2019-10-13T04:59:35.498 INFO:root:teuthology version: 1.0.0-471e0e3 2019-10-13T04:59:35.498 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.OUFD0J.tmp -- --verbose --lock --description fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml} --name pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820 2019-10-13T04:59:35.551 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T04:59:35.620 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820 branch: wip-pdonnell-testing-20191012.025926 description: fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4407820' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-pdonnell-testing-20191012.025926 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd_max_omap_entries_per_request: 10000 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191012.025926 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd suite: fs suite_branch: wip-pdonnell-testing-20191012.025926 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - check-counter: counters: mds: - mds.dir_split - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.4313 2019-10-13T04:59:35.620 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa; will attempt to use it 2019-10-13T04:59:35.621 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks 2019-10-13T04:59:35.621 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-13T04:59:35.623 INFO:teuthology.task.internal:Checking packages... 2019-10-13T04:59:35.635 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd' 2019-10-13T04:59:35.635 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-13T04:59:35.636 INFO:teuthology.packaging:ref: None 2019-10-13T04:59:35.636 INFO:teuthology.packaging:tag: None 2019-10-13T04:59:35.636 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191012.025926 2019-10-13T04:59:35.636 INFO:teuthology.packaging:sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:59:35.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-pdonnell-testing-20191012.025926 2019-10-13T04:59:35.799 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:59:35.800 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-13T04:59:35.807 INFO:teuthology.task.internal:no buildpackages task found 2019-10-13T04:59:35.807 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-13T04:59:35.815 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-13T04:59:35.815 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T04:59:35.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T04:59:45.883 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T04:59:55.901 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:05.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:15.935 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:25.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:35.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:45.987 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:00:56.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:06.008 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:16.031 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:26.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:36.064 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:46.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:01:56.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:06.113 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:16.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:26.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:36.161 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:46.181 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:02:56.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:06.211 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:16.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:26.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:36.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:46.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:03:56.293 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:06.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:16.322 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:26.337 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:36.352 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:46.368 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:04:56.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:05:06.401 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T05:05:16.424 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_pdonnell@teuthology', 'description': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820', 'machine_type': 'smithi'} 2019-10-13T05:05:16.442 DEBUG:teuthology.lock.ops:locked ubuntu@smithi129.front.sepia.ceph.com, ubuntu@smithi071.front.sepia.ceph.com 2019-10-13T05:05:16.581 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T05:05:16.772 INFO:teuthology.provision.fog.smithi129:Scheduling deploy of ubuntu 18.04 2019-10-13T05:05:16.777 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of ubuntu 18.04 2019-10-13T05:05:17.104 INFO:teuthology.orchestra.console:Power off smithi129 2019-10-13T05:05:17.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T05:05:17.160 INFO:teuthology.orchestra.console:Power off smithi071 2019-10-13T05:05:17.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T05:05:17.187 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:17.215 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:21.244 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:21.271 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:25.375 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:25.403 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:25.604 INFO:teuthology.orchestra.console:Power off for smithi129 completed 2019-10-13T05:05:25.705 INFO:teuthology.orchestra.console:Power on smithi129 2019-10-13T05:05:25.705 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T05:05:25.745 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:29.534 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:29.764 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2019-10-13T05:05:29.802 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:29.865 INFO:teuthology.orchestra.console:Power on smithi071 2019-10-13T05:05:29.865 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T05:05:29.905 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:33.938 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:33.969 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T05:05:34.176 INFO:teuthology.orchestra.console:Power on for smithi129 completed 2019-10-13T05:05:34.199 INFO:teuthology.orchestra.console:Power on for smithi071 completed 2019-10-13T05:05:34.277 INFO:teuthology.provision.fog.smithi129:Waiting for deploy to finish 2019-10-13T05:05:34.300 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish 2019-10-13T05:08:20.293 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:29.361 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:38.444 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:47.501 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:50.428 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:56.555 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:08:59.530 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:05.610 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:08.586 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:14.668 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:17.643 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:23.726 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:26.702 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:35.762 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:44.850 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:53.933 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:09:55.260 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T05:10:01.262 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:10:01.273 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T05:10:07.283 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:10:07.293 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T05:10:13.303 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:10:13.314 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T05:10:19.323 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:10:19.334 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T05:10:25.342 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:10:25.447 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:25.447 INFO:teuthology.orchestra.run.smithi129:> hostname 2019-10-13T05:10:25.790 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129.front.sepia.ceph.com 2019-10-13T05:10:25.790 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:25.791 INFO:teuthology.orchestra.run.smithi129:> grep smithi129.front.sepia.ceph.com /etc/hosts 2019-10-13T05:10:25.862 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129 smithi129.front.sepia.ceph.com smithi129 2019-10-13T05:10:25.862 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:25.863 INFO:teuthology.orchestra.run.smithi129:> sudo hostname smithi129 2019-10-13T05:10:26.005 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:26.006 INFO:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/smithi129.front.sepia.ceph.com/smithi129/g' /etc/hosts 2019-10-13T05:10:26.022 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:26.023 INFO:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/172.21.15.129/172.21.15.129/g' /etc/hosts 2019-10-13T05:10:26.107 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:26.108 INFO:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/smithi129.front.sepia.ceph.com/smithi129/g' /etc/hostname 2019-10-13T05:10:26.186 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:10:26.187 INFO:teuthology.orchestra.run.smithi129:> sudo hostname smithi129 2019-10-13T05:10:26.264 INFO:teuthology.provision.fog.smithi129:Deploy complete! 2019-10-13T05:10:26.267 DEBUG:teuthology.parallel:result is None 2019-10-13T05:10:59.954 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:11:00.039 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.039 INFO:teuthology.orchestra.run.smithi071:> hostname 2019-10-13T05:11:00.480 INFO:teuthology.orchestra.run.smithi071.stdout:smithi071.front.sepia.ceph.com 2019-10-13T05:11:00.480 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.480 INFO:teuthology.orchestra.run.smithi071:> grep smithi071.front.sepia.ceph.com /etc/hosts 2019-10-13T05:11:00.552 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71 smithi071.front.sepia.ceph.com smithi071 2019-10-13T05:11:00.552 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.553 INFO:teuthology.orchestra.run.smithi071:> sudo hostname smithi071 2019-10-13T05:11:00.785 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.785 INFO:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/smithi071.front.sepia.ceph.com/smithi071/g' /etc/hosts 2019-10-13T05:11:00.800 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.801 INFO:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/172.21.15.71/172.21.15.71/g' /etc/hosts 2019-10-13T05:11:00.881 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.881 INFO:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/smithi071.front.sepia.ceph.com/smithi071/g' /etc/hostname 2019-10-13T05:11:00.960 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:00.961 INFO:teuthology.orchestra.run.smithi071:> sudo hostname smithi071 2019-10-13T05:11:01.039 INFO:teuthology.provision.fog.smithi071:Deploy complete! 2019-10-13T05:11:01.041 DEBUG:teuthology.parallel:result is None 2019-10-13T05:11:01.223 INFO:teuthology.lock.keys:Checking smithi129.front.sepia.ceph.com 2019-10-13T05:11:01.224 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T05:11:01.236 INFO:teuthology.lock.keys:Checking smithi071.front.sepia.ceph.com 2019-10-13T05:11:01.236 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T05:11:01.249 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: set os type and version on server 2019-10-13T05:11:01.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:11:01.250 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:11:01.335 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:01.335 INFO:teuthology.orchestra.run.smithi129:> true 2019-10-13T05:11:01.559 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:01.560 INFO:teuthology.orchestra.run.smithi129:> uname -m 2019-10-13T05:11:01.630 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2019-10-13T05:11:01.630 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:01.631 INFO:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2019-10-13T05:11:01.701 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2019-10-13T05:11:01.701 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T05:11:01.701 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="18.04" 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T05:11:01.702 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T05:11:01.703 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=bionic 2019-10-13T05:11:01.703 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=bionic 2019-10-13T05:11:01.704 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2019-10-13T05:11:01.717 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: set os type and version on server 2019-10-13T05:11:01.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:11:01.717 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:11:01.801 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:01.801 INFO:teuthology.orchestra.run.smithi071:> true 2019-10-13T05:11:02.039 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:02.039 INFO:teuthology.orchestra.run.smithi071:> uname -m 2019-10-13T05:11:02.109 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2019-10-13T05:11:02.110 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:02.110 INFO:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2019-10-13T05:11:02.183 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Ubuntu" 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:ID=ubuntu 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE=debian 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="18.04" 2019-10-13T05:11:02.184 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T05:11:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T05:11:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T05:11:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T05:11:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_CODENAME=bionic 2019-10-13T05:11:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:UBUNTU_CODENAME=bionic 2019-10-13T05:11:02.186 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2019-10-13T05:11:02.204 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-10-13T05:11:02.234 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMb+Sj+4T/9nm4w1P0FJXLR8/xXnWSHI8tIy559zHUKDvuGlEAk3v9y97kKRV0jlAtxXQKoZhR+0fyJEeOK3PppvH7tUEcqK94JCvGNDoRo+tKwIRrjs3Ux0aQOjB9E41/62sxqby7t7b8K97uwok3Lattw4/cPtCJAov231WLM9Rrve51F1EDQFo5slzsMz9yF92VcHpPUTT2bfpTQUCmx075StocKB+kQXvcKg+NavyePCq1U9udITC6kLPS47g+k6fptrAh6jHNPaFHXQGbfHCWdvqHZyYs52fNVT1iy0Ky0IUvBDTKg2D97KhePQexo/zPsCT3DUpsy+7OTXnR smithi129.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzI9vbs4SRq7J30Hxwo+uim8HbM2JLcX6IOFf0smyWrVynQDgxvjdloZQvh/5F1ADH5eihMnPs+D9i0UL6CldXaMSR3P413KMzfm0TWa3mqA9T1STToiVz1JP/8v4U3FJoQroz3ZfXkBQAdavAe5BnoY0VK0PzCCHF/Vo7FE68n9CM4yO9e/XoBkoV/hV48JdE8ncsiv1ejAMRh9tgEIA1t/MEG548LJSPqMbDg0JZ2QTpg6DzPXd7SRFlb6rUJ5e5BVRSLcTZC0DyB+eKLPDpapeWxVx/4X6/K3VFZ5iuwfa9ZzcB5WIJlsVx4/O0l4R5bxfDAhrI1OHiBDYgYFKr 2019-10-13T05:11:02.234 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T05:11:02.278 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-13T05:11:02.288 INFO:teuthology.task.internal:Saving configuration 2019-10-13T05:11:02.312 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-13T05:11:02.322 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-13T05:11:02.336 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 05:05:16.431721', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi129.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMIN+4Bvm6iM+L6OIhEAqFS6+tfvX0rO5qO7ybS+roEd', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820'} 2019-10-13T05:11:02.348 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 05:05:16.431701', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi071.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIpOWUcxgNsqFUuI4zoUKaGCMnYCUTJE0r3bbhvK7HJU', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820'} 2019-10-13T05:11:02.348 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-13T05:11:02.383 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-10-13T05:11:02.383 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2019-10-13T05:11:02.383 INFO:teuthology.run_tasks:Running task console_log... 2019-10-13T05:11:02.528 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T05:11:02.529 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-13T05:11:02.539 INFO:teuthology.task.internal:Opening connections... 2019-10-13T05:11:02.539 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2019-10-13T05:11:02.541 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:11:02.627 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2019-10-13T05:11:02.627 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:11:02.710 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-13T05:11:02.720 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:02.720 INFO:teuthology.orchestra.run.smithi129:> uname -m 2019-10-13T05:11:02.826 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2019-10-13T05:11:02.826 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:02.827 INFO:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2019-10-13T05:11:02.897 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2019-10-13T05:11:02.897 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T05:11:02.897 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="18.04" 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T05:11:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T05:11:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T05:11:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=bionic 2019-10-13T05:11:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=bionic 2019-10-13T05:11:02.899 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2019-10-13T05:11:02.915 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:02.915 INFO:teuthology.orchestra.run.smithi071:> uname -m 2019-10-13T05:11:02.922 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2019-10-13T05:11:02.922 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:02.923 INFO:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2019-10-13T05:11:02.994 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Ubuntu" 2019-10-13T05:11:02.994 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T05:11:02.994 INFO:teuthology.orchestra.run.smithi071.stdout:ID=ubuntu 2019-10-13T05:11:02.995 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE=debian 2019-10-13T05:11:02.995 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T05:11:02.995 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="18.04" 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_CODENAME=bionic 2019-10-13T05:11:02.996 INFO:teuthology.orchestra.run.smithi071.stdout:UBUNTU_CODENAME=bionic 2019-10-13T05:11:02.997 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2019-10-13T05:11:03.015 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-13T05:11:03.032 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-13T05:11:03.042 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-13T05:11:03.042 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:03.042 INFO:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2019-10-13T05:11:03.046 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:03.046 INFO:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2019-10-13T05:11:03.052 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-13T05:11:03.084 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-13T05:11:03.084 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:03.085 INFO:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2019-10-13T05:11:03.096 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:03.096 INFO:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2019-10-13T05:11:03.304 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-13T05:11:03.342 INFO:teuthology.run_tasks:Running task kernel... 2019-10-13T05:11:03.360 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'mds.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-10-13T05:11:03.360 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-13T05:11:03.361 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:03.363 INFO:teuthology.orchestra.run.smithi129:> uname -r 2019-10-13T05:11:03.370 INFO:teuthology.orchestra.run.smithi129.stdout:4.15.0-58-generic 2019-10-13T05:11:03.371 INFO:teuthology.task.kernel:Running kernel on smithi129: 4.15.0-58-generic 2019-10-13T05:11:03.371 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:03.371 INFO:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2019-10-13T05:11:03.506 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:03.506 INFO:teuthology.orchestra.run.smithi129:> sudo apt-get update 2019-10-13T05:11:03.764 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-13T05:11:03.787 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:11:03.925 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-13T05:11:04.246 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB] 2019-10-13T05:11:04.282 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-13T05:11:04.626 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [591 kB] 2019-10-13T05:11:04.822 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [529 kB] 2019-10-13T05:11:05.237 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [752 kB] 2019-10-13T05:11:05.414 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [177 kB] 2019-10-13T05:11:05.880 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-13T05:11:05.881 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [270 kB] 2019-10-13T05:11:06.038 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-13T05:11:06.199 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-13T05:11:06.392 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-13T05:11:06.633 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-13T05:11:06.794 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-13T05:11:06.815 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-13T05:11:07.420 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-13T05:11:07.471 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-13T05:11:07.905 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-13T05:11:08.067 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-13T05:11:08.145 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-13T05:11:08.229 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-13T05:11:08.689 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-13T05:11:08.845 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-13T05:11:09.015 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-13T05:11:09.167 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-13T05:11:09.329 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-13T05:11:09.490 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-13T05:11:09.618 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 6,739 kB in 6s (1,143 kB/s) 2019-10-13T05:11:10.915 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:11:10.961 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:10.961 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T05:11:11.043 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:11:11.291 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:11:11.291 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:11:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T05:11:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T05:11:11.370 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:11.370 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T05:11:11.450 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:11:11.716 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:11:11.716 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:11:11.841 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2019-10-13T05:11:11.841 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T05:11:11.841 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-13T05:11:11.842 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-13T05:11:11.842 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2019-10-13T05:11:11.842 INFO:teuthology.orchestra.run.smithi129.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-13T05:11:11.843 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2019-10-13T05:11:11.843 INFO:teuthology.orchestra.run.smithi129.stdout: thermald 2019-10-13T05:11:11.878 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2019-10-13T05:11:11.878 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T05:11:11.878 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-13T05:11:11.878 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-13T05:11:11.880 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2019-10-13T05:11:11.881 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-13T05:11:12.054 INFO:teuthology.orchestra.run.smithi129.stdout:3 upgraded, 5 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T05:11:12.054 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 66.3 MB of archives. 2019-10-13T05:11:12.055 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-13T05:11:12.055 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-13T05:11:13.147 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-13T05:11:14.126 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-13T05:11:15.792 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-13T05:11:15.954 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-13T05:11:16.116 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-13T05:11:17.171 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-13T05:11:17.881 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-13T05:11:18.284 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 66.3 MB in 6s (11.0 MB/s) 2019-10-13T05:11:18.441 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-13T05:11:18.476 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-13T05:11:18.479 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:11:18.529 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:20.180 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-13T05:11:20.194 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:11:20.287 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:20.775 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-13T05:11:20.788 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:11:20.825 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:24.688 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:11:24.808 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:11:25.270 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:11:25.598 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-13T05:11:25.612 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-13T05:11:25.657 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T05:11:30.415 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-13T05:11:30.435 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:11:30.473 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:32.842 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:11:32.962 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:11:33.279 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T05:11:33.393 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:33.917 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:34.017 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-13T05:11:34.152 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:34.594 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:11:34.594 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-13T05:11:34.737 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:35.264 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-13T05:11:35.390 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-13T05:11:35.516 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:11:35.693 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-13T05:11:35.694 INFO:teuthology.orchestra.run.smithi129.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:11:49.580 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-13T05:11:49.580 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub' 2019-10-13T05:11:49.608 INFO:teuthology.orchestra.run.smithi129.stdout:Generating grub configuration file ... 2019-10-13T05:11:49.918 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:11:49.945 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:11:50.257 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T05:11:50.288 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T05:11:51.235 INFO:teuthology.orchestra.run.smithi129.stdout:done 2019-10-13T05:11:51.616 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:11:51.616 INFO:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2019-10-13T05:11:51.635 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2019-10-13T05:11:51.635 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2019-10-13T05:11:51.635 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2019-10-13T05:11:51.635 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2019-10-13T05:11:51.636 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 15 2019-10-13T05:11:51.636 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T05:11:51.636 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2019-10-13T05:11:51.636 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2019-10-13T05:11:51.636 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 4.15.0.65.67 2019-10-13T05:11:51.637 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T05:11:51.637 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2019-10-13T05:11:51.637 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2019-10-13T05:11:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2019-10-13T05:11:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2019-10-13T05:11:51.637 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-13T05:11:51.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:51.638 INFO:teuthology.orchestra.run.smithi071:> uname -r 2019-10-13T05:11:51.645 INFO:teuthology.orchestra.run.smithi071.stdout:4.15.0-58-generic 2019-10-13T05:11:51.645 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.15.0-58-generic 2019-10-13T05:11:51.646 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:51.646 INFO:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2019-10-13T05:11:51.865 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:51.865 INFO:teuthology.orchestra.run.smithi071:> sudo apt-get update 2019-10-13T05:11:52.144 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:11:52.144 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-13T05:11:52.284 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-13T05:11:52.668 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [529 kB] 2019-10-13T05:11:52.681 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-13T05:11:53.077 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [752 kB] 2019-10-13T05:11:53.291 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB] 2019-10-13T05:11:53.729 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [591 kB] 2019-10-13T05:11:53.849 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [177 kB] 2019-10-13T05:11:54.295 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-13T05:11:54.360 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [270 kB] 2019-10-13T05:11:54.457 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-13T05:11:54.618 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-13T05:11:54.903 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-13T05:11:55.149 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-13T05:11:55.230 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-13T05:11:55.324 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-13T05:11:55.835 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-13T05:11:56.001 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-13T05:11:56.316 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-13T05:11:56.479 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-13T05:11:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-13T05:11:56.683 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-13T05:11:57.231 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-13T05:11:57.419 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-13T05:11:57.598 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-13T05:11:57.771 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-13T05:11:57.939 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-13T05:11:58.107 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-13T05:11:58.230 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 6,739 kB in 6s (1,094 kB/s) 2019-10-13T05:11:59.527 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:11:59.571 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:59.571 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T05:11:59.651 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:11:59.890 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:11:59.890 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:11:59.961 INFO:teuthology.orchestra.run.smithi071.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T05:11:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T05:11:59.963 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:11:59.963 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T05:12:00.048 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:12:00.277 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:12:00.278 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:12:00.433 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2019-10-13T05:12:00.433 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T05:12:00.433 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-13T05:12:00.433 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-13T05:12:00.434 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2019-10-13T05:12:00.435 INFO:teuthology.orchestra.run.smithi071.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-13T05:12:00.435 INFO:teuthology.orchestra.run.smithi071.stdout:Recommended packages: 2019-10-13T05:12:00.435 INFO:teuthology.orchestra.run.smithi071.stdout: thermald 2019-10-13T05:12:00.464 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2019-10-13T05:12:00.464 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T05:12:00.465 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-13T05:12:00.465 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-13T05:12:00.466 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages will be upgraded: 2019-10-13T05:12:00.466 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-13T05:12:00.664 INFO:teuthology.orchestra.run.smithi071.stdout:3 upgraded, 5 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T05:12:00.665 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 66.3 MB of archives. 2019-10-13T05:12:00.665 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-13T05:12:00.665 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-13T05:12:01.762 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-13T05:12:02.699 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-13T05:12:04.306 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-13T05:12:04.468 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-13T05:12:04.629 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-13T05:12:05.643 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-13T05:12:06.342 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-13T05:12:06.764 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 66.3 MB in 6s (11.3 MB/s) 2019-10-13T05:12:06.904 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-13T05:12:06.964 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-13T05:12:06.965 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:12:07.001 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:08.663 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-13T05:12:08.677 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:12:08.791 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:09.315 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-13T05:12:09.327 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:12:09.365 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:13.187 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:12:13.307 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:12:13.632 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:12:13.769 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:12:14.072 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-13T05:12:14.085 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-13T05:12:14.164 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T05:12:18.896 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-13T05:12:18.916 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T05:12:18.945 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:21.286 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T05:12:21.410 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T05:12:21.742 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T05:12:21.839 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:22.358 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:22.485 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-13T05:12:22.611 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:23.037 INFO:teuthology.orchestra.run.smithi071.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:12:23.037 INFO:teuthology.orchestra.run.smithi071.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-13T05:12:23.171 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:23.631 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-13T05:12:23.757 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-13T05:12:23.884 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T05:12:24.064 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-13T05:12:24.064 INFO:teuthology.orchestra.run.smithi071.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:12:37.768 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-13T05:12:37.768 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub' 2019-10-13T05:12:37.802 INFO:teuthology.orchestra.run.smithi071.stdout:Generating grub configuration file ... 2019-10-13T05:12:38.119 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:12:38.148 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:12:38.467 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T05:12:38.498 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T05:12:39.453 INFO:teuthology.orchestra.run.smithi071.stdout:done 2019-10-13T05:12:39.838 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:39.838 INFO:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2019-10-13T05:12:39.859 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2019-10-13T05:12:39.859 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2019-10-13T05:12:39.859 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2019-10-13T05:12:39.859 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 15 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 4.15.0.65.67 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T05:12:39.860 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2019-10-13T05:12:39.861 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2019-10-13T05:12:39.861 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2019-10-13T05:12:39.861 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2019-10-13T05:12:39.861 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-13T05:12:39.861 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-10-13T05:12:39.861 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-13T05:12:39.862 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-13T05:12:39.862 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-10-13T05:12:39.862 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2019-10-13T05:12:39.862 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:39.862 INFO:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2019-10-13T05:12:39.931 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:39.931 INFO:teuthology.orchestra.run.smithi129:> sudo apt-get update 2019-10-13T05:12:40.191 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T05:12:40.198 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:12:40.359 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T05:12:40.520 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T05:12:41.921 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:12:41.963 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:41.963 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T05:12:42.037 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:12:42.267 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:12:42.268 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:12:42.340 INFO:teuthology.orchestra.run.smithi129.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T05:12:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T05:12:42.341 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:42.342 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T05:12:42.415 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:12:42.637 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:12:42.638 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:12:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-13T05:12:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 0 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T05:12:42.818 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:42.819 INFO:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2019-10-13T05:12:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2019-10-13T05:12:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 15 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2019-10-13T05:12:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 4.15.0.65.67 2019-10-13T05:12:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T05:12:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2019-10-13T05:12:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2019-10-13T05:12:42.845 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2019-10-13T05:12:42.845 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2019-10-13T05:12:42.846 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:42.846 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:12:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.Alc9209Amj 2019-10-13T05:12:42.916 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:42.916 INFO:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Alc9209Amj 2019-10-13T05:12:42.996 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:42.996 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.Alc9209Amj 2019-10-13T05:12:43.160 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.Alc9209Amj is 8KB 2019-10-13T05:12:43.172 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:43.172 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.Alc9209Amj 2019-10-13T05:12:43.179 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:43.179 INFO:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-13T05:12:43.263 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:43.263 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-13T05:12:43.395 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-13T05:12:43.395 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:43.395 INFO:teuthology.orchestra.run.smithi129:> sudo update-grub 2019-10-13T05:12:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub' 2019-10-13T05:12:44.608 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2019-10-13T05:12:44.861 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:12:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:12:45.153 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T05:12:45.165 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T05:12:45.759 INFO:teuthology.orchestra.run.smithi129.stderr:done 2019-10-13T05:12:45.761 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:12:45.761 INFO:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2019-10-13T05:12:45.764 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-10-13T05:12:45.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2019-10-13T05:12:45.765 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:45.765 INFO:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2019-10-13T05:12:45.815 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:45.815 INFO:teuthology.orchestra.run.smithi071:> sudo apt-get update 2019-10-13T05:12:45.939 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T05:12:46.057 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:12:46.215 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T05:12:46.376 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T05:12:47.712 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:12:47.757 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:47.758 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T05:12:47.835 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:12:48.062 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:12:48.062 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:12:48.139 INFO:teuthology.orchestra.run.smithi071.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T05:12:48.141 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T05:12:48.141 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.142 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T05:12:48.214 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:12:48.438 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:12:48.438 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:12:48.628 INFO:teuthology.orchestra.run.smithi071.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-13T05:12:48.628 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 0 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T05:12:48.630 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.630 INFO:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2019-10-13T05:12:48.646 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 15 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2019-10-13T05:12:48.647 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 4.15.0.65.67 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2019-10-13T05:12:48.648 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2019-10-13T05:12:48.649 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.649 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:12:48.718 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.P38vu8WjkG 2019-10-13T05:12:48.719 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.719 INFO:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub/grub.cfg /tmp/tmp.P38vu8WjkG 2019-10-13T05:12:48.802 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.802 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.P38vu8WjkG 2019-10-13T05:12:48.959 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.P38vu8WjkG is 8KB 2019-10-13T05:12:48.971 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.971 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.P38vu8WjkG 2019-10-13T05:12:48.978 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:48.979 INFO:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-13T05:12:49.058 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:49.079 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-13T05:12:49.171 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-13T05:12:49.171 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:49.171 INFO:teuthology.orchestra.run.smithi071:> sudo update-grub 2019-10-13T05:12:50.392 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub' 2019-10-13T05:12:50.408 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2019-10-13T05:12:50.673 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T05:12:50.683 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T05:12:50.955 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T05:12:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T05:12:51.548 INFO:teuthology.orchestra.run.smithi071.stderr:done 2019-10-13T05:12:51.550 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:12:51.550 INFO:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2019-10-13T05:12:51.554 INFO:teuthology.misc:Re-opening connections... 2019-10-13T05:12:51.554 INFO:teuthology.misc:trying to connect to ubuntu@smithi129.front.sepia.ceph.com 2019-10-13T05:12:51.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:12:51.554 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:13:51.556 DEBUG:teuthology.orchestra.remote:timed out 2019-10-13T05:13:51.556 INFO:teuthology.misc:trying to connect to ubuntu@smithi071.front.sepia.ceph.com 2019-10-13T05:13:51.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:13:51.557 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:14:51.558 DEBUG:teuthology.orchestra.remote:timed out 2019-10-13T05:14:51.559 DEBUG:teuthology.misc:waited 120.005068064 2019-10-13T05:14:52.559 INFO:teuthology.misc:trying to connect to ubuntu@smithi129.front.sepia.ceph.com 2019-10-13T05:14:52.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:14:52.560 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:14:52.880 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:52.881 INFO:teuthology.orchestra.run.smithi129:> true 2019-10-13T05:14:53.161 DEBUG:teuthology.misc:waited 121.607486963 2019-10-13T05:14:54.162 INFO:teuthology.misc:trying to connect to ubuntu@smithi071.front.sepia.ceph.com 2019-10-13T05:14:54.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:14:54.163 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:14:54.249 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:54.249 INFO:teuthology.orchestra.run.smithi071:> true 2019-10-13T05:14:54.555 DEBUG:teuthology.misc:waited 123.001758099 2019-10-13T05:14:55.556 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-10-13T05:14:55.556 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-65-generic"... 2019-10-13T05:14:55.556 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:55.557 INFO:teuthology.orchestra.run.smithi129:> uname -r 2019-10-13T05:14:55.564 INFO:teuthology.orchestra.run.smithi129.stdout:4.15.0-65-generic 2019-10-13T05:14:55.564 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-13T05:14:55.564 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-13T05:14:55.564 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-10-13T05:14:55.564 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-65-generic"... 2019-10-13T05:14:55.565 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:55.565 INFO:teuthology.orchestra.run.smithi071:> uname -r 2019-10-13T05:14:55.572 INFO:teuthology.orchestra.run.smithi071.stdout:4.15.0-65-generic 2019-10-13T05:14:55.572 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-13T05:14:55.572 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-13T05:14:56.573 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-10-13T05:14:56.574 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:56.574 INFO:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T05:14:56.679 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2019-10-13T05:14:56.705 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-10-13T05:14:56.705 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:56.705 INFO:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T05:14:56.728 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2019-10-13T05:14:56.751 INFO:teuthology.run_tasks:Running task internal.base... 2019-10-13T05:14:56.761 INFO:teuthology.task.internal:Creating test directory... 2019-10-13T05:14:56.761 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:56.761 INFO:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T05:14:56.795 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:56.796 INFO:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T05:14:56.803 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-10-13T05:14:56.819 INFO:teuthology.run_tasks:Running task internal.archive... 2019-10-13T05:14:56.835 INFO:teuthology.task.internal:Creating archive directory... 2019-10-13T05:14:56.836 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:56.836 INFO:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T05:14:56.846 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:56.846 INFO:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T05:14:56.870 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-10-13T05:14:56.882 INFO:teuthology.task.internal:Enabling coredump saving... 2019-10-13T05:14:56.882 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:56.882 INFO:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T05:14:56.901 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:56.901 INFO:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T05:14:56.921 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T05:14:56.942 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T05:14:56.942 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-10-13T05:14:56.969 INFO:teuthology.task.internal:Configuring sudo... 2019-10-13T05:14:56.969 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:56.969 INFO:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T05:14:56.972 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:56.973 INFO:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T05:14:57.007 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-10-13T05:14:57.021 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-10-13T05:14:57.021 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:57.021 INFO:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T05:14:57.031 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:57.032 INFO:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T05:14:57.071 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:57.071 INFO:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T05:14:57.142 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:57.142 INFO:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T05:14:57.215 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:57.215 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T05:14:57.325 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:57.325 INFO:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T05:14:57.333 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:57.334 INFO:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T05:14:57.411 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:57.411 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T05:14:57.524 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:14:57.524 INFO:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2019-10-13T05:14:57.592 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:14:57.592 INFO:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2019-10-13T05:14:57.645 INFO:teuthology.run_tasks:Running task internal.timer... 2019-10-13T05:14:57.664 INFO:teuthology.task.internal:Starting timer... 2019-10-13T05:14:57.664 INFO:teuthology.run_tasks:Running task pcp... 2019-10-13T05:14:57.702 INFO:teuthology.run_tasks:Running task selinux... 2019-10-13T05:14:57.716 DEBUG:teuthology.task.selinux:Excluding smithi129: OS 'ubuntu' does not support SELinux 2019-10-13T05:14:57.716 DEBUG:teuthology.task.selinux:Excluding smithi071: OS 'ubuntu' does not support SELinux 2019-10-13T05:14:57.716 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-10-13T05:14:57.716 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2019-10-13T05:14:57.716 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-10-13T05:14:57.717 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-10-13T05:14:57.732 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-10-13T05:14:57.756 INFO:teuthology.repo_utils:Fetching master from upstream 2019-10-13T05:14:57.858 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch master 2019-10-13T05:14:57.891 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'set_python_path.yml'}, {'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-10-13T05:14:57.894 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-10-13T05:14:58.300 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-10-13T05:15:02.099 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T05:15:02.122 INFO:teuthology.task.ansible.out: TASK [Check for /usr/bin/python] *********************************************** 2019-10-13T05:15:02.124 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:02 +0000 (0:00:00.073) 0:00:00.073 ******** 2019-10-13T05:15:03.794 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi071.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:03.795 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.003371", "end": "2019-10-13 05:15:04.073573", "rc": 0, "start": "2019-10-13 05:15:04.070202", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-13T05:15:04.040 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi129.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.002584", "end": "2019-10-13 05:15:04.282756", "rc": 0, "start": "2019-10-13 05:15:04.280172", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-13T05:15:04.044 INFO:teuthology.task.ansible.out: TASK [Set ansible_python_interpreter=/usr/bin/python3] ************************* Sunday 13 October 2019 05:15:04 +0000 (0:00:01.921) 0:00:01.995 ******** 2019-10-13T05:15:04.083 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:04.094 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:04.102 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T05:15:04.112 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:04 +0000 (0:00:00.067) 0:00:02.062 ******** 2019-10-13T05:15:04.126 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:04 +0000 (0:00:00.013) 0:00:02.076 ******** 2019-10-13T05:15:05.349 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T05:15:05.357 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.129"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:7261"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570943705", "hour": "05", "iso8601": "2019-10-13T05:15:05Z", "iso8601_basic": "20191013T051505541818", "iso8601_basic_short": "20191013T051505", "iso8601_micro": "2019-10-13T05:15:05.541901Z", "minute": "15", "month": "10", "second": "05", "time": "05:15:05", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.129", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:72:61", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:7261", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:72:61", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67h 2019-10-13T05:15:05.359 INFO:teuthology.task.ansible.out:Cynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "nvme0n1": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["25096892-22de-44c3-a204-057f9b3d312e"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "labels": [], "masters": [], "uuids": ["25096892-22de-44c3-a204-057f9b3d312e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB 2019-10-13T05:15:05.361 INFO:teuthology.task.ansible.out:", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.0 2019-10-13T05:15:05.362 INFO:teuthology.task.ansible.out:0 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contro 2019-10-13T05:15:05.364 INFO:teuthology.task.ansible.out:ller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGTA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292d832"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "an 2019-10-13T05:15:05.366 INFO:teuthology.task.ansible.out:sible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:19:38", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "pr 2019-10-13T05:15:05.367 INFO:teuthology.task.ansible.out:omisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ip 2019-10-13T05:15:05.369 INFO:teuthology.task.ansible.out:xip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:19:39", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ip 2019-10-13T05:15:05.371 INFO:teuthology.task.ansible.out:xip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:72:60", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx 2019-10-13T05:15:05.372 INFO:teuthology.task.ansible.out:_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.129", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:7261", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:72:61", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-hoghcavcshjkvodpkodtmqdtpnruathx ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi129.front.sepia.ceph.com", "ansible_hostname": "smithi129", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "enp3s0f0", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-gener 2019-10-13T05:15:05.374 INFO:teuthology.task.ansible.out:ic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, 2019-10-13T05:15:05.375 INFO:teuthology.task.ansible.out:"ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31473, "ansible_memory_mb": {"nocache": {"free": 31781, "used": 274}, "real": {"free": 31473, "total": 32055, "used": 582}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226440804, "block_size": 4096, "block_total": 240095108, "block_used": 13654304, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501533184, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi129.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel( 2019-10-13T05:15:05.377 INFO:teuthology.task.ansible.out:R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6901895", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD91938", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAKIJEYTYNRb2k+ZdAndnNZuz/BqWEFrEkFDfJAouZdJq0ZjmFXODRZH7yrTX2bVfJWXqryXuPEdah2uV/l1ATdGsHqF9TVj5K4ztht8Iuic/x8VKs10zgMjgHjdUy9pmuAssly9dkO86eM6SZN4Sc7SrYx0XJnATBXv+nYPOmskhAAAAFQDPkHJIOaC2vsQVqdsp4hzJhT5TxQAAAIA6e0tAYEyRDxla9e/CpejqN2yviY3KI3yt6AtZU6T6LjWY+pTKGzyxeIzQCEZ39ysNVuVmABFTfHu7m9Hc3LJFTUDmzk2+SGRJzxUgt+ChGIWnqXn8NSo8NLiYX96FfXLGUvtm+F9GlNOfiWPes7Kjxrqwby944mpJzO99UVW/KwAAAIAEbgHVwg6zLXXKwgS7pkPubWYmZb/wQ87aZPggWZxWNFeS8rTiWARK5+hDLnoolYCyrGt1aBHu4OB/aU8Fzh/on7BRSIferbXUdqpVqsM/yyVMOsjGr30IyWqpmcMPiGmU9UJFC9YTe/NxyrHdWuOQVTquGfth4UA5v8thMLfENw==", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZD4nDdt/ixAC5j8RKxN1EeHnhwbTF2p0obb/UpHzn5ROlTiT12dq/8dpn5rOAlUjChw+IKiKLImqErqHrWkfw=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZD 2019-10-13T05:15:05.378 INFO:teuthology.task.ansible.out:I1NTE5AAAAIMIN+4Bvm6iM+L6OIhEAqFS6+tfvX0rO5qO7ybS+roEd", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCzI9vbs4SRq7J30Hxwo+uim8HbM2JLcX6IOFf0smyWrVynQDgxvjdloZQvh/5F1ADH5eihMnPs+D9i0UL6CldXaMSR3P413KMzfm0TWa3mqA9T1STToiVz1JP/8v4U3FJoQroz3ZfXkBQAdavAe5BnoY0VK0PzCCHF/Vo 2019-10-13T05:15:05.380 INFO:teuthology.task.ansible.out:7FE68n9CM4yO9e/XoBkoV/hV48JdE8ncsiv1ejAMRh9tgEIA1t/MEG548LJSPqMbDg0JZ2QTpg6DzPXd7SRFlb6rUJ5e5BVRSLcTZC0DyB+eKLPDpapeWxVx/4X6/K3VFZ5iuwfa9ZzcB5WIJlsVx4/O0l4R5bxfDAhrI1OHiBDYgYFKr", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 58, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T05:15:05.399 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.71"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:d765"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570943705", "hour": "05", "iso8601": "2019-10-13T05:15:05Z", "iso8601_basic": "20191013T051505593228", "iso8601_basic_short": "20191013T051505", "iso8601_micro": "2019-10-13T05:15:05.593318Z", "minute": "15", "month": "10", "second": "05", "time": "05:15:05", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.71", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:d7:65", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:d765", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:d7:65", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCA 2019-10-13T05:15:05.413 INFO:teuthology.task.ansible.out:xKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "nvme0n1": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "labels": [], "masters": [], "uuids": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", 2019-10-13T05:15:05.416 INFO:teuthology.task.ansible.out: "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 2019-10-13T05:15:05.417 INFO:teuthology.task.ansible.out:Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controll 2019-10-13T05:15:05.419 INFO:teuthology.task.ansible.out:er: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansi 2019-10-13T05:15:05.421 INFO:teuthology.task.ansible.out:ble_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:aa:4a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "prom 2019-10-13T05:15:05.422 INFO:teuthology.task.ansible.out:isc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxi 2019-10-13T05:15:05.424 INFO:teuthology.task.ansible.out:p4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:aa:4b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxi 2019-10-13T05:15:05.425 INFO:teuthology.task.ansible.out:p4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:d7:64", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_g 2019-10-13T05:15:05.427 INFO:teuthology.task.ansible.out:so_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.71", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:d765", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8f:d7:65", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-gmwjhsymokmdcvhbdsrpvvfwluvktwit ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi071.front.sepia.ceph.com", "ansible_hostname": "smithi071", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "enp3s0f0", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic" 2019-10-13T05:15:05.429 INFO:teuthology.task.ansible.out:, "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcas 2019-10-13T05:15:05.431 INFO:teuthology.task.ansible.out:t": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31473, "ansible_memory_mb": {"nocache": {"free": 31782, "used": 273}, "real": {"free": 31473, "total": 32055, "used": 582}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226440811, "block_size": 4096, "block_total": 240095108, "block_used": 13654297, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501561856, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi071.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xe 2019-10-13T05:15:05.432 INFO:teuthology.task.ansible.out:on(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731590", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8AA4A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAJK7vWCJ8jjhW3mOnzg46tgl/mOJhHNCaWJpv3Nf1PPYKIa6gMH/ATBf1AwhwwiBK+X/N8mFCk+ZvvDDizXns/vSYuBRj/yiWABZdHQyJBg/6ESNQctpUzW2Kk3TCs/fMoSudTpVT+XNfqkeTHJe5LzyrnLZjlIE14c7x6/UlZhFAAAAFQDQbV9z2rmLAQd+cv17ABzbj6OjnQAAAIBgT68xWQEOcWSb6JuUS7LMZVUB/tNy17rjbnIHAX4x5paKIFQXzaCtJFAqwifmtQVkq9U8AczFcbkNv0/Mgg4dClAEfk7J6mMmxW9DlV638Ijvr0Bl+YAyo43BuRkgCSOPsKa8W91wfYBNU2kapJDiEf3T9TddiPG4f8iZ/4CQtQAAAIAEJP6Y2cmeBk5Y4rFaZ0WmCyNWhVX3MnxXz1qHyoHAvfl2NPum+3GoX0mB/5RTQXcCsNnI4efn+b5FcfX6fSZRHYS09M2Nni2IZLjvMXH/oi545wnZVb8aZDAiFj6FhXPLlrti7CVMBen7FQQXnB68730A8lfkvNctSQHNBJV0GQ==", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICYRzS7DLsfIeQJ6jyVgqQWPlDHJxuRvLP84djVbr2XPdYDhiuiGyXEsDlQiC07MLZoMjn6tNZOuiDzhSpW 2019-10-13T05:15:05.434 INFO:teuthology.task.ansible.out:HjU=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIIpOWUcxgNsqFUuI4zoUKaGCMnYCUTJE0r3bbhvK7HJU", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDMb+Sj+4T/9nm4w1P0FJXLR8/xXnWSHI8tIy559zHUKDvuGlEAk3v9y97kKRV0jlAtxXQKoZhR+0fyJEeOK3PppvH7tUEcqK94JCvGNDoRo+tKwIRrjs3Ux0aQOjB9E41/62sxqby7t7b8K97uwok3Lattw4/cPtCJAov231WLM9Rrve51F1EDQFo5slzsMz9yF92VcHpPUTT2bfpTQUCmx075StocKB+kQXvcKg+NavyePCq1U9udITC6kLPS47g+k6fptrAh6jHNPaFHXQGbfHCWdvqHZyYs52fNVT1iy0Ky0IUvBDTKg2D97KhePQexo/zPsCT3DUpsy+7OTXnR", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 39, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T05:15:05.538 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:05 +0000 (0:00:01.411) 0:00:03.488 ******** 2019-10-13T05:15:05.573 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:05 +0000 (0:00:00.035) 0:00:03.524 ******** 2019-10-13T05:15:05.828 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2019-10-13T05:15:05.830 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T05:15:05.870 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:05 +0000 (0:00:00.296) 0:00:03.820 ******** 2019-10-13T05:15:05.904 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:05 +0000 (0:00:00.033) 0:00:03.854 ******** 2019-10-13T05:15:06.334 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-10-13T05:15:06.336 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T05:15:06.375 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:06 +0000 (0:00:00.470) 0:00:04.325 ******** 2019-10-13T05:15:06.390 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T05:15:06.395 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:06.430 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:06 +0000 (0:00:00.054) 0:00:04.379 ******** 2019-10-13T05:15:06.458 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:06.581 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.057350", "end": "2019-10-13 05:15:06.840646", "rc": 0, "start": "2019-10-13 05:15:06.783296", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-13T05:15:06.627 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:06 +0000 (0:00:00.197) 0:00:04.577 ******** 2019-10-13T05:15:06.654 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.071900", "end": "2019-10-13 05:15:06.905987", "rc": 0, "start": "2019-10-13 05:15:06.834087", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-13T05:15:06.702 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:06 +0000 (0:00:00.074) 0:00:04.652 ******** 2019-10-13T05:15:06.925 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-10-13T05:15:06.927 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:15:06.979 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:06 +0000 (0:00:00.276) 0:00:04.929 ******** 2019-10-13T05:15:07.024 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:07 +0000 (0:00:00.040) 0:00:04.970 ******** 2019-10-13T05:15:07.868 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-10-13T05:15:07.870 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570943706.99-200276677774773/source", "state": "file", "uid": 0} changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570943707.04-269165240630949/source", "state": "file", "uid": 0} 2019-10-13T05:15:07.919 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:07 +0000 (0:00:00.899) 0:00:05.869 ******** 2019-10-13T05:15:07.956 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:07 +0000 (0:00:00.036) 0:00:05.906 ******** 2019-10-13T05:15:08.272 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-10-13T05:15:08.273 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "change 2019-10-13T05:15:08.275 INFO:teuthology.task.ansible.out:d": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.401 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.402 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.537 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.538 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.664 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.665 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.796 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.807 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.921 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:08.944 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:09.066 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:09.114 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T05:15:09.253 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-10-13T05:15:09.258 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-10-13T05:15:09.486 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:09 +0000 (0:00:01.530) 0:00:07.436 ******** 2019-10-13T05:15:09.529 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:09 +0000 (0:00:00.042) 0:00:07.479 ******** 2019-10-13T05:15:10.228 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T05:15:10.249 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.129"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:7261"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570943710", "hour": "05", "iso8601": "2019-10-13T05:15:10Z", "iso8601_basic": "20191013T051510421852", "iso8601_basic_short": "20191013T051510", "iso8601_micro": "2019-10-13T05:15:10.421933Z", "minute": "15", "month": "10", "second": "10", "time": "05:15:10", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.129", "alias": "enp3s0f1", "broadcast" 2019-10-13T05:15:10.250 INFO:teuthology.task.ansible.out:: "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:72:61", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:7261", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:72:61", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "nvme0n1": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["25096892-22de-44c3-a204-057f9b3d312e"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "labels": [], "masters": [], "uuids": ["25096892-22de-44c3-a204-057f9b3d312e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "3 2019-10-13T05:15:10.252 INFO:teuthology.task.ansible.out:1252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", 2019-10-13T05:15:10.253 INFO:teuthology.task.ansible.out: "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "non 2019-10-13T05:15:10.254 INFO:teuthology.task.ansible.out:e", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"h 2019-10-13T05:15:10.256 INFO:teuthology.task.ansible.out:olders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGTA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292d832"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_ch 2019-10-13T05:15:10.257 INFO:teuthology.task.ansible.out:ecksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:19:38", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx 2019-10-13T05:15:10.259 INFO:teuthology.task.ansible.out:_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:19:39", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "t 2019-10-13T05:15:10.260 INFO:teuthology.task.ansible.out:x_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:72:60", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_c 2019-10-13T05:15:10.262 INFO:teuthology.task.ansible.out:hecksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.129", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:7261", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:72:61", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-mddufcbvjicwnvhdyukdsaqnooyabzzi ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "an 2019-10-13T05:15:10.263 INFO:teuthology.task.ansible.out:sible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi129.front.sepia.ceph.com", "ansible_hostname": "smithi129", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "enp3s0f0", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_c 2019-10-13T05:15:10.264 INFO:teuthology.task.ansible.out:sum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broad 2019-10-13T05:15:10.266 INFO:teuthology.task.ansible.out:cast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31467, "ansible_memory_mb": {"nocache": {"free": 31781, "used": 274}, "real": {"free": 31467, "total": 32055, "used": 588}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi129.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) 2019-10-13T05:15:10.267 INFO:teuthology.task.ansible.out: Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6901895", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD91938", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAKIJEYTYNRb2k+ZdAndnNZuz/BqWEFrEkFDfJAouZdJq0ZjmFXODRZH7yrTX2bVfJWXqryXuPEdah2uV/l1ATdGsHqF9TVj5K4ztht8Iuic/x8VKs10zgMjgHjdUy9pmuAssly9dkO86eM6SZN4Sc7SrYx0XJnATBXv+nYPOmskhAAAAFQDPkHJIOaC2vsQVqdsp4hzJhT5TxQAAAIA6e0tAYEyRDxla9e/CpejqN2yviY3KI3yt6AtZU6T6LjWY+pTKGzyxeIzQCEZ39ysNVuVmABFTfHu7m9Hc3LJFTUDmzk2+SGRJzxUgt+ChGIWnqXn8NSo8NLiYX96FfXLGUvtm+F9GlNOfiWPes7Kjxrqwby944mpJzO99UVW/KwAAAIAEbgHVwg6zLXXKwgS7pkPubWYmZb/wQ87aZPggWZxWNFeS8rTiWARK5+hDLnoolYCyrGt1aBHu4OB/aU8Fzh/on7BRSIferbXUdqpVqsM/yyVMOsjGr30IyWqpmcMPiGmU9UJFC9YTe/NxyrHdWuOQVTquGfth4UA5v8thMLfENw==", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZD4nDdt/ixAC5j8RKxN1EeHnhwbTF2p0obb/UpHzn5ROlTiT12dq/8dpn5rOAlUjChw+IKiKLImqErq 2019-10-13T05:15:10.269 INFO:teuthology.task.ansible.out:HrWkfw=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIMIN+4Bvm6iM+L6OIhEAqFS6+tfvX0rO5qO7ybS+roEd", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCzI9vbs4SRq7J30Hxwo+uim8HbM2JLcX6IOFf0smyWrVynQDgxvjdloZQvh/5F1ADH5eihMnPs+D9i0UL6CldXaMSR3P413KMzfm0TWa3mqA9T1STToiVz1JP/8v4U3FJoQroz3ZfXkBQAdavAe5BnoY0VK0PzCCHF/Vo7FE68n9CM4yO9e/XoBkoV/hV48JdE8ncsiv1ejAMRh9tgEIA1t/MEG548LJSPqMbDg0JZ2QTpg6DzPXd7SRFlb6rUJ5e5BVRSLcTZC0DyB+eKLPDpapeWxVx/4X6/K3VFZ5iuwfa9ZzcB5WIJlsVx4/O0l4R5bxfDAhrI1OHiBDYgYFKr", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 63, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T05:15:10.376 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.846) 0:00:08.326 ******** 2019-10-13T05:15:10.402 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-13T05:15:10.403 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:10.440 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.064) 0:00:08.391 ******** 2019-10-13T05:15:10.465 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-13T05:15:10.467 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:10.503 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.062) 0:00:08.453 ******** 2019-10-13T05:15:10.528 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-13T05:15:10.530 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:10.567 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.064) 0:00:08.517 ******** 2019-10-13T05:15:10.593 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-13T05:15:10.594 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:10.635 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.068) 0:00:08.586 ******** 2019-10-13T05:15:10.780 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:15:10.823 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:10 +0000 (0:00:00.185) 0:00:08.772 ******** 2019-10-13T05:15:11.014 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T05:15:11.015 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T05:15:11.138 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T05:15:11.148 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.71"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:d765"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570943711", "hour": "05", "iso8601": "2019-10-13T05:15:11Z", "iso8601_basic": "20191013T051511355566", "iso8601_basic_short": "20191013T051511", "iso8601_micro": "2019-10-13T05:15:11.355636Z", "minute": "15", "month": "10", "second": "11", "time": "05:15:11", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.71", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:d7:65", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:d765", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:d7:65", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCA 2019-10-13T05:15:11.150 INFO:teuthology.task.ansible.out:xKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "nvme0n1": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "labels": [], "masters": [], "uuids": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", 2019-10-13T05:15:11.151 INFO:teuthology.task.ansible.out: "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 2019-10-13T05:15:11.152 INFO:teuthology.task.ansible.out:Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controll 2019-10-13T05:15:11.154 INFO:teuthology.task.ansible.out:er: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansi 2019-10-13T05:15:11.155 INFO:teuthology.task.ansible.out:ble_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan 2019-10-13T05:15:11.157 INFO:teuthology.task.ansible.out:_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:aa:4a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]" 2019-10-13T05:15:11.158 INFO:teuthology.task.ansible.out:, "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:aa:4b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "o 2019-10-13T05:15:11.160 INFO:teuthology.task.ansible.out:n", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:d7:64", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_o 2019-10-13T05:15:11.161 INFO:teuthology.task.ansible.out:ffload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.71", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:d765", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8f:d7:65", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-atxp 2019-10-13T05:15:11.162 INFO:teuthology.task.ansible.out:xvlbjapukkvwrslbrjfmzcvjyoit ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi071.front.sepia.ceph.com", "ansible_hostname": "smithi071", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "enp3s0f0", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_g 2019-10-13T05:15:11.164 INFO:teuthology.task.ansible.out:ather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31467, "ansible_memory_mb": {"nocache": {"free": 31781, "used": 274}, "real": {"free": 31467, "total": 32055, "used": 588}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226440803, "block_size": 4096, "block_total": 240095108, "block_used": 13654305, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501529088, "size_total": 2019-10-13T05:15:11.165 INFO:teuthology.task.ansible.out:983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi071.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "G 2019-10-13T05:15:11.167 INFO:teuthology.task.ansible.out:enuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731590", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8AA4A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAJK7vWCJ8jjhW3mOnzg46tgl/mOJhHNCaWJpv3Nf1PPYKIa6gMH/ATBf1AwhwwiBK+X/N8mFCk+ZvvDDizXns/vSYuBRj/yiWABZdHQyJBg/6ESNQctpUzW2Kk3TCs/fMoSudTpVT+XNfqkeTHJe5LzyrnLZjlIE14c7x6/UlZhFAAAAFQDQbV9z2rmLAQd+cv17ABzbj6OjnQAAAIBgT68xWQEOcWSb6JuUS7LMZVUB/tNy17rjbnIHAX4x5paKIFQXzaCtJFAqwifmtQVkq9U8AczFcbkNv0/Mgg4dClAEfk7J6mMmxW9DlV638Ijvr0Bl+YAyo43BuRkgCSOPsKa8W91wfYBNU2kapJDiEf3T9TddiPG4f8iZ/4CQtQAAAIAEJP6Y2cmeBk5Y4rFaZ0WmCyNWhVX3MnxXz1qHyoHAvfl2NPum+3GoX0mB/5RTQXcCsNnI4efn+b5FcfX6fSZRHYS09M2Nni2IZLjvMXH/oi545wnZVb8aZDAiFj6FhXPLlrti7CVMBen7FQQXnB68730A8lfkvNctSQHNBJV0GQ==", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICYRzS7DLsfIeQJ6jyVgqQWPlDHJxuRvLP84djVbr2XPdYDhiuiGyXEsDlQiC07MLZoMjn6tNZOuiDzhSpWHjU=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIIpOWUcxgNsqFUuI4zoUKaGCMnYCUTJE0r3bbhvK7HJU", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDMb+Sj+4T/9nm4w1P0FJXLR8/xXnWSHI8tIy559zHUKDvuGlEAk3v9y97kKRV0jlAtxXQKoZhR+0fyJEeOK3PppvH7tUEcqK94JCvGNDoRo+tKwIRrjs3Ux0 2019-10-13T05:15:11.168 INFO:teuthology.task.ansible.out:aQOjB9E41/62sxqby7t7b8K97uwok3Lattw4/cPtCJAov231WLM9Rrve51F1EDQFo5slzsMz9yF92VcHpPUTT2bfpTQUCmx075StocKB+kQXvcKg+NavyePCq1U9udITC6kLPS47g+k6fptrAh6jHNPaFHXQGbfHCWdvqHZyYs52fNVT1iy0Ky0IUvBDTKg2D97KhePQexo/zPsCT3DUpsy+7OTXnR", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 44, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T05:15:11.178 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-13T05:15:11.257 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.435) 0:00:09.208 ******** 2019-10-13T05:15:11.274 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T05:15:11.286 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-13T05:15:11.287 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:11.327 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.069) 0:00:09.277 ******** 2019-10-13T05:15:11.353 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-13T05:15:11.354 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:11.394 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.067) 0:00:09.344 ******** 2019-10-13T05:15:11.410 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T05:15:11.411 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T05:15:11.422 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-13T05:15:11.424 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:11.467 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.069) 0:00:09.413 ******** 2019-10-13T05:15:11.489 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-13T05:15:11.490 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:11.530 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.067) 0:00:09.481 ******** 2019-10-13T05:15:11.547 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T05:15:11.547 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T05:15:11.676 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:15:11.719 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:11 +0000 (0:00:00.188) 0:00:09.669 ******** 2019-10-13T05:15:11.724 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T05:15:11.724 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T05:15:11.824 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T05:15:11.919 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T05:15:11.962 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T05:15:12.058 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-13T05:15:12.091 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T05:15:12.189 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T05:15:12.232 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T05:15:12.327 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T05:15:12.372 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T05:15:12.477 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49 2019-10-13T05:15:12.478 INFO:teuthology.task.ansible.out:a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T05:15:12.510 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T05:15:12.628 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T05:15:12.640 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T05:15:12.765 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T05:15:12.787 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T05:15:12.904 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T05:15:12.915 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T05:15:13.051 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T05:15:13.064 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T05:15:13.199 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T05:15:13.201 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T05:15:13.343 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T05:15:13.344 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T05:15:13.481 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T05:15:13.495 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T05:15:13.626 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T05:15:13.636 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T05:15:13.741 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T05:15:13.793 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T05:15:13.889 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T05:15:13.930 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T05:15:14.025 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T05:15:14.154 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T05:15:14.166 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T05:15:14.302 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T05:15:14.313 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T05:15:14.454 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T05:15:14.465 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T05:15:14.601 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T05:15:14.623 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T05:15:14.739 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T05:15:14.762 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T05:15:14.894 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T05:15:14.906 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T05:15:15.051 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T05:15:15.052 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:15.064 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T05:15:15.201 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T05:15:15.203 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T05:15:15.334 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T05:15:15.347 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T05:15:15.470 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T05:15:15.472 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:15.506 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T05:15:15.621 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T05:15:15.656 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T05:15:15.770 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T05:15:15.813 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T05:15:15.919 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T05:15:15.971 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T05:15:16.058 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T05:15:16.127 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T05:15:16.201 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T05:15:16.264 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T05:15:16.380 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T05:15:16.403 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T05:15:16.510 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T05:15:16.552 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T05:15:16.638 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T05:15:16.692 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T05:15:16.766 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T05:15:16.841 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T05:15:16.906 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T05:15:16.982 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T05:15:17.058 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T05:15:17.143 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T05:15:17.185 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T05:15:17.282 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T05:15:17.338 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T05:15:17.403 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T05:15:17.479 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T05:15:17.553 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T05:15:17.617 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T05:15:17.712 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T05:15:17.755 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T05:15:17.860 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T05:15:17.903 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T05:15:18.018 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T05:15:18.030 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T05:15:18.159 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T05:15:18.172 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T05:15:18.309 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T05:15:18.320 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T05:15:18.448 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T05:15:18.486 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-13T05:15:18.591 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T05:15:18.623 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T05:15:18.709 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T05:15:18.782 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T05:15:18.856 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T05:15:18.930 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T05:15:18.996 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T05:15:19.080 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T05:15:19.133 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-13T05:15:19.227 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T05:15:19.270 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T05:15:19.354 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-13T05:15:19.407 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T05:15:19.501 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T05:15:19.556 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T05:15:19.658 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T05:15:19.702 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T05:15:19.817 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T05:15:19.829 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T05:15:19.984 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T05:15:19.986 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-13T05:15:20.105 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T05:15:20.138 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T05:15:20.242 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T05:15:20.264 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T05:15:20.393 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T05:15:20.425 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T05:15:20.553 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T05:15:20.575 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T05:15:20.692 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T05:15:20.725 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T05:15:20.820 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T05:15:20.884 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T05:15:20.979 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T05:15:21.033 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T05:15:21.117 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T05:15:21.184 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-13T05:15:21.259 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T05:15:21.327 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-13T05:15:21.412 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T05:15:21.460 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-13T05:15:21.542 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T05:15:21.628 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-13T05:15:22.662 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/ 2019-10-13T05:15:22.671 INFO:teuthology.task.ansible.out:Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e 2019-10-13T05:15:22.672 INFO:teuthology.task.ansible.out:9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansib 2019-10-13T05:15:22.673 INFO:teuthology.task.ansible.out:le_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T05:15:22.720 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-13T05:15:22.774 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T05:15:22.858 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T05:15:22.922 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T05:15:23.012 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-13T05:15:23.065 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T05:15:23.150 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-13T05:15:23.218 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T05:15:23.283 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T05:15:23.379 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T05:15:23.420 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T05:15:23.525 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T05:15:23.557 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T05:15:23.672 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-13T05:15:23.705 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T05:15:23.822 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T05:15:23.845 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T05:15:23.983 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T05:15:23.995 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T05:15:24.123 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T05:15:24.135 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T05:15:24.283 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T05:15:24.418 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T05:15:24.429 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T05:15:24.570 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T05:15:24.593 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T05:15:24.752 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T05:15:24.754 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T05:15:24.888 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T05:15:24.901 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T05:15:25.024 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T05:15:25.036 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T05:15:25.163 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T05:15:25.174 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T05:15:25.294 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T05:15:25.326 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T05:15:25.443 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T05:15:25.488 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T05:15:25.592 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T05:15:25.621 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T05:15:25.730 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T05:15:25.771 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T05:15:25.867 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T05:15:25.919 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T05:15:26.016 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T05:15:26.050 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T05:15:26.147 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T05:15:26.201 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T05:15:26.296 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T05:15:26.334 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T05:15:26.421 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T05:15:26.494 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T05:15:26.559 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T05:15:26.635 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T05:15:26.710 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T05:15:26.806 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T05:15:26.842 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T05:15:26.976 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T05:15:27.000 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T05:15:27.140 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T05:15:27.267 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T05:15:27.414 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T05:15:27.529 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-13T05:15:27.562 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T05:15:27.702 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T05:15:28.060 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-13T05:15:28.308 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-13T05:15:28.549 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-13T05:15:28.760 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-13T05:15:29.043 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-13T05:15:29.284 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-13T05:15:29.587 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-13T05:15:29.724 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-13T05:15:30.051 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-13T05:15:30.222 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:30 +0000 (0:00:18.502) 0:00:28.172 ******** 2019-10-13T05:15:30.232 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-13T05:15:30.248 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T05:15:30.289 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:30 +0000 (0:00:00.067) 0:00:28.239 ******** 2019-10-13T05:15:30.343 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T05:15:30.345 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T05:15:30.346 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T05:15:30.348 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T05:15:30.349 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T05:15:30.351 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T05:15:30.352 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-13T05:15:30.353 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0 2019-10-13T05:15:30.355 INFO:teuthology.task.ansible.out:bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name" 2019-10-13T05:15:30.356 INFO:teuthology.task.ansible.out:: "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name" 2019-10-13T05:15:30.358 INFO:teuthology.task.ansible.out:: "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91e 2019-10-13T05:15:30.359 INFO:teuthology.task.ansible.out:d570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d01 2019-10-13T05:15:30.361 INFO:teuthology.task.ansible.out:1a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T05:15:30.405 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-10-13T05:15:30.407 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:30 +0000 (0:00:00.116) 0:00:28.356 ******** 2019-10-13T05:15:30.703 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T05:15:30.704 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-13T05:15:30.866 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:30 +0000 (0:00:00.460) 0:00:28.817 ******** 2019-10-13T05:15:30.893 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T05:15:30.934 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:30 +0000 (0:00:00.067) 0:00:28.884 ******** 2019-10-13T05:15:30.983 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T05:15:30.988 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T05:15:30.989 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T05:15:30.991 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T05:15:30.992 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T05:15:30.994 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T05:15:30.995 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-13T05:15:30.997 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6 2019-10-13T05:15:30.998 INFO:teuthology.task.ansible.out:c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa 2019-10-13T05:15:30.999 INFO:teuthology.task.ansible.out:2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db2 2019-10-13T05:15:31.001 INFO:teuthology.task.ansible.out:93dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.red 2019-10-13T05:15:31.002 INFO:teuthology.task.ansible.out:hat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f 2019-10-13T05:15:31.004 INFO:teuthology.task.ansible.out:57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T05:15:31.068 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:31 +0000 (0:00:00.133) 0:00:29.018 ******** 2019-10-13T05:15:31.311 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:31.323 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi129.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T05:15:31.375 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:31 +0000 (0:00:00.307) 0:00:29.325 ******** 2019-10-13T05:15:31.577 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T05:15:31.577 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T05:15:31.707 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:31.715 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-10-13T05:15:31.715 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T05:15:31.769 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:31 +0000 (0:00:00.394) 0:00:29.720 ******** 2019-10-13T05:15:31.775 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T05:15:31.776 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T05:15:31.919 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T05:15:31.982 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T05:15:32.092 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T05:15:32.146 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T05:15:32.242 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T05:15:32.243 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T05:15:32.284 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T05:15:32.360 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T05:15:32.446 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T05:15:32.508 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-13T05:15:32.510 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T05:15:32.588 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T05:15:32.590 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T05:15:32.651 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T05:15:32.748 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T05:15:32.799 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-13T05:15:32.821 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:32.884 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-13T05:15:32.886 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T05:15:32.989 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T05:15:32.991 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-13T05:15:33.001 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.002 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.013 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.014 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.035 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T05:15:33.158 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T05:15:33.170 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+r 2019-10-13T05:15:33.172 INFO:teuthology.task.ansible.out:R1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.303 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T05:15:33.305 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T05:15:33.325 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T05:15:33.326 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-13T05:15:33.339 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.350 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.351 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.362 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:33.452 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T05:15:33.507 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T05:15:33.606 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T05:15:33.652 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T05:15:33.654 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T05:15:33.749 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T05:15:33.796 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T05:15:33.896 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T05:15:33.949 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T05:15:34.055 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T05:15:34.118 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T05:15:34.182 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T05:15:34.278 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T05:15:34.342 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T05:15:34.344 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T05:15:34.436 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T05:15:34.479 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T05:15:34.481 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T05:15:34.574 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T05:15:34.627 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T05:15:34.727 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T05:15:34.728 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T05:15:34.759 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T05:15:34.875 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T05:15:34.876 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T05:15:34.917 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T05:15:34.919 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T05:15:35.023 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T05:15:35.077 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T05:15:35.150 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T05:15:35.203 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T05:15:35.279 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T05:15:35.281 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T05:15:35.344 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-13T05:15:35.428 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T05:15:35.492 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T05:15:35.576 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T05:15:35.642 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T05:15:35.696 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": 2019-10-13T05:15:35.698 INFO:teuthology.task.ansible.out:true} 2019-10-13T05:15:35.781 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T05:15:35.783 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T05:15:35.835 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T05:15:35.918 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T05:15:35.920 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T05:15:35.950 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T05:15:36.067 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T05:15:36.109 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T05:15:36.111 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T05:15:36.216 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T05:15:36.228 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T05:15:36.229 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T05:15:36.343 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T05:15:36.375 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T05:15:36.501 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T05:15:36.524 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T05:15:36.652 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T05:15:36.674 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T05:15:36.799 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T05:15:36.833 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T05:15:36.960 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T05:15:36.988 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T05:15:37.103 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T05:15:37.156 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T05:15:37.244 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T05:15:37.304 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T05:15:37.389 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T05:15:37.443 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T05:15:37.518 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T05:15:37.520 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T05:15:37.590 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T05:15:37.681 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T05:15:37.683 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T05:15:37.746 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T05:15:37.810 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T05:15:37.874 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T05:15:37.876 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T05:15:37.927 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T05:15:37.998 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T05:15:38.000 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T05:15:38.075 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T05:15:38.153 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T05:15:38.211 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T05:15:38.301 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T05:15:38.357 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T05:15:38.433 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T05:15:38.515 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T05:15:38.584 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T05:15:38.639 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T05:15:38.707 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T05:15:38.797 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T05:15:38.851 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T05:15:38.960 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T05:15:38.961 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T05:15:38.963 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T05:15:38.993 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T05:15:39.099 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T05:15:39.121 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T05:15:39.226 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T05:15:39.258 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T05:15:39.260 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T05:15:39.261 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T05:15:39.373 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T05:15:39.374 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T05:15:39.516 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T05:15:39.518 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T05:15:39.549 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T05:15:39.669 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T05:15:39.703 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T05:15:39.820 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T05:15:39.838 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T05:15:39.839 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T05:15:39.955 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T05:15:39.986 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T05:15:40.096 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T05:15:40.128 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T05:15:40.243 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T05:15:40.275 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T05:15:40.391 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T05:15:40.422 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T05:15:40.540 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T05:15:40.582 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T05:15:40.584 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T05:15:40.677 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T05:15:40.730 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T05:15:40.825 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T05:15:40.827 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T05:15:40.859 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T05:15:40.963 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T05:15:41.012 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T05:15:41.014 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T05:15:41.106 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T05:15:41.160 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T05:15:41.244 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T05:15:41.245 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T05:15:41.308 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T05:15:41.383 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T05:15:41.447 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T05:15:41.542 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T05:15:41.585 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T05:15:41.681 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T05:15:41.725 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T05:15:41.831 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T05:15:41.895 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T05:15:41.969 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T05:15:42.034 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T05:15:42.119 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T05:15:42.173 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T05:15:42.258 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T05:15:42.312 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T05:15:42.419 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T05:15:42.451 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T05:15:42.550 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T05:15:42.616 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T05:15:42.680 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T05:15:42.754 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T05:15:42.808 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T05:15:42.902 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T05:15:42.958 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T05:15:43.037 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T05:15:43.103 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T05:15:43.197 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T05:15:43.240 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T05:15:43.330 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T05:15:43.379 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T05:15:43.486 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T05:15:43.520 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T05:15:43.633 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T05:15:43.655 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T05:15:43.792 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T05:15:43.793 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T05:15:43.929 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T05:15:43.930 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T05:15:44.068 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T05:15:44.079 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T05:15:44.216 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T05:15:44.228 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T05:15:44.343 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T05:15:44.365 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T05:15:44.489 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T05:15:44.491 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T05:15:44.627 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T05:15:44.638 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T05:15:44.753 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T05:15:44.785 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T05:15:44.900 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T05:15:44.932 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-13T05:15:45.037 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-13T05:15:45.059 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T05:15:45.185 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T05:15:45.196 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T05:15:45.324 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-13T05:15:45.325 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T05:15:45.335 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T05:15:45.472 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T05:15:45.474 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T05:15:45.484 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-13T05:15:45.485 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T05:15:45.621 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T05:15:45.622 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T05:15:45.623 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T05:15:45.758 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T05:15:45.780 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T05:15:45.906 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T05:15:45.939 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T05:15:46.046 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T05:15:46.068 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T05:15:46.069 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:46.175 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T05:15:46.207 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T05:15:46.302 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T05:15:46.345 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T05:15:46.454 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T05:15:46.479 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T05:15:46.573 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T05:15:46.615 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-13T05:15:46.720 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-13T05:15:46.769 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-13T05:15:46.863 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-13T05:15:46.901 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T05:15:47.005 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T05:15:47.053 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T05:15:47.147 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T05:15:47.197 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T05:15:47.306 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T05:15:47.341 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T05:15:47.436 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T05:15:47.486 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T05:15:47.616 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:47 +0000 (0:00:15.846) 0:00:45.567 ******** 2019-10-13T05:15:47.623 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T05:15:47.671 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T05:15:47.673 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.716 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:47 +0000 (0:00:00.100) 0:00:45.667 ******** 2019-10-13T05:15:47.725 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.729 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ f 2019-10-13T05:15:47.730 INFO:teuthology.task.ansible.out:f4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.754 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.766 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.778 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.779 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.791 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.792 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.803 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.804 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.806 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.845 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-13T05:15:47.857 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.952 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-13T05:15:47.964 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:47.996 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T05:15:48.114 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T05:15:48.147 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T05:15:48.254 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T05:15:48.286 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T05:15:48.398 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T05:15:48.431 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T05:15:48.432 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.453 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.464 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.476 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.487 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.488 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.499 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.500 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.511 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.522 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.523 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.534 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.535 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.546 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.547 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.558 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.559 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T05:15:48.570 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.571 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.572 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.583 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.584 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.585 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.596 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.597 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.599 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.610 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.610 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.612 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.622 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.623 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.624 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.626 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.636 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.637 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.638 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.649 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.650 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.651 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.662 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.663 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.664 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.680 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.688 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.688 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.690 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.703 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.704 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.706 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.707 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.708 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.719 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.720 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.731 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.733 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.734 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.736 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.738 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.750 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.751 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.754 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.755 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.768 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.769 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.780 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.781 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.783 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.784 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.794 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.795 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.797 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.808 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.809 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.811 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.822 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.823 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.825 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.826 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.828 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.838 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.839 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.841 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.851 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.852 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.854 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.864 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.865 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.867 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.878 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.881 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.882 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.883 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.895 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.896 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.898 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.899 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.901 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.911 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.912 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.914 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.915 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.927 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.928 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.929 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.931 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.942 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.943 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.944 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.955 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.956 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.958 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.968 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.971 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.973 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.975 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.976 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.986 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.987 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.998 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:48.999 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.001 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.011 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.012 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.019 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.026 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.026 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.038 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.039 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.040 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.042 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.043 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.053 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.055 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.057 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.069 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.070 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.071 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.073 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.085 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.087 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.098 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.100 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.101 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.111 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.113 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.114 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.125 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.126 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.137 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.139 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.153 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.156 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.156 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.158 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.182 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.183 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.185 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.187 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.189 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.192 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.234 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:49 +0000 (0:00:01.517) 0:00:47.184 ******** 2019-10-13T05:15:49.240 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.242 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.243 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.245 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.267 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-10-13T05:15:49.268 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.309 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:49 +0000 (0:00:00.074) 0:00:47.259 ******** 2019-10-13T05:15:49.353 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:49 +0000 (0:00:00.037) 0:00:47.297 ******** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:49.397 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:49 +0000 (0:00:00.049) 0:00:47.347 ******** 2019-10-13T05:15:49.520 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T05:15:49.521 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T05:15:49.577 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T05:15:49.673 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T05:15:49.705 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T05:15:49.812 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T05:15:49.855 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T05:15:49.941 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T05:15:50.005 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T05:15:50.081 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T05:15:50.146 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T05:15:50.231 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T05:15:50.285 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T05:15:50.358 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T05:15:50.414 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T05:15:50.487 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T05:15:50.551 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T05:15:50.624 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T05:15:50.688 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T05:15:50.761 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T05:15:50.804 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T05:15:50.898 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-13T05:15:50.950 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-13T05:15:51.019 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-13T05:15:51.082 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-13T05:15:51.157 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-13T05:15:51.221 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-13T05:15:51.284 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-13T05:15:51.357 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-13T05:15:51.411 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-13T05:15:51.495 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-13T05:15:51.559 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T05:15:51.643 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T05:15:51.696 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T05:15:51.698 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:51.786 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T05:15:51.828 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T05:15:51.918 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T05:15:51.972 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T05:15:52.065 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T05:15:52.107 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-13T05:15:52.203 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-13T05:15:52.236 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T05:15:52.348 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T05:15:52.354 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T05:15:52.460 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T05:15:52.482 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T05:15:52.597 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T05:15:52.609 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T05:15:52.725 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T05:15:52.737 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T05:15:52.864 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T05:15:52.865 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T05:15:52.987 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T05:15:53.015 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T05:15:53.110 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T05:15:53.173 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:03.776) 0:00:51.123 ******** 2019-10-13T05:15:53.180 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T05:15:53.203 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T05:15:53.204 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => { "msg": "Host smithi129.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-13T05:15:53.242 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.069) 0:00:51.193 ******** 2019-10-13T05:15:53.290 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T05:15:53.292 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} TASK [users : Remove revoked users] ******************************************** 2019-10-13T05:15:53.293 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T05:15:53.346 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.103) 0:00:51.296 ******** 2019-10-13T05:15:53.385 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.039) 0:00:51.336 ******** 2019-10-13T05:15:53.390 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T05:15:53.391 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => { "msg": "Host smithi071.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-13T05:15:53.437 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.051) 0:00:51.387 ******** 2019-10-13T05:15:53.443 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T05:15:53.445 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.493 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.055) 0:00:51.443 ******** 2019-10-13T05:15:53.499 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T05:15:53.500 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-13T05:15:53.547 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.053) 0:00:51.497 ******** 2019-10-13T05:15:53.554 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.599 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.052) 0:00:51.549 ******** 2019-10-13T05:15:53.605 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T05:15:53.607 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.647 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.047) 0:00:51.597 ******** 2019-10-13T05:15:53.652 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T05:15:53.654 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.696 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.049) 0:00:51.647 ******** 2019-10-13T05:15:53.702 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-13T05:15:53.704 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.745 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.048) 0:00:51.695 ******** 2019-10-13T05:15:53.751 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T05:15:53.753 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.793 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.048) 0:00:51.744 ******** 2019-10-13T05:15:53.799 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T05:15:53.800 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.840 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.046) 0:00:51.790 ******** 2019-10-13T05:15:53.845 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-13T05:15:53.846 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.886 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.046) 0:00:51.836 ******** 2019-10-13T05:15:53.891 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T05:15:53.892 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.932 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.045) 0:00:51.882 ******** 2019-10-13T05:15:53.937 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-13T05:15:53.939 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:53.982 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:53 +0000 (0:00:00.049) 0:00:51.932 ******** 2019-10-13T05:15:53.987 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:53.989 INFO:teuthology.task.ansible.out:TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.030 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.047) 0:00:51.980 ******** 2019-10-13T05:15:54.035 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T05:15:54.036 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.078 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.048) 0:00:52.028 ******** 2019-10-13T05:15:54.083 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-13T05:15:54.085 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.138 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.060) 0:00:52.089 ******** 2019-10-13T05:15:54.147 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T05:15:54.150 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.195 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.057) 0:00:52.146 ******** 2019-10-13T05:15:54.201 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T05:15:54.203 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.244 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.048) 0:00:52.194 ******** 2019-10-13T05:15:54.250 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi129.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.252 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.291 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.047) 0:00:52.241 ******** 2019-10-13T05:15:54.297 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T05:15:54.297 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.337 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.045) 0:00:52.287 ******** 2019-10-13T05:15:54.343 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T05:15:54.344 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.386 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.048) 0:00:52.336 ******** 2019-10-13T05:15:54.391 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-13T05:15:54.392 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.393 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.434 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.047) 0:00:52.384 ******** 2019-10-13T05:15:54.439 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-13T05:15:54.441 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.486 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.052) 0:00:52.436 ******** 2019-10-13T05:15:54.488 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:54.493 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T05:15:54.494 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.537 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.488 ******** 2019-10-13T05:15:54.545 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-13T05:15:54.546 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.593 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.055) 0:00:52.543 ******** 2019-10-13T05:15:54.600 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-13T05:15:54.601 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.644 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.594 ******** 2019-10-13T05:15:54.650 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T05:15:54.651 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.693 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.049) 0:00:52.643 ******** 2019-10-13T05:15:54.695 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:54.700 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:54.702 INFO:teuthology.task.ansible.out:TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.742 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.048) 0:00:52.692 ******** 2019-10-13T05:15:54.743 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:54.749 INFO:teuthology.task.ansible.out: 2019-10-13T05:15:54.751 INFO:teuthology.task.ansible.out:TASK [common : Register with subscription-manager.] **************************** skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.792 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.742 ******** 2019-10-13T05:15:54.798 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T05:15:54.799 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.843 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.793 ******** 2019-10-13T05:15:54.850 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.893 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.843 ******** 2019-10-13T05:15:54.900 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-13T05:15:54.901 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:54.944 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.050) 0:00:52.894 ******** 2019-10-13T05:15:54.950 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:54.998 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:54 +0000 (0:00:00.047) 0:00:52.941 ******** TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-13T05:15:55.000 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.040 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.049) 0:00:52.990 ******** 2019-10-13T05:15:55.046 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-13T05:15:55.048 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.087 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.047) 0:00:53.038 ******** 2019-10-13T05:15:55.093 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-10-13T05:15:55.094 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.135 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.047) 0:00:53.085 ******** 2019-10-13T05:15:55.141 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T05:15:55.143 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.184 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.049) 0:00:53.134 ******** 2019-10-13T05:15:55.190 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-13T05:15:55.192 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.235 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.050) 0:00:53.185 ******** 2019-10-13T05:15:55.241 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T05:15:55.243 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:55.284 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.049) 0:00:53.235 ******** 2019-10-13T05:15:55.290 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T05:15:55.291 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.332 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.048) 0:00:53.283 ******** 2019-10-13T05:15:55.338 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T05:15:55.340 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.379 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.046) 0:00:53.330 ******** 2019-10-13T05:15:55.385 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T05:15:55.386 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:55.426 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.046) 0:00:53.376 ******** 2019-10-13T05:15:55.431 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-13T05:15:55.433 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.474 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.048) 0:00:53.425 ******** 2019-10-13T05:15:55.480 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T05:15:55.481 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.045) 0:00:53.471 ******** 2019-10-13T05:15:55.526 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T05:15:55.527 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.568 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.047) 0:00:53.518 ******** 2019-10-13T05:15:55.573 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-13T05:15:55.575 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.614 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.045) 0:00:53.564 ******** 2019-10-13T05:15:55.620 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-13T05:15:55.621 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.661 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.046) 0:00:53.611 ******** 2019-10-13T05:15:55.666 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T05:15:55.668 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.708 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.047) 0:00:53.659 ******** 2019-10-13T05:15:55.713 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-13T05:15:55.715 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.764 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.055) 0:00:53.714 ******** 2019-10-13T05:15:55.770 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-13T05:15:55.772 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.814 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.050) 0:00:53.765 ******** 2019-10-13T05:15:55.820 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T05:15:55.822 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.866 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.050) 0:00:53.815 ******** 2019-10-13T05:15:55.871 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-13T05:15:55.873 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:55.917 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.051) 0:00:53.867 ******** 2019-10-13T05:15:55.923 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-13T05:15:55.925 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:55.969 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:55 +0000 (0:00:00.051) 0:00:53.919 ******** 2019-10-13T05:15:55.974 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T05:15:55.976 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.021 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.051) 0:00:53.971 ******** 2019-10-13T05:15:56.026 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T05:15:56.071 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.050) 0:00:54.021 ******** 2019-10-13T05:15:56.077 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-13T05:15:56.078 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:15:56.122 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.051) 0:00:54.072 ******** 2019-10-13T05:15:56.128 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T05:15:56.130 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.182 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.059) 0:00:54.131 ******** 2019-10-13T05:15:56.187 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T05:15:56.230 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.049) 0:00:54.180 ******** 2019-10-13T05:15:56.236 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T05:15:56.238 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.282 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.047) 0:00:54.228 ******** 2019-10-13T05:15:56.284 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T05:15:56.291 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.329 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.051) 0:00:54.280 ******** 2019-10-13T05:15:56.335 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T05:15:56.337 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.338 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.379 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.050) 0:00:54.330 ******** 2019-10-13T05:15:56.385 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T05:15:56.387 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.434 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.054) 0:00:54.384 ******** 2019-10-13T05:15:56.441 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi129.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.443 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.489 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.054) 0:00:54.439 ******** 2019-10-13T05:15:56.494 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T05:15:56.495 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.497 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.541 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.052) 0:00:54.491 ******** 2019-10-13T05:15:56.547 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T05:15:56.548 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.595 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.053) 0:00:54.545 ******** 2019-10-13T05:15:56.600 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T05:15:56.602 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.649 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.049) 0:00:54.595 ******** 2019-10-13T05:15:56.672 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T05:15:56.674 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.714 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.069) 0:00:54.664 ******** 2019-10-13T05:15:56.825 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T05:15:56.826 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.086344", "end": "2019-10-13 05:15:57.051344", "rc": 0, "start": "2019-10-13 05:15:56.965000", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T05:15:56.876 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.162) 0:00:54.827 ******** 2019-10-13T05:15:56.885 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.008338", "end": "2019-10-13 05:15:57.118624", "rc": 0, "start": "2019-10-13 05:15:57.110286", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T05:15:56.930 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.053) 0:00:54.880 ******** 2019-10-13T05:15:56.936 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-13T05:15:56.938 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:56.982 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:56 +0000 (0:00:00.052) 0:00:54.932 ******** 2019-10-13T05:15:56.989 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.003 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:57.033 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.050) 0:00:54.983 ******** 2019-10-13T05:15:57.040 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.063 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:15:57.091 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.058) 0:00:55.041 ******** 2019-10-13T05:15:57.098 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.140 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.049) 0:00:55.090 ******** 2019-10-13T05:15:57.146 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T05:15:57.147 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.289 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.149) 0:00:55.240 ******** 2019-10-13T05:15:57.295 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.339 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.049) 0:00:55.289 ******** 2019-10-13T05:15:57.345 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-13T05:15:57.347 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.388 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.049) 0:00:55.339 ******** 2019-10-13T05:15:57.395 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.442 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.047) 0:00:55.386 ******** TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-13T05:15:57.444 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.486 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.049) 0:00:55.436 ******** 2019-10-13T05:15:57.493 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.535 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.049) 0:00:55.485 ******** 2019-10-13T05:15:57.540 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-13T05:15:57.542 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.583 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.048) 0:00:55.534 ******** 2019-10-13T05:15:57.589 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.648 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.064) 0:00:55.598 ******** 2019-10-13T05:15:57.657 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-13T05:15:57.659 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.703 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.055) 0:00:55.654 ******** 2019-10-13T05:15:57.709 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:57.750 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:57 +0000 (0:00:00.047) 0:00:55.701 ******** 2019-10-13T05:15:58.303 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.412009", "end": "2019-10-13 05:15:58.548355", "rc": 0, "start": "2019-10-13 05:15:58.136346", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:15:58.355 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.604) 0:00:56.305 ******** 2019-10-13T05:15:58.369 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.503198", "end": "2019-10-13 05:15:58.570385", "rc": 0, "start": "2019-10-13 05:15:58.067187", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:15:58.441 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.086) 0:00:56.391 ******** 2019-10-13T05:15:58.449 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-13T05:15:58.451 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:58.492 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.050) 0:00:56.442 ******** 2019-10-13T05:15:58.498 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:15:58.539 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.047) 0:00:56.489 ******** 2019-10-13T05:15:58.671 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.044557", "end": "2019-10-13 05:15:58.918006", "rc": 0, "start": "2019-10-13 05:15:58.873449", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:15:58.710 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.171) 0:00:56.660 ******** 2019-10-13T05:15:58.718 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.045049", "end": "2019-10-13 05:15:58.942660", "rc": 0, "start": "2019-10-13 05:15:58.897611", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:15:58.757 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:15:58 +0000 (0:00:00.047) 0:00:56.708 ******** 2019-10-13T05:16:02.072 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:02.080 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-13T05:16:02.082 INFO:teuthology.task.ansible.out: [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:16:02.122 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:02 +0000 (0:00:03.365) 0:01:00.073 ******** 2019-10-13T05:16:02.200 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:02.212 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:16:02.279 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:02 +0000 (0:00:00.156) 0:01:00.229 ******** 2019-10-13T05:16:03.063 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-13T05:16:03.066 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:16:03.105 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.825) 0:01:01.055 ******** 2019-10-13T05:16:03.122 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:16:03.161 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.056) 0:01:01.111 ******** 2019-10-13T05:16:03.167 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-13T05:16:03.168 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:03.209 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.047) 0:01:01.159 ******** 2019-10-13T05:16:03.215 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:03.256 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.047) 0:01:01.207 ******** 2019-10-13T05:16:03.733 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-10-13T05:16:03.734 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-13T05:16:03.784 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.527) 0:01:01.734 ******** 2019-10-13T05:16:03.794 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-13T05:16:03.842 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:03 +0000 (0:00:00.057) 0:01:01.791 ******** 2019-10-13T05:16:04.678 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:16:04.725 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:04 +0000 (0:00:00.883) 0:01:02.675 ******** 2019-10-13T05:16:04.732 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:16:04.782 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:04 +0000 (0:00:00.056) 0:01:02.732 ******** 2019-10-13T05:16:05.116 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T05:16:05.116 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T05:16:05.161 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T05:16:05.449 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T05:16:05.482 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T05:16:05.747 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-13T05:16:05.815 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:05 +0000 (0:00:01.033) 0:01:03.765 ******** 2019-10-13T05:16:05.823 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-13T05:16:05.869 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:05 +0000 (0:00:00.053) 0:01:03.819 ******** 2019-10-13T05:16:05.948 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-13T05:16:05.950 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:16:06.000 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:05 +0000 (0:00:00.131) 0:01:03.951 ******** 2019-10-13T05:16:06.007 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:16:06.069 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:06 +0000 (0:00:00.068) 0:01:04.020 ******** 2019-10-13T05:16:06.342 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T05:16:06.342 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-13T05:16:06.438 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-13T05:16:06.648 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-13T05:16:06.738 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-13T05:16:06.937 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-13T05:16:06.979 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:06 +0000 (0:00:00.909) 0:01:04.930 ******** 2019-10-13T05:16:07.007 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-13T05:16:07.049 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.069) 0:01:04.999 ******** 2019-10-13T05:16:07.056 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-13T05:16:07.057 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-13T05:16:07.101 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.052) 0:01:05.052 ******** 2019-10-13T05:16:07.108 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-13T05:16:07.149 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.047) 0:01:05.100 ******** 2019-10-13T05:16:07.155 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-13T05:16:07.157 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.197 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.047) 0:01:05.147 ******** 2019-10-13T05:16:07.202 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.245 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.047) 0:01:05.195 ******** 2019-10-13T05:16:07.250 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-13T05:16:07.251 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:16:07.296 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.050) 0:01:05.246 ******** 2019-10-13T05:16:07.301 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:16:07.344 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.047) 0:01:05.294 ******** 2019-10-13T05:16:07.349 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-13T05:16:07.350 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.352 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.354 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.398 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.054) 0:01:05.348 ******** 2019-10-13T05:16:07.404 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.405 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.407 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.449 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.051) 0:01:05.400 ******** 2019-10-13T05:16:07.455 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-13T05:16:07.455 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.457 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.501 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.051) 0:01:05.451 ******** 2019-10-13T05:16:07.505 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.506 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:07.546 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:07 +0000 (0:00:00.045) 0:01:05.497 ******** 2019-10-13T05:16:08.425 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T05:16:08.427 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:08.447 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:09.289 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:09.332 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:09 +0000 (0:00:01.785) 0:01:07.282 ******** 2019-10-13T05:16:09.400 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:09.443 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:09 +0000 (0:00:00.111) 0:01:07.393 ******** 2019-10-13T05:16:09.649 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-13T05:16:09.692 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:09 +0000 (0:00:00.248) 0:01:07.642 ******** 2019-10-13T05:16:09.793 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-13T05:16:09.838 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:09 +0000 (0:00:00.146) 0:01:07.789 ******** 2019-10-13T05:16:10.107 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-13T05:16:10.109 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-13T05:16:10.155 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.316) 0:01:08.105 ******** 2019-10-13T05:16:10.182 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-13T05:16:10.232 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.076) 0:01:08.182 ******** 2019-10-13T05:16:10.305 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-13T05:16:10.307 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:16:10.356 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.123) 0:01:08.305 ******** 2019-10-13T05:16:10.384 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:16:10.386 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-13T05:16:10.387 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:10.428 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.072) 0:01:08.378 ******** 2019-10-13T05:16:10.466 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.038) 0:01:08.416 ******** 2019-10-13T05:16:10.473 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:10.517 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.050) 0:01:08.467 ******** 2019-10-13T05:16:10.524 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-13T05:16:10.525 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:16:10.569 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.052) 0:01:08.519 ******** 2019-10-13T05:16:10.576 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T05:16:10.621 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.051) 0:01:08.571 ******** 2019-10-13T05:16:10.628 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-13T05:16:10.629 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:10.673 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.052) 0:01:08.623 ******** 2019-10-13T05:16:10.679 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:10.723 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:10 +0000 (0:00:00.050) 0:01:08.674 ******** 2019-10-13T05:16:11.003 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-13T05:16:11.003 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-13T05:16:11.045 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.321) 0:01:08.995 ******** 2019-10-13T05:16:11.073 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-13T05:16:11.120 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.075) 0:01:09.071 ******** 2019-10-13T05:16:11.678 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-13T05:16:11.680 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:14:42 UTC", "ActiveEnterTimestampMonotonic": "17064380", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target time-sync.target network.target -.mount sysinit.target nss-lookup.target remote-fs.target var-run.mount systemd-journald.socket basic.target system.slice systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:14:42 UTC", "AssertTimestampMonotonic": "17063801", "Before": "shutdown.target plymouth-quit.service getty@tty1.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:14:42 UTC", "ConditionTimestampMonotonic": "17063801", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-10-13T05:16:11.682 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "819", "ExecMainStartTimestamp": "Sun 2019-10-13 05:14:42 UTC", "ExecMainStartTimestampMonotonic": "17064346", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2019-10-13 05:14:42 UTC] ; stop_time=[n/a] ; pid=819 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 05:14:42 UTC", "InactiveExitTimestampMonotonic": "17064380", "InvocationID": "0970e4310a7440b691084f97db03ae95", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "inf 2019-10-13T05:16:11.683 INFO:teuthology.task.ansible.out:inity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "819", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "R 2019-10-13T05:16:11.685 INFO:teuthology.task.ansible.out:estrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:14:42 UTC", "StateChangeTimestampMonotonic": "17064380", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:14:42 UTC", "WatchdogTimestampMonotonic": "17064379", "WatchdogUSec": "0"}} 2019-10-13T05:16:11.724 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.603) 0:01:09.674 ******** 2019-10-13T05:16:11.735 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:14:23 UTC", "ActiveEnterTimestampMonotonic": "16763686", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target nss-lookup.target remote-fs.target -.mount network.target local-fs.target basic.target var-run.mount systemd-journald.socket system.slice sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:14:23 UTC", "AssertTimestampMonotonic": "16763101", "Before": "xdm.service getty@tty1.service multi-user.target plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:14:23 UTC", "ConditionTimestampMonotonic": "16763101", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-10-13T05:16:11.737 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "800", "ExecMainStartTimestamp": "Sun 2019-10-13 05:14:23 UTC", "ExecMainStartTimestampMonotonic": "16763658", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2019-10-13 05:14:23 UTC] ; stop_time=[n/a] ; pid=800 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 05:14:23 UTC", "InactiveExitTimestampMonotonic": "16763686", "InvocationID": "b3dd55315a2a49488d808807ffea8d36", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "inf 2019-10-13T05:16:11.738 INFO:teuthology.task.ansible.out:inity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "800", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "R 2019-10-13T05:16:11.740 INFO:teuthology.task.ansible.out:estrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:14:23 UTC", "StateChangeTimestampMonotonic": "16763686", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:14:23 UTC", "WatchdogTimestampMonotonic": "16763685", "WatchdogUSec": "0"}} 2019-10-13T05:16:11.783 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.058) 0:01:09.733 ******** 2019-10-13T05:16:11.788 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-13T05:16:11.789 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:11.831 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.048) 0:01:09.781 ******** 2019-10-13T05:16:11.837 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:11.881 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.049) 0:01:09.831 ******** 2019-10-13T05:16:11.886 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-13T05:16:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:11.943 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.062) 0:01:09.893 ******** 2019-10-13T05:16:11.950 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:11.951 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:11.994 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:11 +0000 (0:00:00.051) 0:01:09.945 ******** 2019-10-13T05:16:12.000 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-13T05:16:12.002 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.042 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:09.992 ******** 2019-10-13T05:16:12.049 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.090 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.048) 0:01:10.040 ******** 2019-10-13T05:16:12.096 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-13T05:16:12.097 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.139 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.049) 0:01:10.090 ******** 2019-10-13T05:16:12.145 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.189 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.049) 0:01:10.139 ******** 2019-10-13T05:16:12.194 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-13T05:16:12.196 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.236 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.186 ******** 2019-10-13T05:16:12.242 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.288 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.052) 0:01:10.239 ******** 2019-10-13T05:16:12.294 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-13T05:16:12.295 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.336 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.287 ******** 2019-10-13T05:16:12.342 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.384 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.334 ******** 2019-10-13T05:16:12.392 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.434 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.050) 0:01:10.385 ******** 2019-10-13T05:16:12.440 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.483 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.048) 0:01:10.433 ******** 2019-10-13T05:16:12.488 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-13T05:16:12.490 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.530 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.480 ******** 2019-10-13T05:16:12.536 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.577 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.528 ******** 2019-10-13T05:16:12.583 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-13T05:16:12.584 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.626 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.048) 0:01:10.576 ******** 2019-10-13T05:16:12.632 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.673 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.624 ******** 2019-10-13T05:16:12.679 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-13T05:16:12.679 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.720 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.046) 0:01:10.670 ******** 2019-10-13T05:16:12.726 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:12.768 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.047) 0:01:10.718 ******** 2019-10-13T05:16:12.775 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-13T05:16:12.816 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.048) 0:01:10.767 ******** 2019-10-13T05:16:12.823 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-13T05:16:12.875 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.058) 0:01:10.826 ******** 2019-10-13T05:16:12.881 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-13T05:16:12.882 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-13T05:16:12.926 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.050) 0:01:10.876 ******** 2019-10-13T05:16:12.933 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-13T05:16:12.977 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:12 +0000 (0:00:00.050) 0:01:10.927 ******** 2019-10-13T05:16:12.982 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-13T05:16:12.983 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-13T05:16:13.029 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.052) 0:01:10.979 ******** 2019-10-13T05:16:13.036 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-13T05:16:13.084 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.054) 0:01:11.034 ******** 2019-10-13T05:16:13.090 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-13T05:16:13.090 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T05:16:13.141 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.057) 0:01:11.091 ******** 2019-10-13T05:16:13.147 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T05:16:13.189 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.048) 0:01:11.139 ******** 2019-10-13T05:16:13.278 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-13T05:16:13.280 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T05:16:13.321 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.131) 0:01:11.271 ******** 2019-10-13T05:16:13.327 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T05:16:13.369 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.047) 0:01:11.319 ******** 2019-10-13T05:16:13.458 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-13T05:16:13.460 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T05:16:13.500 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.131) 0:01:11.450 ******** 2019-10-13T05:16:13.517 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T05:16:13.564 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.063) 0:01:11.514 ******** 2019-10-13T05:16:13.569 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-13T05:16:13.570 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:13.619 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.055) 0:01:11.570 ******** 2019-10-13T05:16:13.626 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:13.638 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:13.667 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.047) 0:01:11.617 ******** 2019-10-13T05:16:13.672 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-13T05:16:13.673 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:13.685 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:13.715 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.047) 0:01:11.665 ******** 2019-10-13T05:16:13.721 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:13.764 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.048) 0:01:11.714 ******** 2019-10-13T05:16:13.864 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-13T05:16:13.865 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-13T05:16:13.907 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.143) 0:01:11.857 ******** 2019-10-13T05:16:13.945 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-13T05:16:13.997 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:13 +0000 (0:00:00.090) 0:01:11.948 ******** 2019-10-13T05:16:14.359 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-10-13T05:16:14.360 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-13T05:16:14.402 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:14 +0000 (0:00:00.404) 0:01:12.352 ******** 2019-10-13T05:16:14.419 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-13T05:16:14.468 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:14 +0000 (0:00:00.066) 0:01:12.419 ******** 2019-10-13T05:16:14.474 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-13T05:16:14.475 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi071"}, "changed": false} 2019-10-13T05:16:14.522 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:14 +0000 (0:00:00.052) 0:01:12.471 ******** 2019-10-13T05:16:14.528 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi129"}, "changed": false} 2019-10-13T05:16:14.576 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:14 +0000 (0:00:00.054) 0:01:12.526 ******** 2019-10-13T05:16:15.414 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-13T05:16:15.415 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi071.front.sepia.ceph.com", "ansible_hostname": "smithi071", "ansible_nodename": "smithi071"}, "changed": true, "name": "smithi071"} 2019-10-13T05:16:15.465 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:15 +0000 (0:00:00.888) 0:01:13.415 ******** 2019-10-13T05:16:15.515 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi129.front.sepia.ceph.com", "ansible_hostname": "smithi129", "ansible_nodename": "smithi129"}, "changed": true, "name": "smithi129"} 2019-10-13T05:16:15.564 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:15 +0000 (0:00:00.099) 0:01:13.514 ******** 2019-10-13T05:16:15.779 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-10-13T05:16:15.781 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-13T05:16:15.820 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:15 +0000 (0:00:00.256) 0:01:13.771 ******** 2019-10-13T05:16:15.879 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-13T05:16:15.920 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:15 +0000 (0:00:00.099) 0:01:13.870 ******** 2019-10-13T05:16:16.166 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-10-13T05:16:16.167 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-13T05:16:16.209 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.288) 0:01:14.159 ******** 2019-10-13T05:16:16.250 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-13T05:16:16.299 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.089) 0:01:14.249 ******** 2019-10-13T05:16:16.705 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:16.710 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-13T05:16:16.712 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T05:16:16.752 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.453) 0:01:14.703 ******** 2019-10-13T05:16:16.780 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-13T05:16:16.783 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:16.785 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:16.824 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.071) 0:01:14.774 ******** 2019-10-13T05:16:16.831 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-13T05:16:16.833 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T05:16:16.879 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.054) 0:01:14.829 ******** 2019-10-13T05:16:16.889 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:16.933 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.053) 0:01:14.883 ******** 2019-10-13T05:16:16.940 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-13T05:16:16.943 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:16.989 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:16 +0000 (0:00:00.055) 0:01:14.939 ******** 2019-10-13T05:16:16.996 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T05:16:16.997 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.060 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.070) 0:01:15.009 ******** 2019-10-13T05:16:17.067 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T05:16:17.067 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.127 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.068) 0:01:15.078 ******** 2019-10-13T05:16:17.134 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T05:16:17.135 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.200 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.072) 0:01:15.151 ******** 2019-10-13T05:16:17.212 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.269 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.068) 0:01:15.219 ******** 2019-10-13T05:16:17.275 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T05:16:17.277 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.321 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.052) 0:01:15.271 ******** 2019-10-13T05:16:17.328 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T05:16:17.330 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.372 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.051) 0:01:15.323 ******** 2019-10-13T05:16:17.381 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T05:16:17.383 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.437 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.064) 0:01:15.387 ******** 2019-10-13T05:16:17.443 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T05:16:17.445 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.487 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.049) 0:01:15.437 ******** 2019-10-13T05:16:17.493 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T05:16:17.495 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.536 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.049) 0:01:15.486 ******** 2019-10-13T05:16:17.542 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T05:16:17.544 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.597 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.060) 0:01:15.547 ******** 2019-10-13T05:16:17.603 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-13T05:16:17.647 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.050) 0:01:15.598 ******** 2019-10-13T05:16:17.653 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T05:16:17.657 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.697 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.049) 0:01:15.647 ******** 2019-10-13T05:16:17.703 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T05:16:17.745 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.048) 0:01:15.695 ******** 2019-10-13T05:16:17.751 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-13T05:16:17.793 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.047) 0:01:15.743 ******** 2019-10-13T05:16:17.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-13T05:16:17.800 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.840 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.047) 0:01:15.790 ******** 2019-10-13T05:16:17.846 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T05:16:17.890 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.049) 0:01:15.840 ******** 2019-10-13T05:16:17.902 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.938 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.048) 0:01:15.889 ******** 2019-10-13T05:16:17.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-13T05:16:17.946 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:17.986 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:17 +0000 (0:00:00.047) 0:01:15.936 ******** 2019-10-13T05:16:17.992 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-10-13T05:16:17.994 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.041 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.048) 0:01:15.985 ******** TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi129.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.042 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.083 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.048) 0:01:16.034 ******** 2019-10-13T05:16:18.089 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:16:18.090 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.132 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.048) 0:01:16.083 ******** 2019-10-13T05:16:18.138 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-10-13T05:16:18.140 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.180 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.047) 0:01:16.130 ******** 2019-10-13T05:16:18.185 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-10-13T05:16:18.187 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.226 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.046) 0:01:16.176 ******** 2019-10-13T05:16:18.232 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:16:18.234 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.273 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.046) 0:01:16.223 ******** 2019-10-13T05:16:18.277 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T05:16:18.280 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.320 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.043) 0:01:16.267 ******** 2019-10-13T05:16:18.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-10-13T05:16:18.324 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.364 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.047) 0:01:16.314 ******** 2019-10-13T05:16:18.370 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-13T05:16:18.372 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.412 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.046) 0:01:16.361 ******** 2019-10-13T05:16:18.417 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T05:16:18.418 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.464 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.053) 0:01:16.414 ******** 2019-10-13T05:16:18.470 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-13T05:16:18.472 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.526 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.050) 0:01:16.465 ******** TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.571 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.056) 0:01:16.521 ******** 2019-10-13T05:16:18.578 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi071.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.579 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.581 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.582 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.593 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.594 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.595 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi071.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.637 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.065) 0:01:16.587 ******** 2019-10-13T05:16:18.643 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.644 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.645 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.701 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.058) 0:01:16.646 ******** 2019-10-13T05:16:18.703 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.704 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.706 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.707 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.709 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.719 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.720 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T05:16:18.721 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi129.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.761 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.064) 0:01:16.710 ******** 2019-10-13T05:16:18.767 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.768 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.820 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.059) 0:01:16.770 ******** 2019-10-13T05:16:18.826 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T05:16:18.827 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.828 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.830 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:18.890 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.069) 0:01:16.840 ******** 2019-10-13T05:16:18.896 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T05:16:18.938 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.048) 0:01:16.888 ******** 2019-10-13T05:16:18.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T05:16:18.993 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:18 +0000 (0:00:00.055) 0:01:16.944 ******** 2019-10-13T05:16:19.000 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi129.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.002 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.003 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.005 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.048 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.054) 0:01:16.998 ******** 2019-10-13T05:16:19.055 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T05:16:19.057 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.105 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.057) 0:01:17.055 ******** 2019-10-13T05:16:19.112 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T05:16:19.160 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.055) 0:01:17.111 ******** 2019-10-13T05:16:19.167 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T05:16:19.169 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.224 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.063) 0:01:17.174 ******** 2019-10-13T05:16:19.233 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T05:16:19.235 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.279 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.055) 0:01:17.230 ******** 2019-10-13T05:16:19.286 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T05:16:19.287 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.335 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.056) 0:01:17.286 ******** 2019-10-13T05:16:19.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi129.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.342 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.383 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.047) 0:01:17.334 ******** 2019-10-13T05:16:19.390 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-13T05:16:19.391 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.432 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.048) 0:01:17.382 ******** 2019-10-13T05:16:19.438 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T05:16:19.440 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.480 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.047) 0:01:17.430 ******** 2019-10-13T05:16:19.485 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-13T05:16:19.486 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.526 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.046) 0:01:17.476 ******** 2019-10-13T05:16:19.534 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.589 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.062) 0:01:17.539 ******** 2019-10-13T05:16:19.617 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-13T05:16:19.619 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:19.689 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.099) 0:01:17.639 ******** 2019-10-13T05:16:19.698 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-13T05:16:19.699 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005080", "end": "2019-10-13 05:16:19.909897", "rc": 0, "start": "2019-10-13 05:16:19.904817", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:16:19.746 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.057) 0:01:17.696 ******** 2019-10-13T05:16:19.773 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T05:16:19.813 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.067) 0:01:17.763 ******** 2019-10-13T05:16:19.851 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-13T05:16:19.852 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005086", "end": "2019-10-13 05:16:20.063390", "rc": 0, "start": "2019-10-13 05:16:20.058304", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:16:19.898 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.085) 0:01:17.848 ******** 2019-10-13T05:16:19.926 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T05:16:19.983 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:19 +0000 (0:00:00.084) 0:01:17.933 ******** 2019-10-13T05:16:20.165 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T05:16:20.169 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-10-13T05:16:20.218 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:20 +0000 (0:00:00.235) 0:01:18.169 ******** 2019-10-13T05:16:20.256 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:20.303 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:20 +0000 (0:00:00.084) 0:01:18.253 ******** 2019-10-13T05:16:20.319 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T05:16:20.320 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-10-13T05:16:20.360 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:20 +0000 (0:00:00.056) 0:01:18.310 ******** 2019-10-13T05:16:20.387 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-13T05:16:20.388 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:16:20.426 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:20 +0000 (0:00:00.066) 0:01:18.376 ******** 2019-10-13T05:16:21.587 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-13T05:16:21.588 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-13T05:16:21.630 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-13T05:16:22.459 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-13T05:16:22.502 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:22 +0000 (0:00:02.075) 0:01:20.453 ******** 2019-10-13T05:16:22.519 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-13T05:16:22.591 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:22 +0000 (0:00:00.088) 0:01:20.541 ******** 2019-10-13T05:16:23.449 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-13T05:16:23.471 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:16:23.496 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.905) 0:01:21.447 ******** 2019-10-13T05:16:23.514 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:16:23.558 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.061) 0:01:21.508 ******** 2019-10-13T05:16:23.564 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-13T05:16:23.612 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.053) 0:01:21.562 ******** 2019-10-13T05:16:23.676 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.063) 0:01:21.626 ******** 2019-10-13T05:16:23.808 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-13T05:16:23.810 INFO:teuthology.task.ansible.out: [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036030", "end": "2019-10-13 05:16:24.015466", "rc": 0, "start": "2019-10-13 05:16:23.979436", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:16:23.856 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.180) 0:01:21.806 ******** 2019-10-13T05:16:23.863 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.024634", "end": "2019-10-13 05:16:24.075193", "rc": 0, "start": "2019-10-13 05:16:24.050559", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:16:23.912 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:23 +0000 (0:00:00.053) 0:01:21.860 ******** 2019-10-13T05:16:26.576 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-13T05:16:26.576 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:16:26.618 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:26 +0000 (0:00:02.707) 0:01:24.567 ******** 2019-10-13T05:16:26.625 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:16:26.638 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:26.666 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:26 +0000 (0:00:00.049) 0:01:24.617 ******** 2019-10-13T05:16:26.701 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:36.418 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-13T05:16:36.419 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 92 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-13T05:16:36.421 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-13T05:16:36.423 INFO:teuthology.task.ansible.out:6 to remove and 92 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:36.463 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:36 +0000 (0:00:09.797) 0:01:34.414 ******** 2019-10-13T05:16:36.471 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 92 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-13T05:16:36.472 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-13T05:16:36.474 INFO:teuthology.task.ansible.out:6 to remove and 92 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:36.491 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:36.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:36 +0000 (0:00:00.056) 0:01:34.470 ******** 2019-10-13T05:16:36.549 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:59.114 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T05:16:59.115 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-13T05:16:59.117 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (974 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-13T05:16:59.118 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-13T05:16:59.120 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 2019-10-13T05:16:59.121 INFO:teuthology.task.ansible.out: libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (974 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading dat 2019-10-13T05:16:59.123 INFO:teuthology.task.ansible.out:abase ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting up apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubunt 2019-10-13T05:16:59.124 INFO:teuthology.task.ansible.out:u4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:59.172 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:59 +0000 (0:00:22.651) 0:01:57.122 ******** 2019-10-13T05:16:59.199 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T05:16:59.243 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:59 +0000 (0:00:00.071) 0:01:57.194 ******** 2019-10-13T05:16:59.267 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:16:59.600 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T05:16:59.602 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-13T05:16:59.604 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (963 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-13T05:16:59.605 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-13T05:16:59.607 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 2019-10-13T05:16:59.608 INFO:teuthology.task.ansible.out: libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (963 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading dat 2019-10-13T05:16:59.611 INFO:teuthology.task.ansible.out:abase ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting up apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubunt 2019-10-13T05:16:59.613 INFO:teuthology.task.ansible.out:u4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:16:59.657 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:59 +0000 (0:00:00.413) 0:01:57.607 ******** 2019-10-13T05:16:59.686 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T05:16:59.735 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:16:59 +0000 (0:00:00.078) 0:01:57.685 ******** 2019-10-13T05:16:59.759 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:12.390 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:12.391 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages] ********************************************* 2019-10-13T05:17:12.395 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-13T05:17:12.397 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 5s (2085 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-13T05:17:12.398 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-13T05:17:12.400 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 5s (2085 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files a 2019-10-13T05:17:12.401 INFO:teuthology.task.ansible.out:nd directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:17:12.449 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:12 +0000 (0:00:12.712) 0:02:10.398 ******** 2019-10-13T05:17:12.474 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-13T05:17:12.477 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 5s (2264 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-13T05:17:12.487 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-13T05:17:12.489 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http:// 2019-10-13T05:17:12.491 INFO:teuthology.task.ansible.out:archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 5s (2264 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_ 2019-10-13T05:17:12.492 INFO:teuthology.task.ansible.out:amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} [DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:12.546 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:12 +0000 (0:00:00.097) 0:02:10.496 ******** 2019-10-13T05:17:12.568 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:13.537 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-13T05:17:13.539 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:17:13.541 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:17:13.597 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:01.050) 0:02:11.547 ******** 2019-10-13T05:17:13.637 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:00.040) 0:02:11.588 ******** 2019-10-13T05:17:13.740 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-13T05:17:13.742 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-13T05:17:13.787 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:00.149) 0:02:11.738 ******** 2019-10-13T05:17:13.795 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-13T05:17:13.813 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:13.843 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:00.055) 0:02:11.793 ******** 2019-10-13T05:17:13.848 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-13T05:17:13.849 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:13.865 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:13.904 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:00.052) 0:02:11.845 ******** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:13.953 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:13 +0000 (0:00:00.057) 0:02:11.903 ******** 2019-10-13T05:17:14.055 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-13T05:17:14.057 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-13T05:17:14.104 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.151) 0:02:12.055 ******** 2019-10-13T05:17:14.111 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-13T05:17:14.157 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.052) 0:02:12.107 ******** 2019-10-13T05:17:14.463 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-13T05:17:14.464 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-13T05:17:14.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.363) 0:02:12.471 ******** 2019-10-13T05:17:14.527 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-13T05:17:14.581 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.060) 0:02:12.531 ******** 2019-10-13T05:17:14.697 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-13T05:17:14.698 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T05:17:14.755 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.174) 0:02:12.705 ******** 2019-10-13T05:17:14.763 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T05:17:14.811 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.056) 0:02:12.761 ******** 2019-10-13T05:17:14.933 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-13T05:17:14.934 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:14.976 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:14 +0000 (0:00:00.164) 0:02:12.926 ******** 2019-10-13T05:17:14.993 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:15.035 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.059) 0:02:12.985 ******** 2019-10-13T05:17:15.324 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T05:17:15.326 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:16:59 UTC", "ActiveEnterTimestampMonotonic": "153305440", "ActiveExitTimestamp": "Sun 2019-10-13 05:16:46 UTC", "ActiveExitTimestampMonotonic": "140712442", "ActiveState": "active", "After": "-.mount systemd-journald.socket remote-fs.target nss-lookup.target sysinit.target system.slice systemd-tmpfiles-setup.service network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:16:58 UTC", "AssertTimestampMonotonic": "153283554", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:16:58 UTC", "ConditionTimestampMonotonic": "153283553", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-13T05:17:15.328 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6657", "ExecMainStartTimestamp": "Sun 2019-10-13 05:16:59 UTC", "ExecMainStartTimestampMonotonic": "153305429", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sun 2019-10-13 05:16:46 UTC", "InactiveEnterTimestampMonotonic": "140780170", "InactiveExitTimestamp": "Sun 2019-10-13 05:16:58 UTC", "InactiveExitTimestampMonotonic": "153284169", "InvocationID": "55730ae174ff4e998248329aeb404b15", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-13T05:17:15.329 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6657", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-13T05:17:15.331 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:16:59 UTC", "StateChangeTimestampMonotonic": "153305440", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:16:59 UTC", "WatchdogTimestampMonotonic": "153305439", "WatchdogUSec": "0"}} 2019-10-13T05:17:15.380 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.345) 0:02:13.330 ******** 2019-10-13T05:17:15.407 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T05:17:15.408 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.448 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.067) 0:02:13.398 ******** 2019-10-13T05:17:15.485 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T05:17:15.487 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.526 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.077) 0:02:13.476 ******** 2019-10-13T05:17:15.552 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T05:17:15.554 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.599 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.073) 0:02:13.550 ******** 2019-10-13T05:17:15.626 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T05:17:15.628 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.667 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.067) 0:02:13.617 ******** 2019-10-13T05:17:15.694 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T05:17:15.695 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.742 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.075) 0:02:13.692 ******** 2019-10-13T05:17:15.779 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T05:17:15.781 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.820 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.077) 0:02:13.770 ******** 2019-10-13T05:17:15.841 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:15.900 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-13T05:17:15.901 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:15.948 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:15 +0000 (0:00:00.128) 0:02:13.898 ******** 2019-10-13T05:17:15.987 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T05:17:16.036 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.087) 0:02:13.986 ******** 2019-10-13T05:17:16.055 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:16.074 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T05:17:16.076 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.127 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.091) 0:02:14.077 ******** 2019-10-13T05:17:16.150 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:16.188 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi071.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.218 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.091) 0:02:14.168 ******** 2019-10-13T05:17:16.237 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:16.266 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T05:17:16.267 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.309 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.090) 0:02:14.259 ******** 2019-10-13T05:17:16.336 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T05:17:16.338 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.386 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.077) 0:02:14.337 ******** 2019-10-13T05:17:16.394 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T05:17:16.397 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:16:58 UTC", "ActiveEnterTimestampMonotonic": "171792385", "ActiveExitTimestamp": "Sun 2019-10-13 05:16:45 UTC", "ActiveExitTimestampMonotonic": "159176635", "ActiveState": "active", "After": "basic.target systemd-journald.socket nss-lookup.target remote-fs.target systemd-tmpfiles-setup.service sysinit.target -.mount network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:16:58 UTC", "AssertTimestampMonotonic": "171772963", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:16:58 UTC", "ConditionTimestampMonotonic": "171772963", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-13T05:17:16.399 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6631", "ExecMainStartTimestamp": "Sun 2019-10-13 05:16:58 UTC", "ExecMainStartTimestampMonotonic": "171792371", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sun 2019-10-13 05:16:45 UTC", "InactiveEnterTimestampMonotonic": "159252176", "InactiveExitTimestamp": "Sun 2019-10-13 05:16:58 UTC", "InactiveExitTimestampMonotonic": "171773674", "InvocationID": "eb7cabfe3d78481bab3d5db38a720e01", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-13T05:17:16.400 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6631", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-13T05:17:16.402 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:16:58 UTC", "StateChangeTimestampMonotonic": "171792385", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:16:58 UTC", "WatchdogTimestampMonotonic": "171792384", "WatchdogUSec": "0"}} 2019-10-13T05:17:16.443 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.056) 0:02:14.393 ******** 2019-10-13T05:17:16.449 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T05:17:16.450 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.496 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.052) 0:02:14.446 ******** 2019-10-13T05:17:16.502 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T05:17:16.504 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.550 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.053) 0:02:14.500 ******** 2019-10-13T05:17:16.555 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:16.557 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.600 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.050) 0:02:14.550 ******** 2019-10-13T05:17:16.605 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T05:17:16.606 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.648 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.047) 0:02:14.598 ******** 2019-10-13T05:17:16.653 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T05:17:16.654 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.695 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.047) 0:02:14.645 ******** 2019-10-13T05:17:16.700 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T05:17:16.701 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.745 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.049) 0:02:14.695 ******** 2019-10-13T05:17:16.750 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T05:17:16.752 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.797 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.052) 0:02:14.747 ******** 2019-10-13T05:17:16.802 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T05:17:16.803 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.844 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.047) 0:02:14.794 ******** 2019-10-13T05:17:16.849 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T05:17:16.850 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.891 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.046) 0:02:14.841 ******** 2019-10-13T05:17:16.897 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.938 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.046) 0:02:14.888 ******** 2019-10-13T05:17:16.944 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T05:17:16.945 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:16.985 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:16 +0000 (0:00:00.047) 0:02:14.935 ******** 2019-10-13T05:17:16.999 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.035 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.049) 0:02:14.986 ******** 2019-10-13T05:17:17.041 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T05:17:17.042 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.055 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:17.085 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.049) 0:02:15.035 ******** 2019-10-13T05:17:17.102 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-13T05:17:17.103 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.146 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.061) 0:02:15.097 ******** 2019-10-13T05:17:17.152 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:17.153 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.197 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.051) 0:02:15.148 ******** 2019-10-13T05:17:17.203 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T05:17:17.246 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.049) 0:02:15.197 ******** 2019-10-13T05:17:17.254 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T05:17:17.254 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.267 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:17.296 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.049) 0:02:15.246 ******** 2019-10-13T05:17:17.301 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T05:17:17.302 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.343 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.046) 0:02:15.293 ******** 2019-10-13T05:17:17.347 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T05:17:17.348 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.361 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:17.389 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.045) 0:02:15.339 ******** 2019-10-13T05:17:17.394 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T05:17:17.394 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.438 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.049) 0:02:15.388 ******** 2019-10-13T05:17:17.443 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:17.444 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.458 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:17.496 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.054) 0:02:15.443 ******** 2019-10-13T05:17:17.500 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi129.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.541 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.048) 0:02:15.491 ******** 2019-10-13T05:17:17.548 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.588 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.046) 0:02:15.538 ******** 2019-10-13T05:17:17.593 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T05:17:17.595 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.638 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.048) 0:02:15.587 ******** 2019-10-13T05:17:17.643 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-13T05:17:17.643 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.690 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.048) 0:02:15.635 ******** 2019-10-13T05:17:17.692 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.739 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.054) 0:02:15.690 ******** 2019-10-13T05:17:17.773 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:17.775 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.814 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.074) 0:02:15.764 ******** 2019-10-13T05:17:17.830 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T05:17:17.831 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-13T05:17:17.872 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.058) 0:02:15.822 ******** 2019-10-13T05:17:17.880 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.921 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.048) 0:02:15.871 ******** 2019-10-13T05:17:17.947 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T05:17:17.952 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:17.988 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:17 +0000 (0:00:00.067) 0:02:15.939 ******** 2019-10-13T05:17:18.014 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T05:17:18.016 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.061 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.072) 0:02:16.011 ******** 2019-10-13T05:17:18.087 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T05:17:18.089 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.134 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.073) 0:02:16.085 ******** 2019-10-13T05:17:18.161 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T05:17:18.162 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.202 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.067) 0:02:16.152 ******** 2019-10-13T05:17:18.228 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:18.230 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.269 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.067) 0:02:16.220 ******** 2019-10-13T05:17:18.275 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T05:17:18.276 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-13T05:17:18.317 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.048) 0:02:16.268 ******** 2019-10-13T05:17:18.324 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T05:17:18.325 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.366 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.048) 0:02:16.317 ******** 2019-10-13T05:17:18.372 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T05:17:18.374 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.415 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.048) 0:02:16.365 ******** 2019-10-13T05:17:18.420 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T05:17:18.421 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.463 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.047) 0:02:16.413 ******** 2019-10-13T05:17:18.489 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T05:17:18.493 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.550 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.087) 0:02:16.501 ******** 2019-10-13T05:17:18.577 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T05:17:18.578 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.623 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.072) 0:02:16.573 ******** 2019-10-13T05:17:18.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-13T05:17:18.651 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:18.692 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.068) 0:02:16.642 ******** 2019-10-13T05:17:18.782 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T05:17:18.784 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-13T05:17:18.824 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.132) 0:02:16.774 ******** 2019-10-13T05:17:18.841 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T05:17:18.841 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-13T05:17:18.883 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:18 +0000 (0:00:00.058) 0:02:16.833 ******** 2019-10-13T05:17:19.167 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-13T05:17:19.168 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-13T05:17:19.208 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.324) 0:02:17.158 ******** 2019-10-13T05:17:19.224 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T05:17:19.225 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-13T05:17:19.272 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.064) 0:02:17.222 ******** 2019-10-13T05:17:19.300 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T05:17:19.301 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:19.341 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.069) 0:02:17.291 ******** 2019-10-13T05:17:19.575 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-13T05:17:19.577 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-13T05:17:19.622 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.280) 0:02:17.572 ******** 2019-10-13T05:17:19.700 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T05:17:19.700 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-13T05:17:19.741 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.118) 0:02:17.691 ******** 2019-10-13T05:17:19.757 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-13T05:17:19.757 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:19.798 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.057) 0:02:17.749 ******** 2019-10-13T05:17:19.825 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:19.866 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.068) 0:02:17.817 ******** 2019-10-13T05:17:19.892 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T05:17:19.894 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:19.940 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:19 +0000 (0:00:00.073) 0:02:17.890 ******** 2019-10-13T05:17:19.968 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T05:17:19.969 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.011 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.071) 0:02:17.962 ******** 2019-10-13T05:17:20.038 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T05:17:20.041 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.080 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.068) 0:02:18.030 ******** 2019-10-13T05:17:20.085 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-13T05:17:20.088 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-13T05:17:20.130 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.050) 0:02:18.080 ******** 2019-10-13T05:17:20.135 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-13T05:17:20.136 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.177 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.046) 0:02:18.127 ******** 2019-10-13T05:17:20.213 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:20.215 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T05:17:20.255 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.078) 0:02:18.205 ******** 2019-10-13T05:17:20.313 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-13T05:17:20.315 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.325 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.326 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.336 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.347 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.369 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.371 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.386 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.388 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.410 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-13T05:17:20.412 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.423 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.438 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.439 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.450 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.515 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.259) 0:02:18.465 ******** 2019-10-13T05:17:20.522 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-13T05:17:20.562 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.047) 0:02:18.513 ******** 2019-10-13T05:17:20.674 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004169", "end": "2019-10-13 05:17:20.650166", "rc": 0, "start": "2019-10-13 05:17:20.645997", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-13T05:17:20.721 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.158) 0:02:18.671 ******** 2019-10-13T05:17:20.727 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-13T05:17:20.728 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:20.769 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.048) 0:02:18.719 ******** 2019-10-13T05:17:20.773 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T05:17:20.775 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.819 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.050) 0:02:18.770 ******** 2019-10-13T05:17:20.826 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-13T05:17:20.828 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.868 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.048) 0:02:18.818 ******** 2019-10-13T05:17:20.874 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.915 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.046) 0:02:18.865 ******** 2019-10-13T05:17:20.920 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T05:17:20.922 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:20.962 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:20 +0000 (0:00:00.046) 0:02:18.911 ******** 2019-10-13T05:17:20.969 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.007 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.045) 0:02:18.957 ******** 2019-10-13T05:17:21.013 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T05:17:21.014 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.022 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:21.054 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.046) 0:02:19.003 ******** 2019-10-13T05:17:21.079 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T05:17:21.081 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.118 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.064) 0:02:19.068 ******** 2019-10-13T05:17:21.154 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-13T05:17:21.156 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.200 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.082) 0:02:19.150 ******** 2019-10-13T05:17:21.237 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:21.238 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T05:17:21.277 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.076) 0:02:19.227 ******** 2019-10-13T05:17:21.324 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-13T05:17:21.329 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.336 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.348 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.349 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.359 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.370 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.371 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.382 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.393 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.404 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-13T05:17:21.415 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.427 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.428 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:21.429 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.442 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.492 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.214) 0:02:19.442 ******** 2019-10-13T05:17:21.494 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:21.648 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005427", "end": "2019-10-13 05:17:21.619180", "rc": 0, "start": "2019-10-13 05:17:21.613753", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-13T05:17:21.689 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.196) 0:02:19.639 ******** 2019-10-13T05:17:21.736 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T05:17:21.737 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.783 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.094) 0:02:19.733 ******** 2019-10-13T05:17:21.819 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-13T05:17:21.821 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.859 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.076) 0:02:19.810 ******** 2019-10-13T05:17:21.896 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-10-13T05:17:21.898 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:21.939 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.079) 0:02:19.889 ******** 2019-10-13T05:17:21.944 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-13T05:17:21.945 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1570943518, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:17:21.958 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:21.988 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:21 +0000 (0:00:00.049) 0:02:19.939 ******** 2019-10-13T05:17:21.991 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:22.025 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T05:17:22.026 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.072 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.083) 0:02:20.022 ******** 2019-10-13T05:17:22.109 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T05:17:22.110 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={u'block_used': 13654305, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440803, u'size_available': 927501529088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440803, "block_size": 4096, "block_total": 240095108, "block_used": 13654305, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501529088, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-13T05:17:22.153 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.081) 0:02:20.104 ******** 2019-10-13T05:17:22.308 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010956", "end": "2019-10-13 05:17:22.516287", "rc": 0, "start": "2019-10-13 05:17:22.505331", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:22.345 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.191) 0:02:20.295 ******** 2019-10-13T05:17:22.394 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T05:17:22.396 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'block_used': 13654305, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440803, u'size_available': 927501529088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440803, "block_size": 4096, "block_total": 240095108, "block_used": 13654305, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501529088, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.434 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.088) 0:02:20.384 ******** 2019-10-13T05:17:22.460 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T05:17:22.500 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.065) 0:02:20.450 ******** 2019-10-13T05:17:22.537 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T05:17:22.539 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QJJR', u'wwn-0x5000c50091e19c4b'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QJJR', u'wwn': u'0x5000c50091e19c4b', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', u'wwn-0x5000c50091e19c4b-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T05:17:22.540 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.562 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.563 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.575 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.588 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.599 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.601 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.612 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.624 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:22.919 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi129.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1570943469, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T05:17:22.959 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:22 +0000 (0:00:00.459) 0:02:20.909 ******** 2019-10-13T05:17:22.996 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T05:17:22.998 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.038 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:23 +0000 (0:00:00.078) 0:02:20.988 ******** 2019-10-13T05:17:23.074 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T05:17:23.074 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-13T05:17:23.116 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:23 +0000 (0:00:00.078) 0:02:21.066 ******** 2019-10-13T05:17:23.248 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-13T05:17:23.250 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.006868", "end": "2019-10-13 05:17:23.432762", "rc": 0, "start": "2019-10-13 05:17:23.425894", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:23.290 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:23 +0000 (0:00:00.174) 0:02:21.240 ******** 2019-10-13T05:17:23.340 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T05:17:23.341 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.380 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:23 +0000 (0:00:00.089) 0:02:21.330 ******** 2019-10-13T05:17:23.416 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T05:17:23.463 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:23 +0000 (0:00:00.083) 0:02:21.413 ******** 2019-10-13T05:17:23.511 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T05:17:23.513 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGTA', u'wwn-0x5000c5009292d832'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XGTA', u'wwn': u'0x5000c5009292d832', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1', u'wwn-0x5000c5009292d832-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGTA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T05:17:23.514 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c5009292d832"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.516 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.529 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.541 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.542 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:23.554 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.566 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.568 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.580 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:23.592 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:24.395 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE', u'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', u'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.647344", "end": "2019-10-13 05:17:24.596488", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:22.949144", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T05:17:24.397 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:25.420 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN', u'nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.692504", "end": "2019-10-13 05:17:25.592207", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:23.899703", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T05:17:25.422 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:25.650 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.115033", "end": "2019-10-13 05:17:25.847290", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:24.732257", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:26.675 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.121322", "end": "2019-10-13 05:17:26.853462", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:25.732140", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:26.896 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.103247", "end": "2019-10-13 05:17:27.090554", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:25.987307", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:27.929 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.115048", "end": "2019-10-13 05:17:28.104741", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:26.989693", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:28.114 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.104916", "end": "2019-10-13 05:17:28.323906", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:27.218990", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:29.164 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.107202", "end": "2019-10-13 05:17:29.337464", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:28.230262", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:29.365 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ'], u'uuids': [u'aa2d087a-25a9-4433-9cd7-d3a8f3af4d00']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.110612", "end": "2019-10-13 05:17:29.568654", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "labels": [], "masters": [], "uuids": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:28.458042", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:30.412 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi'], u'uuids': [u'25096892-22de-44c3-a204-057f9b3d312e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.120749", "end": "2019-10-13 05:17:30.588014", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "labels": [], "masters": [], "uuids": ["25096892-22de-44c3-a204-057f9b3d312e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:29.467265", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:30.625 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.123147", "end": "2019-10-13 05:17:30.818900", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:29.695753", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:30.692 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:30 +0000 (0:00:07.227) 0:02:28.641 ******** 2019-10-13T05:17:30.739 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T05:17:30.740 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QJJR', u'wwn-0x5000c50091e19c4b'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QJJR', u'wwn': u'0x5000c50091e19c4b', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', u'wwn-0x5000c50091e19c4b-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T05:17:30.742 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.752 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.766 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.778 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.789 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.801 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.813 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.825 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:30.836 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.004 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE', u'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', u'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049758", "end": "2019-10-13 05:17:31.209801", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4n0doq-84WG-BHWC-dWwg-cwkM-LavG-xfEhuE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:31.160043", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-13T05:17:31.006 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0345608 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0345608 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:31.268 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149759", "end": "2019-10-13 05:17:31.466372", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6XyYnUKfxqjG3UqumEQo1PYfxET0dF4oa"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:31.316613", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147387 s, 783 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147387 s, 783 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:31.551 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.155074", "end": "2019-10-13 05:17:31.754684", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6t2tAQG1DI4rxboJA9hc0DWKbflZsUMVQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:31.599610", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150467 s, 767 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150467 s, 767 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:31.665 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T05:17:31.666 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.124525", "end": "2019-10-13 05:17:31.839824", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:30.715299", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T05:17:31.747 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:31 +0000 (0:00:01.055) 0:02:29.697 ******** 2019-10-13T05:17:31.794 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T05:17:31.796 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGTA', u'wwn-0x5000c5009292d832'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XGTA', u'wwn': u'0x5000c5009292d832', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1', u'wwn-0x5000c5009292d832-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA", "wwn-0x5000c5009292d832"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGTA-part1", "wwn-0x5000c5009292d832-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGTA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T05:17:31.797 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c5009292d832"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.809 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.811 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.824 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.150750", "end": "2019-10-13 05:17:32.030483", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6VGrdtacbvPZJfo5ferdubgeSGK011gCS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:31.879733", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147956 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147956 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:31.836 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.837 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.849 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.861 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.863 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:31.877 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:32.058 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN', u'nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049060", "end": "2019-10-13 05:17:32.227755", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2gR65k-HehB-h9VN-eVWH-Qhz4-gYgS-YbRZXG", "nvme-INTEL_SSDPEDMD400G4_CVFT623300HY400BGN", "nvme-nvme.8086-43564654363233333030485934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.178695", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-13T05:17:32.060 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0369971 s, 3.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0369971 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.102 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ'], u'uuids': [u'aa2d087a-25a9-4433-9cd7-d3a8f3af4d00']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154890", "end": "2019-10-13 05:17:32.296745", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6hGqAYmjzLAySUC7k2LLHQTJh77Z63QFZ"], "labels": [], "masters": [], "uuids": ["aa2d087a-25a9-4433-9cd7-d3a8f3af4d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.141855", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152156 s, 758 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152156 s, 758 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.324 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153335", "end": "2019-10-13 05:17:32.505315", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhGCTVuEcOEkeqjNeYNWkNH3Ors06p1tHg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.351980", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148793 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148793 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.379 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.146773", "end": "2019-10-13 05:17:32.566129", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6eYdgxr7epqIa1xy6TtNeVCAxKwGPjJ6zclBUgf3nrFngd7M7IXCCE7jvIgsisG8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.419356", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142443 s, 810 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142443 s, 810 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.440 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:32 +0000 (0:00:00.693) 0:02:30.391 ******** 2019-10-13T05:17:32.623 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.155417", "end": "2019-10-13 05:17:32.781575", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhIe0uWhE9ogFs9BTwAuSTG66hXT58LY0R"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.626158", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151662 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151662 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.702 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.121567", "end": "2019-10-13 05:17:32.900315", "rc": 0, "start": "2019-10-13 05:17:32.778748", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.743 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:32 +0000 (0:00:00.303) 0:02:30.694 ******** 2019-10-13T05:17:32.881 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T05:17:32.882 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.149076", "end": "2019-10-13 05:17:33.048611", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhrMuP9lFQDaOOYdRF32PiHv8vdwnd2FJG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:32.899535", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146193 s, 789 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146193 s, 789 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.946 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-13T05:17:32.948 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043445", "end": "2019-10-13 05:17:33.152725", "rc": 0, "start": "2019-10-13 05:17:33.109280", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:32.988 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:32 +0000 (0:00:00.244) 0:02:30.939 ******** 2019-10-13T05:17:33.016 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T05:17:33.060 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.071) 0:02:31.010 ******** 2019-10-13T05:17:33.098 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-13T05:17:33.100 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.144 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.083) 0:02:31.094 ******** 2019-10-13T05:17:33.152 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T05:17:33.153 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi'], u'uuids': [u'25096892-22de-44c3-a204-057f9b3d312e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.151607", "end": "2019-10-13 05:17:33.322341", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirhG61ZN1PzqHAWm141n3m3oBaURvXUuhDi"], "labels": [], "masters": [], "uuids": ["25096892-22de-44c3-a204-057f9b3d312e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:33.170734", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146998 s, 785 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146998 s, 785 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:33.176 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-13T05:17:33.178 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.218 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.073) 0:02:31.168 ******** 2019-10-13T05:17:33.244 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T05:17:33.246 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.288 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.070) 0:02:31.238 ******** 2019-10-13T05:17:33.325 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-13T05:17:33.326 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.371 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.081) 0:02:31.320 ******** 2019-10-13T05:17:33.429 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T05:17:33.431 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.151303", "end": "2019-10-13 05:17:33.603545", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZNRJpUvr7e2nB2Y3aCu67hCynu0jVirh0ldJdhmvu5mWBwYg37Yw4FMwzLs4V0Es"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 05:17:33.452242", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148729 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148729 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:33.495 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.124) 0:02:31.445 ******** 2019-10-13T05:17:33.500 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T05:17:33.501 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:33.587 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.092) 0:02:31.537 ******** 2019-10-13T05:17:33.638 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T05:17:33.639 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={u'block_used': 13654305, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440803, u'size_available': 927501529088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440803, "block_size": 4096, "block_total": 240095108, "block_used": 13654305, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501529088, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.688 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.101) 0:02:31.638 ******** 2019-10-13T05:17:33.716 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-13T05:17:33.718 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.762 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.073) 0:02:31.712 ******** 2019-10-13T05:17:33.771 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-13T05:17:33.771 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.106207", "end": "2019-10-13 05:17:33.920138", "rc": 0, "start": "2019-10-13 05:17:33.813931", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:33.816 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.053) 0:02:31.766 ******** 2019-10-13T05:17:33.823 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:33.824 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.868 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.052) 0:02:31.818 ******** 2019-10-13T05:17:33.899 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:33.938 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:33 +0000 (0:00:00.070) 0:02:31.889 ******** 2019-10-13T05:17:33.989 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T05:17:33.992 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.038 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.099) 0:02:31.988 ******** 2019-10-13T05:17:34.047 INFO:teuthology.task.ansible.out: 2019-10-13T05:17:34.050 INFO:teuthology.task.ansible.out:TASK [testnode : Check for physical volumes] *********************************** changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054021", "end": "2019-10-13 05:17:34.194277", "rc": 0, "start": "2019-10-13 05:17:34.140256", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:34.096 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.058) 0:02:32.046 ******** 2019-10-13T05:17:34.102 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-13T05:17:34.103 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.148 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.051) 0:02:32.098 ******** 2019-10-13T05:17:34.154 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T05:17:34.204 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.056) 0:02:32.154 ******** 2019-10-13T05:17:34.210 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.253 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.049) 0:02:32.204 ******** 2019-10-13T05:17:34.260 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-13T05:17:34.263 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.304 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.050) 0:02:32.254 ******** 2019-10-13T05:17:34.330 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-13T05:17:34.332 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.372 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.067) 0:02:32.322 ******** 2019-10-13T05:17:34.398 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T05:17:34.400 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.444 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.072) 0:02:32.394 ******** 2019-10-13T05:17:34.487 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.534 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.090) 0:02:32.485 ******** 2019-10-13T05:17:34.668 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T05:17:34.669 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T05:17:34.723 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.188) 0:02:32.673 ******** 2019-10-13T05:17:34.769 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T05:17:34.770 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.821 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.097) 0:02:32.771 ******** 2019-10-13T05:17:34.859 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:34.918 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.097) 0:02:32.869 ******** 2019-10-13T05:17:34.926 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T05:17:34.927 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} 2019-10-13T05:17:34.928 INFO:teuthology.task.ansible.out: [WARNING]: The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. 2019-10-13T05:17:34.971 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:34 +0000 (0:00:00.052) 0:02:32.922 ******** 2019-10-13T05:17:34.978 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:34.979 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:35.020 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.048) 0:02:32.970 ******** 2019-10-13T05:17:35.058 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:35.060 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:35.123 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.103) 0:02:33.073 ******** 2019-10-13T05:17:35.152 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T05:17:35.154 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:35.196 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.072) 0:02:33.146 ******** 2019-10-13T05:17:35.233 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-13T05:17:35.235 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:35.283 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.087) 0:02:33.233 ******** 2019-10-13T05:17:35.320 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-13T05:17:35.322 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:35.368 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.085) 0:02:33.319 ******** 2019-10-13T05:17:35.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T05:17:35.505 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:35.696 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T05:17:35.698 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} 2019-10-13T05:17:35.738 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:35 +0000 (0:00:00.369) 0:02:33.689 ******** 2019-10-13T05:17:35.744 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T05:17:35.745 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:35.976 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.029 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.238 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.249 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.462 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.511 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:36 +0000 (0:00:00.772) 0:02:34.461 ******** 2019-10-13T05:17:36.517 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.637 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T05:17:36.639 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T05:17:36.703 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:36 +0000 (0:00:00.192) 0:02:34.654 ******** 2019-10-13T05:17:36.711 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T05:17:36.715 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.749 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T05:17:36.751 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:36.863 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:36.929 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T05:17:36.931 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T05:17:36.976 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:36 +0000 (0:00:00.272) 0:02:34.926 ******** 2019-10-13T05:17:36.993 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T05:17:36.995 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.107 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T05:17:37.149 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.172) 0:02:35.099 ******** 2019-10-13T05:17:37.155 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T05:17:37.157 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.191 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:37.257 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.320 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.170) 0:02:35.270 ******** 2019-10-13T05:17:37.338 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.444 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.455 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-13T05:17:37.457 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:17:37.512 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.191) 0:02:35.462 ******** 2019-10-13T05:17:37.552 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-10-13T05:17:37.554 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-13T05:17:37.601 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.088) 0:02:35.550 ******** 2019-10-13T05:17:37.608 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T05:17:37.609 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.727 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T05:17:37.775 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.174) 0:02:35.725 ******** 2019-10-13T05:17:37.934 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-13T05:17:37.935 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:17:37.990 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:37 +0000 (0:00:00.214) 0:02:35.940 ******** 2019-10-13T05:17:38.043 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-10-13T05:17:38.045 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-13T05:17:38.086 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:38 +0000 (0:00:00.096) 0:02:36.036 ******** 2019-10-13T05:17:38.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-10-13T05:17:38.114 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true} 2019-10-13T05:17:38.164 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:38 +0000 (0:00:00.078) 0:02:36.114 ******** 2019-10-13T05:17:38.643 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-13T05:17:38.644 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-13T05:17:38.691 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:38 +0000 (0:00:00.527) 0:02:36.642 ******** 2019-10-13T05:17:38.833 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-13T05:17:38.835 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T05:17:38.877 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:38 +0000 (0:00:00.185) 0:02:36.827 ******** 2019-10-13T05:17:39.014 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-13T05:17:39.016 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:17:39.055 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.177) 0:02:37.005 ******** 2019-10-13T05:17:39.332 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi129.front.sepia.ceph.com] => {"changed": true} 2019-10-13T05:17:39.370 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.315) 0:02:37.320 ******** 2019-10-13T05:17:39.419 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-13T05:17:39.421 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-13T05:17:39.467 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.096) 0:02:37.417 ******** 2019-10-13T05:17:39.578 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-13T05:17:39.579 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-13T05:17:39.621 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.154) 0:02:37.571 ******** 2019-10-13T05:17:39.756 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-13T05:17:39.757 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T05:17:39.816 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.195) 0:02:37.767 ******** 2019-10-13T05:17:39.833 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-13T05:17:39.834 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-13T05:17:39.886 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.069) 0:02:37.836 ******** 2019-10-13T05:17:39.955 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-13T05:17:39.956 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T05:17:39.995 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:39 +0000 (0:00:00.109) 0:02:37.946 ******** 2019-10-13T05:17:40.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-13T05:17:40.365 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:40 +0000 (0:00:00.369) 0:02:38.316 ******** 2019-10-13T05:17:40.725 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-13T05:17:40.726 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-13T05:17:40.767 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:40 +0000 (0:00:00.402) 0:02:38.718 ******** 2019-10-13T05:17:40.772 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-13T05:17:40.773 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:17:40.815 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:40 +0000 (0:00:00.047) 0:02:38.765 ******** 2019-10-13T05:17:41.022 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-13T05:17:41.024 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045451", "end": "2019-10-13 05:17:41.229237", "rc": 0, "start": "2019-10-13 05:17:41.183786", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-13T05:17:41.062 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:41 +0000 (0:00:00.247) 0:02:39.013 ******** 2019-10-13T05:17:41.100 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-13T05:17:41.108 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:41.158 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:41 +0000 (0:00:00.095) 0:02:39.109 ******** 2019-10-13T05:17:41.187 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:41.225 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:41 +0000 (0:00:00.067) 0:02:39.176 ******** 2019-10-13T05:17:41.758 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-13T05:17:41.759 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:17:41.807 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:41 +0000 (0:00:00.581) 0:02:39.757 ******** 2019-10-13T05:17:42.024 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-13T05:17:42.025 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.044671", "end": "2019-10-13 05:17:42.198810", "rc": 0, "start": "2019-10-13 05:17:42.154139", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-13T05:17:42.069 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.262) 0:02:40.020 ******** 2019-10-13T05:17:42.075 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-10-13T05:17:42.076 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:17:42.120 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.050) 0:02:40.071 ******** 2019-10-13T05:17:42.126 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-13T05:17:42.127 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:42.173 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.052) 0:02:40.123 ******** 2019-10-13T05:17:42.178 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-13T05:17:42.179 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:42.246 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.072) 0:02:40.195 ******** 2019-10-13T05:17:42.255 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-13T05:17:42.256 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:42.305 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.059) 0:02:40.255 ******** 2019-10-13T05:17:42.564 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-13T05:17:42.565 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-13T05:17:42.606 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.301) 0:02:40.557 ******** 2019-10-13T05:17:42.633 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-13T05:17:42.635 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:42.675 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.068) 0:02:40.625 ******** 2019-10-13T05:17:42.871 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-13T05:17:42.873 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:14:50 UTC", "ActiveEnterTimestampMonotonic": "25287222", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket sysinit.target -.mount basic.target system.slice network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:14:42 UTC", "AssertTimestampMonotonic": "16852079", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:14:42 UTC", "ConditionTimestampMonotonic": "16852079", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-10-13T05:17:42.875 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1132", "ExecMainStartTimestamp": "Sun 2019-10-13 05:14:50 UTC", "ExecMainStartTimestampMonotonic": "25287197", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 05:14:42 UTC", "InactiveExitTimestampMonotonic": "16852599", "InvocationID": "929d7a0fdfe8470184d8a94e98b79fa6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-10-13T05:17:42.876 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1132", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:14:50 UTC", "StateChangeTimestampMonoton 2019-10-13T05:17:42.878 INFO:teuthology.task.ansible.out:ic": "25287222", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:14:50 UTC", "WatchdogTimestampMonotonic": "25287220", "WatchdogUSec": "0"}} 2019-10-13T05:17:42.915 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.239) 0:02:40.865 ******** 2019-10-13T05:17:42.942 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-13T05:17:42.944 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:42.983 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:42 +0000 (0:00:00.068) 0:02:40.933 ******** 2019-10-13T05:17:43.187 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-10-13T05:17:43.188 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:17:43.229 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.246) 0:02:41.179 ******** 2019-10-13T05:17:43.256 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-13T05:17:43.258 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:43.298 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.068) 0:02:41.248 ******** 2019-10-13T05:17:43.635 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-13T05:17:43.636 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-13T05:17:43.677 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.378) 0:02:41.627 ******** 2019-10-13T05:17:43.703 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-13T05:17:43.704 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:43.742 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.065) 0:02:41.693 ******** 2019-10-13T05:17:43.810 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-13T05:17:43.810 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"cache_update_time": 1570943518, "cache_updated": false, "changed": false} 2019-10-13T05:17:43.851 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.108) 0:02:41.801 ******** 2019-10-13T05:17:43.930 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-13T05:17:43.932 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 05:14:31 UTC", "ActiveEnterTimestampMonotonic": "24992109", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target network.target systemd-tmpfiles-setup.service systemd-journald.socket -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 05:14:23 UTC", "AssertTimestampMonotonic": "16520118", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 05:14:23 UTC", "ConditionTimestampMonotonic": "16520118", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-10-13T05:17:43.934 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1114", "ExecMainStartTimestamp": "Sun 2019-10-13 05:14:31 UTC", "ExecMainStartTimestampMonotonic": "24992097", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 05:14:23 UTC", "InactiveExitTimestampMonotonic": "16520713", "InvocationID": "46383d8aac774a09a904b2d472477326", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-10-13T05:17:43.936 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1114", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 05:14:31 UTC", "StateChangeTimestampMonoton 2019-10-13T05:17:43.938 INFO:teuthology.task.ansible.out:ic": "24992109", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 05:14:31 UTC", "WatchdogTimestampMonotonic": "24992108", "WatchdogUSec": "0"}} 2019-10-13T05:17:43.974 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:43 +0000 (0:00:00.123) 0:02:41.924 ******** 2019-10-13T05:17:43.990 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-13T05:17:43.991 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T05:17:44.031 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:44 +0000 (0:00:00.056) 0:02:41.981 ******** 2019-10-13T05:17:44.036 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-13T05:17:44.039 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:44.077 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:44 +0000 (0:00:00.046) 0:02:42.027 ******** 2019-10-13T05:17:44.347 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-13T05:17:44.348 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-13T05:17:44.398 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:44 +0000 (0:00:00.320) 0:02:42.348 ******** 2019-10-13T05:17:44.418 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:44.959 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-13T05:17:44.961 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"cache_update_time": 1570943469, "cache_updated": false, "changed": false} 2019-10-13T05:17:45.002 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:44 +0000 (0:00:00.604) 0:02:42.952 ******** 2019-10-13T05:17:45.132 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-13T05:17:45.134 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T05:17:45.174 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:45 +0000 (0:00:00.172) 0:02:43.124 ******** 2019-10-13T05:17:45.524 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-13T05:17:45.525 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T05:17:45.580 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:45 +0000 (0:00:00.405) 0:02:43.530 ******** 2019-10-13T05:17:45.589 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-13T05:17:45.591 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-13T05:17:45.635 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:45 +0000 (0:00:00.055) 0:02:43.586 ******** 2019-10-13T05:17:45.657 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:46.113 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false} 2019-10-13T05:17:46.156 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.520) 0:02:44.107 ******** 2019-10-13T05:17:46.185 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-13T05:17:46.186 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T05:17:46.228 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.072) 0:02:44.179 ******** 2019-10-13T05:17:46.380 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi129.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T05:17:46.421 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.192) 0:02:44.371 ******** 2019-10-13T05:17:46.432 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1570943451.3861241, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1570943449.7252944, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1570943449.7252944, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-10-13T05:17:46.470 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.048) 0:02:44.420 ******** 2019-10-13T05:17:46.873 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-13T05:17:46.875 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-13T05:17:46.913 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.442) 0:02:44.863 ******** 2019-10-13T05:17:46.919 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-13T05:17:46.920 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false} 2019-10-13T05:17:46.935 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:46.962 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:46 +0000 (0:00:00.049) 0:02:44.913 ******** 2019-10-13T05:17:46.968 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-13T05:17:46.971 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T05:17:47.010 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.047) 0:02:44.960 ******** 2019-10-13T05:17:47.015 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-13T05:17:47.017 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T05:17:47.059 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.048) 0:02:45.009 ******** 2019-10-13T05:17:47.065 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-13T05:17:47.066 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.106 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.047) 0:02:45.056 ******** 2019-10-13T05:17:47.203 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-13T05:17:47.204 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1570943416.8101127, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1570943415.313365, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1570943415.313365, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-10-13T05:17:47.260 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.153) 0:02:45.210 ******** 2019-10-13T05:17:47.266 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-10-13T05:17:47.267 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T05:17:47.308 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.048) 0:02:45.258 ******** 2019-10-13T05:17:47.397 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-13T05:17:47.399 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-13T05:17:47.440 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.132) 0:02:45.390 ******** 2019-10-13T05:17:47.446 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-13T05:17:47.447 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T05:17:47.456 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T05:17:47.489 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.048) 0:02:45.439 ******** 2019-10-13T05:17:47.495 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-13T05:17:47.496 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T05:17:47.537 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.047) 0:02:45.487 ******** 2019-10-13T05:17:47.543 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-13T05:17:47.545 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.588 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.051) 0:02:45.539 ******** 2019-10-13T05:17:47.596 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-13T05:17:47.597 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.650 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.061) 0:02:45.601 ******** 2019-10-13T05:17:47.660 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.663 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.707 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.056) 0:02:45.658 ******** 2019-10-13T05:17:47.745 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-13T05:17:47.747 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.767 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.768 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.769 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.782 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.824 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.116) 0:02:45.774 ******** 2019-10-13T05:17:47.831 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-10-13T05:17:47.832 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T05:17:47.878 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.051) 0:02:45.826 ******** 2019-10-13T05:17:47.882 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:47.929 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:47 +0000 (0:00:00.053) 0:02:45.879 ******** 2019-10-13T05:17:47.967 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.011 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.081) 0:02:45.961 ******** 2019-10-13T05:17:48.017 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-13T05:17:48.017 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T05:17:48.065 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.052) 0:02:46.014 ******** 2019-10-13T05:17:48.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-13T05:17:48.070 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.113 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.049) 0:02:46.063 ******** 2019-10-13T05:17:48.119 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-13T05:17:48.121 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.161 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.047) 0:02:46.111 ******** 2019-10-13T05:17:48.166 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-13T05:17:48.167 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.213 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.051) 0:02:46.162 ******** 2019-10-13T05:17:48.219 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-13T05:17:48.221 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.222 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.223 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.263 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.051) 0:02:46.214 ******** 2019-10-13T05:17:48.269 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-13T05:17:48.270 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.318 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.054) 0:02:46.268 ******** 2019-10-13T05:17:48.324 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-13T05:17:48.325 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.327 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.328 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi129.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.330 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.373 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.052) 0:02:46.321 ******** 2019-10-13T05:17:48.377 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.420 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.049) 0:02:46.371 ******** 2019-10-13T05:17:48.427 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T05:17:48.427 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.472 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.051) 0:02:46.422 ******** 2019-10-13T05:17:48.477 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-10-13T05:17:48.479 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.522 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.050) 0:02:46.473 ******** 2019-10-13T05:17:48.529 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.570 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.048) 0:02:46.521 ******** 2019-10-13T05:17:48.576 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T05:17:48.577 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.618 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.047) 0:02:46.568 ******** 2019-10-13T05:17:48.624 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-13T05:17:48.625 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.669 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.051) 0:02:46.619 ******** 2019-10-13T05:17:48.695 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-13T05:17:48.697 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.753 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.082) 0:02:46.702 ******** 2019-10-13T05:17:48.778 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-13T05:17:48.780 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.834 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.081) 0:02:46.784 ******** 2019-10-13T05:17:48.840 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T05:17:48.840 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009143", "end": "2019-10-13 05:17:48.977893", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 05:17:48.968750", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:48.862 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T05:17:48.864 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:48.909 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:48 +0000 (0:00:00.075) 0:02:46.860 ******** 2019-10-13T05:17:48.951 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi071.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004309", "end": "2019-10-13 05:17:49.133373", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 05:17:49.129064", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:49.006 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.096) 0:02:46.957 ******** 2019-10-13T05:17:49.039 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T05:17:49.041 INFO:teuthology.task.ansible.out:skipping: [smithi129.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T05:17:49.085 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.078) 0:02:47.035 ******** 2019-10-13T05:17:49.093 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T05:17:49.095 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005217", "end": "2019-10-13 05:17:49.259339", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 05:17:49.254122", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.191 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.005464", "end": "2019-10-13 05:17:49.383459", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 05:17:49.377995", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.240 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.155) 0:02:47.190 ******** 2019-10-13T05:17:49.260 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010845", "end": "2019-10-13 05:17:49.414360", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 05:17:49.403515", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.272 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T05:17:49.319 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.078) 0:02:47.269 ******** 2019-10-13T05:17:49.400 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T05:17:49.401 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004359", "end": "2019-10-13 05:17:49.548549", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 05:17:49.544190", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.434 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-13T05:17:49.434 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-13T05:17:49.521 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T05:17:49.522 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003733", "end": "2019-10-13 05:17:49.689734", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 05:17:49.686001", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.662 INFO:teuthology.task.ansible.out:ok: [smithi129.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.005789", "end": "2019-10-13 05:17:49.822469", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 05:17:49.816680", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T05:17:49.713 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.393) 0:02:47.663 ******** 2019-10-13T05:17:49.743 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T05:17:49.795 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 05:17:49 +0000 (0:00:00.082) 0:02:47.745 ******** 2019-10-13T05:17:49.926 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-13T05:17:49.926 INFO:teuthology.task.ansible.out:changed: [smithi129.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-13T05:17:50.050 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-10-13T05:17:50.059 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi071.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 smithi129.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 Sunday 13 October 2019 05:17:50 +0000 (0:00:00.261) 0:02:48.006 ******** =============================================================================== testnode : Upgrade packages -------------------------------------------- 22.65s users : Create all admin users with sudo access. ----------------------- 18.50s users : Update authorized_keys using the keys repo --------------------- 15.85s testnode : Install packages -------------------------------------------- 12.71s testnode : Ensure packages are not present. ----------------------------- 9.80s testnode : Zap all non-root disks --------------------------------------- 7.23s users : Remove revoked users -------------------------------------------- 3.78s common : Update apt cache ----------------------------------------------- 3.37s testnode : Update apt cache. -------------------------------------------- 2.71s testnode : Install apt keys --------------------------------------------- 2.08s Check for /usr/bin/python ----------------------------------------------- 1.92s common : Install nrpe package and dependencies (Ubuntu) ----------------- 1.79s ansible-managed : Add authorized keys for the ansible user. ------------- 1.53s users : Update authorized_keys for each user with literal keys ---------- 1.52s Gathering Facts --------------------------------------------------------- 1.41s testnode : Install non aarch64 packages ------------------------- 2019-10-13T05:17:50.061 INFO:teuthology.task.ansible.out:-------- 1.05s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.03s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.91s testnode : Install python-apt ------------------------------------------- 0.91s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.90s 2019-10-13T05:17:50.336 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2019-10-13T05:17:50.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:17:50.339 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi129.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:17:50.424 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:17:50.424 INFO:teuthology.orchestra.run.smithi129:> true 2019-10-13T05:17:50.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T05:17:50.596 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi071.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T05:17:50.680 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:17:50.680 INFO:teuthology.orchestra.run.smithi071:> true 2019-10-13T05:17:50.885 INFO:teuthology.run_tasks:Running task clock... 2019-10-13T05:17:50.901 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-10-13T05:17:50.902 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:17:50.902 INFO:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T05:17:50.968 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-13T05:17:50.968 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Command line: ntpd -gq 2019-10-13T05:17:50.970 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: proto: precision = 0.123 usec (-23) 2019-10-13T05:17:50.970 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: restrict ::: KOD does nothing without LIMITED. 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen and drop on 0 v6wildcard [::]:123 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen normally on 2 lo 127.0.0.1:123 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen normally on 3 enp3s0f1 172.21.15.129:123 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen normally on 4 lo [::1]:123 2019-10-13T05:17:50.971 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7261%5]:123 2019-10-13T05:17:50.972 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:51 ntpd[7849]: Listening on routing socket on fd #22 for interface updates 2019-10-13T05:17:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T05:17:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-13T05:17:53.970 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:54 ntpd[7849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:17:53.970 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:54 ntpd[7849]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:17:53.970 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:54 ntpd[7849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:17:53.970 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:54 ntpd[7849]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:17:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:56 ntpd[7849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:17:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:56 ntpd[7849]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:17:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:56 ntpd[7849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:17:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:56 ntpd[7849]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:17:57.972 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:58 ntpd[7849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:17:57.972 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:58 ntpd[7849]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:17:57.972 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:58 ntpd[7849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:17:57.972 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:58 ntpd[7849]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:17:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:18:00 ntpd[7849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:17:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:18:00 ntpd[7849]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:17:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:18:00 ntpd[7849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:17:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:18:00 ntpd[7849]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:17:59.974 INFO:teuthology.orchestra.run.smithi129.stderr:13 Oct 05:17:59 ntpd[7849]: can't open /var/log/ntpstats/loopstats.20191013: Permission denied 2019-10-13T05:17:59.974 INFO:teuthology.orchestra.run.smithi129.stdout:13 Oct 05:17:59 ntpd[7849]: ntpd: time set -0.192089 s 2019-10-13T05:17:59.974 INFO:teuthology.orchestra.run.smithi129.stdout:ntpd: time set -0.192089s 2019-10-13T05:18:00.017 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T05:18:00.017 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2019-10-13T05:18:00.017 INFO:teuthology.orchestra.run.smithi129.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:00.017 INFO:teuthology.orchestra.run.smithi129.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:00.017 INFO:teuthology.orchestra.run.smithi129.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:00.018 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:00.019 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:00.019 INFO:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T05:18:00.077 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-13T05:18:00.077 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Command line: ntpd -gq 2019-10-13T05:18:00.079 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: proto: precision = 0.065 usec (-24) 2019-10-13T05:18:00.079 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T05:18:00.079 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: restrict ::: KOD does nothing without LIMITED. 2019-10-13T05:18:00.079 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen and drop on 0 v6wildcard [::]:123 2019-10-13T05:18:00.079 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen normally on 2 lo 127.0.0.1:123 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen normally on 3 enp3s0f1 172.21.15.71:123 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen normally on 4 lo [::1]:123 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d765%5]:123 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:00 ntpd[7876]: Listening on routing socket on fd #22 for interface updates 2019-10-13T05:18:00.080 INFO:teuthology.orchestra.run.smithi071.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T05:18:00.081 INFO:teuthology.orchestra.run.smithi071.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-13T05:18:01.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:01 ntpd[7876]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:18:01.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:01 ntpd[7876]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:18:01.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:01 ntpd[7876]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:18:01.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:01 ntpd[7876]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:18:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:03 ntpd[7876]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:18:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:03 ntpd[7876]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:18:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:03 ntpd[7876]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:18:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:03 ntpd[7876]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:18:05.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:05 ntpd[7876]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:18:05.080 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:05 ntpd[7876]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:18:05.081 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:05 ntpd[7876]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:18:05.081 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:05 ntpd[7876]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:18:07.081 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:07 ntpd[7876]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T05:18:07.082 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:07 ntpd[7876]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T05:18:07.082 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:07 ntpd[7876]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T05:18:07.082 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:07 ntpd[7876]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T05:18:07.082 INFO:teuthology.orchestra.run.smithi071.stderr:13 Oct 05:18:07 ntpd[7876]: can't open /var/log/ntpstats/loopstats.20191013: Permission denied 2019-10-13T05:18:07.084 INFO:teuthology.orchestra.run.smithi071.stdout:13 Oct 05:18:07 ntpd[7876]: ntpd: time set -0.213338 s 2019-10-13T05:18:07.084 INFO:teuthology.orchestra.run.smithi071.stdout:ntpd: time set -0.213338s 2019-10-13T05:18:07.127 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T05:18:07.127 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2019-10-13T05:18:07.127 INFO:teuthology.orchestra.run.smithi071.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:07.128 INFO:teuthology.orchestra.run.smithi071.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:07.128 INFO:teuthology.orchestra.run.smithi071.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:07.128 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T05:18:07.128 INFO:teuthology.run_tasks:Running task install... 2019-10-13T05:18:07.144 DEBUG:teuthology.task.install:project ceph 2019-10-13T05:18:07.145 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}} 2019-10-13T05:18:07.145 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T05:18:07.145 INFO:teuthology.task.install:Using flavor: basic 2019-10-13T05:18:07.152 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-10-13T05:18:07.153 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2019-10-13T05:18:07.153 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:07.153 INFO:teuthology.orchestra.run.smithi129:> sudo lsb_release -is 2019-10-13T05:18:07.213 INFO:teuthology.orchestra.run.smithi129.stdout:Ubuntu 2019-10-13T05:18:07.218 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:18:07.218 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:07.218 INFO:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2019-10-13T05:18:07.219 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:07.219 INFO:teuthology.orchestra.run.smithi129:> sudo apt-key list | grep Ceph 2019-10-13T05:18:07.272 INFO:teuthology.orchestra.run.smithi071.stdout:Ubuntu 2019-10-13T05:18:07.277 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:18:07.278 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:07.278 INFO:teuthology.orchestra.run.smithi071:> sudo apt-key list | grep Ceph 2019-10-13T05:18:07.301 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-13T05:18:07.354 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-13T05:18:07.354 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph.com (release key) 2019-10-13T05:18:07.355 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:07.355 INFO:teuthology.orchestra.run.smithi129:> sudo lsb_release -is 2019-10-13T05:18:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-13T05:18:07.403 INFO:teuthology.orchestra.run.smithi129.stdout:Ubuntu 2019-10-13T05:18:07.407 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-13T05:18:07.407 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph.com (release key) 2019-10-13T05:18:07.408 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:07.408 INFO:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2019-10-13T05:18:07.410 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:18:07.410 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-10-13T05:18:07.410 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-10-13T05:18:07.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:18:07.463 INFO:teuthology.orchestra.run.smithi071.stdout:Ubuntu 2019-10-13T05:18:07.469 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:18:07.469 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-10-13T05:18:07.469 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-10-13T05:18:07.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:18:07.582 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default/ 2019-10-13T05:18:07.583 INFO:teuthology.task.install.deb:Package version is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:18:07.630 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default/ 2019-10-13T05:18:07.630 INFO:teuthology.task.install.deb:Package version is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:18:07.729 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:07.729 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-13T05:18:07.765 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:07.766 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-13T05:18:07.780 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:07.781 INFO:teuthology.orchestra.run.smithi129:> sudo apt-get update 2019-10-13T05:18:07.816 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:07.816 INFO:teuthology.orchestra.run.smithi071:> sudo apt-get update 2019-10-13T05:18:08.022 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T05:18:08.096 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic InRelease 2019-10-13T05:18:08.100 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic InRelease 2019-10-13T05:18:08.117 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-13T05:18:08.121 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-13T05:18:08.136 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T05:18:08.137 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:18:08.139 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T05:18:08.140 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-13T05:18:08.144 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-13T05:18:08.161 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-13T05:18:08.165 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-13T05:18:08.192 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,858 B] 2019-10-13T05:18:08.197 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,858 B] 2019-10-13T05:18:08.288 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T05:18:08.299 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T05:18:08.450 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T05:18:08.460 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T05:18:08.651 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 26.8 kB in 1s (44.4 kB/s) 2019-10-13T05:18:08.688 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 26.8 kB in 1s (40.2 kB/s) 2019-10-13T05:18:09.795 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:18:09.818 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:18:09.836 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:18:09.836 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6132-ga8ee8ac-1bionic ceph-daemon=15.0.0-6132-ga8ee8ac-1bionic ceph-mds=15.0.0-6132-ga8ee8ac-1bionic ceph-mgr=15.0.0-6132-ga8ee8ac-1bionic ceph-common=15.0.0-6132-ga8ee8ac-1bionic ceph-fuse=15.0.0-6132-ga8ee8ac-1bionic ceph-test=15.0.0-6132-ga8ee8ac-1bionic radosgw=15.0.0-6132-ga8ee8ac-1bionic python-ceph=15.0.0-6132-ga8ee8ac-1bionic libcephfs2=15.0.0-6132-ga8ee8ac-1bionic libcephfs-dev=15.0.0-6132-ga8ee8ac-1bionic librados2=15.0.0-6132-ga8ee8ac-1bionic librbd1=15.0.0-6132-ga8ee8ac-1bionic rbd-fuse=15.0.0-6132-ga8ee8ac-1bionic python3-cephfs=15.0.0-6132-ga8ee8ac-1bionic cephfs-shell=15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:18:09.862 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:18:09.863 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6132-ga8ee8ac-1bionic ceph-daemon=15.0.0-6132-ga8ee8ac-1bionic ceph-mds=15.0.0-6132-ga8ee8ac-1bionic ceph-mgr=15.0.0-6132-ga8ee8ac-1bionic ceph-common=15.0.0-6132-ga8ee8ac-1bionic ceph-fuse=15.0.0-6132-ga8ee8ac-1bionic ceph-test=15.0.0-6132-ga8ee8ac-1bionic radosgw=15.0.0-6132-ga8ee8ac-1bionic python-ceph=15.0.0-6132-ga8ee8ac-1bionic libcephfs2=15.0.0-6132-ga8ee8ac-1bionic libcephfs-dev=15.0.0-6132-ga8ee8ac-1bionic librados2=15.0.0-6132-ga8ee8ac-1bionic librbd1=15.0.0-6132-ga8ee8ac-1bionic rbd-fuse=15.0.0-6132-ga8ee8ac-1bionic python3-cephfs=15.0.0-6132-ga8ee8ac-1bionic cephfs-shell=15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:18:09.916 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:18:09.944 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:18:10.154 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:18:10.154 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:18:10.185 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:18:10.186 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:18:10.312 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-13T05:18:10.313 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T05:18:10.314 INFO:teuthology.orchestra.run.smithi071.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T05:18:10.315 INFO:teuthology.orchestra.run.smithi071.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T05:18:10.316 INFO:teuthology.orchestra.run.smithi071.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T05:18:10.317 INFO:teuthology.orchestra.run.smithi071.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T05:18:10.317 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados runc socat ubuntu-fan xmlstarlet 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-13T05:18:10.318 INFO:teuthology.orchestra.run.smithi071.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-13T05:18:10.319 INFO:teuthology.orchestra.run.smithi071.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-10-13T05:18:10.337 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2019-10-13T05:18:10.337 INFO:teuthology.orchestra.run.smithi129.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T05:18:10.338 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T05:18:10.338 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-13T05:18:10.338 INFO:teuthology.orchestra.run.smithi129.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-13T05:18:10.338 INFO:teuthology.orchestra.run.smithi129.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-13T05:18:10.339 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T05:18:10.339 INFO:teuthology.orchestra.run.smithi129.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-13T05:18:10.339 INFO:teuthology.orchestra.run.smithi129.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-13T05:18:10.340 INFO:teuthology.orchestra.run.smithi129.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-13T05:18:10.340 INFO:teuthology.orchestra.run.smithi129.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-13T05:18:10.340 INFO:teuthology.orchestra.run.smithi129.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T05:18:10.340 INFO:teuthology.orchestra.run.smithi129.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T05:18:10.341 INFO:teuthology.orchestra.run.smithi129.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T05:18:10.341 INFO:teuthology.orchestra.run.smithi129.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T05:18:10.341 INFO:teuthology.orchestra.run.smithi129.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T05:18:10.341 INFO:teuthology.orchestra.run.smithi129.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T05:18:10.341 INFO:teuthology.orchestra.run.smithi129.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T05:18:10.342 INFO:teuthology.orchestra.run.smithi129.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T05:18:10.342 INFO:teuthology.orchestra.run.smithi129.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T05:18:10.342 INFO:teuthology.orchestra.run.smithi129.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T05:18:10.342 INFO:teuthology.orchestra.run.smithi129.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T05:18:10.343 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-10-13T05:18:10.343 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados runc socat ubuntu-fan xmlstarlet 2019-10-13T05:18:10.345 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2019-10-13T05:18:10.346 INFO:teuthology.orchestra.run.smithi129.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-13T05:18:10.346 INFO:teuthology.orchestra.run.smithi129.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-13T05:18:10.346 INFO:teuthology.orchestra.run.smithi129.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-13T05:18:10.347 INFO:teuthology.orchestra.run.smithi129.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-13T05:18:10.347 INFO:teuthology.orchestra.run.smithi129.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-13T05:18:10.347 INFO:teuthology.orchestra.run.smithi129.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-13T05:18:10.347 INFO:teuthology.orchestra.run.smithi129.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-13T05:18:10.348 INFO:teuthology.orchestra.run.smithi129.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-13T05:18:10.348 INFO:teuthology.orchestra.run.smithi129.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-13T05:18:10.348 INFO:teuthology.orchestra.run.smithi129.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-13T05:18:10.349 INFO:teuthology.orchestra.run.smithi129.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-13T05:18:10.349 INFO:teuthology.orchestra.run.smithi129.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-10-13T05:18:10.419 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2019-10-13T05:18:10.419 INFO:teuthology.orchestra.run.smithi071.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-13T05:18:10.419 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T05:18:10.420 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T05:18:10.420 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon ceph-osd ceph-test cephfs-shell cgroupfs-mount containerd 2019-10-13T05:18:10.420 INFO:teuthology.orchestra.run.smithi071.stdout: dns-root-data dnsmasq-base docker.io formencode-i18n jq libcephfs-dev 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librgw2 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-10-13T05:18:10.421 INFO:teuthology.orchestra.run.smithi071.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T05:18:10.422 INFO:teuthology.orchestra.run.smithi071.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyperclip python3-rados radosgw rbd-fuse runc socat ubuntu-fan 2019-10-13T05:18:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet 2019-10-13T05:18:10.424 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages will be upgraded: 2019-10-13T05:18:10.424 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 librbd1 2019-10-13T05:18:10.446 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2019-10-13T05:18:10.446 INFO:teuthology.orchestra.run.smithi129.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-13T05:18:10.446 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon ceph-osd ceph-test cephfs-shell cgroupfs-mount containerd 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: dns-root-data dnsmasq-base docker.io formencode-i18n jq libcephfs-dev 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librgw2 2019-10-13T05:18:10.447 INFO:teuthology.orchestra.run.smithi129.stdout: libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T05:18:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T05:18:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T05:18:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T05:18:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T05:18:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-10-13T05:18:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip python3-rados radosgw rbd-fuse runc socat ubuntu-fan 2019-10-13T05:18:10.451 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet 2019-10-13T05:18:10.452 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2019-10-13T05:18:10.452 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 librbd1 2019-10-13T05:18:10.568 INFO:teuthology.orchestra.run.smithi071.stdout:2 upgraded, 119 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T05:18:10.568 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 183 MB of archives. 2019-10-13T05:18:10.568 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 869 MB of additional disk space will be used. 2019-10-13T05:18:10.569 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6132-ga8ee8ac-1bionic [1,489 kB] 2019-10-13T05:18:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:2 upgraded, 119 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T05:18:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 183 MB of archives. 2019-10-13T05:18:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 869 MB of additional disk space will be used. 2019-10-13T05:18:10.596 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6132-ga8ee8ac-1bionic [1,489 kB] 2019-10-13T05:18:10.607 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-13T05:18:10.637 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-13T05:18:10.721 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6132-ga8ee8ac-1bionic [2,966 kB] 2019-10-13T05:18:10.769 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-10-13T05:18:10.795 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6132-ga8ee8ac-1bionic [2,966 kB] 2019-10-13T05:18:10.799 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-10-13T05:18:10.836 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6132-ga8ee8ac-1bionic [426 kB] 2019-10-13T05:18:10.850 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [303 kB] 2019-10-13T05:18:10.878 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.2 kB] 2019-10-13T05:18:10.879 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [111 kB] 2019-10-13T05:18:10.883 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6132-ga8ee8ac-1bionic [10.5 kB] 2019-10-13T05:18:10.883 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6132-ga8ee8ac-1bionic [248 kB] 2019-10-13T05:18:10.891 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6132-ga8ee8ac-1bionic [4,490 kB] 2019-10-13T05:18:11.098 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6132-ga8ee8ac-1bionic [98.8 kB] 2019-10-13T05:18:11.106 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6132-ga8ee8ac-1bionic [329 kB] 2019-10-13T05:18:11.111 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6132-ga8ee8ac-1bionic [17.8 MB] 2019-10-13T05:18:11.118 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-10-13T05:18:11.135 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6132-ga8ee8ac-1bionic [426 kB] 2019-10-13T05:18:11.138 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-10-13T05:18:11.148 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [303 kB] 2019-10-13T05:18:11.213 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.2 kB] 2019-10-13T05:18:11.214 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [111 kB] 2019-10-13T05:18:11.217 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6132-ga8ee8ac-1bionic [10.5 kB] 2019-10-13T05:18:11.218 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6132-ga8ee8ac-1bionic [248 kB] 2019-10-13T05:18:11.229 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6132-ga8ee8ac-1bionic [4,490 kB] 2019-10-13T05:18:11.282 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-10-13T05:18:11.300 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-10-13T05:18:11.541 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-13T05:18:11.559 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-13T05:18:11.709 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6132-ga8ee8ac-1bionic [98.8 kB] 2019-10-13T05:18:11.754 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6132-ga8ee8ac-1bionic [329 kB] 2019-10-13T05:18:11.764 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6132-ga8ee8ac-1bionic [17.8 MB] 2019-10-13T05:18:11.892 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-13T05:18:11.915 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-13T05:18:11.962 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6132-ga8ee8ac-1bionic [4,972 kB] 2019-10-13T05:18:12.145 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-13T05:18:12.177 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6132-ga8ee8ac-1bionic [1,324 kB] 2019-10-13T05:18:12.188 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-13T05:18:12.251 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6132-ga8ee8ac-1bionic [5,895 kB] 2019-10-13T05:18:12.509 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6132-ga8ee8ac-1bionic [22.4 MB] 2019-10-13T05:18:12.577 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-13T05:18:12.656 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-13T05:18:12.818 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-13T05:18:12.907 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-13T05:18:13.075 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-13T05:18:13.189 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-13T05:18:13.317 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-13T05:18:13.431 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-13T05:18:13.574 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,800 B] 2019-10-13T05:18:13.574 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6132-ga8ee8ac-1bionic [12.5 kB] 2019-10-13T05:18:13.575 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [503 kB] 2019-10-13T05:18:13.588 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6132-ga8ee8ac-1bionic [1,822 kB] 2019-10-13T05:18:13.669 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6132-ga8ee8ac-1bionic [3,353 kB] 2019-10-13T05:18:13.749 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-13T05:18:13.770 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6132-ga8ee8ac-1bionic [4,972 kB] 2019-10-13T05:18:13.822 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6132-ga8ee8ac-1bionic [34.3 kB] 2019-10-13T05:18:13.823 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6132-ga8ee8ac-1bionic [1,093 kB] 2019-10-13T05:18:13.881 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6132-ga8ee8ac-1bionic [3,912 B] 2019-10-13T05:18:13.882 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6132-ga8ee8ac-1bionic [13.6 kB] 2019-10-13T05:18:13.882 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6132-ga8ee8ac-1bionic [12.8 kB] 2019-10-13T05:18:13.883 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6132-ga8ee8ac-1bionic [35.3 MB] 2019-10-13T05:18:13.886 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-13T05:18:14.073 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-13T05:18:14.216 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-13T05:18:14.293 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6132-ga8ee8ac-1bionic [1,324 kB] 2019-10-13T05:18:14.315 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-13T05:18:14.442 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6132-ga8ee8ac-1bionic [5,895 kB] 2019-10-13T05:18:14.489 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-13T05:18:14.773 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-13T05:18:14.972 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-13T05:18:15.014 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-13T05:18:15.120 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6132-ga8ee8ac-1bionic [22.4 MB] 2019-10-13T05:18:15.213 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-13T05:18:15.335 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-13T05:18:15.502 INFO:teuthology.orchestra.run.smithi071.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-13T05:18:15.546 INFO:teuthology.orchestra.run.smithi071.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [320 kB] 2019-10-13T05:18:15.549 INFO:teuthology.orchestra.run.smithi071.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.5 kB] 2019-10-13T05:18:15.549 INFO:teuthology.orchestra.run.smithi071.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [115 kB] 2019-10-13T05:18:15.554 INFO:teuthology.orchestra.run.smithi071.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-6132-ga8ee8ac-1bionic [14.3 kB] 2019-10-13T05:18:15.554 INFO:teuthology.orchestra.run.smithi071.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6132-ga8ee8ac-1bionic [17.3 kB] 2019-10-13T05:18:15.555 INFO:teuthology.orchestra.run.smithi071.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,856 B] 2019-10-13T05:18:15.556 INFO:teuthology.orchestra.run.smithi071.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6132-ga8ee8ac-1bionic [5,151 kB] 2019-10-13T05:18:15.579 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-13T05:18:15.740 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-13T05:18:15.743 INFO:teuthology.orchestra.run.smithi071.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-13T05:18:15.795 INFO:teuthology.orchestra.run.smithi071.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [61.9 kB] 2019-10-13T05:18:15.905 INFO:teuthology.orchestra.run.smithi071.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-13T05:18:16.087 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-13T05:18:16.249 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-13T05:18:16.267 INFO:teuthology.orchestra.run.smithi071.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-13T05:18:16.325 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,800 B] 2019-10-13T05:18:16.325 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6132-ga8ee8ac-1bionic [12.5 kB] 2019-10-13T05:18:16.325 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [503 kB] 2019-10-13T05:18:16.339 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6132-ga8ee8ac-1bionic [1,822 kB] 2019-10-13T05:18:16.399 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6132-ga8ee8ac-1bionic [3,353 kB] 2019-10-13T05:18:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-13T05:18:16.428 INFO:teuthology.orchestra.run.smithi071.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-13T05:18:16.509 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6132-ga8ee8ac-1bionic [34.3 kB] 2019-10-13T05:18:16.510 INFO:teuthology.orchestra.run.smithi129.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6132-ga8ee8ac-1bionic [1,093 kB] 2019-10-13T05:18:16.546 INFO:teuthology.orchestra.run.smithi129.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6132-ga8ee8ac-1bionic [3,912 B] 2019-10-13T05:18:16.546 INFO:teuthology.orchestra.run.smithi129.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6132-ga8ee8ac-1bionic [13.6 kB] 2019-10-13T05:18:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6132-ga8ee8ac-1bionic [12.8 kB] 2019-10-13T05:18:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6132-ga8ee8ac-1bionic [35.3 MB] 2019-10-13T05:18:16.590 INFO:teuthology.orchestra.run.smithi071.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-13T05:18:16.753 INFO:teuthology.orchestra.run.smithi129.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-13T05:18:16.957 INFO:teuthology.orchestra.run.smithi071.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-13T05:18:17.107 INFO:teuthology.orchestra.run.smithi129.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-13T05:18:17.331 INFO:teuthology.orchestra.run.smithi071.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-13T05:18:17.462 INFO:teuthology.orchestra.run.smithi129.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-13T05:18:17.702 INFO:teuthology.orchestra.run.smithi071.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-13T05:18:17.707 INFO:teuthology.orchestra.run.smithi129.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [320 kB] 2019-10-13T05:18:17.716 INFO:teuthology.orchestra.run.smithi129.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.5 kB] 2019-10-13T05:18:17.716 INFO:teuthology.orchestra.run.smithi129.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [115 kB] 2019-10-13T05:18:17.719 INFO:teuthology.orchestra.run.smithi129.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-6132-ga8ee8ac-1bionic [14.3 kB] 2019-10-13T05:18:17.720 INFO:teuthology.orchestra.run.smithi129.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6132-ga8ee8ac-1bionic [17.3 kB] 2019-10-13T05:18:17.721 INFO:teuthology.orchestra.run.smithi129.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,856 B] 2019-10-13T05:18:17.721 INFO:teuthology.orchestra.run.smithi129.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6132-ga8ee8ac-1bionic [5,151 kB] 2019-10-13T05:18:17.784 INFO:teuthology.orchestra.run.smithi129.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-13T05:18:17.889 INFO:teuthology.orchestra.run.smithi129.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [61.9 kB] 2019-10-13T05:18:18.051 INFO:teuthology.orchestra.run.smithi071.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-13T05:18:18.152 INFO:teuthology.orchestra.run.smithi129.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-13T05:18:18.314 INFO:teuthology.orchestra.run.smithi129.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-13T05:18:18.448 INFO:teuthology.orchestra.run.smithi071.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-13T05:18:18.609 INFO:teuthology.orchestra.run.smithi071.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-13T05:18:18.661 INFO:teuthology.orchestra.run.smithi129.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-13T05:18:18.902 INFO:teuthology.orchestra.run.smithi129.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-13T05:18:18.980 INFO:teuthology.orchestra.run.smithi071.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-13T05:18:19.222 INFO:teuthology.orchestra.run.smithi071.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-13T05:18:19.479 INFO:teuthology.orchestra.run.smithi129.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-13T05:18:19.641 INFO:teuthology.orchestra.run.smithi129.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-13T05:18:19.837 INFO:teuthology.orchestra.run.smithi071.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-13T05:18:19.882 INFO:teuthology.orchestra.run.smithi129.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-13T05:18:19.998 INFO:teuthology.orchestra.run.smithi071.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-13T05:18:20.124 INFO:teuthology.orchestra.run.smithi129.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-13T05:18:20.267 INFO:teuthology.orchestra.run.smithi071.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-13T05:18:20.486 INFO:teuthology.orchestra.run.smithi129.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-13T05:18:20.536 INFO:teuthology.orchestra.run.smithi071.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-13T05:18:20.930 INFO:teuthology.orchestra.run.smithi129.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-13T05:18:20.946 INFO:teuthology.orchestra.run.smithi071.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-13T05:18:21.284 INFO:teuthology.orchestra.run.smithi129.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-13T05:18:21.408 INFO:teuthology.orchestra.run.smithi071.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-13T05:18:21.667 INFO:teuthology.orchestra.run.smithi129.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-13T05:18:21.812 INFO:teuthology.orchestra.run.smithi071.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-13T05:18:22.010 INFO:teuthology.orchestra.run.smithi129.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-13T05:18:22.244 INFO:teuthology.orchestra.run.smithi071.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-13T05:18:22.475 INFO:teuthology.orchestra.run.smithi129.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-13T05:18:22.605 INFO:teuthology.orchestra.run.smithi071.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-13T05:18:23.084 INFO:teuthology.orchestra.run.smithi071.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-13T05:18:23.229 INFO:teuthology.orchestra.run.smithi129.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-13T05:18:23.860 INFO:teuthology.orchestra.run.smithi071.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-13T05:18:24.491 INFO:teuthology.orchestra.run.smithi129.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-13T05:18:25.147 INFO:teuthology.orchestra.run.smithi071.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-13T05:18:26.064 INFO:teuthology.orchestra.run.smithi129.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-13T05:18:26.226 INFO:teuthology.orchestra.run.smithi129.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-13T05:18:26.568 INFO:teuthology.orchestra.run.smithi129.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi071.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-13T05:18:26.913 INFO:teuthology.orchestra.run.smithi071.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-13T05:18:26.925 INFO:teuthology.orchestra.run.smithi129.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi071.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi129.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-13T05:18:27.440 INFO:teuthology.orchestra.run.smithi129.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-13T05:18:27.672 INFO:teuthology.orchestra.run.smithi071.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-13T05:18:28.068 INFO:teuthology.orchestra.run.smithi071.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-13T05:18:28.230 INFO:teuthology.orchestra.run.smithi071.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-13T05:18:28.448 INFO:teuthology.orchestra.run.smithi129.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-13T05:18:29.168 INFO:teuthology.orchestra.run.smithi129.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-13T05:18:29.249 INFO:teuthology.orchestra.run.smithi071.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-13T05:18:29.582 INFO:teuthology.orchestra.run.smithi129.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-13T05:18:30.001 INFO:teuthology.orchestra.run.smithi071.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-13T05:18:30.286 INFO:teuthology.orchestra.run.smithi129.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-13T05:18:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-13T05:18:30.448 INFO:teuthology.orchestra.run.smithi129.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-13T05:18:30.610 INFO:teuthology.orchestra.run.smithi129.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-13T05:18:30.879 INFO:teuthology.orchestra.run.smithi129.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-13T05:18:31.172 INFO:teuthology.orchestra.run.smithi071.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-13T05:18:31.206 INFO:teuthology.orchestra.run.smithi129.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-13T05:18:31.334 INFO:teuthology.orchestra.run.smithi071.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi129.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi071.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-13T05:18:31.725 INFO:teuthology.orchestra.run.smithi129.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-13T05:18:31.781 INFO:teuthology.orchestra.run.smithi071.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-13T05:18:32.117 INFO:teuthology.orchestra.run.smithi071.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-13T05:18:32.370 INFO:teuthology.orchestra.run.smithi129.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-13T05:18:32.387 INFO:teuthology.orchestra.run.smithi071.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi071.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-13T05:18:32.711 INFO:teuthology.orchestra.run.smithi129.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-13T05:18:32.953 INFO:teuthology.orchestra.run.smithi129.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-13T05:18:33.308 INFO:teuthology.orchestra.run.smithi071.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-13T05:18:33.373 INFO:teuthology.orchestra.run.smithi129.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-13T05:18:33.668 INFO:teuthology.orchestra.run.smithi071.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-13T05:18:33.852 INFO:teuthology.orchestra.run.smithi129.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-13T05:18:33.941 INFO:teuthology.orchestra.run.smithi071.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-13T05:18:34.277 INFO:teuthology.orchestra.run.smithi129.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-13T05:18:34.381 INFO:teuthology.orchestra.run.smithi071.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-13T05:18:34.690 INFO:teuthology.orchestra.run.smithi129.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-13T05:18:34.872 INFO:teuthology.orchestra.run.smithi071.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-13T05:18:35.014 INFO:teuthology.orchestra.run.smithi129.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-13T05:18:35.310 INFO:teuthology.orchestra.run.smithi071.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-13T05:18:35.543 INFO:teuthology.orchestra.run.smithi129.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-13T05:18:35.747 INFO:teuthology.orchestra.run.smithi071.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-13T05:18:35.989 INFO:teuthology.orchestra.run.smithi129.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-13T05:18:36.077 INFO:teuthology.orchestra.run.smithi071.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-13T05:18:36.568 INFO:teuthology.orchestra.run.smithi129.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-10-13T05:18:36.654 INFO:teuthology.orchestra.run.smithi071.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-13T05:18:36.809 INFO:teuthology.orchestra.run.smithi129.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-13T05:18:36.970 INFO:teuthology.orchestra.run.smithi129.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-13T05:18:37.113 INFO:teuthology.orchestra.run.smithi071.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-13T05:18:37.132 INFO:teuthology.orchestra.run.smithi129.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi129.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-13T05:18:37.703 INFO:teuthology.orchestra.run.smithi071.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-10-13T05:18:37.899 INFO:teuthology.orchestra.run.smithi129.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-13T05:18:37.945 INFO:teuthology.orchestra.run.smithi071.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-13T05:18:38.060 INFO:teuthology.orchestra.run.smithi129.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-13T05:18:38.107 INFO:teuthology.orchestra.run.smithi071.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-13T05:18:38.222 INFO:teuthology.orchestra.run.smithi129.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-13T05:18:38.268 INFO:teuthology.orchestra.run.smithi071.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-13T05:18:38.607 INFO:teuthology.orchestra.run.smithi129.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-13T05:18:38.831 INFO:teuthology.orchestra.run.smithi071.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-13T05:18:39.095 INFO:teuthology.orchestra.run.smithi071.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-13T05:18:39.257 INFO:teuthology.orchestra.run.smithi071.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-13T05:18:39.286 INFO:teuthology.orchestra.run.smithi129.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-13T05:18:39.418 INFO:teuthology.orchestra.run.smithi071.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-13T05:18:39.689 INFO:teuthology.orchestra.run.smithi129.stdout:Get:114 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-13T05:18:39.828 INFO:teuthology.orchestra.run.smithi071.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi129.stdout:Get:115 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-13T05:18:40.183 INFO:teuthology.orchestra.run.smithi129.stdout:Get:116 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-13T05:18:40.525 INFO:teuthology.orchestra.run.smithi129.stdout:Get:117 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-13T05:18:40.549 INFO:teuthology.orchestra.run.smithi071.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-13T05:18:40.767 INFO:teuthology.orchestra.run.smithi129.stdout:Get:118 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-13T05:18:40.957 INFO:teuthology.orchestra.run.smithi071.stdout:Get:114 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-13T05:18:41.206 INFO:teuthology.orchestra.run.smithi129.stdout:Get:119 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-13T05:18:41.307 INFO:teuthology.orchestra.run.smithi071.stdout:Get:115 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi129.stdout:Get:120 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi071.stdout:Get:116 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-13T05:18:41.609 INFO:teuthology.orchestra.run.smithi129.stdout:Get:121 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-13T05:18:41.835 INFO:teuthology.orchestra.run.smithi071.stdout:Get:117 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-13T05:18:42.105 INFO:teuthology.orchestra.run.smithi071.stdout:Get:118 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-13T05:18:42.413 INFO:teuthology.orchestra.run.smithi129.stdout:Preconfiguring packages ... 2019-10-13T05:18:42.562 INFO:teuthology.orchestra.run.smithi071.stdout:Get:119 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-13T05:18:42.575 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 183 MB in 31s (5,852 kB/s) 2019-10-13T05:18:42.814 INFO:teuthology.orchestra.run.smithi071.stdout:Get:120 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-13T05:18:42.976 INFO:teuthology.orchestra.run.smithi071.stdout:Get:121 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-13T05:18:43.407 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-uritemplate. 2019-10-13T05:18:43.460 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-13T05:18:43.464 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-13T05:18:43.505 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T05:18:43.807 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyparsing. 2019-10-13T05:18:43.822 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-10-13T05:18:43.827 INFO:teuthology.orchestra.run.smithi071.stdout:Preconfiguring packages ... 2019-10-13T05:18:43.866 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T05:18:44.006 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 183 MB in 33s (5,593 kB/s) 2019-10-13T05:18:44.123 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-uritemplate. 2019-10-13T05:18:44.171 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-13T05:18:44.175 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-13T05:18:44.185 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyperclip. 2019-10-13T05:18:44.199 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-10-13T05:18:44.213 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T05:18:44.235 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-10-13T05:18:44.441 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyparsing. 2019-10-13T05:18:44.456 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-10-13T05:18:44.480 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cmd2. 2019-10-13T05:18:44.491 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T05:18:44.494 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-10-13T05:18:44.530 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T05:18:44.807 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package pigz. 2019-10-13T05:18:44.810 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyperclip. 2019-10-13T05:18:44.823 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../004-pigz_2.4-1_amd64.deb ... 2019-10-13T05:18:44.825 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-10-13T05:18:44.866 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking pigz (2.4-1) ... 2019-10-13T05:18:44.868 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-10-13T05:18:45.155 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cmd2. 2019-10-13T05:18:45.171 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-10-13T05:18:45.213 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T05:18:45.227 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package bridge-utils. 2019-10-13T05:18:45.242 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../005-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-13T05:18:45.286 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-13T05:18:45.441 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package pigz. 2019-10-13T05:18:45.455 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../004-pigz_2.4-1_amd64.deb ... 2019-10-13T05:18:45.491 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking pigz (2.4-1) ... 2019-10-13T05:18:45.581 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-13T05:18:45.595 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../006-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-13T05:18:45.665 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T05:18:45.819 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package bridge-utils. 2019-10-13T05:18:45.831 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../005-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-13T05:18:45.868 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-13T05:18:45.990 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../007-librbd1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:46.119 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librbd1 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T05:18:46.214 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-13T05:18:46.228 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../006-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-13T05:18:46.314 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T05:18:46.637 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../008-librados2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:46.639 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../007-librbd1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:46.735 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librbd1 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T05:18:46.756 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librados2 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T05:18:47.245 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../008-librados2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:47.351 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs2. 2019-10-13T05:18:47.365 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../009-libcephfs2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:47.372 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librados2 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T05:18:47.410 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:47.695 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-rados. 2019-10-13T05:18:47.710 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../010-python-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:47.754 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:47.951 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs2. 2019-10-13T05:18:47.965 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../009-libcephfs2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:48.009 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.065 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-13T05:18:48.078 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../011-python-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:18:48.115 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-rados. 2019-10-13T05:18:48.294 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../010-python-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:48.336 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.359 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-cephfs. 2019-10-13T05:18:48.373 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../012-python-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:48.409 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.614 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-13T05:18:48.628 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../011-python-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:18:48.661 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-ceph-common. 2019-10-13T05:18:48.678 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../013-python-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:18:48.689 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.719 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:48.967 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-cephfs. 2019-10-13T05:18:48.982 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../012-python-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:49.024 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:49.030 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-prettytable. 2019-10-13T05:18:49.046 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../014-python-prettytable_0.7.2-3_all.deb ... 2019-10-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-13T05:18:49.286 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-ceph-common. 2019-10-13T05:18:49.299 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../013-python-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:18:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:49.341 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-rbd. 2019-10-13T05:18:49.355 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../015-python-rbd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:49.401 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:49.580 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-prettytable. 2019-10-13T05:18:49.595 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../014-python-prettytable_0.7.2-3_all.deb ... 2019-10-13T05:18:49.630 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-13T05:18:49.686 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-13T05:18:49.701 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../016-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-13T05:18:49.737 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T05:18:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-rbd. 2019-10-13T05:18:49.888 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../015-python-rbd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:49.932 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:50.025 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librgw2. 2019-10-13T05:18:50.051 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../017-librgw2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:50.072 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:50.293 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-13T05:18:50.309 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../016-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-13T05:18:50.352 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T05:18:50.622 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librgw2. 2019-10-13T05:18:50.637 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../017-librgw2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:50.666 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-rgw. 2019-10-13T05:18:50.671 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:50.680 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../018-python-rgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:50.724 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-13T05:18:51.018 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../019-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-13T05:18:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T05:18:51.290 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-rgw. 2019-10-13T05:18:51.306 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../018-python-rgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:51.321 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-13T05:18:51.335 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../020-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-13T05:18:51.348 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:51.371 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T05:18:51.659 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-13T05:18:51.673 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../019-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-13T05:18:51.718 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T05:18:51.750 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liboath0. 2019-10-13T05:18:51.776 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../021-liboath0_2.6.1-1_amd64.deb ... 2019-10-13T05:18:51.799 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-13T05:18:52.004 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-13T05:18:52.017 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../020-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-13T05:18:52.062 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T05:18:52.103 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libradosstriper1. 2019-10-13T05:18:52.117 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../022-libradosstriper1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:52.160 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:52.365 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liboath0. 2019-10-13T05:18:52.378 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../021-liboath0_2.6.1-1_amd64.deb ... 2019-10-13T05:18:52.415 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-13T05:18:52.584 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-common. 2019-10-13T05:18:52.595 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../023-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:52.638 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:52.718 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libradosstriper1. 2019-10-13T05:18:52.732 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../022-libradosstriper1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:52.776 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:53.221 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-common. 2019-10-13T05:18:53.236 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../023-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-base. 2019-10-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../024-ceph-base_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:54.353 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:54.909 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-base. 2019-10-13T05:18:54.925 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../024-ceph-base_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:18:54.976 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:18:55.282 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-bcrypt. 2019-10-13T05:18:55.301 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../025-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-13T05:18:55.340 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-13T05:18:55.693 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-13T05:18:55.706 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../026-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-13T05:18:55.743 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-13T05:18:55.813 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-bcrypt. 2019-10-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../025-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-13T05:18:56.147 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-jwt. 2019-10-13T05:18:56.163 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../027-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-13T05:18:56.166 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-13T05:18:56.180 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../026-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-13T05:18:56.205 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-13T05:18:56.224 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-13T05:18:56.525 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-openssl. 2019-10-13T05:18:56.538 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../028-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-13T05:18:56.569 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-jwt. 2019-10-13T05:18:56.575 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T05:18:56.584 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../027-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-13T05:18:56.628 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-13T05:18:56.886 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-logutils. 2019-10-13T05:18:56.901 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../029-python-logutils_0.3.3-5_all.deb ... 2019-10-13T05:18:56.939 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-openssl. 2019-10-13T05:18:56.944 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-13T05:18:56.953 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../028-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-13T05:18:56.997 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T05:18:57.255 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-markupsafe. 2019-10-13T05:18:57.270 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../030-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-13T05:18:57.308 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-logutils. 2019-10-13T05:18:57.314 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-13T05:18:57.321 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../029-python-logutils_0.3.3-5_all.deb ... 2019-10-13T05:18:57.364 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-13T05:18:57.591 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-mako. 2019-10-13T05:18:57.608 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../031-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-13T05:18:57.650 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-13T05:18:57.653 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-markupsafe. 2019-10-13T05:18:57.666 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../030-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-13T05:18:57.703 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../032-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-mako. 2019-10-13T05:18:58.028 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-13T05:18:58.028 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../031-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-13T05:18:58.072 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-13T05:18:58.347 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-singledispatch. 2019-10-13T05:18:58.362 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../033-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-13T05:18:58.406 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-13T05:18:58.442 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-13T05:18:58.457 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../032-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-13T05:18:58.501 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-13T05:18:58.683 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-simplejson. 2019-10-13T05:18:58.699 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../034-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-13T05:18:58.741 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-13T05:18:58.822 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-singledispatch. 2019-10-13T05:18:58.836 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../033-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-13T05:18:59.052 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-webob. 2019-10-13T05:18:59.066 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../035-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-13T05:18:59.113 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T05:18:59.170 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-simplejson. 2019-10-13T05:18:59.184 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../034-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-13T05:18:59.229 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-13T05:18:59.429 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-bs4. 2019-10-13T05:18:59.443 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../036-python-bs4_4.6.0-1_all.deb ... 2019-10-13T05:18:59.488 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-13T05:18:59.574 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-webob. 2019-10-13T05:18:59.589 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../035-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-13T05:18:59.634 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T05:18:59.854 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-waitress. 2019-10-13T05:18:59.869 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../037-python-waitress_1.0.1-1_all.deb ... 2019-10-13T05:18:59.913 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-13T05:18:59.953 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-bs4. 2019-10-13T05:18:59.967 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../036-python-bs4_4.6.0-1_all.deb ... 2019-10-13T05:19:00.005 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-13T05:19:00.191 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-dnspython. 2019-10-13T05:19:00.207 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../038-python-dnspython_1.15.0-1_all.deb ... 2019-10-13T05:19:00.249 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-13T05:19:00.311 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-waitress. 2019-10-13T05:19:00.327 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../037-python-waitress_1.0.1-1_all.deb ... 2019-10-13T05:19:00.361 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-13T05:19:00.519 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package formencode-i18n. 2019-10-13T05:19:00.535 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../039-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-13T05:19:00.578 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T05:19:00.639 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-dnspython. 2019-10-13T05:19:00.660 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../038-python-dnspython_1.15.0-1_all.deb ... 2019-10-13T05:19:00.697 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-13T05:19:00.872 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-formencode. 2019-10-13T05:19:00.886 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../040-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-13T05:19:00.931 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T05:19:01.028 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package formencode-i18n. 2019-10-13T05:19:01.041 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../039-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-13T05:19:01.086 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T05:19:01.259 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-tempita. 2019-10-13T05:19:01.273 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../041-python-tempita_0.5.2-2_all.deb ... 2019-10-13T05:19:01.317 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-13T05:19:01.372 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-formencode. 2019-10-13T05:19:01.386 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../040-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-13T05:19:01.430 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T05:19:01.594 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-paste. 2019-10-13T05:19:01.608 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../042-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-13T05:19:01.644 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T05:19:01.742 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-tempita. 2019-10-13T05:19:01.756 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../041-python-tempita_0.5.2-2_all.deb ... 2019-10-13T05:19:01.801 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-13T05:19:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-13T05:19:01.996 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-13T05:19:02.039 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T05:19:02.112 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-paste. 2019-10-13T05:19:02.125 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../042-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-13T05:19:02.178 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T05:19:02.342 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-13T05:19:02.349 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../044-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-13T05:19:02.391 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-13T05:19:02.507 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-13T05:19:02.521 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-13T05:19:02.565 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T05:19:02.703 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-webtest. 2019-10-13T05:19:02.718 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../045-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-13T05:19:02.761 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T05:19:02.902 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-13T05:19:02.917 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../044-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-13T05:19:03.131 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pecan. 2019-10-13T05:19:03.145 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../046-python-pecan_1.2.1-2_all.deb ... 2019-10-13T05:19:03.203 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-13T05:19:03.281 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-webtest. 2019-10-13T05:19:03.295 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../045-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-13T05:19:03.338 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-certifi. 2019-10-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../047-python-certifi_2018.1.18-2_all.deb ... 2019-10-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-13T05:19:03.708 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pecan. 2019-10-13T05:19:03.721 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../046-python-pecan_1.2.1-2_all.deb ... 2019-10-13T05:19:03.762 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-13T05:19:03.913 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-chardet. 2019-10-13T05:19:03.927 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../048-python-chardet_3.0.4-1_all.deb ... 2019-10-13T05:19:03.971 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-13T05:19:04.122 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-certifi. 2019-10-13T05:19:04.136 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../047-python-certifi_2018.1.18-2_all.deb ... 2019-10-13T05:19:04.180 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-13T05:19:04.283 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-urllib3. 2019-10-13T05:19:04.299 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../049-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-13T05:19:04.341 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T05:19:04.508 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-chardet. 2019-10-13T05:19:04.522 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../048-python-chardet_3.0.4-1_all.deb ... 2019-10-13T05:19:04.567 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-13T05:19:04.669 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-requests. 2019-10-13T05:19:04.683 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../050-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-13T05:19:04.719 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T05:19:04.895 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-urllib3. 2019-10-13T05:19:04.909 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../049-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-13T05:19:04.956 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T05:19:05.055 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-werkzeug. 2019-10-13T05:19:05.070 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../051-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-13T05:19:05.113 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T05:19:05.282 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-requests. 2019-10-13T05:19:05.296 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../050-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-13T05:19:05.348 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T05:19:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr. 2019-10-13T05:19:05.489 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../052-ceph-mgr_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:05.533 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:05.668 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-werkzeug. 2019-10-13T05:19:05.682 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../051-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-13T05:19:05.718 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T05:19:05.996 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mon. 2019-10-13T05:19:06.012 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../053-ceph-mon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:06.054 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:06.088 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr. 2019-10-13T05:19:06.105 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../052-ceph-mgr_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:06.146 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:06.635 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mon. 2019-10-13T05:19:06.649 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../053-ceph-mon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:06.693 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:06.949 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-osd. 2019-10-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../054-ceph-osd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:07.922 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-osd. 2019-10-13T05:19:07.937 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../054-ceph-osd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:07.980 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:09.137 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph. 2019-10-13T05:19:09.153 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../055-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:09.203 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:09.456 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package runc. 2019-10-13T05:19:09.470 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../056-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T05:19:09.514 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T05:19:09.818 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph. 2019-10-13T05:19:09.833 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../055-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:09.876 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:10.017 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package containerd. 2019-10-13T05:19:10.032 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../057-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T05:19:10.075 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T05:19:10.129 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package runc. 2019-10-13T05:19:10.144 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../056-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T05:19:10.187 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T05:19:10.708 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package containerd. 2019-10-13T05:19:10.721 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../057-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T05:19:10.766 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T05:19:11.996 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package docker.io. 2019-10-13T05:19:12.009 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../058-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-13T05:19:12.093 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T05:19:12.665 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package docker.io. 2019-10-13T05:19:12.676 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../058-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-13T05:19:12.750 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T05:19:14.993 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-daemon. 2019-10-13T05:19:15.010 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../059-ceph-daemon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:15.051 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:15.379 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-fuse. 2019-10-13T05:19:15.392 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../060-ceph-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:15.429 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:15.692 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-daemon. 2019-10-13T05:19:15.710 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../059-ceph-daemon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:15.807 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mds. 2019-10-13T05:19:15.821 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../061-ceph-mds_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:15.870 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:16.079 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-fuse. 2019-10-13T05:19:16.093 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../060-ceph-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:16.137 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:16.335 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-13T05:19:16.348 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../062-python-repoze.lru_0.7-1_all.deb ... 2019-10-13T05:19:16.384 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-13T05:19:16.590 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mds. 2019-10-13T05:19:16.604 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../061-ceph-mds_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:16.637 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjs-underscore. 2019-10-13T05:19:16.651 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:16.652 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../063-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-13T05:19:16.695 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T05:19:17.008 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-13T05:19:17.021 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../064-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-13T05:19:17.064 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T05:19:17.094 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-13T05:19:17.109 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../062-python-repoze.lru_0.7-1_all.deb ... 2019-10-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-13T05:19:17.386 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-routes. 2019-10-13T05:19:17.398 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../065-python-routes_2.4.1-1_all.deb ... 2019-10-13T05:19:17.442 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-13T05:19:17.455 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libjs-underscore. 2019-10-13T05:19:17.468 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../063-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-13T05:19:17.505 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T05:19:17.778 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-13T05:19:17.778 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-13T05:19:17.788 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../064-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-13T05:19:17.795 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../066-ceph-mgr-dashboard_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:17.833 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T05:19:17.836 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-routes. 2019-10-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../065-python-routes_2.4.1-1_all.deb ... 2019-10-13T05:19:18.210 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-13T05:19:18.531 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-13T05:19:18.546 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../066-ceph-mgr-dashboard_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:18.589 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../067-ceph-mgr-diskprediction-cloud_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:19.053 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-decorator. 2019-10-13T05:19:19.068 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../068-python-decorator_4.1.2-1_all.deb ... 2019-10-13T05:19:19.128 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-13T05:19:19.404 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-13T05:19:19.420 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../067-ceph-mgr-diskprediction-cloud_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:19.448 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-scipy. 2019-10-13T05:19:19.454 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:19.462 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../069-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-13T05:19:19.506 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T05:19:19.749 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-decorator. 2019-10-13T05:19:19.766 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../068-python-decorator_4.1.2-1_all.deb ... 2019-10-13T05:19:19.807 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-13T05:19:20.144 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-scipy. 2019-10-13T05:19:20.157 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../069-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-13T05:19:20.202 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T05:19:20.672 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-13T05:19:20.687 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../070-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-13T05:19:20.740 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-13T05:19:21.217 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-joblib. 2019-10-13T05:19:21.233 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../071-python-joblib_0.11-1_all.deb ... 2019-10-13T05:19:21.276 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-13T05:19:21.319 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-13T05:19:21.335 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../070-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-13T05:19:21.377 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-13T05:19:21.629 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-sklearn. 2019-10-13T05:19:21.644 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../072-python-sklearn_0.19.1-3_all.deb ... 2019-10-13T05:19:21.687 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-13T05:19:21.848 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-joblib. 2019-10-13T05:19:21.862 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../071-python-joblib_0.11-1_all.deb ... 2019-10-13T05:19:21.918 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-13T05:19:22.218 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-13T05:19:22.233 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../073-ceph-mgr-diskprediction-local_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:22.260 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-sklearn. 2019-10-13T05:19:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../072-python-sklearn_0.19.1-3_all.deb ... 2019-10-13T05:19:22.285 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:22.318 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-13T05:19:22.950 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-13T05:19:22.965 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../073-ceph-mgr-diskprediction-local_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:23.008 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:23.029 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-monotonic. 2019-10-13T05:19:23.045 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../074-python-monotonic_1.1-2_all.deb ... 2019-10-13T05:19:23.088 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-13T05:19:23.382 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-fasteners. 2019-10-13T05:19:23.399 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../075-python-fasteners_0.12.0-3_all.deb ... 2019-10-13T05:19:23.440 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-13T05:19:23.670 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-monotonic. 2019-10-13T05:19:23.686 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../074-python-monotonic_1.1-2_all.deb ... 2019-10-13T05:19:23.728 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-13T05:19:23.752 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-httplib2. 2019-10-13T05:19:23.766 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../076-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-13T05:19:23.810 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T05:19:24.031 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-fasteners. 2019-10-13T05:19:24.045 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../075-python-fasteners_0.12.0-3_all.deb ... 2019-10-13T05:19:24.098 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-13T05:19:24.130 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pyasn1. 2019-10-13T05:19:24.143 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../077-python-pyasn1_0.4.2-3_all.deb ... 2019-10-13T05:19:24.179 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-13T05:19:24.401 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-httplib2. 2019-10-13T05:19:24.415 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../076-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-13T05:19:24.451 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T05:19:24.509 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-13T05:19:24.522 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../078-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-13T05:19:24.567 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T05:19:24.746 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pyasn1. 2019-10-13T05:19:24.762 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../077-python-pyasn1_0.4.2-3_all.deb ... 2019-10-13T05:19:24.804 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-13T05:19:24.878 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-rsa. 2019-10-13T05:19:24.891 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../079-python-rsa_3.4.2-1_all.deb ... 2019-10-13T05:19:24.936 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-13T05:19:25.126 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-13T05:19:25.142 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../078-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-13T05:19:25.184 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T05:19:25.325 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-13T05:19:25.338 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../080-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-13T05:19:25.383 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:25.480 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-rsa. 2019-10-13T05:19:25.493 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../079-python-rsa_3.4.2-1_all.deb ... 2019-10-13T05:19:25.537 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-13T05:19:25.844 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-oauth2client. 2019-10-13T05:19:25.858 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../081-python-oauth2client_4.1.2-3_all.deb ... 2019-10-13T05:19:25.902 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-13T05:19:25.926 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-13T05:19:25.940 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../080-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-13T05:19:25.985 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-websocket. 2019-10-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../082-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T05:19:26.430 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-oauth2client. 2019-10-13T05:19:26.447 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../081-python-oauth2client_4.1.2-3_all.deb ... 2019-10-13T05:19:26.488 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-13T05:19:26.658 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-yaml. 2019-10-13T05:19:26.671 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../083-python-yaml_3.12-1build2_amd64.deb ... 2019-10-13T05:19:26.716 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-websocket. 2019-10-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../082-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T05:19:27.044 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-kubernetes. 2019-10-13T05:19:27.059 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../084-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-13T05:19:27.102 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T05:19:27.246 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-yaml. 2019-10-13T05:19:27.259 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../083-python-yaml_3.12-1build2_amd64.deb ... 2019-10-13T05:19:27.295 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:27.634 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-kubernetes. 2019-10-13T05:19:27.649 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../084-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-13T05:19:27.692 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T05:19:27.883 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-13T05:19:27.898 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../086-ceph-mgr-rook_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:27.942 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:28.122 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-13T05:19:28.137 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:28.180 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:28.261 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-13T05:19:28.277 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../087-ceph-mgr-ssh_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:28.319 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:28.500 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-13T05:19:28.514 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../086-ceph-mgr-rook_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:28.558 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:28.655 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-13T05:19:28.670 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../088-libonig4_6.7.0-1_amd64.deb ... 2019-10-13T05:19:28.713 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-13T05:19:28.886 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-13T05:19:28.901 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../087-ceph-mgr-ssh_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:28.944 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-13T05:19:29.090 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../089-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-13T05:19:29.133 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T05:19:29.272 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-13T05:19:29.286 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../088-libonig4_6.7.0-1_amd64.deb ... 2019-10-13T05:19:29.331 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-13T05:19:29.419 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package jq. 2019-10-13T05:19:29.435 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../090-jq_1.5+dfsg-2_amd64.deb ... 2019-10-13T05:19:29.477 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-13T05:19:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-13T05:19:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../089-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-13T05:19:29.738 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package socat. 2019-10-13T05:19:29.752 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../091-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-13T05:19:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T05:19:29.796 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-13T05:19:30.045 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package jq. 2019-10-13T05:19:30.060 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../090-jq_1.5+dfsg-2_amd64.deb ... 2019-10-13T05:19:30.103 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-13T05:19:30.167 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-13T05:19:30.181 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../092-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-13T05:19:30.233 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T05:19:30.339 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package socat. 2019-10-13T05:19:30.354 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../091-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-13T05:19:30.397 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-13T05:19:30.511 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package xmlstarlet. 2019-10-13T05:19:30.525 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-13T05:19:30.569 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-13T05:19:30.793 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-13T05:19:30.809 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../092-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-13T05:19:30.860 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T05:19:30.898 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-test. 2019-10-13T05:19:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../094-ceph-test_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:30.964 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:31.138 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package xmlstarlet. 2019-10-13T05:19:31.152 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-13T05:19:31.196 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-13T05:19:31.551 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-test. 2019-10-13T05:19:31.565 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../094-ceph-test_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:31.609 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:33.676 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rados. 2019-10-13T05:19:33.690 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../095-python3-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:33.734 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:34.062 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-10-13T05:19:34.079 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../096-python3-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:34.121 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:34.354 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rados. 2019-10-13T05:19:34.369 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../095-python3-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:34.398 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cephfs. 2019-10-13T05:19:34.412 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:34.414 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../097-python3-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:34.457 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:34.757 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-10-13T05:19:34.772 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../096-python3-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:34.776 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-colorama. 2019-10-13T05:19:34.790 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../098-python3-colorama_0.3.7-1_all.deb ... 2019-10-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:34.834 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-10-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package cephfs-shell. 2019-10-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../099-cephfs-shell_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cephfs. 2019-10-13T05:19:35.133 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../097-python3-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:35.153 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:35.185 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:35.480 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-colorama. 2019-10-13T05:19:35.496 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../098-python3-colorama_0.3.7-1_all.deb ... 2019-10-13T05:19:35.530 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-10-13T05:19:35.531 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-13T05:19:35.545 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../100-cgroupfs-mount_1.4_all.deb ... 2019-10-13T05:19:35.599 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package cephfs-shell. 2019-10-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../099-cephfs-shell_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T05:19:35.816 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:35.868 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package dns-root-data. 2019-10-13T05:19:35.884 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../101-dns-root-data_2018013001_all.deb ... 2019-10-13T05:19:35.934 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-13T05:19:36.203 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-13T05:19:36.217 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../100-cgroupfs-mount_1.4_all.deb ... 2019-10-13T05:19:36.270 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-13T05:19:36.271 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-13T05:19:36.285 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../102-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-13T05:19:36.489 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package dns-root-data. 2019-10-13T05:19:36.503 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../101-dns-root-data_2018013001_all.deb ... 2019-10-13T05:19:36.539 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-13T05:19:36.640 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-13T05:19:36.655 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../103-libcephfs-dev_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:36.698 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../102-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-13T05:19:36.909 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-13T05:19:37.009 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-attr. 2019-10-13T05:19:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../104-python-attr_17.4.0-2_all.deb ... 2019-10-13T05:19:37.068 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-13T05:19:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-13T05:19:37.251 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../103-libcephfs-dev_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:37.296 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:37.347 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-ceph. 2019-10-13T05:19:37.362 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../105-python-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:37.403 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:37.557 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-attr. 2019-10-13T05:19:37.572 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../104-python-attr_17.4.0-2_all.deb ... 2019-10-13T05:19:37.607 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-13T05:19:37.699 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-funcsigs. 2019-10-13T05:19:37.714 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../106-python-funcsigs_1.0.2-4_all.deb ... 2019-10-13T05:19:37.756 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-13T05:19:38.050 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-webencodings. 2019-10-13T05:19:38.067 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../107-python-webencodings_0.5-2_all.deb ... 2019-10-13T05:19:38.108 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-13T05:19:38.293 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-ceph. 2019-10-13T05:19:38.308 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../105-python-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:38.351 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:38.394 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-html5lib. 2019-10-13T05:19:38.411 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../108-python-html5lib_0.999999999-1_all.deb ... 2019-10-13T05:19:38.452 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-13T05:19:38.678 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-funcsigs. 2019-10-13T05:19:38.694 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../106-python-funcsigs_1.0.2-4_all.deb ... 2019-10-13T05:19:38.737 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-13T05:19:38.806 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-13T05:19:38.821 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../109-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-13T05:19:38.872 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T05:19:39.015 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-webencodings. 2019-10-13T05:19:39.031 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../107-python-webencodings_0.5-2_all.deb ... 2019-10-13T05:19:39.065 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-13T05:19:39.319 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-html5lib. 2019-10-13T05:19:39.333 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../108-python-html5lib_0.999999999-1_all.deb ... 2019-10-13T05:19:39.377 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-13T05:19:39.680 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-13T05:19:39.696 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../109-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-13T05:19:39.730 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-openid. 2019-10-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../110-python-openid_2.2.5-6_all.deb ... 2019-10-13T05:19:39.869 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-13T05:19:40.101 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-openid. 2019-10-13T05:19:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../110-python-openid_2.2.5-6_all.deb ... 2019-10-13T05:19:40.150 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-13T05:19:40.214 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastescript. 2019-10-13T05:19:40.229 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../111-python-pastescript_2.0.2-2_all.deb ... 2019-10-13T05:19:40.272 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-13T05:19:40.496 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pastescript. 2019-10-13T05:19:40.510 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../111-python-pastescript_2.0.2-2_all.deb ... 2019-10-13T05:19:40.554 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-13T05:19:40.592 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pluggy. 2019-10-13T05:19:40.606 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../112-python-pluggy_0.6.0-1_all.deb ... 2019-10-13T05:19:40.650 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-13T05:19:40.858 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pluggy. 2019-10-13T05:19:40.874 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../112-python-pluggy_0.6.0-1_all.deb ... 2019-10-13T05:19:40.916 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-13T05:19:40.962 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-py. 2019-10-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../113-python-py_1.5.2-1_all.deb ... 2019-10-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-13T05:19:41.228 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-py. 2019-10-13T05:19:41.243 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../113-python-py_1.5.2-1_all.deb ... 2019-10-13T05:19:41.286 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-13T05:19:41.331 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pyinotify. 2019-10-13T05:19:41.348 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../114-python-pyinotify_0.9.6-1_all.deb ... 2019-10-13T05:19:41.389 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pyinotify. 2019-10-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../114-python-pyinotify_0.9.6-1_all.deb ... 2019-10-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-13T05:19:41.700 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pytest. 2019-10-13T05:19:41.716 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../115-python-pytest_3.3.2-2_all.deb ... 2019-10-13T05:19:41.767 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-13T05:19:41.984 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pytest. 2019-10-13T05:19:41.998 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../115-python-pytest_3.3.2-2_all.deb ... 2019-10-13T05:19:42.042 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-13T05:19:42.112 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-scgi. 2019-10-13T05:19:42.126 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../116-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-13T05:19:42.170 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-13T05:19:42.396 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-scgi. 2019-10-13T05:19:42.412 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../116-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-13T05:19:42.440 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-13T05:19:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../117-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-13T05:19:42.454 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-13T05:19:42.498 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:42.707 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-13T05:19:42.723 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../117-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-13T05:19:42.765 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package radosgw. 2019-10-13T05:19:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../118-radosgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:43.144 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package radosgw. 2019-10-13T05:19:43.158 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../118-radosgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:43.203 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:43.605 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package rbd-fuse. 2019-10-13T05:19:43.620 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../119-rbd-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:43.662 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:43.889 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package rbd-fuse. 2019-10-13T05:19:43.901 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../119-rbd-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T05:19:43.939 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:44.038 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-13T05:19:44.055 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../120-ubuntu-fan_0.12.10_all.deb ... 2019-10-13T05:19:44.096 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-13T05:19:44.250 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-13T05:19:44.268 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../120-ubuntu-fan_0.12.10_all.deb ... 2019-10-13T05:19:44.299 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-13T05:19:44.393 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-13T05:19:44.596 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-13T05:19:44.832 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-13T05:19:44.837 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-13T05:19:45.025 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-13T05:19:45.038 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:45.508 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-13T05:19:45.693 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-13T05:19:45.749 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-13T05:19:45.922 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-13T05:19:45.968 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T05:19:46.100 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T05:19:46.197 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-13T05:19:46.323 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-13T05:19:46.526 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-13T05:19:46.637 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-13T05:19:46.730 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-13T05:19:46.918 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-13T05:19:47.004 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-13T05:19:47.130 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T05:19:47.209 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T05:19:47.260 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T05:19:47.339 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T05:19:47.399 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T05:19:47.454 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T05:19:47.501 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T05:19:47.556 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T05:19:47.721 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:47.736 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:48.002 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-13T05:19:48.033 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-13T05:19:48.134 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-13T05:19:48.151 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-13T05:19:48.365 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-py (1.5.2-1) ... 2019-10-13T05:19:48.383 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-py (1.5.2-1) ... 2019-10-13T05:19:48.596 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-13T05:19:48.614 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-joblib (0.11-1) ... 2019-10-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-joblib (0.11-1) ... 2019-10-13T05:19:49.060 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-13T05:19:49.067 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-13T05:19:49.188 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-13T05:19:49.197 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-13T05:19:49.553 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-10-13T05:19:49.603 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-10-13T05:19:49.824 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up dns-root-data (2018013001) ... 2019-10-13T05:19:49.891 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up dns-root-data (2018013001) ... 2019-10-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T05:19:50.018 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T05:19:50.246 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T05:19:50.321 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T05:19:50.373 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T05:19:50.461 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T05:19:50.485 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T05:19:50.715 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:50.793 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:50.816 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-13T05:19:50.919 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-13T05:19:50.957 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-13T05:19:51.101 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-13T05:19:51.161 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-13T05:19:51.289 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-13T05:19:51.600 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T05:19:51.783 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T05:19:51.819 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-13T05:19:51.987 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-13T05:19:52.161 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T05:19:52.331 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T05:19:52.471 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-13T05:19:52.641 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-13T05:19:52.669 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-13T05:19:52.846 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-13T05:19:52.873 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-13T05:19:53.050 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-13T05:19:53.090 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T05:19:53.250 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T05:19:53.337 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-13T05:19:53.787 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-13T05:19:53.795 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-13T05:19:53.983 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T05:19:53.999 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-13T05:19:54.056 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-13T05:19:54.185 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T05:19:54.209 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T05:19:54.275 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-13T05:19:54.335 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T05:19:54.420 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T05:19:54.536 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T05:19:54.555 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T05:19:54.797 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:19:54.890 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:19:54.923 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-13T05:19:55.016 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:19:55.124 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-13T05:19:55.150 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-13T05:19:55.358 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-13T05:19:55.426 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T05:19:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-13T05:19:55.594 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T05:19:55.646 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T05:19:55.786 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T05:19:55.822 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T05:19:56.005 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:56.014 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T05:19:56.130 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-13T05:19:56.232 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T05:19:56.332 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-13T05:19:56.358 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-13T05:19:56.551 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-13T05:19:56.945 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T05:19:57.155 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T05:19:57.181 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T05:19:57.391 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T05:20:01.141 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T05:20:01.317 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-13T05:20:01.377 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T05:20:01.510 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-13T05:20:01.553 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-13T05:20:01.678 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-13T05:20:01.746 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-13T05:20:01.887 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T05:20:01.931 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-13T05:20:02.013 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T05:20:02.124 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T05:20:02.139 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-13T05:20:02.215 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-13T05:20:02.251 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T05:20:02.376 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-13T05:20:02.382 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T05:20:02.460 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-13T05:20:02.483 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up pigz (2.4-1) ... 2019-10-13T05:20:02.593 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-13T05:20:02.628 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T05:20:02.728 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up pigz (2.4-1) ... 2019-10-13T05:20:02.785 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-13T05:20:02.855 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-13T05:20:03.047 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-13T05:20:03.053 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-13T05:20:03.299 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-13T05:20:03.431 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-10-13T05:20:03.483 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-13T05:20:03.676 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-10-13T05:20:03.698 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-13T05:20:03.875 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T05:20:03.952 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-13T05:20:04.084 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-13T05:20:04.137 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T05:20:04.210 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T05:20:04.355 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T05:20:04.641 INFO:teuthology.orchestra.run.smithi071.stdout:Adding group `docker' (GID 122) ... 2019-10-13T05:20:04.750 INFO:teuthology.orchestra.run.smithi071.stdout:Done. 2019-10-13T05:20:04.887 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-13T05:20:04.897 INFO:teuthology.orchestra.run.smithi129.stdout:Adding group `docker' (GID 122) ... 2019-10-13T05:20:05.012 INFO:teuthology.orchestra.run.smithi129.stdout:Done. 2019-10-13T05:20:05.150 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-13T05:20:08.065 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T05:20:08.393 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T05:20:09.052 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-13T05:20:09.261 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T05:20:09.388 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-13T05:20:09.563 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T05:20:09.589 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T05:20:09.882 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T05:20:09.932 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-13T05:20:10.175 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-13T05:20:10.243 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-13T05:20:10.385 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T05:20:10.469 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-13T05:20:10.511 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-13T05:20:10.662 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librados2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:10.663 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T05:20:10.780 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-13T05:20:10.796 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:10.906 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librados2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:10.922 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-13T05:20:11.032 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:11.158 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-13T05:20:11.559 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-13T05:20:11.837 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-13T05:20:12.125 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-13T05:20:12.385 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:12.807 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:12.907 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T05:20:13.008 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:13.260 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:13.386 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T05:20:13.418 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-13T05:20:13.653 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:13.787 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:13.905 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-13T05:20:13.980 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.106 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.131 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.257 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.458 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.482 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-13T05:20:14.584 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.793 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:14.885 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.003 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.006 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T05:20:15.007 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T05:20:15.104 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-13T05:20:15.287 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librbd1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.297 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.420 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.497 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T05:20:15.497 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T05:20:15.546 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-13T05:20:15.773 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.790 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librbd1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.882 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:15.916 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-13T05:20:16.260 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:16.358 INFO:teuthology.orchestra.run.smithi071.stdout:Adding group ceph....done 2019-10-13T05:20:16.386 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user ceph....done 2019-10-13T05:20:16.853 INFO:teuthology.orchestra.run.smithi129.stdout:Adding group ceph....done 2019-10-13T05:20:16.927 INFO:teuthology.orchestra.run.smithi071.stdout:Setting system user ceph properties....done 2019-10-13T05:20:17.086 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user ceph....done 2019-10-13T05:20:17.330 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-13T05:20:17.439 INFO:teuthology.orchestra.run.smithi129.stdout:Setting system user ceph properties....done 2019-10-13T05:20:17.596 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-13T05:20:17.625 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-13T05:20:17.869 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T05:20:17.869 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T05:20:18.137 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.251 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.368 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T05:20:18.368 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T05:20:18.377 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.512 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T05:20:18.722 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.789 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.847 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:18.907 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:19.033 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T05:20:19.176 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-13T05:20:19.234 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:19.361 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:19.466 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-13T05:20:19.539 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-13T05:20:19.571 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-13T05:20:19.692 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-13T05:20:19.768 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:19.995 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-13T05:20:20.068 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-13T05:20:20.092 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-13T05:20:20.281 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:20.315 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T05:20:20.316 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T05:20:20.604 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:20.827 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T05:20:20.827 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T05:20:20.831 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T05:20:20.831 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T05:20:21.108 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:21.113 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T05:20:21.618 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:21.623 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:21.828 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T05:20:21.828 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T05:20:21.840 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T05:20:21.840 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T05:20:22.099 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.119 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.217 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T05:20:22.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T05:20:22.343 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.487 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.620 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.631 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.746 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.774 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.889 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:22.917 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:23.031 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T05:20:23.059 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:23.185 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:23.258 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T05:20:23.312 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:23.446 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T05:20:23.581 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T05:20:23.673 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:23.758 INFO:teuthology.orchestra.run.smithi071.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T05:20:23.761 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:23.761 INFO:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-10-13T05:20:23.799 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T05:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2019-10-13T05:20:24.090 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2019-10-13T05:20:24.090 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2019-10-13T05:20:24.247 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2019-10-13T05:20:24.247 INFO:teuthology.orchestra.run.smithi071.stdout: libbison-dev libfl-dev libfl2 zlib1g-dev 2019-10-13T05:20:24.249 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2019-10-13T05:20:24.250 INFO:teuthology.orchestra.run.smithi071.stdout: bison-doc flex-doc libssl-doc 2019-10-13T05:20:24.274 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2019-10-13T05:20:24.275 INFO:teuthology.orchestra.run.smithi071.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-10-13T05:20:24.306 INFO:teuthology.orchestra.run.smithi129.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T05:20:24.308 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:24.308 INFO:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-10-13T05:20:24.391 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2019-10-13T05:20:24.457 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T05:20:24.457 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 2,737 kB of archives. 2019-10-13T05:20:24.457 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-10-13T05:20:24.457 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-10-13T05:20:24.640 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2019-10-13T05:20:24.641 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2019-10-13T05:20:24.803 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2019-10-13T05:20:24.804 INFO:teuthology.orchestra.run.smithi129.stdout: libbison-dev libfl-dev libfl2 zlib1g-dev 2019-10-13T05:20:24.806 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2019-10-13T05:20:24.806 INFO:teuthology.orchestra.run.smithi129.stdout: bison-doc flex-doc libssl-doc 2019-10-13T05:20:24.830 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2019-10-13T05:20:24.831 INFO:teuthology.orchestra.run.smithi129.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-10-13T05:20:25.011 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T05:20:25.012 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 2,737 kB of archives. 2019-10-13T05:20:25.012 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-10-13T05:20:25.012 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-10-13T05:20:25.016 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-10-13T05:20:25.537 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-10-13T05:20:25.596 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-10-13T05:20:26.082 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-10-13T05:20:26.133 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-10-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-10-13T05:20:26.652 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-10-13T05:20:27.005 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-10-13T05:20:27.067 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-10-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-10-13T05:20:27.328 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.4 [1,566 kB] 2019-10-13T05:20:27.404 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-10-13T05:20:27.566 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-10-13T05:20:27.727 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.4 [1,566 kB] 2019-10-13T05:20:28.298 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 2,737 kB in 4s (756 kB/s) 2019-10-13T05:20:28.446 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package flex. 2019-10-13T05:20:28.498 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143618 files and directories currently installed.) 2019-10-13T05:20:28.502 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-10-13T05:20:28.544 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking flex (2.6.4-6) ... 2019-10-13T05:20:28.652 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 2,737 kB in 3s (797 kB/s) 2019-10-13T05:20:28.788 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package flex. 2019-10-13T05:20:28.842 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143618 files and directories currently installed.) 2019-10-13T05:20:28.846 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-10-13T05:20:28.886 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking flex (2.6.4-6) ... 2019-10-13T05:20:29.023 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-10-13T05:20:29.037 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../1-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T05:20:29.082 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:29.241 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-10-13T05:20:29.256 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T05:20:29.290 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:29.426 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package bison. 2019-10-13T05:20:29.444 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../2-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T05:20:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package bison. 2019-10-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T05:20:29.698 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-10-13T05:20:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../3-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-10-13T05:20:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T05:20:30.005 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-10-13T05:20:30.020 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-10-13T05:20:30.064 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T05:20:30.157 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-10-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../4-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-10-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T05:20:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-10-13T05:20:30.322 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-10-13T05:20:30.358 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T05:20:30.576 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libfl2:amd64. 2019-10-13T05:20:30.591 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../5-libfl2_2.6.4-6_amd64.deb ... 2019-10-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-10-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libfl2:amd64. 2019-10-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-libfl2_2.6.4-6_amd64.deb ... 2019-10-13T05:20:30.829 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-10-13T05:20:30.938 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-10-13T05:20:30.952 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../6-libfl-dev_2.6.4-6_amd64.deb ... 2019-10-13T05:20:30.996 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T05:20:31.090 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-10-13T05:20:31.104 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-libfl-dev_2.6.4-6_amd64.deb ... 2019-10-13T05:20:31.148 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-10-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../7-libssl-dev_1.1.1-1ubuntu2.1~18.04.4_amd64.deb ... 2019-10-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T05:20:31.392 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-10-13T05:20:31.409 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-libssl-dev_1.1.1-1ubuntu2.1~18.04.4_amd64.deb ... 2019-10-13T05:20:31.451 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T05:20:31.783 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up flex (2.6.4-6) ... 2019-10-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up flex (2.6.4-6) ... 2019-10-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-10-13T05:20:32.022 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-10-13T05:20:32.313 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T05:20:32.407 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T05:20:32.439 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:32.533 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:32.590 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:32.681 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:32.716 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-10-13T05:20:32.802 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-10-13T05:20:32.884 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T05:20:32.970 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T05:20:34.586 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:34.655 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T05:20:34.675 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-10-13T05:20:34.745 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-10-13T05:20:34.746 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T05:20:34.816 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T05:20:34.871 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T05:20:34.941 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T05:20:35.068 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T05:20:35.140 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:35.194 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi071.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T05:20:35.594 DEBUG:teuthology.parallel:result is None 2019-10-13T05:20:35.643 INFO:teuthology.orchestra.run.smithi129.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T05:20:35.644 DEBUG:teuthology.parallel:result is None 2019-10-13T05:20:35.644 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:35.645 INFO:teuthology.orchestra.run.smithi129:> sudo lsb_release -is 2019-10-13T05:20:35.690 INFO:teuthology.orchestra.run.smithi129.stdout:Ubuntu 2019-10-13T05:20:35.695 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:20:35.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:20:35.849 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:35.850 INFO:teuthology.orchestra.run.smithi129:> dpkg-query -W -f '${Version}' ceph 2019-10-13T05:20:35.875 INFO:teuthology.orchestra.run.smithi129.stdout:15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:20:35.876 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:20:35.876 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132-ga8ee8ac-1bionic is installed. 2019-10-13T05:20:35.876 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:35.877 INFO:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2019-10-13T05:20:35.932 INFO:teuthology.orchestra.run.smithi071.stdout:Ubuntu 2019-10-13T05:20:35.937 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T05:20:35.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:20:36.093 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.093 INFO:teuthology.orchestra.run.smithi071:> dpkg-query -W -f '${Version}' ceph 2019-10-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi071.stdout:15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:20:36.130 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T05:20:36.130 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132-ga8ee8ac-1bionic is installed. 2019-10-13T05:20:36.131 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-10-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T05:20:36.184 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-10-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T05:20:36.313 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.313 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T05:20:36.334 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.335 INFO:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T05:20:36.424 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-10-13T05:20:36.429 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.429 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T05:20:36.477 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.477 INFO:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T05:20:36.556 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.556 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T05:20:36.605 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.605 INFO:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T05:20:36.687 INFO:teuthology.run_tasks:Running task ceph... 2019-10-13T05:20:36.765 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-10-13T05:20:36.766 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.766 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2019-10-13T05:20:36.770 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.770 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2019-10-13T05:20:36.788 INFO:tasks.ceph:Disabling ceph logrotate... 2019-10-13T05:20:36.789 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.789 INFO:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T05:20:36.855 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.855 INFO:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T05:20:36.871 INFO:tasks.ceph:Creating extra log directories... 2019-10-13T05:20:36.872 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:36.872 INFO:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T05:20:36.939 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.939 INFO:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T05:20:36.955 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2019-10-13T05:20:36.956 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2019-10-13T05:20:36.956 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2019-10-13T05:20:36.956 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:36.956 INFO:teuthology.orchestra.run.smithi129:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T05:20:37.068 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.068 INFO:teuthology.orchestra.run.smithi129:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:37.171 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:37.172 INFO:teuthology.orchestra.run.smithi071:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T05:20:37.220 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:37.220 INFO:teuthology.orchestra.run.smithi071:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:37.323 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-10-13T05:20:37.323 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10000, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd op complaint time': 180}}} 2019-10-13T05:20:37.324 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, {'ceph': None}, {'ceph-fuse': None}, {'check-counter': {'counters': {'mds': ['mds.dir_split']}}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.4313', 'owner': 'scheduled_pdonnell@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'targets': {'smithi129.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzI9vbs4SRq7J30Hxwo+uim8HbM2JLcX6IOFf0smyWrVynQDgxvjdloZQvh/5F1ADH5eihMnPs+D9i0UL6CldXaMSR3P413KMzfm0TWa3mqA9T1STToiVz1JP/8v4U3FJoQroz3ZfXkBQAdavAe5BnoY0VK0PzCCHF/Vo7FE68n9CM4yO9e/XoBkoV/hV48JdE8ncsiv1ejAMRh9tgEIA1t/MEG548LJSPqMbDg0JZ2QTpg6DzPXd7SRFlb6rUJ5e5BVRSLcTZC0DyB+eKLPDpapeWxVx/4X6/K3VFZ5iuwfa9ZzcB5WIJlsVx4/O0l4R5bxfDAhrI1OHiBDYgYFKr', 'smithi071.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMb+Sj+4T/9nm4w1P0FJXLR8/xXnWSHI8tIy559zHUKDvuGlEAk3v9y97kKRV0jlAtxXQKoZhR+0fyJEeOK3PppvH7tUEcqK94JCvGNDoRo+tKwIRrjs3Ux0aQOjB9E41/62sxqby7t7b8K97uwok3Lattw4/cPtCJAov231WLM9Rrve51F1EDQFo5slzsMz9yF92VcHpPUTT2bfpTQUCmx075StocKB+kQXvcKg+NavyePCq1U9udITC6kLPS47g+k6fptrAh6jHNPaFHXQGbfHCWdvqHZyYs52fNVT1iy0Ky0IUvBDTKg2D97KhePQexo/zPsCT3DUpsy+7OTXnR'}, 'job_id': '4407820', 'tube': 'smithi', 'suite_branch': 'wip-pdonnell-testing-20191012.025926', 'priority': 95, 'os_version': '18.04', 'first_in_suite': False, 'branch': 'wip-pdonnell-testing-20191012.025926', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': 'pdonnell@redhat.com', 'archive_path': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407820', 'description': 'fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'name': 'pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10000, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd op complaint time': 180}}, 'fs': 'xfs', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95, 'bdev async discard': True}}}, 'install': {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, 'admin_socket': {'branch': 'wip-pdonnell-testing-20191012.025926'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'branch': 'wip-pdonnell-testing-20191012.025926'}}, 'os_type': 'ubuntu', 'suite_sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T05:20:37.324 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:37.324 INFO:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T05:20:37.392 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.392 INFO:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T05:20:37.401 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:37.401 INFO:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T05:20:37.468 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.469 INFO:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T05:20:37.578 DEBUG:teuthology.orchestra.remote:smithi129:/scratch_devs is 72B 2019-10-13T05:20:37.632 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T05:20:37.632 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.632 INFO:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T05:20:37.640 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-13T05:20:37.641 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:37.641 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 450 Links: 1 2019-10-13T05:20:37.642 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:37.643 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2019-10-13 05:17:36.551205609 +0000 2019-10-13T05:20:37.643 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2019-10-13 05:17:36.455253611 +0000 2019-10-13T05:20:37.643 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2019-10-13 05:17:36.455253611 +0000 2019-10-13T05:20:37.643 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2019-10-13T05:20:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2019-10-13T05:20:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2019-10-13T05:20:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000410797 s, 1.2 MB/s 2019-10-13T05:20:37.654 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.654 INFO:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T05:20:37.726 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-13T05:20:37.727 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:37.727 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 456 Links: 1 2019-10-13T05:20:37.727 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:37.727 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2019-10-13 05:17:36.687137606 +0000 2019-10-13T05:20:37.727 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2019-10-13 05:17:36.683139606 +0000 2019-10-13T05:20:37.728 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2019-10-13 05:17:36.683139606 +0000 2019-10-13T05:20:37.728 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2019-10-13T05:20:37.736 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2019-10-13T05:20:37.736 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2019-10-13T05:20:37.736 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000388573 s, 1.3 MB/s 2019-10-13T05:20:37.741 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.741 INFO:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T05:20:37.815 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-13T05:20:37.816 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:37.816 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 462 Links: 1 2019-10-13T05:20:37.816 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:37.816 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2019-10-13 05:17:36.879041601 +0000 2019-10-13T05:20:37.817 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2019-10-13 05:17:36.875043602 +0000 2019-10-13T05:20:37.817 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2019-10-13 05:17:36.875043602 +0000 2019-10-13T05:20:37.817 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2019-10-13T05:20:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2019-10-13T05:20:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2019-10-13T05:20:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000335672 s, 1.5 MB/s 2019-10-13T05:20:37.829 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.829 INFO:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T05:20:37.901 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-13T05:20:37.901 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 468 Links: 1 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2019-10-13 05:19:51.549843522 +0000 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2019-10-13 05:17:37.162899595 +0000 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2019-10-13 05:17:37.162899595 +0000 2019-10-13T05:20:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2019-10-13T05:20:37.910 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2019-10-13T05:20:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2019-10-13T05:20:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000399581 s, 1.3 MB/s 2019-10-13T05:20:37.915 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T05:20:37.915 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T05:20:37.916 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:37.916 INFO:teuthology.orchestra.run.smithi129:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T05:20:37.987 INFO:teuthology.orchestra.run.smithi129.stdout:lrwxrwxrwx 1 root root 9 Oct 13 05:14 /dev/disk/by-id/wwn-0x5000c5009292d832 -> ../../sda 2019-10-13T05:20:37.988 INFO:teuthology.orchestra.run.smithi129.stdout:lrwxrwxrwx 1 root root 10 Oct 13 05:14 /dev/disk/by-id/wwn-0x5000c5009292d832-part1 -> ../../sda1 2019-10-13T05:20:37.988 INFO:tasks.ceph:dev map: {} 2019-10-13T05:20:37.998 DEBUG:teuthology.orchestra.remote:smithi071:/scratch_devs is 72B 2019-10-13T05:20:38.009 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T05:20:38.009 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.010 INFO:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 439 Links: 1 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2019-10-13 05:17:35.964600037 +0000 2019-10-13T05:20:38.021 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2019-10-13 05:17:35.960602038 +0000 2019-10-13T05:20:38.022 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2019-10-13 05:17:35.960602038 +0000 2019-10-13T05:20:38.022 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2019-10-13T05:20:38.026 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2019-10-13T05:20:38.026 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2019-10-13T05:20:38.026 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000385184 s, 1.3 MB/s 2019-10-13T05:20:38.030 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.030 INFO:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T05:20:38.102 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-13T05:20:38.103 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:38.103 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 445 Links: 1 2019-10-13T05:20:38.103 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:38.103 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2019-10-13 05:17:36.192486033 +0000 2019-10-13T05:20:38.103 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2019-10-13 05:17:36.188488032 +0000 2019-10-13T05:20:38.104 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2019-10-13 05:17:36.188488032 +0000 2019-10-13T05:20:38.104 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2019-10-13T05:20:38.112 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2019-10-13T05:20:38.112 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2019-10-13T05:20:38.112 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000304931 s, 1.7 MB/s 2019-10-13T05:20:38.124 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.124 INFO:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T05:20:38.198 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-13T05:20:38.198 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 451 Links: 1 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2019-10-13 05:17:36.572296023 +0000 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2019-10-13 05:17:36.456354027 +0000 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2019-10-13 05:17:36.456354027 +0000 2019-10-13T05:20:38.199 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2019-10-13T05:20:38.207 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2019-10-13T05:20:38.208 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2019-10-13T05:20:38.208 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000338981 s, 1.5 MB/s 2019-10-13T05:20:38.213 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.213 INFO:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T05:20:38.285 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-13T05:20:38.285 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T05:20:38.285 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 457 Links: 1 2019-10-13T05:20:38.285 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:20:38.285 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2019-10-13 05:19:51.402454683 +0000 2019-10-13T05:20:38.286 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2019-10-13 05:17:36.684240021 +0000 2019-10-13T05:20:38.286 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2019-10-13 05:17:36.684240021 +0000 2019-10-13T05:20:38.286 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2019-10-13T05:20:38.295 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2019-10-13T05:20:38.295 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2019-10-13T05:20:38.295 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000418704 s, 1.2 MB/s 2019-10-13T05:20:38.300 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T05:20:38.300 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T05:20:38.300 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.300 INFO:teuthology.orchestra.run.smithi071:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T05:20:38.374 INFO:teuthology.orchestra.run.smithi071.stdout:lrwxrwxrwx 1 root root 9 Oct 13 05:14 /dev/disk/by-id/wwn-0x5000c50091e19c4b -> ../../sda 2019-10-13T05:20:38.374 INFO:teuthology.orchestra.run.smithi071.stdout:lrwxrwxrwx 1 root root 10 Oct 13 05:14 /dev/disk/by-id/wwn-0x5000c50091e19c4b-part1 -> ../../sda1 2019-10-13T05:20:38.375 INFO:tasks.ceph:dev map: {} 2019-10-13T05:20:38.375 INFO:tasks.ceph:Generating config... 2019-10-13T05:20:38.376 INFO:tasks.ceph:[mds] mds debug frag = True 2019-10-13T05:20:38.376 INFO:tasks.ceph:[mds] mds bal split bits = 3 2019-10-13T05:20:38.376 INFO:tasks.ceph:[mds] mds bal split size = 100 2019-10-13T05:20:38.376 INFO:tasks.ceph:[mds] debug mds = 20 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] mds bal merge size = 5 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] debug ms = 1 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] mds bal frag = True 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] mds op complaint time = 180 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2019-10-13T05:20:38.377 INFO:tasks.ceph:[mds] osd op complaint time = 180 2019-10-13T05:20:38.378 INFO:tasks.ceph:[mds] mds verify scatter = True 2019-10-13T05:20:38.378 INFO:tasks.ceph:[client] debug ms = 1 2019-10-13T05:20:38.378 INFO:tasks.ceph:[client] debug client = 20 2019-10-13T05:20:38.378 INFO:tasks.ceph:[client] client mount timeout = 600 2019-10-13T05:20:38.378 INFO:tasks.ceph:[mon] debug mon = 20 2019-10-13T05:20:38.378 INFO:tasks.ceph:[mon] debug paxos = 20 2019-10-13T05:20:38.378 INFO:tasks.ceph:[mon] mon op complaint time = 120 2019-10-13T05:20:38.379 INFO:tasks.ceph:[mon] debug ms = 1 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] debug ms = 1 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] filestore flush min = 0 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] debug osd = 25 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-10-13T05:20:38.379 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] osd_max_omap_entries_per_request = 10000 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] debug journal = 20 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-10-13T05:20:38.380 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-10-13T05:20:38.381 INFO:tasks.ceph:[osd] bdev async discard = True 2019-10-13T05:20:38.381 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-10-13T05:20:38.381 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-10-13T05:20:38.381 INFO:tasks.ceph:[osd] debug filestore = 20 2019-10-13T05:20:38.381 INFO:tasks.ceph:[osd] bdev enable discard = True 2019-10-13T05:20:38.382 INFO:tasks.ceph:[osd] osd op complaint time = 180 2019-10-13T05:20:38.382 INFO:tasks.ceph:Setting up mon.a... 2019-10-13T05:20:38.382 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.382 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-10-13T05:20:38.418 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2019-10-13T05:20:38.420 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.420 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-10-13T05:20:38.519 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.520 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T05:20:38.602 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129'), ('mon.c', '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]'), ('mon.b', '172.21.15.71')] 2019-10-13T05:20:38.602 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.602 INFO:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.129 --addv c '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]' --add b 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:38.701 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:38.702 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid 678bb30f-383e-4566-b2c7-8d4fe3e6c97c 2019-10-13T05:20:38.702 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2019-10-13T05:20:38.702 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 678bb30f-383e-4566-b2c7-8d4fe3e6c97c 2019-10-13T05:20:38.702 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2019-10-13T05:20:38.701844+0000 2019-10-13T05:20:38.703 INFO:teuthology.orchestra.run.smithi129.stdout:created 2019-10-13T05:20:38.701844+0000 2019-10-13T05:20:38.703 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 0 (unknown) 2019-10-13T05:20:38.703 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2019-10-13T05:20:38.703 INFO:teuthology.orchestra.run.smithi129.stdout:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2019-10-13T05:20:38.704 INFO:teuthology.orchestra.run.smithi129.stdout:2: [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] mon.c 2019-10-13T05:20:38.704 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-10-13T05:20:38.705 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 678bb30f-383e-4566-b2c7-8d4fe3e6c97c... 2019-10-13T05:20:38.708 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:38.709 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T05:20:38.713 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.713 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T05:20:38.805 INFO:tasks.ceph:Creating admin key on mon.a... 2019-10-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-10-13T05:20:38.846 INFO:tasks.ceph:Copying monmap to all nodes... 2019-10-13T05:20:38.922 DEBUG:teuthology.orchestra.remote:smithi129:/etc/ceph/ceph.keyring is 206B 2019-10-13T05:20:38.992 DEBUG:teuthology.orchestra.remote:smithi129:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-10-13T05:20:39.005 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2019-10-13T05:20:39.005 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.005 INFO:teuthology.orchestra.run.smithi129:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T05:20:39.062 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.063 INFO:teuthology.orchestra.run.smithi129:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:39.172 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2019-10-13T05:20:39.172 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:39.172 INFO:teuthology.orchestra.run.smithi071:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T05:20:39.230 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:39.231 INFO:teuthology.orchestra.run.smithi071:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:39.343 INFO:tasks.ceph:Setting up mon nodes... 2019-10-13T05:20:39.344 INFO:tasks.ceph:Setting up mgr nodes... 2019-10-13T05:20:39.344 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.344 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T05:20:39.438 INFO:tasks.ceph:Setting up mds nodes... 2019-10-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T05:20:39.488 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.489 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2019-10-13T05:20:39.573 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:39.573 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T05:20:39.618 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T05:20:39.733 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:39.734 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2019-10-13T05:20:39.813 INFO:tasks.ceph_client:Setting up client nodes... 2019-10-13T05:20:39.814 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.814 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-10-13T05:20:39.850 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-10-13T05:20:39.862 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-10-13T05:20:39.862 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {}} 2019-10-13T05:20:39.862 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.863 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-10-13T05:20:39.942 INFO:tasks.ceph:{} 2019-10-13T05:20:39.943 INFO:tasks.ceph:{} 2019-10-13T05:20:39.943 INFO:tasks.ceph:osd.0 2019-10-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-10-13T05:20:40.023 INFO:tasks.ceph:{} 2019-10-13T05:20:40.023 INFO:tasks.ceph:{} 2019-10-13T05:20:40.023 INFO:tasks.ceph:osd.1 2019-10-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-10-13T05:20:40.103 INFO:tasks.ceph:{} 2019-10-13T05:20:40.103 INFO:tasks.ceph:{} 2019-10-13T05:20:40.103 INFO:tasks.ceph:osd.2 2019-10-13T05:20:40.103 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:40.103 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-10-13T05:20:40.184 INFO:tasks.ceph:{} 2019-10-13T05:20:40.184 INFO:tasks.ceph:{} 2019-10-13T05:20:40.185 INFO:tasks.ceph:osd.3 2019-10-13T05:20:40.185 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:40.185 INFO:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.301+0000 7f9fdbe81d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-10-13T05:20:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.301+0000 7f9fdbe81d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-10-13T05:20:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.301+0000 7f9fdbe81d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T05:20:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.301+0000 7f9fdbe81d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T05:20:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.301+0000 7f9fdbe81d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T05:20:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:40.305+0000 7f9fdbe81d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-10-13T05:20:42.753 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:42.754 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-10-13T05:20:42.770 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:42.770 INFO:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:42.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-10-13T05:20:42.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-10-13T05:20:42.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T05:20:42.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T05:20:42.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T05:20:42.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:42.869+0000 7f013d136d80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-10-13T05:20:45.341 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:45.342 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-10-13T05:20:45.428 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:45.428 INFO:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:45.532 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-10-13T05:20:45.532 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-10-13T05:20:45.532 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T05:20:45.532 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T05:20:45.533 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T05:20:45.533 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:45.529+0000 7f0be2fded80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-10-13T05:20:47.980 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:47.980 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-10-13T05:20:48.060 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:48.060 INFO:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-10-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-10-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:20:48.161+0000 7f935e901d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-10-13T05:20:50.620 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:20:50.620 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-10-13T05:20:50.701 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:50.701 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-10-13T05:20:50.718 INFO:tasks.ceph:{} 2019-10-13T05:20:50.718 INFO:tasks.ceph:{} 2019-10-13T05:20:50.718 INFO:tasks.ceph:osd.4 2019-10-13T05:20:50.718 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:50.718 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-10-13T05:20:50.797 INFO:tasks.ceph:{} 2019-10-13T05:20:50.797 INFO:tasks.ceph:{} 2019-10-13T05:20:50.797 INFO:tasks.ceph:osd.5 2019-10-13T05:20:50.797 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:50.798 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-10-13T05:20:50.877 INFO:tasks.ceph:{} 2019-10-13T05:20:50.878 INFO:tasks.ceph:{} 2019-10-13T05:20:50.878 INFO:tasks.ceph:osd.6 2019-10-13T05:20:50.878 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:50.878 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-10-13T05:20:50.962 INFO:tasks.ceph:{} 2019-10-13T05:20:50.962 INFO:tasks.ceph:{} 2019-10-13T05:20:50.963 INFO:tasks.ceph:osd.7 2019-10-13T05:20:50.963 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:50.963 INFO:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:51.081 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-10-13T05:20:51.081 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-10-13T05:20:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T05:20:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T05:20:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T05:20:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:51.074+0000 7f1532760d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-10-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-10-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:53.628 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-10-13T05:20:53.628 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-10-13T05:20:53.628 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T05:20:53.628 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T05:20:53.629 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T05:20:53.629 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:53.622+0000 7fd3800add80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-10-13T05:20:56.100 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:56.101 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-10-13T05:20:56.182 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:56.182 INFO:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:56.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-10-13T05:20:56.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-10-13T05:20:56.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T05:20:56.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T05:20:56.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T05:20:56.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:56.278+0000 7f5068291d80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-10-13T05:20:58.736 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:58.736 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-10-13T05:20:58.818 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:20:58.818 INFO:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:20:58.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-10-13T05:20:58.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-10-13T05:20:58.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T05:20:58.928 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T05:20:58.928 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T05:20:58.928 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:20:58.922+0000 7f8ab6a5ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-10-13T05:21:01.392 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:01.392 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-10-13T05:21:01.472 INFO:tasks.ceph:Reading keys from all nodes... 2019-10-13T05:21:01.472 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.473 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:01.480 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.BHChQSsdjw 2019-10-13T05:21:01.481 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.481 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.BHChQSsdjw 2019-10-13T05:21:01.561 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.561 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.BHChQSsdjw 2019-10-13T05:21:01.713 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.BHChQSsdjw is 56B 2019-10-13T05:21:01.726 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.727 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.BHChQSsdjw 2019-10-13T05:21:01.734 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.734 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:01.804 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.oASam0HC7Y 2019-10-13T05:21:01.805 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.805 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.oASam0HC7Y 2019-10-13T05:21:01.884 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:01.885 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.oASam0HC7Y 2019-10-13T05:21:02.039 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.oASam0HC7Y is 56B 2019-10-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.oASam0HC7Y 2019-10-13T05:21:02.059 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.059 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:02.132 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.GkluT8smTf 2019-10-13T05:21:02.132 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.132 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.GkluT8smTf 2019-10-13T05:21:02.214 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.214 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.GkluT8smTf 2019-10-13T05:21:02.366 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.GkluT8smTf is 56B 2019-10-13T05:21:02.419 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.420 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.GkluT8smTf 2019-10-13T05:21:02.428 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.428 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:02.503 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.dYdOVBtADA 2019-10-13T05:21:02.503 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.503 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.dYdOVBtADA 2019-10-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.dYdOVBtADA 2019-10-13T05:21:02.737 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.dYdOVBtADA is 56B 2019-10-13T05:21:02.756 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.756 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.dYdOVBtADA 2019-10-13T05:21:02.763 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.763 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:02.838 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.ImE4I6TojI 2019-10-13T05:21:02.839 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.839 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.ImE4I6TojI 2019-10-13T05:21:02.920 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:02.920 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.ImE4I6TojI 2019-10-13T05:21:03.071 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.ImE4I6TojI is 56B 2019-10-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.ImE4I6TojI 2019-10-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:03.161 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.RxfZgAAVOe 2019-10-13T05:21:03.161 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:03.162 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.RxfZgAAVOe 2019-10-13T05:21:03.242 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:03.242 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.RxfZgAAVOe 2019-10-13T05:21:03.394 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.RxfZgAAVOe is 56B 2019-10-13T05:21:03.406 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:03.406 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.RxfZgAAVOe 2019-10-13T05:21:03.414 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.414 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:03.421 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.npsiegIrVb 2019-10-13T05:21:03.422 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.422 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.npsiegIrVb 2019-10-13T05:21:03.509 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.509 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.npsiegIrVb 2019-10-13T05:21:03.661 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.npsiegIrVb is 56B 2019-10-13T05:21:03.673 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.673 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.npsiegIrVb 2019-10-13T05:21:03.681 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.682 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:03.752 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.LKESAdScTd 2019-10-13T05:21:03.753 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.753 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.LKESAdScTd 2019-10-13T05:21:03.833 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:03.833 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.LKESAdScTd 2019-10-13T05:21:03.993 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.LKESAdScTd is 56B 2019-10-13T05:21:04.005 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.005 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.LKESAdScTd 2019-10-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.kdA03uDBpn 2019-10-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.kdA03uDBpn 2019-10-13T05:21:04.165 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.165 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.kdA03uDBpn 2019-10-13T05:21:04.317 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.kdA03uDBpn is 56B 2019-10-13T05:21:04.367 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.367 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.kdA03uDBpn 2019-10-13T05:21:04.375 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.375 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:04.447 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.BhaVquMf8y 2019-10-13T05:21:04.448 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.448 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.BhaVquMf8y 2019-10-13T05:21:04.527 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.528 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.BhaVquMf8y 2019-10-13T05:21:04.685 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.BhaVquMf8y is 56B 2019-10-13T05:21:04.697 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.697 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.BhaVquMf8y 2019-10-13T05:21:04.704 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.705 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:04.775 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.FK0UKGQYcV 2019-10-13T05:21:04.775 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.776 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.FK0UKGQYcV 2019-10-13T05:21:04.858 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:04.858 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.FK0UKGQYcV 2019-10-13T05:21:05.009 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.FK0UKGQYcV is 56B 2019-10-13T05:21:05.021 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.021 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.FK0UKGQYcV 2019-10-13T05:21:05.028 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.029 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:05.099 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.vfXtVEeFeZ 2019-10-13T05:21:05.100 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.100 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.vfXtVEeFeZ 2019-10-13T05:21:05.180 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.180 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.vfXtVEeFeZ 2019-10-13T05:21:05.331 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.vfXtVEeFeZ is 56B 2019-10-13T05:21:05.343 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.344 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.vfXtVEeFeZ 2019-10-13T05:21:05.361 DEBUG:teuthology.orchestra.remote:smithi129:/etc/ceph/ceph.client.0.keyring is 59B 2019-10-13T05:21:05.373 INFO:tasks.ceph:Adding keys to all mons... 2019-10-13T05:21:05.374 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.374 INFO:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T05:21:05.408 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.409 INFO:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T05:21:05.419 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAntKJdPuQTFxAAyGKGEo3mmVnb7C6Z1d7V4w== 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.a] 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAntKJdolcAHRAAZ1UnLLqHnP0ji1amAoUnTg== 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAotKJdIr1DEhAA9tpnL/iVpNSqvnwD/m8TZQ== 2019-10-13T05:21:05.420 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAqtKJdvGMGNBAAnTn01EEWtPli490JF3Ovaw== 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAttKJdAqOyHxAAJZ8OFonXQm7Ry7+g7emOLg== 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAwtKJdfiXICRAAIh56BxJVz7ejylCsLvk9CQ== 2019-10-13T05:21:05.421 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAntKJdDAPKGRAAJnVHYaykugaWwQDDXaDIqw== 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.b] 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAntKJdDOTSJBAALRqstgHH8Q6QhvnxuwPO/Q== 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAztKJdo4bRBBAADh8uXx2c8i3u7lDxI93mLw== 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2019-10-13T05:21:05.422 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA1tKJdGnJqJRAAXDRXx9l5bcCkzEacLW0FZw== 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA4tKJds3vqEBAAWtbRGY/B9QMyEyZ9UPLqBg== 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA6tKJdwgRANxAAT/eHIfhdEjpGZiSFBWfkOw== 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2019-10-13T05:21:05.423 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAntKJdmM2wMhAAptrKweaMRJRjpVH4hZXMKA== 2019-10-13T05:21:05.454 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2019-10-13T05:21:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAntKJdPuQTFxAAyGKGEo3mmVnb7C6Z1d7V4w== 2019-10-13T05:21:05.454 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2019-10-13T05:21:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAntKJdolcAHRAAZ1UnLLqHnP0ji1amAoUnTg== 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAotKJdIr1DEhAA9tpnL/iVpNSqvnwD/m8TZQ== 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAqtKJdvGMGNBAAnTn01EEWtPli490JF3Ovaw== 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAttKJdAqOyHxAAJZ8OFonXQm7Ry7+g7emOLg== 2019-10-13T05:21:05.455 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAwtKJdfiXICRAAIh56BxJVz7ejylCsLvk9CQ== 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAntKJdDAPKGRAAJnVHYaykugaWwQDDXaDIqw== 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAntKJdDOTSJBAALRqstgHH8Q6QhvnxuwPO/Q== 2019-10-13T05:21:05.456 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAztKJdo4bRBBAADh8uXx2c8i3u7lDxI93mLw== 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA1tKJdGnJqJRAAXDRXx9l5bcCkzEacLW0FZw== 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA4tKJds3vqEBAAWtbRGY/B9QMyEyZ9UPLqBg== 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2019-10-13T05:21:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA6tKJdwgRANxAAT/eHIfhdEjpGZiSFBWfkOw== 2019-10-13T05:21:05.458 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2019-10-13T05:21:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAntKJdmM2wMhAAptrKweaMRJRjpVH4hZXMKA== 2019-10-13T05:21:05.460 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.460 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T05:21:05.470 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.470 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T05:21:05.554 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.554 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.558 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.558 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.673 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.673 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.677 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.677 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.785 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.785 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.793 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.793 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.870 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.870 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.876 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.876 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.983 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:05.983 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:05.987 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:05.987 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.094 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.094 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T05:21:06.098 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.098 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T05:21:06.206 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.206 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.210 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.210 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.321 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.321 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.325 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.325 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.431 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.431 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.435 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.435 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.542 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.542 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.547 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.547 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.655 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.655 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.660 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.660 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T05:21:06.766 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:06.766 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T05:21:06.769 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.770 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T05:21:06.880 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-10-13T05:21:06.880 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.880 INFO:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-10-13T05:21:06.959 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:06.960 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T05:21:07.080 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.080 INFO:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-10-13T05:21:07.094 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.094 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-10-13T05:21:07.110 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.110 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T05:21:07.247 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.247 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-10-13T05:21:07.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.263 INFO:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-10-13T05:21:07.327 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.327 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:07.331 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.331 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:07.343 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.344 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T05:21:07.493 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.493 INFO:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-10-13T05:21:07.508 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.508 INFO:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:21:07.575 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.576 INFO:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T05:21:07.583 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-10-13T05:21:07.583 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.583 INFO:teuthology.orchestra.run.smithi129:> which systemctl 2019-10-13T05:21:07.653 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/systemctl 2019-10-13T05:21:07.654 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-13T05:21:07.654 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.654 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-13T05:21:07.721 INFO:tasks.ceph.mon.a:Started 2019-10-13T05:21:07.721 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.721 INFO:teuthology.orchestra.run.smithi071:> which systemctl 2019-10-13T05:21:07.729 INFO:teuthology.orchestra.run.smithi071.stdout:/bin/systemctl 2019-10-13T05:21:07.729 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-13T05:21:07.730 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.730 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-13T05:21:07.796 INFO:tasks.ceph.mon.b:Started 2019-10-13T05:21:07.797 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-13T05:21:07.797 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.797 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-13T05:21:07.801 INFO:tasks.ceph.mon.c:Started 2019-10-13T05:21:07.801 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-10-13T05:21:07.801 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-10-13T05:21:07.802 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.802 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-10-13T05:21:07.806 INFO:tasks.ceph.mgr.y:Started 2019-10-13T05:21:07.806 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-10-13T05:21:07.806 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:07.806 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-10-13T05:21:07.810 INFO:tasks.ceph.mgr.x:Started 2019-10-13T05:21:07.810 INFO:tasks.ceph:Setting crush tunables to default 2019-10-13T05:21:07.810 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:07.810 INFO:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2019-10-13T05:21:18.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 Processor -- start 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 -- start start 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00a7a30 con 0x7f65e00a1c80 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00aa3e0 con 0x7f65e0144a10 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65e5a21700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00aa610 con 0x7f65e00a7520 2019-10-13T05:21:18.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.905+0000 7f65deffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65dffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 0x7f65e014eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58382/0 (socket says 172.21.15.129:58382) 2019-10-13T05:21:18.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65dffff700 1 -- 172.21.15.129:0/3264528247 learned_addr learned my addr 172.21.15.129:0/3264528247 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:07.909+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.113+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.113+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.113+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.113+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.113+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.517+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.517+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.517+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.517+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:08.517+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:09.321+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:09.321+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:09.321+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:09.321+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:09.321+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:10.925+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.130 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:10.925+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:10.925+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:10.925+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:10.925+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2019-10-13T05:21:18.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:14.129+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:14.129+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T05:21:18.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:14.129+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:14.129+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:14.129+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 6.400000 2019-10-13T05:21:18.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.793+0000 7f65dffff700 1 -- 172.21.15.129:0/3264528247 >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 legacy=0x7f65e014eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-10-13T05:21:18.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.793+0000 7f65dffff700 1 -- 172.21.15.129:0/3264528247 >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 legacy=0x7f65e014eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-10-13T05:21:18.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.793+0000 7f65dffff700 1 --1- 172.21.15.129:0/3264528247 >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 0x7f65e014eb80 :-1 s=OPENED pgs=6 cs=1 l=1).handle_message read tag failed 2019-10-13T05:21:18.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.793+0000 7f65dffff700 1 --1- 172.21.15.129:0/3264528247 >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 0x7f65e014eb80 :-1 s=OPENED pgs=6 cs=1 l=1).fault on lossy channel, failing 2019-10-13T05:21:18.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 msgr2=0x7f65e00a9e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-10-13T05:21:18.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.129:3300/0 conn(0x7f65e00a7520 0x7f65e00a9e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:18.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 -- 172.21.15.129:0/3264528247 >> v1:172.21.15.129:6789/0 conn(0x7f65e0144a10 legacy=0x7f65e014eb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-10-13T05:21:18.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 -- 172.21.15.129:0/3264528247 >> v1:172.21.15.71:6789/0 conn(0x7f65e00a1c80 legacy=0x7f65e00a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:18.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:18.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:18.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 -- 172.21.15.129:0/3264528247 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00aa610 con 0x7f65cc0030f0 2019-10-13T05:21:18.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65dd7fa700 1 -- 172.21.15.129:0/3264528247 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc008830 con 0x7f65cc005c00 2019-10-13T05:21:18.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65df7fe700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 msgr2=0x7f65cc0055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65df7fe700 1 -- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 msgr2=0x7f65cc0055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 msgr2=0x7f65cc008100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65deffd700 1 -- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 msgr2=0x7f65cc008100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T05:21:18.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T05:21:18.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:17.909+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T05:21:18.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65deffd700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 -- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 msgr2=0x7f65cc008100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65cc005c00 0x7f65cc008100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 -- 172.21.15.129:0/3264528247 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0144e30 con 0x7f65cc0030f0 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f65d0001000 tx=0x7f65d4009d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> v2:172.21.15.71:3300/0 conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f65d0001000 tx=0x7f65d4009d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f65d0001000 tx=0x7f65d4009d60).ready entity=mon.0 client_cookie=11bcebe44d527166 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:18.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f65d0001000 tx=0x7f65d4009d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65de7fc700 1 -- 172.21.15.129:0/3264528247 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d0011840 con 0x7f65cc0030f0 2019-10-13T05:21:18.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f65d0001000 tx=0x7f65d4009d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65df7fe700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f65d0001000 tx=0x7f65d4009d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65de7fc700 1 -- 172.21.15.129:0/3264528247 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0011460 con 0x7f65cc0030f0 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65de7fc700 1 -- 172.21.15.129:0/3264528247 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d0006350 con 0x7f65cc0030f0 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 -- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 msgr2=0x7f65cc0055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f65d0001000 tx=0x7f65d4009d60).stop 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65dffff700 1 -- 172.21.15.129:0/3264528247 reap_dead start 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 -- 172.21.15.129:0/3264528247 shutdown_connections 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 --2- 172.21.15.129:0/3264528247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0030f0 0x7f65cc0055b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:18.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 -- 172.21.15.129:0/3264528247 shutdown_connections 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 -- 172.21.15.129:0/3264528247 wait complete. 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.113+0000 7f65e5a21700 1 -- 172.21.15.129:0/3264528247 >> 172.21.15.129:0/3264528247 conn(0x7f65e005e870 msgr2=0x7f65e009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 Processor -- start 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- start start 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f65e00a5cb0 0x7f65e01212b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:18.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65e0121800 0x7f65e00a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:18.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e009d070 con 0x7f65e00a4cf0 2019-10-13T05:21:18.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00a6bb0 con 0x7f65e00a5cb0 2019-10-13T05:21:18.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00a62b0 con 0x7f65e0121800 2019-10-13T05:21:18.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65e0121800 0x7f65e00a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65e0121800 0x7f65e00a47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40472/0 (socket says 172.21.15.129:40472) 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 -- 172.21.15.129:0/1240308115 learned_addr learned my addr 172.21.15.129:0/1240308115 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65df7fe700 1 --2- 172.21.15.129:0/1240308115 >> v2:172.21.15.71:3300/0 conn(0x7f65e00a5cb0 0x7f65e01212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 -- 172.21.15.129:0/1240308115 >> v2:172.21.15.71:3300/0 conn(0x7f65e00a5cb0 msgr2=0x7f65e01212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> v2:172.21.15.71:3300/0 conn(0x7f65e00a5cb0 0x7f65e01212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 -- 172.21.15.129:0/1240308115 >> v1:172.21.15.71:6789/0 conn(0x7f65e00a4cf0 legacy=0x7f65e0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 -- 172.21.15.129:0/1240308115 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00a67f0 con 0x7f65e0121800 2019-10-13T05:21:18.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> v2:172.21.15.129:3300/0 conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> v2:172.21.15.129:3300/0 conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d4055820 con 0x7f65e0121800 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d4007150 con 0x7f65e0121800 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e00a5110 con 0x7f65e0121800 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d4055e00 con 0x7f65e0121800 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e0070a00 con 0x7f65e0121800 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 341+0+0 (secure 0 0 0) 0x7f65d4005640 con 0x7f65e0121800 2019-10-13T05:21:18.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.117+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f65d4009850 con 0x7f65e0121800 2019-10-13T05:21:18.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.145+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e00555c0 con 0x7f65e0121800 2019-10-13T05:21:18.150 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.149+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.149+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7f65d40560c0 con 0x7f65e0121800 2019-10-13T05:21:18.244 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.241+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f65e005a210 con 0x7f65e0121800 2019-10-13T05:21:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.249+0000 7f65deffd700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.249+0000 7f65c7fff700 1 -- 172.21.15.129:0/1240308115 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f65d40562e0 con 0x7f65e0121800 2019-10-13T05:21:18.259 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2019-10-13T05:21:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 msgr2=0x7f65e00a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:18.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f65e0129520 tx=0x7f65d4002f70).stop 2019-10-13T05:21:18.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 shutdown_connections 2019-10-13T05:21:18.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 --2- 172.21.15.129:0/1240308115 >> v2:172.21.15.71:3300/0 conn(0x7f65e00a5cb0 0x7f65e01212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:18.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 --2- 172.21.15.129:0/1240308115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65e0121800 0x7f65e00a47a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:18.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 shutdown_connections 2019-10-13T05:21:18.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 wait complete. 2019-10-13T05:21:18.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:18.257+0000 7f65e5a21700 1 -- 172.21.15.129:0/1240308115 >> 172.21.15.129:0/1240308115 conn(0x7f65e00a17a0 msgr2=0x7f65e0063800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:18.274 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-10-13T05:21:18.275 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.275 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:18.283 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.3VMuPkuN5w 2019-10-13T05:21:18.283 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.283 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.3VMuPkuN5w 2019-10-13T05:21:18.362 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.362 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.3VMuPkuN5w 2019-10-13T05:21:18.513 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.3VMuPkuN5w is 37B 2019-10-13T05:21:18.524 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.524 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.3VMuPkuN5w 2019-10-13T05:21:18.532 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.532 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:18.602 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.iGnLZRe4ra 2019-10-13T05:21:18.603 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.603 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.iGnLZRe4ra 2019-10-13T05:21:18.682 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.683 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.iGnLZRe4ra 2019-10-13T05:21:18.832 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.iGnLZRe4ra is 37B 2019-10-13T05:21:18.884 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.884 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.iGnLZRe4ra 2019-10-13T05:21:18.891 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.892 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:18.961 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.7ZwpXp0Pl1 2019-10-13T05:21:18.962 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:18.962 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.7ZwpXp0Pl1 2019-10-13T05:21:19.044 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.044 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.7ZwpXp0Pl1 2019-10-13T05:21:19.197 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.7ZwpXp0Pl1 is 37B 2019-10-13T05:21:19.208 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.208 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.7ZwpXp0Pl1 2019-10-13T05:21:19.214 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.215 INFO:teuthology.orchestra.run.smithi129:> mktemp 2019-10-13T05:21:19.284 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.73cjcPZMXc 2019-10-13T05:21:19.285 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.285 INFO:teuthology.orchestra.run.smithi129:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.73cjcPZMXc 2019-10-13T05:21:19.365 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.365 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.73cjcPZMXc 2019-10-13T05:21:19.522 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.73cjcPZMXc is 37B 2019-10-13T05:21:19.536 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:19.536 INFO:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.73cjcPZMXc 2019-10-13T05:21:19.544 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.544 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:19.551 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.IAGmqyybkC 2019-10-13T05:21:19.551 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.551 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.IAGmqyybkC 2019-10-13T05:21:19.631 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.632 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.IAGmqyybkC 2019-10-13T05:21:19.787 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.IAGmqyybkC is 37B 2019-10-13T05:21:19.840 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.840 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.IAGmqyybkC 2019-10-13T05:21:19.848 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.848 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:19.920 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.GOpGOfD1h5 2019-10-13T05:21:19.920 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:19.923 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.GOpGOfD1h5 2019-10-13T05:21:20.004 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.005 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.GOpGOfD1h5 2019-10-13T05:21:20.158 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.GOpGOfD1h5 is 37B 2019-10-13T05:21:20.169 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.169 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.GOpGOfD1h5 2019-10-13T05:21:20.177 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.177 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:20.248 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.woXzOvBk43 2019-10-13T05:21:20.248 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.248 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.woXzOvBk43 2019-10-13T05:21:20.329 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.329 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.woXzOvBk43 2019-10-13T05:21:20.491 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.woXzOvBk43 is 37B 2019-10-13T05:21:20.503 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.503 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.woXzOvBk43 2019-10-13T05:21:20.510 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.510 INFO:teuthology.orchestra.run.smithi071:> mktemp 2019-10-13T05:21:20.580 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.rGbs88BUnb 2019-10-13T05:21:20.580 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.580 INFO:teuthology.orchestra.run.smithi071:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.rGbs88BUnb 2019-10-13T05:21:20.660 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.660 INFO:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.rGbs88BUnb 2019-10-13T05:21:20.815 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.rGbs88BUnb is 37B 2019-10-13T05:21:20.826 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.826 INFO:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.rGbs88BUnb 2019-10-13T05:21:20.834 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:20.834 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 76f2fbe2-33c9-4e71-b7de-e1be85d1477f 0 2019-10-13T05:21:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 Processor -- start 2019-10-13T05:21:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 -- start start 2019-10-13T05:21:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f61b00a1c80 0x7f61b00a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b0144a10 0x7f61b0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b00a7a30 con 0x7f61b00a7520 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b00a96b0 con 0x7f61b00a1c80 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b51ef700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00a9950 con 0x7f61b0144a10 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f61b00a1c80 0x7f61b00a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61aed9d700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f61b00a7520 0x7f61b00a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43284/0 (socket says 172.21.15.71:43284) 2019-10-13T05:21:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61b49ee700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b0144a10 0x7f61b0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61aed9d700 1 -- 172.21.15.71:0/3128205298 learned_addr learned my addr 172.21.15.71:0/3128205298 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 -- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b0144a10 msgr2=0x7f61b0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b0144a10 0x7f61b0144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 -- 172.21.15.71:0/3128205298 >> v1:172.21.15.71:6789/0 conn(0x7f61b00a7520 legacy=0x7f61b00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 -- 172.21.15.71:0/3128205298 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b00a1940 con 0x7f61b00a1c80 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61ae59c700 1 -- 172.21.15.71:0/3128205298 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829805771 0 0) 0x7f61b00a7a30 con 0x7f61b00a7520 2019-10-13T05:21:21.018 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> v2:172.21.15.71:3300/0 conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> v2:172.21.15.71:3300/0 conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.006+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).ready entity=mon.0 client_cookie=51f19828d0b45181 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61ae59c700 1 -- 172.21.15.71:0/3128205298 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a0008ba0 con 0x7f61b00a1c80 2019-10-13T05:21:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.020 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61af59e700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61ae59c700 1 -- 172.21.15.71:0/3128205298 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a00087c0 con 0x7f61b00a1c80 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61ae59c700 1 -- 172.21.15.71:0/3128205298 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a0011740 con 0x7f61b00a1c80 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 -- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 msgr2=0x7f61b00a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f61b00a7ea0 tx=0x7f61a8007a90).stop 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 -- 172.21.15.71:0/3128205298 shutdown_connections 2019-10-13T05:21:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b0144a10 0x7f61b0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.022 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 --2- 172.21.15.71:0/3128205298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61b00a1c80 0x7f61b00a13f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.022 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 -- 172.21.15.71:0/3128205298 shutdown_connections 2019-10-13T05:21:21.022 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 -- 172.21.15.71:0/3128205298 wait complete. 2019-10-13T05:21:21.022 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.010+0000 7f61b51ef700 1 -- 172.21.15.71:0/3128205298 >> 172.21.15.71:0/3128205298 conn(0x7f61b005e870 msgr2=0x7f61b009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 Processor -- start 2019-10-13T05:21:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- start start 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f61b00a59a0 0x7f61b0143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b01193b0 con 0x7f61b0143550 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b0121b10 con 0x7f61b00a59a0 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b0121db0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.71:54134/0 (socket says 172.21.15.71:54134) 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 -- 172.21.15.71:0/139755333 learned_addr learned my addr 172.21.15.71:0/139755333 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b49ee700 1 --2- 172.21.15.71:0/139755333 >> v2:172.21.15.129:3300/0 conn(0x7f61b00a59a0 0x7f61b0143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 -- 172.21.15.71:0/139755333 >> v2:172.21.15.129:3300/0 conn(0x7f61b00a59a0 msgr2=0x7f61b0143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> v2:172.21.15.129:3300/0 conn(0x7f61b00a59a0 0x7f61b0143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 -- 172.21.15.71:0/139755333 >> v1:172.21.15.129:6789/0 conn(0x7f61b0143550 legacy=0x7f61b00a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 -- 172.21.15.71:0/139755333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b011a5f0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a0007350 con 0x7f61b00aa1d0 2019-10-13T05:21:21.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a0008da0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a0005910 con 0x7f61b00aa1d0 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b01195e0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b0119b10 con 0x7f61b00aa1d0 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 2) v1 ==== 16220+0+0 (secure 0 0 0) 0x7f61a0005c20 con 0x7f61b00aa1d0 2019-10-13T05:21:21.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.014+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f61a001eac0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.022+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b00555c0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.022+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.026+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7f61a0007730 con 0x7f61b00aa1d0 2019-10-13T05:21:21.137 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.134+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76f2fbe2-33c9-4e71-b7de-e1be85d1477f", "id": 0} v 0) v1 -- 0x7f61b0122700 con 0x7f61b00aa1d0 2019-10-13T05:21:21.149 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.142+0000 7f61af59e700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.149 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.142+0000 7f6197fff700 1 -- 172.21.15.71:0/139755333 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76f2fbe2-33c9-4e71-b7de-e1be85d1477f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f61a000abf0 con 0x7f61b00aa1d0 2019-10-13T05:21:21.153 INFO:teuthology.orchestra.run.smithi071.stdout:0 2019-10-13T05:21:21.156 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 msgr2=0x7f61b0077560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.156 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f61a0008360 tx=0x7f61a8007e40).stop 2019-10-13T05:21:21.156 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 shutdown_connections 2019-10-13T05:21:21.156 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 --2- 172.21.15.71:0/139755333 >> v2:172.21.15.129:3300/0 conn(0x7f61b00a59a0 0x7f61b0143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.156 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 --2- 172.21.15.71:0/139755333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f61b00aa1d0 0x7f61b0077560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.157 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 shutdown_connections 2019-10-13T05:21:21.157 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 wait complete. 2019-10-13T05:21:21.157 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.150+0000 7f61b51ef700 1 -- 172.21.15.71:0/139755333 >> 172.21.15.71:0/139755333 conn(0x7f61b007e580 msgr2=0x7f61b00a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.166 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:21.166 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new b7fe1be3-7c39-4273-b0a8-19549cb83be1 1 2019-10-13T05:21:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 Processor -- start 2019-10-13T05:21:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 -- start start 2019-10-13T05:21:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff3800a1c70 0x7ff3800a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 0x7ff380144e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800a7a20 con 0x7ff3800a7510 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800a96a0 con 0x7ff3800a1c70 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff387fa6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800a9940 con 0x7ff380144a00 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff3800a1c70 0x7ff3800a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff3800a1c70 0x7ff3800a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40146/0 (socket says 172.21.15.71:40146) 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386d44700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 0x7ff380144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 -- 172.21.15.71:0/4283290640 learned_addr learned my addr 172.21.15.71:0/4283290640 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 -- 172.21.15.71:0/4283290640 >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 msgr2=0x7ff380144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 0x7ff380144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 -- 172.21.15.71:0/4283290640 >> v1:172.21.15.129:6789/0 conn(0x7ff3800a7510 legacy=0x7ff3800a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 -- 172.21.15.71:0/4283290640 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800a1930 con 0x7ff3800a1c70 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386d44700 1 --2- 172.21.15.71:0/4283290640 >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 0x7ff380144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> v2:172.21.15.71:3300/0 conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff3800a9860 tx=0x7ff378007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> v2:172.21.15.71:3300/0 conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff3800a9860 tx=0x7ff378007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.258+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7ff3800a9860 tx=0x7ff378007e40).ready entity=mon.0 client_cookie=335df319ec8f4b81 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7ff3800a9860 tx=0x7ff378007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff385541700 1 -- 172.21.15.71:0/4283290640 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c008c80 con 0x7ff3800a1c70 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7ff3800a9860 tx=0x7ff378007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff385541700 1 -- 172.21.15.71:0/4283290640 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c0088a0 con 0x7ff3800a1c70 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff386543700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7ff3800a9860 tx=0x7ff378007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 -- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 msgr2=0x7ff3800a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff385541700 1 -- 172.21.15.71:0/4283290640 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c011750 con 0x7ff3800a1c70 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7ff3800a9860 tx=0x7ff378007e40).stop 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 -- 172.21.15.71:0/4283290640 shutdown_connections 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 --2- 172.21.15.71:0/4283290640 >> v2:172.21.15.129:3300/0 conn(0x7ff380144a00 0x7ff380144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 --2- 172.21.15.71:0/4283290640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3800a1c70 0x7ff3800a13e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 -- 172.21.15.71:0/4283290640 shutdown_connections 2019-10-13T05:21:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 -- 172.21.15.71:0/4283290640 wait complete. 2019-10-13T05:21:21.279 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.266+0000 7ff387fa6700 1 -- 172.21.15.71:0/4283290640 >> 172.21.15.71:0/4283290640 conn(0x7ff38005e860 msgr2=0x7ff38009c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.279 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 Processor -- start 2019-10-13T05:21:21.279 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- start start 2019-10-13T05:21:21.283 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 0x7ff3800a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff38012ca70 0x7ff38012eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800a21c0 con 0x7ff3800a5f50 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38009d060 con 0x7ff38012ca70 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff380078fa0 con 0x7ff380064d00 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff38012ca70 0x7ff38012eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff38012ca70 0x7ff38012eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40162/0 (socket says 172.21.15.71:40162) 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 -- 172.21.15.71:0/2823283201 learned_addr learned my addr 172.21.15.71:0/2823283201 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff385d42700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 0x7ff3800a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 -- 172.21.15.71:0/2823283201 >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 msgr2=0x7ff3800a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 0x7ff3800a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 -- 172.21.15.71:0/2823283201 >> v1:172.21.15.129:6789/0 conn(0x7ff3800a5f50 legacy=0x7ff380123470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 -- 172.21.15.71:0/2823283201 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff380123c70 con 0x7ff38012ca70 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff385d42700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 0x7ff3800a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.71:3300/0 conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.71:3300/0 conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.286 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3780058b0 con 0x7ff38012ca70 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff378011ab0 con 0x7ff38012ca70 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3800a4540 con 0x7ff38012ca70 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37800c7b0 con 0x7ff38012ca70 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38012f3f0 con 0x7ff38012ca70 2019-10-13T05:21:21.287 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff37800cac0 con 0x7ff38012ca70 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 0x7ff368029e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.270+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7ff378035340 con 0x7ff38012ca70 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.274+0000 7ff386543700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 0x7ff368029e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.274+0000 7ff386543700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 0x7ff368029e20 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.288 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.274+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3800555b0 con 0x7ff38012ca70 2019-10-13T05:21:21.289 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.278+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.289 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.278+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff37800cdb0 con 0x7ff38012ca70 2019-10-13T05:21:21.432 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.426+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b7fe1be3-7c39-4273-b0a8-19549cb83be1", "id": 1} v 0) v1 -- 0x7ff380067b80 con 0x7ff38012ca70 2019-10-13T05:21:21.439 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.434+0000 7ff386d44700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.439 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.434+0000 7ff372ffd700 1 -- 172.21.15.71:0/2823283201 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7fe1be3-7c39-4273-b0a8-19549cb83be1", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff37800eac0 con 0x7ff38012ca70 2019-10-13T05:21:21.439 INFO:teuthology.orchestra.run.smithi071.stdout:1 2019-10-13T05:21:21.444 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 msgr2=0x7ff368029e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.444 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 0x7ff368029e20 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.444 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 msgr2=0x7ff38012eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.444 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ff3800a23f0 tx=0x7ff374001250).stop 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 shutdown_connections 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff368027920 0x7ff368029e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 --2- 172.21.15.71:0/2823283201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff38012ca70 0x7ff38012eea0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 --2- 172.21.15.71:0/2823283201 >> v2:172.21.15.129:3300/0 conn(0x7ff380064d00 0x7ff3800a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 shutdown_connections 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 wait complete. 2019-10-13T05:21:21.445 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.438+0000 7ff387fa6700 1 -- 172.21.15.71:0/2823283201 >> 172.21.15.71:0/2823283201 conn(0x7ff3800a2d80 msgr2=0x7ff38007df10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.456 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:21.456 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 6528b803-919d-4663-9be9-f2dcebe08ddc 2 2019-10-13T05:21:21.556 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 Processor -- start 2019-10-13T05:21:21.557 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 -- start start 2019-10-13T05:21:21.557 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f02d00a1c80 0x7f02d00a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.557 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d00a7520 0x7f02d00a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.557 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f02d0144a10 0x7f02d0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.558 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d00a96b0 con 0x7f02d00a1c80 2019-10-13T05:21:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d00a9950 con 0x7f02d0144a10 2019-10-13T05:21:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02d4fe2700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00a9bf0 con 0x7f02d00a7520 2019-10-13T05:21:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f02d00a1c80 0x7f02d00a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f02d00a1c80 0x7f02d00a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40166/0 (socket says 172.21.15.71:40166) 2019-10-13T05:21:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 -- 172.21.15.71:0/686366268 learned_addr learned my addr 172.21.15.71:0/686366268 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.560 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ce59c700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d00a7520 0x7f02d00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.560 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02cf59e700 1 --2- 172.21.15.71:0/686366268 >> v2:172.21.15.129:3300/0 conn(0x7f02d0144a10 0x7f02d0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.560 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 -- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d00a7520 msgr2=0x7f02d00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:21.560 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d00a7520 0x7f02d00a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 -- 172.21.15.71:0/686366268 >> v2:172.21.15.129:3300/0 conn(0x7f02d0144a10 msgr2=0x7f02d0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> v2:172.21.15.129:3300/0 conn(0x7f02d0144a10 0x7f02d0144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 -- 172.21.15.71:0/686366268 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00a7a30 con 0x7f02d00a1c80 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> v2:172.21.15.71:3300/0 conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> v2:172.21.15.71:3300/0 conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.546+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).ready entity=mon.0 client_cookie=e970325564340ecd server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.562 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.562 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02cdd9b700 1 -- 172.21.15.71:0/686366268 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02c0008c80 con 0x7f02d00a1c80 2019-10-13T05:21:21.563 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02ced9d700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02cdd9b700 1 -- 172.21.15.71:0/686366268 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c00088a0 con 0x7f02d00a1c80 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02cdd9b700 1 -- 172.21.15.71:0/686366268 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02c0011710 con 0x7f02d00a1c80 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 -- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 msgr2=0x7f02d00a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f02d00a7ea0 tx=0x7f02c8003370).stop 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 -- 172.21.15.71:0/686366268 shutdown_connections 2019-10-13T05:21:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/686366268 >> v2:172.21.15.129:3300/0 conn(0x7f02d0144a10 0x7f02d0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.565 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02d00a1c80 0x7f02d00a13f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.565 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/686366268 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d00a7520 0x7f02d00a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.565 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 -- 172.21.15.71:0/686366268 shutdown_connections 2019-10-13T05:21:21.565 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 -- 172.21.15.71:0/686366268 wait complete. 2019-10-13T05:21:21.565 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.550+0000 7f02d4fe2700 1 -- 172.21.15.71:0/686366268 >> 172.21.15.71:0/686366268 conn(0x7f02d005e870 msgr2=0x7f02d009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.569 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 Processor -- start 2019-10-13T05:21:21.569 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- start start 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00a51c0 con 0x7f02d0121f70 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d0126e20 con 0x7f02d00a6ac0 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00a6ee0 con 0x7f02d01266a0 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02ced9d700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f02d0121f70 0x7f02d011c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43320/0 (socket says 172.21.15.71:43320) 2019-10-13T05:21:21.570 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02ced9d700 1 -- 172.21.15.71:0/3263937428 learned_addr learned my addr 172.21.15.71:0/3263937428 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492248865 0 0) 0x7f02c0008480 con 0x7f02d0121f70 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b40041b0 con 0x7f02d0121f70 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834006835 0 0) 0x7f02d0126e20 con 0x7f02d00a6ac0 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 -- 172.21.15.71:0/3263937428 >> v1:172.21.15.129:6789/0 conn(0x7f02d00a6ac0 legacy=0x7f02d011cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.571 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 -- 172.21.15.71:0/3263937428 >> v1:172.21.15.71:6789/0 conn(0x7f02d0121f70 legacy=0x7f02d011c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 -- 172.21.15.71:0/3263937428 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0126ac0 con 0x7f02d01266a0 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.572 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.573 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02c8012c40 con 0x7f02d01266a0 2019-10-13T05:21:21.573 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.573 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8012860 con 0x7f02d01266a0 2019-10-13T05:21:21.573 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d00a42b0 con 0x7f02d01266a0 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02c8009580 con 0x7f02d01266a0 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.554+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d009c620 con 0x7f02d01266a0 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f02c8009890 con 0x7f02d01266a0 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02bf7fe700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 0x7f02b402b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.574 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f02c8034c40 con 0x7f02d01266a0 2019-10-13T05:21:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02ced9d700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 0x7f02b402b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.558+0000 7f02ced9d700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 0x7f02b402b3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.562+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d00555c0 con 0x7f02d01266a0 2019-10-13T05:21:21.576 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.566+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.576 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.566+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f02c8005b80 con 0x7f02d01266a0 2019-10-13T05:21:21.721 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.714+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6528b803-919d-4663-9be9-f2dcebe08ddc", "id": 2} v 0) v1 -- 0x7f02d009cc30 con 0x7f02d01266a0 2019-10-13T05:21:21.731 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.726+0000 7f02cf59e700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.731 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.726+0000 7f02bf7fe700 1 -- 172.21.15.71:0/3263937428 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6528b803-919d-4663-9be9-f2dcebe08ddc", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f02c8005da0 con 0x7f02d01266a0 2019-10-13T05:21:21.736 INFO:teuthology.orchestra.run.smithi071.stdout:2 2019-10-13T05:21:21.739 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 msgr2=0x7f02b402b3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.739 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 0x7f02b402b3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 msgr2=0x7f02d00a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f02c80032b0 tx=0x7f02c8010db0).stop 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 shutdown_connections 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f02b4028ea0 0x7f02b402b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 --2- 172.21.15.71:0/3263937428 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f02d01266a0 0x7f02d00a4c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 shutdown_connections 2019-10-13T05:21:21.741 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 wait complete. 2019-10-13T05:21:21.741 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.734+0000 7f02d4fe2700 1 -- 172.21.15.71:0/3263937428 >> 172.21.15.71:0/3263937428 conn(0x7f02d00a2ce0 msgr2=0x7f02d00a7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.750 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:21.750 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 25d2f3bc-a14f-488e-8103-ee1794e0249b 3 2019-10-13T05:21:21.845 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 Processor -- start 2019-10-13T05:21:21.845 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 -- start start 2019-10-13T05:21:21.846 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf540a1c80 0x7fcf540a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.847 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:21.847 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf540a7a30 con 0x7fcf540a7520 2019-10-13T05:21:21.847 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540a96b0 con 0x7fcf540a1c80 2019-10-13T05:21:21.848 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf59283700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540a9950 con 0x7fcf54144a10 2019-10-13T05:21:21.848 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.848 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.71:54178/0 (socket says 172.21.15.71:54178) 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 -- 172.21.15.71:0/1327761488 learned_addr learned my addr 172.21.15.71:0/1327761488 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf537fe700 1 --2- 172.21.15.71:0/1327761488 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a1c80 0x7fcf540a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 -- 172.21.15.71:0/1327761488 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a1c80 msgr2=0x7fcf540a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a1c80 0x7fcf540a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 -- 172.21.15.71:0/1327761488 >> v1:172.21.15.129:6789/0 conn(0x7fcf540a7520 legacy=0x7fcf540a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 -- 172.21.15.71:0/1327761488 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf540a1940 con 0x7fcf54144a10 2019-10-13T05:21:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.834+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).ready entity=mon.2 client_cookie=1e0d385173a7706 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf527fc700 1 -- 172.21.15.71:0/1327761488 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf4c010b90 con 0x7fcf54144a10 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf53fff700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.850 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf527fc700 1 -- 172.21.15.71:0/1327761488 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4c0107b0 con 0x7fcf54144a10 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf527fc700 1 -- 172.21.15.71:0/1327761488 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf4c0112b0 con 0x7fcf54144a10 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 msgr2=0x7fcf54144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7fcf4c008b00 tx=0x7fcf3c007710).stop 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- 172.21.15.71:0/1327761488 shutdown_connections 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 --2- 172.21.15.71:0/1327761488 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcf54144a10 0x7fcf54144e30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 --2- 172.21.15.71:0/1327761488 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a1c80 0x7fcf540a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- 172.21.15.71:0/1327761488 shutdown_connections 2019-10-13T05:21:21.852 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- 172.21.15.71:0/1327761488 wait complete. 2019-10-13T05:21:21.852 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- 172.21.15.71:0/1327761488 >> 172.21.15.71:0/1327761488 conn(0x7fcf5405e870 msgr2=0x7fcf5409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:21.852 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 Processor -- start 2019-10-13T05:21:21.852 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.838+0000 7fcf59283700 1 -- start start 2019-10-13T05:21:21.855 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf540a05b0 0x7fcf540a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.856 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf5409f6b0 0x7fcf54134630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.856 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf540a4940 con 0x7fcf54063850 2019-10-13T05:21:21.856 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540a4710 con 0x7fcf5409f6b0 2019-10-13T05:21:21.856 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540a4c80 con 0x7fcf540a05b0 2019-10-13T05:21:21.856 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf5409f6b0 0x7fcf54134630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf5409f6b0 0x7fcf54134630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40190/0 (socket says 172.21.15.71:40190) 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 -- 172.21.15.71:0/1394059389 learned_addr learned my addr 172.21.15.71:0/1394059389 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf537fe700 1 --2- 172.21.15.71:0/1394059389 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a05b0 0x7fcf540a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 -- 172.21.15.71:0/1394059389 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a05b0 msgr2=0x7fcf540a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a05b0 0x7fcf540a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:21.857 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 -- 172.21.15.71:0/1394059389 >> v1:172.21.15.129:6789/0 conn(0x7fcf54063850 legacy=0x7fcf54063c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 -- 172.21.15.71:0/1394059389 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf54134b80 con 0x7fcf5409f6b0 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> v2:172.21.15.71:3300/0 conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> v2:172.21.15.71:3300/0 conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.858 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf4400c9a0 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4400c5c0 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf54064470 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf44010d70 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5411ba30 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fcf44007420 con 0x7fcf5409f6b0 2019-10-13T05:21:21.859 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.860 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 0x7fcf2c029ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:21.860 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7fcf44034b60 con 0x7fcf5409f6b0 2019-10-13T05:21:21.860 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf537fe700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 0x7fcf2c029ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:21.860 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.842+0000 7fcf537fe700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 0x7fcf2c029ea0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:21.860 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.850+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf540555c0 con 0x7fcf5409f6b0 2019-10-13T05:21:21.864 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.858+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:21.864 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:21.858+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fcf44025730 con 0x7fcf5409f6b0 2019-10-13T05:21:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.014+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "25d2f3bc-a14f-488e-8103-ee1794e0249b", "id": 3} v 0) v1 -- 0x7fcf5409d070 con 0x7fcf5409f6b0 2019-10-13T05:21:22.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.022+0000 7fcf52ffd700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.022+0000 7fcf4bfff700 1 -- 172.21.15.71:0/1394059389 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25d2f3bc-a14f-488e-8103-ee1794e0249b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf44025ef0 con 0x7fcf5409f6b0 2019-10-13T05:21:22.035 INFO:teuthology.orchestra.run.smithi071.stdout:3 2019-10-13T05:21:22.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 msgr2=0x7fcf2c029ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 0x7fcf2c029ea0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 msgr2=0x7fcf54134630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fcf540a48d0 tx=0x7fcf3c007930).stop 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 shutdown_connections 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 --2- 172.21.15.71:0/1394059389 >> v2:172.21.15.129:3300/0 conn(0x7fcf540a05b0 0x7fcf540a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcf2c0279a0 0x7fcf2c029ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 --2- 172.21.15.71:0/1394059389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf5409f6b0 0x7fcf54134630 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 shutdown_connections 2019-10-13T05:21:22.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 wait complete. 2019-10-13T05:21:22.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.034+0000 7fcf59283700 1 -- 172.21.15.71:0/1394059389 >> 172.21.15.71:0/1394059389 conn(0x7fcf540a6ce0 msgr2=0x7fcf5409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.049 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:22.049 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb 4 2019-10-13T05:21:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.142+0000 7f87af603700 1 Processor -- start 2019-10-13T05:21:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.142+0000 7f87af603700 1 -- start start 2019-10-13T05:21:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f87a8144a10 0x7f87a8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a80a7a30 con 0x7f87a80a1c80 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a8150ba0 con 0x7f87a80a7520 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a80a96b0 con 0x7f87a8144a10 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87adba0700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f87a80a1c80 0x7f87a80a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43342/0 (socket says 172.21.15.71:43342) 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87adba0700 1 -- 172.21.15.71:0/2552564634 learned_addr learned my addr 172.21.15.71:0/2552564634 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> v2:172.21.15.129:3300/0 conn(0x7f87a8144a10 0x7f87a8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503064929 0 0) 0x7f87a80a7a30 con 0x7f87a80a1c80 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87940030a0 con 0x7f87a80a1c80 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3859933544 0 0) 0x7f87940030a0 con 0x7f87a80a1c80 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2552564634 >> v1:172.21.15.129:6789/0 conn(0x7f87a80a7520 legacy=0x7f87a80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2552564634 >> v1:172.21.15.71:6789/0 conn(0x7f87a80a1c80 legacy=0x7f87a80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2552564634 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8151de0 con 0x7f87a8144a10 2019-10-13T05:21:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> v2:172.21.15.129:3300/0 conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f87a4001950 tx=0x7f87a4001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.163 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> v2:172.21.15.129:3300/0 conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f87a4001950 tx=0x7f87a4001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.163 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f87a4001950 tx=0x7f87a4001990).ready entity=mon.1 client_cookie=9d5d7d5bd42ebd8e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.163 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f87a4001950 tx=0x7f87a4001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.163 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a4001750 con 0x7f87a8144a10 2019-10-13T05:21:22.163 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f87a4001950 tx=0x7f87a4001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f87a4001950 tx=0x7f87a4001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a4001560 con 0x7f87a8144a10 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87acb9e700 1 -- 172.21.15.71:0/2552564634 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a0010a50 con 0x7f87a8144a10 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 -- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 msgr2=0x7f87a8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.146+0000 7f87af603700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f87a4001950 tx=0x7f87a4001990).stop 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- 172.21.15.71:0/2552564634 shutdown_connections 2019-10-13T05:21:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 --2- 172.21.15.71:0/2552564634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87a8144a10 0x7f87a8144e30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- 172.21.15.71:0/2552564634 shutdown_connections 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- 172.21.15.71:0/2552564634 wait complete. 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- 172.21.15.71:0/2552564634 >> 172.21.15.71:0/2552564634 conn(0x7f87a805e870 msgr2=0x7f87a809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 Processor -- start 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- start start 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87a812cfe0 0x7f87a812f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.165 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a8124500 con 0x7f87a8060830 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a8060c50 con 0x7f87a81235d0 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87af603700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a80a0780 con 0x7f87a812cfe0 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87a812cfe0 0x7f87a812f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87a812cfe0 0x7f87a812f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40202/0 (socket says 172.21.15.71:40202) 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2927314924 learned_addr learned my addr 172.21.15.71:0/2927314924 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:22.166 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260795861 0 0) 0x7f87a8060c50 con 0x7f87a81235d0 2019-10-13T05:21:22.167 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87900030a0 con 0x7f87a81235d0 2019-10-13T05:21:22.167 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2927314924 >> v1:172.21.15.129:6789/0 conn(0x7f87a81235d0 legacy=0x7f87a805f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.167 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2927314924 >> v1:172.21.15.71:6789/0 conn(0x7f87a8060830 legacy=0x7f87a8123d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.167 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 -- 172.21.15.71:0/2927314924 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80624a0 con 0x7f87a812cfe0 2019-10-13T05:21:22.167 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> v2:172.21.15.71:3300/0 conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.150+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> v2:172.21.15.71:3300/0 conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a0005cb0 con 0x7f87a812cfe0 2019-10-13T05:21:22.168 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0008cf0 con 0x7f87a812cfe0 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a812f960 con 0x7f87a812cfe0 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a0007540 con 0x7f87a812cfe0 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a812fe30 con 0x7f87a812cfe0 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.169 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f87a000fae0 con 0x7f87a812cfe0 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 0x7f879002a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f87a0034ad0 con 0x7f87a812cfe0 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87adba0700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 0x7f879002a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.154+0000 7f87adba0700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 0x7f879002a290 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.170 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.158+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a80555c0 con 0x7f87a812cfe0 2019-10-13T05:21:22.172 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.166+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.173 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.166+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f87a001d340 con 0x7f87a812cfe0 2019-10-13T05:21:22.335 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.330+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb", "id": 4} v 0) v1 -- 0x7f87a809d070 con 0x7f87a812cfe0 2019-10-13T05:21:22.341 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.334+0000 7f87ae3a1700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.341 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.334+0000 7f879e7fc700 1 -- 172.21.15.71:0/2927314924 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f87a001d5a0 con 0x7f87a812cfe0 2019-10-13T05:21:22.343 INFO:teuthology.orchestra.run.smithi071.stdout:4 2019-10-13T05:21:22.345 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 msgr2=0x7f879002a290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.345 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 0x7f879002a290 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 msgr2=0x7f87a812f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f87a00086e0 tx=0x7f87a0010b40).stop 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 shutdown_connections 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8790027d90 0x7f879002a290 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 --2- 172.21.15.71:0/2927314924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87a812cfe0 0x7f87a812f410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.346 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 shutdown_connections 2019-10-13T05:21:22.347 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 wait complete. 2019-10-13T05:21:22.347 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.338+0000 7f87af603700 1 -- 172.21.15.71:0/2927314924 >> 172.21.15.71:0/2927314924 conn(0x7f87a805e770 msgr2=0x7f87a80a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.355 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:22.355 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new d15bcd3d-1a88-44e6-b0cd-7736730331c6 5 2019-10-13T05:21:22.452 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 Processor -- start 2019-10-13T05:21:22.452 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 -- start start 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe9140a7520 0x7fe9140a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9140a7a30 con 0x7fe9140a1c80 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9140a96b0 con 0x7fe9140a7520 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe919899700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140a9950 con 0x7fe914144a10 2019-10-13T05:21:22.453 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe9137fe700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe9140a1c80 0x7fe9140a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43356/0 (socket says 172.21.15.71:43356) 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe9137fe700 1 -- 172.21.15.71:0/3819643859 learned_addr learned my addr 172.21.15.71:0/3819643859 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe912ffd700 1 --2- 172.21.15.71:0/3819643859 >> v2:172.21.15.129:3300/0 conn(0x7fe9140a7520 0x7fe9140a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.442+0000 7fe9127fc700 1 -- 172.21.15.71:0/3819643859 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782993056 0 0) 0x7fe9140a7a30 con 0x7fe9140a1c80 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe9127fc700 1 -- 172.21.15.71:0/3819643859 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9000030a0 con 0x7fe9140a1c80 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 -- 172.21.15.71:0/3819643859 >> v2:172.21.15.129:3300/0 conn(0x7fe9140a7520 msgr2=0x7fe9140a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:22.454 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> v2:172.21.15.129:3300/0 conn(0x7fe9140a7520 0x7fe9140a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 -- 172.21.15.71:0/3819643859 >> v1:172.21.15.71:6789/0 conn(0x7fe9140a1c80 legacy=0x7fe9140a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 -- 172.21.15.71:0/3819643859 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140aa3e0 con 0x7fe914144a10 2019-10-13T05:21:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe90c008780 tx=0x7fe8fc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe90c008780 tx=0x7fe8fc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7fe90c008780 tx=0x7fe8fc007710).ready entity=mon.2 client_cookie=4be427af30435a07 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fe90c008780 tx=0x7fe8fc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe9127fc700 1 -- 172.21.15.71:0/3819643859 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe90c008ba0 con 0x7fe914144a10 2019-10-13T05:21:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fe90c008780 tx=0x7fe8fc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe913fff700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fe90c008780 tx=0x7fe8fc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe9127fc700 1 -- 172.21.15.71:0/3819643859 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c0087c0 con 0x7fe914144a10 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe9127fc700 1 -- 172.21.15.71:0/3819643859 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe90c011650 con 0x7fe914144a10 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 -- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 msgr2=0x7fe914144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7fe90c008780 tx=0x7fe8fc007710).stop 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 -- 172.21.15.71:0/3819643859 shutdown_connections 2019-10-13T05:21:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 --2- 172.21.15.71:0/3819643859 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe914144a10 0x7fe914144e30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.458 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 --2- 172.21.15.71:0/3819643859 >> v2:172.21.15.129:3300/0 conn(0x7fe9140a7520 0x7fe9140a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:22.458 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 -- 172.21.15.71:0/3819643859 shutdown_connections 2019-10-13T05:21:22.458 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 -- 172.21.15.71:0/3819643859 wait complete. 2019-10-13T05:21:22.458 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.446+0000 7fe919899700 1 -- 172.21.15.71:0/3819643859 >> 172.21.15.71:0/3819643859 conn(0x7fe91405e870 msgr2=0x7fe91409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.461 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 Processor -- start 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- start start 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91411ab70 con 0x7fe914141d30 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9140644b0 con 0x7fe914142150 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91411c7c0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe912ffd700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe914141d30 0x7fe914063cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43362/0 (socket says 172.21.15.71:43362) 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe912ffd700 1 -- 172.21.15.71:0/989426925 learned_addr learned my addr 172.21.15.71:0/989426925 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242336590 0 0) 0x7fe91411ab70 con 0x7fe914141d30 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f80030a0 con 0x7fe914141d30 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 -- 172.21.15.71:0/989426925 >> v1:172.21.15.129:6789/0 conn(0x7fe914142150 legacy=0x7fe914077190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 -- 172.21.15.71:0/989426925 >> v1:172.21.15.71:6789/0 conn(0x7fe914141d30 legacy=0x7fe914063cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 -- 172.21.15.71:0/989426925 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91407a9c0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9040122c0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe904005bc0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9140789a0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9040128a0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe914078cd0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.450+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fe904013060 con 0x7fe9140a1fc0 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.454+0000 7fe90bfff700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 0x7fe8f802a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.454+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7fe904034d70 con 0x7fe9140a1fc0 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.454+0000 7fe912ffd700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 0x7fe8f802a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.454+0000 7fe912ffd700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 0x7fe8f802a2d0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.458+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9140555c0 con 0x7fe9140a1fc0 2019-10-13T05:21:22.468 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.462+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.469 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.462+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fe904012b70 con 0x7fe9140a1fc0 2019-10-13T05:21:22.619 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.614+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d15bcd3d-1a88-44e6-b0cd-7736730331c6", "id": 5} v 0) v1 -- 0x7fe914067b90 con 0x7fe9140a1fc0 2019-10-13T05:21:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.622+0000 7fe9137fe700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.622+0000 7fe90bfff700 1 -- 172.21.15.71:0/989426925 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d15bcd3d-1a88-44e6-b0cd-7736730331c6", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe90400a500 con 0x7fe9140a1fc0 2019-10-13T05:21:22.634 INFO:teuthology.orchestra.run.smithi071.stdout:5 2019-10-13T05:21:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.630+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 msgr2=0x7fe8f802a2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.630+0000 7fe919899700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 0x7fe8f802a2d0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.630+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 msgr2=0x7fe91411a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.630+0000 7fe919899700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7fe904004710 tx=0x7fe904010c00).stop 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 shutdown_connections 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fe8f8027dd0 0x7fe8f802a2d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 --2- 172.21.15.71:0/989426925 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fe9140a1fc0 0x7fe91411a620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 shutdown_connections 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 wait complete. 2019-10-13T05:21:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.634+0000 7fe919899700 1 -- 172.21.15.71:0/989426925 >> 172.21.15.71:0/989426925 conn(0x7fe914144120 msgr2=0x7fe914142760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.650 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:22.650 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 460e48b8-6810-40b9-b018-121ed3eea51d 6 2019-10-13T05:21:22.742 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.730+0000 7f1e33b41700 1 Processor -- start 2019-10-13T05:21:22.742 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.730+0000 7f1e33b41700 1 -- start start 2019-10-13T05:21:22.742 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1e2c144a10 0x7f1e2c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:22.742 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0a7a30 con 0x7f1e2c0a1c80 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c150ba0 con 0x7f1e2c0a7520 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c0a96b0 con 0x7f1e2c144a10 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e320de700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1e2c0a1c80 0x7f1e2c0a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43370/0 (socket says 172.21.15.71:43370) 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e320de700 1 -- 172.21.15.71:0/1479991386 learned_addr learned my addr 172.21.15.71:0/1479991386 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1e2c144a10 0x7f1e2c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.743 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 -- 172.21.15.71:0/1479991386 >> v1:172.21.15.129:6789/0 conn(0x7f1e2c0a7520 legacy=0x7f1e2c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 -- 172.21.15.71:0/1479991386 >> v1:172.21.15.71:6789/0 conn(0x7f1e2c0a1c80 legacy=0x7f1e2c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e310dc700 1 -- 172.21.15.71:0/1479991386 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989031683 0 0) 0x7f1e2c150ba0 con 0x7f1e2c0a7520 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 -- 172.21.15.71:0/1479991386 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c151de0 con 0x7f1e2c144a10 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> v2:172.21.15.71:3300/0 conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1e24008b00 tx=0x7f1e28001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> v2:172.21.15.71:3300/0 conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1e24008b00 tx=0x7f1e28001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.744 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f1e24008b00 tx=0x7f1e28001150).ready entity=mon.0 client_cookie=aff2b3d98e330b0a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.746 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f1e24008b00 tx=0x7f1e28001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.746 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f1e24008b00 tx=0x7f1e28001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e310dc700 1 -- 172.21.15.71:0/1479991386 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e24010860 con 0x7f1e2c144a10 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e328df700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f1e24008b00 tx=0x7f1e28001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e310dc700 1 -- 172.21.15.71:0/1479991386 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24008b40 con 0x7f1e2c144a10 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e310dc700 1 -- 172.21.15.71:0/1479991386 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e240110c0 con 0x7f1e2c144a10 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 msgr2=0x7f1e2c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f1e24008b00 tx=0x7f1e28001150).stop 2019-10-13T05:21:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- 172.21.15.71:0/1479991386 shutdown_connections 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 --2- 172.21.15.71:0/1479991386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e2c144a10 0x7f1e2c144e30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- 172.21.15.71:0/1479991386 shutdown_connections 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- 172.21.15.71:0/1479991386 wait complete. 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.734+0000 7f1e33b41700 1 -- 172.21.15.71:0/1479991386 >> 172.21.15.71:0/1479991386 conn(0x7f1e2c05e870 msgr2=0x7f1e2c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 Processor -- start 2019-10-13T05:21:22.748 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- start start 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1e2c12c0f0 0x7f1e2c12e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0a04b0 con 0x7f1e2c079600 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c079a20 con 0x7f1e2c12c0f0 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c079cb0 con 0x7f1e2c123920 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e320de700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1e2c079600 0x7f1e2c09fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.71:56922/0 (socket says 172.21.15.71:56922) 2019-10-13T05:21:22.752 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e320de700 1 -- 172.21.15.71:0/3191008830 learned_addr learned my addr 172.21.15.71:0/3191008830 (peer_addr_for_me v1:172.21.15.71:0/0) 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e328df700 1 --2- 172.21.15.71:0/3191008830 >> v2:172.21.15.129:3300/0 conn(0x7f1e2c12c0f0 0x7f1e2c12e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112665309 0 0) 0x7f1e2c0a04b0 con 0x7f1e2c079600 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e140030a0 con 0x7f1e2c079600 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 -- 172.21.15.71:0/3191008830 >> v2:172.21.15.129:3300/0 conn(0x7f1e2c12c0f0 msgr2=0x7f1e2c12e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> v2:172.21.15.129:3300/0 conn(0x7f1e2c12c0f0 0x7f1e2c12e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.753 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 -- 172.21.15.71:0/3191008830 >> v1:172.21.15.129:6789/0 conn(0x7f1e2c079600 legacy=0x7f1e2c09fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 -- 172.21.15.71:0/3191008830 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c0a0850 con 0x7f1e2c123920 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.754 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c006b20 con 0x7f1e2c123920 2019-10-13T05:21:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c006740 con 0x7f1e2c123920 2019-10-13T05:21:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c010e40 con 0x7f1e2c123920 2019-10-13T05:21:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c124290 con 0x7f1e2c123920 2019-10-13T05:21:22.756 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c07a840 con 0x7f1e2c123920 2019-10-13T05:21:22.756 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.738+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.756 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1e1c011150 con 0x7f1e2c123920 2019-10-13T05:21:22.756 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.757 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e22ffd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 0x7f1e1402a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:22.757 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f1e1c034ac0 con 0x7f1e2c123920 2019-10-13T05:21:22.757 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e320de700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 0x7f1e1402a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:22.757 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.742+0000 7f1e320de700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 0x7f1e1402a370 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:22.758 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.746+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c0555c0 con 0x7f1e2c123920 2019-10-13T05:21:22.759 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.754+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.759 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.754+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f1e1c025e50 con 0x7f1e2c123920 2019-10-13T05:21:22.907 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.902+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd new", "uuid": "460e48b8-6810-40b9-b018-121ed3eea51d", "id": 6} v 0) v1 -- 0x7f1e2c07a330 con 0x7f1e2c123920 2019-10-13T05:21:22.915 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.910+0000 7f1e318dd700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:22.916 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.910+0000 7f1e22ffd700 1 -- 172.21.15.71:0/3191008830 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "460e48b8-6810-40b9-b018-121ed3eea51d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1e1c011440 con 0x7f1e2c123920 2019-10-13T05:21:22.922 INFO:teuthology.orchestra.run.smithi071.stdout:6 2019-10-13T05:21:22.925 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 msgr2=0x7f1e1402a370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.925 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 0x7f1e1402a370 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 msgr2=0x7f1e2c123d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e240102f0).stop 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 shutdown_connections 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1e14027e70 0x7f1e1402a370 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 --2- 172.21.15.71:0/3191008830 >> v2:172.21.15.129:3300/0 conn(0x7f1e2c12c0f0 0x7f1e2c12e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.926 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 --2- 172.21.15.71:0/3191008830 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1e2c123920 0x7f1e2c123d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:22.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 shutdown_connections 2019-10-13T05:21:22.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 wait complete. 2019-10-13T05:21:22.927 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:22.918+0000 7f1e33b41700 1 -- 172.21.15.71:0/3191008830 >> 172.21.15.71:0/3191008830 conn(0x7f1e2c0a1c00 msgr2=0x7f1e2c0a7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:22.936 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:22.936 INFO:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 3b87653f-36c2-470a-b30d-c9cc5a20c689 7 2019-10-13T05:21:23.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 Processor -- start 2019-10-13T05:21:23.027 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 -- start start 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298144a10 0x7fb298144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2980a7a30 con 0x7fb2980a7520 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb298150ba0 con 0x7fb2980a1c80 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2980a96b0 con 0x7fb298144a10 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298144a10 0x7fb298144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:23.028 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298144a10 0x7fb298144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40240/0 (socket says 172.21.15.71:40240) 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 -- 172.21.15.71:0/123430010 learned_addr learned my addr 172.21.15.71:0/123430010 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 -- 172.21.15.71:0/123430010 >> v1:172.21.15.129:6789/0 conn(0x7fb2980a1c80 legacy=0x7fb2980a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 -- 172.21.15.71:0/123430010 >> v1:172.21.15.71:6789/0 conn(0x7fb2980a7520 legacy=0x7fb2980a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 -- 172.21.15.71:0/123430010 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298151de0 con 0x7fb298144a10 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> v2:172.21.15.71:3300/0 conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb290008b00 tx=0x7fb290010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> v2:172.21.15.71:3300/0 conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb290008b00 tx=0x7fb290010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.029 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fb290008b00 tx=0x7fb290010020).ready entity=mon.0 client_cookie=c524008fe8b4f2af server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fb290008b00 tx=0x7fb290010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fb290008b00 tx=0x7fb290010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29759e700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fb290008b00 tx=0x7fb290010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb295d9b700 1 -- 172.21.15.71:0/123430010 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb29000a160 con 0x7fb298144a10 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb295d9b700 1 -- 172.21.15.71:0/123430010 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290011a90 con 0x7fb298144a10 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb295d9b700 1 -- 172.21.15.71:0/123430010 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb29000a740 con 0x7fb298144a10 2019-10-13T05:21:23.030 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 -- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 msgr2=0x7fb298144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.018+0000 7fb29c904700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fb290008b00 tx=0x7fb290010020).stop 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 -- 172.21.15.71:0/123430010 shutdown_connections 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 --2- 172.21.15.71:0/123430010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298144a10 0x7fb298144e30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 -- 172.21.15.71:0/123430010 shutdown_connections 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 -- 172.21.15.71:0/123430010 wait complete. 2019-10-13T05:21:23.031 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 -- 172.21.15.71:0/123430010 >> 172.21.15.71:0/123430010 conn(0x7fb29805e870 msgr2=0x7fb29809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:23.034 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 Processor -- start 2019-10-13T05:21:23.034 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.022+0000 7fb29c904700 1 -- start start 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb2981116a0 0x7fb2981235c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb298126b80 0x7fb29807d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298123b10 0x7fb29807dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb298127100 con 0x7fb298123b10 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2981273a0 con 0x7fb298126b80 2019-10-13T05:21:23.035 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb298127640 con 0x7fb2981116a0 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298123b10 0x7fb29807dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb298123b10 0x7fb29807dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40244/0 (socket says 172.21.15.71:40244) 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 -- 172.21.15.71:0/4216316852 learned_addr learned my addr 172.21.15.71:0/4216316852 (peer_addr_for_me v2:172.21.15.71:0/0) 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb296d9d700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb2981116a0 0x7fb2981235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29659c700 1 --2- 172.21.15.71:0/4216316852 >> v2:172.21.15.129:3300/0 conn(0x7fb298126b80 0x7fb29807d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 -- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb2981116a0 msgr2=0x7fb2981235c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.036 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb2981116a0 0x7fb2981235c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.037 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 -- 172.21.15.71:0/4216316852 >> v2:172.21.15.129:3300/0 conn(0x7fb298126b80 msgr2=0x7fb29807d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.037 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> v2:172.21.15.129:3300/0 conn(0x7fb298126b80 0x7fb29807d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.037 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 -- 172.21.15.71:0/4216316852 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298116f40 con 0x7fb298123b10 2019-10-13T05:21:23.037 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> v2:172.21.15.71:3300/0 conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> v2:172.21.15.71:3300/0 conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb29000f790 con 0x7fb298123b10 2019-10-13T05:21:23.038 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29000a360 con 0x7fb298123b10 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb290005aa0 con 0x7fb298123b10 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2981278e0 con 0x7fb298123b10 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2980a6f60 con 0x7fb298123b10 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.026+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb2900100a0 con 0x7fb298123b10 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.030+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.039 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.030+0000 7fb2877fe700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 0x7fb27c029e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:23.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.030+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7fb2900352d0 con 0x7fb298123b10 2019-10-13T05:21:23.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.030+0000 7fb296d9d700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 0x7fb27c029e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:23.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.030+0000 7fb296d9d700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 0x7fb27c029e70 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:23.040 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.034+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2980555c0 con 0x7fb298123b10 2019-10-13T05:21:23.045 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.042+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.046 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.042+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb29000ade0 con 0x7fb298123b10 2019-10-13T05:21:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.198+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3b87653f-36c2-470a-b30d-c9cc5a20c689", "id": 7} v 0) v1 -- 0x7fb298067b90 con 0x7fb298123b10 2019-10-13T05:21:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.206+0000 7fb29759e700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.206+0000 7fb2877fe700 1 -- 172.21.15.71:0/4216316852 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b87653f-36c2-470a-b30d-c9cc5a20c689", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb29000fbc0 con 0x7fb298123b10 2019-10-13T05:21:23.211 INFO:teuthology.orchestra.run.smithi071.stdout:7 2019-10-13T05:21:23.214 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 msgr2=0x7fb27c029e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 0x7fb27c029e70 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 msgr2=0x7fb29807dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fb2981272c0 tx=0x7fb28c001200).stop 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 shutdown_connections 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb27c027970 0x7fb27c029e70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb298123b10 0x7fb29807dc10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb2981116a0 0x7fb2981235c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 --2- 172.21.15.71:0/4216316852 >> v2:172.21.15.129:3300/0 conn(0x7fb298126b80 0x7fb29807d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 shutdown_connections 2019-10-13T05:21:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 wait complete. 2019-10-13T05:21:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2019-10-13T05:21:23.210+0000 7fb29c904700 1 -- 172.21.15.71:0/4216316852 >> 172.21.15.71:0/4216316852 conn(0x7fb29807ec10 msgr2=0x7fb2980a8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:23.224 INFO:tasks.ceph.osd.0:Restarting daemon 2019-10-13T05:21:23.224 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:23.225 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-10-13T05:21:23.228 INFO:tasks.ceph.osd.0:Started 2019-10-13T05:21:23.228 INFO:tasks.ceph.osd.1:Restarting daemon 2019-10-13T05:21:23.228 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:23.229 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-10-13T05:21:23.235 INFO:tasks.ceph.osd.1:Started 2019-10-13T05:21:23.236 INFO:tasks.ceph.osd.2:Restarting daemon 2019-10-13T05:21:23.236 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:23.236 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-10-13T05:21:23.240 INFO:tasks.ceph.osd.2:Started 2019-10-13T05:21:23.240 INFO:tasks.ceph.osd.3:Restarting daemon 2019-10-13T05:21:23.240 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:23.240 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-10-13T05:21:23.253 INFO:tasks.ceph.osd.3:Started 2019-10-13T05:21:23.253 INFO:tasks.ceph.osd.4:Restarting daemon 2019-10-13T05:21:23.254 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:23.254 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-10-13T05:21:23.257 INFO:tasks.ceph.osd.4:Started 2019-10-13T05:21:23.257 INFO:tasks.ceph.osd.5:Restarting daemon 2019-10-13T05:21:23.258 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:23.258 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-10-13T05:21:23.261 INFO:tasks.ceph.osd.5:Started 2019-10-13T05:21:23.261 INFO:tasks.ceph.osd.6:Restarting daemon 2019-10-13T05:21:23.262 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:23.262 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-10-13T05:21:23.265 INFO:tasks.ceph.osd.6:Started 2019-10-13T05:21:23.266 INFO:tasks.ceph.osd.7:Restarting daemon 2019-10-13T05:21:23.266 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:23.266 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-10-13T05:21:23.276 INFO:tasks.ceph.osd.7:Started 2019-10-13T05:21:23.276 INFO:tasks.ceph:Waiting for OSDs to come up 2019-10-13T05:21:23.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T05:21:23.587+0000 7f9153227d80 -1 Falling back to public interface 2019-10-13T05:21:23.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T05:21:23.599+0000 7ffb34d69d80 -1 Falling back to public interface 2019-10-13T05:21:23.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T05:21:23.615+0000 7fb4ef635d80 -1 Falling back to public interface 2019-10-13T05:21:23.623 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:21:23.618+0000 7f99ab0b4d80 -1 Falling back to public interface 2019-10-13T05:21:23.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T05:21:23.826+0000 7f0926d27d80 -1 Falling back to public interface 2019-10-13T05:21:23.855 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:21:23.850+0000 7f28f3758d80 -1 Falling back to public interface 2019-10-13T05:21:23.864 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:21:23.858+0000 7f9d8b263d80 -1 Falling back to public interface 2019-10-13T05:21:23.878 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:21:23.874+0000 7fbf7ad35d80 -1 Falling back to public interface 2019-10-13T05:21:24.086 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:24.086 INFO:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:24.162 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 Processor -- start 2019-10-13T05:21:24.163 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 -- start start 2019-10-13T05:21:24.163 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead40a1c80 0x7fead40a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:24.163 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead40a7a30 con 0x7fead4144a10 2019-10-13T05:21:24.163 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead40a1940 con 0x7fead40a7520 2019-10-13T05:21:24.164 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead40a96b0 con 0x7fead40a1c80 2019-10-13T05:21:24.164 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9e79700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fead40a7520 0x7fead40a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58476/0 (socket says 172.21.15.129:58476) 2019-10-13T05:21:24.164 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9e79700 1 -- 172.21.15.129:0/3723402194 learned_addr learned my addr 172.21.15.129:0/3723402194 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:24.164 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead40a1c80 0x7fead40a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.164 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117442375 0 0) 0x7fead40a1940 con 0x7fead40a7520 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac00030a0 con 0x7fead40a7520 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 150507452 0 0) 0x7feac00030a0 con 0x7fead40a7520 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead40a1c80 msgr2=0x7fead40a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead40a1c80 0x7fead40a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 >> v1:172.21.15.71:6789/0 conn(0x7fead4144a10 legacy=0x7fead414eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4144e30 con 0x7fead40a7520 2019-10-13T05:21:24.165 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7feac8001750 con 0x7fead40a7520 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 >> v1:172.21.15.129:6789/0 conn(0x7fead40a7520 legacy=0x7fead40a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feac0004410 0x7feac0004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feac0004d80 0x7feac000b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40a96b0 con 0x7feac0004410 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.166 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac00051a0 con 0x7feac00036d0 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac004fbc0 con 0x7feac0004d80 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9e79700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feac0004410 0x7feac0004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadae7b700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feac0004d80 0x7feac000b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 -- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feac0004d80 msgr2=0x7feac000b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feac0004d80 0x7feac000b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:24.167 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 -- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feac0004410 msgr2=0x7feac0004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feac0004410 0x7feac0004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 -- 172.21.15.129:0/3723402194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac004fe40 con 0x7feac00036d0 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9e79700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feac0004410 0x7feac0004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feac00053e0 tx=0x7feac8001d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feac00053e0 tx=0x7feac8001d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.168 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7feac00053e0 tx=0x7feac8001d20).ready entity=mon.1 client_cookie=7bf0448846877055 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:24.169 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7feac00053e0 tx=0x7feac8001d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.169 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead0008ba0 con 0x7feac00036d0 2019-10-13T05:21:24.169 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7feac00053e0 tx=0x7feac8001d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.169 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feada67a700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7feac00053e0 tx=0x7feac8001d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.169 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead00087c0 con 0x7feac00036d0 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7fead9678700 1 -- 172.21.15.129:0/3723402194 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead00115b0 con 0x7feac00036d0 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 -- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 msgr2=0x7feac0003af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadc0dd700 1 --2- 172.21.15.129:0/3723402194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac00036d0 0x7feac0003af0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7feac00053e0 tx=0x7feac8001d20).stop 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.154+0000 7feadae7b700 1 -- 172.21.15.129:0/3723402194 reap_dead start 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- 172.21.15.129:0/3723402194 shutdown_connections 2019-10-13T05:21:24.170 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- 172.21.15.129:0/3723402194 shutdown_connections 2019-10-13T05:21:24.171 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- 172.21.15.129:0/3723402194 wait complete. 2019-10-13T05:21:24.171 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- 172.21.15.129:0/3723402194 >> 172.21.15.129:0/3723402194 conn(0x7fead405e870 msgr2=0x7fead409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:24.171 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 Processor -- start 2019-10-13T05:21:24.171 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- start start 2019-10-13T05:21:24.176 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fead40a6bc0 0x7fead411f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.176 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead409fba0 0x7fead413a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead4078bc0 con 0x7fead411fdc0 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead40a7c60 con 0x7fead40a6bc0 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadc0dd700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead4078f40 con 0x7fead409fba0 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7fead9e79700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fead411fdc0 0x7fead409f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58488/0 (socket says 172.21.15.129:58488) 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7fead9e79700 1 -- 172.21.15.129:0/1891050709 learned_addr learned my addr 172.21.15.129:0/1891050709 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead409fba0 0x7fead413a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.177 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feada67a700 1 --2- 172.21.15.129:0/1891050709 >> v2:172.21.15.71:3300/0 conn(0x7fead40a6bc0 0x7fead411f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769980704 0 0) 0x7fead4078bc0 con 0x7fead411fdc0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab8002e50 con 0x7fead411fdc0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 429181385 0 0) 0x7feab8002e50 con 0x7fead411fdc0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead4078bc0 con 0x7fead411fdc0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7feac8001750 con 0x7fead411fdc0 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead409fba0 msgr2=0x7fead413a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.178 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fead409fba0 0x7fead413a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 >> v2:172.21.15.71:3300/0 conn(0x7fead40a6bc0 msgr2=0x7fead411f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> v2:172.21.15.71:3300/0 conn(0x7fead40a6bc0 0x7fead411f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 >> v1:172.21.15.129:6789/0 conn(0x7fead411fdc0 legacy=0x7fead409f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab80049c0 0x7feab8004de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feab8003790 0x7feab8003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.179 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40a7c60 con 0x7feab80049c0 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead4078f40 con 0x7feab8004100 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab800bbc0 con 0x7feab8003790 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7fead9e79700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feab8003790 0x7feab8003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.158+0000 7feada67a700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab80049c0 0x7feab8004de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 -- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feab8003790 msgr2=0x7feab8003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.180 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7feab8003790 0x7feab8003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 -- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab80049c0 msgr2=0x7feab8004de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab80049c0 0x7feab8004de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead41201e0 con 0x7feab8004100 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead409e8a0 con 0x7feab8004100 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feada67a700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab80049c0 0x7feab8004de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.181 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead409fba0 con 0x7feab8004100 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc0066d0 con 0x7feab8004100 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.182 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc001bf0 con 0x7feab8004100 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc006cd0 con 0x7feab8004100 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7feacc013060 con 0x7feab8004100 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7feab8030440 0x7feab8032940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:24.183 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7feacc034540 con 0x7feab8004100 2019-10-13T05:21:24.184 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feada67a700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7feab8030440 0x7feab8032940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:24.184 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.162+0000 7feada67a700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7feab8030440 0x7feab8032940 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:24.184 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.170+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab00031f0 con 0x7feab8004100 2019-10-13T05:21:24.185 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.174+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.185 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.174+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7feacc012710 con 0x7feab8004100 2019-10-13T05:21:24.344 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.334+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feab0003630 con 0x7feab8004100 2019-10-13T05:21:24.344 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.334+0000 7feadae7b700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:24.344 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.334+0000 7feac6ffd700 1 -- 172.21.15.129:0/1891050709 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7feacc008be0 con 0x7feab8004100 2019-10-13T05:21:24.345 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7feab8030440 msgr2=0x7feab8032940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.345 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7feab8030440 0x7feab8032940 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:24.345 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 msgr2=0x7feab800b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:24.345 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 --2- 172.21.15.129:0/1891050709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feab8004100 0x7feab800b670 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7feac8001520 tx=0x7fead0008110).stop 2019-10-13T05:21:24.345 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadae7b700 1 -- 172.21.15.129:0/1891050709 reap_dead start 2019-10-13T05:21:24.346 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 shutdown_connections 2019-10-13T05:21:24.346 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 shutdown_connections 2019-10-13T05:21:24.346 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 wait complete. 2019-10-13T05:21:24.346 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:24.338+0000 7feadc0dd700 1 -- 172.21.15.129:0/1891050709 >> 172.21.15.129:0/1891050709 conn(0x7fead409c700 msgr2=0x7fead4123140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:24.347 INFO:teuthology.misc.health.smithi129.stdout: 2019-10-13T05:21:24.347 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:23.205612+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:24.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-10-13T05:21:24.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T05:21:24.742+0000 7f9153227d80 -1 osd.0 0 log_to_monitors {default=true} 2019-10-13T05:21:24.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T05:21:24.750+0000 7ffb34d69d80 -1 osd.2 0 log_to_monitors {default=true} 2019-10-13T05:21:24.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T05:21:24.758+0000 7fb4ef635d80 -1 osd.3 0 log_to_monitors {default=true} 2019-10-13T05:21:24.795 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:21:24.790+0000 7f99ab0b4d80 -1 osd.7 0 log_to_monitors {default=true} 2019-10-13T05:21:24.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T05:21:24.982+0000 7f0926d27d80 -1 osd.1 0 log_to_monitors {default=true} 2019-10-13T05:21:25.033 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:21:25.026+0000 7f9d8b263d80 -1 osd.5 0 log_to_monitors {default=true} 2019-10-13T05:21:25.043 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:21:25.038+0000 7f28f3758d80 -1 osd.6 0 log_to_monitors {default=true} 2019-10-13T05:21:25.043 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:21:25.038+0000 7fbf7ad35d80 -1 osd.4 0 log_to_monitors {default=true} 2019-10-13T05:21:26.420 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T05:21:26.413+0000 7f913c45c700 -1 osd.0 0 waiting for initial osdmap 2019-10-13T05:21:26.421 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:21:26.417+0000 7f99942e9700 -1 osd.7 0 waiting for initial osdmap 2019-10-13T05:21:26.421 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:21:26.417+0000 7f9d74498700 -1 osd.5 0 waiting for initial osdmap 2019-10-13T05:21:26.422 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:21:26.417+0000 7f28dc98d700 -1 osd.6 0 waiting for initial osdmap 2019-10-13T05:21:26.428 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T05:21:26.421+0000 7ffb1df9e700 -1 osd.2 0 waiting for initial osdmap 2019-10-13T05:21:26.428 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T05:21:26.421+0000 7f090ff5c700 -1 osd.1 0 waiting for initial osdmap 2019-10-13T05:21:26.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T05:21:26.421+0000 7fb4d886a700 -1 osd.3 0 waiting for initial osdmap 2019-10-13T05:21:26.432 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:21:26.429+0000 7fbf63f6a700 -1 osd.4 0 waiting for initial osdmap 2019-10-13T05:21:31.163 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:31.163 INFO:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:31.275 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 Processor -- start 2019-10-13T05:21:31.276 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 -- start start 2019-10-13T05:21:31.276 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f15b8144a10 0x7f15b8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:31.276 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80a7a30 con 0x7f15b80a7520 2019-10-13T05:21:31.276 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8150ba0 con 0x7f15b80a1c80 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15c02ef700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b80a96b0 con 0x7f15b8144a10 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15be88c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f15b80a1c80 0x7f15b80a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58720/0 (socket says 172.21.15.129:58720) 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15be88c700 1 -- 172.21.15.129:0/2160137022 learned_addr learned my addr 172.21.15.129:0/2160137022 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> v2:172.21.15.129:3300/0 conn(0x7f15b8144a10 0x7f15b8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068834581 0 0) 0x7f15b8150ba0 con 0x7f15b80a1c80 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a40030a0 con 0x7f15b80a1c80 2019-10-13T05:21:31.277 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163306415 0 0) 0x7f15b80a7a30 con 0x7f15b80a7520 2019-10-13T05:21:31.278 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bf08d700 1 -- 172.21.15.129:0/2160137022 >> v1:172.21.15.129:6789/0 conn(0x7f15b80a1c80 legacy=0x7f15b80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.278 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bf08d700 1 -- 172.21.15.129:0/2160137022 >> v1:172.21.15.71:6789/0 conn(0x7f15b80a7520 legacy=0x7f15b80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.278 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.257+0000 7f15bf08d700 1 -- 172.21.15.129:0/2160137022 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8151de0 con 0x7f15b8144a10 2019-10-13T05:21:31.278 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> v2:172.21.15.129:3300/0 conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f15b0008780 tx=0x7f15b000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.278 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> v2:172.21.15.129:3300/0 conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f15b0008780 tx=0x7f15b000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.279 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f15b0008780 tx=0x7f15b000fbe0).ready entity=mon.1 client_cookie=8ffa38c0296106ba server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.279 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f15b0008780 tx=0x7f15b000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.279 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b0011260 con 0x7f15b8144a10 2019-10-13T05:21:31.279 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f15b0008780 tx=0x7f15b000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.280 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f15b0008780 tx=0x7f15b000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.280 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0008bb0 con 0x7f15b8144a10 2019-10-13T05:21:31.280 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15bd88a700 1 -- 172.21.15.129:0/2160137022 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b0011840 con 0x7f15b8144a10 2019-10-13T05:21:31.281 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 -- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 msgr2=0x7f15b8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.281 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f15b0008780 tx=0x7f15b000fbe0).stop 2019-10-13T05:21:31.281 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 -- 172.21.15.129:0/2160137022 shutdown_connections 2019-10-13T05:21:31.281 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 --2- 172.21.15.129:0/2160137022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b8144a10 0x7f15b8144e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.282 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 -- 172.21.15.129:0/2160137022 shutdown_connections 2019-10-13T05:21:31.282 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 -- 172.21.15.129:0/2160137022 wait complete. 2019-10-13T05:21:31.282 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.261+0000 7f15c02ef700 1 -- 172.21.15.129:0/2160137022 >> 172.21.15.129:0/2160137022 conn(0x7f15b805e870 msgr2=0x7f15b809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:31.283 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 Processor -- start 2019-10-13T05:21:31.283 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 -- start start 2019-10-13T05:21:31.283 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15b8079a80 0x7f15b8078090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.293 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80785e0 con 0x7f15b80aa1d0 2019-10-13T05:21:31.294 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8078920 con 0x7f15b811fac0 2019-10-13T05:21:31.294 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80792a0 con 0x7f15b8079a80 2019-10-13T05:21:31.294 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f15b811fac0 0x7f15b811fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58728/0 (socket says 172.21.15.129:58728) 2019-10-13T05:21:31.294 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 learned_addr learned my addr 172.21.15.129:0/2586627582 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:31.295 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15be08b700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15b8079a80 0x7f15b8078090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.295 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494879994 0 0) 0x7f15b8078920 con 0x7f15b811fac0 2019-10-13T05:21:31.295 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a00030a0 con 0x7f15b811fac0 2019-10-13T05:21:31.295 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181486121 0 0) 0x7f15b80785e0 con 0x7f15b80aa1d0 2019-10-13T05:21:31.295 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b8078920 con 0x7f15b80aa1d0 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2945434331 0 0) 0x7f15a00030a0 con 0x7f15b811fac0 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15b80785e0 con 0x7f15b811fac0 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7f15b0011260 con 0x7f15b811fac0 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15b8079a80 msgr2=0x7f15b8078090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15b8079a80 0x7f15b8078090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.296 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 >> v1:172.21.15.129:6789/0 conn(0x7f15b811fac0 legacy=0x7f15b811fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.297 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 >> v1:172.21.15.71:6789/0 conn(0x7f15b80aa1d0 legacy=0x7f15b8126120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.297 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15a0003aa0 0x7f15a0004410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.297 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15a0003f50 0x7f15a00049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.297 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80792a0 con 0x7f15a0003f50 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000bd80 con 0x7f15a0005000 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000bfc0 con 0x7f15a0003aa0 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15be08b700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15b8079a80 0x7f15b8078090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.298 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15be88c700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15a0003aa0 0x7f15a0004410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15be08b700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15a0003f50 0x7f15a00049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15a0003aa0 msgr2=0x7f15a0004410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15a0003aa0 0x7f15a0004410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15a0003f50 msgr2=0x7f15a00049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15a0003f50 0x7f15a00049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.299 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b81305a0 con 0x7f15a0005000 2019-10-13T05:21:31.300 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.265+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b8079a80 con 0x7f15a0005000 2019-10-13T05:21:31.300 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15be88c700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f15a0003aa0 0x7f15a0004410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:31.300 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15be08b700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15a0003f50 0x7f15a00049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T05:21:31.300 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8078090 con 0x7f15a0005000 2019-10-13T05:21:31.300 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.301 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.301 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.301 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.301 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b000f970 con 0x7f15a0005000 2019-10-13T05:21:31.301 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.302 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0007180 con 0x7f15a0005000 2019-10-13T05:21:31.302 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b000abf0 con 0x7f15a0005000 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f15b0005530 con 0x7f15a0005000 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f15a0030aa0 0x7f15a0032fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.303 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f15b0035900 con 0x7f15a0005000 2019-10-13T05:21:31.304 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15be88c700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f15a0030aa0 0x7f15a0032fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.304 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.269+0000 7f15be88c700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f15a0030aa0 0x7f15a0032fa0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.304 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.273+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15940031f0 con 0x7f15a0005000 2019-10-13T05:21:31.304 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.277+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.305 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.277+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f15b0005820 con 0x7f15a0005000 2019-10-13T05:21:31.444 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.437+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1594003630 con 0x7f15a0005000 2019-10-13T05:21:31.445 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15bf08d700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.446 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15af7fe700 1 -- 172.21.15.129:0/2586627582 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10439 (secure 0 0 0) 0x7f15b0005a40 con 0x7f15a0005000 2019-10-13T05:21:31.447 INFO:teuthology.misc.health.smithi129.stdout: 2019-10-13T05:21:31.447 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":14,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:28.442780+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:31.449 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f15a0030aa0 msgr2=0x7f15a0032fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.450 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f15a0030aa0 0x7f15a0032fa0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.450 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 msgr2=0x7f15a000b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.450 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 --2- 172.21.15.129:0/2586627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15a0005000 0x7f15a000b6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f15b0008fc0 tx=0x7f15a8001380).stop 2019-10-13T05:21:31.450 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 reap_dead start 2019-10-13T05:21:31.450 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15bf08d700 1 -- 172.21.15.129:0/2586627582 reap_dead start 2019-10-13T05:21:31.451 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 shutdown_connections 2019-10-13T05:21:31.451 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.441+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 shutdown_connections 2019-10-13T05:21:31.451 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.445+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 wait complete. 2019-10-13T05:21:31.451 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:31.445+0000 7f15c02ef700 1 -- 172.21.15.129:0/2586627582 >> 172.21.15.129:0/2586627582 conn(0x7f15b80a7520 msgr2=0x7f15b807c950 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:31.459 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T05:21:31.459 INFO:tasks.ceph:Creating RBD pool 2019-10-13T05:21:31.460 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:31.460 INFO:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool create rbd 8 2019-10-13T05:21:31.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 Processor -- start 2019-10-13T05:21:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 -- start start 2019-10-13T05:21:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c1c0a82d0 0x7f4c1c0a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c144920 0x7f4c1c144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c0a8c40 con 0x7f4c1c09d160 2019-10-13T05:21:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c0a1300 con 0x7f4c1c144920 2019-10-13T05:21:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c215e8700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c0a15a0 con 0x7f4c1c0a82d0 2019-10-13T05:21:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c144920 0x7f4c1c144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c144920 0x7f4c1c144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40774/0 (socket says 172.21.15.129:40774) 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 -- 172.21.15.129:0/681614886 learned_addr learned my addr 172.21.15.129:0/681614886 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1affd700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c1c0a82d0 0x7f4c1c0a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 -- 172.21.15.129:0/681614886 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c1c0a82d0 msgr2=0x7f4c1c0a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c1c0a82d0 0x7f4c1c0a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 -- 172.21.15.129:0/681614886 >> v1:172.21.15.71:6789/0 conn(0x7f4c1c09d160 legacy=0x7f4c1c0a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 -- 172.21.15.129:0/681614886 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c145290 con 0x7f4c1c144920 2019-10-13T05:21:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.553+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> v2:172.21.15.129:3300/0 conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4c14008b00 tx=0x7f4c10007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> v2:172.21.15.129:3300/0 conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4c14008b00 tx=0x7f4c10007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f4c14008b00 tx=0x7f4c10007710).ready entity=mon.1 client_cookie=c8968c940577bd06 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f4c14008b00 tx=0x7f4c10007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f4c14008b00 tx=0x7f4c10007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1a7fc700 1 -- 172.21.15.129:0/681614886 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c14010b90 con 0x7f4c1c144920 2019-10-13T05:21:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f4c14008b00 tx=0x7f4c10007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1a7fc700 1 -- 172.21.15.129:0/681614886 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c140107b0 con 0x7f4c1c144920 2019-10-13T05:21:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c1a7fc700 1 -- 172.21.15.129:0/681614886 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c140112b0 con 0x7f4c1c144920 2019-10-13T05:21:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 -- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 msgr2=0x7f4c1c144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f4c14008b00 tx=0x7f4c10007710).stop 2019-10-13T05:21:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 -- 172.21.15.129:0/681614886 shutdown_connections 2019-10-13T05:21:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c144920 0x7f4c1c144d40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 --2- 172.21.15.129:0/681614886 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c1c0a82d0 0x7f4c1c0a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 -- 172.21.15.129:0/681614886 shutdown_connections 2019-10-13T05:21:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 -- 172.21.15.129:0/681614886 wait complete. 2019-10-13T05:21:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.557+0000 7f4c215e8700 1 -- 172.21.15.129:0/681614886 >> 172.21.15.129:0/681614886 conn(0x7f4c1c05e870 msgr2=0x7f4c1c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 Processor -- start 2019-10-13T05:21:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- start start 2019-10-13T05:21:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c1c0a9500 0x7f4c1c072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c0726a0 0x7f4c1c074ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c064c70 con 0x7f4c1c0645c0 2019-10-13T05:21:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c063de0 con 0x7f4c1c0a9500 2019-10-13T05:21:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c064080 con 0x7f4c1c0726a0 2019-10-13T05:21:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c0726a0 0x7f4c1c074ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4c1c0726a0 0x7f4c1c074ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40780/0 (socket says 172.21.15.129:40780) 2019-10-13T05:21:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 -- 172.21.15.129:0/2180402233 learned_addr learned my addr 172.21.15.129:0/2180402233 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1affd700 1 --2- 172.21.15.129:0/2180402233 >> v2:172.21.15.71:3300/0 conn(0x7f4c1c0a9500 0x7f4c1c072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 -- 172.21.15.129:0/2180402233 >> v2:172.21.15.71:3300/0 conn(0x7f4c1c0a9500 msgr2=0x7f4c1c072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> v2:172.21.15.71:3300/0 conn(0x7f4c1c0a9500 0x7f4c1c072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 -- 172.21.15.129:0/2180402233 >> v1:172.21.15.71:6789/0 conn(0x7f4c1c0645c0 legacy=0x7f4c1c0a4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 -- 172.21.15.129:0/2180402233 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c11b890 con 0x7f4c1c0726a0 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> v2:172.21.15.129:3300/0 conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> v2:172.21.15.129:3300/0 conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c14005aa0 con 0x7f4c1c0726a0 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c14010d90 con 0x7f4c1c0726a0 2019-10-13T05:21:31.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c1400f470 con 0x7f4c1c0726a0 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c0a5d00 con 0x7f4c1c0726a0 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.561+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c075020 con 0x7f4c1c0726a0 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4c1400f780 con 0x7f4c1c0726a0 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c03fff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 0x7f4bfc029db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f4c14034d40 con 0x7f4c1c0726a0 2019-10-13T05:21:31.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c1b7fe700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 0x7f4bfc029db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:31.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.565+0000 7f4c1b7fe700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 0x7f4bfc029db0 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:31.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.569+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c0a9920 con 0x7f4c1c0726a0 2019-10-13T05:21:31.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.573+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:31.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.573+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4c1401da60 con 0x7f4c1c0726a0 2019-10-13T05:21:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:31.721+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f4c1c0555c0 con 0x7f4c1c0726a0 2019-10-13T05:21:32.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.477+0000 7f4c1bfff700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.477+0000 7f4c03fff700 1 -- 172.21.15.129:0/2180402233 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4c1401dc80 con 0x7f4c1c0726a0 2019-10-13T05:21:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'rbd' created 2019-10-13T05:21:32.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 msgr2=0x7f4bfc029db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 0x7f4bfc029db0 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 msgr2=0x7f4c1c074ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7f4c14003250 tx=0x7f4c10007550).stop 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 shutdown_connections 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c1c0726a0 0x7f4c1c074ad0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 --2- 172.21.15.129:0/2180402233 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4bfc0278b0 0x7f4bfc029db0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:32.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 --2- 172.21.15.129:0/2180402233 >> v2:172.21.15.71:3300/0 conn(0x7f4c1c0a9500 0x7f4c1c072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 shutdown_connections 2019-10-13T05:21:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 wait complete. 2019-10-13T05:21:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.489+0000 7f4c215e8700 1 -- 172.21.15.129:0/2180402233 >> 172.21.15.129:0/2180402233 conn(0x7f4c1c0a75b0 msgr2=0x7f4c1c124a80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:32.507 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:32.507 INFO:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-10-13T05:21:32.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 Processor -- start 2019-10-13T05:21:32.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 -- start start 2019-10-13T05:21:32.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:32.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80a7a30 con 0x7fd8f8144a10 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80aa3e0 con 0x7fd8f80a1c80 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8fcb71700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80aa610 con 0x7fd8f80a7520 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8f6d9d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd8f80a1c80 0x7fd8f80a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58756/0 (socket says 172.21.15.129:58756) 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8f6d9d700 1 -- 172.21.15.129:0/2067975296 learned_addr learned my addr 172.21.15.129:0/2067975296 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.601+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843064588 0 0) 0x7fd8f80aa3e0 con 0x7fd8f80a1c80 2019-10-13T05:21:32.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e00030a0 con 0x7fd8f80a1c80 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344773722 0 0) 0x7fd8f80a7a30 con 0x7fd8f8144a10 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f80aa3e0 con 0x7fd8f8144a10 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 -- 172.21.15.129:0/2067975296 >> v1:172.21.15.129:6789/0 conn(0x7fd8f80a1c80 legacy=0x7fd8f80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 -- 172.21.15.129:0/2067975296 >> v1:172.21.15.71:6789/0 conn(0x7fd8f8144a10 legacy=0x7fd8f814eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 -- 172.21.15.129:0/2067975296 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8144e30 con 0x7fd8f80a7520 2019-10-13T05:21:32.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).ready entity=mon.2 client_cookie=e2c697f3ad2d41a3 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec011260 con 0x7fd8f80a7520 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec008bb0 con 0x7fd8f80a7520 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f659c700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8f5d9b700 1 -- 172.21.15.129:0/2067975296 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec011840 con 0x7fd8f80a7520 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 -- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 msgr2=0x7fd8f80a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7fd8ec002f30 tx=0x7fd8ec00fbe0).stop 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 -- 172.21.15.129:0/2067975296 shutdown_connections 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/2067975296 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd8f80a7520 0x7fd8f80a9e90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 -- 172.21.15.129:0/2067975296 shutdown_connections 2019-10-13T05:21:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 -- 172.21.15.129:0/2067975296 wait complete. 2019-10-13T05:21:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.605+0000 7fd8fcb71700 1 -- 172.21.15.129:0/2067975296 >> 172.21.15.129:0/2067975296 conn(0x7fd8f805e870 msgr2=0x7fd8f809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 Processor -- start 2019-10-13T05:21:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 -- start start 2019-10-13T05:21:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd8f8113d00 0x7fd8f814f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f8150030 con 0x7fd8f81161d0 2019-10-13T05:21:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f8131bf0 con 0x7fd8f807b0c0 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8fcb71700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f807b5d0 con 0x7fd8f8113d00 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f6d9d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd8f807b0c0 0x7fd8f8119840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58760/0 (socket says 172.21.15.129:58760) 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> v2:172.21.15.129:3300/0 conn(0x7fd8f8113d00 0x7fd8f814f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f6d9d700 1 -- 172.21.15.129:0/482118 learned_addr learned my addr 172.21.15.129:0/482118 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494473534 0 0) 0x7fd8f8150030 con 0x7fd8f81161d0 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 -- 172.21.15.129:0/482118 >> v1:172.21.15.129:6789/0 conn(0x7fd8f807b0c0 legacy=0x7fd8f8119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 -- 172.21.15.129:0/482118 >> v1:172.21.15.71:6789/0 conn(0x7fd8f81161d0 legacy=0x7fd8f814f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 -- 172.21.15.129:0/482118 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8132e30 con 0x7fd8f8113d00 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> v2:172.21.15.129:3300/0 conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> v2:172.21.15.129:3300/0 conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec011da0 con 0x7fd8f8113d00 2019-10-13T05:21:32.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.609+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec011460 con 0x7fd8f8113d00 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f8130be0 con 0x7fd8f8113d00 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec00a5c0 con 0x7fd8f8113d00 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f81310e0 con 0x7fd8f8113d00 2019-10-13T05:21:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd8ec00f850 con 0x7fd8f8113d00 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8e77fe700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 0x7fd8dc029c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7fd8ec035290 con 0x7fd8f8113d00 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8f6d9d700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 0x7fd8dc029c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:32.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.613+0000 7fd8f6d9d700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 0x7fd8dc029c60 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:32.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.617+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f80555c0 con 0x7fd8f8113d00 2019-10-13T05:21:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.625+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:32.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.625+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd8ec01dda0 con 0x7fd8f8113d00 2019-10-13T05:21:32.788 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:32.781+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd8f809d070 con 0x7fd8f8113d00 2019-10-13T05:21:33.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.489+0000 7fd8f659c700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.489+0000 7fd8e77fe700 1 -- 172.21.15.129:0/482118 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd8ec01e000 con 0x7fd8f8113d00 2019-10-13T05:21:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:enabled application 'rbd' on pool 'rbd' 2019-10-13T05:21:33.506 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 msgr2=0x7fd8dc029c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.506 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 0x7fd8dc029c60 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 msgr2=0x7fd8f814f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7fd8e80010d0 tx=0x7fd8e8001110).stop 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 shutdown_connections 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd8dc027760 0x7fd8dc029c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 --2- 172.21.15.129:0/482118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8113d00 0x7fd8f814f2e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 shutdown_connections 2019-10-13T05:21:33.507 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 wait complete. 2019-10-13T05:21:33.508 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.501+0000 7fd8fcb71700 1 -- 172.21.15.129:0/482118 >> 172.21.15.129:0/482118 conn(0x7fd8f809c620 msgr2=0x7fd8f8117cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:33.517 INFO:tasks.ceph:Setting up CephFS filesystem... 2019-10-13T05:21:33.517 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:33.517 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:33.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 Processor -- start 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 -- start start 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1480a7a30 con 0x7fa148144a10 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1480aa3e0 con 0x7fa1480a1c80 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14e4ed700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480aa610 con 0x7fa1480a7520 2019-10-13T05:21:33.618 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14ca8a700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa1480a1c80 0x7fa1480a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58772/0 (socket says 172.21.15.129:58772) 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa14ca8a700 1 -- 172.21.15.129:0/2442883745 learned_addr learned my addr 172.21.15.129:0/2442883745 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.605+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401289281 0 0) 0x7fa1480aa3e0 con 0x7fa1480a1c80 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1300030a0 con 0x7fa1480a1c80 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351655818 0 0) 0x7fa1480a7a30 con 0x7fa148144a10 2019-10-13T05:21:33.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1480aa3e0 con 0x7fa148144a10 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 -- 172.21.15.129:0/2442883745 >> v1:172.21.15.129:6789/0 conn(0x7fa1480a1c80 legacy=0x7fa1480a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 776557187 0 0) 0x7fa1300030a0 con 0x7fa1480a1c80 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 -- 172.21.15.129:0/2442883745 >> v1:172.21.15.71:6789/0 conn(0x7fa148144a10 legacy=0x7fa14814eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 -- 172.21.15.129:0/2442883745 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148144e30 con 0x7fa1480a7520 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa13c008780 tx=0x7fa138001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa13c008780 tx=0x7fa138001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fa13c008780 tx=0x7fa138001950).ready entity=mon.2 client_cookie=40c10ee0026c35c5 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa13c008780 tx=0x7fa138001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa138001750 con 0x7fa1480a7520 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa13c008780 tx=0x7fa138001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa138001560 con 0x7fa1480a7520 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa147fff700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa13c008780 tx=0x7fa138001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa1477fe700 1 -- 172.21.15.129:0/2442883745 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa13c00a350 con 0x7fa1480a7520 2019-10-13T05:21:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 -- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 msgr2=0x7fa1480a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fa13c008780 tx=0x7fa138001950).stop 2019-10-13T05:21:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 -- 172.21.15.129:0/2442883745 shutdown_connections 2019-10-13T05:21:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/2442883745 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa1480a7520 0x7fa1480a9e90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 -- 172.21.15.129:0/2442883745 shutdown_connections 2019-10-13T05:21:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 -- 172.21.15.129:0/2442883745 wait complete. 2019-10-13T05:21:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.609+0000 7fa14e4ed700 1 -- 172.21.15.129:0/2442883745 >> 172.21.15.129:0/2442883745 conn(0x7fa14805e870 msgr2=0x7fa14809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:33.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 Processor -- start 2019-10-13T05:21:33.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 -- start start 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa148133050 con 0x7fa1480650a0 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa148132250 con 0x7fa14813ac50 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14e4ed700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148135990 con 0x7fa14811ec80 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14d28b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa14813ac50 0x7fa14813d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58782/0 (socket says 172.21.15.129:58782) 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14d28b700 1 -- 172.21.15.129:0/4062297239 learned_addr learned my addr 172.21.15.129:0/4062297239 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433452204 0 0) 0x7fa148132250 con 0x7fa14813ac50 2019-10-13T05:21:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1280030a0 con 0x7fa14813ac50 2019-10-13T05:21:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652127075 0 0) 0x7fa148133050 con 0x7fa1480650a0 2019-10-13T05:21:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.613+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa148132250 con 0x7fa1480650a0 2019-10-13T05:21:33.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 -- 172.21.15.129:0/4062297239 >> v1:172.21.15.129:6789/0 conn(0x7fa14813ac50 legacy=0x7fa14813d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 -- 172.21.15.129:0/4062297239 >> v1:172.21.15.71:6789/0 conn(0x7fa1480650a0 legacy=0x7fa148135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 -- 172.21.15.129:0/4062297239 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1480a3dd0 con 0x7fa14811ec80 2019-10-13T05:21:33.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa13800c850 con 0x7fa14811ec80 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13800c470 con 0x7fa14811ec80 2019-10-13T05:21:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa138004420 con 0x7fa14811ec80 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148063850 con 0x7fa14811ec80 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14813e890 con 0x7fa14811ec80 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa138004730 con 0x7fa14811ec80 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 0x7fa12802a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fa1380345e0 con 0x7fa14811ec80 2019-10-13T05:21:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa147fff700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 0x7fa12802a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.617+0000 7fa147fff700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 0x7fa12802a4d0 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.621+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1481366d0 con 0x7fa14811ec80 2019-10-13T05:21:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.629+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.629+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa138025990 con 0x7fa14811ec80 2019-10-13T05:21:33.798 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.793+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1480555c0 con 0x7fa14811ec80 2019-10-13T05:21:33.800 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.793+0000 7fa14ca8a700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.800 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.793+0000 7fa1457fa700 1 -- 172.21.15.129:0/4062297239 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11826 (secure 0 0 0) 0x7fa13800b0d0 con 0x7fa14811ec80 2019-10-13T05:21:33.801 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:33.801 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":16,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:33.476444+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:33.803 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 msgr2=0x7fa12802a4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 0x7fa12802a4d0 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 msgr2=0x7fa148132b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7fa138000fa0 tx=0x7fa13800c3f0).stop 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 shutdown_connections 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa128027fd0 0x7fa12802a4d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.804 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 --2- 172.21.15.129:0/4062297239 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa14811ec80 0x7fa148132b00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.805 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 shutdown_connections 2019-10-13T05:21:33.805 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 wait complete. 2019-10-13T05:21:33.805 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.797+0000 7fa14e4ed700 1 -- 172.21.15.129:0/4062297239 >> 172.21.15.129:0/4062297239 conn(0x7fa1480a7a10 msgr2=0x7fa148122ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:33.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:31.757741+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T05:21:33.815 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:33.815 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T05:21:33.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 Processor -- start 2019-10-13T05:21:33.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 -- start start 2019-10-13T05:21:33.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7540a82d0 0x7ff7540a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:33.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff754144920 0x7ff754144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:33.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7540a8c40 con 0x7ff75409d160 2019-10-13T05:21:33.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7540a1300 con 0x7ff7540a82d0 2019-10-13T05:21:33.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff75b0c5700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540a15a0 con 0x7ff754144920 2019-10-13T05:21:33.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7540a82d0 0x7ff7540a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff759e63700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff754144920 0x7ff754144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7540a82d0 0x7ff7540a86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.129:42646/0 (socket says 172.21.15.129:42646) 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 -- 172.21.15.129:0/599260229 learned_addr learned my addr 172.21.15.129:0/599260229 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 -- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff754144920 msgr2=0x7ff754144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff754144920 0x7ff754144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 -- 172.21.15.129:0/599260229 >> v1:172.21.15.71:6789/0 conn(0x7ff75409d160 legacy=0x7ff7540a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 -- 172.21.15.129:0/599260229 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754145290 con 0x7ff7540a82d0 2019-10-13T05:21:33.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> v2:172.21.15.71:3300/0 conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff744008780 tx=0x7ff74c007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> v2:172.21.15.71:3300/0 conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff744008780 tx=0x7ff74c007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.901+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7ff744008780 tx=0x7ff74c007a90).ready entity=mon.0 client_cookie=cf2942e4ff2bea2e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff744008780 tx=0x7ff74c007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff744008780 tx=0x7ff74c007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff74bfff700 1 -- 172.21.15.129:0/599260229 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff744008c80 con 0x7ff7540a82d0 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff758e61700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff744008780 tx=0x7ff74c007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff74bfff700 1 -- 172.21.15.129:0/599260229 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7440088a0 con 0x7ff7540a82d0 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff74bfff700 1 -- 172.21.15.129:0/599260229 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7440119f0 con 0x7ff7540a82d0 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 msgr2=0x7ff7540a86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7ff744008780 tx=0x7ff74c007a90).stop 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- 172.21.15.129:0/599260229 shutdown_connections 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff754144920 0x7ff754144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/599260229 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7540a82d0 0x7ff7540a86f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- 172.21.15.129:0/599260229 shutdown_connections 2019-10-13T05:21:33.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- 172.21.15.129:0/599260229 wait complete. 2019-10-13T05:21:33.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- 172.21.15.129:0/599260229 >> 172.21.15.129:0/599260229 conn(0x7ff75405e870 msgr2=0x7ff75409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:33.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 Processor -- start 2019-10-13T05:21:33.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.905+0000 7ff75b0c5700 1 -- start start 2019-10-13T05:21:33.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff7540a2d20 0x7ff754143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:33.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff7541439e0 0x7ff754143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:33.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7540a4370 con 0x7ff754126260 2019-10-13T05:21:33.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7540a4890 con 0x7ff7541439e0 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540a4b30 con 0x7ff7540a2d20 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff7541439e0 0x7ff754143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff7541439e0 0x7ff754143e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40826/0 (socket says 172.21.15.129:40826) 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 -- 172.21.15.129:0/1838592131 learned_addr learned my addr 172.21.15.129:0/1838592131 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff758e61700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff7540a2d20 0x7ff754143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 -- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff7540a2d20 msgr2=0x7ff754143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff7540a2d20 0x7ff754143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 -- 172.21.15.129:0/1838592131 >> v1:172.21.15.71:6789/0 conn(0x7ff754126260 legacy=0x7ff754123aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 -- 172.21.15.129:0/1838592131 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540a3e60 con 0x7ff7541439e0 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> v2:172.21.15.129:3300/0 conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> v2:172.21.15.129:3300/0 conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff74c005c90 con 0x7ff7541439e0 2019-10-13T05:21:33.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7540a5070 con 0x7ff7541439e0 2019-10-13T05:21:33.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c0058b0 con 0x7ff7541439e0 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff74c011c70 con 0x7ff7541439e0 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7541242a0 con 0x7ff7541439e0 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff74c007180 con 0x7ff7541439e0 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 0x7ff734029e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:33.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff74c034a10 con 0x7ff7541439e0 2019-10-13T05:21:33.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.909+0000 7ff759662700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 0x7ff734029e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:33.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.913+0000 7ff759662700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 0x7ff734029e40 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:33.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.917+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7540555c0 con 0x7ff7541439e0 2019-10-13T05:21:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.925+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:33.925+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff74c00e160 con 0x7ff7541439e0 2019-10-13T05:21:34.086 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff7540a8470 con 0x7ff7541439e0 2019-10-13T05:21:34.087 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff759e63700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.087 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff749ffb700 1 -- 172.21.15.129:0/1838592131 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff74c007470 con 0x7ff7541439e0 2019-10-13T05:21:34.087 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 msgr2=0x7ff734029e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 0x7ff734029e40 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 msgr2=0x7ff754143e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7ff7540a4a50 tx=0x7ff74c003370).stop 2019-10-13T05:21:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 shutdown_connections 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff734027940 0x7ff734029e40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7541439e0 0x7ff754143e00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 --2- 172.21.15.129:0/1838592131 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff7540a2d20 0x7ff754143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 shutdown_connections 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 wait complete. 2019-10-13T05:21:34.091 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.081+0000 7ff75b0c5700 1 -- 172.21.15.129:0/1838592131 >> 172.21.15.129:0/1838592131 conn(0x7ff7540a1840 msgr2=0x7ff754122c90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:34.099 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:34.100 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2019-10-13T05:21:34.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 Processor -- start 2019-10-13T05:21:34.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- start start 2019-10-13T05:21:34.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbeb80a1c80 0x7fbeb80a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:34.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb8144a10 0x7fbeb8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:34.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb80a7a30 con 0x7fbeb80a7520 2019-10-13T05:21:34.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80a96b0 con 0x7fbeb80a1c80 2019-10-13T05:21:34.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80a9950 con 0x7fbeb8144a10 2019-10-13T05:21:34.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb8144a10 0x7fbeb8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb8144a10 0x7fbeb8144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40830/0 (socket says 172.21.15.129:40830) 2019-10-13T05:21:34.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 -- 172.21.15.129:0/3207708190 learned_addr learned my addr 172.21.15.129:0/3207708190 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:34.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebeb74700 1 --2- 172.21.15.129:0/3207708190 >> v2:172.21.15.71:3300/0 conn(0x7fbeb80a1c80 0x7fbeb80a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 -- 172.21.15.129:0/3207708190 >> v2:172.21.15.71:3300/0 conn(0x7fbeb80a1c80 msgr2=0x7fbeb80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> v2:172.21.15.71:3300/0 conn(0x7fbeb80a1c80 0x7fbeb80a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 -- 172.21.15.129:0/3207708190 >> v1:172.21.15.71:6789/0 conn(0x7fbeb80a7520 legacy=0x7fbeb80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 -- 172.21.15.129:0/3207708190 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb80a1940 con 0x7fbeb8144a10 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> v2:172.21.15.129:3300/0 conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> v2:172.21.15.129:3300/0 conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.202 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).ready entity=mon.1 client_cookie=f23b173241f756b4 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebf375700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebdb72700 1 -- 172.21.15.129:0/3207708190 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb0010cc0 con 0x7fbeb8144a10 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebdb72700 1 -- 172.21.15.129:0/3207708190 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb00108e0 con 0x7fbeb8144a10 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbebdb72700 1 -- 172.21.15.129:0/3207708190 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb00114a0 con 0x7fbeb8144a10 2019-10-13T05:21:34.203 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 msgr2=0x7fbeb8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7fbeb00032b0 tx=0x7fbeb4002ff0).stop 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- 172.21.15.129:0/3207708190 shutdown_connections 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 --2- 172.21.15.129:0/3207708190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb8144a10 0x7fbeb8144e30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 --2- 172.21.15.129:0/3207708190 >> v2:172.21.15.71:3300/0 conn(0x7fbeb80a1c80 0x7fbeb80a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.185+0000 7fbec05d7700 1 -- 172.21.15.129:0/3207708190 shutdown_connections 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- 172.21.15.129:0/3207708190 wait complete. 2019-10-13T05:21:34.204 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- 172.21.15.129:0/3207708190 >> 172.21.15.129:0/3207708190 conn(0x7fbeb805e870 msgr2=0x7fbeb809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:34.208 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 Processor -- start 2019-10-13T05:21:34.208 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- start start 2019-10-13T05:21:34.208 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb80aa030 0x7fbeb81124e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb80a66f0 con 0x7fbeb814d7d0 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb812a580 con 0x7fbeb8112a30 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbec05d7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80a5cb0 con 0x7fbeb80aa030 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb80aa030 0x7fbeb81124e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeb80aa030 0x7fbeb81124e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40834/0 (socket says 172.21.15.129:40834) 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 -- 172.21.15.129:0/264912736 learned_addr learned my addr 172.21.15.129:0/264912736 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 -- 172.21.15.129:0/264912736 >> v1:172.21.15.129:6789/0 conn(0x7fbeb8112a30 legacy=0x7fbeb8129d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 -- 172.21.15.129:0/264912736 >> v1:172.21.15.71:6789/0 conn(0x7fbeb814d7d0 legacy=0x7fbeb814dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 -- 172.21.15.129:0/264912736 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb814e3f0 con 0x7fbeb80aa030 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.189+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> v2:172.21.15.129:3300/0 conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> v2:172.21.15.129:3300/0 conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb4012770 con 0x7fbeb80aa030 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb4008ba0 con 0x7fbeb80aa030 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb4012d50 con 0x7fbeb80aa030 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb8132410 con 0x7fbeb80aa030 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb8132960 con 0x7fbeb80aa030 2019-10-13T05:21:34.211 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fbeb4005580 con 0x7fbeb80aa030 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 0x7fbea0029e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fbeb40350a0 con 0x7fbeb80aa030 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebe373700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 0x7fbea0029e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.193+0000 7fbebe373700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 0x7fbea0029e10 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.201+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb80aa450 con 0x7fbeb80aa030 2019-10-13T05:21:34.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.205+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.205+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fbeb4026430 con 0x7fbeb80aa030 2019-10-13T05:21:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.361+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbeb80555c0 con 0x7fbeb80aa030 2019-10-13T05:21:34.369 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.361+0000 7fbebeb74700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.369 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.361+0000 7fbeab7fe700 1 -- 172.21.15.129:0/264912736 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7fbeb40070f0 con 0x7fbeb80aa030 2019-10-13T05:21:34.370 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:34.370 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2019-10-13T05:21:34.370 INFO:teuthology.orchestra.run.smithi129.stdout: { 2019-10-13T05:21:34.370 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2019-10-13T05:21:34.370 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": "rbd" 2019-10-13T05:21:34.371 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T05:21:34.371 INFO:teuthology.orchestra.run.smithi129.stdout:] 2019-10-13T05:21:34.372 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 msgr2=0x7fbea0029e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.372 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 0x7fbea0029e10 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.372 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 msgr2=0x7fbeb81124e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.372 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7fbeb4002f30 tx=0x7fbeb4010c40).stop 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 shutdown_connections 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbea0027910 0x7fbea0029e10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 --2- 172.21.15.129:0/264912736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeb80aa030 0x7fbeb81124e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 shutdown_connections 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 wait complete. 2019-10-13T05:21:34.373 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.365+0000 7fbec05d7700 1 -- 172.21.15.129:0/264912736 >> 172.21.15.129:0/264912736 conn(0x7fbeb80a27f0 msgr2=0x7fbeb80a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:34.381 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2019-10-13T05:21:34.382 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:34.382 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2019-10-13T05:21:34.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 Processor -- start 2019-10-13T05:21:34.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 -- start start 2019-10-13T05:21:34.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc09d150 0x7f28fc0a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:34.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc0a86e0 0x7f28fc14eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0a0fc0 con 0x7f28fc0a82c0 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0a12f0 con 0x7f28fc09d150 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0a1590 con 0x7f28fc0a86e0 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc09d150 0x7f28fc0a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc09d150 0x7f28fc0a69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40842/0 (socket says 172.21.15.129:40842) 2019-10-13T05:21:34.486 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 -- 172.21.15.129:0/3420002199 learned_addr learned my addr 172.21.15.129:0/3420002199 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc0a86e0 0x7f28fc14eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 -- 172.21.15.129:0/3420002199 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc0a86e0 msgr2=0x7f28fc14eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc0a86e0 0x7f28fc14eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 -- 172.21.15.129:0/3420002199 >> v1:172.21.15.71:6789/0 conn(0x7f28fc0a82c0 legacy=0x7f28fc144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fa7fc700 1 -- 172.21.15.129:0/3420002199 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637533336 0 0) 0x7f28fc0a0fc0 con 0x7f28fc0a82c0 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 -- 172.21.15.129:0/3420002199 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0a8b00 con 0x7f28fc09d150 2019-10-13T05:21:34.487 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> v2:172.21.15.129:3300/0 conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.488 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> v2:172.21.15.129:3300/0 conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.489 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).ready entity=mon.1 client_cookie=b2cbfe229205e210 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fa7fc700 1 -- 172.21.15.129:0/3420002199 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28ec008c80 con 0x7f28fc09d150 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fa7fc700 1 -- 172.21.15.129:0/3420002199 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec0088a0 con 0x7f28fc09d150 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 -- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 msgr2=0x7f28fc0a69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.491 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f28fa7fc700 1 -- 172.21.15.129:0/3420002199 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28ec0119f0 con 0x7f28fc09d150 2019-10-13T05:21:34.491 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.469+0000 7f29016a9700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f28fc0a14b0 tx=0x7f28f4007710).stop 2019-10-13T05:21:34.491 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- 172.21.15.129:0/3420002199 shutdown_connections 2019-10-13T05:21:34.491 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc0a86e0 0x7f28fc14eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 --2- 172.21.15.129:0/3420002199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc09d150 0x7f28fc0a69e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- 172.21.15.129:0/3420002199 shutdown_connections 2019-10-13T05:21:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- 172.21.15.129:0/3420002199 wait complete. 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- 172.21.15.129:0/3420002199 >> 172.21.15.129:0/3420002199 conn(0x7f28fc05e860 msgr2=0x7f28fc09c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 Processor -- start 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- start start 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc115b10 0x7f28fc1164e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f28fc078340 0x7f28fc078760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc1176a0 0x7f28fc117ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc116a30 con 0x7f28fc078340 2019-10-13T05:21:34.494 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0a2510 con 0x7f28fc1176a0 2019-10-13T05:21:34.494 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f29016a9700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0a27b0 con 0x7f28fc115b10 2019-10-13T05:21:34.494 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f28fbfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc1176a0 0x7f28fc117ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.494 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f28fbfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28fc1176a0 0x7f28fc117ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40850/0 (socket says 172.21.15.129:40850) 2019-10-13T05:21:34.494 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f28fbfff700 1 -- 172.21.15.129:0/3040521772 learned_addr learned my addr 172.21.15.129:0/3040521772 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f28faffd700 1 --2- 172.21.15.129:0/3040521772 >> v2:172.21.15.71:3300/0 conn(0x7f28fc078340 0x7f28fc078760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.473+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc115b10 0x7f28fc1164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 -- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc115b10 msgr2=0x7f28fc1164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc115b10 0x7f28fc1164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 -- 172.21.15.129:0/3040521772 >> v2:172.21.15.71:3300/0 conn(0x7f28fc078340 msgr2=0x7f28fc078760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:34.495 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> v2:172.21.15.71:3300/0 conn(0x7f28fc078340 0x7f28fc078760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 -- 172.21.15.129:0/3040521772 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0a1ea0 con 0x7f28fc1176a0 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> v2:172.21.15.129:3300/0 conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> v2:172.21.15.129:3300/0 conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f4008bc0 con 0x7f28fc1176a0 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc0a2cf0 con 0x7f28fc1176a0 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f40087e0 con 0x7f28fc1176a0 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f4005720 con 0x7f28fc1176a0 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc12bac0 con 0x7f28fc1176a0 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f28f400b250 con 0x7f28fc1176a0 2019-10-13T05:21:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 0x7f28e0029df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f28f40347e0 con 0x7f28fc1176a0 2019-10-13T05:21:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 0x7f28e0029df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.477+0000 7f28fb7fe700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 0x7f28e0029df0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:34.499 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.485+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc0555b0 con 0x7f28fc1176a0 2019-10-13T05:21:34.503 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.497+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:34.503 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.497+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f28f4005bf0 con 0x7f28fc1176a0 2019-10-13T05:21:34.657 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:34.649+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f28fc14e840 con 0x7f28fc1176a0 2019-10-13T05:21:35.523 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.517+0000 7f28fbfff700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.523 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.517+0000 7f28f3fff700 1 -- 172.21.15.129:0/3040521772 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f28f400b540 con 0x7f28fc1176a0 2019-10-13T05:21:35.571 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2019-10-13T05:21:35.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 msgr2=0x7f28e0029df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 0x7f28e0029df0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 msgr2=0x7f28fc117ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f28fc0a2970 tx=0x7f28e4001540).stop 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 shutdown_connections 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1176a0 0x7f28fc117ac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f28fc115b10 0x7f28fc1164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f28e00278f0 0x7f28e0029df0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 --2- 172.21.15.129:0/3040521772 >> v2:172.21.15.71:3300/0 conn(0x7f28fc078340 0x7f28fc078760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 shutdown_connections 2019-10-13T05:21:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 wait complete. 2019-10-13T05:21:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.569+0000 7f29016a9700 1 -- 172.21.15.129:0/3040521772 >> 172.21.15.129:0/3040521772 conn(0x7f28fc09f730 msgr2=0x7f28fc07de70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:35.591 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:35.591 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2019-10-13T05:21:35.708 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 Processor -- start 2019-10-13T05:21:35.709 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 -- start start 2019-10-13T05:21:35.709 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35dc09d160 0x7f35dc0a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:35.709 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f35dc0a82d0 0x7f35dc0a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:35.709 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc0a8c40 con 0x7f35dc144920 2019-10-13T05:21:35.710 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc0a1300 con 0x7f35dc0a82d0 2019-10-13T05:21:35.710 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e252c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc0a15a0 con 0x7f35dc09d160 2019-10-13T05:21:35.710 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e0ac9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35dc09d160 0x7f35dc0a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:35.710 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e12ca700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f35dc144920 0x7f35dc0a9560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58830/0 (socket says 172.21.15.129:58830) 2019-10-13T05:21:35.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35e12ca700 1 -- 172.21.15.129:0/1618445910 learned_addr learned my addr 172.21.15.129:0/1618445910 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:35.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.689+0000 7f35dbfff700 1 --2- 172.21.15.129:0/1618445910 >> v2:172.21.15.71:3300/0 conn(0x7f35dc0a82d0 0x7f35dc0a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:35.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1618445910 >> v2:172.21.15.71:3300/0 conn(0x7f35dc0a82d0 msgr2=0x7f35dc0a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:35.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> v2:172.21.15.71:3300/0 conn(0x7f35dc0a82d0 0x7f35dc0a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:35.712 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1618445910 >> v1:172.21.15.129:6789/0 conn(0x7f35dc144920 legacy=0x7f35dc0a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.712 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1618445910 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0a0fd0 con 0x7f35dc09d160 2019-10-13T05:21:35.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35db7fe700 1 -- 172.21.15.129:0/1618445910 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635378675 0 0) 0x7f35dc0a8c40 con 0x7f35dc144920 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> v2:172.21.15.129:3300/0 conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35d0002f30 tx=0x7f35c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> v2:172.21.15.129:3300/0 conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35d0002f30 tx=0x7f35c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f35d0002f30 tx=0x7f35c8007710).ready entity=mon.1 client_cookie=45096df6c8003a2a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f35d0002f30 tx=0x7f35c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35db7fe700 1 -- 172.21.15.129:0/1618445910 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d0008ba0 con 0x7f35dc09d160 2019-10-13T05:21:35.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f35d0002f30 tx=0x7f35c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f35d0002f30 tx=0x7f35c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35db7fe700 1 -- 172.21.15.129:0/1618445910 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d00087c0 con 0x7f35dc09d160 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35db7fe700 1 -- 172.21.15.129:0/1618445910 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d0011700 con 0x7f35dc09d160 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 -- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 msgr2=0x7f35dc0a69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f35d0002f30 tx=0x7f35c8007710).stop 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 -- 172.21.15.129:0/1618445910 shutdown_connections 2019-10-13T05:21:35.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 --2- 172.21.15.129:0/1618445910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc09d160 0x7f35dc0a69f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 --2- 172.21.15.129:0/1618445910 >> v2:172.21.15.71:3300/0 conn(0x7f35dc0a82d0 0x7f35dc0a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 -- 172.21.15.129:0/1618445910 shutdown_connections 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 -- 172.21.15.129:0/1618445910 wait complete. 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.693+0000 7f35e252c700 1 -- 172.21.15.129:0/1618445910 >> 172.21.15.129:0/1618445910 conn(0x7f35dc05e870 msgr2=0x7f35dc09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 Processor -- start 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- start start 2019-10-13T05:21:35.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35dc125750 0x7f35dc123710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:35.722 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc133eb0 con 0x7f35dc0a44b0 2019-10-13T05:21:35.722 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc122640 con 0x7f35dc0a4090 2019-10-13T05:21:35.722 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc123f20 con 0x7f35dc125750 2019-10-13T05:21:35.722 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35dc125750 0x7f35dc123710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35dbfff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f35dc0a4090 0x7f35dc1336b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58834/0 (socket says 172.21.15.129:58834) 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35dbfff700 1 -- 172.21.15.129:0/1604496321 learned_addr learned my addr 172.21.15.129:0/1604496321 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35dc125750 0x7f35dc123710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40862/0 (socket says 172.21.15.129:40862) 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1604496321 >> v1:172.21.15.129:6789/0 conn(0x7f35dc0a4090 legacy=0x7f35dc1336b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273998910 0 0) 0x7f35d4001340 con 0x7f35dc0a44b0 2019-10-13T05:21:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1604496321 >> v1:172.21.15.71:6789/0 conn(0x7f35dc0a44b0 legacy=0x7f35dc13a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 -- 172.21.15.129:0/1604496321 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc123c60 con 0x7f35dc125750 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> v2:172.21.15.129:3300/0 conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> v2:172.21.15.129:3300/0 conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d0007340 con 0x7f35dc125750 2019-10-13T05:21:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d0008da0 con 0x7f35dc125750 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc133030 con 0x7f35dc125750 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d0005910 con 0x7f35dc125750 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.697+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc13bc90 con 0x7f35dc125750 2019-10-13T05:21:35.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f35d0005bb0 con 0x7f35dc125750 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35d97fa700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 0x7f35bc029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f35d0078a70 con 0x7f35dc125750 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35dbfff700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 0x7f35bc029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.701+0000 7f35dbfff700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 0x7f35bc029c90 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.705+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc125b70 con 0x7f35dc125750 2019-10-13T05:21:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.713+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.713+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f35dc123c60 con 0x7f35dc125750 2019-10-13T05:21:35.875 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:35.869+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f35dc0555c0 con 0x7f35dc125750 2019-10-13T05:21:36.529 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.521+0000 7f35e0ac9700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.530 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.521+0000 7f35d97fa700 1 -- 172.21.15.129:0/1604496321 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v19) v1 ==== 125+0+0 (secure 0 0 0) 0x7f35d00585a0 con 0x7f35dc125750 2019-10-13T05:21:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 msgr2=0x7f35bc029c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 0x7f35bc029c90 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 msgr2=0x7f35dc123710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f35dc0652c0 tx=0x7f35c8007a40).stop 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 shutdown_connections 2019-10-13T05:21:36.544 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f35bc027790 0x7f35bc029c90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 --2- 172.21.15.129:0/1604496321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35dc125750 0x7f35dc123710 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 shutdown_connections 2019-10-13T05:21:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 wait complete. 2019-10-13T05:21:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.537+0000 7f35e252c700 1 -- 172.21.15.129:0/1604496321 >> 172.21.15.129:0/1604496321 conn(0x7f35dc0a75b0 msgr2=0x7f35dc0a93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:36.554 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:36.554 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2019-10-13T05:21:36.659 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 Processor -- start 2019-10-13T05:21:36.659 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 -- start start 2019-10-13T05:21:36.659 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee009d160 0x7faee00a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:36.659 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faee00a86f0 0x7faee014eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00a0fd0 con 0x7faee00a82d0 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00a1300 con 0x7faee00a86f0 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00a15a0 con 0x7faee009d160 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee009d160 0x7faee00a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee009d160 0x7faee00a69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40870/0 (socket says 172.21.15.129:40870) 2019-10-13T05:21:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 -- 172.21.15.129:0/2130679883 learned_addr learned my addr 172.21.15.129:0/2130679883 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee5235700 1 --2- 172.21.15.129:0/2130679883 >> v2:172.21.15.71:3300/0 conn(0x7faee00a86f0 0x7faee014eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 -- 172.21.15.129:0/2130679883 >> v2:172.21.15.71:3300/0 conn(0x7faee00a86f0 msgr2=0x7faee014eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> v2:172.21.15.71:3300/0 conn(0x7faee00a86f0 0x7faee014eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 -- 172.21.15.129:0/2130679883 >> v1:172.21.15.71:6789/0 conn(0x7faee00a82d0 legacy=0x7faee0144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 -- 172.21.15.129:0/2130679883 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee00a8b10 con 0x7faee009d160 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> v2:172.21.15.129:3300/0 conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faec8002f30 tx=0x7faed8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> v2:172.21.15.129:3300/0 conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faec8002f30 tx=0x7faed8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7faec8002f30 tx=0x7faed8007710).ready entity=mon.1 client_cookie=defa9dab3323cb49 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7faec8002f30 tx=0x7faed8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faedf7fe700 1 -- 172.21.15.129:0/2130679883 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec8008c80 con 0x7faee009d160 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7faec8002f30 tx=0x7faed8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee4a34700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7faec8002f30 tx=0x7faed8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faedf7fe700 1 -- 172.21.15.129:0/2130679883 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec80088a0 con 0x7faee009d160 2019-10-13T05:21:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faedf7fe700 1 -- 172.21.15.129:0/2130679883 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec80119b0 con 0x7faee009d160 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 -- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 msgr2=0x7faee00a69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.649+0000 7faee6497700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7faec8002f30 tx=0x7faed8007710).stop 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- 172.21.15.129:0/2130679883 shutdown_connections 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 --2- 172.21.15.129:0/2130679883 >> v2:172.21.15.71:3300/0 conn(0x7faee00a86f0 0x7faee014eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 --2- 172.21.15.129:0/2130679883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee009d160 0x7faee00a69f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- 172.21.15.129:0/2130679883 shutdown_connections 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- 172.21.15.129:0/2130679883 wait complete. 2019-10-13T05:21:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- 172.21.15.129:0/2130679883 >> 172.21.15.129:0/2130679883 conn(0x7faee005e870 msgr2=0x7faee009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 Processor -- start 2019-10-13T05:21:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- start start 2019-10-13T05:21:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee00a2000 0x7faee00a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faee0073160 0x7faee0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee0065020 con 0x7faee0123610 2019-10-13T05:21:36.668 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00a3e60 con 0x7faee00a2000 2019-10-13T05:21:36.668 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee6497700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee01244b0 con 0x7faee0073160 2019-10-13T05:21:36.668 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faee4a34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee00a2000 0x7faee00a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.668 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faee00a2000 0x7faee00a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40872/0 (socket says 172.21.15.129:40872) 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.653+0000 7faedffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7faee0123610 0x7faee0143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58844/0 (socket says 172.21.15.129:58844) 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 -- 172.21.15.129:0/4246606089 learned_addr learned my addr 172.21.15.129:0/4246606089 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee5235700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faee0073160 0x7faee0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 -- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faee0073160 msgr2=0x7faee0075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faee0073160 0x7faee0075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361499800 0 0) 0x7faee00a0fd0 con 0x7faee0123610 2019-10-13T05:21:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 -- 172.21.15.129:0/4246606089 >> v1:172.21.15.129:6789/0 conn(0x7faee0123610 legacy=0x7faee0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 -- 172.21.15.129:0/4246606089 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0123a30 con 0x7faee00a2000 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> v2:172.21.15.129:3300/0 conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> v2:172.21.15.129:3300/0 conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec8007160 con 0x7faee00a2000 2019-10-13T05:21:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8016060 con 0x7faee00a2000 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee0124020 con 0x7faee00a2000 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec8005740 con 0x7faee00a2000 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee0075ae0 con 0x7faee00a2000 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7faec80059e0 con 0x7faee00a2000 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 0x7faeb8029ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5557+0+0 (secure 0 0 0) 0x7faec8034d30 con 0x7faee00a2000 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.657+0000 7faedffff700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 0x7faeb8029ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.661+0000 7faedffff700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 0x7faeb8029ca0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.665+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee00555c0 con 0x7faee00a2000 2019-10-13T05:21:36.675 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.669+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.675 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.669+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7faec801d840 con 0x7faee00a2000 2019-10-13T05:21:36.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.825+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7faee0067b90 con 0x7faee00a2000 2019-10-13T05:21:36.836 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:21:36.827+0000 7f0e653fd700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-10-13T05:21:36.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.841+0000 7faee4a34700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.841+0000 7faedd7fa700 1 -- 172.21.15.129:0/4246606089 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 172+0+0 (secure 0 0 0) 0x7faec801daa0 con 0x7faee00a2000 2019-10-13T05:21:36.848 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 2 and data pool 3 2019-10-13T05:21:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 msgr2=0x7faeb8029ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 0x7faeb8029ca0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 msgr2=0x7faee00a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7faee00a4020 tx=0x7faed8007550).stop 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 shutdown_connections 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faeb80277a0 0x7faeb8029ca0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faee0073160 0x7faee0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 --2- 172.21.15.129:0/4246606089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00a2000 0x7faee00a5200 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 shutdown_connections 2019-10-13T05:21:36.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 wait complete. 2019-10-13T05:21:36.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.845+0000 7faee6497700 1 -- 172.21.15.129:0/4246606089 >> 172.21.15.129:0/4246606089 conn(0x7faee00a8370 msgr2=0x7faee009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:36.861 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:36.861 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:36.953 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.941+0000 7fa9fe3bb700 1 Processor -- start 2019-10-13T05:21:36.953 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.941+0000 7fa9fe3bb700 1 -- start start 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa9f80a7520 0x7fa9f80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f8144a10 0x7fa9f8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80a7a30 con 0x7fa9f80a1c80 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80a96b0 con 0x7fa9f80a7520 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80a9950 con 0x7fa9f8144a10 2019-10-13T05:21:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f8144a10 0x7fa9f8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f8144a10 0x7fa9f8144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40884/0 (socket says 172.21.15.129:40884) 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 -- 172.21.15.129:0/1179689615 learned_addr learned my addr 172.21.15.129:0/1179689615 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fc958700 1 --1- 172.21.15.129:0/1179689615 >> v1:172.21.15.129:6789/0 conn(0x7fa9f80a1c80 0x7fa9f80a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58850/0 (socket says 172.21.15.129:58850) 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9f7fff700 1 --2- 172.21.15.129:0/1179689615 >> v2:172.21.15.71:3300/0 conn(0x7fa9f80a7520 0x7fa9f80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 -- 172.21.15.129:0/1179689615 >> v2:172.21.15.71:3300/0 conn(0x7fa9f80a7520 msgr2=0x7fa9f80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> v2:172.21.15.71:3300/0 conn(0x7fa9f80a7520 0x7fa9f80a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 -- 172.21.15.129:0/1179689615 >> v1:172.21.15.129:6789/0 conn(0x7fa9f80a1c80 legacy=0x7fa9f80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 -- 172.21.15.129:0/1179689615 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80aa3e0 con 0x7fa9f8144a10 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> v2:172.21.15.129:3300/0 conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> v2:172.21.15.129:3300/0 conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).ready entity=mon.1 client_cookie=9f64fa0ebf286461 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fd159700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9f77fe700 1 -- 172.21.15.129:0/1179689615 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9f0010cc0 con 0x7fa9f8144a10 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9f77fe700 1 -- 172.21.15.129:0/1179689615 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f00108e0 con 0x7fa9f8144a10 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 msgr2=0x7fa9f8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9f77fe700 1 -- 172.21.15.129:0/1179689615 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9f00114a0 con 0x7fa9f8144a10 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fa9f00032b0 tx=0x7fa9ec002ff0).stop 2019-10-13T05:21:36.957 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/1179689615 shutdown_connections 2019-10-13T05:21:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/1179689615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f8144a10 0x7fa9f8144e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/1179689615 >> v2:172.21.15.71:3300/0 conn(0x7fa9f80a7520 0x7fa9f80a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/1179689615 shutdown_connections 2019-10-13T05:21:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/1179689615 wait complete. 2019-10-13T05:21:36.959 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.945+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/1179689615 >> 172.21.15.129:0/1179689615 conn(0x7fa9f805e870 msgr2=0x7fa9f809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 Processor -- start 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- start start 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa9f8076870 0x7fa9f809f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f809f970 0x7fa9f809fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80788f0 con 0x7fa9f80a05a0 2019-10-13T05:21:36.962 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f8078110 con 0x7fa9f809f970 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80783b0 con 0x7fa9f8076870 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f809f970 0x7fa9f809fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9f809f970 0x7fa9f809fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40890/0 (socket says 172.21.15.129:40890) 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 -- 172.21.15.129:0/2884011375 learned_addr learned my addr 172.21.15.129:0/2884011375 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9f7fff700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa9f8076870 0x7fa9f809f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 -- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa9f8076870 msgr2=0x7fa9f809f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.963 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa9f8076870 0x7fa9f809f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 -- 172.21.15.129:0/2884011375 >> v1:172.21.15.71:6789/0 conn(0x7fa9f80a05a0 legacy=0x7fa9f8123a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 -- 172.21.15.129:0/2884011375 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80774d0 con 0x7fa9f809f970 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> v2:172.21.15.129:3300/0 conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> v2:172.21.15.129:3300/0 conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9f0005aa0 con 0x7fa9f809f970 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f00110c0 con 0x7fa9f809f970 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f8076c90 con 0x7fa9f809f970 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9f000f680 con 0x7fa9f809f970 2019-10-13T05:21:36.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f8124200 con 0x7fa9f809f970 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.949+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa9f000f990 con 0x7fa9f809f970 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9f57fa700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 0x7fa9d8029f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fa9f0034ec0 con 0x7fa9f809f970 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9fc958700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 0x7fa9d8029f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:36.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.953+0000 7fa9fc958700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 0x7fa9d8029f10 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:36.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.957+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f80555c0 con 0x7fa9f809f970 2019-10-13T05:21:36.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.965+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:36.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:36.965+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa9f001db10 con 0x7fa9f809f970 2019-10-13T05:21:37.128 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.121+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9f8067b90 con 0x7fa9f809f970 2019-10-13T05:21:37.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.121+0000 7fa9fd159700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.129 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.121+0000 7fa9f57fa700 1 -- 172.21.15.129:0/2884011375 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14685 (secure 0 0 0) 0x7fa9f001dd70 con 0x7fa9f809f970 2019-10-13T05:21:37.131 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:37.131 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:36.832609+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 msgr2=0x7fa9d8029f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 0x7fa9d8029f10 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 msgr2=0x7fa9f809fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7fa9f80764b0 tx=0x7fa9ec005530).stop 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 shutdown_connections 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa9d8027a10 0x7fa9d8029f10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9f809f970 0x7fa9f809fd90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 --2- 172.21.15.129:0/2884011375 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa9f8076870 0x7fa9f809f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 shutdown_connections 2019-10-13T05:21:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 wait complete. 2019-10-13T05:21:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.125+0000 7fa9fe3bb700 1 -- 172.21.15.129:0/2884011375 >> 172.21.15.129:0/2884011375 conn(0x7fa9f80a9130 msgr2=0x7fa9f8142980 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:37.143 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:37.143 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 Processor -- start 2019-10-13T05:21:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 -- start start 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc388144a10 0x7fc388144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3880a7a30 con 0x7fc3880a1c80 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3880a96b0 con 0x7fc388144a10 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.233+0000 7fc38d9d8700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3880a9950 con 0x7fc3880a7520 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.129:43794/0 (socket says 172.21.15.129:43794) 2019-10-13T05:21:37.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 -- 172.21.15.129:0/201039372 learned_addr learned my addr 172.21.15.129:0/201039372 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:37.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc387fff700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc388144a10 0x7fc388144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 -- 172.21.15.129:0/201039372 >> v2:172.21.15.71:3300/0 conn(0x7fc388144a10 msgr2=0x7fc388144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:37.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> v2:172.21.15.71:3300/0 conn(0x7fc388144a10 0x7fc388144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 -- 172.21.15.129:0/201039372 >> v1:172.21.15.71:6789/0 conn(0x7fc3880a1c80 legacy=0x7fc3880a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 -- 172.21.15.129:0/201039372 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880aa3e0 con 0x7fc3880a7520 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc378002f30 tx=0x7fc380007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc378002f30 tx=0x7fc380007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fc378002f30 tx=0x7fc380007710).ready entity=mon.2 client_cookie=1a931d5a768c2892 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fc378002f30 tx=0x7fc380007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc3867fc700 1 -- 172.21.15.129:0/201039372 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc378008c80 con 0x7fc3880a7520 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fc378002f30 tx=0x7fc380007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc386ffd700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fc378002f30 tx=0x7fc380007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc3867fc700 1 -- 172.21.15.129:0/201039372 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3780088a0 con 0x7fc3880a7520 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 -- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 msgr2=0x7fc3880a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc3867fc700 1 -- 172.21.15.129:0/201039372 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3780119b0 con 0x7fc3880a7520 2019-10-13T05:21:37.249 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fc378002f30 tx=0x7fc380007710).stop 2019-10-13T05:21:37.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 -- 172.21.15.129:0/201039372 shutdown_connections 2019-10-13T05:21:37.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/201039372 >> v2:172.21.15.71:3300/0 conn(0x7fc388144a10 0x7fc388144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:37.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/201039372 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc3880a7520 0x7fc3880a9e90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 -- 172.21.15.129:0/201039372 shutdown_connections 2019-10-13T05:21:37.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 -- 172.21.15.129:0/201039372 wait complete. 2019-10-13T05:21:37.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.237+0000 7fc38d9d8700 1 -- 172.21.15.129:0/201039372 >> 172.21.15.129:0/201039372 conn(0x7fc38805e870 msgr2=0x7fc38809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:37.255 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 Processor -- start 2019-10-13T05:21:37.255 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 -- start start 2019-10-13T05:21:37.255 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc3880a2410 0x7fc38805f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc388060170 0x7fc38807b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388111bf0 con 0x7fc38807be50 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3880a0360 con 0x7fc3880a2410 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3880a0600 con 0x7fc388060170 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc388060170 0x7fc38807b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc388060170 0x7fc38807b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40902/0 (socket says 172.21.15.129:40902) 2019-10-13T05:21:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc387fff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc38807be50 0x7fc3881113f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58872/0 (socket says 172.21.15.129:58872) 2019-10-13T05:21:37.257 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 -- 172.21.15.129:0/1976361866 learned_addr learned my addr 172.21.15.129:0/1976361866 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:37.257 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc3877fe700 1 --2- 172.21.15.129:0/1976361866 >> v2:172.21.15.71:3300/0 conn(0x7fc3880a2410 0x7fc38805f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.257 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 -- 172.21.15.129:0/1976361866 >> v2:172.21.15.71:3300/0 conn(0x7fc3880a2410 msgr2=0x7fc38805f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.257 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> v2:172.21.15.71:3300/0 conn(0x7fc3880a2410 0x7fc38805f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.257 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 -- 172.21.15.129:0/1976361866 >> v1:172.21.15.129:6789/0 conn(0x7fc38807be50 legacy=0x7fc3881113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 -- 172.21.15.129:0/1976361866 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388111e20 con 0x7fc388060170 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> v2:172.21.15.129:3300/0 conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> v2:172.21.15.129:3300/0 conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc378007160 con 0x7fc388060170 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc378016060 con 0x7fc388060170 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38809fd80 con 0x7fc388060170 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.241+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc378005720 con 0x7fc388060170 2019-10-13T05:21:37.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3881317c0 con 0x7fc388060170 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc3780059f0 con 0x7fc388060170 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc37ffff700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 0x7fc36c029d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fc378034c00 con 0x7fc388060170 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc3877fe700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 0x7fc36c029d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.245+0000 7fc3877fe700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 0x7fc36c029d90 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.249+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3880555c0 con 0x7fc388060170 2019-10-13T05:21:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.257+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.257+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc388111e20 con 0x7fc388060170 2019-10-13T05:21:37.377 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:37.377 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:37.381 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:37.381 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:37.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.429+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38807db10 con 0x7fc388060170 2019-10-13T05:21:37.437 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.429+0000 7fc386ffd700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.437 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.429+0000 7fc37ffff700 1 -- 172.21.15.129:0/1976361866 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14685 (secure 0 0 0) 0x7fc37801d610 con 0x7fc388060170 2019-10-13T05:21:37.438 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:37.439 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:36.832609+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:37.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 msgr2=0x7fc36c029d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 0x7fc36c029d90 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 msgr2=0x7fc38807b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fc3880615c0 tx=0x7fc3700015d0).stop 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 shutdown_connections 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc36c027890 0x7fc36c029d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/1976361866 >> v2:172.21.15.71:3300/0 conn(0x7fc3880a2410 0x7fc38805f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 --2- 172.21.15.129:0/1976361866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc388060170 0x7fc38807b900 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 shutdown_connections 2019-10-13T05:21:37.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 wait complete. 2019-10-13T05:21:37.443 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.433+0000 7fc38d9d8700 1 -- 172.21.15.129:0/1976361866 >> 172.21.15.129:0/1976361866 conn(0x7fc38809c9a0 msgr2=0x7fc3880a60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:37.451 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:37.451 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2019-10-13T05:21:37.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 Processor -- start 2019-10-13T05:21:37.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 -- start start 2019-10-13T05:21:37.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcb7c09d160 0x7fcb7c0a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:37.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0a8c40 con 0x7fcb7c144920 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0a1300 con 0x7fcb7c09d160 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0a15a0 con 0x7fcb7c0a82d0 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7bfff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fcb7c144920 0x7fcb7c0a9560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58882/0 (socket says 172.21.15.129:58882) 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7bfff700 1 -- 172.21.15.129:0/2253869015 learned_addr learned my addr 172.21.15.129:0/2253869015 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7b7fe700 1 --2- 172.21.15.129:0/2253869015 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c09d160 0x7fcb7c0a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7a7fc700 1 -- 172.21.15.129:0/2253869015 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081044301 0 0) 0x7fcb7c0a8c40 con 0x7fcb7c144920 2019-10-13T05:21:37.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7a7fc700 1 -- 172.21.15.129:0/2253869015 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb680030a0 con 0x7fcb7c144920 2019-10-13T05:21:37.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 -- 172.21.15.129:0/2253869015 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c09d160 msgr2=0x7fcb7c0a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:37.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c09d160 0x7fcb7c0a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 -- 172.21.15.129:0/2253869015 >> v1:172.21.15.129:6789/0 conn(0x7fcb7c144920 legacy=0x7fcb7c0a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 -- 172.21.15.129:0/2253869015 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0a0fd0 con 0x7fcb7c0a82d0 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcb6c002f30 tx=0x7fcb64007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcb6c002f30 tx=0x7fcb64007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fcb6c002f30 tx=0x7fcb64007710).ready entity=mon.2 client_cookie=a5481988abeaf1af server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fcb6c002f30 tx=0x7fcb64007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fcb6c002f30 tx=0x7fcb64007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7a7fc700 1 -- 172.21.15.129:0/2253869015 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb6c008ba0 con 0x7fcb7c0a82d0 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7affd700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fcb6c002f30 tx=0x7fcb64007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7a7fc700 1 -- 172.21.15.129:0/2253869015 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c0087c0 con 0x7fcb7c0a82d0 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb7a7fc700 1 -- 172.21.15.129:0/2253869015 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb6c011700 con 0x7fcb7c0a82d0 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 -- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 msgr2=0x7fcb7c0a86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.617+0000 7fcb81375700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fcb6c002f30 tx=0x7fcb64007710).stop 2019-10-13T05:21:37.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 -- 172.21.15.129:0/2253869015 shutdown_connections 2019-10-13T05:21:37.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 --2- 172.21.15.129:0/2253869015 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c09d160 0x7fcb7c0a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:37.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 --2- 172.21.15.129:0/2253869015 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c0a82d0 0x7fcb7c0a86f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 -- 172.21.15.129:0/2253869015 shutdown_connections 2019-10-13T05:21:37.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 -- 172.21.15.129:0/2253869015 wait complete. 2019-10-13T05:21:37.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 -- 172.21.15.129:0/2253869015 >> 172.21.15.129:0/2253869015 conn(0x7fcb7c05e870 msgr2=0x7fcb7c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:37.635 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 Processor -- start 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.621+0000 7fcb81375700 1 -- start start 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcb7c07ae50 0x7fcb7c07b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c11d800 con 0x7fcb7c0a96a0 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c11f110 con 0x7fcb7c07ae50 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0a3e60 con 0x7fcb7c07b7c0 2019-10-13T05:21:37.636 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7bfff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcb7c07ae50 0x7fcb7c07b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7bfff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.129:43816/0 (socket says 172.21.15.129:43816) 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7bfff700 1 -- 172.21.15.129:0/3716961861 learned_addr learned my addr 172.21.15.129:0/3716961861 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468840540 0 0) 0x7fcb7c11d800 con 0x7fcb7c0a96a0 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb600030a0 con 0x7fcb7c0a96a0 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 -- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 msgr2=0x7fcb7c13ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 -- 172.21.15.129:0/3716961861 >> v1:172.21.15.71:6789/0 conn(0x7fcb7c0a96a0 legacy=0x7fcb7c11d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 -- 172.21.15.129:0/3716961861 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c124640 con 0x7fcb7c07ae50 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7bfff700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> v2:172.21.15.71:3300/0 conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.638 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb6c007160 con 0x7fcb7c07ae50 2019-10-13T05:21:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c008da0 con 0x7fcb7c07ae50 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c07bbe0 con 0x7fcb7c07ae50 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb6c005530 con 0x7fcb7c07ae50 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c11f3b0 con 0x7fcb7c07ae50 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fcb6c005840 con 0x7fcb7c07ae50 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 0x7fcb6002a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.625+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fcb6c034510 con 0x7fcb7c07ae50 2019-10-13T05:21:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.629+0000 7fcb7b7fe700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 0x7fcb6002a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.629+0000 7fcb7b7fe700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 0x7fcb6002a2e0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:37.642 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.633+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c11ef00 con 0x7fcb7c07ae50 2019-10-13T05:21:37.648 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.641+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:37.649 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.641+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fcb6c01d360 con 0x7fcb7c07ae50 2019-10-13T05:21:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:37.801+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fcb7c0555c0 con 0x7fcb7c07ae50 2019-10-13T05:21:38.552 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.545+0000 7fcb7affd700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.552 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.545+0000 7fcb73fff700 1 -- 172.21.15.129:0/3716961861 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcb6c01cf00 con 0x7fcb7c07ae50 2019-10-13T05:21:38.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 msgr2=0x7fcb6002a2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 0x7fcb6002a2e0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 msgr2=0x7fcb7c07b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fcb7c0aa030 tx=0x7fcb74001840).stop 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 shutdown_connections 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fcb7c07b7c0 0x7fcb7c13ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fcb60027de0 0x7fcb6002a2e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 --2- 172.21.15.129:0/3716961861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb7c07ae50 0x7fcb7c07b270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 shutdown_connections 2019-10-13T05:21:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 wait complete. 2019-10-13T05:21:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.569+0000 7fcb81375700 1 -- 172.21.15.129:0/3716961861 >> 172.21.15.129:0/3716961861 conn(0x7fcb7c05e770 msgr2=0x7fcb7c07db40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:38.589 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:38.589 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T05:21:38.694 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.681+0000 7fe000714700 1 Processor -- start 2019-10-13T05:21:38.694 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.681+0000 7fe000714700 1 -- start start 2019-10-13T05:21:38.694 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff80a1c80 0x7fdff80a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 0x7fdff8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff80a7a30 con 0x7fdff80a7520 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff80a96b0 con 0x7fdff8144a10 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff80a9950 con 0x7fdff80a1c80 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff80a1c80 0x7fdff80a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff80a1c80 0x7fdff80a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40922/0 (socket says 172.21.15.129:40922) 2019-10-13T05:21:38.696 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 -- 172.21.15.129:0/3768460280 learned_addr learned my addr 172.21.15.129:0/3768460280 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:38.696 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdfff4b2700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 0x7fdff8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.696 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 -- 172.21.15.129:0/3768460280 >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 msgr2=0x7fdff8144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:38.696 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 0x7fdff8144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:38.696 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 -- 172.21.15.129:0/3768460280 >> v1:172.21.15.129:6789/0 conn(0x7fdff80a7520 legacy=0x7fdff80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 -- 172.21.15.129:0/3768460280 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff80a1940 con 0x7fdff80a1c80 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdfff4b2700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 0x7fdff8144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.129:3300/0 conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdff4002f30 tx=0x7fdff0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.129:3300/0 conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdff4002f30 tx=0x7fdff0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7fdff4002f30 tx=0x7fdff0007e40).ready entity=mon.1 client_cookie=6343c6b8e04988e5 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fdff4002f30 tx=0x7fdff0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.697 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fdff4002f30 tx=0x7fdff0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffdcaf700 1 -- 172.21.15.129:0/3768460280 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff4008c80 con 0x7fdff80a1c80 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fdff4002f30 tx=0x7fdff0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffdcaf700 1 -- 172.21.15.129:0/3768460280 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff40088a0 con 0x7fdff80a1c80 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fdffdcaf700 1 -- 172.21.15.129:0/3768460280 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff4011750 con 0x7fdff80a1c80 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 msgr2=0x7fdff80a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7fdff4002f30 tx=0x7fdff0007e40).stop 2019-10-13T05:21:38.698 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- 172.21.15.129:0/3768460280 shutdown_connections 2019-10-13T05:21:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 --2- 172.21.15.129:0/3768460280 >> v2:172.21.15.71:3300/0 conn(0x7fdff8144a10 0x7fdff8144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 --2- 172.21.15.129:0/3768460280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff80a1c80 0x7fdff80a13f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.685+0000 7fe000714700 1 -- 172.21.15.129:0/3768460280 shutdown_connections 2019-10-13T05:21:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- 172.21.15.129:0/3768460280 wait complete. 2019-10-13T05:21:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- 172.21.15.129:0/3768460280 >> 172.21.15.129:0/3768460280 conn(0x7fdff805e870 msgr2=0x7fdff809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:38.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 Processor -- start 2019-10-13T05:21:38.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- start start 2019-10-13T05:21:38.711 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdff807bc50 0x7fdff814df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff809e230 0x7fdff809e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 0x7fdff80781b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff80a6a00 con 0x7fdff8077d90 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff80754c0 con 0x7fdff809e230 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fe000714700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff809df90 con 0x7fdff807bc50 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff809e230 0x7fdff809e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdff809e230 0x7fdff809e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40928/0 (socket says 172.21.15.129:40928) 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 -- 172.21.15.129:0/3681872095 learned_addr learned my addr 172.21.15.129:0/3681872095 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdff807bc50 0x7fdff814df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdfff4b2700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 0x7fdff80781b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 -- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdff807bc50 msgr2=0x7fdff814df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdff807bc50 0x7fdff814df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 -- 172.21.15.129:0/3681872095 >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 msgr2=0x7fdff80781b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 0x7fdff80781b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdffe4b0700 1 -- 172.21.15.129:0/3681872095 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff80a2a80 con 0x7fdff809e230 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.689+0000 7fdfff4b2700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 0x7fdff80781b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.129:3300/0 conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.129:3300/0 conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfe800c9a0 con 0x7fdff809e230 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe800c5c0 con 0x7fdff809e230 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff80a5160 con 0x7fdff809e230 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfe8010cb0 con 0x7fdff809e230 2019-10-13T05:21:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff8078700 con 0x7fdff809e230 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fdfe8007420 con 0x7fdff809e230 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 0x7fdfe0029e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fdfe8034a70 con 0x7fdff809e230 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 0x7fdfe0029e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.693+0000 7fdffecb1700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 0x7fdfe0029e20 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.697+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff80555c0 con 0x7fdff809e230 2019-10-13T05:21:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.701+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.701+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fdfe8025590 con 0x7fdff809e230 2019-10-13T05:21:38.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.857+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdff80a6310 con 0x7fdff809e230 2019-10-13T05:21:38.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.857+0000 7fdffe4b0700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.857+0000 7fdfef7fe700 1 -- 172.21.15.129:0/3681872095 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7fdfe8003c40 con 0x7fdff809e230 2019-10-13T05:21:38.866 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 3 2019-10-13T05:21:38.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:38.866 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T05:21:36.833107+0000","modified":"2019-10-13T05:21:38.536320+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 msgr2=0x7fdfe0029e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 0x7fdfe0029e20 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 msgr2=0x7fdff809e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fdff8075680 tx=0x7fdff00078d0).stop 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 shutdown_connections 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdfe0027920 0x7fdfe0029e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> v2:172.21.15.71:3300/0 conn(0x7fdff8077d90 0x7fdff80781b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdff807bc50 0x7fdff814df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 --2- 172.21.15.129:0/3681872095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdff809e230 0x7fdff809e650 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 shutdown_connections 2019-10-13T05:21:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 wait complete. 2019-10-13T05:21:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.861+0000 7fe000714700 1 -- 172.21.15.129:0/3681872095 >> 172.21.15.129:0/3681872095 conn(0x7fdff80a7520 msgr2=0x7fdff81429f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:38.877 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:38.877 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.957+0000 7f4c95cd3700 1 Processor -- start 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 -- start start 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 0x7f4c90144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c900a7a30 con 0x7f4c900a7520 2019-10-13T05:21:38.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c900a96b0 con 0x7f4c90144a10 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c95cd3700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c900a9950 con 0x7f4c900a1c80 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8f7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4c900a7520 0x7f4c900a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58910/0 (socket says 172.21.15.129:58910) 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8f7fe700 1 -- 172.21.15.129:0/72598927 learned_addr learned my addr 172.21.15.129:0/72598927 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c94a71700 1 --2- 172.21.15.129:0/72598927 >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 0x7f4c90144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8effd700 1 -- 172.21.15.129:0/72598927 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117160882 0 0) 0x7f4c900a7a30 con 0x7f4c900a7520 2019-10-13T05:21:38.973 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8effd700 1 -- 172.21.15.129:0/72598927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c780030a0 con 0x7f4c900a7520 2019-10-13T05:21:38.974 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 -- 172.21.15.129:0/72598927 >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 msgr2=0x7f4c90144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:38.974 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 0x7f4c90144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:38.974 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 -- 172.21.15.129:0/72598927 >> v1:172.21.15.129:6789/0 conn(0x7f4c900a7520 legacy=0x7f4c900a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.974 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 -- 172.21.15.129:0/72598927 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c900a1940 con 0x7f4c900a1c80 2019-10-13T05:21:38.974 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c94a71700 1 --2- 172.21.15.129:0/72598927 >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 0x7f4c90144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).ready entity=mon.2 client_cookie=5c2e43e5521f75fe server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8effd700 1 -- 172.21.15.129:0/72598927 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c80011670 con 0x7f4c900a1c80 2019-10-13T05:21:38.975 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.961+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c8effd700 1 -- 172.21.15.129:0/72598927 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c80008b40 con 0x7f4c900a1c80 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c8effd700 1 -- 172.21.15.129:0/72598927 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c80011c50 con 0x7f4c900a1c80 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 msgr2=0x7f4c900a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4c80008b00 tx=0x7f4c88007ac0).stop 2019-10-13T05:21:38.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- 172.21.15.129:0/72598927 shutdown_connections 2019-10-13T05:21:38.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/72598927 >> v2:172.21.15.71:3300/0 conn(0x7f4c90144a10 0x7f4c90144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:38.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/72598927 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c900a1c80 0x7f4c900a13f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:38.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- 172.21.15.129:0/72598927 shutdown_connections 2019-10-13T05:21:38.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- 172.21.15.129:0/72598927 wait complete. 2019-10-13T05:21:38.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- 172.21.15.129:0/72598927 >> 172.21.15.129:0/72598927 conn(0x7f4c9005e870 msgr2=0x7f4c9009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 Processor -- start 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 -- start start 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.965+0000 7f4c95cd3700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c95cd3700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c90123a30 con 0x7f4c90141cd0 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c95cd3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c90075480 con 0x7f4c9007d960 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c95cd3700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90124550 con 0x7f4c90121790 2019-10-13T05:21:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8f7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4c9007d960 0x7f4c90123d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58916/0 (socket says 172.21.15.129:58916) 2019-10-13T05:21:38.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8f7fe700 1 -- 172.21.15.129:0/2251366431 learned_addr learned my addr 172.21.15.129:0/2251366431 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:38.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186581376 0 0) 0x7f4c90075480 con 0x7f4c9007d960 2019-10-13T05:21:38.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c700030a0 con 0x7f4c9007d960 2019-10-13T05:21:38.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480634086 0 0) 0x7f4c90123a30 con 0x7f4c90141cd0 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c90075480 con 0x7f4c90141cd0 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 -- 172.21.15.129:0/2251366431 >> v1:172.21.15.129:6789/0 conn(0x7f4c9007d960 legacy=0x7f4c90123d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 -- 172.21.15.129:0/2251366431 >> v1:172.21.15.71:6789/0 conn(0x7f4c90141cd0 legacy=0x7f4c9011a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 -- 172.21.15.129:0/2251366431 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c900650c0 con 0x7f4c90121790 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.983 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c80005cb0 con 0x7f4c90121790 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c80011870 con 0x7f4c90121790 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c80007540 con 0x7f4c90121790 2019-10-13T05:21:38.984 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c90123370 con 0x7f4c90121790 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c901420f0 con 0x7f4c90121790 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4c80015180 con 0x7f4c90121790 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 0x7f4c7002a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.969+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f4c80034dd0 con 0x7f4c90121790 2019-10-13T05:21:38.985 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.973+0000 7f4c8f7fe700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 0x7f4c7002a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:38.986 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.973+0000 7f4c8f7fe700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 0x7f4c7002a530 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:38.986 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.973+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c900555c0 con 0x7f4c90121790 2019-10-13T05:21:38.990 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.985+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:38.990 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:38.985+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4c800082c0 con 0x7f4c90121790 2019-10-13T05:21:39.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.145+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c90067b90 con 0x7f4c90121790 2019-10-13T05:21:39.153 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.145+0000 7f4c8ffff700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.153 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.145+0000 7f4c8cff9700 1 -- 172.21.15.129:0/2251366431 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7f4c8001d9a0 con 0x7f4c90121790 2019-10-13T05:21:39.154 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:39.154 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 msgr2=0x7f4c7002a530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 0x7f4c7002a530 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 msgr2=0x7f4c90122e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f4c800086e0 tx=0x7f4c8000f700).stop 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 shutdown_connections 2019-10-13T05:21:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4c70028030 0x7f4c7002a530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 --2- 172.21.15.129:0/2251366431 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4c90121790 0x7f4c90122e20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 shutdown_connections 2019-10-13T05:21:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 wait complete. 2019-10-13T05:21:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.149+0000 7f4c95cd3700 1 -- 172.21.15.129:0/2251366431 >> 172.21.15.129:0/2251366431 conn(0x7f4c90059eb0 msgr2=0x7f4c900a9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.168 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-10-13T05:21:39.168 INFO:tasks.ceph.mds.a:Restarting daemon 2019-10-13T05:21:39.168 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:39.168 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-10-13T05:21:39.172 INFO:tasks.ceph.mds.a:Started 2019-10-13T05:21:39.173 INFO:tasks.ceph.mds.b:Restarting daemon 2019-10-13T05:21:39.173 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:39.173 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-10-13T05:21:39.176 INFO:tasks.ceph.mds.b:Started 2019-10-13T05:21:39.177 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:39.177 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:39.177 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-10-13T05:21:39.236 INFO:tasks.ceph.mds.b.smithi071.stdout:starting mds.b at 2019-10-13T05:21:39.237 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2019-10-13T05:21:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 Processor -- start 2019-10-13T05:21:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 -- start start 2019-10-13T05:21:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f551c0a1c80 0x7f551c0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 0x7f551c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c0a7a30 con 0x7f551c144a10 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c0a96b0 con 0x7f551c0a1c80 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5522a8f700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0a9950 con 0x7f551c0a7520 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f551c0a1c80 0x7f551c0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552182d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f551c144a10 0x7f551c0a60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58938/0 (socket says 172.21.15.129:58938) 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552182d700 1 -- 172.21.15.129:0/365172098 learned_addr learned my addr 172.21.15.129:0/365172098 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552082b700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 0x7f551c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 -- 172.21.15.129:0/365172098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 msgr2=0x7f551c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 0x7f551c0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 -- 172.21.15.129:0/365172098 >> v1:172.21.15.129:6789/0 conn(0x7f551c144a10 legacy=0x7f551c0a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f5513fff700 1 -- 172.21.15.129:0/365172098 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811002627 0 0) 0x7f551c0a7a30 con 0x7f551c144a10 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 -- 172.21.15.129:0/365172098 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c0a1940 con 0x7f551c0a1c80 2019-10-13T05:21:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552082b700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 0x7f551c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.261+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> v2:172.21.15.129:3300/0 conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f551c0a9870 tx=0x7f550c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> v2:172.21.15.129:3300/0 conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f551c0a9870 tx=0x7f550c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f551c0a9870 tx=0x7f550c007710).ready entity=mon.1 client_cookie=62f27d7db0512c84 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f551c0a9870 tx=0x7f550c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5513fff700 1 -- 172.21.15.129:0/365172098 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5518008ba0 con 0x7f551c0a1c80 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f551c0a9870 tx=0x7f550c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f551c0a9870 tx=0x7f550c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5513fff700 1 -- 172.21.15.129:0/365172098 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55180087c0 con 0x7f551c0a1c80 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5513fff700 1 -- 172.21.15.129:0/365172098 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5518011700 con 0x7f551c0a1c80 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 msgr2=0x7f551c0a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f551c0a9870 tx=0x7f550c007710).stop 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- 172.21.15.129:0/365172098 shutdown_connections 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c0a1c80 0x7f551c0a13f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 --2- 172.21.15.129:0/365172098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f551c0a7520 0x7f551c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- 172.21.15.129:0/365172098 shutdown_connections 2019-10-13T05:21:39.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- 172.21.15.129:0/365172098 wait complete. 2019-10-13T05:21:39.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- 172.21.15.129:0/365172098 >> 172.21.15.129:0/365172098 conn(0x7f551c05e870 msgr2=0x7f551c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 Processor -- start 2019-10-13T05:21:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- start start 2019-10-13T05:21:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f551c142200 0x7f551c07c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c063c20 con 0x7f551c063800 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c0a06e0 con 0x7f551c0a2410 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5522a8f700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c063f80 con 0x7f551c142200 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f551c0a2410 0x7f551c141a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58942/0 (socket says 172.21.15.129:58942) 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552102c700 1 -- 172.21.15.129:0/402549548 learned_addr learned my addr 172.21.15.129:0/402549548 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f552082b700 1 --2- 172.21.15.129:0/402549548 >> v2:172.21.15.71:3300/0 conn(0x7f551c142200 0x7f551c07c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257827564 0 0) 0x7f551c0a06e0 con 0x7f551c0a2410 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.265+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc0030a0 con 0x7f551c0a2410 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527207834 0 0) 0x7f551c063c20 con 0x7f551c063800 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f551c0a06e0 con 0x7f551c063800 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2505769717 0 0) 0x7f54fc0030a0 con 0x7f551c0a2410 2019-10-13T05:21:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f551c063c20 con 0x7f551c0a2410 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7f5518008460 con 0x7f551c0a2410 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 >> v2:172.21.15.71:3300/0 conn(0x7f551c142200 msgr2=0x7f551c07c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 --2- 172.21.15.129:0/402549548 >> v2:172.21.15.71:3300/0 conn(0x7f551c142200 0x7f551c07c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 >> v1:172.21.15.129:6789/0 conn(0x7f551c0a2410 legacy=0x7f551c141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 >> v1:172.21.15.71:6789/0 conn(0x7f551c063800 legacy=0x7f551c11c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54fc003fb0 0x7f54fc004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f54fc009650 0x7f54fc00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c063f80 con 0x7f54fc003fb0 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc005250 con 0x7f54fc003aa0 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc00c050 con 0x7f54fc009650 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552082b700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54fc003fb0 0x7f54fc004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552182d700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f54fc009650 0x7f54fc00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 -- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f54fc009650 msgr2=0x7f54fc00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f54fc009650 0x7f54fc00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 -- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54fc003fb0 msgr2=0x7f54fc004a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54fc003fb0 0x7f54fc004a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c0a4cb0 con 0x7f54fc003aa0 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552182d700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f54fc009650 0x7f54fc00ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T05:21:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c0a4f80 con 0x7f54fc003aa0 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552182d700 1 -- 172.21.15.129:0/402549548 reap_dead start 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c063800 con 0x7f54fc003aa0 2019-10-13T05:21:39.286 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5518009b20 con 0x7f54fc003aa0 2019-10-13T05:21:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518007180 con 0x7f54fc003aa0 2019-10-13T05:21:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5518005740 con 0x7f54fc003aa0 2019-10-13T05:21:39.288 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.288 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5518005940 con 0x7f54fc003aa0 2019-10-13T05:21:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 0x7f54fc02dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f5518035070 con 0x7f54fc003aa0 2019-10-13T05:21:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.269+0000 7f552082b700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 0x7f54fc02dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.273+0000 7f552082b700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 0x7f54fc02dc20 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.290 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.277+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f40031f0 con 0x7f54fc003aa0 2019-10-13T05:21:39.290 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.281+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.290 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.281+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f5518005c30 con 0x7f54fc003aa0 2019-10-13T05:21:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.441+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f4003630 con 0x7f54fc003aa0 2019-10-13T05:21:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.441+0000 7f552102c700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.441+0000 7f5511ffb700 1 -- 172.21.15.129:0/402549548 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7f551801cba0 con 0x7f54fc003aa0 2019-10-13T05:21:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:39.449 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 msgr2=0x7f54fc02dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 0x7f54fc02dc20 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 msgr2=0x7f54fc004410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f5518007940 tx=0x7f550c00d090).stop 2019-10-13T05:21:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 shutdown_connections 2019-10-13T05:21:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f54fc02b720 0x7f54fc02dc20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 --2- 172.21.15.129:0/402549548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc003aa0 0x7f54fc004410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 shutdown_connections 2019-10-13T05:21:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 wait complete. 2019-10-13T05:21:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.445+0000 7f5522a8f700 1 -- 172.21.15.129:0/402549548 >> 172.21.15.129:0/402549548 conn(0x7f551c059eb0 msgr2=0x7f551c142bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.463 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:31.757741+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:34.685611+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:35.908349+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T05:21:39.463 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:39.463 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T05:21:39.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 Processor -- start 2019-10-13T05:21:39.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 -- start start 2019-10-13T05:21:39.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd62809d160 0x7fd6280a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6280a86f0 0x7fd62814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6280a0fd0 con 0x7fd6280a82d0 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6280a1300 con 0x7fd62809d160 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6280a15a0 con 0x7fd6280a86f0 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6280a86f0 0x7fd62814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd627fff700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd62809d160 0x7fd6280a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd6277fe700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd6280a82d0 0x7fd628144920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:53544/0 (socket says 172.21.15.129:53544) 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd6277fe700 1 -- 172.21.15.129:0/2223620037 learned_addr learned my addr 172.21.15.129:0/2223620037 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 -- 172.21.15.129:0/2223620037 >> v2:172.21.15.71:3300/0 conn(0x7fd62809d160 msgr2=0x7fd6280a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> v2:172.21.15.71:3300/0 conn(0x7fd62809d160 0x7fd6280a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd626ffd700 1 -- 172.21.15.129:0/2223620037 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148884707 0 0) 0x7fd6280a0fd0 con 0x7fd6280a82d0 2019-10-13T05:21:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 -- 172.21.15.129:0/2223620037 >> v1:172.21.15.71:6789/0 conn(0x7fd6280a82d0 legacy=0x7fd628144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 -- 172.21.15.129:0/2223620037 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6280a8b10 con 0x7fd6280a86f0 2019-10-13T05:21:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> v2:172.21.15.129:3300/0 conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6200032b0 tx=0x7fd618007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> v2:172.21.15.129:3300/0 conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6200032b0 tx=0x7fd618007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fd6200032b0 tx=0x7fd618007710).ready entity=mon.1 client_cookie=2a4169212fb4f52e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fd6200032b0 tx=0x7fd618007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd626ffd700 1 -- 172.21.15.129:0/2223620037 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6200117a0 con 0x7fd6280a86f0 2019-10-13T05:21:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fd6200032b0 tx=0x7fd618007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd626ffd700 1 -- 172.21.15.129:0/2223620037 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620008b40 con 0x7fd6280a86f0 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fd6200032b0 tx=0x7fd618007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 -- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 msgr2=0x7fd62814eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd626ffd700 1 -- 172.21.15.129:0/2223620037 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd620011d80 con 0x7fd6280a86f0 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.557+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fd6200032b0 tx=0x7fd618007710).stop 2019-10-13T05:21:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/2223620037 shutdown_connections 2019-10-13T05:21:39.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/2223620037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6280a86f0 0x7fd62814eb20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/2223620037 >> v2:172.21.15.71:3300/0 conn(0x7fd62809d160 0x7fd6280a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/2223620037 shutdown_connections 2019-10-13T05:21:39.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/2223620037 wait complete. 2019-10-13T05:21:39.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/2223620037 >> 172.21.15.129:0/2223620037 conn(0x7fd62805e870 msgr2=0x7fd62809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 Processor -- start 2019-10-13T05:21:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- start start 2019-10-13T05:21:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd628129b60 0x7fd628129f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd628078400 0x7fd628078820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd628078d70 con 0x7fd628059f30 2019-10-13T05:21:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62807a2c0 con 0x7fd628129b60 2019-10-13T05:21:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62807a560 con 0x7fd628078400 2019-10-13T05:21:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd628078400 0x7fd628078820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd628078400 0x7fd628078820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40994/0 (socket says 172.21.15.129:40994) 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 -- 172.21.15.129:0/445558732 learned_addr learned my addr 172.21.15.129:0/445558732 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd6277fe700 1 --2- 172.21.15.129:0/445558732 >> v2:172.21.15.71:3300/0 conn(0x7fd628129b60 0x7fd628129f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 -- 172.21.15.129:0/445558732 >> v2:172.21.15.71:3300/0 conn(0x7fd628129b60 msgr2=0x7fd628129f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> v2:172.21.15.71:3300/0 conn(0x7fd628129b60 0x7fd628129f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 -- 172.21.15.129:0/445558732 >> v1:172.21.15.71:6789/0 conn(0x7fd628059f30 legacy=0x7fd6280a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 -- 172.21.15.129:0/445558732 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628079f40 con 0x7fd628078400 2019-10-13T05:21:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> v2:172.21.15.129:3300/0 conn(0x7fd628078400 0x7fd628078820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> v2:172.21.15.129:3300/0 conn(0x7fd628078400 0x7fd628078820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd620005ac0 con 0x7fd628078400 2019-10-13T05:21:39.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6200119a0 con 0x7fd628078400 2019-10-13T05:21:39.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62807aaa0 con 0x7fd628078400 2019-10-13T05:21:39.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd620007350 con 0x7fd628078400 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.561+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd628130fb0 con 0x7fd628078400 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd620015180 con 0x7fd628078400 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd624ff9700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 0x7fd608029d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd620034f90 con 0x7fd628078400 2019-10-13T05:21:39.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd627fff700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 0x7fd608029d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.565+0000 7fd627fff700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 0x7fd608029d90 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.569+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6280555c0 con 0x7fd628078400 2019-10-13T05:21:39.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.577+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.577+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd6200075a0 con 0x7fd628078400 2019-10-13T05:21:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd6280a3100 con 0x7fd628078400 2019-10-13T05:21:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62c9b8700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd624ff9700 1 -- 172.21.15.129:0/445558732 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd62001db70 con 0x7fd628078400 2019-10-13T05:21:39.735 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:39.737 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 msgr2=0x7fd608029d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.737 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 0x7fd608029d90 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.737 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 msgr2=0x7fd628078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.737 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fd620007c30 tx=0x7fd61c000fc0).stop 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 shutdown_connections 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd628078400 0x7fd628078820 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/445558732 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd608027890 0x7fd608029d90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 --2- 172.21.15.129:0/445558732 >> v2:172.21.15.71:3300/0 conn(0x7fd628129b60 0x7fd628129f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 shutdown_connections 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 wait complete. 2019-10-13T05:21:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.729+0000 7fd62dc1a700 1 -- 172.21.15.129:0/445558732 >> 172.21.15.129:0/445558732 conn(0x7fd6280a75b0 msgr2=0x7fd62811bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.746 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:39.746 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T05:21:39.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 Processor -- start 2019-10-13T05:21:39.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 -- start start 2019-10-13T05:21:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a86f0 0x7fd17c14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0a0fd0 con 0x7fd17c0a82d0 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0a8b10 con 0x7fd17c09d160 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd1822b3700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0a1300 con 0x7fd17c0a86f0 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd180850700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd17c09d160 0x7fd17c0a69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58968/0 (socket says 172.21.15.129:58968) 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.833+0000 7fd180850700 1 -- 172.21.15.129:0/71521356 learned_addr learned my addr 172.21.15.129:0/71521356 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd181051700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a86f0 0x7fd17c14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104460441 0 0) 0x7fd17c0a8b10 con 0x7fd17c09d160 2019-10-13T05:21:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1680030a0 con 0x7fd17c09d160 2019-10-13T05:21:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793258757 0 0) 0x7fd17c0a0fd0 con 0x7fd17c0a82d0 2019-10-13T05:21:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c0a8b10 con 0x7fd17c0a82d0 2019-10-13T05:21:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 249095251 0 0) 0x7fd1680030a0 con 0x7fd17c09d160 2019-10-13T05:21:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a86f0 msgr2=0x7fd17c14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a86f0 0x7fd17c14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 >> v1:172.21.15.71:6789/0 conn(0x7fd17c0a82d0 legacy=0x7fd17c144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c145120 con 0x7fd17c09d160 2019-10-13T05:21:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7fd16c001750 con 0x7fd17c09d160 2019-10-13T05:21:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 >> v1:172.21.15.129:6789/0 conn(0x7fd17c09d160 legacy=0x7fd17c0a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd168004410 0x7fd168004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd168004d80 0x7fd16804f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0a1300 con 0x7fd168004d80 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16804fbc0 con 0x7fd168003930 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16804fd80 con 0x7fd168004410 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17bfff700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd168004410 0x7fd168004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd181051700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd168004d80 0x7fd16804f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 -- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd168004410 msgr2=0x7fd168004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd168004410 0x7fd168004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 -- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd168004d80 msgr2=0x7fd16804f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd168004d80 0x7fd16804f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 -- 172.21.15.129:0/71521356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168050000 con 0x7fd168003930 2019-10-13T05:21:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd16c001e80 tx=0x7fd174007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd16c001e80 tx=0x7fd174007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7fd16c001e80 tx=0x7fd174007c30).ready entity=mon.1 client_cookie=7386b36152bc7d6e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd16c001e80 tx=0x7fd174007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd16c011370 con 0x7fd168003930 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd16c001e80 tx=0x7fd174007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c010f90 con 0x7fd168003930 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd180850700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd16c001e80 tx=0x7fd174007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd17b7fe700 1 -- 172.21.15.129:0/71521356 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd16c011a30 con 0x7fd168003930 2019-10-13T05:21:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 -- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 msgr2=0x7fd168003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 --2- 172.21.15.129:0/71521356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd168003930 0x7fd168003d50 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7fd16c001e80 tx=0x7fd174007c30).stop 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd181051700 1 -- 172.21.15.129:0/71521356 reap_dead start 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 -- 172.21.15.129:0/71521356 shutdown_connections 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 -- 172.21.15.129:0/71521356 shutdown_connections 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 -- 172.21.15.129:0/71521356 wait complete. 2019-10-13T05:21:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.837+0000 7fd1822b3700 1 -- 172.21.15.129:0/71521356 >> 172.21.15.129:0/71521356 conn(0x7fd17c05e870 msgr2=0x7fd17c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.841+0000 7fd1822b3700 1 Processor -- start 2019-10-13T05:21:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.841+0000 7fd1822b3700 1 -- start start 2019-10-13T05:21:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.841+0000 7fd1822b3700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a5690 0x7fd17c119ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.841+0000 7fd1822b3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd17c11ba20 0x7fd17c0a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1822b3700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c119960 con 0x7fd17c0a1f90 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1822b3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c11c880 con 0x7fd17c11ba20 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1822b3700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c064ac0 con 0x7fd17c0a5690 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd17c11ba20 0x7fd17c0a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd17c11ba20 0x7fd17c0a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41012/0 (socket says 172.21.15.129:41012) 2019-10-13T05:21:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 -- 172.21.15.129:0/119161401 learned_addr learned my addr 172.21.15.129:0/119161401 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd180850700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a5690 0x7fd17c119ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 -- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a5690 msgr2=0x7fd17c119ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a5690 0x7fd17c119ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 -- 172.21.15.129:0/119161401 >> v1:172.21.15.71:6789/0 conn(0x7fd17c0a1f90 legacy=0x7fd17c13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 -- 172.21.15.129:0/119161401 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c11be40 con 0x7fd17c11ba20 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> v2:172.21.15.129:3300/0 conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> v2:172.21.15.129:3300/0 conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd164011bc0 con 0x7fd17c11ba20 2019-10-13T05:21:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640117e0 con 0x7fd17c11ba20 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c065200 con 0x7fd17c11ba20 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd16400e840 con 0x7fd17c11ba20 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c0a23b0 con 0x7fd17c11ba20 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd16400ea40 con 0x7fd17c11ba20 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 0x7fd160029d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd164034b10 con 0x7fd17c11ba20 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.845+0000 7fd180850700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 0x7fd160029d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.849+0000 7fd180850700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 0x7fd160029d80 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.849+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c064f60 con 0x7fd17c11ba20 2019-10-13T05:21:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.857+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:39.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:39.857+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd16400ed30 con 0x7fd17c11ba20 2019-10-13T05:21:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd17c0555c0 con 0x7fd17c11ba20 2019-10-13T05:21:40.022 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd17bfff700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.022 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1797fa700 1 -- 172.21.15.129:0/119161401 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd164025ea0 con 0x7fd17c11ba20 2019-10-13T05:21:40.022 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:40.025 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 msgr2=0x7fd160029d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.025 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 0x7fd160029d80 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.026 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 msgr2=0x7fd17c0a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.026 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7fd17c11ca40 tx=0x7fd16c005100).stop 2019-10-13T05:21:40.026 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 shutdown_connections 2019-10-13T05:21:40.026 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd160027880 0x7fd160029d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd17c0a5690 0x7fd17c119ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 --2- 172.21.15.129:0/119161401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17c11ba20 0x7fd17c0a1a40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 shutdown_connections 2019-10-13T05:21:40.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 wait complete. 2019-10-13T05:21:40.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.017+0000 7fd1822b3700 1 -- 172.21.15.129:0/119161401 >> 172.21.15.129:0/119161401 conn(0x7fd17c0a0d20 msgr2=0x7fd17c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:40.035 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:40.035 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T05:21:40.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 Processor -- start 2019-10-13T05:21:40.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 -- start start 2019-10-13T05:21:40.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f359409d160 0x7f35940a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:40.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940a0fd0 con 0x7f35940a82d0 2019-10-13T05:21:40.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940a8b10 con 0x7f35940a86f0 2019-10-13T05:21:40.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35940a1300 con 0x7f359409d160 2019-10-13T05:21:40.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f359409d160 0x7f35940a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3598df5700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f35940a86f0 0x7f359414eb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58992/0 (socket says 172.21.15.129:58992) 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3598df5700 1 -- 172.21.15.129:0/3257015443 learned_addr learned my addr 172.21.15.129:0/3257015443 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> v2:172.21.15.129:3300/0 conn(0x7f359409d160 0x7f35940a69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41020/0 (socket says 172.21.15.129:41020) 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 -- 172.21.15.129:0/3257015443 >> v1:172.21.15.129:6789/0 conn(0x7f35940a86f0 legacy=0x7f359414eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 -- 172.21.15.129:0/3257015443 >> v1:172.21.15.71:6789/0 conn(0x7f35940a82d0 legacy=0x7f3594144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.142 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 -- 172.21.15.129:0/3257015443 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940a6f40 con 0x7f359409d160 2019-10-13T05:21:40.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3592ffd700 1 -- 172.21.15.129:0/3257015443 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010248034 0 0) 0x7f35940a0fd0 con 0x7f35940a82d0 2019-10-13T05:21:40.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> v2:172.21.15.129:3300/0 conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f357c008780 tx=0x7f357c00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> v2:172.21.15.129:3300/0 conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f357c008780 tx=0x7f357c00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.143 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f357c008780 tx=0x7f357c00fbe0).ready entity=mon.1 client_cookie=7d3991c3a83d375a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f357c008780 tx=0x7f357c00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3592ffd700 1 -- 172.21.15.129:0/3257015443 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f357c011260 con 0x7f359409d160 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f357c008780 tx=0x7f357c00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3593fff700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f357c008780 tx=0x7f357c00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3592ffd700 1 -- 172.21.15.129:0/3257015443 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c008bb0 con 0x7f359409d160 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f3592ffd700 1 -- 172.21.15.129:0/3257015443 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f357c011840 con 0x7f359409d160 2019-10-13T05:21:40.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 -- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 msgr2=0x7f35940a69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.125+0000 7f359a057700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f357c008780 tx=0x7f357c00fbe0).stop 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- 172.21.15.129:0/3257015443 shutdown_connections 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 --2- 172.21.15.129:0/3257015443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f359409d160 0x7f35940a69f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- 172.21.15.129:0/3257015443 shutdown_connections 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- 172.21.15.129:0/3257015443 wait complete. 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- 172.21.15.129:0/3257015443 >> 172.21.15.129:0/3257015443 conn(0x7f359405e870 msgr2=0x7f359409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:40.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 Processor -- start 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- start start 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35940a5640 0x7f3594063dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3594064c50 con 0x7f35940a9540 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940a4610 con 0x7f3594126ea0 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f359a057700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35940a4370 con 0x7f35940a5640 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f3593fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35940a5640 0x7f3594063dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f35937fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3594126ea0 0x7f35940a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58994/0 (socket says 172.21.15.129:58994) 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f35937fe700 1 -- 172.21.15.129:0/2533585297 learned_addr learned my addr 172.21.15.129:0/2533585297 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.129+0000 7f3593fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35940a5640 0x7f3594063dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41026/0 (socket says 172.21.15.129:41026) 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 -- 172.21.15.129:0/2533585297 >> v1:172.21.15.129:6789/0 conn(0x7f3594126ea0 legacy=0x7f35940a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 -- 172.21.15.129:0/2533585297 >> v1:172.21.15.71:6789/0 conn(0x7f35940a9540 legacy=0x7f35940781a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664743785 0 0) 0x7f35940a4610 con 0x7f3594126ea0 2019-10-13T05:21:40.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 -- 172.21.15.129:0/2533585297 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940a3e60 con 0x7f35940a5640 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> v2:172.21.15.129:3300/0 conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> v2:172.21.15.129:3300/0 conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f357c007340 con 0x7f35940a5640 2019-10-13T05:21:40.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c011460 con 0x7f35940a5640 2019-10-13T05:21:40.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f357c00aa60 con 0x7f35940a5640 2019-10-13T05:21:40.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359405f000 con 0x7f35940a5640 2019-10-13T05:21:40.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359411b680 con 0x7f35940a5640 2019-10-13T05:21:40.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.150 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f357c00fc30 con 0x7f35940a5640 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 0x7f356c029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f357c0356b0 con 0x7f35940a5640 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.133+0000 7f35937fe700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 0x7f356c029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.137+0000 7f35937fe700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 0x7f356c029c90 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.141+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3594127460 con 0x7f35940a5640 2019-10-13T05:21:40.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.145+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.145+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f357c00ad30 con 0x7f35940a5640 2019-10-13T05:21:40.302 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f35940555c0 con 0x7f35940a5640 2019-10-13T05:21:40.303 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f3593fff700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.303 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f3590ff9700 1 -- 172.21.15.129:0/2533585297 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7f357c00f870 con 0x7f35940a5640 2019-10-13T05:21:40.303 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:40.305 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 msgr2=0x7f356c029c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.305 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 0x7f356c029c90 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 msgr2=0x7f3594063dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f357c007d70 tx=0x7f357c00f560).stop 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 shutdown_connections 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f356c027790 0x7f356c029c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 --2- 172.21.15.129:0/2533585297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35940a5640 0x7f3594063dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.297+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 shutdown_connections 2019-10-13T05:21:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.301+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 wait complete. 2019-10-13T05:21:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.301+0000 7f359a057700 1 -- 172.21.15.129:0/2533585297 >> 172.21.15.129:0/2533585297 conn(0x7f35940a2600 msgr2=0x7f35940a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:40.316 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-10-13T05:21:40.316 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-10-13T05:21:40.316 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:40.316 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-10-13T05:21:40.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.405+0000 7fef4798f700 1 Processor -- start 2019-10-13T05:21:40.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- start start 2019-10-13T05:21:40.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef400a7520 0x7fef400a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:40.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fef40144a10 0x7fef40144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:40.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400a7a30 con 0x7fef400a1c80 2019-10-13T05:21:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400a96b0 con 0x7fef400a7520 2019-10-13T05:21:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef400a9950 con 0x7fef40144a10 2019-10-13T05:21:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef400a7520 0x7fef400a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef400a7520 0x7fef400a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41034/0 (socket says 172.21.15.129:41034) 2019-10-13T05:21:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 -- 172.21.15.129:0/3247082532 learned_addr learned my addr 172.21.15.129:0/3247082532 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4672d700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fef40144a10 0x7fef40144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 -- 172.21.15.129:0/3247082532 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fef40144a10 msgr2=0x7fef40144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fef40144a10 0x7fef40144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 -- 172.21.15.129:0/3247082532 >> v1:172.21.15.71:6789/0 conn(0x7fef400a1c80 legacy=0x7fef400a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 -- 172.21.15.129:0/3247082532 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400aa3e0 con 0x7fef400a7520 2019-10-13T05:21:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> v2:172.21.15.129:3300/0 conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fef400a7ea0 tx=0x7fef38007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> v2:172.21.15.129:3300/0 conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fef400a7ea0 tx=0x7fef38007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fef400a7ea0 tx=0x7fef38007a90).ready entity=mon.1 client_cookie=84a4d75ab1dbe343 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fef400a7ea0 tx=0x7fef38007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef44f2a700 1 -- 172.21.15.129:0/3247082532 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef34008c80 con 0x7fef400a7520 2019-10-13T05:21:40.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fef400a7ea0 tx=0x7fef38007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef44f2a700 1 -- 172.21.15.129:0/3247082532 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef340088a0 con 0x7fef400a7520 2019-10-13T05:21:40.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4572b700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fef400a7ea0 tx=0x7fef38007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef44f2a700 1 -- 172.21.15.129:0/3247082532 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef340119f0 con 0x7fef400a7520 2019-10-13T05:21:40.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 msgr2=0x7fef400a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fef400a7ea0 tx=0x7fef38007a90).stop 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- 172.21.15.129:0/3247082532 shutdown_connections 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fef40144a10 0x7fef40144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 --2- 172.21.15.129:0/3247082532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400a7520 0x7fef400a9e90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- 172.21.15.129:0/3247082532 shutdown_connections 2019-10-13T05:21:40.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- 172.21.15.129:0/3247082532 wait complete. 2019-10-13T05:21:40.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.409+0000 7fef4798f700 1 -- 172.21.15.129:0/3247082532 >> 172.21.15.129:0/3247082532 conn(0x7fef4005e870 msgr2=0x7fef4009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:40.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 Processor -- start 2019-10-13T05:21:40.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 -- start start 2019-10-13T05:21:40.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef4009fa10 0x7fef400a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:40.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fef40075180 0x7fef400775b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400a51a0 con 0x7fef40064fe0 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400a3dd0 con 0x7fef40075180 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4798f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400a4420 con 0x7fef4009fa10 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef45f2c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef4009fa10 0x7fef400a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef45f2c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fef4009fa10 0x7fef400a4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41040/0 (socket says 172.21.15.129:41040) 2019-10-13T05:21:40.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef45f2c700 1 -- 172.21.15.129:0/1922597426 learned_addr learned my addr 172.21.15.129:0/1922597426 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:40.433 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4572b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fef40064fe0 0x7fef40111630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59012/0 (socket says 172.21.15.129:59012) 2019-10-13T05:21:40.433 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.413+0000 7fef4672d700 1 --2- 172.21.15.129:0/1922597426 >> v2:172.21.15.71:3300/0 conn(0x7fef40075180 0x7fef400775b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.433 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 -- 172.21.15.129:0/1922597426 >> v2:172.21.15.71:3300/0 conn(0x7fef40075180 msgr2=0x7fef400775b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.433 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> v2:172.21.15.71:3300/0 conn(0x7fef40075180 0x7fef400775b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021116975 0 0) 0x7fef400a51a0 con 0x7fef40064fe0 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 -- 172.21.15.129:0/1922597426 >> v1:172.21.15.129:6789/0 conn(0x7fef40064fe0 legacy=0x7fef40111630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 -- 172.21.15.129:0/1922597426 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef40121000 con 0x7fef4009fa10 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> v2:172.21.15.129:3300/0 conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> v2:172.21.15.129:3300/0 conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef3c00ca90 con 0x7fef4009fa10 2019-10-13T05:21:40.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c00c6b0 con 0x7fef4009fa10 2019-10-13T05:21:40.437 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef3c010f60 con 0x7fef4009fa10 2019-10-13T05:21:40.438 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400a5fb0 con 0x7fef4009fa10 2019-10-13T05:21:40.438 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef40077b00 con 0x7fef4009fa10 2019-10-13T05:21:40.438 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.438 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fef3c011270 con 0x7fef4009fa10 2019-10-13T05:21:40.439 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 0x7fef28029d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:40.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fef3c034bd0 con 0x7fef4009fa10 2019-10-13T05:21:40.441 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef4572b700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 0x7fef28029d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:40.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.417+0000 7fef4572b700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 0x7fef28029d10 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:40.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.425+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef400555c0 con 0x7fef4009fa10 2019-10-13T05:21:40.442 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.429+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.443 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.429+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fef3c025f60 con 0x7fef4009fa10 2019-10-13T05:21:40.597 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.589+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fef4009c870 con 0x7fef4009fa10 2019-10-13T05:21:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.593+0000 7fef45f2c700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.593+0000 7fef327fc700 1 -- 172.21.15.129:0/1922597426 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+68495 (secure 0 0 0) 0x7fef3c011560 con 0x7fef4009fa10 2019-10-13T05:21:40.606 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:40.607 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 msgr2=0x7fef28029d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.607 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 0x7fef28029d10 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 msgr2=0x7fef400a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:40.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7fef3c000e90 tx=0x7fef380078d0).stop 2019-10-13T05:21:40.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 shutdown_connections 2019-10-13T05:21:40.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fef28027810 0x7fef28029d10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 --2- 172.21.15.129:0/1922597426 >> v2:172.21.15.71:3300/0 conn(0x7fef40075180 0x7fef400775b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 --2- 172.21.15.129:0/1922597426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef4009fa10 0x7fef400a4c50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:40.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 shutdown_connections 2019-10-13T05:21:40.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 wait complete. 2019-10-13T05:21:40.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:40.601+0000 7fef4798f700 1 -- 172.21.15.129:0/1922597426 >> 172.21.15.129:0/1922597426 conn(0x7fef400a2320 msgr2=0x7fef400a66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:40.620 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":13713},{"type":"v1","addr":"172.21.15.71:6801","nonce":13713}]},"active_addr":"172.21.15.71:6801/13713","active_change":"2019-10-13T05:21:20.255059+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-10-13T05:21:40.626 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-10-13T05:21:41.434 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:41.434 INFO:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:41.527 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 Processor -- start 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 -- start start 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 0x7fee4c0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fee4c144a10 0x7fee4c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0a7a30 con 0x7fee4c0a7520 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c0a96b0 con 0x7fee4c144a10 2019-10-13T05:21:41.528 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee53104700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0a9950 con 0x7fee4c0a1c80 2019-10-13T05:21:41.529 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee516a1700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 0x7fee4c0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.529 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee50ea0700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fee4c0a7520 0x7fee4c0a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:53608/0 (socket says 172.21.15.129:53608) 2019-10-13T05:21:41.529 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee50ea0700 1 -- 172.21.15.129:0/59354144 learned_addr learned my addr 172.21.15.129:0/59354144 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:41.529 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fee4c144a10 0x7fee4c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.529 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee3ffff700 1 -- 172.21.15.129:0/59354144 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255177619 0 0) 0x7fee4c0a7a30 con 0x7fee4c0a7520 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee3ffff700 1 -- 172.21.15.129:0/59354144 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee380030a0 con 0x7fee4c0a7520 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 -- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 msgr2=0x7fee4c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 0x7fee4c0a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 -- 172.21.15.129:0/59354144 >> v1:172.21.15.71:6789/0 conn(0x7fee4c0a7520 legacy=0x7fee4c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 -- 172.21.15.129:0/59354144 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0a1940 con 0x7fee4c144a10 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee516a1700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 0x7fee4c0a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:41.530 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> v2:172.21.15.71:3300/0 conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.531 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> v2:172.21.15.71:3300/0 conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.531 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).ready entity=mon.0 client_cookie=80cb978d696903d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.531 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.531 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee3ffff700 1 -- 172.21.15.129:0/59354144 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee44011670 con 0x7fee4c144a10 2019-10-13T05:21:41.531 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.517+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee3ffff700 1 -- 172.21.15.129:0/59354144 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44008b40 con 0x7fee4c144a10 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee51ea2700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee3ffff700 1 -- 172.21.15.129:0/59354144 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee44011c50 con 0x7fee4c144a10 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 msgr2=0x7fee4c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fee4c0a9870 tx=0x7fee48007ac0).stop 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- 172.21.15.129:0/59354144 shutdown_connections 2019-10-13T05:21:41.532 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee4c144a10 0x7fee4c144e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.533 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 --2- 172.21.15.129:0/59354144 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c0a1c80 0x7fee4c0a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:41.533 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- 172.21.15.129:0/59354144 shutdown_connections 2019-10-13T05:21:41.533 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- 172.21.15.129:0/59354144 wait complete. 2019-10-13T05:21:41.533 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- 172.21.15.129:0/59354144 >> 172.21.15.129:0/59354144 conn(0x7fee4c05e870 msgr2=0x7fee4c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:41.536 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 Processor -- start 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.521+0000 7fee53104700 1 -- start start 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fee4c11dfc0 0x7fee4c09d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c09de60 0x7fee4c11ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c11b990 con 0x7fee4c11cfc0 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c11e890 con 0x7fee4c11dfc0 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c11e5f0 con 0x7fee4c09de60 2019-10-13T05:21:41.537 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fee4c11dfc0 0x7fee4c09d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fee4c11dfc0 0x7fee4c09d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41056/0 (socket says 172.21.15.129:41056) 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 -- 172.21.15.129:0/2888634106 learned_addr learned my addr 172.21.15.129:0/2888634106 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee50ea0700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c09de60 0x7fee4c11ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 -- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c09de60 msgr2=0x7fee4c11ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c09de60 0x7fee4c11ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 -- 172.21.15.129:0/2888634106 >> v1:172.21.15.129:6789/0 conn(0x7fee4c11cfc0 legacy=0x7fee4c1313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.538 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 -- 172.21.15.129:0/2888634106 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c09e280 con 0x7fee4c11dfc0 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> v2:172.21.15.129:3300/0 conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> v2:172.21.15.129:3300/0 conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee48011cb0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.539 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee480118d0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c142530 con 0x7fee4c11dfc0 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee48005910 con 0x7fee4c11dfc0 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c11d3e0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.525+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fee48007140 con 0x7fee4c11dfc0 2019-10-13T05:21:41.540 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.541 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee3dffb700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 0x7fee2c029da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.541 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fee48036cc0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.541 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee50ea0700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 0x7fee2c029da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.541 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.529+0000 7fee50ea0700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 0x7fee2c029da0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.541 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.533+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c11e3e0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.544 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.537+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.545 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.537+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fee48005bb0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.702 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.697+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee4c0555c0 con 0x7fee4c11dfc0 2019-10-13T05:21:41.703 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.697+0000 7fee516a1700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.704 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.697+0000 7fee3dffb700 1 -- 172.21.15.129:0/2888634106 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7fee4800e760 con 0x7fee4c11dfc0 2019-10-13T05:21:41.705 INFO:teuthology.misc.health.smithi129.stdout: 2019-10-13T05:21:41.706 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:41.709 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 msgr2=0x7fee2c029da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.709 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 0x7fee2c029da0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.709 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 msgr2=0x7fee4c09d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7fee48002f30 tx=0x7fee40001160).stop 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 shutdown_connections 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fee2c0278a0 0x7fee2c029da0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee4c11dfc0 0x7fee4c09d910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 --2- 172.21.15.129:0/2888634106 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fee4c09de60 0x7fee4c11ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 shutdown_connections 2019-10-13T05:21:41.710 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 wait complete. 2019-10-13T05:21:41.711 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:41.701+0000 7fee53104700 1 -- 172.21.15.129:0/2888634106 >> 172.21.15.129:0/2888634106 conn(0x7fee4c1433a0 msgr2=0x7fee4c123cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:41.722 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T05:21:41.722 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:41.722 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 Processor -- start 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 -- start start 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a7520 0x7fab040a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fab04144a10 0x7fab04144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040a7a30 con 0x7fab040a1c80 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab040a96b0 con 0x7fab04144a10 2019-10-13T05:21:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab0aff5700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040a9950 con 0x7fab040a7520 2019-10-13T05:21:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fab04144a10 0x7fab04144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fab04144a10 0x7fab04144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41068/0 (socket says 172.21.15.129:41068) 2019-10-13T05:21:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09592700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fab040a1c80 0x7fab040a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59034/0 (socket says 172.21.15.129:59034) 2019-10-13T05:21:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 -- 172.21.15.129:0/3756439615 learned_addr learned my addr 172.21.15.129:0/3756439615 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab08d91700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a7520 0x7fab040a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 -- 172.21.15.129:0/3756439615 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a7520 msgr2=0x7fab040a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a7520 0x7fab040a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 -- 172.21.15.129:0/3756439615 >> v1:172.21.15.129:6789/0 conn(0x7fab040a1c80 legacy=0x7fab040a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 -- 172.21.15.129:0/3756439615 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040aa3e0 con 0x7fab04144a10 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7faafbfff700 1 -- 172.21.15.129:0/3756439615 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706431454 0 0) 0x7fab040a7a30 con 0x7fab040a1c80 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> v2:172.21.15.129:3300/0 conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab040a9870 tx=0x7faaf4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> v2:172.21.15.129:3300/0 conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab040a9870 tx=0x7faaf4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.809+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fab040a9870 tx=0x7faaf4007710).ready entity=mon.1 client_cookie=988df7d03edbbbcc server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fab040a9870 tx=0x7faaf4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7faafbfff700 1 -- 172.21.15.129:0/3756439615 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faafc010b90 con 0x7fab04144a10 2019-10-13T05:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fab040a9870 tx=0x7faaf4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7faafbfff700 1 -- 172.21.15.129:0/3756439615 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc0107b0 con 0x7fab04144a10 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab09d93700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fab040a9870 tx=0x7faaf4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 msgr2=0x7fab04144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fab040a9870 tx=0x7faaf4007710).stop 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7faafbfff700 1 -- 172.21.15.129:0/3756439615 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faafc0112b0 con 0x7fab04144a10 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- 172.21.15.129:0/3756439615 shutdown_connections 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab04144a10 0x7fab04144e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.824 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 --2- 172.21.15.129:0/3756439615 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a7520 0x7fab040a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:41.825 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- 172.21.15.129:0/3756439615 shutdown_connections 2019-10-13T05:21:41.825 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- 172.21.15.129:0/3756439615 wait complete. 2019-10-13T05:21:41.825 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- 172.21.15.129:0/3756439615 >> 172.21.15.129:0/3756439615 conn(0x7fab0405e870 msgr2=0x7fab0409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:41.829 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 Processor -- start 2019-10-13T05:21:41.829 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.813+0000 7fab0aff5700 1 -- start start 2019-10-13T05:21:41.829 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a8a50 0x7fab0412a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.829 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fab04064cc0 0x7fab041316d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.831 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab04078420 con 0x7fab040a5bd0 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab04064780 con 0x7fab04064cc0 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab04064a20 con 0x7fab040a8a50 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fab040a5bd0 0x7fab04077c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59042/0 (socket says 172.21.15.129:59042) 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 -- 172.21.15.129:0/1140308900 learned_addr learned my addr 172.21.15.129:0/1140308900 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab09592700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a8a50 0x7fab0412a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab09d93700 1 --2- 172.21.15.129:0/1140308900 >> v2:172.21.15.71:3300/0 conn(0x7fab04064cc0 0x7fab041316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586402251 0 0) 0x7fab04078420 con 0x7fab040a5bd0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae40030a0 con 0x7fab040a5bd0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 117758818 0 0) 0x7faae40030a0 con 0x7fab040a5bd0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab04078420 con 0x7fab040a5bd0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7faaf40015d0 con 0x7fab040a5bd0 2019-10-13T05:21:41.833 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a8a50 msgr2=0x7fab0412a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fab040a8a50 0x7fab0412a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 >> v2:172.21.15.71:3300/0 conn(0x7fab04064cc0 msgr2=0x7fab041316d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> v2:172.21.15.71:3300/0 conn(0x7fab04064cc0 0x7fab041316d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 >> v1:172.21.15.129:6789/0 conn(0x7fab040a5bd0 legacy=0x7fab04077c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faae4003890 0x7faae4003cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faae4004dd0 0x7faae400b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04064780 con 0x7faae4003890 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04064a20 con 0x7faae4004410 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae400bb20 con 0x7faae4004dd0 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab09d93700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faae4004dd0 0x7faae400b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 -- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faae4004dd0 msgr2=0x7faae400b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7faae4004dd0 0x7faae400b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 -- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faae4003890 msgr2=0x7faae4003cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab09592700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faae4003890 0x7faae4003cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faae4003890 0x7faae4003cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040650e0 con 0x7faae4004410 2019-10-13T05:21:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab09592700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faae4003890 0x7faae4003cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T05:21:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab04078820 con 0x7faae4004410 2019-10-13T05:21:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab040a8bf0 con 0x7faae4004410 2019-10-13T05:21:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf4005910 con 0x7faae4004410 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf4005530 con 0x7faae4004410 2019-10-13T05:21:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf400f330 con 0x7faae4004410 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.817+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7faaf400f530 con 0x7faae4004410 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.821+0000 7faaf9ffb700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faae4030410 0x7faae4032910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.821+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7faaf4011070 con 0x7faae4004410 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.821+0000 7fab09592700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faae4030410 0x7faae4032910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:41.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.821+0000 7fab09592700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faae4030410 0x7faae4032910 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:41.840 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.825+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0031f0 con 0x7faae4004410 2019-10-13T05:21:41.840 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.829+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.841 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.829+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7faaf40267c0 con 0x7faae4004410 2019-10-13T05:21:41.992 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.985+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faadc003630 con 0x7faae4004410 2019-10-13T05:21:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab08d91700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7faaf9ffb700 1 -- 172.21.15.129:0/1140308900 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7faaf400ba00 con 0x7faae4004410 2019-10-13T05:21:41.995 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:41.995 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faae4030410 msgr2=0x7faae4032910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab0aff5700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7faae4030410 0x7faae4032910 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 msgr2=0x7faae4004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab0aff5700 1 --2- 172.21.15.129:0/1140308900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faae4004410 0x7faae4004830 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7faaf4001960 tx=0x7fab00001360).stop 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab09d93700 1 -- 172.21.15.129:0/1140308900 reap_dead start 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab09d93700 1 -- 172.21.15.129:0/1140308900 reap_dead start 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.989+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 shutdown_connections 2019-10-13T05:21:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.993+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 shutdown_connections 2019-10-13T05:21:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.993+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 wait complete. 2019-10-13T05:21:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:41.993+0000 7fab0aff5700 1 -- 172.21.15.129:0/1140308900 >> 172.21.15.129:0/1140308900 conn(0x7fab040a7a10 msgr2=0x7fab0407d980 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.008 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.008 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-10-13T05:21:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 Processor -- start 2019-10-13T05:21:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 -- start start 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd4ac144a10 0x7fd4ac144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0a7a30 con 0x7fd4ac0a1c80 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac150ba0 con 0x7fd4ac0a7520 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0a96b0 con 0x7fd4ac144a10 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4ab7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd4ac0a7520 0x7fd4ac0a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59054/0 (socket says 172.21.15.129:59054) 2019-10-13T05:21:42.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4ab7fe700 1 -- 172.21.15.129:0/1099460399 learned_addr learned my addr 172.21.15.129:0/1099460399 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> v2:172.21.15.71:3300/0 conn(0x7fd4ac144a10 0x7fd4ac144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766988755 0 0) 0x7fd4ac150ba0 con 0x7fd4ac0a7520 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4980030a0 con 0x7fd4ac0a7520 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2930245804 0 0) 0x7fd4980030a0 con 0x7fd4ac0a7520 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1099460399 >> v1:172.21.15.129:6789/0 conn(0x7fd4ac0a7520 legacy=0x7fd4ac0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1099460399 >> v1:172.21.15.71:6789/0 conn(0x7fd4ac0a1c80 legacy=0x7fd4ac0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1099460399 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac151de0 con 0x7fd4ac144a10 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> v2:172.21.15.71:3300/0 conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd494001110 tx=0x7fd49c001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> v2:172.21.15.71:3300/0 conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd494001110 tx=0x7fd49c001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fd494001110 tx=0x7fd49c001d10).ready entity=mon.0 client_cookie=5c8a1acb6ba346d8 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fd494001110 tx=0x7fd49c001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd49c001ad0 con 0x7fd4ac144a10 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fd494001110 tx=0x7fd49c001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c0018e0 con 0x7fd4ac144a10 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fd494001110 tx=0x7fd49c001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 msgr2=0x7fd4ac144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fd494001110 tx=0x7fd49c001d10).stop 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.097+0000 7fd4aaffd700 1 -- 172.21.15.129:0/1099460399 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4a4008d50 con 0x7fd4ac144a10 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1099460399 shutdown_connections 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1099460399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4ac144a10 0x7fd4ac144e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1099460399 shutdown_connections 2019-10-13T05:21:42.112 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1099460399 wait complete. 2019-10-13T05:21:42.112 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1099460399 >> 172.21.15.129:0/1099460399 conn(0x7fd4ac05e870 msgr2=0x7fd4ac09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.115 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 Processor -- start 2019-10-13T05:21:42.115 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- start start 2019-10-13T05:21:42.115 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd4ac0a48c0 0x7fd4ac063800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.115 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0a0630 con 0x7fd4ac064e40 2019-10-13T05:21:42.115 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac1325f0 con 0x7fd4ac112930 2019-10-13T05:21:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b1b2b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0a4ce0 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b08c9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd4ac0a48c0 0x7fd4ac063800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b08c9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd4ac0a48c0 0x7fd4ac063800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41090/0 (socket says 172.21.15.129:41090) 2019-10-13T05:21:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.101+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1011511111 learned_addr learned my addr 172.21.15.129:0/1011511111 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1011511111 >> v1:172.21.15.129:6789/0 conn(0x7fd4ac112930 legacy=0x7fd4ac115540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1011511111 >> v1:172.21.15.71:6789/0 conn(0x7fd4ac064e40 legacy=0x7fd4ac114d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1011511111 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0a3dd0 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> v2:172.21.15.129:3300/0 conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> v2:172.21.15.129:3300/0 conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4a4007160 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a4016060 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac063d50 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4a4005930 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac1315e0 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.119 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.119 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd4a4005c40 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.119 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4b08c9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.119 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd490027840 0x7fd490029d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.119 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd4a4034cc0 con 0x7fd4ac0a48c0 2019-10-13T05:21:42.120 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4abfff700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd490027840 0x7fd490029d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.120 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.105+0000 7fd4abfff700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd490027840 0x7fd490029d40 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.121 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.113+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] conn(0x7fd4ac1174b0 0x7fd4ac131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.121 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.113+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 --> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4ac0555c0 con 0x7fd4ac1174b0 2019-10-13T05:21:42.121 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.113+0000 7fd4ab7fe700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] conn(0x7fd4ac1174b0 0x7fd4ac131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.121 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.113+0000 7fd4ab7fe700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] conn(0x7fd4ac1174b0 0x7fd4ac131bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.121 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.113+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== osd.0 v2:172.21.15.129:6800/13880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fd4ac0555c0 con 0x7fd4ac1174b0 2019-10-13T05:21:42.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.137+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 --> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4ac05a210 con 0x7fd4ac1174b0 2019-10-13T05:21:42.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4a8ff9700 1 -- 172.21.15.129:0/1011511111 <== osd.0 v2:172.21.15.129:6800/13880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd4ac05a210 con 0x7fd4ac1174b0 2019-10-13T05:21:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] conn(0x7fd4ac1174b0 msgr2=0x7fd4ac131bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:6800/13880,v1:172.21.15.129:6801/13880] conn(0x7fd4ac1174b0 0x7fd4ac131bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd490027840 msgr2=0x7fd490029d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd490027840 0x7fd490029d40 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 msgr2=0x7fd4ac063800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fd4a4007790 tx=0x7fd4a40118b0).stop 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b08c9700 1 -- 172.21.15.129:0/1011511111 reap_dead start 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 shutdown_connections 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 --2- 172.21.15.129:0/1011511111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ac0a48c0 0x7fd4ac063800 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 shutdown_connections 2019-10-13T05:21:42.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 wait complete. 2019-10-13T05:21:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.141+0000 7fd4b1b2b700 1 -- 172.21.15.129:0/1011511111 >> 172.21.15.129:0/1011511111 conn(0x7fd4ac05e770 msgr2=0x7fd4ac116cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.157 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.157 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.158 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-10-13T05:21:42.259 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.245+0000 7f38e5051700 1 Processor -- start 2019-10-13T05:21:42.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.245+0000 7f38e5051700 1 -- start start 2019-10-13T05:21:42.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f38e00a7520 0x7f38e00a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0144a10 0x7f38e0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.260 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e00a7a30 con 0x7f38e00a1c80 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e00a96b0 con 0x7f38e00a7520 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00a9950 con 0x7f38e0144a10 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38df59e700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f38e00a1c80 0x7f38e00a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59072/0 (socket says 172.21.15.129:59072) 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38df59e700 1 -- 172.21.15.129:0/4082101612 learned_addr learned my addr 172.21.15.129:0/4082101612 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> v2:172.21.15.129:3300/0 conn(0x7f38e00a7520 0x7f38e00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e4850700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0144a10 0x7f38e0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 -- 172.21.15.129:0/4082101612 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0144a10 msgr2=0x7f38e0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0144a10 0x7f38e0144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 -- 172.21.15.129:0/4082101612 >> v1:172.21.15.129:6789/0 conn(0x7f38e00a1c80 legacy=0x7f38e00a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 -- 172.21.15.129:0/4082101612 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00aa3e0 con 0x7f38e00a7520 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38de59c700 1 -- 172.21.15.129:0/4082101612 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715594764 0 0) 0x7f38e00a7a30 con 0x7f38e00a1c80 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> v2:172.21.15.129:3300/0 conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> v2:172.21.15.129:3300/0 conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).ready entity=mon.1 client_cookie=eb7104c764213bcf server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38de59c700 1 -- 172.21.15.129:0/4082101612 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38d4008ba0 con 0x7f38e00a7520 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38ded9d700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38de59c700 1 -- 172.21.15.129:0/4082101612 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d40087c0 con 0x7f38e00a7520 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38de59c700 1 -- 172.21.15.129:0/4082101612 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38d4011740 con 0x7f38e00a7520 2019-10-13T05:21:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 -- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 msgr2=0x7f38e00a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.249+0000 7f38e5051700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f38e00a7ea0 tx=0x7f38d8007710).stop 2019-10-13T05:21:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 -- 172.21.15.129:0/4082101612 shutdown_connections 2019-10-13T05:21:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0144a10 0x7f38e0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 --2- 172.21.15.129:0/4082101612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38e00a7520 0x7f38e00a9e90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 -- 172.21.15.129:0/4082101612 shutdown_connections 2019-10-13T05:21:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 -- 172.21.15.129:0/4082101612 wait complete. 2019-10-13T05:21:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 -- 172.21.15.129:0/4082101612 >> 172.21.15.129:0/4082101612 conn(0x7f38e005e870 msgr2=0x7f38e009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 Processor -- start 2019-10-13T05:21:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.253+0000 7f38e5051700 1 -- start start 2019-10-13T05:21:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f38e011cb50 0x7f38e0122a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e0122fc0 con 0x7f38e00a1c80 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e0132af0 con 0x7f38e011cb50 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e009d070 con 0x7f38e0132180 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e4850700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f38e00a1c80 0x7f38e00a20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59080/0 (socket says 172.21.15.129:59080) 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e4850700 1 -- 172.21.15.129:0/3079587516 learned_addr learned my addr 172.21.15.129:0/3079587516 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38df59e700 1 --2- 172.21.15.129:0/3079587516 >> v2:172.21.15.71:3300/0 conn(0x7f38e011cb50 0x7f38e0122a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736791584 0 0) 0x7f38e0122fc0 con 0x7f38e00a1c80 2019-10-13T05:21:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c00030a0 con 0x7f38e00a1c80 2019-10-13T05:21:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 -- 172.21.15.129:0/3079587516 >> v2:172.21.15.71:3300/0 conn(0x7f38e011cb50 msgr2=0x7f38e0122a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> v2:172.21.15.71:3300/0 conn(0x7f38e011cb50 0x7f38e0122a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 -- 172.21.15.129:0/3079587516 >> v1:172.21.15.129:6789/0 conn(0x7f38e00a1c80 legacy=0x7f38e00a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 -- 172.21.15.129:0/3079587516 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00a28a0 con 0x7f38e0132180 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38d400f620 con 0x7f38e0132180 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e0075170 con 0x7f38e0132180 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d4008da0 con 0x7f38e0132180 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38d4005530 con 0x7f38e0132180 2019-10-13T05:21:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e00756a0 con 0x7f38e0132180 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.257+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f38d400f820 con 0x7f38e0132180 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38ded9d700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38c7fff700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f38c0027dd0 0x7f38c002a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f38d4034790 con 0x7f38e0132180 2019-10-13T05:21:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38df59e700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f38c0027dd0 0x7f38c002a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38df59e700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f38c0027dd0 0x7f38c002a2d0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38e5051700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] conn(0x7f38e0075d60 0x7f38e00a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 --> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38e00555c0 con 0x7f38e0075d60 2019-10-13T05:21:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38e4850700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] conn(0x7f38e0075d60 0x7f38e00a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.261+0000 7f38e4850700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] conn(0x7f38e0075d60 0x7f38e00a66c0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.265+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== osd.1 v2:172.21.15.129:6824/13881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f38e00555c0 con 0x7f38e0075d60 2019-10-13T05:21:42.296 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.289+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 --> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f38e011c5f0 con 0x7f38e0075d60 2019-10-13T05:21:42.296 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.289+0000 7f38c7fff700 1 -- 172.21.15.129:0/3079587516 <== osd.1 v2:172.21.15.129:6824/13881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f38e011c5f0 con 0x7f38e0075d60 2019-10-13T05:21:42.298 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 >> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] conn(0x7f38e0075d60 msgr2=0x7f38e00a66c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.298 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.129:6824/13881,v1:172.21.15.129:6825/13881] conn(0x7f38e0075d60 0x7f38e00a66c0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.298 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f38c0027dd0 msgr2=0x7f38c002a2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f38c0027dd0 0x7f38c002a2d0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 msgr2=0x7f38e01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 --2- 172.21.15.129:0/3079587516 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f38e0132180 0x7f38e01325a0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f38d4002ed0 tx=0x7f38d0001500).stop 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e4850700 1 -- 172.21.15.129:0/3079587516 reap_dead start 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 shutdown_connections 2019-10-13T05:21:42.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 shutdown_connections 2019-10-13T05:21:42.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 wait complete. 2019-10-13T05:21:42.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.293+0000 7f38e5051700 1 -- 172.21.15.129:0/3079587516 >> 172.21.15.129:0/3079587516 conn(0x7f38e00a1780 msgr2=0x7f38e0143dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.309 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.309 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.309 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-10-13T05:21:42.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.405+0000 7f5c00597700 1 Processor -- start 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.405+0000 7f5c00597700 1 -- start start 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5bf8144a10 0x7f5bf8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf80a7a30 con 0x7f5bf80a7520 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf80a96b0 con 0x7f5bf8144a10 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf80a9950 con 0x7f5bf80a1c80 2019-10-13T05:21:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfe333700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5bf80a7520 0x7f5bf80a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59088/0 (socket says 172.21.15.129:59088) 2019-10-13T05:21:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfe333700 1 -- 172.21.15.129:0/3712547678 learned_addr learned my addr 172.21.15.129:0/3712547678 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bff335700 1 --2- 172.21.15.129:0/3712547678 >> v2:172.21.15.71:3300/0 conn(0x7f5bf8144a10 0x7f5bf8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726387863 0 0) 0x7f5bf80a7a30 con 0x7f5bf80a7520 2019-10-13T05:21:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be40030a0 con 0x7f5bf80a7520 2019-10-13T05:21:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2386429436 0 0) 0x7f5be40030a0 con 0x7f5bf80a7520 2019-10-13T05:21:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 -- 172.21.15.129:0/3712547678 >> v2:172.21.15.71:3300/0 conn(0x7f5bf8144a10 msgr2=0x7f5bf8144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> v2:172.21.15.71:3300/0 conn(0x7f5bf8144a10 0x7f5bf8144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 -- 172.21.15.129:0/3712547678 >> v1:172.21.15.129:6789/0 conn(0x7f5bf80a7520 legacy=0x7f5bf80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 -- 172.21.15.129:0/3712547678 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf80a1940 con 0x7f5bf80a1c80 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5be8001990 tx=0x7f5be80019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5be8001990 tx=0x7f5be80019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f5be8001990 tx=0x7f5be80019d0).ready entity=mon.2 client_cookie=5f4374118de62b85 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f5be8001990 tx=0x7f5be80019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f5be8001990 tx=0x7f5be80019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f5be8001990 tx=0x7f5be80019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be8001750 con 0x7f5bf80a1c80 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be8001560 con 0x7f5bf80a1c80 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5bfdb32700 1 -- 172.21.15.129:0/3712547678 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bf4005740 con 0x7f5bf80a1c80 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 -- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 msgr2=0x7f5bf80a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.409+0000 7f5c00597700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f5be8001990 tx=0x7f5be80019d0).stop 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- 172.21.15.129:0/3712547678 shutdown_connections 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 --2- 172.21.15.129:0/3712547678 >> v2:172.21.15.71:3300/0 conn(0x7f5bf8144a10 0x7f5bf8144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 --2- 172.21.15.129:0/3712547678 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf80a1c80 0x7f5bf80a13f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- 172.21.15.129:0/3712547678 shutdown_connections 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- 172.21.15.129:0/3712547678 wait complete. 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- 172.21.15.129:0/3712547678 >> 172.21.15.129:0/3712547678 conn(0x7f5bf805e870 msgr2=0x7f5bf809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 Processor -- start 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- start start 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5bf811dcb0 0x7f5bf811d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf81286b0 0x7f5bf8143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf8128b80 con 0x7f5bf8124ad0 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf80a07e0 con 0x7f5bf811dcb0 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5c00597700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811b610 con 0x7f5bf81286b0 2019-10-13T05:21:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5bf811dcb0 0x7f5bf811d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5bf811dcb0 0x7f5bf811d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41124/0 (socket says 172.21.15.129:41124) 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 -- 172.21.15.129:0/237028980 learned_addr learned my addr 172.21.15.129:0/237028980 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfe333700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf81286b0 0x7f5bf8143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 -- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf81286b0 msgr2=0x7f5bf8143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5bf81286b0 0x7f5bf8143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 -- 172.21.15.129:0/237028980 >> v1:172.21.15.129:6789/0 conn(0x7f5bf8124ad0 legacy=0x7f5bf8125c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 -- 172.21.15.129:0/237028980 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf811d570 con 0x7f5bf811dcb0 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> v2:172.21.15.129:3300/0 conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> v2:172.21.15.129:3300/0 conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.413+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bf4007390 con 0x7f5bf811dcb0 2019-10-13T05:21:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf400f430 con 0x7f5bf811dcb0 2019-10-13T05:21:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf81440b0 con 0x7f5bf811dcb0 2019-10-13T05:21:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bf4008d40 con 0x7f5bf811dcb0 2019-10-13T05:21:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf8126440 con 0x7f5bf811dcb0 2019-10-13T05:21:42.433 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfeb34700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5bf400c340 con 0x7f5bf811dcb0 2019-10-13T05:21:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5be0027960 0x7f5be0029e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f5bf4034b60 con 0x7f5bf811dcb0 2019-10-13T05:21:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfe333700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5be0027960 0x7f5be0029e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.417+0000 7f5bfe333700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5be0027960 0x7f5be0029e60 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.421+0000 7f5c00597700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] conn(0x7f5bf8054340 0x7f5bf8054760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.421+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 --> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bf811ce30 con 0x7f5bf8054340 2019-10-13T05:21:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.421+0000 7f5bff335700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] conn(0x7f5bf8054340 0x7f5bf8054760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.421+0000 7f5bff335700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] conn(0x7f5bf8054340 0x7f5bf8054760 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.425+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== osd.2 v2:172.21.15.129:6808/13879 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f5bf0009a10 con 0x7f5bf8054340 2019-10-13T05:21:42.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 --> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5bf80555c0 con 0x7f5bf8054340 2019-10-13T05:21:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5bef7fe700 1 -- 172.21.15.129:0/237028980 <== osd.2 v2:172.21.15.129:6808/13879 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5bf80555c0 con 0x7f5bf8054340 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] conn(0x7f5bf8054340 msgr2=0x7f5bf8054760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:6808/13879,v1:172.21.15.129:6809/13879] conn(0x7f5bf8054340 0x7f5bf8054760 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5be0027960 msgr2=0x7f5be0029e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5be0027960 0x7f5be0029e60 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 msgr2=0x7f5bf811d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f5bf4008360 tx=0x7f5be80010b0).stop 2019-10-13T05:21:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5bff335700 1 -- 172.21.15.129:0/237028980 reap_dead start 2019-10-13T05:21:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 shutdown_connections 2019-10-13T05:21:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 --2- 172.21.15.129:0/237028980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf811dcb0 0x7f5bf811d020 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 shutdown_connections 2019-10-13T05:21:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 wait complete. 2019-10-13T05:21:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.449+0000 7f5c00597700 1 -- 172.21.15.129:0/237028980 >> 172.21.15.129:0/237028980 conn(0x7f5bf805a030 msgr2=0x7f5bf80a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.466 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.467 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.467 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-10-13T05:21:42.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 Processor -- start 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 -- start start 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 0x7f1a4c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c0a7a30 con 0x7f1a4c0a1c80 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c0a96b0 con 0x7f1a4c0a7520 2019-10-13T05:21:42.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a535a6700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0a9950 con 0x7f1a4c144a10 2019-10-13T05:21:42.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41138/0 (socket says 172.21.15.129:41138) 2019-10-13T05:21:42.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 -- 172.21.15.129:0/372491758 learned_addr learned my addr 172.21.15.129:0/372491758 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a52344700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 0x7f1a4c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 -- 172.21.15.129:0/372491758 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 msgr2=0x7f1a4c144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 0x7f1a4c144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 -- 172.21.15.129:0/372491758 >> v1:172.21.15.71:6789/0 conn(0x7f1a4c0a1c80 legacy=0x7f1a4c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 -- 172.21.15.129:0/372491758 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0aa3e0 con 0x7f1a4c0a7520 2019-10-13T05:21:42.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a52344700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 0x7f1a4c144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> v2:172.21.15.129:3300/0 conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> v2:172.21.15.129:3300/0 conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.561+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).ready entity=mon.1 client_cookie=959dbd247764bf10 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a50b41700 1 -- 172.21.15.129:0/372491758 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3c008c80 con 0x7f1a4c0a7520 2019-10-13T05:21:42.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a51342700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a50b41700 1 -- 172.21.15.129:0/372491758 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c0088a0 con 0x7f1a4c0a7520 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a50b41700 1 -- 172.21.15.129:0/372491758 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3c011790 con 0x7f1a4c0a7520 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 -- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 msgr2=0x7f1a4c0a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f1a4c0a7ea0 tx=0x7f1a44007e40).stop 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 -- 172.21.15.129:0/372491758 shutdown_connections 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c144a10 0x7f1a4c144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 --2- 172.21.15.129:0/372491758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a4c0a7520 0x7f1a4c0a9e90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 -- 172.21.15.129:0/372491758 shutdown_connections 2019-10-13T05:21:42.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 -- 172.21.15.129:0/372491758 wait complete. 2019-10-13T05:21:42.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 -- 172.21.15.129:0/372491758 >> 172.21.15.129:0/372491758 conn(0x7f1a4c05e870 msgr2=0x7f1a4c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.569+0000 7f1a535a6700 1 Processor -- start 2019-10-13T05:21:42.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- start start 2019-10-13T05:21:42.586 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.586 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 0x7f1a4c12a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.586 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c079000 con 0x7f1a4c078260 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c07a690 con 0x7f1a4c12a290 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c079540 con 0x7f1a4c079d20 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a52344700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 0x7f1a4c12a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a52344700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 0x7f1a4c12a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41144/0 (socket says 172.21.15.129:41144) 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a52344700 1 -- 172.21.15.129:0/3236098522 learned_addr learned my addr 172.21.15.129:0/3236098522 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.587 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791206380 0 0) 0x7f1a4c079000 con 0x7f1a4c078260 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a340030a0 con 0x7f1a4c078260 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 -- 172.21.15.129:0/3236098522 >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 msgr2=0x7f1a4c12a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 0x7f1a4c12a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 -- 172.21.15.129:0/3236098522 >> v1:172.21.15.71:6789/0 conn(0x7f1a4c078260 legacy=0x7f1a4c078680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 -- 172.21.15.129:0/3236098522 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c11c900 con 0x7f1a4c079d20 2019-10-13T05:21:42.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a52344700 1 --2- 172.21.15.129:0/3236098522 >> v2:172.21.15.129:3300/0 conn(0x7f1a4c12a290 0x7f1a4c12a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:42.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a4800c980 con 0x7f1a4c079d20 2019-10-13T05:21:42.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4800c5a0 con 0x7f1a4c079d20 2019-10-13T05:21:42.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a48010c70 con 0x7f1a4c079d20 2019-10-13T05:21:42.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c12ac00 con 0x7f1a4c079d20 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.573+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c1316e0 con 0x7f1a4c079d20 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a51b43700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1a48007420 con 0x7f1a4c079d20 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a427fc700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1a34027df0 0x7f1a3402a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f1a48034b30 con 0x7f1a4c079d20 2019-10-13T05:21:42.591 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a51342700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1a34027df0 0x7f1a3402a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.577+0000 7f1a51342700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1a34027df0 0x7f1a3402a2f0 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.593+0000 7f1a535a6700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] conn(0x7f1a4c07ce60 0x7f1a4c054340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.599 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.593+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 --> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a4c0555c0 con 0x7f1a4c07ce60 2019-10-13T05:21:42.599 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.593+0000 7f1a52344700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] conn(0x7f1a4c07ce60 0x7f1a4c054340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.599 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.593+0000 7f1a52344700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] conn(0x7f1a4c07ce60 0x7f1a4c054340 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.600 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.593+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== osd.3 v2:172.21.15.129:6816/13902 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f1a4c0555c0 con 0x7f1a4c07ce60 2019-10-13T05:21:42.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.621+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 --> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a4c07d280 con 0x7f1a4c07ce60 2019-10-13T05:21:42.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a427fc700 1 -- 172.21.15.129:0/3236098522 <== osd.3 v2:172.21.15.129:6816/13902 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a4c07d280 con 0x7f1a4c07ce60 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 >> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] conn(0x7f1a4c07ce60 msgr2=0x7f1a4c054340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.129:6816/13902,v1:172.21.15.129:6817/13902] conn(0x7f1a4c07ce60 0x7f1a4c054340 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1a34027df0 msgr2=0x7f1a3402a2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f1a34027df0 0x7f1a3402a2f0 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 msgr2=0x7f1a4c07a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 --2- 172.21.15.129:0/3236098522 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1a4c079d20 0x7f1a4c07a140 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f1a48001430 tx=0x7f1a4400b640).stop 2019-10-13T05:21:42.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a52344700 1 -- 172.21.15.129:0/3236098522 reap_dead start 2019-10-13T05:21:42.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 shutdown_connections 2019-10-13T05:21:42.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 shutdown_connections 2019-10-13T05:21:42.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 wait complete. 2019-10-13T05:21:42.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.625+0000 7f1a535a6700 1 -- 172.21.15.129:0/3236098522 >> 172.21.15.129:0/3236098522 conn(0x7f1a4c07ec10 msgr2=0x7f1a4c0a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.641 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.642 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.642 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-10-13T05:21:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 Processor -- start 2019-10-13T05:21:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 -- start start 2019-10-13T05:21:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 0x7f65d00a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65d00a7520 0x7f65d00a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00a7a30 con 0x7f65d0144a10 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00a96b0 con 0x7f65d00a1c80 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00a9950 con 0x7f65d00a7520 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65d00a7520 0x7f65d00a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d757b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f65d0144a10 0x7f65d00a60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59124/0 (socket says 172.21.15.129:59124) 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d757b700 1 -- 172.21.15.129:0/3236191438 learned_addr learned my addr 172.21.15.129:0/3236191438 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 0x7f65d00a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 -- 172.21.15.129:0/3236191438 >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 msgr2=0x7f65d00a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 0x7f65d00a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 -- 172.21.15.129:0/3236191438 >> v1:172.21.15.129:6789/0 conn(0x7f65d0144a10 legacy=0x7f65d00a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d5d78700 1 -- 172.21.15.129:0/3236191438 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746539313 0 0) 0x7f65d00a7a30 con 0x7f65d0144a10 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 -- 172.21.15.129:0/3236191438 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00a1940 con 0x7f65d00a7520 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 0x7f65d00a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.129:3300/0 conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f65cc0032b0 tx=0x7f65c4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.129:3300/0 conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f65cc0032b0 tx=0x7f65c4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f65cc0032b0 tx=0x7f65c4007710).ready entity=mon.1 client_cookie=a9be2672dea6b18c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f65cc0032b0 tx=0x7f65c4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d5d78700 1 -- 172.21.15.129:0/3236191438 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65cc011890 con 0x7f65d00a7520 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f65cc0032b0 tx=0x7f65c4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d5d78700 1 -- 172.21.15.129:0/3236191438 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc008b40 con 0x7f65d00a7520 2019-10-13T05:21:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d6579700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f65cc0032b0 tx=0x7f65c4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d5d78700 1 -- 172.21.15.129:0/3236191438 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65cc0103b0 con 0x7f65d00a7520 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 -- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 msgr2=0x7f65d00a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.737+0000 7f65d87dd700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f65cc0032b0 tx=0x7f65c4007710).stop 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 -- 172.21.15.129:0/3236191438 shutdown_connections 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 --2- 172.21.15.129:0/3236191438 >> v2:172.21.15.71:3300/0 conn(0x7f65d00a1c80 0x7f65d00a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 --2- 172.21.15.129:0/3236191438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d00a7520 0x7f65d00a9e90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 -- 172.21.15.129:0/3236191438 shutdown_connections 2019-10-13T05:21:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 -- 172.21.15.129:0/3236191438 wait complete. 2019-10-13T05:21:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.741+0000 7f65d87dd700 1 -- 172.21.15.129:0/3236191438 >> 172.21.15.129:0/3236191438 conn(0x7f65d005e870 msgr2=0x7f65d009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.756 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 Processor -- start 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- start start 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65d009fd40 0x7f65d009da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65d0122730 0x7f65d0132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d0132cb0 con 0x7f65d0132890 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00a0240 con 0x7f65d009fd40 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00a04e0 con 0x7f65d0122730 2019-10-13T05:21:42.757 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65d009fd40 0x7f65d009da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65d009fd40 0x7f65d009da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41156/0 (socket says 172.21.15.129:41156) 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 -- 172.21.15.129:0/225727906 learned_addr learned my addr 172.21.15.129:0/225727906 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6579700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65d0122730 0x7f65d0132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 -- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65d0122730 msgr2=0x7f65d0132340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65d0122730 0x7f65d0132340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 -- 172.21.15.129:0/225727906 >> v1:172.21.15.71:6789/0 conn(0x7f65d0132890 legacy=0x7f65d00a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.758 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 -- 172.21.15.129:0/225727906 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00a6dc0 con 0x7f65d009fd40 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6579700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f65d0122730 0x7f65d0132340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> v2:172.21.15.129:3300/0 conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> v2:172.21.15.129:3300/0 conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65c4012600 con 0x7f65d009fd40 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4008bc0 con 0x7f65d009fd40 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65c4012be0 con 0x7f65d009fd40 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d00a2480 con 0x7f65d009fd40 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.745+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d00a2a30 con 0x7f65d009fd40 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.760 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65d6d7a700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f65c40055b0 con 0x7f65d009fd40 2019-10-13T05:21:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65c37fe700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f65b80278f0 0x7f65b8029df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f65c4034b20 con 0x7f65d009fd40 2019-10-13T05:21:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65d6579700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f65b80278f0 0x7f65b8029df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.749+0000 7f65d6579700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f65b80278f0 0x7f65b8029df0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.753+0000 7f65d87dd700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] conn(0x7f65d0054340 0x7f65d0054760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.753+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 --> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f65d00555c0 con 0x7f65d0054340 2019-10-13T05:21:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.753+0000 7f65d757b700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] conn(0x7f65d0054340 0x7f65d0054760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.763 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.753+0000 7f65d757b700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] conn(0x7f65d0054340 0x7f65d0054760 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.763 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.757+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== osd.4 v2:172.21.15.71:6826/14134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f65d00555c0 con 0x7f65d0054340 2019-10-13T05:21:42.787 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 --> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f65d007e920 con 0x7f65d0054340 2019-10-13T05:21:42.787 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65c37fe700 1 -- 172.21.15.129:0/225727906 <== osd.4 v2:172.21.15.71:6826/14134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f65d007e920 con 0x7f65d0054340 2019-10-13T05:21:42.788 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] conn(0x7f65d0054340 msgr2=0x7f65d0054760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6826/14134,v1:172.21.15.71:6827/14134] conn(0x7f65d0054340 0x7f65d0054760 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f65b80278f0 msgr2=0x7f65b8029df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f65b80278f0 0x7f65b8029df0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 msgr2=0x7f65d009da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f65d00a0400 tx=0x7f65cc008500).stop 2019-10-13T05:21:42.789 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d757b700 1 -- 172.21.15.129:0/225727906 reap_dead start 2019-10-13T05:21:42.790 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 shutdown_connections 2019-10-13T05:21:42.790 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 --2- 172.21.15.129:0/225727906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65d009fd40 0x7f65d009da20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.790 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 shutdown_connections 2019-10-13T05:21:42.790 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 wait complete. 2019-10-13T05:21:42.790 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.781+0000 7f65d87dd700 1 -- 172.21.15.129:0/225727906 >> 172.21.15.129:0/225727906 conn(0x7f65d007e4c0 msgr2=0x7f65d00a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.799 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.799 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.799 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-10-13T05:21:42.907 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 Processor -- start 2019-10-13T05:21:42.907 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 -- start start 2019-10-13T05:21:42.907 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a1c80 0x7f5b580a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.907 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58144a10 0x7f5b58144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b580a7a30 con 0x7f5b580a7520 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580a96b0 con 0x7f5b580a1c80 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5f990700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580a9950 con 0x7f5b58144a10 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a1c80 0x7f5b580a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a1c80 0x7f5b580a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41166/0 (socket says 172.21.15.129:41166) 2019-10-13T05:21:42.908 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 -- 172.21.15.129:0/3179433010 learned_addr learned my addr 172.21.15.129:0/3179433010 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5e72e700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58144a10 0x7f5b58144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 -- 172.21.15.129:0/3179433010 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58144a10 msgr2=0x7f5b58144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58144a10 0x7f5b58144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 -- 172.21.15.129:0/3179433010 >> v1:172.21.15.71:6789/0 conn(0x7f5b580a7520 legacy=0x7f5b580a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 -- 172.21.15.129:0/3179433010 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580a1940 con 0x7f5b580a1c80 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> v2:172.21.15.129:3300/0 conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> v2:172.21.15.129:3300/0 conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.897+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).ready entity=mon.1 client_cookie=e78372ccafc19439 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5cf2b700 1 -- 172.21.15.129:0/3179433010 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54008c80 con 0x7f5b580a1c80 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5cf2b700 1 -- 172.21.15.129:0/3179433010 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b540088a0 con 0x7f5b580a1c80 2019-10-13T05:21:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5cf2b700 1 -- 172.21.15.129:0/3179433010 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b540119f0 con 0x7f5b580a1c80 2019-10-13T05:21:42.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 msgr2=0x7f5b580a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f5b580a7ea0 tx=0x7f5b50007a90).stop 2019-10-13T05:21:42.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- 172.21.15.129:0/3179433010 shutdown_connections 2019-10-13T05:21:42.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58144a10 0x7f5b58144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:42.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- 172.21.15.129:0/3179433010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a1c80 0x7f5b580a13f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- 172.21.15.129:0/3179433010 shutdown_connections 2019-10-13T05:21:42.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- 172.21.15.129:0/3179433010 wait complete. 2019-10-13T05:21:42.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- 172.21.15.129:0/3179433010 >> 172.21.15.129:0/3179433010 conn(0x7f5b5805e870 msgr2=0x7f5b5809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 Processor -- start 2019-10-13T05:21:42.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- start start 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a8790 0x7f5b580a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58141b30 0x7f5b58141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5b580a51d0 0x7f5b580a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580a40e0 con 0x7f5b580a51d0 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580a4380 con 0x7f5b580a8790 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5f990700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580a4620 con 0x7f5b58141b30 2019-10-13T05:21:42.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a8790 0x7f5b580a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5b580a8790 0x7f5b580a6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41176/0 (socket says 172.21.15.129:41176) 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5df2d700 1 -- 172.21.15.129:0/1924473945 learned_addr learned my addr 172.21.15.129:0/1924473945 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5e72e700 1 --2- 172.21.15.129:0/1924473945 >> v2:172.21.15.71:3300/0 conn(0x7f5b580a51d0 0x7f5b580a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.901+0000 7f5b5d72c700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58141b30 0x7f5b58141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 -- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58141b30 msgr2=0x7f5b58141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f5b58141b30 0x7f5b58141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 -- 172.21.15.129:0/1924473945 >> v2:172.21.15.71:3300/0 conn(0x7f5b580a51d0 msgr2=0x7f5b580a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> v2:172.21.15.71:3300/0 conn(0x7f5b580a51d0 0x7f5b580a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 -- 172.21.15.129:0/1924473945 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580a3dd0 con 0x7f5b580a8790 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> v2:172.21.15.129:3300/0 conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> v2:172.21.15.129:3300/0 conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54007350 con 0x7f5b580a8790 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54011610 con 0x7f5b580a8790 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b581424a0 con 0x7f5b580a8790 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54005910 con 0x7f5b580a8790 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b580a5b40 con 0x7f5b580a8790 2019-10-13T05:21:42.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5b54005c20 con 0x7f5b580a8790 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5df2d700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5b40027950 0x7f5b40029e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f5b54034c20 con 0x7f5b580a8790 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5d72c700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5b40027950 0x7f5b40029e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.905+0000 7f5b5d72c700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5b40027950 0x7f5b40029e50 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.913+0000 7f5b5f990700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] conn(0x7f5b58122990 0x7f5b58122db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:42.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.913+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 --> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b58142fb0 con 0x7f5b58122990 2019-10-13T05:21:42.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.913+0000 7f5b5e72e700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] conn(0x7f5b58122990 0x7f5b58122db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:42.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.913+0000 7f5b5e72e700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] conn(0x7f5b58122990 0x7f5b58122db0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:42.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.913+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== osd.5 v2:172.21.15.71:6818/14132 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f5b58142fb0 con 0x7f5b58122990 2019-10-13T05:21:42.950 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 --> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b58123570 con 0x7f5b58122990 2019-10-13T05:21:42.951 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b4e7fc700 1 -- 172.21.15.129:0/1924473945 <== osd.5 v2:172.21.15.71:6818/14132 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5b58123570 con 0x7f5b58122990 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] conn(0x7f5b58122990 msgr2=0x7f5b58122db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6818/14132,v1:172.21.15.71:6819/14132] conn(0x7f5b58122990 0x7f5b58122db0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5b40027950 msgr2=0x7f5b40029e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5b40027950 0x7f5b40029e50 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 msgr2=0x7f5b580a6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f5b580a42a0 tx=0x7f5b500078d0).stop 2019-10-13T05:21:42.954 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5e72e700 1 -- 172.21.15.129:0/1924473945 reap_dead start 2019-10-13T05:21:42.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 shutdown_connections 2019-10-13T05:21:42.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 --2- 172.21.15.129:0/1924473945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b580a8790 0x7f5b580a6ba0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:42.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 shutdown_connections 2019-10-13T05:21:42.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 wait complete. 2019-10-13T05:21:42.955 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:42.945+0000 7f5b5f990700 1 -- 172.21.15.129:0/1924473945 >> 172.21.15.129:0/1924473945 conn(0x7f5b580a7520 msgr2=0x7f5b5807db00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:42.963 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:42.964 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-10-13T05:21:43.070 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 Processor -- start 2019-10-13T05:21:43.070 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 -- start start 2019-10-13T05:21:43.070 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89d80a1c80 0x7f89d80a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f89d8144a10 0x7f89d8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d80a7a30 con 0x7f89d80a7520 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d80a96b0 con 0x7f89d80a1c80 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89dd621700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d80a9950 con 0x7f89d8144a10 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89d7fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f89d8144a10 0x7f89d8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.071 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89d7fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f89d8144a10 0x7f89d8144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41186/0 (socket says 172.21.15.129:41186) 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89d7fff700 1 -- 172.21.15.129:0/3103526920 learned_addr learned my addr 172.21.15.129:0/3103526920 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.057+0000 7f89d77fe700 1 --2- 172.21.15.129:0/3103526920 >> v2:172.21.15.71:3300/0 conn(0x7f89d80a1c80 0x7f89d80a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 -- 172.21.15.129:0/3103526920 >> v2:172.21.15.71:3300/0 conn(0x7f89d80a1c80 msgr2=0x7f89d80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> v2:172.21.15.71:3300/0 conn(0x7f89d80a1c80 0x7f89d80a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 -- 172.21.15.129:0/3103526920 >> v1:172.21.15.71:6789/0 conn(0x7f89d80a7520 legacy=0x7f89d80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 -- 172.21.15.129:0/3103526920 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d80a1940 con 0x7f89d8144a10 2019-10-13T05:21:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> v2:172.21.15.129:3300/0 conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f89c8002f30 tx=0x7f89c0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> v2:172.21.15.129:3300/0 conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f89c8002f30 tx=0x7f89c0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f89c8002f30 tx=0x7f89c0007710).ready entity=mon.1 client_cookie=91c480b3b0ab6f73 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f89c8002f30 tx=0x7f89c0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f89c8002f30 tx=0x7f89c0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d67fc700 1 -- 172.21.15.129:0/3103526920 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c8008c80 con 0x7f89d8144a10 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d7fff700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f89c8002f30 tx=0x7f89c0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d67fc700 1 -- 172.21.15.129:0/3103526920 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c80088a0 con 0x7f89d8144a10 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89d67fc700 1 -- 172.21.15.129:0/3103526920 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c80118c0 con 0x7f89d8144a10 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 -- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 msgr2=0x7f89d8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f89c8002f30 tx=0x7f89c0007710).stop 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 -- 172.21.15.129:0/3103526920 shutdown_connections 2019-10-13T05:21:43.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 --2- 172.21.15.129:0/3103526920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89d8144a10 0x7f89d8144e30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 --2- 172.21.15.129:0/3103526920 >> v2:172.21.15.71:3300/0 conn(0x7f89d80a1c80 0x7f89d80a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 -- 172.21.15.129:0/3103526920 shutdown_connections 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 -- 172.21.15.129:0/3103526920 wait complete. 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.061+0000 7f89dd621700 1 -- 172.21.15.129:0/3103526920 >> 172.21.15.129:0/3103526920 conn(0x7f89d805e870 msgr2=0x7f89d809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 Processor -- start 2019-10-13T05:21:43.075 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- start start 2019-10-13T05:21:43.078 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89d809f790 0x7f89d811df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.079 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f89d8073160 0x7f89d8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.079 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d811ee50 con 0x7f89d811e490 2019-10-13T05:21:43.079 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d809d070 con 0x7f89d809f790 2019-10-13T05:21:43.079 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d809fc60 con 0x7f89d8073160 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d6ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f89d811e490 0x7f89d809ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59160/0 (socket says 172.21.15.129:59160) 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d6ffd700 1 -- 172.21.15.129:0/34146213 learned_addr learned my addr 172.21.15.129:0/34146213 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> v2:172.21.15.71:3300/0 conn(0x7f89d809f790 0x7f89d811df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d7fff700 1 --2- 172.21.15.129:0/34146213 >> v2:172.21.15.129:3300/0 conn(0x7f89d8073160 0x7f89d8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593240332 0 0) 0x7f89d811ee50 con 0x7f89d811e490 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b00030a0 con 0x7f89d811e490 2019-10-13T05:21:43.080 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 -- 172.21.15.129:0/34146213 >> v2:172.21.15.129:3300/0 conn(0x7f89d8073160 msgr2=0x7f89d8075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> v2:172.21.15.129:3300/0 conn(0x7f89d8073160 0x7f89d8075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 -- 172.21.15.129:0/34146213 >> v1:172.21.15.129:6789/0 conn(0x7f89d811e490 legacy=0x7f89d809ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 -- 172.21.15.129:0/34146213 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d80a0780 con 0x7f89d809f790 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> v2:172.21.15.71:3300/0 conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> v2:172.21.15.71:3300/0 conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.081 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c0012740 con 0x7f89d809f790 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d811b910 con 0x7f89d809f790 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0007150 con 0x7f89d809f790 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c0012d20 con 0x7f89d809f790 2019-10-13T05:21:43.082 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d8075ae0 con 0x7f89d809f790 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.065+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f89c0010170 con 0x7f89d809f790 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89d77fe700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89cffff700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f89b0027dd0 0x7f89b002a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f89c0034d00 con 0x7f89d809f790 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89d6ffd700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f89b0027dd0 0x7f89b002a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.083 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.069+0000 7f89d6ffd700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f89b0027dd0 0x7f89b002a2d0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.084 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.073+0000 7f89dd621700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] conn(0x7f89d8054340 0x7f89d8054760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.084 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.073+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 --> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89d80555c0 con 0x7f89d8054340 2019-10-13T05:21:43.085 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.073+0000 7f89d7fff700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] conn(0x7f89d8054340 0x7f89d8054760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.085 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.073+0000 7f89d7fff700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] conn(0x7f89d8054340 0x7f89d8054760 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.085 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.077+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== osd.6 v2:172.21.15.71:6810/14133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f89d80555c0 con 0x7f89d8054340 2019-10-13T05:21:43.106 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 --> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f89d805a210 con 0x7f89d8054340 2019-10-13T05:21:43.107 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89cffff700 1 -- 172.21.15.129:0/34146213 <== osd.6 v2:172.21.15.71:6810/14133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f89d805a210 con 0x7f89d8054340 2019-10-13T05:21:43.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] conn(0x7f89d8054340 msgr2=0x7f89d8054760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.108 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6810/14133,v1:172.21.15.71:6811/14133] conn(0x7f89d8054340 0x7f89d8054760 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f89b0027dd0 msgr2=0x7f89b002a2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f89b0027dd0 0x7f89b002a2d0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 msgr2=0x7f89d811df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7f89c0002f30 tx=0x7f89c0002ff0).stop 2019-10-13T05:21:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89d7fff700 1 -- 172.21.15.129:0/34146213 reap_dead start 2019-10-13T05:21:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 shutdown_connections 2019-10-13T05:21:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 --2- 172.21.15.129:0/34146213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89d809f790 0x7f89d811df40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 shutdown_connections 2019-10-13T05:21:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 wait complete. 2019-10-13T05:21:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.101+0000 7f89dd621700 1 -- 172.21.15.129:0/34146213 >> 172.21.15.129:0/34146213 conn(0x7f89d80a6c00 msgr2=0x7f89d80a61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.118 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:43.119 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:43.119 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-10-13T05:21:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 Processor -- start 2019-10-13T05:21:43.225 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 -- start start 2019-10-13T05:21:43.225 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fec0c0a7520 0x7fec0c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.225 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c144a10 0x7fec0c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.225 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0a7a30 con 0x7fec0c0a1c80 2019-10-13T05:21:43.225 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c0a96b0 con 0x7fec0c144a10 2019-10-13T05:21:43.226 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec11cb1700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0a9950 con 0x7fec0c0a7520 2019-10-13T05:21:43.226 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec10a4f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c144a10 0x7fec0c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.226 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec10a4f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c144a10 0x7fec0c144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41202/0 (socket says 172.21.15.129:41202) 2019-10-13T05:21:43.226 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec10a4f700 1 -- 172.21.15.129:0/738789535 learned_addr learned my addr 172.21.15.129:0/738789535 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.213+0000 7fec0b7fe700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fec0c0a7520 0x7fec0c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 -- 172.21.15.129:0/738789535 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fec0c0a7520 msgr2=0x7fec0c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fec0c0a7520 0x7fec0c0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 -- 172.21.15.129:0/738789535 >> v1:172.21.15.71:6789/0 conn(0x7fec0c0a1c80 legacy=0x7fec0c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 -- 172.21.15.129:0/738789535 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c0aa3e0 con 0x7fec0c144a10 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> v2:172.21.15.129:3300/0 conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fec0c0a9870 tx=0x7fec00007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.227 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> v2:172.21.15.129:3300/0 conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fec0c0a9870 tx=0x7fec00007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7fec0c0a9870 tx=0x7fec00007710).ready entity=mon.1 client_cookie=69c1f7e453ed08a1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7fec0c0a9870 tx=0x7fec00007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7fec0c0a9870 tx=0x7fec00007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec0affd700 1 -- 172.21.15.129:0/738789535 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec04010b90 con 0x7fec0c144a10 2019-10-13T05:21:43.228 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec10a4f700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7fec0c0a9870 tx=0x7fec00007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec0affd700 1 -- 172.21.15.129:0/738789535 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec040107b0 con 0x7fec0c144a10 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec0affd700 1 -- 172.21.15.129:0/738789535 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec040112b0 con 0x7fec0c144a10 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 -- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 msgr2=0x7fec0c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7fec0c0a9870 tx=0x7fec00007710).stop 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 -- 172.21.15.129:0/738789535 shutdown_connections 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c144a10 0x7fec0c144e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 --2- 172.21.15.129:0/738789535 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fec0c0a7520 0x7fec0c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 -- 172.21.15.129:0/738789535 shutdown_connections 2019-10-13T05:21:43.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 -- 172.21.15.129:0/738789535 wait complete. 2019-10-13T05:21:43.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.217+0000 7fec11cb1700 1 -- 172.21.15.129:0/738789535 >> 172.21.15.129:0/738789535 conn(0x7fec0c05e870 msgr2=0x7fec0c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 Processor -- start 2019-10-13T05:21:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 -- start start 2019-10-13T05:21:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c0a8280 0x7fec0c111d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fec0c128d00 0x7fec0c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0a6dd0 con 0x7fec0c09f5e0 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c0a3dd0 con 0x7fec0c128d00 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec11cb1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c09fb60 con 0x7fec0c0a8280 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c0a8280 0x7fec0c111d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec10a4f700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fec0c09f5e0 0x7fec0c0a65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59178/0 (socket says 172.21.15.129:59178) 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fec0c0a8280 0x7fec0c111d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41204/0 (socket says 172.21.15.129:41204) 2019-10-13T05:21:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec10a4f700 1 -- 172.21.15.129:0/250789877 learned_addr learned my addr 172.21.15.129:0/250789877 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0b7fe700 1 --2- 172.21.15.129:0/250789877 >> v2:172.21.15.71:3300/0 conn(0x7fec0c128d00 0x7fec0c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 -- 172.21.15.129:0/250789877 >> v2:172.21.15.71:3300/0 conn(0x7fec0c128d00 msgr2=0x7fec0c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> v2:172.21.15.71:3300/0 conn(0x7fec0c128d00 0x7fec0c0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030210605 0 0) 0x7fec0c0a6dd0 con 0x7fec0c09f5e0 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 -- 172.21.15.129:0/250789877 >> v1:172.21.15.129:6789/0 conn(0x7fec0c09f5e0 legacy=0x7fec0c0a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 -- 172.21.15.129:0/250789877 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c09fe00 con 0x7fec0c0a8280 2019-10-13T05:21:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> v2:172.21.15.129:3300/0 conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> v2:172.21.15.129:3300/0 conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.221+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.250 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febfc00ca70 con 0x7fec0c0a8280 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00c690 con 0x7fec0c0a8280 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c1159e0 con 0x7fec0c0a8280 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febfc013140 con 0x7fec0c0a8280 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c1122b0 con 0x7fec0c0a8280 2019-10-13T05:21:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7febfc013450 con 0x7fec0c0a8280 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0bfff700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7febec027870 0x7febec029d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7febfc036ad0 con 0x7fec0c0a8280 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0b7fe700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7febec027870 0x7febec029d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.225+0000 7fec0b7fe700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7febec027870 0x7febec029d70 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.252 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.229+0000 7fec11cb1700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] conn(0x7fec0c0a57c0 0x7fec0c054340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.229+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 --> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec0c0555c0 con 0x7fec0c0a57c0 2019-10-13T05:21:43.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.229+0000 7fec10a4f700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] conn(0x7fec0c0a57c0 0x7fec0c054340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.229+0000 7fec10a4f700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] conn(0x7fec0c0a57c0 0x7fec0c054340 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.253 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.229+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== osd.7 v2:172.21.15.71:6802/14135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fec0c0555c0 con 0x7fec0c0a57c0 2019-10-13T05:21:43.261 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.253+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 --> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec0c0a5e80 con 0x7fec0c0a57c0 2019-10-13T05:21:43.262 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec08ff9700 1 -- 172.21.15.129:0/250789877 <== osd.7 v2:172.21.15.71:6802/14135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fec0c0a5e80 con 0x7fec0c0a57c0 2019-10-13T05:21:43.263 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] conn(0x7fec0c0a57c0 msgr2=0x7fec0c054340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6802/14135,v1:172.21.15.71:6803/14135] conn(0x7fec0c0a57c0 0x7fec0c054340 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7febec027870 msgr2=0x7febec029d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7febec027870 0x7febec029d70 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 msgr2=0x7fec0c111d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7febfc000e90 tx=0x7fec00007550).stop 2019-10-13T05:21:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec10a4f700 1 -- 172.21.15.129:0/250789877 reap_dead start 2019-10-13T05:21:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 shutdown_connections 2019-10-13T05:21:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 --2- 172.21.15.129:0/250789877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c0a8280 0x7fec0c111d60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 shutdown_connections 2019-10-13T05:21:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 wait complete. 2019-10-13T05:21:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.257+0000 7fec11cb1700 1 -- 172.21.15.129:0/250789877 >> 172.21.15.129:0/250789877 conn(0x7fec0c0a87d0 msgr2=0x7fec0c1190a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.273 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:43.274 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:43.274 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-13T05:21:43.379 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 Processor -- start 2019-10-13T05:21:43.379 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 -- start start 2019-10-13T05:21:43.379 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 0x7efc1c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc1c144a10 0x7efc1c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0a7a30 con 0x7efc1c0a1c80 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0a96b0 con 0x7efc1c144a10 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc2284b700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0a9950 con 0x7efc1c0a7520 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc1bfff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 0x7efc1c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc1c144a10 0x7efc1c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc1c144a10 0x7efc1c144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.129:43050/0 (socket says 172.21.15.129:43050) 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc1bfff700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 0x7efc1c0a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.129:44114/0 (socket says 172.21.15.129:44114) 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 -- 172.21.15.129:0/868584610 learned_addr learned my addr 172.21.15.129:0/868584610 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc20de8700 1 --1- 172.21.15.129:0/868584610 >> v1:172.21.15.71:6789/0 conn(0x7efc1c0a1c80 0x7efc1c0a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:53774/0 (socket says 172.21.15.129:53774) 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 -- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 msgr2=0x7efc1c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 0x7efc1c0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 -- 172.21.15.129:0/868584610 >> v1:172.21.15.71:6789/0 conn(0x7efc1c0a1c80 legacy=0x7efc1c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 -- 172.21.15.129:0/868584610 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0aa3e0 con 0x7efc1c144a10 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> v2:172.21.15.71:3300/0 conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc04001110 tx=0x7efc04001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> v2:172.21.15.71:3300/0 conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc04001110 tx=0x7efc04001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.369+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7efc04001110 tx=0x7efc04001150).ready entity=mon.0 client_cookie=83b16473d7b07683 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7efc04001110 tx=0x7efc04001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc1b7fe700 1 -- 172.21.15.129:0/868584610 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc140072b0 con 0x7efc1c144a10 2019-10-13T05:21:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7efc04001110 tx=0x7efc04001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc1b7fe700 1 -- 172.21.15.129:0/868584610 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc14006cd0 con 0x7efc1c144a10 2019-10-13T05:21:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc215e9700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7efc04001110 tx=0x7efc04001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 msgr2=0x7efc1c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc1b7fe700 1 -- 172.21.15.129:0/868584610 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc14007890 con 0x7efc1c144a10 2019-10-13T05:21:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7efc04001110 tx=0x7efc04001150).stop 2019-10-13T05:21:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- 172.21.15.129:0/868584610 shutdown_connections 2019-10-13T05:21:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1c144a10 0x7efc1c144e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 --2- 172.21.15.129:0/868584610 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c0a7520 0x7efc1c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- 172.21.15.129:0/868584610 shutdown_connections 2019-10-13T05:21:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- 172.21.15.129:0/868584610 wait complete. 2019-10-13T05:21:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- 172.21.15.129:0/868584610 >> 172.21.15.129:0/868584610 conn(0x7efc1c05e870 msgr2=0x7efc1c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 Processor -- start 2019-10-13T05:21:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- start start 2019-10-13T05:21:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0a07c0 con 0x7efc1c09f490 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0a04a0 con 0x7efc1c061160 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.373+0000 7efc2284b700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c05f8a0 con 0x7efc1c061580 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc1bfff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7efc1c061160 0x7efc1c1183b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59194/0 (socket says 172.21.15.129:59194) 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc1bfff700 1 -- 172.21.15.129:0/1152434352 learned_addr learned my addr 172.21.15.129:0/1152434352 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999776849 0 0) 0x7efc1c0a04a0 con 0x7efc1c061160 2019-10-13T05:21:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc000030a0 con 0x7efc1c061160 2019-10-13T05:21:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695246784 0 0) 0x7efc04001150 con 0x7efc1c09f490 2019-10-13T05:21:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c0a04a0 con 0x7efc1c09f490 2019-10-13T05:21:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 -- 172.21.15.129:0/1152434352 >> v1:172.21.15.129:6789/0 conn(0x7efc1c061160 legacy=0x7efc1c1183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 -- 172.21.15.129:0/1152434352 >> v1:172.21.15.71:6789/0 conn(0x7efc1c09f490 legacy=0x7efc1c07d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 -- 172.21.15.129:0/1152434352 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c142a00 con 0x7efc1c061580 2019-10-13T05:21:43.394 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.394 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.394 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc14056860 con 0x7efc1c061580 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc140074b0 con 0x7efc1c061580 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.395 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c05f440 con 0x7efc1c061580 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc14011450 con 0x7efc1c061580 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c11c120 con 0x7efc1c061580 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7efc140117b0 con 0x7efc1c061580 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 0x7efc0002a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.396 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.377+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7efc14079510 con 0x7efc1c061580 2019-10-13T05:21:43.397 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.381+0000 7efc20de8700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 0x7efc0002a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.397 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.381+0000 7efc20de8700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 0x7efc0002a460 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.397 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.385+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c0555c0 con 0x7efc1c061580 2019-10-13T05:21:43.400 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.393+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.400 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.393+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7efc14011aa0 con 0x7efc1c061580 2019-10-13T05:21:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.557+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7efc1c144330 con 0x7efc1c061580 2019-10-13T05:21:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.557+0000 7efc215e9700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.557+0000 7efc197fa700 1 -- 172.21.15.129:0/1152434352 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc14011cc0 con 0x7efc1c061580 2019-10-13T05:21:43.565 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2019-10-13T05:21:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 msgr2=0x7efc0002a460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 0x7efc0002a460 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 msgr2=0x7efc1c11bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7efc14008b90 tx=0x7efc1400f670).stop 2019-10-13T05:21:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 shutdown_connections 2019-10-13T05:21:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7efc00027f60 0x7efc0002a460 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 --2- 172.21.15.129:0/1152434352 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7efc1c061580 0x7efc1c11bbd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 shutdown_connections 2019-10-13T05:21:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 wait complete. 2019-10-13T05:21:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.561+0000 7efc2284b700 1 -- 172.21.15.129:0/1152434352 >> 172.21.15.129:0/1152434352 conn(0x7efc1c143320 msgr2=0x7efc1c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2019-10-13T05:21:43.576 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:43.576 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-13T05:21:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.657+0000 7f4a7d35e700 1 Processor -- start 2019-10-13T05:21:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.657+0000 7f4a7d35e700 1 -- start start 2019-10-13T05:21:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a780a1c80 0x7f4a780a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780a7a30 con 0x7f4a780a7520 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780a1940 con 0x7f4a78144a10 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a780a96b0 con 0x7f4a780a1c80 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a780a1c80 0x7f4a780a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a780a1c80 0x7f4a780a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41230/0 (socket says 172.21.15.129:41230) 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 -- 172.21.15.129:0/4071784517 learned_addr learned my addr 172.21.15.129:0/4071784517 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 -- 172.21.15.129:0/4071784517 >> v1:172.21.15.129:6789/0 conn(0x7f4a78144a10 legacy=0x7f4a7814eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 -- 172.21.15.129:0/4071784517 >> v1:172.21.15.71:6789/0 conn(0x7f4a780a7520 legacy=0x7f4a780a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 -- 172.21.15.129:0/4071784517 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78144e30 con 0x7f4a780a1c80 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> v2:172.21.15.129:3300/0 conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a70008b00 tx=0x7f4a70010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> v2:172.21.15.129:3300/0 conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a70008b00 tx=0x7f4a70010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f4a70008b00 tx=0x7f4a70010020).ready entity=mon.1 client_cookie=aee7e88cbb0bdeb6 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f4a70008b00 tx=0x7f4a70010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a767fc700 1 -- 172.21.15.129:0/4071784517 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7000a160 con 0x7f4a780a1c80 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f4a70008b00 tx=0x7f4a70010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a777fe700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f4a70008b00 tx=0x7f4a70010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a767fc700 1 -- 172.21.15.129:0/4071784517 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a70011a90 con 0x7f4a780a1c80 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a767fc700 1 -- 172.21.15.129:0/4071784517 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7000a740 con 0x7f4a780a1c80 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 msgr2=0x7f4a780a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f4a70008b00 tx=0x7f4a70010020).stop 2019-10-13T05:21:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- 172.21.15.129:0/4071784517 shutdown_connections 2019-10-13T05:21:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/4071784517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a780a1c80 0x7f4a780a13f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- 172.21.15.129:0/4071784517 shutdown_connections 2019-10-13T05:21:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- 172.21.15.129:0/4071784517 wait complete. 2019-10-13T05:21:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.661+0000 7f4a7d35e700 1 -- 172.21.15.129:0/4071784517 >> 172.21.15.129:0/4071784517 conn(0x7f4a7805e870 msgr2=0x7f4a7809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 Processor -- start 2019-10-13T05:21:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 -- start start 2019-10-13T05:21:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4a78065150 0x7f4a78113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a7807b4e0 0x7f4a7807b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7807d460 con 0x7f4a7812ac10 2019-10-13T05:21:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7807d810 con 0x7f4a7807b4e0 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a7d35e700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7807dab0 con 0x7f4a78065150 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a77fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a7807b4e0 0x7f4a7807b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a77fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a7807b4e0 0x7f4a7807b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41238/0 (socket says 172.21.15.129:41238) 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a77fff700 1 -- 172.21.15.129:0/1309356820 learned_addr learned my addr 172.21.15.129:0/1309356820 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.665+0000 7f4a777fe700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4a78065150 0x7f4a78113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 -- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4a78065150 msgr2=0x7f4a78113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4a78065150 0x7f4a78113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 -- 172.21.15.129:0/1309356820 >> v1:172.21.15.71:6789/0 conn(0x7f4a7812ac10 legacy=0x7f4a7812b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 -- 172.21.15.129:0/1309356820 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78064c90 con 0x7f4a7807b4e0 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568514292 0 0) 0x7f4a7807d460 con 0x7f4a7812ac10 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> v2:172.21.15.129:3300/0 conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> v2:172.21.15.129:3300/0 conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=READY pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a68003540 con 0x7f4a7807b4e0 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68003160 con 0x7f4a7807b4e0 2019-10-13T05:21:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7807be50 con 0x7f4a7807b4e0 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a68003b20 con 0x7f4a7807b4e0 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a78132290 con 0x7f4a7807b4e0 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4a68009450 con 0x7f4a7807b4e0 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 0x7f4a58029d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:43.681 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f4a68034420 con 0x7f4a7807b4e0 2019-10-13T05:21:43.682 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a777fe700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 0x7f4a58029d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:43.682 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.669+0000 7f4a777fe700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 0x7f4a58029d60 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:43.682 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.673+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a780555c0 con 0x7f4a7807b4e0 2019-10-13T05:21:43.685 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.677+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.685 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.677+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4a680257b0 con 0x7f4a7807b4e0 2019-10-13T05:21:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.837+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4a780a3180 con 0x7f4a7807b4e0 2019-10-13T05:21:43.844 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.837+0000 7f4a77fff700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.837+0000 7f4a5ffff700 1 -- 172.21.15.129:0/1309356820 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a68025b50 con 0x7f4a7807b4e0 2019-10-13T05:21:43.845 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 msgr2=0x7f4a58029d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 0x7f4a58029d60 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 msgr2=0x7f4a7807b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 secure :-1 s=READY pgs=133 cs=0 l=1 rx=0x7f4a7807d9d0 tx=0x7f4a700104b0).stop 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 shutdown_connections 2019-10-13T05:21:43.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f4a58027860 0x7f4a58029d60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7807b4e0 0x7f4a7807b900 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 --2- 172.21.15.129:0/1309356820 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f4a78065150 0x7f4a78113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:43.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 shutdown_connections 2019-10-13T05:21:43.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 wait complete. 2019-10-13T05:21:43.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:43.841+0000 7f4a7d35e700 1 -- 172.21.15.129:0/1309356820 >> 172.21.15.129:0/1309356820 conn(0x7f4a7809c850 msgr2=0x7f4a78117cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:43.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2019-10-13T05:21:44.857 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:44.857 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-13T05:21:44.964 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 Processor -- start 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 -- start start 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa44c0a1c80 0x7fa44c0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c0a7a30 con 0x7fa44c144a10 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c0a1940 con 0x7fa44c0a7520 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa4545a2700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c0a96b0 con 0x7fa44c0a1c80 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa45233e700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa44c0a7520 0x7fa44c0a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59216/0 (socket says 172.21.15.129:59216) 2019-10-13T05:21:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa45233e700 1 -- 172.21.15.129:0/3698238333 learned_addr learned my addr 172.21.15.129:0/3698238333 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa452b3f700 1 --2- 172.21.15.129:0/3698238333 >> v2:172.21.15.71:3300/0 conn(0x7fa44c0a1c80 0x7fa44c0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608961636 0 0) 0x7fa44c0a1940 con 0x7fa44c0a7520 2019-10-13T05:21:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4380030a0 con 0x7fa44c0a7520 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 913506329 0 0) 0x7fa4380030a0 con 0x7fa44c0a7520 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 >> v2:172.21.15.71:3300/0 conn(0x7fa44c0a1c80 msgr2=0x7fa44c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 --2- 172.21.15.129:0/3698238333 >> v2:172.21.15.71:3300/0 conn(0x7fa44c0a1c80 0x7fa44c0a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 >> v1:172.21.15.71:6789/0 conn(0x7fa44c144a10 legacy=0x7fa44c14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c144e30 con 0x7fa44c0a7520 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7fa43c001750 con 0x7fa44c0a7520 2019-10-13T05:21:44.967 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 >> v1:172.21.15.129:6789/0 conn(0x7fa44c0a7520 legacy=0x7fa44c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4380036d0 0x7fa438003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa438004410 0x7fa438004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0a96b0 con 0x7fa438004410 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa452b3f700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4380036d0 0x7fa438003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43804fbf0 con 0x7fa4380036d0 2019-10-13T05:21:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43804fe30 con 0x7fa438004d80 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa45233e700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa438004410 0x7fa438004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 -- 172.21.15.129:0/3698238333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4380036d0 msgr2=0x7fa438003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4380036d0 0x7fa438003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 -- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa438004410 msgr2=0x7fa438004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa438004410 0x7fa438004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.953+0000 7fa453340700 1 -- 172.21.15.129:0/3698238333 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438050130 con 0x7fa438004d80 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa452b3f700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4380036d0 0x7fa438003af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa444001000 tx=0x7fa43c001b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 -- 172.21.15.129:0/3698238333 reap_dead start 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa444001000 tx=0x7fa43c001b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fa444001000 tx=0x7fa43c001b20).ready entity=mon.2 client_cookie=3a9517fcba602586 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fa444001000 tx=0x7fa43c001b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa444011490 con 0x7fa438004d80 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fa444001000 tx=0x7fa43c001b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa453340700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fa444001000 tx=0x7fa43c001b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4440110b0 con 0x7fa438004d80 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa451b3d700 1 -- 172.21.15.129:0/3698238333 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa444011a90 con 0x7fa438004d80 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 -- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 msgr2=0x7fa43800b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fa444001000 tx=0x7fa43c001b20).stop 2019-10-13T05:21:44.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 -- 172.21.15.129:0/3698238333 shutdown_connections 2019-10-13T05:21:44.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 --2- 172.21.15.129:0/3698238333 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa438004d80 0x7fa43800b660 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:44.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 -- 172.21.15.129:0/3698238333 shutdown_connections 2019-10-13T05:21:44.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 -- 172.21.15.129:0/3698238333 wait complete. 2019-10-13T05:21:44.972 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.957+0000 7fa4545a2700 1 -- 172.21.15.129:0/3698238333 >> 172.21.15.129:0/3698238333 conn(0x7fa44c05e870 msgr2=0x7fa44c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:44.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 Processor -- start 2019-10-13T05:21:44.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- start start 2019-10-13T05:21:44.976 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c135200 con 0x7fa44c068c30 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c1265f0 con 0x7fa44c143700 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c077300 con 0x7fa44c134de0 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa453340700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa44c143700 0x7fa44c0a8720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59230/0 (socket says 172.21.15.129:59230) 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa453340700 1 -- 172.21.15.129:0/4236313874 learned_addr learned my addr 172.21.15.129:0/4236313874 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116185186 0 0) 0x7fa44c1265f0 con 0x7fa44c143700 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4340030a0 con 0x7fa44c143700 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010053310 0 0) 0x7fa44c135200 con 0x7fa44c068c30 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c1265f0 con 0x7fa44c068c30 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 -- 172.21.15.129:0/4236313874 >> v1:172.21.15.129:6789/0 conn(0x7fa44c143700 legacy=0x7fa44c0a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 -- 172.21.15.129:0/4236313874 >> v1:172.21.15.71:6789/0 conn(0x7fa44c068c30 legacy=0x7fa44c0a7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 -- 172.21.15.129:0/4236313874 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0a0840 con 0x7fa44c134de0 2019-10-13T05:21:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa43c0129a0 con 0x7fa44c134de0 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c0125c0 con 0x7fa44c134de0 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c125d50 con 0x7fa44c134de0 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa43c008560 con 0x7fa44c134de0 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.961+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c143b20 con 0x7fa44c134de0 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa43c0011a0 con 0x7fa44c134de0 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.980 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa4437fe700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 0x7fa43402a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:44.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fa43c034900 con 0x7fa44c134de0 2019-10-13T05:21:44.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa452b3f700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 0x7fa43402a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:44.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa452b3f700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 0x7fa43402a4f0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:44.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.965+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c0555c0 con 0x7fa44c134de0 2019-10-13T05:21:44.981 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.977+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:44.982 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:44.977+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa43c0087b0 con 0x7fa44c134de0 2019-10-13T05:21:45.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.137+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa44c07d480 con 0x7fa44c134de0 2019-10-13T05:21:45.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.137+0000 7fa45233e700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:45.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.137+0000 7fa4437fe700 1 -- 172.21.15.129:0/4236313874 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa43c008120 con 0x7fa44c134de0 2019-10-13T05:21:45.145 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2019-10-13T05:21:45.146 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 msgr2=0x7fa43402a4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:45.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 0x7fa43402a4f0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:45.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 msgr2=0x7fa44c125460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:45.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fa43c001950 tx=0x7fa43c010b10).stop 2019-10-13T05:21:45.147 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 shutdown_connections 2019-10-13T05:21:45.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fa434027ff0 0x7fa43402a4f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:45.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 --2- 172.21.15.129:0/4236313874 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fa44c134de0 0x7fa44c125460 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:45.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 shutdown_connections 2019-10-13T05:21:45.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 wait complete. 2019-10-13T05:21:45.149 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:45.141+0000 7fa4545a2700 1 -- 172.21.15.129:0/4236313874 >> 172.21.15.129:0/4236313874 conn(0x7fa44c0aa5f0 msgr2=0x7fa44c07cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:45.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2019-10-13T05:21:46.157 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:46.157 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-13T05:21:46.266 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.253+0000 7f6aa042c700 1 Processor -- start 2019-10-13T05:21:46.266 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.253+0000 7f6aa042c700 1 -- start start 2019-10-13T05:21:46.266 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a980a1c80 0x7f6a980a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.266 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a980a7520 0x7f6a980a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.266 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a980a7a30 con 0x7f6a98144a10 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a980a96b0 con 0x7f6a980a7520 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a980a9950 con 0x7f6a980a1c80 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a980a1c80 0x7f6a980a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a980a1c80 0x7f6a980a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41266/0 (socket says 172.21.15.129:41266) 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/303544974 learned_addr learned my addr 172.21.15.129:0/303544974 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e1c8700 1 --2- 172.21.15.129:0/303544974 >> v2:172.21.15.71:3300/0 conn(0x7f6a980a7520 0x7f6a980a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.267 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/303544974 >> v2:172.21.15.71:3300/0 conn(0x7f6a980a7520 msgr2=0x7f6a980a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> v2:172.21.15.71:3300/0 conn(0x7f6a980a7520 0x7f6a980a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/303544974 >> v1:172.21.15.71:6789/0 conn(0x7f6a98144a10 legacy=0x7f6a980a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/303544974 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980a1940 con 0x7f6a980a1c80 2019-10-13T05:21:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> v2:172.21.15.129:3300/0 conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a94002f30 tx=0x7f6a88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> v2:172.21.15.129:3300/0 conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a94002f30 tx=0x7f6a88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f6a94002f30 tx=0x7f6a88007710).ready entity=mon.1 client_cookie=e9652c63bcf980cd server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6a94002f30 tx=0x7f6a88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6a94002f30 tx=0x7f6a88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9d9c7700 1 -- 172.21.15.129:0/303544974 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a94008c80 con 0x7f6a980a1c80 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6a94002f30 tx=0x7f6a88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9d9c7700 1 -- 172.21.15.129:0/303544974 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a940088a0 con 0x7f6a980a1c80 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6a9d9c7700 1 -- 172.21.15.129:0/303544974 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a940119b0 con 0x7f6a980a1c80 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 msgr2=0x7f6a980a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f6a94002f30 tx=0x7f6a88007710).stop 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- 172.21.15.129:0/303544974 shutdown_connections 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 --2- 172.21.15.129:0/303544974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a980a1c80 0x7f6a980a13f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 --2- 172.21.15.129:0/303544974 >> v2:172.21.15.71:3300/0 conn(0x7f6a980a7520 0x7f6a980a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- 172.21.15.129:0/303544974 shutdown_connections 2019-10-13T05:21:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- 172.21.15.129:0/303544974 wait complete. 2019-10-13T05:21:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.257+0000 7f6aa042c700 1 -- 172.21.15.129:0/303544974 >> 172.21.15.129:0/303544974 conn(0x7f6a9805e870 msgr2=0x7f6a9809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 Processor -- start 2019-10-13T05:21:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- start start 2019-10-13T05:21:46.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a9809f730 0x7f6a98141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6a980a6a20 0x7f6a981183a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.274 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a980a02c0 con 0x7f6a981420e0 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9809d070 con 0x7f6a9809f730 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9809dda0 con 0x7f6a980a6a20 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e1c8700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6a981420e0 0x7f6a980a6220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59244/0 (socket says 172.21.15.129:59244) 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e1c8700 1 -- 172.21.15.129:0/170735877 learned_addr learned my addr 172.21.15.129:0/170735877 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> v2:172.21.15.71:3300/0 conn(0x7f6a9809f730 0x7f6a98141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9f1ca700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6a980a6a20 0x7f6a981183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887089933 0 0) 0x7f6a980a02c0 con 0x7f6a981420e0 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a800030a0 con 0x7f6a981420e0 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2157382204 0 0) 0x7f6a800030a0 con 0x7f6a981420e0 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6a980a6a20 msgr2=0x7f6a981183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6a980a6a20 0x7f6a981183a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/170735877 >> v1:172.21.15.129:6789/0 conn(0x7f6a981420e0 legacy=0x7f6a980a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 -- 172.21.15.129:0/170735877 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980a0720 con 0x7f6a9809f730 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> v2:172.21.15.71:3300/0 conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> v2:172.21.15.71:3300/0 conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88001550 con 0x7f6a9809f730 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9809fb50 con 0x7f6a9809f730 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a94016060 con 0x7f6a9809f730 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a94007350 con 0x7f6a9809f730 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.261+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a981188f0 con 0x7f6a9809f730 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f6a9400f810 con 0x7f6a9809f730 2019-10-13T05:21:46.278 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a8f7fe700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 0x7f6a8002a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f6a94034510 con 0x7f6a9809f730 2019-10-13T05:21:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a9e1c8700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 0x7f6a8002a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.265+0000 7f6a9e1c8700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 0x7f6a8002a290 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.269+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a980555c0 con 0x7f6a9809f730 2019-10-13T05:21:46.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.277+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.284 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.277+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f6a94007790 con 0x7f6a9809f730 2019-10-13T05:21:46.450 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.445+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6a9809db70 con 0x7f6a9809f730 2019-10-13T05:21:46.451 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.445+0000 7f6a9e9c9700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.451 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.445+0000 7f6a8f7fe700 1 -- 172.21.15.129:0/170735877 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a9401cbf0 con 0x7f6a9809f730 2019-10-13T05:21:46.451 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.445+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 msgr2=0x7f6a8002a290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.445+0000 7f6aa042c700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 0x7f6a8002a290 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 msgr2=0x7f6a98141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f6a88001790 tx=0x7f6a880017d0).stop 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 shutdown_connections 2019-10-13T05:21:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6a980a6a20 0x7f6a981183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.455 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6a80027d90 0x7f6a8002a290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.455 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 --2- 172.21.15.129:0/170735877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a9809f730 0x7f6a98141b90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.455 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 shutdown_connections 2019-10-13T05:21:46.455 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 wait complete. 2019-10-13T05:21:46.456 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.449+0000 7f6aa042c700 1 -- 172.21.15.129:0/170735877 >> 172.21.15.129:0/170735877 conn(0x7f6a980a7520 msgr2=0x7f6a98142910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:46.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.1 2019-10-13T05:21:46.464 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:46.465 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-10-13T05:21:46.561 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 Processor -- start 2019-10-13T05:21:46.561 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 -- start start 2019-10-13T05:21:46.561 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a1c80 0x7fb8e40a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.561 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e40a7520 0x7fb8e40a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.561 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40a7a30 con 0x7fb8e4144a10 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e40a96b0 con 0x7fb8e40a1c80 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.549+0000 7fb8e9980700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40a9950 con 0x7fb8e40a7520 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a1c80 0x7fb8e40a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a1c80 0x7fb8e40a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41280/0 (socket says 172.21.15.129:41280) 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 -- 172.21.15.129:0/3426873249 learned_addr learned my addr 172.21.15.129:0/3426873249 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e40a7520 0x7fb8e40a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 -- 172.21.15.129:0/3426873249 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e40a7520 msgr2=0x7fb8e40a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e40a7520 0x7fb8e40a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 -- 172.21.15.129:0/3426873249 >> v1:172.21.15.71:6789/0 conn(0x7fb8e4144a10 legacy=0x7fb8e40a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 -- 172.21.15.129:0/3426873249 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40a1940 con 0x7fb8e40a1c80 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).ready entity=mon.1 client_cookie=439660a2fceca40b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e27fc700 1 -- 172.21.15.129:0/3426873249 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc008c80 con 0x7fb8e40a1c80 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e27fc700 1 -- 172.21.15.129:0/3426873249 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc0088a0 con 0x7fb8e40a1c80 2019-10-13T05:21:46.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e27fc700 1 -- 172.21.15.129:0/3426873249 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc0119b0 con 0x7fb8e40a1c80 2019-10-13T05:21:46.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 -- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 msgr2=0x7fb8e40a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7fb8e40a9870 tx=0x7fb8d4007710).stop 2019-10-13T05:21:46.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 -- 172.21.15.129:0/3426873249 shutdown_connections 2019-10-13T05:21:46.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a1c80 0x7fb8e40a13f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3426873249 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e40a7520 0x7fb8e40a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 -- 172.21.15.129:0/3426873249 shutdown_connections 2019-10-13T05:21:46.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 -- 172.21.15.129:0/3426873249 wait complete. 2019-10-13T05:21:46.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.553+0000 7fb8e9980700 1 -- 172.21.15.129:0/3426873249 >> 172.21.15.129:0/3426873249 conn(0x7fb8e405e870 msgr2=0x7fb8e409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:46.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 Processor -- start 2019-10-13T05:21:46.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 -- start start 2019-10-13T05:21:46.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a3dd0 0x7fb8e4123a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.570 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb8e41183b0 0x7fb8e411a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4123f60 con 0x7fb8e407bdf0 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e4142450 con 0x7fb8e41183b0 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e9980700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e4064320 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e2ffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a3dd0 0x7fb8e4123a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e2ffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a3dd0 0x7fb8e4123a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41286/0 (socket says 172.21.15.129:41286) 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e2ffd700 1 -- 172.21.15.129:0/3651421735 learned_addr learned my addr 172.21.15.129:0/3651421735 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:46.571 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.557+0000 7fb8e3fff700 1 --2- 172.21.15.129:0/3651421735 >> v2:172.21.15.71:3300/0 conn(0x7fb8e41183b0 0x7fb8e411a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 -- 172.21.15.129:0/3651421735 >> v2:172.21.15.71:3300/0 conn(0x7fb8e41183b0 msgr2=0x7fb8e411a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> v2:172.21.15.71:3300/0 conn(0x7fb8e41183b0 0x7fb8e411a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 -- 172.21.15.129:0/3651421735 >> v1:172.21.15.71:6789/0 conn(0x7fb8e407bdf0 legacy=0x7fb8e4141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 -- 172.21.15.129:0/3651421735 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e409d070 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.572 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> v2:172.21.15.129:3300/0 conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d4012c30 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d4012850 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e4063850 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d4005910 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e4065650 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb8d4005c20 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 0x7fb8c8029e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fb8d40353b0 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.561+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 0x7fb8c8029e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.565+0000 7fb8e37fe700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 0x7fb8c8029e80 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.565+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e4065440 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.573+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.573+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb8d4010170 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.735 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.729+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb8e40555c0 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.729+0000 7fb8e2ffd700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.729+0000 7fb8dbfff700 1 -- 172.21.15.129:0/3651421735 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8d4007160 con 0x7fb8e40a3dd0 2019-10-13T05:21:46.736 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:46.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 msgr2=0x7fb8c8029e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 0x7fb8c8029e80 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 msgr2=0x7fb8e4123a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.738 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fb8d4002f30 tx=0x7fb8dc001540).stop 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 shutdown_connections 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8c8027980 0x7fb8c8029e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3651421735 >> v2:172.21.15.71:3300/0 conn(0x7fb8e41183b0 0x7fb8e411a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 --2- 172.21.15.129:0/3651421735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e40a3dd0 0x7fb8e4123a10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 shutdown_connections 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 wait complete. 2019-10-13T05:21:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.733+0000 7fb8e9980700 1 -- 172.21.15.129:0/3651421735 >> 172.21.15.129:0/3651421735 conn(0x7fb8e40aa2c0 msgr2=0x7fb8e407dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:46.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2019-10-13T05:21:46.752 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:46.752 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-10-13T05:21:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 Processor -- start 2019-10-13T05:21:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 -- start start 2019-10-13T05:21:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8280a1c80 0x7fb8280a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb8280a7520 0x7fb8280a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb828144a10 0x7fb828144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8280a96b0 con 0x7fb8280a7520 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8280a9950 con 0x7fb8280a1c80 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280a9bf0 con 0x7fb828144a10 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8280a1c80 0x7fb8280a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.853 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8280a1c80 0x7fb8280a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41292/0 (socket says 172.21.15.129:41292) 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 -- 172.21.15.129:0/201830071 learned_addr learned my addr 172.21.15.129:0/201830071 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82e7bd700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb828144a10 0x7fb828144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/201830071 >> v2:172.21.15.71:3300/0 conn(0x7fb8280a7520 0x7fb8280a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 -- 172.21.15.129:0/201830071 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb828144a10 msgr2=0x7fb828144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb828144a10 0x7fb828144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 -- 172.21.15.129:0/201830071 >> v2:172.21.15.71:3300/0 conn(0x7fb8280a7520 msgr2=0x7fb8280a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> v2:172.21.15.71:3300/0 conn(0x7fb8280a7520 0x7fb8280a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 -- 172.21.15.129:0/201830071 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280a7a30 con 0x7fb8280a1c80 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> v2:172.21.15.129:3300/0 conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb8280a9870 tx=0x7fb818007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> v2:172.21.15.129:3300/0 conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb8280a9870 tx=0x7fb818007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7fb8280a9870 tx=0x7fb818007710).ready entity=mon.1 client_cookie=839a9bdd2875776b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7fb8280a9870 tx=0x7fb818007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7fb8280a9870 tx=0x7fb818007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82cfba700 1 -- 172.21.15.129:0/201830071 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8240088a0 con 0x7fb8280a1c80 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7fb8280a9870 tx=0x7fb818007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82cfba700 1 -- 172.21.15.129:0/201830071 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8240073c0 con 0x7fb8280a1c80 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82cfba700 1 -- 172.21.15.129:0/201830071 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb824006560 con 0x7fb8280a1c80 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 -- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 msgr2=0x7fb8280a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.841+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7fb8280a9870 tx=0x7fb818007710).stop 2019-10-13T05:21:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 -- 172.21.15.129:0/201830071 shutdown_connections 2019-10-13T05:21:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb828144a10 0x7fb828144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/201830071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8280a1c80 0x7fb8280a13f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/201830071 >> v2:172.21.15.71:3300/0 conn(0x7fb8280a7520 0x7fb8280a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 -- 172.21.15.129:0/201830071 shutdown_connections 2019-10-13T05:21:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 -- 172.21.15.129:0/201830071 wait complete. 2019-10-13T05:21:46.858 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 -- 172.21.15.129:0/201830071 >> 172.21.15.129:0/201830071 conn(0x7fb82805e870 msgr2=0x7fb82809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:46.861 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.845+0000 7fb82fa1f700 1 Processor -- start 2019-10-13T05:21:46.861 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- start start 2019-10-13T05:21:46.861 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb82811aa90 0x7fb8281343c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8281422e0 con 0x7fb828134910 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280a8540 con 0x7fb828063d70 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb828064d10 con 0x7fb82811aa90 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb82811aa90 0x7fb8281343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb82811aa90 0x7fb8281343c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41300/0 (socket says 172.21.15.129:41300) 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 -- 172.21.15.129:0/2299946639 learned_addr learned my addr 172.21.15.129:0/2299946639 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:46.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 -- 172.21.15.129:0/2299946639 >> v1:172.21.15.129:6789/0 conn(0x7fb828063d70 legacy=0x7fb828141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 -- 172.21.15.129:0/2299946639 >> v1:172.21.15.71:6789/0 conn(0x7fb828134910 legacy=0x7fb828077190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206390256 0 0) 0x7fb8280a8540 con 0x7fb828063d70 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 -- 172.21.15.129:0/2299946639 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828065130 con 0x7fb82811aa90 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> v2:172.21.15.129:3300/0 conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> v2:172.21.15.129:3300/0 conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb818005970 con 0x7fb82811aa90 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818007160 con 0x7fb82811aa90 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb828134d30 con 0x7fb82811aa90 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb818012490 con 0x7fb82811aa90 2019-10-13T05:21:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.849+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8280789a0 con 0x7fb82811aa90 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb818010170 con 0x7fb82811aa90 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb81e7fc700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 0x7fb810029d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fb818034d60 con 0x7fb82811aa90 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 0x7fb810029d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.853+0000 7fb82dfbc700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 0x7fb810029d40 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:46.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.857+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8280555c0 con 0x7fb82811aa90 2019-10-13T05:21:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.865+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:46.865+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb818012d90 con 0x7fb82811aa90 2019-10-13T05:21:47.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.021+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb828134120 con 0x7fb82811aa90 2019-10-13T05:21:47.027 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.021+0000 7fb82d7bb700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.028 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.021+0000 7fb81e7fc700 1 -- 172.21.15.129:0/2299946639 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8180127a0 con 0x7fb82811aa90 2019-10-13T05:21:47.028 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:47.030 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 msgr2=0x7fb810029d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.030 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 0x7fb810029d40 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 msgr2=0x7fb8281343c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7fb818002f30 tx=0x7fb818010dd0).stop 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 shutdown_connections 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb810027840 0x7fb810029d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 --2- 172.21.15.129:0/2299946639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82811aa90 0x7fb8281343c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 shutdown_connections 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 wait complete. 2019-10-13T05:21:47.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.025+0000 7fb82fa1f700 1 -- 172.21.15.129:0/2299946639 >> 172.21.15.129:0/2299946639 conn(0x7fb8280a7520 msgr2=0x7fb828123ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2019-10-13T05:21:47.041 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:47.041 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-13T05:21:47.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 Processor -- start 2019-10-13T05:21:47.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 -- start start 2019-10-13T05:21:47.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc00a1c80 0x7f6bc00a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc0144a10 0x7f6bc0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc00a7a30 con 0x7f6bc00a7520 2019-10-13T05:21:47.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc00a96b0 con 0x7f6bc0144a10 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc82ab700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00a9950 con 0x7f6bc00a1c80 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc7049700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc0144a10 0x7f6bc0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc7049700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc0144a10 0x7f6bc0144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41310/0 (socket says 172.21.15.129:41310) 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc7049700 1 -- 172.21.15.129:0/3980646059 learned_addr learned my addr 172.21.15.129:0/3980646059 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.125+0000 7f6bc6848700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc00a1c80 0x7f6bc00a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 -- 172.21.15.129:0/3980646059 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc00a1c80 msgr2=0x7f6bc00a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc00a1c80 0x7f6bc00a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 -- 172.21.15.129:0/3980646059 >> v1:172.21.15.71:6789/0 conn(0x7f6bc00a7520 legacy=0x7f6bc00a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 -- 172.21.15.129:0/3980646059 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00a1940 con 0x7f6bc0144a10 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> v2:172.21.15.129:3300/0 conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> v2:172.21.15.129:3300/0 conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).ready entity=mon.1 client_cookie=3e950c955c86e735 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.139 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc5846700 1 -- 172.21.15.129:0/3980646059 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb8010b90 con 0x7f6bc0144a10 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc7049700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc5846700 1 -- 172.21.15.129:0/3980646059 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb80107b0 con 0x7f6bc0144a10 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc5846700 1 -- 172.21.15.129:0/3980646059 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb80112b0 con 0x7f6bc0144a10 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 -- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 msgr2=0x7f6bc0144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f6bc00a9870 tx=0x7f6bbc007710).stop 2019-10-13T05:21:47.141 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 -- 172.21.15.129:0/3980646059 shutdown_connections 2019-10-13T05:21:47.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0144a10 0x7f6bc0144e30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/3980646059 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc00a1c80 0x7f6bc00a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 -- 172.21.15.129:0/3980646059 shutdown_connections 2019-10-13T05:21:47.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 -- 172.21.15.129:0/3980646059 wait complete. 2019-10-13T05:21:47.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.129+0000 7f6bc82ab700 1 -- 172.21.15.129:0/3980646059 >> 172.21.15.129:0/3980646059 conn(0x7f6bc005e870 msgr2=0x7f6bc009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.148 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 Processor -- start 2019-10-13T05:21:47.150 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 -- start start 2019-10-13T05:21:47.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc009d570 0x7f6bc014d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc014dd20 0x7f6bc012e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc00773e0 con 0x7f6bc0075740 2019-10-13T05:21:47.151 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc014e6a0 con 0x7f6bc009d570 2019-10-13T05:21:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc82ab700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc009d070 con 0x7f6bc014dd20 2019-10-13T05:21:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc009d570 0x7f6bc014d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6848700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6bc0075740 0x7f6bc0123ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59282/0 (socket says 172.21.15.129:59282) 2019-10-13T05:21:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6bc009d570 0x7f6bc014d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41314/0 (socket says 172.21.15.129:41314) 2019-10-13T05:21:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6848700 1 -- 172.21.15.129:0/4150362098 learned_addr learned my addr 172.21.15.129:0/4150362098 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc7049700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc014dd20 0x7f6bc012e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 -- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc014dd20 msgr2=0x7f6bc012e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc014dd20 0x7f6bc012e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 -- 172.21.15.129:0/4150362098 >> v1:172.21.15.129:6789/0 conn(0x7f6bc0075740 legacy=0x7f6bc0123ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 -- 172.21.15.129:0/4150362098 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc014e140 con 0x7f6bc009d570 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> v2:172.21.15.129:3300/0 conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> v2:172.21.15.129:3300/0 conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.133+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.154 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb000cb60 con 0x7f6bc009d570 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000c780 con 0x7f6bc009d570 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc01242c0 con 0x7f6bc009d570 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb0010ee0 con 0x7f6bc009d570 2019-10-13T05:21:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc012ea20 con 0x7f6bc009d570 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f6bb00111f0 con 0x7f6bc009d570 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 0x7f6ba8029db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f6bb0034c60 con 0x7f6bc009d570 2019-10-13T05:21:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6848700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 0x7f6ba8029db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.137+0000 7f6bc6848700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 0x7f6ba8029db0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.141+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc009d990 con 0x7f6bc009d570 2019-10-13T05:21:47.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.145+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.157 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.145+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f6bb0025ff0 con 0x7f6bc009d570 2019-10-13T05:21:47.309 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.301+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6bc0077610 con 0x7f6bc009d570 2019-10-13T05:21:47.310 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc6047700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.310 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bb77fe700 1 -- 172.21.15.129:0/4150362098 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bb0010100 con 0x7f6bc009d570 2019-10-13T05:21:47.310 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:47.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 msgr2=0x7f6ba8029db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 0x7f6ba8029db0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 msgr2=0x7f6bc014d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f6bc009d230 tx=0x7f6bbc002ff0).stop 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 shutdown_connections 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f6ba80278b0 0x7f6ba8029db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f6bc014dd20 0x7f6bc012e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 --2- 172.21.15.129:0/4150362098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc009d570 0x7f6bc014d7d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 shutdown_connections 2019-10-13T05:21:47.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 wait complete. 2019-10-13T05:21:47.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.305+0000 7f6bc82ab700 1 -- 172.21.15.129:0/4150362098 >> 172.21.15.129:0/4150362098 conn(0x7f6bc007e5b0 msgr2=0x7f6bc01429b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2019-10-13T05:21:47.322 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:47.322 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-10-13T05:21:47.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.405+0000 7fc5cf1d3700 1 Processor -- start 2019-10-13T05:21:47.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.405+0000 7fc5cf1d3700 1 -- start start 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 0x7fc5c80a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5c8144a10 0x7fc5c8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80a7a30 con 0x7fc5c80a7520 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80a96b0 con 0x7fc5c8144a10 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80a9950 con 0x7fc5c80a1c80 2019-10-13T05:21:47.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5ccf6f700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc5c80a7520 0x7fc5c80a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59292/0 (socket says 172.21.15.129:59292) 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5ccf6f700 1 -- 172.21.15.129:0/447107426 learned_addr learned my addr 172.21.15.129:0/447107426 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5c8144a10 0x7fc5c8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cd770700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 0x7fc5c80a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 -- 172.21.15.129:0/447107426 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 msgr2=0x7fc5c80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 0x7fc5c80a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 -- 172.21.15.129:0/447107426 >> v1:172.21.15.129:6789/0 conn(0x7fc5c80a7520 legacy=0x7fc5c80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5c7fff700 1 -- 172.21.15.129:0/447107426 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871673691 0 0) 0x7fc5c80a7a30 con 0x7fc5c80a7520 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 -- 172.21.15.129:0/447107426 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80a1940 con 0x7fc5c8144a10 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cd770700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 0x7fc5c80a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> v2:172.21.15.129:3300/0 conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> v2:172.21.15.129:3300/0 conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).ready entity=mon.1 client_cookie=28672a7bf8906896 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5c7fff700 1 -- 172.21.15.129:0/447107426 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c0008ba0 con 0x7fc5c8144a10 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cdf71700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5c7fff700 1 -- 172.21.15.129:0/447107426 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c00087c0 con 0x7fc5c8144a10 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5c7fff700 1 -- 172.21.15.129:0/447107426 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c0011740 con 0x7fc5c8144a10 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 msgr2=0x7fc5c8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7fc5c80a9870 tx=0x7fc5b0007710).stop 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/447107426 shutdown_connections 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8144a10 0x7fc5c8144e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/447107426 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fc5c80a1c80 0x7fc5c80a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/447107426 shutdown_connections 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/447107426 wait complete. 2019-10-13T05:21:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.409+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/447107426 >> 172.21.15.129:0/447107426 conn(0x7fc5c805e870 msgr2=0x7fc5c809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 Processor -- start 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 -- start start 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5c8143990 0x7fc5c811c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c811d3a0 con 0x7fc5c8122ba0 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8122fc0 con 0x7fc5c80a6310 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cf1d3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c811afd0 con 0x7fc5c8143990 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cd770700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5c8143990 0x7fc5c811c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cd770700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5c8143990 0x7fc5c811c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41326/0 (socket says 172.21.15.129:41326) 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.413+0000 7fc5cd770700 1 -- 172.21.15.129:0/1981948436 learned_addr learned my addr 172.21.15.129:0/1981948436 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 -- 172.21.15.129:0/1981948436 >> v1:172.21.15.129:6789/0 conn(0x7fc5c80a6310 legacy=0x7fc5c807d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 -- 172.21.15.129:0/1981948436 >> v1:172.21.15.71:6789/0 conn(0x7fc5c8122ba0 legacy=0x7fc5c8138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 -- 172.21.15.129:0/1981948436 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80a3dd0 con 0x7fc5c8143990 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> v2:172.21.15.129:3300/0 conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> v2:172.21.15.129:3300/0 conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b0007160 con 0x7fc5c8143990 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0008bc0 con 0x7fc5c8143990 2019-10-13T05:21:47.429 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b0005b90 con 0x7fc5c8143990 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c807df00 con 0x7fc5c8143990 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c8139b30 con 0x7fc5c8143990 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc5b00122f0 con 0x7fc5c8143990 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 0x7fc5ac029de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fc5b0034ed0 con 0x7fc5c8143990 2019-10-13T05:21:47.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5ccf6f700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 0x7fc5ac029de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.417+0000 7fc5ccf6f700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 0x7fc5ac029de0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.431 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.425+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c80555c0 con 0x7fc5c8143990 2019-10-13T05:21:47.437 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.429+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.437 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.429+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc5b00125e0 con 0x7fc5c8143990 2019-10-13T05:21:47.601 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.593+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc5c8123b00 con 0x7fc5c8143990 2019-10-13T05:21:47.601 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.593+0000 7fc5cd770700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.601 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.593+0000 7fc5c5ffb700 1 -- 172.21.15.129:0/1981948436 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc5b0012800 con 0x7fc5c8143990 2019-10-13T05:21:47.602 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:47.604 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 msgr2=0x7fc5ac029de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.604 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 0x7fc5ac029de0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.605 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 msgr2=0x7fc5c811c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.605 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7fc5b0002f30 tx=0x7fc5b0010c30).stop 2019-10-13T05:21:47.605 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 shutdown_connections 2019-10-13T05:21:47.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fc5ac0278e0 0x7fc5ac029de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 --2- 172.21.15.129:0/1981948436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c8143990 0x7fc5c811c810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 shutdown_connections 2019-10-13T05:21:47.608 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 wait complete. 2019-10-13T05:21:47.609 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.597+0000 7fc5cf1d3700 1 -- 172.21.15.129:0/1981948436 >> 172.21.15.129:0/1981948436 conn(0x7fc5c80a7a10 msgr2=0x7fc5c8123300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.613 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2019-10-13T05:21:47.614 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:47.614 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-10-13T05:21:47.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 Processor -- start 2019-10-13T05:21:47.714 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- start start 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 0x7f50500a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f50500a7520 0x7f50500a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5050144a10 0x7f5050144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50500a96b0 con 0x7f50500a1c80 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50500a9950 con 0x7f5050144a10 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50500a9bf0 con 0x7f50500a7520 2019-10-13T05:21:47.715 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5050144a10 0x7f5050144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5050144a10 0x7f5050144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41338/0 (socket says 172.21.15.129:41338) 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 -- 172.21.15.129:0/3175283464 learned_addr learned my addr 172.21.15.129:0/3175283464 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5055399700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f50500a7520 0x7f50500a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5055b9a700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 0x7f50500a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 -- 172.21.15.129:0/3175283464 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f50500a7520 msgr2=0x7f50500a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f50500a7520 0x7f50500a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.716 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 -- 172.21.15.129:0/3175283464 >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 msgr2=0x7f50500a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 0x7f50500a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 -- 172.21.15.129:0/3175283464 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50500a7a30 con 0x7f5050144a10 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5055b9a700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 0x7f50500a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.129:3300/0 conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f50500a9870 tx=0x7f5044007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.129:3300/0 conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f50500a9870 tx=0x7f5044007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f50500a9870 tx=0x7f5044007710).ready entity=mon.1 client_cookie=7c1e2dc6d227d815 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f50500a9870 tx=0x7f5044007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5054b98700 1 -- 172.21.15.129:0/3175283464 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50400088a0 con 0x7f5050144a10 2019-10-13T05:21:47.718 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f50500a9870 tx=0x7f5044007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f505639b700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f50500a9870 tx=0x7f5044007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5054b98700 1 -- 172.21.15.129:0/3175283464 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50400073c0 con 0x7f5050144a10 2019-10-13T05:21:47.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f5054b98700 1 -- 172.21.15.129:0/3175283464 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5040006560 con 0x7f5050144a10 2019-10-13T05:21:47.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 msgr2=0x7f5050144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f50500a9870 tx=0x7f5044007710).stop 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- 172.21.15.129:0/3175283464 shutdown_connections 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050144a10 0x7f5050144e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- 172.21.15.129:0/3175283464 >> v2:172.21.15.71:3300/0 conn(0x7f50500a1c80 0x7f50500a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 --2- 172.21.15.129:0/3175283464 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f50500a7520 0x7f50500a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- 172.21.15.129:0/3175283464 shutdown_connections 2019-10-13T05:21:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- 172.21.15.129:0/3175283464 wait complete. 2019-10-13T05:21:47.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.705+0000 7f50575fd700 1 -- 172.21.15.129:0/3175283464 >> 172.21.15.129:0/3175283464 conn(0x7f505005e870 msgr2=0x7f505009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 Processor -- start 2019-10-13T05:21:47.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 -- start start 2019-10-13T05:21:47.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5050123a60 0x7f50501226b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5050143c60 0x7f50500a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.724 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5050122c00 con 0x7f5050132f90 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505007d250 con 0x7f5050123a60 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505007c240 con 0x7f5050143c60 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5050143c60 0x7f50500a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f5055b9a700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5050132f90 0x7f505007c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59310/0 (socket says 172.21.15.129:59310) 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f5055b9a700 1 -- 172.21.15.129:0/1308816608 learned_addr learned my addr 172.21.15.129:0/1308816608 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:47.725 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f5055399700 1 --2- 172.21.15.129:0/1308816608 >> v2:172.21.15.71:3300/0 conn(0x7f5050123a60 0x7f50501226b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 -- 172.21.15.129:0/1308816608 >> v2:172.21.15.71:3300/0 conn(0x7f5050123a60 msgr2=0x7f50501226b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> v2:172.21.15.71:3300/0 conn(0x7f5050123a60 0x7f50501226b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 -- 172.21.15.129:0/1308816608 >> v1:172.21.15.129:6789/0 conn(0x7f5050132f90 legacy=0x7f505007c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574866128 0 0) 0x7f5050122c00 con 0x7f5050132f90 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 -- 172.21.15.129:0/1308816608 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050123e80 con 0x7f5050143c60 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> v2:172.21.15.129:3300/0 conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.726 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> v2:172.21.15.129:3300/0 conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.727 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f504000f5f0 con 0x7f5050143c60 2019-10-13T05:21:47.728 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.728 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040008aa0 con 0x7f5050143c60 2019-10-13T05:21:47.728 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5050144080 con 0x7f5050143c60 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.709+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50400124f0 con 0x7f5050143c60 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50500a59b0 con 0x7f5050143c60 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f50400150a0 con 0x7f5050143c60 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f504e7fc700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 0x7f5030029d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:47.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f5040034650 con 0x7f5050143c60 2019-10-13T05:21:47.730 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f5055b9a700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 0x7f5030029d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:47.730 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.713+0000 7f5055b9a700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 0x7f5030029d50 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:47.730 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.721+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50500555c0 con 0x7f5050143c60 2019-10-13T05:21:47.732 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.725+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.732 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.725+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f50400140a0 con 0x7f5050143c60 2019-10-13T05:21:47.888 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.881+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f505007c400 con 0x7f5050143c60 2019-10-13T05:21:47.889 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.881+0000 7f505639b700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:47.889 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.881+0000 7f504e7fc700 1 -- 172.21.15.129:0/1308816608 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f504001d180 con 0x7f5050143c60 2019-10-13T05:21:47.889 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:47.892 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 msgr2=0x7f5030029d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.892 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 0x7f5030029d50 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.892 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 msgr2=0x7f50500a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:47.892 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f5040007b00 tx=0x7f50480078d0).stop 2019-10-13T05:21:47.892 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 shutdown_connections 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5050143c60 0x7f50500a5460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 --2- 172.21.15.129:0/1308816608 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f5030027850 0x7f5030029d50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 --2- 172.21.15.129:0/1308816608 >> v2:172.21.15.71:3300/0 conn(0x7f5050123a60 0x7f50501226b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 shutdown_connections 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 wait complete. 2019-10-13T05:21:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.885+0000 7f50575fd700 1 -- 172.21.15.129:0/1308816608 >> 172.21.15.129:0/1308816608 conn(0x7f50500a8fe0 msgr2=0x7f505009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:47.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2019-10-13T05:21:47.902 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:47.902 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-10-13T05:21:48.004 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 Processor -- start 2019-10-13T05:21:48.004 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 -- start start 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1824144a10 0x7f1824144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240a7a30 con 0x7f18240a1c80 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824150ba0 con 0x7f18240a7520 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1828ec7700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240a96b0 con 0x7f1824144a10 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f182259c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f18240a7520 0x7f18240a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59318/0 (socket says 172.21.15.129:59318) 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f182259c700 1 -- 172.21.15.129:0/2735023647 learned_addr learned my addr 172.21.15.129:0/2735023647 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f182359e700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1824144a10 0x7f1824144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134583269 0 0) 0x7f1824150ba0 con 0x7f18240a7520 2019-10-13T05:21:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18040030a0 con 0x7f18240a7520 2019-10-13T05:21:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3663281878 0 0) 0x7f18040030a0 con 0x7f18240a7520 2019-10-13T05:21:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1824144a10 msgr2=0x7f1824144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1824144a10 0x7f1824144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 >> v1:172.21.15.71:6789/0 conn(0x7f18240a1c80 legacy=0x7f18240a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824151de0 con 0x7f18240a7520 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7f180c001ad0 con 0x7f18240a7520 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 >> v1:172.21.15.129:6789/0 conn(0x7f18240a7520 legacy=0x7f18240a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1804003700 0x7f1804003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1804004d80 0x7f180400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240a96b0 con 0x7f1804003700 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18040051a0 con 0x7f1804004410 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180404fbc0 con 0x7f1804004d80 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f182359e700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1804004d80 0x7f180400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.993+0000 7f1822d9d700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1804003700 0x7f1804003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 -- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1804004d80 msgr2=0x7f180400b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f1804004d80 0x7f180400b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 -- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1804003700 msgr2=0x7f1804003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1804003700 0x7f1804003b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 -- 172.21.15.129:0/2735023647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180404fe40 con 0x7f1804004410 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1822d9d700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1804003700 0x7f1804003b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f18040053e0 tx=0x7f1814001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f18040053e0 tx=0x7f1814001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7f18040053e0 tx=0x7f1814001310).ready entity=mon.1 client_cookie=2cce7f491ed3d75c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f18040053e0 tx=0x7f1814001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c0057d0 con 0x7f1804004410 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f18040053e0 tx=0x7f1814001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182259c700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f18040053e0 tx=0x7f1814001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c0082e0 con 0x7f1804004410 2019-10-13T05:21:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1821d9b700 1 -- 172.21.15.129:0/2735023647 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c005db0 con 0x7f1804004410 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 msgr2=0x7f1804004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 --2- 172.21.15.129:0/2735023647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1804004410 0x7f1804004830 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7f18040053e0 tx=0x7f1814001310).stop 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f182359e700 1 -- 172.21.15.129:0/2735023647 reap_dead start 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- 172.21.15.129:0/2735023647 shutdown_connections 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- 172.21.15.129:0/2735023647 shutdown_connections 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- 172.21.15.129:0/2735023647 wait complete. 2019-10-13T05:21:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- 172.21.15.129:0/2735023647 >> 172.21.15.129:0/2735023647 conn(0x7f182405e870 msgr2=0x7f182409c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 Processor -- start 2019-10-13T05:21:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:47.997+0000 7f1828ec7700 1 -- start start 2019-10-13T05:21:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1824119030 0x7f182407c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182409e770 con 0x7f182407cdd0 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182409d8c0 con 0x7f1824119030 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182409db60 con 0x7f182409de00 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182259c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f182407cdd0 0x7f182409f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59334/0 (socket says 172.21.15.129:59334) 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182259c700 1 -- 172.21.15.129:0/4107673801 learned_addr learned my addr 172.21.15.129:0/4107673801 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1822d9d700 1 --2- 172.21.15.129:0/4107673801 >> v2:172.21.15.71:3300/0 conn(0x7f1824119030 0x7f182407c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22745140 0 0) 0x7f182409e770 con 0x7f182407cdd0 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17fc0030a0 con 0x7f182407cdd0 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 -- 172.21.15.129:0/4107673801 >> v2:172.21.15.71:3300/0 conn(0x7f1824119030 msgr2=0x7f182407c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> v2:172.21.15.71:3300/0 conn(0x7f1824119030 0x7f182407c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 -- 172.21.15.129:0/4107673801 >> v1:172.21.15.129:6789/0 conn(0x7f182407cdd0 legacy=0x7f182409f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 -- 172.21.15.129:0/4107673801 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824123780 con 0x7f182409de00 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c005920 con 0x7f182409de00 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c007160 con 0x7f182409de00 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c008550 con 0x7f182409de00 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182409f8e0 con 0x7f182409de00 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182412e190 con 0x7f182409de00 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f181c00b250 con 0x7f182409de00 2019-10-13T05:21:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.001+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.005+0000 7f181b7fe700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 0x7f17fc02a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.005+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f181c0346a0 con 0x7f182409de00 2019-10-13T05:21:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.005+0000 7f1822d9d700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 0x7f17fc02a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.005+0000 7f1822d9d700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 0x7f17fc02a260 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.009+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18240555c0 con 0x7f182409de00 2019-10-13T05:21:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.017+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.017+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f181c008750 con 0x7f182409de00 2019-10-13T05:21:48.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.177+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f18240a8e50 con 0x7f182409de00 2019-10-13T05:21:48.184 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.177+0000 7f182359e700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.184 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.177+0000 7f181b7fe700 1 -- 172.21.15.129:0/4107673801 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f181c008ca0 con 0x7f182409de00 2019-10-13T05:21:48.186 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2019-10-13T05:21:48.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 msgr2=0x7f17fc02a260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 0x7f17fc02a260 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 msgr2=0x7f182409e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f180c0018a0 tx=0x7f18140012f0).stop 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 shutdown_connections 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f182409de00 0x7f182409e220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 --2- 172.21.15.129:0/4107673801 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f17fc027d60 0x7f17fc02a260 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 --2- 172.21.15.129:0/4107673801 >> v2:172.21.15.71:3300/0 conn(0x7f1824119030 0x7f182407c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 shutdown_connections 2019-10-13T05:21:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 wait complete. 2019-10-13T05:21:48.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.181+0000 7f1828ec7700 1 -- 172.21.15.129:0/4107673801 >> 172.21.15.129:0/4107673801 conn(0x7f18240a7520 msgr2=0x7f18240a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2019-10-13T05:21:48.198 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-10-13T05:21:48.198 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:48.198 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T05:21:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 Processor -- start 2019-10-13T05:21:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 -- start start 2019-10-13T05:21:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7dd8144a10 0x7f7dd8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80a7a30 con 0x7f7dd80a7520 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd8150ba0 con 0x7f7dd80a1c80 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.293+0000 7f7de0479700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd80a96b0 con 0x7f7dd8144a10 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7dd8144a10 0x7f7dd8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7dde215700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7dd80a7520 0x7f7dd80a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:53926/0 (socket says 172.21.15.129:53926) 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7dde215700 1 -- 172.21.15.129:0/3020577216 learned_addr learned my addr 172.21.15.129:0/3020577216 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddda14700 1 -- 172.21.15.129:0/3020577216 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077669075 0 0) 0x7f7dd80a7a30 con 0x7f7dd80a7520 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddda14700 1 -- 172.21.15.129:0/3020577216 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc40030a0 con 0x7f7dd80a7520 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 -- 172.21.15.129:0/3020577216 >> v1:172.21.15.129:6789/0 conn(0x7f7dd80a1c80 legacy=0x7f7dd80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 -- 172.21.15.129:0/3020577216 >> v1:172.21.15.71:6789/0 conn(0x7f7dd80a7520 legacy=0x7f7dd80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 -- 172.21.15.129:0/3020577216 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8151de0 con 0x7f7dd8144a10 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> v2:172.21.15.129:3300/0 conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> v2:172.21.15.129:3300/0 conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).ready entity=mon.1 client_cookie=62e46ee57ac92c86 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddda14700 1 -- 172.21.15.129:0/3020577216 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd0011260 con 0x7f7dd8144a10 2019-10-13T05:21:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddda14700 1 -- 172.21.15.129:0/3020577216 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0008bb0 con 0x7f7dd8144a10 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddf217700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7ddda14700 1 -- 172.21.15.129:0/3020577216 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd0011840 con 0x7f7dd8144a10 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 -- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 msgr2=0x7f7dd8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7f7dd0008780 tx=0x7f7dd000fbe0).stop 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 -- 172.21.15.129:0/3020577216 shutdown_connections 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 --2- 172.21.15.129:0/3020577216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd8144a10 0x7f7dd8144e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 -- 172.21.15.129:0/3020577216 shutdown_connections 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 -- 172.21.15.129:0/3020577216 wait complete. 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.297+0000 7f7de0479700 1 -- 172.21.15.129:0/3020577216 >> 172.21.15.129:0/3020577216 conn(0x7f7dd805e870 msgr2=0x7f7dd809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 Processor -- start 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- start start 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7dd807a760 0x7f7dd8079920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f7dd8079e70 0x7f7dd80a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd8117150 con 0x7f7dd80a8610 2019-10-13T05:21:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd8116990 con 0x7f7dd807a760 2019-10-13T05:21:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80a6af0 con 0x7f7dd8079e70 2019-10-13T05:21:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7dd807a760 0x7f7dd8079920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddf217700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7dd80a8610 0x7f7dd8115770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59348/0 (socket says 172.21.15.129:59348) 2019-10-13T05:21:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddf217700 1 -- 172.21.15.129:0/904029030 learned_addr learned my addr 172.21.15.129:0/904029030 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7dde215700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f7dd8079e70 0x7f7dd80a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 -- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f7dd8079e70 msgr2=0x7f7dd80a80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f7dd8079e70 0x7f7dd80a80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 -- 172.21.15.129:0/904029030 >> v1:172.21.15.129:6789/0 conn(0x7f7dd80a8610 legacy=0x7f7dd8115770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002371576 0 0) 0x7f7dd8117150 con 0x7f7dd80a8610 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 -- 172.21.15.129:0/904029030 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80a3dd0 con 0x7f7dd807a760 2019-10-13T05:21:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> v2:172.21.15.129:3300/0 conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> v2:172.21.15.129:3300/0 conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd400ca70 con 0x7f7dd807a760 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd400c690 con 0x7f7dd807a760 2019-10-13T05:21:48.319 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd4010f60 con 0x7f7dd807a760 2019-10-13T05:21:48.319 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd807b960 con 0x7f7dd807a760 2019-10-13T05:21:48.319 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.301+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd80a8a30 con 0x7f7dd807a760 2019-10-13T05:21:48.319 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.320 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f7dd4011270 con 0x7f7dd807a760 2019-10-13T05:21:48.320 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.321 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7dcb7fe700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 0x7f7dc0029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.321 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f7dd4034ad0 con 0x7f7dd807a760 2019-10-13T05:21:48.321 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7dde215700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 0x7f7dc0029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.321 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.305+0000 7f7dde215700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 0x7f7dc0029d00 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.322 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.309+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd807c730 con 0x7f7dd807a760 2019-10-13T05:21:48.322 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.317+0000 7f7ddea16700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.322 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.317+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f7dd4025e60 con 0x7f7dd807a760 2019-10-13T05:21:48.478 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.473+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 --> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7dd80555c0 con 0x7f7dc0027800 2019-10-13T05:21:48.485 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.481+0000 7f7dcb7fe700 1 -- 172.21.15.129:0/904029030 <== mgr.4104 v2:172.21.15.71:6800/13713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80341 (crc 0 0 0) 0x7f7dd80555c0 con 0x7f7dc0027800 2019-10-13T05:21:48.494 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:48.494 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 msgr2=0x7f7dc0029d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 0x7f7dc0029d00 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 msgr2=0x7f7dd8079920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f7dd807ad40 tx=0x7f7dcc001340).stop 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 shutdown_connections 2019-10-13T05:21:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f7dc0027800 0x7f7dc0029d00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd807a760 0x7f7dd8079920 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 --2- 172.21.15.129:0/904029030 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f7dd8079e70 0x7f7dd80a80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 shutdown_connections 2019-10-13T05:21:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 wait complete. 2019-10-13T05:21:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.489+0000 7f7de0479700 1 -- 172.21.15.129:0/904029030 >> 172.21.15.129:0/904029030 conn(0x7f7dd80a9050 msgr2=0x7f7dd80a2170 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.507 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2019-10-13T05:21:48.266828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8394624,"kb_used_data":5888,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746580096,"statfs":{"total":773094113280,"available":764498018304,"internally_reserved":8589934592,"allocated":6029312,"data_stored":607668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001264"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.851972+0000","last_change":"2019-10-13T05:21:36.850565+0000","last_active":"2019-10-13T05:21:36.851972+0000","last_peered":"2019-10-13T05:21:36.851972+0000","last_clean":"2019-10-13T05:21:36.851972+0000","last_became_active":"2019-10-13T05:21:36.849754+0000","last_became_peered":"2019-10-13T05:21:36.849754+0000","last_unstale":"2019-10-13T05:21:36.851972+0000","last_undegraded":"2019-10-13T05:21:36.851972+0000","last_fullsized":"2019-10-13T05:21:36.851972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.841685+0000","last_change":"2019-10-13T05:21:34.091136+0000","last_active":"2019-10-13T05:21:36.841685+0000","last_peered":"2019-10-13T05:21:36.841685+0000","last_clean":"2019-10-13T05:21:36.841685+0000","last_became_active":"2019-10-13T05:21:34.090386+0000","last_became_peered":"2019-10-13T05:21:34.090386+0000","last_unstale":"2019-10-13T05:21:36.841685+0000","last_undegraded":"2019-10-13T05:21:36.841685+0000","last_fullsized":"2019-10-13T05:21:36.841685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.916975+0000","last_change":"2019-10-13T05:21:36.688806+0000","last_active":"2019-10-13T05:21:39.916975+0000","last_peered":"2019-10-13T05:21:39.916975+0000","last_clean":"2019-10-13T05:21:39.916975+0000","last_became_active":"2019-10-13T05:21:36.687995+0000","last_became_peered":"2019-10-13T05:21:36.687995+0000","last_unstale":"2019-10-13T05:21:39.916975+0000","last_undegraded":"2019-10-13T05:21:39.916975+0000","last_fullsized":"2019-10-13T05:21:39.916975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.763699+0000","last_change":"2019-10-13T05:21:37.762797+0000","last_active":"2019-10-13T05:21:37.763699+0000","last_peered":"2019-10-13T05:21:37.763699+0000","last_clean":"2019-10-13T05:21:37.763699+0000","last_became_active":"2019-10-13T05:21:37.762354+0000","last_became_peered":"2019-10-13T05:21:37.762354+0000","last_unstale":"2019-10-13T05:21:37.763699+0000","last_undegraded":"2019-10-13T05:21:37.763699+0000","last_fullsized":"2019-10-13T05:21:37.763699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.757827+0000","last_change":"2019-10-13T05:21:33.499568+0000","last_active":"2019-10-13T05:21:37.757827+0000","last_peered":"2019-10-13T05:21:37.757827+0000","last_clean":"2019-10-13T05:21:37.757827+0000","last_became_active":"2019-10-13T05:21:33.499124+0000","last_became_peered":"2019-10-13T05:21:33.499124+0000","last_unstale":"2019-10-13T05:21:37.757827+0000","last_undegraded":"2019-10-13T05:21:37.757827+0000","last_fullsized":"2019-10-13T05:21:37.757827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":"17","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919198+0000","last_change":"2019-10-13T05:21:36.688467+0000","last_active":"2019-10-13T05:21:39.919198+0000","last_peered":"2019-10-13T05:21:39.919198+0000","last_clean":"2019-10-13T05:21:39.919198+0000","last_became_active":"2019-10-13T05:21:36.687973+0000","last_became_peered":"2019-10-13T05:21:36.687973+0000","last_unstale":"2019-10-13T05:21:39.919198+0000","last_undegraded":"2019-10-13T05:21:39.919198+0000","last_fullsized":"2019-10-13T05:21:39.919198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.852058+0000","last_change":"2019-10-13T05:21:36.849279+0000","last_active":"2019-10-13T05:21:36.852058+0000","last_peered":"2019-10-13T05:21:36.852058+0000","last_clean":"2019-10-13T05:21:36.852058+0000","last_became_active":"2019-10-13T05:21:36.848992+0000","last_became_peered":"2019-10-13T05:21:36.848992+0000","last_unstale":"2019-10-13T05:21:36.852058+0000","last_undegraded":"2019-10-13T05:21:36.852058+0000","last_fullsized":"2019-10-13T05:21:36.852058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.842552+0000","last_change":"2019-10-13T05:21:33.499961+0000","last_active":"2019-10-13T05:21:36.842552+0000","last_peered":"2019-10-13T05:21:36.842552+0000","last_clean":"2019-10-13T05:21:36.842552+0000","last_became_active":"2019-10-13T05:21:33.499518+0000","last_became_peered":"2019-10-13T05:21:33.499518+0000","last_unstale":"2019-10-13T05:21:36.842552+0000","last_undegraded":"2019-10-13T05:21:36.842552+0000","last_fullsized":"2019-10-13T05:21:36.842552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919904+0000","last_change":"2019-10-13T05:21:36.533229+0000","last_active":"2019-10-13T05:21:39.919904+0000","last_peered":"2019-10-13T05:21:39.919904+0000","last_clean":"2019-10-13T05:21:39.919904+0000","last_became_active":"2019-10-13T05:21:36.532929+0000","last_became_peered":"2019-10-13T05:21:36.532929+0000","last_unstale":"2019-10-13T05:21:39.919904+0000","last_undegraded":"2019-10-13T05:21:39.919904+0000","last_fullsized":"2019-10-13T05:21:39.919904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.763476+0000","last_change":"2019-10-13T05:21:37.762836+0000","last_active":"2019-10-13T05:21:37.763476+0000","last_peered":"2019-10-13T05:21:37.763476+0000","last_clean":"2019-10-13T05:21:37.763476+0000","last_became_active":"2019-10-13T05:21:37.762447+0000","last_became_peered":"2019-10-13T05:21:37.762447+0000","last_unstale":"2019-10-13T05:21:37.763476+0000","last_undegraded":"2019-10-13T05:21:37.763476+0000","last_fullsized":"2019-10-13T05:21:37.763476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.845771+0000","last_change":"2019-10-13T05:21:33.497347+0000","last_active":"2019-10-13T05:21:36.845771+0000","last_peered":"2019-10-13T05:21:36.845771+0000","last_clean":"2019-10-13T05:21:36.845771+0000","last_became_active":"2019-10-13T05:21:33.497007+0000","last_became_peered":"2019-10-13T05:21:33.497007+0000","last_unstale":"2019-10-13T05:21:36.845771+0000","last_undegraded":"2019-10-13T05:21:36.845771+0000","last_fullsized":"2019-10-13T05:21:36.845771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.757757+0000","last_change":"2019-10-13T05:21:36.527884+0000","last_active":"2019-10-13T05:21:37.757757+0000","last_peered":"2019-10-13T05:21:37.757757+0000","last_clean":"2019-10-13T05:21:37.757757+0000","last_became_active":"2019-10-13T05:21:36.527519+0000","last_became_peered":"2019-10-13T05:21:36.527519+0000","last_unstale":"2019-10-13T05:21:37.757757+0000","last_undegraded":"2019-10-13T05:21:37.757757+0000","last_fullsized":"2019-10-13T05:21:37.757757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.846458+0000","last_change":"2019-10-13T05:21:36.845532+0000","last_active":"2019-10-13T05:21:36.846458+0000","last_peered":"2019-10-13T05:21:36.846458+0000","last_clean":"2019-10-13T05:21:36.846458+0000","last_became_active":"2019-10-13T05:21:36.844901+0000","last_became_peered":"2019-10-13T05:21:36.844901+0000","last_unstale":"2019-10-13T05:21:36.846458+0000","last_undegraded":"2019-10-13T05:21:36.846458+0000","last_fullsized":"2019-10-13T05:21:36.846458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.952101+0000","last_change":"2019-10-13T05:21:33.499724+0000","last_active":"2019-10-13T05:21:37.952101+0000","last_peered":"2019-10-13T05:21:37.952101+0000","last_clean":"2019-10-13T05:21:37.952101+0000","last_became_active":"2019-10-13T05:21:33.499351+0000","last_became_peered":"2019-10-13T05:21:33.499351+0000","last_unstale":"2019-10-13T05:21:37.952101+0000","last_undegraded":"2019-10-13T05:21:37.952101+0000","last_fullsized":"2019-10-13T05:21:37.952101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.920569+0000","last_change":"2019-10-13T05:21:36.535530+0000","last_active":"2019-10-13T05:21:39.920569+0000","last_peered":"2019-10-13T05:21:39.920569+0000","last_clean":"2019-10-13T05:21:39.920569+0000","last_became_active":"2019-10-13T05:21:36.535206+0000","last_became_peered":"2019-10-13T05:21:36.535206+0000","last_unstale":"2019-10-13T05:21:39.920569+0000","last_undegraded":"2019-10-13T05:21:39.920569+0000","last_fullsized":"2019-10-13T05:21:39.920569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.850536+0000","last_change":"2019-10-13T05:21:36.849029+0000","last_active":"2019-10-13T05:21:36.850536+0000","last_peered":"2019-10-13T05:21:36.850536+0000","last_clean":"2019-10-13T05:21:36.850536+0000","last_became_active":"2019-10-13T05:21:36.848571+0000","last_became_peered":"2019-10-13T05:21:36.848571+0000","last_unstale":"2019-10-13T05:21:36.850536+0000","last_undegraded":"2019-10-13T05:21:36.850536+0000","last_fullsized":"2019-10-13T05:21:36.850536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.466631+0000","last_change":"2019-10-13T05:21:34.090940+0000","last_active":"2019-10-13T05:21:37.466631+0000","last_peered":"2019-10-13T05:21:37.466631+0000","last_clean":"2019-10-13T05:21:37.466631+0000","last_became_active":"2019-10-13T05:21:34.090335+0000","last_became_peered":"2019-10-13T05:21:34.090335+0000","last_unstale":"2019-10-13T05:21:37.466631+0000","last_undegraded":"2019-10-13T05:21:37.466631+0000","last_fullsized":"2019-10-13T05:21:37.466631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:37.844022+0000","last_change":"2019-10-13T05:21:36.533426+0000","last_active":"2019-10-13T05:21:37.844022+0000","last_peered":"2019-10-13T05:21:37.844022+0000","last_clean":"2019-10-13T05:21:37.844022+0000","last_became_active":"2019-10-13T05:21:36.532814+0000","last_became_peered":"2019-10-13T05:21:36.532814+0000","last_unstale":"2019-10-13T05:21:37.844022+0000","last_undegraded":"2019-10-13T05:21:37.844022+0000","last_fullsized":"2019-10-13T05:21:37.844022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.857350+0000","last_change":"2019-10-13T05:21:36.852431+0000","last_active":"2019-10-13T05:21:36.857350+0000","last_peered":"2019-10-13T05:21:36.857350+0000","last_clean":"2019-10-13T05:21:36.857350+0000","last_became_active":"2019-10-13T05:21:36.852190+0000","last_became_peered":"2019-10-13T05:21:36.852190+0000","last_unstale":"2019-10-13T05:21:36.857350+0000","last_undegraded":"2019-10-13T05:21:36.857350+0000","last_fullsized":"2019-10-13T05:21:36.857350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.467022+0000","last_change":"2019-10-13T05:21:34.091019+0000","last_active":"2019-10-13T05:21:37.467022+0000","last_peered":"2019-10-13T05:21:37.467022+0000","last_clean":"2019-10-13T05:21:37.467022+0000","last_became_active":"2019-10-13T05:21:34.089768+0000","last_became_peered":"2019-10-13T05:21:34.089768+0000","last_unstale":"2019-10-13T05:21:37.467022+0000","last_undegraded":"2019-10-13T05:21:37.467022+0000","last_fullsized":"2019-10-13T05:21:37.467022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919053+0000","last_change":"2019-10-13T05:21:36.532329+0000","last_active":"2019-10-13T05:21:39.919053+0000","last_peered":"2019-10-13T05:21:39.919053+0000","last_clean":"2019-10-13T05:21:39.919053+0000","last_became_active":"2019-10-13T05:21:36.531585+0000","last_became_peered":"2019-10-13T05:21:36.531585+0000","last_unstale":"2019-10-13T05:21:39.919053+0000","last_undegraded":"2019-10-13T05:21:39.919053+0000","last_fullsized":"2019-10-13T05:21:39.919053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.850838+0000","last_change":"2019-10-13T05:21:36.849965+0000","last_active":"2019-10-13T05:21:36.850838+0000","last_peered":"2019-10-13T05:21:36.850838+0000","last_clean":"2019-10-13T05:21:36.850838+0000","last_became_active":"2019-10-13T05:21:36.849632+0000","last_became_peered":"2019-10-13T05:21:36.849632+0000","last_unstale":"2019-10-13T05:21:36.850838+0000","last_undegraded":"2019-10-13T05:21:36.850838+0000","last_fullsized":"2019-10-13T05:21:36.850838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.952106+0000","last_change":"2019-10-13T05:21:34.090601+0000","last_active":"2019-10-13T05:21:37.952106+0000","last_peered":"2019-10-13T05:21:37.952106+0000","last_clean":"2019-10-13T05:21:37.952106+0000","last_became_active":"2019-10-13T05:21:34.090284+0000","last_became_peered":"2019-10-13T05:21:34.090284+0000","last_unstale":"2019-10-13T05:21:37.952106+0000","last_undegraded":"2019-10-13T05:21:37.952106+0000","last_fullsized":"2019-10-13T05:21:37.952106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.917881+0000","last_change":"2019-10-13T05:21:36.529432+0000","last_active":"2019-10-13T05:21:39.917881+0000","last_peered":"2019-10-13T05:21:39.917881+0000","last_clean":"2019-10-13T05:21:39.917881+0000","last_became_active":"2019-10-13T05:21:36.529027+0000","last_became_peered":"2019-10-13T05:21:36.529027+0000","last_unstale":"2019-10-13T05:21:39.917881+0000","last_undegraded":"2019-10-13T05:21:39.917881+0000","last_fullsized":"2019-10-13T05:21:39.917881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049296,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322544,"statfs":{"total":96636764160,"available":95562285056,"internally_reserved":1073741824,"allocated":720896,"data_stored":75387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.507}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.852}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.914}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.289}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.802}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.141}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049344,"kb_used_data":752,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322496,"statfs":{"total":96636764160,"available":95562235904,"internally_reserved":1073741824,"allocated":770048,"data_stored":76913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.830}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.978}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.675}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.639}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.946}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.094}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.135}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049296,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322544,"statfs":{"total":96636764160,"available":95562285056,"internally_reserved":1073741824,"allocated":720896,"data_stored":75387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.764}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.581}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.038}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.243}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.148}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.303}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049328,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322512,"statfs":{"total":96636764160,"available":95562252288,"internally_reserved":1073741824,"allocated":753664,"data_stored":75953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.565}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.960}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.536}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.018}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.894}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.254}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.292}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049312,"kb_used_data":720,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322528,"statfs":{"total":96636764160,"available":95562268672,"internally_reserved":1073741824,"allocated":737280,"data_stored":75421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.646}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.805}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.936}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.854}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.056}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.153}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":76147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.511}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.609}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.777}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.963}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.983}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.433}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049328,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322512,"statfs":{"total":96636764160,"available":95562252288,"internally_reserved":1073741824,"allocated":753664,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.686}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.533}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.072}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.134}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.757}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.496}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049344,"kb_used_data":752,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322496,"statfs":{"total":96636764160,"available":95562235904,"internally_reserved":1073741824,"allocated":770048,"data_stored":76913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.779}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.690}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.153}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.420}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.112}]}]}]}} 2019-10-13T05:21:48.511 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:48.511 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T05:21:48.612 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 Processor -- start 2019-10-13T05:21:48.612 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 -- start start 2019-10-13T05:21:48.612 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb8e8144a10 0x7fb8e8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e80a7a30 con 0x7fb8e80a1c80 2019-10-13T05:21:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e80a96b0 con 0x7fb8e8144a10 2019-10-13T05:21:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ef77e700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80a9950 con 0x7fb8e80a7520 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.129:44280/0 (socket says 172.21.15.129:44280) 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 -- 172.21.15.129:0/2414782269 learned_addr learned my addr 172.21.15.129:0/2414782269 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ee51c700 1 --2- 172.21.15.129:0/2414782269 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8144a10 0x7fb8e8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 -- 172.21.15.129:0/2414782269 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8144a10 msgr2=0x7fb8e8144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8144a10 0x7fb8e8144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 -- 172.21.15.129:0/2414782269 >> v1:172.21.15.71:6789/0 conn(0x7fb8e80a1c80 legacy=0x7fb8e80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 -- 172.21.15.129:0/2414782269 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e80aa3e0 con 0x7fb8e80a7520 2019-10-13T05:21:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).ready entity=mon.2 client_cookie=b6a159eedbac149c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.601+0000 7fb8ecd19700 1 -- 172.21.15.129:0/2414782269 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d8010c40 con 0x7fb8e80a7520 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ecd19700 1 -- 172.21.15.129:0/2414782269 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8010860 con 0x7fb8e80a7520 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ecd19700 1 -- 172.21.15.129:0/2414782269 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d80114a0 con 0x7fb8e80a7520 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 msgr2=0x7fb8e80a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7fb8d80032b0 tx=0x7fb8e0007710).stop 2019-10-13T05:21:48.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- 172.21.15.129:0/2414782269 shutdown_connections 2019-10-13T05:21:48.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/2414782269 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8144a10 0x7fb8e8144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/2414782269 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e80a7520 0x7fb8e80a9e90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.617 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- 172.21.15.129:0/2414782269 shutdown_connections 2019-10-13T05:21:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- 172.21.15.129:0/2414782269 wait complete. 2019-10-13T05:21:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- 172.21.15.129:0/2414782269 >> 172.21.15.129:0/2414782269 conn(0x7fb8e805e870 msgr2=0x7fb8e809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 Processor -- start 2019-10-13T05:21:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.605+0000 7fb8ef77e700 1 -- start start 2019-10-13T05:21:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 0x7fb8e812e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e8131e30 con 0x7fb8e80a6960 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e8065790 con 0x7fb8e8131a10 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e809e2d0 con 0x7fb8e8122030 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ed51a700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 0x7fb8e812e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ee51c700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb8e80a6960 0x7fb8e80a56c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:53946/0 (socket says 172.21.15.129:53946) 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ee51c700 1 -- 172.21.15.129:0/1761202640 learned_addr learned my addr 172.21.15.129:0/1761202640 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 -- 172.21.15.129:0/1761202640 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 msgr2=0x7fb8e812e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 0x7fb8e812e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 -- 172.21.15.129:0/1761202640 >> v1:172.21.15.71:6789/0 conn(0x7fb8e80a6960 legacy=0x7fb8e80a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788080275 0 0) 0x7fb8e8131e30 con 0x7fb8e80a6960 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 -- 172.21.15.129:0/1761202640 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e80a6d80 con 0x7fb8e8122030 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/1761202640 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 0x7fb8e812e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d000ca70 con 0x7fb8e8122030 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d000c690 con 0x7fb8e8122030 2019-10-13T05:21:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80a5ec0 con 0x7fb8e8122030 2019-10-13T05:21:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d0010e60 con 0x7fb8e8122030 2019-10-13T05:21:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e80a6470 con 0x7fb8e8122030 2019-10-13T05:21:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.609+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb8d0011100 con 0x7fb8e8122030 2019-10-13T05:21:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8e67fc700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 0x7fb8cc029c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fb8d0034c10 con 0x7fb8e8122030 2019-10-13T05:21:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 0x7fb8cc029c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.613+0000 7fb8ed51a700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 0x7fb8cc029c70 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.617+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e80555c0 con 0x7fb8e8122030 2019-10-13T05:21:48.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.625+0000 7fb8edd1b700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.625+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb8d00113f0 con 0x7fb8e8122030 2019-10-13T05:21:48.796 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.789+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 --> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8e8067b90 con 0x7fb8cc027770 2019-10-13T05:21:48.803 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.797+0000 7fb8e67fc700 1 -- 172.21.15.129:0/1761202640 <== mgr.4104 v2:172.21.15.71:6800/13713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80341 (crc 0 0 0) 0x7fb8e8067b90 con 0x7fb8cc027770 2019-10-13T05:21:48.813 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:48.814 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2019-10-13T05:21:48.815 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 msgr2=0x7fb8cc029c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 0x7fb8cc029c70 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 msgr2=0x7fb8e8132490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fb8d0001430 tx=0x7fb8d8007ad0).stop 2019-10-13T05:21:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 shutdown_connections 2019-10-13T05:21:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb8e8122030 0x7fb8e8132490 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/1761202640 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb8cc027770 0x7fb8cc029c70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 --2- 172.21.15.129:0/1761202640 >> v2:172.21.15.71:3300/0 conn(0x7fb8e8131a10 0x7fb8e812e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 shutdown_connections 2019-10-13T05:21:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 wait complete. 2019-10-13T05:21:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:48.809+0000 7fb8ef77e700 1 -- 172.21.15.129:0/1761202640 >> 172.21.15.129:0/1761202640 conn(0x7fb8e8059eb0 msgr2=0x7fb8e811f900 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.827 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2019-10-13T05:21:48.266828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8394624,"kb_used_data":5888,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746580096,"statfs":{"total":773094113280,"available":764498018304,"internally_reserved":8589934592,"allocated":6029312,"data_stored":607668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001264"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.851972+0000","last_change":"2019-10-13T05:21:36.850565+0000","last_active":"2019-10-13T05:21:36.851972+0000","last_peered":"2019-10-13T05:21:36.851972+0000","last_clean":"2019-10-13T05:21:36.851972+0000","last_became_active":"2019-10-13T05:21:36.849754+0000","last_became_peered":"2019-10-13T05:21:36.849754+0000","last_unstale":"2019-10-13T05:21:36.851972+0000","last_undegraded":"2019-10-13T05:21:36.851972+0000","last_fullsized":"2019-10-13T05:21:36.851972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.841685+0000","last_change":"2019-10-13T05:21:34.091136+0000","last_active":"2019-10-13T05:21:36.841685+0000","last_peered":"2019-10-13T05:21:36.841685+0000","last_clean":"2019-10-13T05:21:36.841685+0000","last_became_active":"2019-10-13T05:21:34.090386+0000","last_became_peered":"2019-10-13T05:21:34.090386+0000","last_unstale":"2019-10-13T05:21:36.841685+0000","last_undegraded":"2019-10-13T05:21:36.841685+0000","last_fullsized":"2019-10-13T05:21:36.841685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.916975+0000","last_change":"2019-10-13T05:21:36.688806+0000","last_active":"2019-10-13T05:21:39.916975+0000","last_peered":"2019-10-13T05:21:39.916975+0000","last_clean":"2019-10-13T05:21:39.916975+0000","last_became_active":"2019-10-13T05:21:36.687995+0000","last_became_peered":"2019-10-13T05:21:36.687995+0000","last_unstale":"2019-10-13T05:21:39.916975+0000","last_undegraded":"2019-10-13T05:21:39.916975+0000","last_fullsized":"2019-10-13T05:21:39.916975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.763699+0000","last_change":"2019-10-13T05:21:37.762797+0000","last_active":"2019-10-13T05:21:37.763699+0000","last_peered":"2019-10-13T05:21:37.763699+0000","last_clean":"2019-10-13T05:21:37.763699+0000","last_became_active":"2019-10-13T05:21:37.762354+0000","last_became_peered":"2019-10-13T05:21:37.762354+0000","last_unstale":"2019-10-13T05:21:37.763699+0000","last_undegraded":"2019-10-13T05:21:37.763699+0000","last_fullsized":"2019-10-13T05:21:37.763699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.757827+0000","last_change":"2019-10-13T05:21:33.499568+0000","last_active":"2019-10-13T05:21:37.757827+0000","last_peered":"2019-10-13T05:21:37.757827+0000","last_clean":"2019-10-13T05:21:37.757827+0000","last_became_active":"2019-10-13T05:21:33.499124+0000","last_became_peered":"2019-10-13T05:21:33.499124+0000","last_unstale":"2019-10-13T05:21:37.757827+0000","last_undegraded":"2019-10-13T05:21:37.757827+0000","last_fullsized":"2019-10-13T05:21:37.757827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":"17","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919198+0000","last_change":"2019-10-13T05:21:36.688467+0000","last_active":"2019-10-13T05:21:39.919198+0000","last_peered":"2019-10-13T05:21:39.919198+0000","last_clean":"2019-10-13T05:21:39.919198+0000","last_became_active":"2019-10-13T05:21:36.687973+0000","last_became_peered":"2019-10-13T05:21:36.687973+0000","last_unstale":"2019-10-13T05:21:39.919198+0000","last_undegraded":"2019-10-13T05:21:39.919198+0000","last_fullsized":"2019-10-13T05:21:39.919198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.852058+0000","last_change":"2019-10-13T05:21:36.849279+0000","last_active":"2019-10-13T05:21:36.852058+0000","last_peered":"2019-10-13T05:21:36.852058+0000","last_clean":"2019-10-13T05:21:36.852058+0000","last_became_active":"2019-10-13T05:21:36.848992+0000","last_became_peered":"2019-10-13T05:21:36.848992+0000","last_unstale":"2019-10-13T05:21:36.852058+0000","last_undegraded":"2019-10-13T05:21:36.852058+0000","last_fullsized":"2019-10-13T05:21:36.852058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.842552+0000","last_change":"2019-10-13T05:21:33.499961+0000","last_active":"2019-10-13T05:21:36.842552+0000","last_peered":"2019-10-13T05:21:36.842552+0000","last_clean":"2019-10-13T05:21:36.842552+0000","last_became_active":"2019-10-13T05:21:33.499518+0000","last_became_peered":"2019-10-13T05:21:33.499518+0000","last_unstale":"2019-10-13T05:21:36.842552+0000","last_undegraded":"2019-10-13T05:21:36.842552+0000","last_fullsized":"2019-10-13T05:21:36.842552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919904+0000","last_change":"2019-10-13T05:21:36.533229+0000","last_active":"2019-10-13T05:21:39.919904+0000","last_peered":"2019-10-13T05:21:39.919904+0000","last_clean":"2019-10-13T05:21:39.919904+0000","last_became_active":"2019-10-13T05:21:36.532929+0000","last_became_peered":"2019-10-13T05:21:36.532929+0000","last_unstale":"2019-10-13T05:21:39.919904+0000","last_undegraded":"2019-10-13T05:21:39.919904+0000","last_fullsized":"2019-10-13T05:21:39.919904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.763476+0000","last_change":"2019-10-13T05:21:37.762836+0000","last_active":"2019-10-13T05:21:37.763476+0000","last_peered":"2019-10-13T05:21:37.763476+0000","last_clean":"2019-10-13T05:21:37.763476+0000","last_became_active":"2019-10-13T05:21:37.762447+0000","last_became_peered":"2019-10-13T05:21:37.762447+0000","last_unstale":"2019-10-13T05:21:37.763476+0000","last_undegraded":"2019-10-13T05:21:37.763476+0000","last_fullsized":"2019-10-13T05:21:37.763476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.845771+0000","last_change":"2019-10-13T05:21:33.497347+0000","last_active":"2019-10-13T05:21:36.845771+0000","last_peered":"2019-10-13T05:21:36.845771+0000","last_clean":"2019-10-13T05:21:36.845771+0000","last_became_active":"2019-10-13T05:21:33.497007+0000","last_became_peered":"2019-10-13T05:21:33.497007+0000","last_unstale":"2019-10-13T05:21:36.845771+0000","last_undegraded":"2019-10-13T05:21:36.845771+0000","last_fullsized":"2019-10-13T05:21:36.845771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.757757+0000","last_change":"2019-10-13T05:21:36.527884+0000","last_active":"2019-10-13T05:21:37.757757+0000","last_peered":"2019-10-13T05:21:37.757757+0000","last_clean":"2019-10-13T05:21:37.757757+0000","last_became_active":"2019-10-13T05:21:36.527519+0000","last_became_peered":"2019-10-13T05:21:36.527519+0000","last_unstale":"2019-10-13T05:21:37.757757+0000","last_undegraded":"2019-10-13T05:21:37.757757+0000","last_fullsized":"2019-10-13T05:21:37.757757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.846458+0000","last_change":"2019-10-13T05:21:36.845532+0000","last_active":"2019-10-13T05:21:36.846458+0000","last_peered":"2019-10-13T05:21:36.846458+0000","last_clean":"2019-10-13T05:21:36.846458+0000","last_became_active":"2019-10-13T05:21:36.844901+0000","last_became_peered":"2019-10-13T05:21:36.844901+0000","last_unstale":"2019-10-13T05:21:36.846458+0000","last_undegraded":"2019-10-13T05:21:36.846458+0000","last_fullsized":"2019-10-13T05:21:36.846458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.952101+0000","last_change":"2019-10-13T05:21:33.499724+0000","last_active":"2019-10-13T05:21:37.952101+0000","last_peered":"2019-10-13T05:21:37.952101+0000","last_clean":"2019-10-13T05:21:37.952101+0000","last_became_active":"2019-10-13T05:21:33.499351+0000","last_became_peered":"2019-10-13T05:21:33.499351+0000","last_unstale":"2019-10-13T05:21:37.952101+0000","last_undegraded":"2019-10-13T05:21:37.952101+0000","last_fullsized":"2019-10-13T05:21:37.952101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.920569+0000","last_change":"2019-10-13T05:21:36.535530+0000","last_active":"2019-10-13T05:21:39.920569+0000","last_peered":"2019-10-13T05:21:39.920569+0000","last_clean":"2019-10-13T05:21:39.920569+0000","last_became_active":"2019-10-13T05:21:36.535206+0000","last_became_peered":"2019-10-13T05:21:36.535206+0000","last_unstale":"2019-10-13T05:21:39.920569+0000","last_undegraded":"2019-10-13T05:21:39.920569+0000","last_fullsized":"2019-10-13T05:21:39.920569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.850536+0000","last_change":"2019-10-13T05:21:36.849029+0000","last_active":"2019-10-13T05:21:36.850536+0000","last_peered":"2019-10-13T05:21:36.850536+0000","last_clean":"2019-10-13T05:21:36.850536+0000","last_became_active":"2019-10-13T05:21:36.848571+0000","last_became_peered":"2019-10-13T05:21:36.848571+0000","last_unstale":"2019-10-13T05:21:36.850536+0000","last_undegraded":"2019-10-13T05:21:36.850536+0000","last_fullsized":"2019-10-13T05:21:36.850536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.466631+0000","last_change":"2019-10-13T05:21:34.090940+0000","last_active":"2019-10-13T05:21:37.466631+0000","last_peered":"2019-10-13T05:21:37.466631+0000","last_clean":"2019-10-13T05:21:37.466631+0000","last_became_active":"2019-10-13T05:21:34.090335+0000","last_became_peered":"2019-10-13T05:21:34.090335+0000","last_unstale":"2019-10-13T05:21:37.466631+0000","last_undegraded":"2019-10-13T05:21:37.466631+0000","last_fullsized":"2019-10-13T05:21:37.466631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:37.844022+0000","last_change":"2019-10-13T05:21:36.533426+0000","last_active":"2019-10-13T05:21:37.844022+0000","last_peered":"2019-10-13T05:21:37.844022+0000","last_clean":"2019-10-13T05:21:37.844022+0000","last_became_active":"2019-10-13T05:21:36.532814+0000","last_became_peered":"2019-10-13T05:21:36.532814+0000","last_unstale":"2019-10-13T05:21:37.844022+0000","last_undegraded":"2019-10-13T05:21:37.844022+0000","last_fullsized":"2019-10-13T05:21:37.844022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.857350+0000","last_change":"2019-10-13T05:21:36.852431+0000","last_active":"2019-10-13T05:21:36.857350+0000","last_peered":"2019-10-13T05:21:36.857350+0000","last_clean":"2019-10-13T05:21:36.857350+0000","last_became_active":"2019-10-13T05:21:36.852190+0000","last_became_peered":"2019-10-13T05:21:36.852190+0000","last_unstale":"2019-10-13T05:21:36.857350+0000","last_undegraded":"2019-10-13T05:21:36.857350+0000","last_fullsized":"2019-10-13T05:21:36.857350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.467022+0000","last_change":"2019-10-13T05:21:34.091019+0000","last_active":"2019-10-13T05:21:37.467022+0000","last_peered":"2019-10-13T05:21:37.467022+0000","last_clean":"2019-10-13T05:21:37.467022+0000","last_became_active":"2019-10-13T05:21:34.089768+0000","last_became_peered":"2019-10-13T05:21:34.089768+0000","last_unstale":"2019-10-13T05:21:37.467022+0000","last_undegraded":"2019-10-13T05:21:37.467022+0000","last_fullsized":"2019-10-13T05:21:37.467022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.919053+0000","last_change":"2019-10-13T05:21:36.532329+0000","last_active":"2019-10-13T05:21:39.919053+0000","last_peered":"2019-10-13T05:21:39.919053+0000","last_clean":"2019-10-13T05:21:39.919053+0000","last_became_active":"2019-10-13T05:21:36.531585+0000","last_became_peered":"2019-10-13T05:21:36.531585+0000","last_unstale":"2019-10-13T05:21:39.919053+0000","last_undegraded":"2019-10-13T05:21:39.919053+0000","last_fullsized":"2019-10-13T05:21:39.919053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:36.850838+0000","last_change":"2019-10-13T05:21:36.849965+0000","last_active":"2019-10-13T05:21:36.850838+0000","last_peered":"2019-10-13T05:21:36.850838+0000","last_clean":"2019-10-13T05:21:36.850838+0000","last_became_active":"2019-10-13T05:21:36.849632+0000","last_became_peered":"2019-10-13T05:21:36.849632+0000","last_unstale":"2019-10-13T05:21:36.850838+0000","last_undegraded":"2019-10-13T05:21:36.850838+0000","last_fullsized":"2019-10-13T05:21:36.850838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:36.513983+0000","last_clean_scrub_stamp":"2019-10-13T05:21:36.513983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2019-10-13T05:21:37.952106+0000","last_change":"2019-10-13T05:21:34.090601+0000","last_active":"2019-10-13T05:21:37.952106+0000","last_peered":"2019-10-13T05:21:37.952106+0000","last_clean":"2019-10-13T05:21:37.952106+0000","last_became_active":"2019-10-13T05:21:34.090284+0000","last_became_peered":"2019-10-13T05:21:34.090284+0000","last_unstale":"2019-10-13T05:21:37.952106+0000","last_undegraded":"2019-10-13T05:21:37.952106+0000","last_fullsized":"2019-10-13T05:21:37.952106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:32.463516+0000","last_clean_scrub_stamp":"2019-10-13T05:21:32.463516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2019-10-13T05:21:39.917881+0000","last_change":"2019-10-13T05:21:36.529432+0000","last_active":"2019-10-13T05:21:39.917881+0000","last_peered":"2019-10-13T05:21:39.917881+0000","last_clean":"2019-10-13T05:21:39.917881+0000","last_became_active":"2019-10-13T05:21:36.529027+0000","last_became_peered":"2019-10-13T05:21:36.529027+0000","last_unstale":"2019-10-13T05:21:39.917881+0000","last_undegraded":"2019-10-13T05:21:39.917881+0000","last_fullsized":"2019-10-13T05:21:39.917881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T05:21:35.502513+0000","last_clean_scrub_stamp":"2019-10-13T05:21:35.502513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049296,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322544,"statfs":{"total":96636764160,"available":95562285056,"internally_reserved":1073741824,"allocated":720896,"data_stored":75387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.507}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.852}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.914}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.289}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.802}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.141}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049344,"kb_used_data":752,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322496,"statfs":{"total":96636764160,"available":95562235904,"internally_reserved":1073741824,"allocated":770048,"data_stored":76913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.830}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.978}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.675}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.639}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.946}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.094}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.135}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049296,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322544,"statfs":{"total":96636764160,"available":95562285056,"internally_reserved":1073741824,"allocated":720896,"data_stored":75387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.764}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.581}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.038}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.243}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.148}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.303}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049328,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322512,"statfs":{"total":96636764160,"available":95562252288,"internally_reserved":1073741824,"allocated":753664,"data_stored":75953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.565}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.960}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.536}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.018}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.894}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.254}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.292}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049312,"kb_used_data":720,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322528,"statfs":{"total":96636764160,"available":95562268672,"internally_reserved":1073741824,"allocated":737280,"data_stored":75421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.646}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.805}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.936}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.854}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.056}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.153}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":76147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.511}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.609}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.777}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.963}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.983}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.433}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049328,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322512,"statfs":{"total":96636764160,"available":95562252288,"internally_reserved":1073741824,"allocated":753664,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.686}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.533}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.072}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.134}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.757}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.496}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049344,"kb_used_data":752,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322496,"statfs":{"total":96636764160,"available":95562235904,"internally_reserved":1073741824,"allocated":770048,"data_stored":76913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.779}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.690}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.153}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.420}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.112}]}]}]}} 2019-10-13T05:21:48.831 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-10-13T05:21:48.831 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-10-13T05:21:48.831 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:48.832 INFO:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2019-10-13T05:21:48.919 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 Processor -- start 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 -- start start 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 0x7fd6400a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 0x7fd6400a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd640144a10 0x7fd640144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400a96b0 con 0x7fd6400a7520 2019-10-13T05:21:48.920 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400a9950 con 0x7fd640144a10 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd647bb2700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400a9bf0 con 0x7fd6400a1c80 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd640144a10 0x7fd640144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd640144a10 0x7fd640144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41396/0 (socket says 172.21.15.129:41396) 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 -- 172.21.15.129:0/2345101409 learned_addr learned my addr 172.21.15.129:0/2345101409 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd64594e700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 0x7fd6400a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd64614f700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 0x7fd6400a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.921 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 -- 172.21.15.129:0/2345101409 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 msgr2=0x7fd6400a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 0x7fd6400a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 -- 172.21.15.129:0/2345101409 >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 msgr2=0x7fd6400a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 0x7fd6400a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 -- 172.21.15.129:0/2345101409 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400a7a30 con 0x7fd640144a10 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd64594e700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 0x7fd6400a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd64614f700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 0x7fd6400a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:48.922 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.129:3300/0 conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6400a9870 tx=0x7fd630007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.129:3300/0 conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6400a9870 tx=0x7fd630007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.909+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fd6400a9870 tx=0x7fd630007710).ready entity=mon.1 client_cookie=6c89eb3b733f2947 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fd6400a9870 tx=0x7fd630007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd64514d700 1 -- 172.21.15.129:0/2345101409 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6380070c0 con 0x7fd640144a10 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fd6400a9870 tx=0x7fd630007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd646950700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fd6400a9870 tx=0x7fd630007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd64514d700 1 -- 172.21.15.129:0/2345101409 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638006be0 con 0x7fd640144a10 2019-10-13T05:21:48.923 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd64514d700 1 -- 172.21.15.129:0/2345101409 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6380076a0 con 0x7fd640144a10 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 msgr2=0x7fd640144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fd6400a9870 tx=0x7fd630007710).stop 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- 172.21.15.129:0/2345101409 shutdown_connections 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd640144a10 0x7fd640144e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 --2- 172.21.15.129:0/2345101409 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fd6400a1c80 0x7fd6400a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.924 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 --2- 172.21.15.129:0/2345101409 >> v2:172.21.15.71:3300/0 conn(0x7fd6400a7520 0x7fd6400a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:48.925 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- 172.21.15.129:0/2345101409 shutdown_connections 2019-10-13T05:21:48.925 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- 172.21.15.129:0/2345101409 wait complete. 2019-10-13T05:21:48.925 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- 172.21.15.129:0/2345101409 >> 172.21.15.129:0/2345101409 conn(0x7fd64005e870 msgr2=0x7fd64009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:48.928 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 Processor -- start 2019-10-13T05:21:48.928 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.913+0000 7fd647bb2700 1 -- start start 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6400a40b0 0x7fd640064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd640064790 0x7fd64013b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400a44d0 con 0x7fd6401204e0 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6401213d0 con 0x7fd640064790 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd640121630 con 0x7fd6400a40b0 2019-10-13T05:21:48.929 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6400a40b0 0x7fd640064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6400a40b0 0x7fd640064240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41398/0 (socket says 172.21.15.129:41398) 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 -- 172.21.15.129:0/844801946 learned_addr learned my addr 172.21.15.129:0/844801946 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd646950700 1 --2- 172.21.15.129:0/844801946 >> v2:172.21.15.71:3300/0 conn(0x7fd640064790 0x7fd64013b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 -- 172.21.15.129:0/844801946 >> v2:172.21.15.71:3300/0 conn(0x7fd640064790 msgr2=0x7fd64013b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> v2:172.21.15.71:3300/0 conn(0x7fd640064790 0x7fd64013b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955187933 0 0) 0x7fd6400a44d0 con 0x7fd6401204e0 2019-10-13T05:21:48.930 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 -- 172.21.15.129:0/844801946 >> v1:172.21.15.129:6789/0 conn(0x7fd6401204e0 legacy=0x7fd6401349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 -- 172.21.15.129:0/844801946 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400a3dd0 con 0x7fd6400a40b0 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> v2:172.21.15.129:3300/0 conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> v2:172.21.15.129:3300/0 conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6300124f0 con 0x7fd6400a40b0 2019-10-13T05:21:48.931 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630007bc0 con 0x7fd6400a40b0 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640121b70 con 0x7fd6400a40b0 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd630012ad0 con 0x7fd6400a40b0 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6401351c0 con 0x7fd6400a40b0 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.932 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd630009180 con 0x7fd6400a40b0 2019-10-13T05:21:48.933 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.917+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.933 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.921+0000 7fd636ffd700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 0x7fd628029d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:48.933 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.921+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd630034850 con 0x7fd6400a40b0 2019-10-13T05:21:48.933 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.921+0000 7fd64614f700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 0x7fd628029d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:48.933 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.921+0000 7fd64614f700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 0x7fd628029d90 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:48.934 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.925+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6400555c0 con 0x7fd6400a40b0 2019-10-13T05:21:48.935 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.929+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:48.935 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:48.929+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd630009470 con 0x7fd6400a40b0 2019-10-13T05:21:49.131 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.105+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7fd640067b90 con 0x7fd6400a40b0 2019-10-13T05:21:49.131 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.105+0000 7fd64594e700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.131 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.105+0000 7fd636ffd700 1 -- 172.21.15.129:0/844801946 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7fd630009690 con 0x7fd6400a40b0 2019-10-13T05:21:49.131 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 msgr2=0x7fd628029d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.131 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 0x7fd628029d90 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 msgr2=0x7fd640064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7fd630002ef0 tx=0x7fd63c002f70).stop 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 shutdown_connections 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fd628027890 0x7fd628029d90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 --2- 172.21.15.129:0/844801946 >> v2:172.21.15.71:3300/0 conn(0x7fd640064790 0x7fd64013b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 --2- 172.21.15.129:0/844801946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6400a40b0 0x7fd640064240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.132 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 shutdown_connections 2019-10-13T05:21:49.133 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 wait complete. 2019-10-13T05:21:49.133 INFO:teuthology.misc.health.smithi129.stderr:2019-10-13T05:21:49.109+0000 7fd647bb2700 1 -- 172.21.15.129:0/844801946 >> 172.21.15.129:0/844801946 conn(0x7fd6400a3120 msgr2=0x7fd6400a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.133 INFO:teuthology.misc.health.smithi129.stdout:HEALTH_OK 2019-10-13T05:21:49.133 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2019-10-13T05:21:49.134 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:49.134 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:49.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 Processor -- start 2019-10-13T05:21:49.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 -- start start 2019-10-13T05:21:49.229 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4dc0a7520 0x7fb4dc0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0a7a30 con 0x7fb4dc144a10 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc0a96b0 con 0x7fb4dc0a7520 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4e105f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc0a9950 con 0x7fb4dc0a1c80 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.217+0000 7fb4db59e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41406/0 (socket says 172.21.15.129:41406) 2019-10-13T05:21:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 -- 172.21.15.129:0/1750177797 learned_addr learned my addr 172.21.15.129:0/1750177797 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/1750177797 >> v2:172.21.15.71:3300/0 conn(0x7fb4dc0a7520 0x7fb4dc0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 -- 172.21.15.129:0/1750177797 >> v2:172.21.15.71:3300/0 conn(0x7fb4dc0a7520 msgr2=0x7fb4dc0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> v2:172.21.15.71:3300/0 conn(0x7fb4dc0a7520 0x7fb4dc0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 -- 172.21.15.129:0/1750177797 >> v1:172.21.15.71:6789/0 conn(0x7fb4dc144a10 legacy=0x7fb4dc0a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 -- 172.21.15.129:0/1750177797 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0a1940 con 0x7fb4dc0a1c80 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> v2:172.21.15.129:3300/0 conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> v2:172.21.15.129:3300/0 conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).ready entity=mon.1 client_cookie=6bd2005c20e4e56f server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4da59c700 1 -- 172.21.15.129:0/1750177797 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4cc008c80 con 0x7fb4dc0a1c80 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4db59e700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4da59c700 1 -- 172.21.15.129:0/1750177797 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc0088a0 con 0x7fb4dc0a1c80 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4da59c700 1 -- 172.21.15.129:0/1750177797 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4cc0119b0 con 0x7fb4dc0a1c80 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 -- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 msgr2=0x7fb4dc0a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7fb4cc002f30 tx=0x7fb4c4007a90).stop 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 -- 172.21.15.129:0/1750177797 shutdown_connections 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 --2- 172.21.15.129:0/1750177797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4dc0a1c80 0x7fb4dc0a13f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 --2- 172.21.15.129:0/1750177797 >> v2:172.21.15.71:3300/0 conn(0x7fb4dc0a7520 0x7fb4dc0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.233 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 -- 172.21.15.129:0/1750177797 shutdown_connections 2019-10-13T05:21:49.234 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 -- 172.21.15.129:0/1750177797 wait complete. 2019-10-13T05:21:49.234 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.221+0000 7fb4e105f700 1 -- 172.21.15.129:0/1750177797 >> 172.21.15.129:0/1750177797 conn(0x7fb4dc05e870 msgr2=0x7fb4dc09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.237 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 Processor -- start 2019-10-13T05:21:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 -- start start 2019-10-13T05:21:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc078830 con 0x7fb4dc0a6530 2019-10-13T05:21:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc077fa0 con 0x7fb4dc0a2480 2019-10-13T05:21:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e105f700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc075560 con 0x7fb4dc0769b0 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e085e700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb4dc0a2480 0x7fb4dc12e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59386/0 (socket says 172.21.15.129:59386) 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4e085e700 1 -- 172.21.15.129:0/3580169907 learned_addr learned my addr 172.21.15.129:0/3580169907 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552680821 0 0) 0x7fb4dc077fa0 con 0x7fb4dc0a2480 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c00030a0 con 0x7fb4dc0a2480 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626903431 0 0) 0x7fb4dc078830 con 0x7fb4dc0a6530 2019-10-13T05:21:49.239 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc077fa0 con 0x7fb4dc0a6530 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 -- 172.21.15.129:0/3580169907 >> v1:172.21.15.129:6789/0 conn(0x7fb4dc0a2480 legacy=0x7fb4dc12e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 -- 172.21.15.129:0/3580169907 >> v1:172.21.15.71:6789/0 conn(0x7fb4dc0a6530 legacy=0x7fb4dc0a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 -- 172.21.15.129:0/3580169907 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc075850 con 0x7fb4dc0769b0 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.225+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.241 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.241 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c4005aa0 con 0x7fb4dc0769b0 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4011c60 con 0x7fb4dc0769b0 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0765f0 con 0x7fb4dc0769b0 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c400c7e0 con 0x7fb4dc0769b0 2019-10-13T05:21:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc0761f0 con 0x7fb4dc0769b0 2019-10-13T05:21:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb4c400ca80 con 0x7fb4dc0769b0 2019-10-13T05:21:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 0x7fb4c002a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fb4c4034d20 con 0x7fb4dc0769b0 2019-10-13T05:21:49.244 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4db59e700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 0x7fb4c002a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.244 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.229+0000 7fb4db59e700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 0x7fb4c002a420 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.244 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.233+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc0555c0 con 0x7fb4dc0769b0 2019-10-13T05:21:49.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.241+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.247 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.241+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb4c400cd70 con 0x7fb4dc0769b0 2019-10-13T05:21:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.409+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4dc05cf50 con 0x7fb4dc0769b0 2019-10-13T05:21:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.413+0000 7fb4dad9d700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.413+0000 7fb4d3fff700 1 -- 172.21.15.129:0/3580169907 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7fb4c4025840 con 0x7fb4dc0769b0 2019-10-13T05:21:49.420 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:49.421 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 msgr2=0x7fb4c002a420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 0x7fb4c002a420 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 msgr2=0x7fb4dc0782e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.425 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7fb4c40032b0 tx=0x7fb4c4011180).stop 2019-10-13T05:21:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 shutdown_connections 2019-10-13T05:21:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fb4c0027f20 0x7fb4c002a420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 --2- 172.21.15.129:0/3580169907 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fb4dc0769b0 0x7fb4dc0782e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 shutdown_connections 2019-10-13T05:21:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 wait complete. 2019-10-13T05:21:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.417+0000 7fb4e105f700 1 -- 172.21.15.129:0/3580169907 >> 172.21.15.129:0/3580169907 conn(0x7fb4dc07e5b0 msgr2=0x7fb4dc0a7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:31.757741+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:34.685611+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:35.908349+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T05:21:49.446 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:49.446 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T05:21:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 Processor -- start 2019-10-13T05:21:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 -- start start 2019-10-13T05:21:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad009d160 0x7fbad00a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbad00a82d0 0x7fbad00a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 0x7fbad0144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad00a1300 con 0x7fbad00a82d0 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad00a15a0 con 0x7fbad009d160 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad68a0700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00a1840 con 0x7fbad0144920 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad4e3d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad009d160 0x7fbad00a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad4e3d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad009d160 0x7fbad00a69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41424/0 (socket says 172.21.15.129:41424) 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad4e3d700 1 -- 172.21.15.129:0/992510720 learned_addr learned my addr 172.21.15.129:0/992510720 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:49.550 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbacffff700 1 --2- 172.21.15.129:0/992510720 >> v2:172.21.15.71:3300/0 conn(0x7fbad00a82d0 0x7fbad00a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.537+0000 7fbad563e700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 0x7fbad0144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 -- 172.21.15.129:0/992510720 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 msgr2=0x7fbad0144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 0x7fbad0144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 -- 172.21.15.129:0/992510720 >> v2:172.21.15.71:3300/0 conn(0x7fbad00a82d0 msgr2=0x7fbad00a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> v2:172.21.15.71:3300/0 conn(0x7fbad00a82d0 0x7fbad00a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 -- 172.21.15.129:0/992510720 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad00a8c40 con 0x7fbad009d160 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad563e700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 0x7fbad0144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> v2:172.21.15.129:3300/0 conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.553 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> v2:172.21.15.129:3300/0 conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.554 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).ready entity=mon.1 client_cookie=887171527dca50d9 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.554 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.554 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbacf7fe700 1 -- 172.21.15.129:0/992510720 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab80088a0 con 0x7fbad009d160 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbacf7fe700 1 -- 172.21.15.129:0/992510720 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab80073c0 con 0x7fbad009d160 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbacf7fe700 1 -- 172.21.15.129:0/992510720 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab8006560 con 0x7fbad009d160 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 msgr2=0x7fbad00a69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.555 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7fbad00a14c0 tx=0x7fbac0007a90).stop 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- 172.21.15.129:0/992510720 shutdown_connections 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0144920 0x7fbad0144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 --2- 172.21.15.129:0/992510720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad009d160 0x7fbad00a69f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 --2- 172.21.15.129:0/992510720 >> v2:172.21.15.71:3300/0 conn(0x7fbad00a82d0 0x7fbad00a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- 172.21.15.129:0/992510720 shutdown_connections 2019-10-13T05:21:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- 172.21.15.129:0/992510720 wait complete. 2019-10-13T05:21:49.557 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- 172.21.15.129:0/992510720 >> 172.21.15.129:0/992510720 conn(0x7fbad005e870 msgr2=0x7fbad009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.557 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 Processor -- start 2019-10-13T05:21:49.557 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.541+0000 7fbad68a0700 1 -- start start 2019-10-13T05:21:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad00a96a0 0x7fbad011fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbad0120f80 0x7fbad0063b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0064090 0x7fbad00a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad01209e0 con 0x7fbad0120f80 2019-10-13T05:21:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad01214c0 con 0x7fbad00a96a0 2019-10-13T05:21:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0121760 con 0x7fbad0064090 2019-10-13T05:21:49.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad00a96a0 0x7fbad011fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad00a96a0 0x7fbad011fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41432/0 (socket says 172.21.15.129:41432) 2019-10-13T05:21:49.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 -- 172.21.15.129:0/3157636179 learned_addr learned my addr 172.21.15.129:0/3157636179 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:49.565 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbacffff700 1 --2- 172.21.15.129:0/3157636179 >> v2:172.21.15.71:3300/0 conn(0x7fbad0120f80 0x7fbad0063b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad563e700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0064090 0x7fbad00a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 -- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0064090 msgr2=0x7fbad00a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0064090 0x7fbad00a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 -- 172.21.15.129:0/3157636179 >> v2:172.21.15.71:3300/0 conn(0x7fbad0120f80 msgr2=0x7fbad0063b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> v2:172.21.15.71:3300/0 conn(0x7fbad0120f80 0x7fbad0063b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 -- 172.21.15.129:0/3157636179 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0121a00 con 0x7fbad00a96a0 2019-10-13T05:21:49.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> v2:172.21.15.129:3300/0 conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> v2:172.21.15.129:3300/0 conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.567 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=READY pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab80590b0 con 0x7fbad00a96a0 2019-10-13T05:21:49.576 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8008aa0 con 0x7fbad00a96a0 2019-10-13T05:21:49.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad0120310 con 0x7fbad00a96a0 2019-10-13T05:21:49.577 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab8059690 con 0x7fbad00a96a0 2019-10-13T05:21:49.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad00a2710 con 0x7fbad00a96a0 2019-10-13T05:21:49.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.578 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fbab80599a0 con 0x7fbad00a96a0 2019-10-13T05:21:49.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.545+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.549+0000 7fbacd7fa700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 0x7fbaa8029ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.549+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fbab8078710 con 0x7fbad00a96a0 2019-10-13T05:21:49.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.549+0000 7fbacffff700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 0x7fbaa8029ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.579 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.549+0000 7fbacffff700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 0x7fbaa8029ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.580 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.553+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad00555c0 con 0x7fbad00a96a0 2019-10-13T05:21:49.581 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.561+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.582 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.561+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fbab8061160 con 0x7fbad00a96a0 2019-10-13T05:21:49.719 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.713+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fbad0067b90 con 0x7fbad00a96a0 2019-10-13T05:21:49.720 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.713+0000 7fbad4e3d700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.721 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.713+0000 7fbacd7fa700 1 -- 172.21.15.129:0/3157636179 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbab80076f0 con 0x7fbad00a96a0 2019-10-13T05:21:49.721 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 msgr2=0x7fbaa8029ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 0x7fbaa8029ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 msgr2=0x7fbad011fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 secure :-1 s=READY pgs=166 cs=0 l=1 rx=0x7fbad00aa510 tx=0x7fbac00078d0).stop 2019-10-13T05:21:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 shutdown_connections 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fbad0064090 0x7fbad00a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad00a96a0 0x7fbad011fdc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fbaa80279a0 0x7fbaa8029ea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 --2- 172.21.15.129:0/3157636179 >> v2:172.21.15.71:3300/0 conn(0x7fbad0120f80 0x7fbad0063b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 shutdown_connections 2019-10-13T05:21:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 wait complete. 2019-10-13T05:21:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.717+0000 7fbad68a0700 1 -- 172.21.15.129:0/3157636179 >> 172.21.15.129:0/3157636179 conn(0x7fbad00a0e30 msgr2=0x7fbad0121e70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.733 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:49.733 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T05:21:49.839 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 Processor -- start 2019-10-13T05:21:49.840 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 -- start start 2019-10-13T05:21:49.840 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff8040a86e0 0x7ff80414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:49.840 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8040a0fc0 con 0x7ff8040a82c0 2019-10-13T05:21:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8040a8b00 con 0x7ff80409d150 2019-10-13T05:21:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff808df6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8040a12f0 con 0x7ff8040a86e0 2019-10-13T05:21:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff802d9d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff80409d150 0x7ff8040a69e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59406/0 (socket says 172.21.15.129:59406) 2019-10-13T05:21:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff802d9d700 1 -- 172.21.15.129:0/2572219691 learned_addr learned my addr 172.21.15.129:0/2572219691 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.825+0000 7ff80359e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff8040a86e0 0x7ff80414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024774923 0 0) 0x7ff8040a8b00 con 0x7ff80409d150 2019-10-13T05:21:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f00030a0 con 0x7ff80409d150 2019-10-13T05:21:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036319739 0 0) 0x7ff8040a0fc0 con 0x7ff8040a82c0 2019-10-13T05:21:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8040a8b00 con 0x7ff8040a82c0 2019-10-13T05:21:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 -- 172.21.15.129:0/2572219691 >> v1:172.21.15.129:6789/0 conn(0x7ff80409d150 legacy=0x7ff8040a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 -- 172.21.15.129:0/2572219691 >> v1:172.21.15.71:6789/0 conn(0x7ff8040a82c0 legacy=0x7ff804144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 -- 172.21.15.129:0/2572219691 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804145110 con 0x7ff8040a86e0 2019-10-13T05:21:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> v2:172.21.15.129:3300/0 conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> v2:172.21.15.129:3300/0 conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).ready entity=mon.1 client_cookie=8ed0e0012b03a153 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.846 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc011260 con 0x7ff8040a86e0 2019-10-13T05:21:49.846 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.846 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff80359e700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.846 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc008bb0 con 0x7ff8040a86e0 2019-10-13T05:21:49.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff801d9b700 1 -- 172.21.15.129:0/2572219691 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc011840 con 0x7ff8040a86e0 2019-10-13T05:21:49.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 -- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 msgr2=0x7ff80414fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7ff7fc008780 tx=0x7ff7fc00fbe0).stop 2019-10-13T05:21:49.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 -- 172.21.15.129:0/2572219691 shutdown_connections 2019-10-13T05:21:49.847 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 --2- 172.21.15.129:0/2572219691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8040a86e0 0x7ff80414fb20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 -- 172.21.15.129:0/2572219691 shutdown_connections 2019-10-13T05:21:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 -- 172.21.15.129:0/2572219691 wait complete. 2019-10-13T05:21:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.829+0000 7ff808df6700 1 -- 172.21.15.129:0/2572219691 >> 172.21.15.129:0/2572219691 conn(0x7ff80405e860 msgr2=0x7ff80409c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 Processor -- start 2019-10-13T05:21:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 -- start start 2019-10-13T05:21:49.862 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804117cf0 con 0x7ff80407bba0 2019-10-13T05:21:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80407afe0 con 0x7ff8040a1ff0 2019-10-13T05:21:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff808df6700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804118030 con 0x7ff804064340 2019-10-13T05:21:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff802d9d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff8040a1ff0 0x7ff80407b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59412/0 (socket says 172.21.15.129:59412) 2019-10-13T05:21:49.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff802d9d700 1 -- 172.21.15.129:0/3143016648 learned_addr learned my addr 172.21.15.129:0/3143016648 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:49.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769132635 0 0) 0x7ff80407afe0 con 0x7ff8040a1ff0 2019-10-13T05:21:49.864 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e80030a0 con 0x7ff8040a1ff0 2019-10-13T05:21:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191616485 0 0) 0x7ff804117cf0 con 0x7ff80407bba0 2019-10-13T05:21:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.833+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80407afe0 con 0x7ff80407bba0 2019-10-13T05:21:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 -- 172.21.15.129:0/3143016648 >> v1:172.21.15.129:6789/0 conn(0x7ff8040a1ff0 legacy=0x7ff80407b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 -- 172.21.15.129:0/3143016648 >> v1:172.21.15.71:6789/0 conn(0x7ff80407bba0 legacy=0x7ff80407a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:49.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 -- 172.21.15.129:0/3143016648 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804063990 con 0x7ff804064340 2019-10-13T05:21:49.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.866 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc007340 con 0x7ff804064340 2019-10-13T05:21:49.867 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff804118570 con 0x7ff804064340 2019-10-13T05:21:49.870 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.870 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc011460 con 0x7ff804064340 2019-10-13T05:21:49.871 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff804133310 con 0x7ff804064340 2019-10-13T05:21:49.871 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.871 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc005910 con 0x7ff804064340 2019-10-13T05:21:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff7fc00fc60 con 0x7ff804064340 2019-10-13T05:21:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 0x7ff7e802a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:49.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7ff7fc035190 con 0x7ff804064340 2019-10-13T05:21:49.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff802d9d700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 0x7ff7e802a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:49.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.837+0000 7ff802d9d700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 0x7ff7e802a4d0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:49.873 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.841+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8040555b0 con 0x7ff804064340 2019-10-13T05:21:49.874 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.849+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:49.874 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:49.849+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff7fc00f660 con 0x7ff804064340 2019-10-13T05:21:50.010 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.005+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff804119be0 con 0x7ff804064340 2019-10-13T05:21:50.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.005+0000 7ff80359e700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.011 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.005+0000 7ff7fb7fe700 1 -- 172.21.15.129:0/3143016648 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff7fc00f880 con 0x7ff804064340 2019-10-13T05:21:50.013 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 msgr2=0x7ff7e802a4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 0x7ff7e802a4d0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 msgr2=0x7ff80407aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7ff7fc007d70 tx=0x7ff7fc008e90).stop 2019-10-13T05:21:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 shutdown_connections 2019-10-13T05:21:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7ff7e8027fd0 0x7ff7e802a4d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 --2- 172.21.15.129:0/3143016648 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7ff804064340 0x7ff80407aa90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 shutdown_connections 2019-10-13T05:21:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 wait complete. 2019-10-13T05:21:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.009+0000 7ff808df6700 1 -- 172.21.15.129:0/3143016648 >> 172.21.15.129:0/3143016648 conn(0x7ff8040a75a0 msgr2=0x7ff8041446e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:50.029 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:50.030 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T05:21:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 Processor -- start 2019-10-13T05:21:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 -- start start 2019-10-13T05:21:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3de00a82d0 0x7f3de00a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a8c40 con 0x7f3de0144920 2019-10-13T05:21:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a0fd0 con 0x7f3de009d160 2019-10-13T05:21:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de00a1300 con 0x7f3de00a82d0 2019-10-13T05:21:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3de00a82d0 0x7f3de00a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3de00a82d0 0x7f3de00a86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41452/0 (socket says 172.21.15.129:41452) 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 -- 172.21.15.129:0/3395395732 learned_addr learned my addr 172.21.15.129:0/3395395732 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 -- 172.21.15.129:0/3395395732 >> v1:172.21.15.129:6789/0 conn(0x7f3de009d160 legacy=0x7f3de00a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 -- 172.21.15.129:0/3395395732 >> v1:172.21.15.71:6789/0 conn(0x7f3de0144920 legacy=0x7f3de014eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 -- 172.21.15.129:0/3395395732 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0144d40 con 0x7f3de00a82d0 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> v2:172.21.15.129:3300/0 conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> v2:172.21.15.129:3300/0 conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=READY pgs=171 cs=0 l=1 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).ready entity=mon.1 client_cookie=3abf2b94ce9097a1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddf7fe700 1 -- 172.21.15.129:0/3395395732 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc8011510 con 0x7f3de00a82d0 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddffff700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddf7fe700 1 -- 172.21.15.129:0/3395395732 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc8008c90 con 0x7f3de00a82d0 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3ddf7fe700 1 -- 172.21.15.129:0/3395395732 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc8011af0 con 0x7f3de00a82d0 2019-10-13T05:21:50.126 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 -- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 msgr2=0x7f3de00a86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.113+0000 7f3de6949700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 secure :-1 s=READY pgs=171 cs=0 l=1 rx=0x7f3dc8008780 tx=0x7f3dc800fca0).stop 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 -- 172.21.15.129:0/3395395732 shutdown_connections 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 --2- 172.21.15.129:0/3395395732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00a82d0 0x7f3de00a86f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 -- 172.21.15.129:0/3395395732 shutdown_connections 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 -- 172.21.15.129:0/3395395732 wait complete. 2019-10-13T05:21:50.127 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 -- 172.21.15.129:0/3395395732 >> 172.21.15.129:0/3395395732 conn(0x7f3de005e870 msgr2=0x7f3de009c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:50.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 Processor -- start 2019-10-13T05:21:50.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.117+0000 7f3de6949700 1 -- start start 2019-10-13T05:21:50.131 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a5480 con 0x7f3de00a4140 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de01214c0 con 0x7f3de0125180 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a5f40 con 0x7f3de011fcd0 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de4ee6700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3de0125180 0x7f3de011e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59426/0 (socket says 172.21.15.129:59426) 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de4ee6700 1 -- 172.21.15.129:0/3797316823 learned_addr learned my addr 172.21.15.129:0/3797316823 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.132 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093561617 0 0) 0x7f3de01214c0 con 0x7f3de0125180 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc40030a0 con 0x7f3de0125180 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058633726 0 0) 0x7f3de00a5480 con 0x7f3de00a4140 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de01214c0 con 0x7f3de00a4140 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2242365890 0 0) 0x7f3dc40030a0 con 0x7f3de0125180 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de00a5480 con 0x7f3de0125180 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7f3dd8001a10 con 0x7f3de0125180 2019-10-13T05:21:50.133 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 -- 172.21.15.129:0/3797316823 >> v1:172.21.15.129:6789/0 conn(0x7f3de0125180 legacy=0x7f3de011e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 -- 172.21.15.129:0/3797316823 >> v1:172.21.15.71:6789/0 conn(0x7f3de00a4140 legacy=0x7f3de0137cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 -- 172.21.15.129:0/3797316823 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0121130 con 0x7f3de011fcd0 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.134 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd0006c90 con 0x7f3de011fcd0 2019-10-13T05:21:50.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd00068b0 con 0x7f3de011fcd0 2019-10-13T05:21:50.135 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de011f0d0 con 0x7f3de011fcd0 2019-10-13T05:21:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd0002600 con 0x7f3de011fcd0 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de013b510 con 0x7f3de011fcd0 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.121+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f3dd0002920 con 0x7f3de011fcd0 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3ddd7fa700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 0x7f3dc402a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f3dd0034b10 con 0x7f3de011fcd0 2019-10-13T05:21:50.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3de4ee6700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 0x7f3dc402a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.125+0000 7f3de4ee6700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 0x7f3dc402a8d0 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.138 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.129+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de01200f0 con 0x7f3de011fcd0 2019-10-13T05:21:50.144 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.137+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.145 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.137+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f3dd0025ec0 con 0x7f3de011fcd0 2019-10-13T05:21:50.295 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.289+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3de00555c0 con 0x7f3de011fcd0 2019-10-13T05:21:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.289+0000 7f3de56e7700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.289+0000 7f3ddd7fa700 1 -- 172.21.15.129:0/3797316823 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3dd0002c10 con 0x7f3de011fcd0 2019-10-13T05:21:50.296 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 msgr2=0x7f3dc402a8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 0x7f3dc402a8d0 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 msgr2=0x7f3de00a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f3de011fa90 tx=0x7f3dc80101b0).stop 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 shutdown_connections 2019-10-13T05:21:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f3dc40283d0 0x7f3dc402a8d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 --2- 172.21.15.129:0/3797316823 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f3de011fcd0 0x7f3de00a59f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 shutdown_connections 2019-10-13T05:21:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 wait complete. 2019-10-13T05:21:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.293+0000 7f3de6949700 1 -- 172.21.15.129:0/3797316823 >> 172.21.15.129:0/3797316823 conn(0x7f3de007eae0 msgr2=0x7f3de00a8210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:50.308 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:50.309 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T05:21:50.402 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.389+0000 7f88af0c4700 1 Processor -- start 2019-10-13T05:21:50.402 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.389+0000 7f88af0c4700 1 -- start start 2019-10-13T05:21:50.402 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a80a7520 0x7f88a80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:50.403 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8144a10 0x7f88a8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:50.405 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80a7a30 con 0x7f88a80a1c80 2019-10-13T05:21:50.405 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80a96b0 con 0x7f88a8144a10 2019-10-13T05:21:50.405 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80a9950 con 0x7f88a80a7520 2019-10-13T05:21:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8144a10 0x7f88a8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8144a10 0x7f88a8144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41470/0 (socket says 172.21.15.129:41470) 2019-10-13T05:21:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 -- 172.21.15.129:0/2673238445 learned_addr learned my addr 172.21.15.129:0/2673238445 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ace60700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a80a7520 0x7f88a80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 -- 172.21.15.129:0/2673238445 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a80a7520 msgr2=0x7f88a80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a80a7520 0x7f88a80a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 -- 172.21.15.129:0/2673238445 >> v1:172.21.15.71:6789/0 conn(0x7f88a80a1c80 legacy=0x7f88a80a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 -- 172.21.15.129:0/2673238445 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80aa3e0 con 0x7f88a8144a10 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> v2:172.21.15.129:3300/0 conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f88a80a9870 tx=0x7f88a4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> v2:172.21.15.129:3300/0 conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f88a80a9870 tx=0x7f88a4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.407 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f88a80a9870 tx=0x7f88a4007710).ready entity=mon.1 client_cookie=897f058c3e90970c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f88a80a9870 tx=0x7f88a4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f889bfff700 1 -- 172.21.15.129:0/2673238445 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a0010b90 con 0x7f88a8144a10 2019-10-13T05:21:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f88a80a9870 tx=0x7f88a4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.408 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f889bfff700 1 -- 172.21.15.129:0/2673238445 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a00107b0 con 0x7f88a8144a10 2019-10-13T05:21:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88ade62700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f88a80a9870 tx=0x7f88a4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f889bfff700 1 -- 172.21.15.129:0/2673238445 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a00112b0 con 0x7f88a8144a10 2019-10-13T05:21:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 msgr2=0x7f88a8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f88a80a9870 tx=0x7f88a4007710).stop 2019-10-13T05:21:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- 172.21.15.129:0/2673238445 shutdown_connections 2019-10-13T05:21:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8144a10 0x7f88a8144e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.413 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 --2- 172.21.15.129:0/2673238445 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a80a7520 0x7f88a80a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:50.413 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- 172.21.15.129:0/2673238445 shutdown_connections 2019-10-13T05:21:50.413 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- 172.21.15.129:0/2673238445 wait complete. 2019-10-13T05:21:50.413 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.393+0000 7f88af0c4700 1 -- 172.21.15.129:0/2673238445 >> 172.21.15.129:0/2673238445 conn(0x7f88a805e870 msgr2=0x7f88a809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 Processor -- start 2019-10-13T05:21:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 -- start start 2019-10-13T05:21:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a807b690 0x7f88a8141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8075810 0x7f88a80a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a8077d70 con 0x7f88a80a68c0 2019-10-13T05:21:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80780b0 con 0x7f88a8075810 2019-10-13T05:21:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.397+0000 7f88af0c4700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a8078350 con 0x7f88a807b690 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8075810 0x7f88a80a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a8075810 0x7f88a80a6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41474/0 (socket says 172.21.15.129:41474) 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 -- 172.21.15.129:0/1637192728 learned_addr learned my addr 172.21.15.129:0/1637192728 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ad661700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a807b690 0x7f88a8141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 -- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a807b690 msgr2=0x7f88a8141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a807b690 0x7f88a8141f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 -- 172.21.15.129:0/1637192728 >> v1:172.21.15.129:6789/0 conn(0x7f88a80a68c0 legacy=0x7f88a8131880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 -- 172.21.15.129:0/1637192728 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80a6ce0 con 0x7f88a8075810 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> v2:172.21.15.129:3300/0 conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> v2:172.21.15.129:3300/0 conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a4012c10 con 0x7f88a8075810 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4012830 con 0x7f88a8075810 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a81424e0 con 0x7f88a8075810 2019-10-13T05:21:50.421 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a4011540 con 0x7f88a8075810 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a80a0500 con 0x7f88a8075810 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f88a4011740 con 0x7f88a8075810 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 0x7f8888029e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f88a40351f0 con 0x7f88a8075810 2019-10-13T05:21:50.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ad661700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 0x7f8888029e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:50.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.401+0000 7f88ad661700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 0x7f8888029e70 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:50.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.405+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a80555c0 con 0x7f88a8075810 2019-10-13T05:21:50.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.413+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.423 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.413+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f88a4026620 con 0x7f88a8075810 2019-10-13T05:21:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.577+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a8144330 con 0x7f88a8075810 2019-10-13T05:21:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.577+0000 7f88ace60700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.577+0000 7f8899ffb700 1 -- 172.21.15.129:0/1637192728 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2105 (secure 0 0 0) 0x7f88a4011a30 con 0x7f88a8075810 2019-10-13T05:21:50.586 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:50.586 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.129:6833/1374113332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1374113332},{"type":"v1","addr":"172.21.15.129:6833","nonce":1374113332}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T05:21:36.833107+0000","modified":"2019-10-13T05:21:40.901183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4326":{"gid":4326,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.71:6835/3596066442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3596066442},{"type":"v1","addr":"172.21.15.71:6835","nonce":3596066442}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T05:21:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 7 2019-10-13T05:21:50.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 msgr2=0x7f8888029e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 0x7f8888029e70 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 msgr2=0x7f88a80a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:50.588 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f88a807c030 tx=0x7f889c001000).stop 2019-10-13T05:21:50.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 shutdown_connections 2019-10-13T05:21:50.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f8888027970 0x7f8888029e70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f88a807b690 0x7f88a8141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.589 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 --2- 172.21.15.129:0/1637192728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a8075810 0x7f88a80a6370 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 shutdown_connections 2019-10-13T05:21:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 wait complete. 2019-10-13T05:21:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:50.581+0000 7f88af0c4700 1 -- 172.21.15.129:0/1637192728 >> 172.21.15.129:0/1637192728 conn(0x7f88a80a2d90 msgr2=0x7f88a807dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:50.802 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4326}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4326': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4326, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3596066442, u'type': u'v2', u'addr': u'172.21.15.71:6834'}, {u'nonce': 3596066442, u'type': u'v1', u'addr': u'172.21.15.71:6835'}]}, u'addr': u'172.21.15.71:6835/3596066442'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T05:21:36.833107+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T05:21:40.901183+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T05:21:50.803 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T05:21:50.803 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:50.803 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T05:21:50.897 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2019-10-13T05:21:50.897 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "678bb30f-383e-4566-b2c7-8d4fe3e6c97c", 2019-10-13T05:21:50.897 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 0, 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4326, 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 10.992263467000001, 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 6, 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 21, 2019-10-13T05:21:50.898 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 21, 2019-10-13T05:21:50.899 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 11.658778153 2019-10-13T05:21:50.899 INFO:teuthology.orchestra.run.smithi071.stdout:} 2019-10-13T05:21:50.904 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "678bb30f-383e-4566-b2c7-8d4fe3e6c97c", "whoami": 0, "id": 4326, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.992263467000001, "mdsmap_epoch": 6, "osdmap_epoch": 21, "osdmap_epoch_barrier": 21, "uptime": 11.658778153 } 2019-10-13T05:21:50.904 INFO:teuthology.run_tasks:Running task ceph-fuse... 2019-10-13T05:21:50.924 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2019-10-13T05:21:50.924 INFO:tasks.ceph_fuse:config is None 2019-10-13T05:21:50.924 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2019-10-13T05:21:50.925 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2019-10-13T05:21:50.925 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:50.925 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T05:21:51.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 Processor -- start 2019-10-13T05:21:51.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 -- start start 2019-10-13T05:21:51.033 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0a7a30 con 0x7fac6c0a7520 2019-10-13T05:21:51.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0a1940 con 0x7fac6c144a10 2019-10-13T05:21:51.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac717a4700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0a96b0 con 0x7fac6c0a1c80 2019-10-13T05:21:51.034 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6bfff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fac6c144a10 0x7fac6c14eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59452/0 (socket says 172.21.15.129:59452) 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6bfff700 1 -- 172.21.15.129:0/3854664256 learned_addr learned my addr 172.21.15.129:0/3854664256 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581208744 0 0) 0x7fac6c0a1940 con 0x7fac6c144a10 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac4c0030a0 con 0x7fac6c144a10 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575330913 0 0) 0x7fac6c0a7a30 con 0x7fac6c0a7520 2019-10-13T05:21:51.035 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c0a1940 con 0x7fac6c0a7520 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 -- 172.21.15.129:0/3854664256 >> v1:172.21.15.129:6789/0 conn(0x7fac6c144a10 legacy=0x7fac6c14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 -- 172.21.15.129:0/3854664256 >> v1:172.21.15.71:6789/0 conn(0x7fac6c0a7520 legacy=0x7fac6c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 -- 172.21.15.129:0/3854664256 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c144e30 con 0x7fac6c0a1c80 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac54008b00 tx=0x7fac5400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac54008b00 tx=0x7fac5400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7fac54008b00 tx=0x7fac5400ff60).ready entity=mon.2 client_cookie=271f92af7730e0a9 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fac54008b00 tx=0x7fac5400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.037 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac540119c0 con 0x7fac6c0a1c80 2019-10-13T05:21:51.037 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fac54008b00 tx=0x7fac5400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.037 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac540115e0 con 0x7fac6c0a1c80 2019-10-13T05:21:51.037 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.021+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fac54008b00 tx=0x7fac5400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac6a7fc700 1 -- 172.21.15.129:0/3854664256 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac5400a160 con 0x7fac6c0a1c80 2019-10-13T05:21:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 msgr2=0x7fac6c0a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7fac54008b00 tx=0x7fac5400ff60).stop 2019-10-13T05:21:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- 172.21.15.129:0/3854664256 shutdown_connections 2019-10-13T05:21:51.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 --2- 172.21.15.129:0/3854664256 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fac6c0a1c80 0x7fac6c0a13f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- 172.21.15.129:0/3854664256 shutdown_connections 2019-10-13T05:21:51.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- 172.21.15.129:0/3854664256 wait complete. 2019-10-13T05:21:51.039 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- 172.21.15.129:0/3854664256 >> 172.21.15.129:0/3854664256 conn(0x7fac6c05e870 msgr2=0x7fac6c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.040 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 Processor -- start 2019-10-13T05:21:51.040 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.025+0000 7fac717a4700 1 -- start start 2019-10-13T05:21:51.044 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fac6c115f30 0x7fac6c11a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.044 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fac6c117e50 0x7fac6c07db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.044 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c118270 con 0x7fac6c07d6e0 2019-10-13T05:21:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c1173d0 con 0x7fac6c115f30 2019-10-13T05:21:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c07e050 con 0x7fac6c117e50 2019-10-13T05:21:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fac6c117e50 0x7fac6c07db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6b7fe700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fac6c115f30 0x7fac6c11a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fac6c117e50 0x7fac6c07db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41490/0 (socket says 172.21.15.129:41490) 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 -- 172.21.15.129:0/2284311567 learned_addr learned my addr 172.21.15.129:0/2284311567 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 -- 172.21.15.129:0/2284311567 >> v2:172.21.15.71:3300/0 conn(0x7fac6c115f30 msgr2=0x7fac6c11a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> v2:172.21.15.71:3300/0 conn(0x7fac6c115f30 0x7fac6c11a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 -- 172.21.15.129:0/2284311567 >> v1:172.21.15.71:6789/0 conn(0x7fac6c07d6e0 legacy=0x7fac6c117650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 -- 172.21.15.129:0/2284311567 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c111360 con 0x7fac6c117e50 2019-10-13T05:21:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> v2:172.21.15.129:3300/0 conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> v2:172.21.15.129:3300/0 conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac64005b30 con 0x7fac6c117e50 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac64005750 con 0x7fac6c117e50 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c12fd30 con 0x7fac6c117e50 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac640062b0 con 0x7fac6c117e50 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c1302b0 con 0x7fac6c117e50 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.029+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fac64006580 con 0x7fac6c117e50 2019-10-13T05:21:51.048 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac63fff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 0x7fac44029eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fac64034a90 con 0x7fac6c117e50 2019-10-13T05:21:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 0x7fac44029eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.033+0000 7fac6b7fe700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 0x7fac44029eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.037+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0555c0 con 0x7fac6c117e50 2019-10-13T05:21:51.050 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.041+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.050 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.041+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fac64025e20 con 0x7fac6c117e50 2019-10-13T05:21:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.201+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac6c067b90 con 0x7fac6c117e50 2019-10-13T05:21:51.207 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.201+0000 7fac6bfff700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.207 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.201+0000 7fac63fff700 1 -- 172.21.15.129:0/2284311567 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14672 (secure 0 0 0) 0x7fac64006870 con 0x7fac6c117e50 2019-10-13T05:21:51.209 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:51.210 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"678bb30f-383e-4566-b2c7-8d4fe3e6c97c","created":"2019-10-13T05:21:17.925040+0000","modified":"2019-10-13T05:21:37.836112+0000","last_up_change":"2019-10-13T05:21:27.419947+0000","last_in_change":"2019-10-13T05:21:27.419947+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T05:21:31.757741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T05:21:34.685611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T05:21:35.908349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"76f2fbe2-33c9-4e71-b7de-e1be85d1477f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":13880},{"type":"v1","addr":"172.21.15.129:6801","nonce":13880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":13880},{"type":"v1","addr":"172.21.15.129:6803","nonce":13880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":13880},{"type":"v1","addr":"172.21.15.129:6807","nonce":13880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":13880},{"type":"v1","addr":"172.21.15.129:6805","nonce":13880}]},"public_addr":"172.21.15.129:6801/13880","cluster_addr":"172.21.15.129:6803/13880","heartbeat_back_addr":"172.21.15.129:6807/13880","heartbeat_front_addr":"172.21.15.129:6805/13880","state":["exists","up"]},{"osd":1,"uuid":"b7fe1be3-7c39-4273-b0a8-19549cb83be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":13881},{"type":"v1","addr":"172.21.15.129:6825","nonce":13881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":13881},{"type":"v1","addr":"172.21.15.129:6827","nonce":13881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":13881},{"type":"v1","addr":"172.21.15.129:6831","nonce":13881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":13881},{"type":"v1","addr":"172.21.15.129:6829","nonce":13881}]},"public_addr":"172.21.15.129:6825/13881","cluster_addr":"172.21.15.129:6827/13881","heartbeat_back_addr":"172.21.15.129:6831/13881","heartbeat_front_addr":"172.21.15.129:6829/13881","state":["exists","up"]},{"osd":2,"uuid":"6528b803-919d-4663-9be9-f2dcebe08ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":13879},{"type":"v1","addr":"172.21.15.129:6809","nonce":13879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":13879},{"type":"v1","addr":"172.21.15.129:6811","nonce":13879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":13879},{"type":"v1","addr":"172.21.15.129:6815","nonce":13879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":13879},{"type":"v1","addr":"172.21.15.129:6813","nonce":13879}]},"public_addr":"172.21.15.129:6809/13879","cluster_addr":"172.21.15.129:6811/13879","heartbeat_back_addr":"172.21.15.129:6815/13879","heartbeat_front_addr":"172.21.15.129:6813/13879","state":["exists","up"]},{"osd":3,"uuid":"25d2f3bc-a14f-488e-8103-ee1794e0249b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":13902},{"type":"v1","addr":"172.21.15.129:6817","nonce":13902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":13902},{"type":"v1","addr":"172.21.15.129:6819","nonce":13902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":13902},{"type":"v1","addr":"172.21.15.129:6823","nonce":13902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":13902},{"type":"v1","addr":"172.21.15.129:6821","nonce":13902}]},"public_addr":"172.21.15.129:6817/13902","cluster_addr":"172.21.15.129:6819/13902","heartbeat_back_addr":"172.21.15.129:6823/13902","heartbeat_front_addr":"172.21.15.129:6821/13902","state":["exists","up"]},{"osd":4,"uuid":"bf0e5ca3-85f6-40bb-b4c5-cbe758dbaedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":14134},{"type":"v1","addr":"172.21.15.71:6827","nonce":14134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":14134},{"type":"v1","addr":"172.21.15.71:6829","nonce":14134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":14134},{"type":"v1","addr":"172.21.15.71:6833","nonce":14134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":14134},{"type":"v1","addr":"172.21.15.71:6831","nonce":14134}]},"public_addr":"172.21.15.71:6827/14134","cluster_addr":"172.21.15.71:6829/14134","heartbeat_back_addr":"172.21.15.71:6833/14134","heartbeat_front_addr":"172.21.15.71:6831/14134","state":["exists","up"]},{"osd":5,"uuid":"d15bcd3d-1a88-44e6-b0cd-7736730331c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":14132},{"type":"v1","addr":"172.21.15.71:6819","nonce":14132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":14132},{"type":"v1","addr":"172.21.15.71:6821","nonce":14132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":14132},{"type":"v1","addr":"172.21.15.71:6825","nonce":14132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":14132},{"type":"v1","addr":"172.21.15.71:6823","nonce":14132}]},"public_addr":"172.21.15.71:6819/14132","cluster_addr":"172.21.15.71:6821/14132","heartbeat_back_addr":"172.21.15.71:6825/14132","heartbeat_front_addr":"172.21.15.71:6823/14132","state":["exists","up"]},{"osd":6,"uuid":"460e48b8-6810-40b9-b018-121ed3eea51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":14133},{"type":"v1","addr":"172.21.15.71:6811","nonce":14133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":14133},{"type":"v1","addr":"172.21.15.71:6813","nonce":14133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":14133},{"type":"v1","addr":"172.21.15.71:6817","nonce":14133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":14133},{"type":"v1","addr":"172.21.15.71:6815","nonce":14133}]},"public_addr":"172.21.15.71:6811/14133","cluster_addr":"172.21.15.71:6813/14133","heartbeat_back_addr":"172.21.15.71:6817/14133","heartbeat_front_addr":"172.21.15.71:6815/14133","state":["exists","up"]},{"osd":7,"uuid":"3b87653f-36c2-470a-b30d-c9cc5a20c689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":14135},{"type":"v1","addr":"172.21.15.71:6803","nonce":14135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":14135},{"type":"v1","addr":"172.21.15.71:6805","nonce":14135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":14135},{"type":"v1","addr":"172.21.15.71:6809","nonce":14135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":14135},{"type":"v1","addr":"172.21.15.71:6807","nonce":14135}]},"public_addr":"172.21.15.71:6803/14135","cluster_addr":"172.21.15.71:6805/14135","heartbeat_back_addr":"172.21.15.71:6809/14135","heartbeat_front_addr":"172.21.15.71:6807/14135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.725528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.944785+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.766354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.784979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.088000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.024115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:26.006770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T05:21:25.802464+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T05:21:51.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 msgr2=0x7fac44029eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 0x7fac44029eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.212 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 msgr2=0x7fac6c07db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7fac64004390 tx=0x7fac54008500).stop 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 shutdown_connections 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6c117e50 0x7fac6c07db00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 --2- 172.21.15.129:0/2284311567 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fac440279b0 0x7fac44029eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 --2- 172.21.15.129:0/2284311567 >> v2:172.21.15.71:3300/0 conn(0x7fac6c115f30 0x7fac6c11a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 shutdown_connections 2019-10-13T05:21:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 wait complete. 2019-10-13T05:21:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.205+0000 7fac717a4700 1 -- 172.21.15.129:0/2284311567 >> 172.21.15.129:0/2284311567 conn(0x7fac6c0a7520 msgr2=0x7fac6c118cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:31.757741+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:34.685611+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T05:21:35.908349+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T05:21:51.223 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:51.223 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T05:21:51.331 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.317+0000 7fdb2fa99700 1 Processor -- start 2019-10-13T05:21:51.331 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.317+0000 7fdb2fa99700 1 -- start start 2019-10-13T05:21:51.331 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb2809d160 0x7fdb280a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.331 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb280a82d0 0x7fdb280a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb280a8c40 con 0x7fdb28144920 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb280a1300 con 0x7fdb280a82d0 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280a15a0 con 0x7fdb2809d160 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb280a82d0 0x7fdb280a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb280a82d0 0x7fdb280a86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41496/0 (socket says 172.21.15.129:41496) 2019-10-13T05:21:51.332 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 -- 172.21.15.129:0/3637539763 learned_addr learned my addr 172.21.15.129:0/3637539763 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb2809d160 0x7fdb280a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 -- 172.21.15.129:0/3637539763 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb2809d160 msgr2=0x7fdb280a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb2809d160 0x7fdb280a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 -- 172.21.15.129:0/3637539763 >> v1:172.21.15.71:6789/0 conn(0x7fdb28144920 legacy=0x7fdb280a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 -- 172.21.15.129:0/3637539763 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280a0fd0 con 0x7fdb280a82d0 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> v2:172.21.15.129:3300/0 conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdb24002f30 tx=0x7fdb1c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> v2:172.21.15.129:3300/0 conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdb24002f30 tx=0x7fdb1c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7fdb24002f30 tx=0x7fdb1c007710).ready entity=mon.1 client_cookie=b9a44ce73cdb7d08 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fdb24002f30 tx=0x7fdb1c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d034700 1 -- 172.21.15.129:0/3637539763 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb24008c80 con 0x7fdb280a82d0 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fdb24002f30 tx=0x7fdb1c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fdb24002f30 tx=0x7fdb1c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d034700 1 -- 172.21.15.129:0/3637539763 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb240088a0 con 0x7fdb280a82d0 2019-10-13T05:21:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2d034700 1 -- 172.21.15.129:0/3637539763 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb240119b0 con 0x7fdb280a82d0 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 msgr2=0x7fdb280a86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7fdb24002f30 tx=0x7fdb1c007710).stop 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3637539763 shutdown_connections 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb2809d160 0x7fdb280a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.321+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3637539763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb280a82d0 0x7fdb280a86f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.335 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.325+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3637539763 shutdown_connections 2019-10-13T05:21:51.336 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.325+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3637539763 wait complete. 2019-10-13T05:21:51.336 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.325+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3637539763 >> 172.21.15.129:0/3637539763 conn(0x7fdb2805e870 msgr2=0x7fdb2809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.325+0000 7fdb2fa99700 1 Processor -- start 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.325+0000 7fdb2fa99700 1 -- start start 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 0x7fdb2807a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb28123590 con 0x7fdb280781d0 2019-10-13T05:21:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb28079570 con 0x7fdb280a46b0 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28079810 con 0x7fdb28124080 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e837700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdb280781d0 0x7fdb280785f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59476/0 (socket says 172.21.15.129:59476) 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e837700 1 -- 172.21.15.129:0/3380176263 learned_addr learned my addr 172.21.15.129:0/3380176263 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3380176263 >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 0x7fdb2807a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175293468 0 0) 0x7fdb28123590 con 0x7fdb280781d0 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb100030a0 con 0x7fdb280781d0 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 -- 172.21.15.129:0/3380176263 >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 msgr2=0x7fdb2807a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 0x7fdb2807a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 -- 172.21.15.129:0/3380176263 >> v1:172.21.15.129:6789/0 conn(0x7fdb280781d0 legacy=0x7fdb280785f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 -- 172.21.15.129:0/3380176263 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28124650 con 0x7fdb28124080 2019-10-13T05:21:51.342 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3380176263 >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 0x7fdb2807a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb1c005530 con 0x7fdb28124080 2019-10-13T05:21:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c007150 con 0x7fdb28124080 2019-10-13T05:21:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb280a3e60 con 0x7fdb28124080 2019-10-13T05:21:51.345 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb1c005b10 con 0x7fdb28124080 2019-10-13T05:21:51.345 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb28078f90 con 0x7fdb28124080 2019-10-13T05:21:51.345 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.345 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fdb1c010170 con 0x7fdb28124080 2019-10-13T05:21:51.346 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.329+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.346 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.333+0000 7fdb1affd700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 0x7fdb1002a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.346 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.333+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fdb1c034830 con 0x7fdb28124080 2019-10-13T05:21:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.333+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 0x7fdb1002a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.333+0000 7fdb2d835700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 0x7fdb1002a280 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.348 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.333+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb280555c0 con 0x7fdb28124080 2019-10-13T05:21:51.350 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.345+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.350 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.345+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fdb28124650 con 0x7fdb28124080 2019-10-13T05:21:51.505 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.497+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdb28067b90 con 0x7fdb28124080 2019-10-13T05:21:51.505 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.501+0000 7fdb2e036700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.506 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.501+0000 7fdb1affd700 1 -- 172.21.15.129:0/3380176263 <== mon.2 v2:172.21.15.71:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdb1c013060 con 0x7fdb28124080 2019-10-13T05:21:51.508 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 msgr2=0x7fdb1002a280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 0x7fdb1002a280 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 msgr2=0x7fdb28079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7fdb1c002f30 tx=0x7fdb240078b0).stop 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 shutdown_connections 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7fdb28124080 0x7fdb28079ab0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3380176263 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7fdb10027d80 0x7fdb1002a280 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 --2- 172.21.15.129:0/3380176263 >> v2:172.21.15.71:3300/0 conn(0x7fdb280a46b0 0x7fdb2807a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.511 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 shutdown_connections 2019-10-13T05:21:51.512 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 wait complete. 2019-10-13T05:21:51.512 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.505+0000 7fdb2fa99700 1 -- 172.21.15.129:0/3380176263 >> 172.21.15.129:0/3380176263 conn(0x7fdb2805e770 msgr2=0x7fdb2807de80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.521 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:51.521 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T05:21:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 Processor -- start 2019-10-13T05:21:51.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 -- start start 2019-10-13T05:21:51.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 0x7f97a80a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80a0fd0 con 0x7f97a80a82d0 2019-10-13T05:21:51.623 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a80a1300 con 0x7f97a809d160 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97ade92700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80a15a0 con 0x7f97a80a86f0 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97acc30700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97a7fff700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 0x7f97a80a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97acc30700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3301/0 says I am v2:172.21.15.129:44412/0 (socket says 172.21.15.129:44412) 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97acc30700 1 -- 172.21.15.129:0/3425924829 learned_addr learned my addr 172.21.15.129:0/3425924829 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:51.626 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.609+0000 7f97a77fe700 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f97a80a82d0 0x7f97a8144920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.129:54068/0 (socket says 172.21.15.129:54068) 2019-10-13T05:21:51.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 -- 172.21.15.129:0/3425924829 >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 msgr2=0x7f97a80a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:51.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 0x7f97a80a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 -- 172.21.15.129:0/3425924829 >> v1:172.21.15.71:6789/0 conn(0x7f97a80a82d0 legacy=0x7f97a8144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.627 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 -- 172.21.15.129:0/3425924829 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80a8b10 con 0x7f97a80a86f0 2019-10-13T05:21:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97a7fff700 1 --2- 172.21.15.129:0/3425924829 >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 0x7f97a80a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f97a0008780 tx=0x7f9798007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f97a0008780 tx=0x7f9798007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f97a0008780 tx=0x7f9798007e40).ready entity=mon.2 client_cookie=59a0205258d4f1e1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f97a0008780 tx=0x7f9798007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f97a0008780 tx=0x7f9798007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97a6ffd700 1 -- 172.21.15.129:0/3425924829 <== mon.2 v2:172.21.15.71:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97a0008c80 con 0x7f97a80a86f0 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97acc30700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f97a0008780 tx=0x7f9798007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97a6ffd700 1 -- 172.21.15.129:0/3425924829 <== mon.2 v2:172.21.15.71:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a00088a0 con 0x7f97a80a86f0 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97a6ffd700 1 -- 172.21.15.129:0/3425924829 <== mon.2 v2:172.21.15.71:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97a0011790 con 0x7f97a80a86f0 2019-10-13T05:21:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 -- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 msgr2=0x7f97a814eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f97a0008780 tx=0x7f9798007e40).stop 2019-10-13T05:21:51.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 -- 172.21.15.129:0/3425924829 shutdown_connections 2019-10-13T05:21:51.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 --2- 172.21.15.129:0/3425924829 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f97a80a86f0 0x7f97a814eb20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 --2- 172.21.15.129:0/3425924829 >> v2:172.21.15.71:3300/0 conn(0x7f97a809d160 0x7f97a80a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.630 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 -- 172.21.15.129:0/3425924829 shutdown_connections 2019-10-13T05:21:51.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 -- 172.21.15.129:0/3425924829 wait complete. 2019-10-13T05:21:51.631 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.613+0000 7f97ade92700 1 -- 172.21.15.129:0/3425924829 >> 172.21.15.129:0/3425924829 conn(0x7f97a805e870 msgr2=0x7f97a809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 Processor -- start 2019-10-13T05:21:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- start start 2019-10-13T05:21:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97a8124930 0x7f97a8119c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80a3e60 con 0x7f97a81238c0 2019-10-13T05:21:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a8125100 con 0x7f97a80a96a0 2019-10-13T05:21:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a8124e60 con 0x7f97a8124930 2019-10-13T05:21:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a77fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f97a80a96a0 0x7f97a8116400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59488/0 (socket says 172.21.15.129:59488) 2019-10-13T05:21:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a77fe700 1 -- 172.21.15.129:0/1581792816 learned_addr learned my addr 172.21.15.129:0/1581792816 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:51.634 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> v2:172.21.15.71:3300/0 conn(0x7f97a8124930 0x7f97a8119c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546611155 0 0) 0x7f97a8125100 con 0x7f97a80a96a0 2019-10-13T05:21:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97800030a0 con 0x7f97a80a96a0 2019-10-13T05:21:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 -- 172.21.15.129:0/1581792816 >> v1:172.21.15.129:6789/0 conn(0x7f97a80a96a0 legacy=0x7f97a8116400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882310163 0 0) 0x7f97a80a3e60 con 0x7f97a81238c0 2019-10-13T05:21:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 -- 172.21.15.129:0/1581792816 >> v1:172.21.15.71:6789/0 conn(0x7f97a81238c0 legacy=0x7f97a80a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 -- 172.21.15.129:0/1581792816 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80a0220 con 0x7f97a8124930 2019-10-13T05:21:51.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> v2:172.21.15.71:3300/0 conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> v2:172.21.15.71:3300/0 conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.639 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97a0007160 con 0x7f97a8124930 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a811a170 con 0x7f97a8124930 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a00113b0 con 0x7f97a8124930 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a811a640 con 0x7f97a8124930 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.617+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97a0005530 con 0x7f97a8124930 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f97a0007f10 con 0x7f97a8124930 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97a4ff9700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 0x7f978002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f97a00346b0 con 0x7f97a8124930 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97a7fff700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 0x7f978002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.621+0000 7f97a7fff700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 0x7f978002a240 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.625+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a80555c0 con 0x7f97a8124930 2019-10-13T05:21:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.633+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.633+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f97a001cd10 con 0x7f97a8124930 2019-10-13T05:21:51.793 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.785+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f97a8065350 con 0x7f97a8124930 2019-10-13T05:21:51.795 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.789+0000 7f97acc30700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.795 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.789+0000 7f97a4ff9700 1 -- 172.21.15.129:0/1581792816 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7f97a00079d0 con 0x7f97a8124930 2019-10-13T05:21:51.796 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:51.798 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 msgr2=0x7f978002a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.798 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 0x7f978002a240 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 msgr2=0x7f97a8119c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f97a0008360 tx=0x7f97a0008b80).stop 2019-10-13T05:21:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 shutdown_connections 2019-10-13T05:21:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f9780027d40 0x7f978002a240 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 --2- 172.21.15.129:0/1581792816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97a8124930 0x7f97a8119c20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.800 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 shutdown_connections 2019-10-13T05:21:51.800 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 wait complete. 2019-10-13T05:21:51.800 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.793+0000 7f97ade92700 1 -- 172.21.15.129:0/1581792816 >> 172.21.15.129:0/1581792816 conn(0x7f97a80a8650 msgr2=0x7f97a809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.808 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:51.808 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T05:21:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 Processor -- start 2019-10-13T05:21:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 -- start start 2019-10-13T05:21:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f04d80a86f0 0x7f04d814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d80a0fd0 con 0x7f04d809d160 2019-10-13T05:21:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d80a8b10 con 0x7f04d80a82d0 2019-10-13T05:21:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04e02d4700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d80a1300 con 0x7f04d80a86f0 2019-10-13T05:21:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04de070700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f04d80a82d0 0x7f04d8144920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59494/0 (socket says 172.21.15.129:59494) 2019-10-13T05:21:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04de070700 1 -- 172.21.15.129:0/2278561737 learned_addr learned my addr 172.21.15.129:0/2278561737 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.893+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> v2:172.21.15.71:3300/0 conn(0x7f04d80a86f0 0x7f04d814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449248113 0 0) 0x7f04d80a8b10 con 0x7f04d80a82d0 2019-10-13T05:21:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c40030a0 con 0x7f04d80a82d0 2019-10-13T05:21:51.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 28666372 0 0) 0x7f04c40030a0 con 0x7f04d80a82d0 2019-10-13T05:21:51.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 >> v2:172.21.15.71:3300/0 conn(0x7f04d80a86f0 msgr2=0x7f04d814fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:51.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 --2- 172.21.15.129:0/2278561737 >> v2:172.21.15.71:3300/0 conn(0x7f04d80a86f0 0x7f04d814fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.913 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 >> v1:172.21.15.71:6789/0 conn(0x7f04d809d160 legacy=0x7f04d80a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d8145120 con 0x7f04d80a82d0 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1138480238 0 0) 0x7f04c8001750 con 0x7f04d80a82d0 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 >> v1:172.21.15.129:6789/0 conn(0x7f04d80a82d0 legacy=0x7f04d8144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04c40036d0 0x7f04c4003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04c4004410 0x7f04c4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80a1300 con 0x7f04c40036d0 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c404fbf0 con 0x7f04c4004d80 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c404fe30 con 0x7f04c4004410 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04de070700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04c4004410 0x7f04c4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04de871700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04c40036d0 0x7f04c4003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 -- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04c4004410 msgr2=0x7f04c4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04c4004410 0x7f04c4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 -- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04c40036d0 msgr2=0x7f04c4003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04c40036d0 0x7f04c4003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 -- 172.21.15.129:0/2278561737 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4050130 con 0x7f04c4004d80 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f04d0008b00 tx=0x7f04c0001370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 -- 172.21.15.129:0/2278561737 reap_dead start 2019-10-13T05:21:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f04d0008b00 tx=0x7f04c0001370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f04d0008b00 tx=0x7f04c0001370).ready entity=mon.1 client_cookie=99caae3dabc28138 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f04d0008b00 tx=0x7f04c0001370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d0005aa0 con 0x7f04c4004d80 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f04d0008b00 tx=0x7f04c0001370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04df072700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f04d0008b00 tx=0x7f04c0001370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0008d70 con 0x7f04c4004d80 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04dd86f700 1 -- 172.21.15.129:0/2278561737 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d00092b0 con 0x7f04c4004d80 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 -- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 msgr2=0x7f04c400b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f04d0008b00 tx=0x7f04c0001370).stop 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 -- 172.21.15.129:0/2278561737 shutdown_connections 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 --2- 172.21.15.129:0/2278561737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c4004d80 0x7f04c400b660 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.918 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 -- 172.21.15.129:0/2278561737 shutdown_connections 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 -- 172.21.15.129:0/2278561737 wait complete. 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.897+0000 7f04e02d4700 1 -- 172.21.15.129:0/2278561737 >> 172.21.15.129:0/2278561737 conn(0x7f04d805e870 msgr2=0x7f04d809c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 Processor -- start 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- start start 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04d81228e0 0x7f04d8124450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 0x7f04d807a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.919 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d80a4870 con 0x7f04d807b1d0 2019-10-13T05:21:51.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d80a4dd0 con 0x7f04d81228e0 2019-10-13T05:21:51.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80a5070 con 0x7f04d807a1f0 2019-10-13T05:21:51.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de871700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f04d807b1d0 0x7f04d80a5310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59506/0 (socket says 172.21.15.129:59506) 2019-10-13T05:21:51.920 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04d81228e0 0x7f04d8124450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de871700 1 -- 172.21.15.129:0/1918835987 learned_addr learned my addr 172.21.15.129:0/1918835987 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04df072700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 0x7f04d807a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409404656 0 0) 0x7f04d80a4870 con 0x7f04d807b1d0 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc0030a0 con 0x7f04d807b1d0 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 -- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 msgr2=0x7f04d807a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 0x7f04d807a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:51.921 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 -- 172.21.15.129:0/1918835987 >> v1:172.21.15.129:6789/0 conn(0x7f04d807b1d0 legacy=0x7f04d80a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:51.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 -- 172.21.15.129:0/1918835987 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d813b480 con 0x7f04d81228e0 2019-10-13T05:21:51.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04df072700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 0x7f04d807a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T05:21:51.922 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> v2:172.21.15.129:3300/0 conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> v2:172.21.15.129:3300/0 conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c8012920 con 0x7f04d81228e0 2019-10-13T05:21:51.923 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d813a470 con 0x7f04d81228e0 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8012540 con 0x7f04d81228e0 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d813a9c0 con 0x7f04d81228e0 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.901+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c8008370 con 0x7f04d81228e0 2019-10-13T05:21:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f04c8008650 con 0x7f04d81228e0 2019-10-13T05:21:51.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04d77fe700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 0x7f04bc02a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:51.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f04c8034830 con 0x7f04d81228e0 2019-10-13T05:21:51.925 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04de871700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 0x7f04bc02a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.905+0000 7f04de871700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 0x7f04bc02a2a0 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.909+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d8122d00 con 0x7f04d81228e0 2019-10-13T05:21:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.913+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:51.913+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f04c8008940 con 0x7f04d81228e0 2019-10-13T05:21:52.069 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f04d80555c0 con 0x7f04d81228e0 2019-10-13T05:21:52.070 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04de070700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.070 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04d77fe700 1 -- 172.21.15.129:0/1918835987 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7f04c800b0f0 con 0x7f04d81228e0 2019-10-13T05:21:52.070 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 8 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 msgr2=0x7f04bc02a2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 0x7f04bc02a2a0 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 msgr2=0x7f04d8124450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f04d8134060 tx=0x7f04d0005e90).stop 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 shutdown_connections 2019-10-13T05:21:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f04bc027da0 0x7f04bc02a2a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f04d807a1f0 0x7f04d807a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 --2- 172.21.15.129:0/1918835987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04d81228e0 0x7f04d8124450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 shutdown_connections 2019-10-13T05:21:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 wait complete. 2019-10-13T05:21:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.065+0000 7f04e02d4700 1 -- 172.21.15.129:0/1918835987 >> 172.21.15.129:0/1918835987 conn(0x7f04d81446f0 msgr2=0x7f04d80a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:52.084 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2019-10-13T05:21:52.084 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.084 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T05:21:52.180 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 Processor -- start 2019-10-13T05:21:52.180 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 -- start start 2019-10-13T05:21:52.180 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 --2- >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f512c0a7520 0x7f512c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:52.180 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f512c144a10 0x7f512c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T05:21:52.180 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c0a7a30 con 0x7f512c0a1c80 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c0a96b0 con 0x7f512c144a10 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5132e13700 1 -- --> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0a9950 con 0x7f512c0a7520 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f51313b0700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f512c0a1c80 0x7f512c0a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59514/0 (socket says 172.21.15.129:59514) 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5131bb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f512c144a10 0x7f512c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f51313b0700 1 -- 172.21.15.129:0/4271551815 learned_addr learned my addr 172.21.15.129:0/4271551815 (peer_addr_for_me v1:172.21.15.129:0/0) 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5130baf700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f512c0a7520 0x7f512c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:52.181 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5131bb1700 1 -- 172.21.15.129:0/4271551815 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f512c0a7520 msgr2=0x7f512c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f512c0a7520 0x7f512c0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5131bb1700 1 -- 172.21.15.129:0/4271551815 >> v1:172.21.15.129:6789/0 conn(0x7f512c0a1c80 legacy=0x7f512c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5131bb1700 1 -- 172.21.15.129:0/4271551815 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c0aa3e0 con 0x7f512c144a10 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.169+0000 7f5123fff700 1 -- 172.21.15.129:0/4271551815 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210532857 0 0) 0x7f512c0a7a30 con 0x7f512c0a1c80 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> v2:172.21.15.129:3300/0 conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f512c0a9870 tx=0x7f511c002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> v2:172.21.15.129:3300/0 conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f512c0a9870 tx=0x7f511c002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.182 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7f512c0a9870 tx=0x7f511c002ff0).ready entity=mon.1 client_cookie=8219d37c52b1e6d6 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f512c0a9870 tx=0x7f511c002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f512c0a9870 tx=0x7f511c002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5123fff700 1 -- 172.21.15.129:0/4271551815 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5124010860 con 0x7f512c144a10 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5131bb1700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f512c0a9870 tx=0x7f511c002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5123fff700 1 -- 172.21.15.129:0/4271551815 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124008b40 con 0x7f512c144a10 2019-10-13T05:21:52.183 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 -- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 msgr2=0x7f512c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.184 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5123fff700 1 -- 172.21.15.129:0/4271551815 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51240110c0 con 0x7f512c144a10 2019-10-13T05:21:52.184 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7f512c0a9870 tx=0x7f511c002ff0).stop 2019-10-13T05:21:52.184 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 -- 172.21.15.129:0/4271551815 shutdown_connections 2019-10-13T05:21:52.185 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c144a10 0x7f512c144e30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.185 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 --2- 172.21.15.129:0/4271551815 >> [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] conn(0x7f512c0a7520 0x7f512c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T05:21:52.185 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 -- 172.21.15.129:0/4271551815 shutdown_connections 2019-10-13T05:21:52.185 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 -- 172.21.15.129:0/4271551815 wait complete. 2019-10-13T05:21:52.185 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 -- 172.21.15.129:0/4271551815 >> 172.21.15.129:0/4271551815 conn(0x7f512c05e870 msgr2=0x7f512c09c620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:52.188 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.173+0000 7f5132e13700 1 Processor -- start 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- start start 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f512c11f7f0 0x7f512c11fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f512c137f80 0x7f512c13a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c0a0680 con 0x7f512c0a8280 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c124770 con 0x7f512c137f80 2019-10-13T05:21:52.189 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c126d50 con 0x7f512c11f7f0 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f512c11f7f0 0x7f512c11fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f512c11f7f0 0x7f512c11fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41552/0 (socket says 172.21.15.129:41552) 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 -- 172.21.15.129:0/860302809 learned_addr learned my addr 172.21.15.129:0/860302809 (peer_addr_for_me v2:172.21.15.129:0/0) 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5131bb1700 1 --2- 172.21.15.129:0/860302809 >> v2:172.21.15.71:3300/0 conn(0x7f512c137f80 0x7f512c13a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 -- 172.21.15.129:0/860302809 >> v2:172.21.15.71:3300/0 conn(0x7f512c137f80 msgr2=0x7f512c13a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.190 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> v2:172.21.15.71:3300/0 conn(0x7f512c137f80 0x7f512c13a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 -- 172.21.15.129:0/860302809 >> v1:172.21.15.129:6789/0 conn(0x7f512c0a8280 legacy=0x7f512c123f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 -- 172.21.15.129:0/860302809 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c120160 con 0x7f512c11f7f0 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> v2:172.21.15.129:3300/0 conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> v2:172.21.15.129:3300/0 conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:52.191 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.192 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f511c005530 con 0x7f512c11f7f0 2019-10-13T05:21:52.192 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.193 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.193 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c008b20 con 0x7f512c11f7f0 2019-10-13T05:21:52.193 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c125110 con 0x7f512c11f7f0 2019-10-13T05:21:52.193 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f511c005b10 con 0x7f512c11f7f0 2019-10-13T05:21:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c13a900 con 0x7f512c11f7f0 2019-10-13T05:21:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.177+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f511c012430 con 0x7f512c11f7f0 2019-10-13T05:21:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f5121ffb700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 0x7f510c029ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T05:21:52.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f511c034660 con 0x7f512c11f7f0 2019-10-13T05:21:52.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f51313b0700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 0x7f510c029ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T05:21:52.195 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.181+0000 7f51313b0700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 0x7f510c029ea0 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T05:21:52.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.185+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c0555c0 con 0x7f512c11f7f0 2019-10-13T05:21:52.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.189+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.196 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.189+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f511c005de0 con 0x7f512c11f7f0 2019-10-13T05:21:52.347 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.341+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f512c124f00 con 0x7f512c11f7f0 2019-10-13T05:21:52.348 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.341+0000 7f5130baf700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T05:21:52.348 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.341+0000 7f5121ffb700 1 -- 172.21.15.129:0/860302809 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2105 (secure 0 0 0) 0x7f511c00ebd0 con 0x7f512c11f7f0 2019-10-13T05:21:52.349 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 7 2019-10-13T05:21:52.349 INFO:teuthology.orchestra.run.smithi129.stdout: 2019-10-13T05:21:52.349 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4348,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.129:6833/1374113332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1374113332},{"type":"v1","addr":"172.21.15.129:6833","nonce":1374113332}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T05:21:36.833107+0000","modified":"2019-10-13T05:21:40.901183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4326":{"gid":4326,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.71:6835/3596066442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3596066442},{"type":"v1","addr":"172.21.15.71:6835","nonce":3596066442}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T05:21:52.353 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 msgr2=0x7f510c029ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.353 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 0x7f510c029ea0 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.353 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 msgr2=0x7f512c11fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7f511c002ef0 tx=0x7f511c002f70).stop 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 shutdown_connections 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.71:6800/13713,v1:172.21.15.71:6801/13713] conn(0x7f510c0279a0 0x7f510c029ea0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 --2- 172.21.15.129:0/860302809 >> v2:172.21.15.71:3300/0 conn(0x7f512c137f80 0x7f512c13a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 --2- 172.21.15.129:0/860302809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f512c11f7f0 0x7f512c11fc10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 shutdown_connections 2019-10-13T05:21:52.354 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 wait complete. 2019-10-13T05:21:52.355 INFO:teuthology.orchestra.run.smithi129.stderr:2019-10-13T05:21:52.345+0000 7f5132e13700 1 -- 172.21.15.129:0/860302809 >> 172.21.15.129:0/860302809 conn(0x7f512c09c920 msgr2=0x7f512c1207b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T05:21:52.566 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4326}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4326': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4326, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3596066442, u'type': u'v2', u'addr': u'172.21.15.71:6834'}, {u'nonce': 3596066442, u'type': u'v1', u'addr': u'172.21.15.71:6835'}]}, u'addr': u'172.21.15.71:6835/3596066442'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T05:21:36.833107+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T05:21:40.901183+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T05:21:52.566 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T05:21:52.566 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:21:52.566 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T05:21:52.660 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2019-10-13T05:21:52.660 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "678bb30f-383e-4566-b2c7-8d4fe3e6c97c", 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 0, 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4326, 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 12.755561859, 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 6, 2019-10-13T05:21:52.661 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 21, 2019-10-13T05:21:52.662 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 21, 2019-10-13T05:21:52.662 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 13.422067362 2019-10-13T05:21:52.662 INFO:teuthology.orchestra.run.smithi071.stdout:} 2019-10-13T05:21:52.667 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "678bb30f-383e-4566-b2c7-8d4fe3e6c97c", "whoami": 0, "id": 4326, "want_state": "up:active", "state": "up:active", "rank_uptime": 12.755561859, "mdsmap_epoch": 6, "osdmap_epoch": 21, "osdmap_epoch_barrier": 21, "uptime": 13.422067362 } 2019-10-13T05:21:52.667 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2019-10-13T05:21:52.667 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2019-10-13T05:21:52.667 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi129.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2019-10-13T05:21:52.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.668 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.668 INFO:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:52.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.679 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.680 INFO:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T05:21:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T05:21:52.762 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-10-13T05:21:52.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.763 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.763 INFO:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2019-10-13T05:21:52.838 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2019-10-13T05:21:52.838 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.839 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:52.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.906 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.906 INFO:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T05:21:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T05:21:52.926 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-10-13T05:21:52.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.927 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.927 INFO:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2019-10-13T05:21:52.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi129.stderr:2019-10-13T05:21:52.961+0000 7ffb1ff8ef40 -1 init, newargv = 0x55d516a969a0 newargc=7 2019-10-13T05:21:52.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi129.stderr:ceph-fuse[15971]: starting ceph client 2019-10-13T05:21:52.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi129.stderr:ceph-fuse[15971]: starting fuse 2019-10-13T05:21:52.998 INFO:teuthology.orchestra.run.smithi129.stdout:49 2019-10-13T05:21:52.999 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2019-10-13T05:21:52.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:52.999 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:52.999 INFO:teuthology.orchestra.run.smithi129:> sudo python2 -c ' 2019-10-13T05:21:52.999 INFO:teuthology.orchestra.run.smithi129:> import glob 2019-10-13T05:21:52.999 INFO:teuthology.orchestra.run.smithi129:> import re 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> import os 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> import subprocess 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> def find_socket(client_name): 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> files = glob.glob(asok_path) 2019-10-13T05:21:53.000 INFO:teuthology.orchestra.run.smithi129:> 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> # Given a non-glob path, it better be there 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> if "*" not in asok_path: 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> assert(len(files) == 1) 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> return files[0] 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> for f in files: 2019-10-13T05:21:53.001 INFO:teuthology.orchestra.run.smithi129:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> if os.path.exists("/proc/{0}".format(pid)): 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> return f 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> print find_socket("client.0") 2019-10-13T05:21:53.002 INFO:teuthology.orchestra.run.smithi129:> ' 2019-10-13T05:21:53.102 INFO:teuthology.orchestra.run.smithi129.stdout:/var/run/ceph/ceph-client.0.15971.asok 2019-10-13T05:21:53.102 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.15971.asok 2019-10-13T05:21:53.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:53.103 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.103 INFO:teuthology.orchestra.run.smithi129:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.15971.asok status 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout: "metadata": { 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_sha1": "a8ee8ac95ca822967c9dffe2a705f1d5502812bd", 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev)", 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout: "entity_id": "0", 2019-10-13T05:21:53.291 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": "smithi129", 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "pid": "15971", 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "root": "/" 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_count": 0, 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_pinned_count": 0, 2019-10-13T05:21:53.292 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4657, 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "inst": { 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "name": { 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "client", 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "num": 4657 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2019-10-13T05:21:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 73160150 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2019-10-13T05:21:53.294 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 73160150 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "inst_str": "client.4657 172.21.15.129:0/73160150", 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "addr_str": "172.21.15.129:0/73160150", 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "inode_count": 1, 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_epoch": 6, 2019-10-13T05:21:53.295 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch": 21, 2019-10-13T05:21:53.296 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch_barrier": 0, 2019-10-13T05:21:53.296 INFO:teuthology.orchestra.run.smithi129.stdout: "blacklisted": false 2019-10-13T05:21:53.296 INFO:teuthology.orchestra.run.smithi129.stdout:} 2019-10-13T05:21:53.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:53.297 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.297 INFO:teuthology.orchestra.run.smithi129:> stat --file-system '--printf=%T 2019-10-13T05:21:53.298 INFO:teuthology.orchestra.run.smithi129:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.310 INFO:teuthology.orchestra.run.smithi129.stdout:fuseblk 2019-10-13T05:21:53.310 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:21:53.311 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.311 INFO:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.395 INFO:teuthology.run_tasks:Running task check-counter... 2019-10-13T05:21:53.407 INFO:teuthology.run_tasks:Running task workunit... 2019-10-13T05:21:53.427 INFO:tasks.workunit:Pulling workunits from ref a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:21:53.427 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-10-13T05:21:53.427 INFO:tasks.workunit:timeout=3h 2019-10-13T05:21:53.427 INFO:tasks.workunit:cleanup=True 2019-10-13T05:21:53.428 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.428 INFO:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.457 INFO:teuthology.orchestra.run.smithi129.stdout: File: /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.457 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2019-10-13T05:21:53.457 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 31h/49d Inode: 1 Links: 2 2019-10-13T05:21:53.457 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T05:21:53.457 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2019-10-13T05:21:53.458 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2019-10-13 05:21:39.903827962 +0000 2019-10-13T05:21:53.458 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2019-10-13 05:21:53.387725989 +0000 2019-10-13T05:21:53.458 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2019-10-13T05:21:53.458 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2019-10-13T05:21:53.458 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.459 INFO:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2019-10-13T05:21:53.556 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:21:53.556 INFO:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T05:21:53.686 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-10-13T05:22:07.418 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:22:07.418 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:07.422 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:22:07.422 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:37.461 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:22:37.461 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:37.465 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:22:37.465 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:Note: checking out 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'. 2019-10-13T05:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-10-13T05:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2019-10-13T05:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by performing another checkout. 2019-10-13T05:22:41.295 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:41.295 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2019-10-13T05:22:41.295 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-10-13T05:22:41.295 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:41.295 INFO:tasks.workunit.client.0.smithi129.stderr: git checkout -b 2019-10-13T05:22:41.296 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:41.296 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at a8ee8ac95c Merge PR #29626 into wip-pdonnell-testing-20191012.025926 2019-10-13T05:22:41.299 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:22:41.299 INFO:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-10-13T05:22:41.306 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-10-13T05:22:41.308 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T05:22:41.308 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-10-13T05:22:42.042 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-10-13T05:22:42.145 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-10-13T05:22:42.199 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T05:22:42.201 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T05:22:42.201 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-10-13T05:22:42.256 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T05:22:42.333 DEBUG:teuthology.orchestra.remote:smithi129:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-10-13T05:22:42.345 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2019-10-13T05:22:42.346 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2019-10-13T05:22:42.346 INFO:teuthology.orchestra.run.smithi129:Running (workunit test suites/ffsb.sh): 2019-10-13T05:22:42.346 INFO:teuthology.orchestra.run.smithi129:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2019-10-13T05:22:42.399 INFO:tasks.workunit.client.0.smithi129.stderr:--2019-10-13 05:22:42-- http://download.ceph.com/qa/ffsb.tar.bz2 2019-10-13T05:22:42.426 INFO:tasks.workunit.client.0.smithi129.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2019-10-13T05:22:42.447 INFO:tasks.workunit.client.0.smithi129.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2019-10-13T05:22:42.467 INFO:tasks.workunit.client.0.smithi129.stderr:HTTP request sent, awaiting response... 200 OK 2019-10-13T05:22:42.467 INFO:tasks.workunit.client.0.smithi129.stderr:Length: 128931 (126K) [application/octet-stream] 2019-10-13T05:22:42.472 INFO:tasks.workunit.client.0.smithi129.stderr:Saving to: 'ffsb.tar.bz2' 2019-10-13T05:22:42.477 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:42.509 INFO:tasks.workunit.client.0.smithi129.stderr: 0K .......... .......... .......... .......... .......... 39% 1.35M 0s 2019-10-13T05:22:42.528 INFO:tasks.workunit.client.0.smithi129.stderr: 50K .......... .......... .......... .......... .......... 79% 2.53M 0s 2019-10-13T05:22:42.530 INFO:tasks.workunit.client.0.smithi129.stderr: 100K .......... .......... ..... 100% 11.1M=0.06s 2019-10-13T05:22:42.531 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:42.531 INFO:tasks.workunit.client.0.smithi129.stderr:2019-10-13 05:22:42 (2.13 MB/s) - 'ffsb.tar.bz2' saved [128931/128931] 2019-10-13T05:22:42.531 INFO:tasks.workunit.client.0.smithi129.stderr: 2019-10-13T05:22:42.565 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ 2019-10-13T05:22:42.568 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_fs.c 2019-10-13T05:22:42.574 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/INSTALL 2019-10-13T05:22:42.580 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/fileops.c 2019-10-13T05:22:42.584 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/missing 2019-10-13T05:22:42.589 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/config.h.in 2019-10-13T05:22:42.591 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/config.guess 2019-10-13T05:22:42.596 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/fh.h 2019-10-13T05:22:42.600 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/COPYING 2019-10-13T05:22:42.603 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/util.h 2019-10-13T05:22:42.606 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/main.c 2019-10-13T05:22:42.609 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ChangeLog 2019-10-13T05:22:42.612 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rand.c 2019-10-13T05:22:42.615 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_fs.h 2019-10-13T05:22:42.618 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/install-sh 2019-10-13T05:22:42.621 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rwlock.h 2019-10-13T05:22:42.624 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/parser.h 2019-10-13T05:22:42.627 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/filelist.h 2019-10-13T05:22:42.630 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/LICENSE 2019-10-13T05:22:42.635 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/Makefile.in 2019-10-13T05:22:42.639 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/filelist.c 2019-10-13T05:22:42.643 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/configure 2019-10-13T05:22:42.654 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/README 2019-10-13T05:22:42.658 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/cirlist.c 2019-10-13T05:22:42.662 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/metaops.c 2019-10-13T05:22:42.665 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/registration-description 2019-10-13T05:22:42.668 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/list.c 2019-10-13T05:22:42.671 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/list.h 2019-10-13T05:22:42.674 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rand.h 2019-10-13T05:22:42.676 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/parser.c 2019-10-13T05:22:42.679 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_tg.c 2019-10-13T05:22:42.684 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/cirlist.h 2019-10-13T05:22:42.685 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/AUTHORS 2019-10-13T05:22:42.687 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/aclocal.m4 2019-10-13T05:22:42.692 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/configure.in 2019-10-13T05:22:42.695 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/examples/ 2019-10-13T05:22:42.697 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/examples/profile_everything 2019-10-13T05:22:42.702 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/depcomp 2019-10-13T05:22:42.707 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_stats.c 2019-10-13T05:22:42.710 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/util.c 2019-10-13T05:22:42.714 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/Makefile.am 2019-10-13T05:22:42.717 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_thread.c 2019-10-13T05:22:42.720 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rbt.c 2019-10-13T05:22:42.725 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rbt.h 2019-10-13T05:22:42.729 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_tg.h 2019-10-13T05:22:42.732 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/stamp-h.in 2019-10-13T05:22:42.735 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/config.sub 2019-10-13T05:22:42.739 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_thread.h 2019-10-13T05:22:42.742 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/fileops.h 2019-10-13T05:22:42.745 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_op.h 2019-10-13T05:22:42.748 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/public-description 2019-10-13T05:22:42.751 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/mkinstalldirs 2019-10-13T05:22:42.754 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb.h 2019-10-13T05:22:42.757 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/metaops.h 2019-10-13T05:22:42.760 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_op.c 2019-10-13T05:22:42.763 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/fh.c 2019-10-13T05:22:42.766 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/NEWS 2019-10-13T05:22:42.769 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/rwlock.c 2019-10-13T05:22:42.772 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_stats.h 2019-10-13T05:22:42.775 INFO:tasks.workunit.client.0.smithi129.stdout:ffsb-6.0-rc2/ffsb_fc.c 2019-10-13T05:22:42.794 INFO:tasks.workunit.client.0.smithi129.stdout:patching file parser.c 2019-10-13T05:22:43.187 INFO:tasks.workunit.client.0.smithi129.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2019-10-13T05:22:44.207 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether build environment is sane... yes 2019-10-13T05:22:44.247 INFO:tasks.workunit.client.0.smithi129.stdout:checking for a thread-safe mkdir -p... /bin/mkdir -p 2019-10-13T05:22:44.249 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gawk... gawk 2019-10-13T05:22:44.292 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether make sets $(MAKE)... yes 2019-10-13T05:22:44.406 INFO:tasks.workunit.client.0.smithi129.stdout:checking build system type... x86_64-unknown-linux-gnu 2019-10-13T05:22:44.407 INFO:tasks.workunit.client.0.smithi129.stdout:checking host system type... x86_64-unknown-linux-gnu 2019-10-13T05:22:44.446 INFO:tasks.workunit.client.0.smithi129.stdout:checking for style of include used by make... GNU 2019-10-13T05:22:44.456 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gcc... gcc 2019-10-13T05:22:44.611 INFO:tasks.workunit.client.0.smithi129.stdout:checking for C compiler default output file name... a.out 2019-10-13T05:22:44.618 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether the C compiler works... yes 2019-10-13T05:22:44.626 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether we are cross compiling... no 2019-10-13T05:22:44.734 INFO:tasks.workunit.client.0.smithi129.stdout:checking for suffix of executables... 2019-10-13T05:22:44.822 INFO:tasks.workunit.client.0.smithi129.stdout:checking for suffix of object files... o 2019-10-13T05:22:44.911 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether we are using the GNU C compiler... yes 2019-10-13T05:22:45.022 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether gcc accepts -g... yes 2019-10-13T05:22:45.155 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gcc option to accept ISO C89... none needed 2019-10-13T05:22:45.480 INFO:tasks.workunit.client.0.smithi129.stdout:checking dependency style of gcc... gcc3 2019-10-13T05:22:45.613 INFO:tasks.workunit.client.0.smithi129.stdout:checking how to run the C preprocessor... gcc -E 2019-10-13T05:22:45.737 INFO:tasks.workunit.client.0.smithi129.stdout:checking for grep that handles long lines and -e... /bin/grep 2019-10-13T05:22:45.740 INFO:tasks.workunit.client.0.smithi129.stdout:checking for egrep... /bin/grep -E 2019-10-13T05:22:45.762 INFO:tasks.workunit.client.0.smithi129.stdout:checking for AIX... no 2019-10-13T05:22:45.799 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gcc... (cached) gcc 2019-10-13T05:22:45.827 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether we are using the GNU C compiler... (cached) yes 2019-10-13T05:22:45.830 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether gcc accepts -g... (cached) yes 2019-10-13T05:22:45.831 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2019-10-13T05:22:45.833 INFO:tasks.workunit.client.0.smithi129.stdout:checking dependency style of gcc... (cached) gcc3 2019-10-13T05:22:45.968 INFO:tasks.workunit.client.0.smithi129.stdout:checking for main in -lm... yes 2019-10-13T05:22:46.094 INFO:tasks.workunit.client.0.smithi129.stdout:checking for main in -lpthread... yes 2019-10-13T05:22:46.449 INFO:tasks.workunit.client.0.smithi129.stdout:checking for ANSI C header files... yes 2019-10-13T05:22:46.556 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2019-10-13T05:22:46.680 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/types.h... yes 2019-10-13T05:22:46.817 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/stat.h... yes 2019-10-13T05:22:46.943 INFO:tasks.workunit.client.0.smithi129.stdout:checking for stdlib.h... yes 2019-10-13T05:22:47.054 INFO:tasks.workunit.client.0.smithi129.stdout:checking for string.h... yes 2019-10-13T05:22:47.193 INFO:tasks.workunit.client.0.smithi129.stdout:checking for memory.h... yes 2019-10-13T05:22:47.319 INFO:tasks.workunit.client.0.smithi129.stdout:checking for strings.h... yes 2019-10-13T05:22:47.435 INFO:tasks.workunit.client.0.smithi129.stdout:checking for inttypes.h... yes 2019-10-13T05:22:47.568 INFO:tasks.workunit.client.0.smithi129.stdout:checking for stdint.h... yes 2019-10-13T05:22:47.697 INFO:tasks.workunit.client.0.smithi129.stdout:checking for unistd.h... yes 2019-10-13T05:22:47.842 INFO:tasks.workunit.client.0.smithi129.stdout:checking pthread.h usability... yes 2019-10-13T05:22:47.906 INFO:tasks.workunit.client.0.smithi129.stdout:checking pthread.h presence... yes 2019-10-13T05:22:47.907 INFO:tasks.workunit.client.0.smithi129.stdout:checking for pthread.h... yes 2019-10-13T05:22:48.039 INFO:tasks.workunit.client.0.smithi129.stdout:checking fcntl.h usability... yes 2019-10-13T05:22:48.091 INFO:tasks.workunit.client.0.smithi129.stdout:checking fcntl.h presence... yes 2019-10-13T05:22:48.093 INFO:tasks.workunit.client.0.smithi129.stdout:checking for fcntl.h... yes 2019-10-13T05:22:48.231 INFO:tasks.workunit.client.0.smithi129.stdout:checking limits.h usability... yes 2019-10-13T05:22:48.293 INFO:tasks.workunit.client.0.smithi129.stdout:checking limits.h presence... yes 2019-10-13T05:22:48.296 INFO:tasks.workunit.client.0.smithi129.stdout:checking for limits.h... yes 2019-10-13T05:22:48.335 INFO:tasks.workunit.client.0.smithi129.stdout:checking for stdint.h... (cached) yes 2019-10-13T05:22:48.458 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/time.h usability... yes 2019-10-13T05:22:48.511 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/time.h presence... yes 2019-10-13T05:22:48.513 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/time.h... yes 2019-10-13T05:22:48.553 INFO:tasks.workunit.client.0.smithi129.stdout:checking for unistd.h... (cached) yes 2019-10-13T05:22:48.659 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/vfs.h usability... yes 2019-10-13T05:22:48.721 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/vfs.h presence... yes 2019-10-13T05:22:48.723 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/vfs.h... yes 2019-10-13T05:22:48.862 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/limits.h usability... no 2019-10-13T05:22:48.931 INFO:tasks.workunit.client.0.smithi129.stdout:checking sys/limits.h presence... no 2019-10-13T05:22:48.933 INFO:tasks.workunit.client.0.smithi129.stdout:checking for sys/limits.h... no 2019-10-13T05:22:49.012 INFO:tasks.workunit.client.0.smithi129.stdout:checking for an ANSI C-conforming const... yes 2019-10-13T05:22:49.099 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether time.h and sys/time.h may both be included... yes 2019-10-13T05:22:49.105 INFO:tasks.workunit.client.0.smithi129.stdout:checking for function prototypes... yes 2019-10-13T05:22:49.226 INFO:tasks.workunit.client.0.smithi129.stdout:checking whether setvbuf arguments are reversed... no 2019-10-13T05:22:49.381 INFO:tasks.workunit.client.0.smithi129.stdout:checking for system... yes 2019-10-13T05:22:49.553 INFO:tasks.workunit.client.0.smithi129.stdout:checking for gettimeofday... yes 2019-10-13T05:22:49.743 INFO:tasks.workunit.client.0.smithi129.stdout:checking for mkdir... yes 2019-10-13T05:22:49.912 INFO:tasks.workunit.client.0.smithi129.stdout:checking for strerror... yes 2019-10-13T05:22:50.056 INFO:tasks.workunit.client.0.smithi129.stdout:checking for open64... yes 2019-10-13T05:22:50.224 INFO:tasks.workunit.client.0.smithi129.stdout:checking for stat64... yes 2019-10-13T05:22:50.394 INFO:tasks.workunit.client.0.smithi129.stdout:checking for fseeko64... yes 2019-10-13T05:22:50.569 INFO:tasks.workunit.client.0.smithi129.stdout:checking for lrand48_r... yes 2019-10-13T05:22:50.750 INFO:tasks.workunit.client.0.smithi129.stdout:checking for srand48_r... yes 2019-10-13T05:22:50.837 INFO:tasks.workunit.client.0.smithi129.stdout:configure: creating ./config.status 2019-10-13T05:22:51.148 INFO:tasks.workunit.client.0.smithi129.stdout:config.status: creating Makefile 2019-10-13T05:22:51.192 INFO:tasks.workunit.client.0.smithi129.stdout:config.status: creating config.h 2019-10-13T05:22:51.244 INFO:tasks.workunit.client.0.smithi129.stdout:config.status: executing depfiles commands 2019-10-13T05:22:51.394 INFO:tasks.workunit.client.0.smithi129.stdout:config.status: executing default commands 2019-10-13T05:22:51.603 INFO:tasks.workunit.client.0.smithi129.stdout:make all-am 2019-10-13T05:22:51.661 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T05:22:51.661 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2019-10-13T05:22:51.774 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c: In function 'ffsb_readfile': 2019-10-13T05:22:51.774 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:138:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr: printf("Error: read size %llu bytes too big " 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:138:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 5 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr: printf("Error: read size %llu bytes too big " 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:51.775 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:140:29: note: format string is defined here 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr: " for file of size %llu bytes\n" 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:144:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr: printf("minimum file size must be at least " 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:51.776 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:145:12: note: format string is defined here 2019-10-13T05:22:51.777 INFO:tasks.workunit.client.0.smithi129.stderr: " %llu bytes\n", minfilesize); 2019-10-13T05:22:51.777 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:51.777 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:51.777 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c: In function 'ffsb_stat': 2019-10-13T05:22:51.778 INFO:tasks.workunit.client.0.smithi129.stderr:fileops.c:542:2: warning: implicit declaration of function 'fhstat'; did you mean 'fstat'? [-Wimplicit-function-declaration] 2019-10-13T05:22:51.778 INFO:tasks.workunit.client.0.smithi129.stderr: fhstat(curfile->name, ft, fs); 2019-10-13T05:22:51.778 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~ 2019-10-13T05:22:51.778 INFO:tasks.workunit.client.0.smithi129.stderr: fstat 2019-10-13T05:22:51.916 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2019-10-13T05:22:51.929 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2019-10-13T05:22:52.067 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2019-10-13T05:22:52.079 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2019-10-13T05:22:52.274 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/main.Tpo .deps/main.Po 2019-10-13T05:22:52.289 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c: In function 'fhwrite': 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c:180:18: warning: format '%d' expects argument of type 'int', but argument 2 has type 'ssize_t {aka long int}' [-Wformat=] 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr: printf("Wrote %d instead of %d bytes.\n" 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr: ~^ 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr: %ld 2019-10-13T05:22:52.393 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c: In function 'fhseek': 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c:212:41: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr: fprintf(stderr, "tried to seek to %lld\n", offset); 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr: %ld 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c:214:20: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr: fprintf(stderr, "tried to seek from current " 2019-10-13T05:22:52.394 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:52.395 INFO:tasks.workunit.client.0.smithi129.stderr:fh.c:215:21: note: format string is defined here 2019-10-13T05:22:52.395 INFO:tasks.workunit.client.0.smithi129.stderr: "position to %lld\n", offset); 2019-10-13T05:22:52.395 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:52.395 INFO:tasks.workunit.client.0.smithi129.stderr: %ld 2019-10-13T05:22:52.493 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2019-10-13T05:22:52.509 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2019-10-13T05:22:52.719 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2019-10-13T05:22:52.735 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2019-10-13T05:22:52.885 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2019-10-13T05:22:52.898 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2019-10-13T05:22:53.007 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2019-10-13T05:22:53.022 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2019-10-13T05:22:53.153 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2019-10-13T05:22:53.171 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2019-10-13T05:22:53.405 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2019-10-13T05:22:53.418 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2019-10-13T05:22:53.501 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_tg.c: In function 'tg_print_config_helper': 2019-10-13T05:22:53.501 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_tg.c:274:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.501 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t read_size = %llu\t(%s)\n", tg->read_size, 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ ~~~~~~~~~~~~~ 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_tg.c:283:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t write_size = %llu\t(%s)\n", tg->write_size, 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ ~~~~~~~~~~~~~~ 2019-10-13T05:22:53.502 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:53.625 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2019-10-13T05:22:53.638 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2019-10-13T05:22:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c: In function 'verify_file': 2019-10-13T05:22:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:217:19: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.734 INFO:tasks.workunit.client.0.smithi129.stderr: printf("size %llu bytes for file %s is invalid\n", 2019-10-13T05:22:53.734 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:53.734 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c: In function 'fs_print_config': 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:595:22: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~~~^ 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr: %12lu 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr: fs->size_weights[i].size, 2019-10-13T05:22:53.736 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:595:43: warning: unknown conversion type character ')' in format [-Wformat=] 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:603:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t min file size = %llu\t(%s)\n", fs->minfilesize, 2019-10-13T05:22:53.737 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:605:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t max file size = %llu\t(%s)\n", fs->maxfilesize, 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:616:41: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2019-10-13T05:22:53.740 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t current utilization = %.2f\%\n", getfsutil(fs->basedir)*100); 2019-10-13T05:22:53.740 INFO:tasks.workunit.client.0.smithi129.stderr: ^~ 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:618:42: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr: printf("\t desired utilization = %.2lf%\n", fs->desired_fsutil * 100); 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr: ^~ 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c: In function 'fs_needs_stats': 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_fs.c:627:24: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:53.741 INFO:tasks.workunit.client.0.smithi129.stderr: return (fs != NULL) ? (int)fs->fsd.config : 0; 2019-10-13T05:22:53.742 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:53.886 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2019-10-13T05:22:53.903 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2019-10-13T05:22:54.072 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2019-10-13T05:22:54.087 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2019-10-13T05:22:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c: In function 'print_op_results': 2019-10-13T05:22:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c:83:23: warning: embedded '\0' in format [-Wformat-contains-nul] 2019-10-13T05:22:54.192 INFO:tasks.workunit.client.0.smithi129.stderr: sprintf(buf, "%s/sec\0", buf); 2019-10-13T05:22:54.192 INFO:tasks.workunit.client.0.smithi129.stderr: ^~ 2019-10-13T05:22:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c:86:19: warning: embedded '\0' in format [-Wformat-contains-nul] 2019-10-13T05:22:54.192 INFO:tasks.workunit.client.0.smithi129.stderr: sprintf(buf, "NA\0"); 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr: ^~ 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c: In function 'print_results': 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c:112:61: warning: unknown conversion type character 'T' in format [-Wformat=] 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:ffsb_op.c:112:74: warning: unknown conversion type character 'O' in format [-Wformat=] 2019-10-13T05:22:54.193 INFO:tasks.workunit.client.0.smithi129.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2019-10-13T05:22:54.194 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.267 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2019-10-13T05:22:54.291 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:util.c: In function 'ffsb_bench_gettimeofday': 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:util.c:333:46: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr: printf("average time for gettimeofday(): %llu nsec\n", average); 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:util.c: In function 'ffsb_bench_getpid': 2019-10-13T05:22:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:util.c:350:40: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T05:22:54.389 INFO:tasks.workunit.client.0.smithi129.stderr: printf("average time for getpid(): %llu nsec\n", average); 2019-10-13T05:22:54.389 INFO:tasks.workunit.client.0.smithi129.stderr: ~~~^ 2019-10-13T05:22:54.389 INFO:tasks.workunit.client.0.smithi129.stderr: %lu 2019-10-13T05:22:54.498 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/util.Tpo .deps/util.Po 2019-10-13T05:22:54.514 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2019-10-13T05:22:54.613 INFO:tasks.workunit.client.0.smithi129.stderr:In file included from parser.h:22:0, 2019-10-13T05:22:54.613 INFO:tasks.workunit.client.0.smithi129.stderr: from parser.c:25: 2019-10-13T05:22:54.613 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c: In function 'init_filesys': 2019-10-13T05:22:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.615 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.615 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.616 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.616 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T05:22:54.623 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.624 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T05:22:54.624 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.625 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.625 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.625 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T05:22:54.626 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.627 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.627 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T05:22:54.627 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.633 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.633 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.634 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.634 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.634 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.634 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.634 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.635 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.635 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.635 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.635 INFO:tasks.workunit.client.0.smithi129.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.637 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.638 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.639 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.639 INFO:tasks.workunit.client.0.smithi129.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c: In function 'init_tg_stats': 2019-10-13T05:22:54.640 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.641 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, 2019-10-13T05:22:54.642 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.643 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, 2019-10-13T05:22:54.644 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.645 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.646 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr: ^ 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T05:22:54.647 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~ 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr: container_of(ptr, type, member) 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~ 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T05:22:54.648 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~ 2019-10-13T05:22:54.649 INFO:tasks.workunit.client.0.smithi129.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T05:22:54.649 INFO:tasks.workunit.client.0.smithi129.stderr: list_for_each_entry(tmp_list, 2019-10-13T05:22:54.649 INFO:tasks.workunit.client.0.smithi129.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T05:22:54.894 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2019-10-13T05:22:54.909 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2019-10-13T05:22:55.054 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2019-10-13T05:22:55.070 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2019-10-13T05:22:55.231 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2019-10-13T05:22:55.247 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2019-10-13T05:22:55.335 INFO:tasks.workunit.client.0.smithi129.stdout:mv -f .deps/list.Tpo .deps/list.Po 2019-10-13T05:22:55.359 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2019-10-13T05:22:55.618 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T05:22:55.664 INFO:tasks.workunit.client.0.smithi129.stdout:FFSB version 6.0-RC2 started 2019-10-13T05:22:55.664 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.672 INFO:tasks.workunit.client.0.smithi129.stdout:benchmark time = 300 2019-10-13T05:22:55.672 INFO:tasks.workunit.client.0.smithi129.stdout:ThreadGroup 0 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout:================ 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: num_threads = 32 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: read_random = off 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: read_size = 0 (0B) 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: read_blocksize = 0 (0B) 2019-10-13T05:22:55.673 INFO:tasks.workunit.client.0.smithi129.stdout: read_skip = off 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: read_skipsize = 0 (0B) 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: write_random = on 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: write_size = 5242880 (5MB) 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: fsync_file = 0 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: write_blocksize = 4096 (4KB) 2019-10-13T05:22:55.674 INFO:tasks.workunit.client.0.smithi129.stdout: wait time = 0 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: op weights 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: read = 0 (0.00%) 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: readall = 0 (0.00%) 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: write = 1 (100.00%) 2019-10-13T05:22:55.675 INFO:tasks.workunit.client.0.smithi129.stdout: create = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: append = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: delete = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: metaop = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: createdir = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: stat = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: writeall = 0 (0.00%) 2019-10-13T05:22:55.676 INFO:tasks.workunit.client.0.smithi129.stdout: writeall_fsync = 0 (0.00%) 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout: open_close = 0 (0.00%) 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout: write_fsync = 0 (0.00%) 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout: create_fsync = 0 (0.00%) 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout: append_fsync = 0 (0.00%) 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout:FileSystem . 2019-10-13T05:22:55.677 INFO:tasks.workunit.client.0.smithi129.stdout:========== 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: num_dirs = 0 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: starting files = 128 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: min file size = 104857600 (100MB) 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: max file size = 104857600 (100MB) 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: directio = off 2019-10-13T05:22:55.678 INFO:tasks.workunit.client.0.smithi129.stdout: alignedio = on 2019-10-13T05:22:55.679 INFO:tasks.workunit.client.0.smithi129.stdout: bufferedio = off 2019-10-13T05:22:55.679 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.679 INFO:tasks.workunit.client.0.smithi129.stdout: aging is off 2019-10-13T05:22:55.679 INFO:tasks.workunit.client.0.smithi129.stdout: current utilization = 0.00% 2019-10-13T05:22:55.680 INFO:tasks.workunit.client.0.smithi129.stdout: 2019-10-13T05:22:55.680 INFO:tasks.workunit.client.0.smithi129.stdout:checking existing fs: . 2019-10-13T05:22:55.680 INFO:tasks.workunit.client.0.smithi129.stdout:recreating new fileset 2019-10-13T05:22:55.680 INFO:tasks.workunit.client.0.smithi129.stderr:opendir: No such file or directory 2019-10-13T05:23:07.565 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:23:07.566 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:07.569 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:23:07.569 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:37.657 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:23:37.657 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:37.661 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:23:37.661 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:07.762 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:24:07.762 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:07.766 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:24:07.766 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:37.882 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:24:37.882 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:37.885 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:24:37.886 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:08.022 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:25:08.022 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:08.026 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:25:08.026 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:38.142 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:25:38.142 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:38.145 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:25:38.146 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:09.163 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:26:09.164 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:09.167 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:26:09.167 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:40.736 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:26:40.736 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:40.740 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:26:40.741 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:10.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:27:10.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:10.851 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:27:10.851 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:40.942 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:27:40.942 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:40.946 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:27:40.946 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:11.062 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:28:11.062 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:11.066 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:28:11.067 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:41.186 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:28:41.186 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:41.190 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:28:41.190 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:11.519 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:29:11.519 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:11.523 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:29:11.523 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:41.557 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:29:41.557 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:41.561 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:29:41.562 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:11.679 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:30:11.680 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:11.683 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:30:11.683 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:41.828 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:30:41.828 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:41.832 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:30:41.832 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:11.939 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:31:11.940 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:11.943 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:31:11.944 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:42.093 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:31:42.093 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:42.098 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:31:42.098 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:12.212 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:32:12.213 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:12.217 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:32:12.217 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:42.275 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:32:42.275 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:42.279 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:32:42.291 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:12.396 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:33:12.396 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:12.400 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:33:12.400 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:42.441 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:33:42.441 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:42.445 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:33:42.445 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:12.477 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:34:12.478 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:12.482 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:34:12.482 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:43.769 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:34:43.769 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:43.773 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:34:43.773 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:48.005 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:34:48.000+0000 7f0e67c02700 -1 log_channel(cluster) log [ERR] : Health check failed: mon c is very low on available space (MON_DISK_CRIT) 2019-10-13T05:34:53.143 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:34:53.136+0000 7f0e67c02700 -1 log_channel(cluster) log [ERR] : Health check update: mons b,c are very low on available space (MON_DISK_CRIT) 2019-10-13T05:35:14.684 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:35:14.684 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:14.688 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:35:14.688 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:33.802 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:35:33.795+0000 7fbf6e985700 -1 bdev(0x563d211ba000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.804 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:35:33.795+0000 7f28e73a8700 -1 bdev(0x55a19fafc000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.806 INFO:tasks.ceph.osd.4.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbf6e985700 time 2019-10-13T05:35:33.798740+0000 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x563d15ef002c] 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.807 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:35:33.799+0000 7fbf6e985700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbf6e985700 time 2019-10-13T05:35:33.798740+0000 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x563d15ef002c] 2019-10-13T05:35:33.808 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:33.809 INFO:tasks.ceph.osd.4.smithi071.stderr: in thread 7fbf6e985700 thread_name:bstore_aio 2019-10-13T05:35:33.810 INFO:tasks.ceph.osd.6.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f28e73a8700 time 2019-10-13T05:35:33.801741+0000 2019-10-13T05:35:33.810 INFO:tasks.ceph.osd.6.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a19436002c] 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f28e73a8700 time 2019-10-13T05:35:33.801741+0000 2019-10-13T05:35:33.811 INFO:tasks.ceph.osd.6.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a19436002c] 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.812 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.6.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.6.smithi071.stderr: in thread 7f28e73a8700 thread_name:bstore_aio 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (()+0x12890) [0x7fbf78e9f890] 2019-10-13T05:35:33.813 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (gsignal()+0xc7) [0x7fbf77b51e97] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (abort()+0x141) [0x7fbf77b53801] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x563d15ef00fd] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 7: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: 8: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr:2019-10-13T05:35:33.803+0000 7fbf6e985700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.814 INFO:tasks.ceph.osd.4.smithi071.stderr: in thread 7fbf6e985700 thread_name:bstore_aio 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (()+0x12890) [0x7fbf78e9f890] 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (gsignal()+0xc7) [0x7fbf77b51e97] 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (abort()+0x141) [0x7fbf77b53801] 2019-10-13T05:35:33.815 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x563d15ef00fd] 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: 7: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: 8: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.816 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.817 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.817 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (()+0x12890) [0x7f28f18c2890] 2019-10-13T05:35:33.817 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f28f0574e97] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (abort()+0x141) [0x7f28f0576801] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1943600fd] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 7: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr: 8: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.818 INFO:tasks.ceph.osd.6.smithi071.stderr:2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: in thread 7f28e73a8700 thread_name:bstore_aio 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (()+0x12890) [0x7f28f18c2890] 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f28f0574e97] 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (abort()+0x141) [0x7f28f0576801] 2019-10-13T05:35:33.819 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1943600fd] 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: 7: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: 8: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.820 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.826 INFO:tasks.ceph.osd.4.smithi071.stderr: -6> 2019-10-13T05:35:33.795+0000 7fbf6e985700 -1 bdev(0x563d211ba000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.827 INFO:tasks.ceph.osd.4.smithi071.stderr: -1> 2019-10-13T05:35:33.799+0000 7fbf6e985700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbf6e985700 time 2019-10-13T05:35:33.798740+0000 2019-10-13T05:35:33.827 INFO:tasks.ceph.osd.4.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.827 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.828 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.828 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x563d15ef002c] 2019-10-13T05:35:33.828 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.828 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.828 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.829 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.829 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.829 INFO:tasks.ceph.osd.4.smithi071.stderr: 0> 2019-10-13T05:35:33.803+0000 7fbf6e985700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.829 INFO:tasks.ceph.osd.4.smithi071.stderr: in thread 7fbf6e985700 thread_name:bstore_aio 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (()+0x12890) [0x7fbf78e9f890] 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (gsignal()+0xc7) [0x7fbf77b51e97] 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (abort()+0x141) [0x7fbf77b53801] 2019-10-13T05:35:33.830 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x563d15ef00fd] 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: 7: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: 8: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.831 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.834 INFO:tasks.ceph.osd.4.smithi071.stderr: -9999> 2019-10-13T05:35:33.795+0000 7fbf6e985700 -1 bdev(0x563d211ba000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.834 INFO:tasks.ceph.osd.4.smithi071.stderr: -9998> 2019-10-13T05:35:33.799+0000 7fbf6e985700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbf6e985700 time 2019-10-13T05:35:33.798740+0000 2019-10-13T05:35:33.835 INFO:tasks.ceph.osd.4.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.835 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.835 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.835 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x563d15ef002c] 2019-10-13T05:35:33.835 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.836 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.836 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: -9997> 2019-10-13T05:35:33.803+0000 7fbf6e985700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: in thread 7fbf6e985700 thread_name:bstore_aio 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.837 INFO:tasks.ceph.osd.4.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 1: (()+0x12890) [0x7fbf78e9f890] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 2: (gsignal()+0xc7) [0x7fbf77b51e97] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 3: (abort()+0x141) [0x7fbf77b53801] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x563d15ef00fd] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x563d165592f9] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x563d1655b24d] 2019-10-13T05:35:33.838 INFO:tasks.ceph.osd.4.smithi071.stderr: 7: (()+0x76db) [0x7fbf78e946db] 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.4.smithi071.stderr: 8: (clone()+0x3f) [0x7fbf77c3488f] 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.4.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.4.smithi071.stderr: 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.6.smithi071.stderr: -4> 2019-10-13T05:35:33.795+0000 7f28e73a8700 -1 bdev(0x55a19fafc000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.6.smithi071.stderr: -1> 2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f28e73a8700 time 2019-10-13T05:35:33.801741+0000 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.6.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.839 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a19436002c] 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.840 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: 0> 2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: in thread 7f28e73a8700 thread_name:bstore_aio 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (()+0x12890) [0x7f28f18c2890] 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f28f0574e97] 2019-10-13T05:35:33.841 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (abort()+0x141) [0x7f28f0576801] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1943600fd] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 7: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 8: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.842 INFO:tasks.ceph.osd.6.smithi071.stderr: -9999> 2019-10-13T05:35:33.795+0000 7f28e73a8700 -1 bdev(0x55a19fafc000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: -9998> 2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f28e73a8700 time 2019-10-13T05:35:33.801741+0000 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a19436002c] 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.843 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: -9997> 2019-10-13T05:35:33.803+0000 7f28e73a8700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: in thread 7f28e73a8700 thread_name:bstore_aio 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.844 INFO:tasks.ceph.osd.6.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 1: (()+0x12890) [0x7f28f18c2890] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f28f0574e97] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 3: (abort()+0x141) [0x7f28f0576801] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1943600fd] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a1949c92f9] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a1949cb24d] 2019-10-13T05:35:33.845 INFO:tasks.ceph.osd.6.smithi071.stderr: 7: (()+0x76db) [0x7f28f18b76db] 2019-10-13T05:35:33.846 INFO:tasks.ceph.osd.6.smithi071.stderr: 8: (clone()+0x3f) [0x7f28f065788f] 2019-10-13T05:35:33.846 INFO:tasks.ceph.osd.6.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:33.846 INFO:tasks.ceph.osd.6.smithi071.stderr: 2019-10-13T05:35:33.910 INFO:tasks.ceph.osd.6.smithi071.stderr:daemon-helper: command failed with exit status 5 2019-10-13T05:35:33.911 INFO:tasks.ceph.osd.4.smithi071.stderr:daemon-helper: command failed with exit status 5 2019-10-13T05:35:34.494 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:35:34.487+0000 7f99962ed700 -1 bdev(0x558114b9ca80 /var/lib/ceph/osd/ceph-7/block) _sync_write pwritev error: (28) No space left on device 2019-10-13T05:35:34.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2019-10-13T05:35:35.181 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 bdev(0x55e78143e000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9d7eeb3700 time 2019-10-13T05:35:35.178525+0000 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55e7756fb02c] 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.185 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9d7eeb3700 time 2019-10-13T05:35:35.178525+0000 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.186 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55e7756fb02c] 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.187 INFO:tasks.ceph.osd.5.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:35.188 INFO:tasks.ceph.osd.5.smithi071.stderr: in thread 7f9d7eeb3700 thread_name:bstore_aio 2019-10-13T05:35:35.188 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:35:35.179+0000 7f999ed04700 -1 bdev(0x558114b9ce00 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.188 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.189 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (()+0x12890) [0x7f9d893cd890] 2019-10-13T05:35:35.189 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f9d8807fe97] 2019-10-13T05:35:35.189 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (abort()+0x141) [0x7f9d88081801] 2019-10-13T05:35:35.189 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55e7756fb0fd] 2019-10-13T05:35:35.189 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr: 7: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr: 8: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr:2019-10-13T05:35:35.179+0000 7f9d7eeb3700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr: in thread 7f9d7eeb3700 thread_name:bstore_aio 2019-10-13T05:35:35.190 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.191 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.191 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (()+0x12890) [0x7f9d893cd890] 2019-10-13T05:35:35.191 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f9d8807fe97] 2019-10-13T05:35:35.192 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (abort()+0x141) [0x7f9d88081801] 2019-10-13T05:35:35.192 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55e7756fb0fd] 2019-10-13T05:35:35.192 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.192 INFO:tasks.ceph.osd.5.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.192 INFO:tasks.ceph.osd.5.smithi071.stderr: 7: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.193 INFO:tasks.ceph.osd.5.smithi071.stderr: 8: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.193 INFO:tasks.ceph.osd.5.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.193 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.194 INFO:tasks.ceph.osd.7.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f999ed04700 time 2019-10-13T05:35:35.183900+0000 2019-10-13T05:35:35.194 INFO:tasks.ceph.osd.7.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.194 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x5581095e702c] 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.195 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:35:35.183+0000 7f999ed04700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f999ed04700 time 2019-10-13T05:35:35.183900+0000 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x5581095e702c] 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.196 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: in thread 7f999ed04700 thread_name:bstore_aio 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.197 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (()+0x12890) [0x7f99a921e890] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f99a7ed0e97] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (abort()+0x141) [0x7f99a7ed2801] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x5581095e70fd] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 7: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.198 INFO:tasks.ceph.osd.7.smithi071.stderr: 8: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.199 INFO:tasks.ceph.osd.7.smithi071.stderr:2019-10-13T05:35:35.187+0000 7f999ed04700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.199 INFO:tasks.ceph.osd.7.smithi071.stderr: in thread 7f999ed04700 thread_name:bstore_aio 2019-10-13T05:35:35.199 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.199 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (()+0x12890) [0x7f99a921e890] 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f99a7ed0e97] 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (abort()+0x141) [0x7f99a7ed2801] 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x5581095e70fd] 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.200 INFO:tasks.ceph.osd.7.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.201 INFO:tasks.ceph.osd.7.smithi071.stderr: 7: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.201 INFO:tasks.ceph.osd.7.smithi071.stderr: 8: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.201 INFO:tasks.ceph.osd.7.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.201 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.215 INFO:tasks.ceph.osd.5.smithi071.stderr: -18> 2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 bdev(0x55e78143e000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.215 INFO:tasks.ceph.osd.5.smithi071.stderr: -17> 2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9d7eeb3700 time 2019-10-13T05:35:35.178525+0000 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55e7756fb02c] 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.216 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.217 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.217 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.217 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.217 INFO:tasks.ceph.osd.5.smithi071.stderr: 0> 2019-10-13T05:35:35.179+0000 7f9d7eeb3700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.217 INFO:tasks.ceph.osd.5.smithi071.stderr: in thread 7f9d7eeb3700 thread_name:bstore_aio 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (()+0x12890) [0x7f9d893cd890] 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f9d8807fe97] 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (abort()+0x141) [0x7f9d88081801] 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55e7756fb0fd] 2019-10-13T05:35:35.218 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.219 INFO:tasks.ceph.osd.5.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.219 INFO:tasks.ceph.osd.5.smithi071.stderr: 7: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.219 INFO:tasks.ceph.osd.5.smithi071.stderr: 8: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.219 INFO:tasks.ceph.osd.5.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.219 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.220 INFO:tasks.ceph.osd.5.smithi071.stderr: -9999> 2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 bdev(0x55e78143e000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.220 INFO:tasks.ceph.osd.5.smithi071.stderr: -9998> 2019-10-13T05:35:35.175+0000 7f9d7eeb3700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9d7eeb3700 time 2019-10-13T05:35:35.178525+0000 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55e7756fb02c] 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.221 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.222 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.222 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.222 INFO:tasks.ceph.osd.5.smithi071.stderr: -9997> 2019-10-13T05:35:35.179+0000 7f9d7eeb3700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.222 INFO:tasks.ceph.osd.5.smithi071.stderr: in thread 7f9d7eeb3700 thread_name:bstore_aio 2019-10-13T05:35:35.222 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 1: (()+0x12890) [0x7f9d893cd890] 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f9d8807fe97] 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 3: (abort()+0x141) [0x7f9d88081801] 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55e7756fb0fd] 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55e775d642f9] 2019-10-13T05:35:35.223 INFO:tasks.ceph.osd.5.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55e775d6624d] 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.5.smithi071.stderr: 7: (()+0x76db) [0x7f9d893c26db] 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.5.smithi071.stderr: 8: (clone()+0x3f) [0x7f9d8816288f] 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.5.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.5.smithi071.stderr: 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.7.smithi071.stderr: -250> 2019-10-13T05:35:34.487+0000 7f99962ed700 -1 bdev(0x558114b9ca80 /var/lib/ceph/osd/ceph-7/block) _sync_write pwritev error: (28) No space left on device 2019-10-13T05:35:35.224 INFO:tasks.ceph.osd.7.smithi071.stderr: -2> 2019-10-13T05:35:35.179+0000 7f999ed04700 -1 bdev(0x558114b9ce00 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.225 INFO:tasks.ceph.osd.7.smithi071.stderr: -1> 2019-10-13T05:35:35.183+0000 7f999ed04700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f999ed04700 time 2019-10-13T05:35:35.183900+0000 2019-10-13T05:35:35.225 INFO:tasks.ceph.osd.7.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.225 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.225 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.226 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x5581095e702c] 2019-10-13T05:35:35.226 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.226 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.226 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: 0> 2019-10-13T05:35:35.187+0000 7f999ed04700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: in thread 7f999ed04700 thread_name:bstore_aio 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.227 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (()+0x12890) [0x7f99a921e890] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f99a7ed0e97] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (abort()+0x141) [0x7f99a7ed2801] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x5581095e70fd] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 7: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.228 INFO:tasks.ceph.osd.7.smithi071.stderr: 8: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.229 INFO:tasks.ceph.osd.7.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.229 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.229 INFO:tasks.ceph.osd.7.smithi071.stderr: -9999> 2019-10-13T05:35:34.487+0000 7f99962ed700 -1 bdev(0x558114b9ca80 /var/lib/ceph/osd/ceph-7/block) _sync_write pwritev error: (28) No space left on device 2019-10-13T05:35:35.229 INFO:tasks.ceph.osd.7.smithi071.stderr: -9998> 2019-10-13T05:35:35.179+0000 7f999ed04700 -1 bdev(0x558114b9ce00 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T05:35:35.230 INFO:tasks.ceph.osd.7.smithi071.stderr: -9997> 2019-10-13T05:35:35.183+0000 7f999ed04700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f999ed04700 time 2019-10-13T05:35:35.183900+0000 2019-10-13T05:35:35.230 INFO:tasks.ceph.osd.7.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T05:35:35.230 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.230 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.230 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x5581095e702c] 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.231 INFO:tasks.ceph.osd.7.smithi071.stderr: -9996> 2019-10-13T05:35:35.187+0000 7f999ed04700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:35.232 INFO:tasks.ceph.osd.7.smithi071.stderr: in thread 7f999ed04700 thread_name:bstore_aio 2019-10-13T05:35:35.232 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.232 INFO:tasks.ceph.osd.7.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:35.232 INFO:tasks.ceph.osd.7.smithi071.stderr: 1: (()+0x12890) [0x7f99a921e890] 2019-10-13T05:35:35.232 INFO:tasks.ceph.osd.7.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f99a7ed0e97] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 3: (abort()+0x141) [0x7f99a7ed2801] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x5581095e70fd] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x558109c502f9] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x558109c5224d] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 7: (()+0x76db) [0x7f99a92136db] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: 8: (clone()+0x3f) [0x7f99a7fb388f] 2019-10-13T05:35:35.233 INFO:tasks.ceph.osd.7.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:35.234 INFO:tasks.ceph.osd.7.smithi071.stderr: 2019-10-13T05:35:35.304 INFO:tasks.ceph.osd.5.smithi071.stderr:daemon-helper: command failed with exit status 5 2019-10-13T05:35:35.319 INFO:tasks.ceph.osd.7.smithi071.stderr:daemon-helper: command failed with exit status 5 2019-10-13T05:35:36.194 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:35:36.187+0000 7f0e63bfa700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.194 INFO:tasks.ceph.mon.b.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.195 INFO:tasks.ceph.mon.c.smithi071.stderr:2019-10-13T05:35:36.187+0000 7f2ca29a9700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.195 INFO:tasks.ceph.mon.c.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.199 INFO:tasks.ceph.mon.c.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2ca29a9700 time 2019-10-13T05:35:36.192432+0000 2019-10-13T05:35:36.199 INFO:tasks.ceph.mon.c.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.199 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.199 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2caf186008] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.200 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.201 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.202 INFO:tasks.ceph.mon.c.smithi071.stderr:2019-10-13T05:35:36.191+0000 7f2ca29a9700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2ca29a9700 time 2019-10-13T05:35:36.192432+0000 2019-10-13T05:35:36.202 INFO:tasks.ceph.mon.c.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.202 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.203 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.203 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2caf186008] 2019-10-13T05:35:36.203 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.203 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.204 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.204 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.205 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.205 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.205 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.205 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.205 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.206 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.206 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.206 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.206 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.206 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.208 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.210 INFO:tasks.ceph.mon.c.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:36.210 INFO:tasks.ceph.mon.c.smithi071.stderr: in thread 7f2ca29a9700 thread_name:ms_dispatch 2019-10-13T05:35:36.212 INFO:tasks.ceph.mon.b.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f0e63bfa700 time 2019-10-13T05:35:36.191376+0000 2019-10-13T05:35:36.219 INFO:tasks.ceph.mon.b.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.219 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.219 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f0e71bda008] 2019-10-13T05:35:36.219 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.220 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.221 INFO:tasks.ceph.mon.b.smithi071.stderr:*** Caught signal (Aborted) ** 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr: in thread 7f0e63bfa700 thread_name:fn_monstore 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:35:36.195+0000 7f0e63bfa700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f0e63bfa700 time 2019-10-13T05:35:36.191376+0000 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f0e71bda008] 2019-10-13T05:35:36.222 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.232 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.232 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.232 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.232 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.233 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.233 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.233 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.233 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.234 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (()+0x12890) [0x7f0e70a47890] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f0e6fb3fe97] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (abort()+0x141) [0x7f0e6fb41801] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f0e71bda0d9] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.235 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.236 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr: 15: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr: 16: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr: 17: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr:2019-10-13T05:35:36.199+0000 7f0e63bfa700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.237 INFO:tasks.ceph.mon.b.smithi071.stderr: in thread 7f0e63bfa700 thread_name:fn_monstore 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (()+0x12890) [0x7f0e70a47890] 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f0e6fb3fe97] 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (abort()+0x141) [0x7f0e6fb41801] 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f0e71bda0d9] 2019-10-13T05:35:36.238 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.239 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: 15: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: 16: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: 17: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.240 INFO:tasks.ceph.mon.b.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (()+0x12890) [0x7f2cadff3890] 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f2cad0ebe97] 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (abort()+0x141) [0x7f2cad0ed801] 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2caf1860d9] 2019-10-13T05:35:36.241 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.242 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.243 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.243 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.243 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.243 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.243 INFO:tasks.ceph.mon.c.smithi071.stderr: 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.244 INFO:tasks.ceph.mon.c.smithi071.stderr: 17: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.244 INFO:tasks.ceph.mon.c.smithi071.stderr: 18: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.244 INFO:tasks.ceph.mon.c.smithi071.stderr:2019-10-13T05:35:36.195+0000 7f2ca29a9700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: in thread 7f2ca29a9700 thread_name:ms_dispatch 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (()+0x12890) [0x7f2cadff3890] 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f2cad0ebe97] 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (abort()+0x141) [0x7f2cad0ed801] 2019-10-13T05:35:36.245 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2caf1860d9] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.246 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 17: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.247 INFO:tasks.ceph.mon.c.smithi071.stderr: 18: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.248 INFO:tasks.ceph.mon.c.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.248 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.249 INFO:tasks.ceph.mon.c.smithi071.stderr: -2> 2019-10-13T05:35:36.187+0000 7f2ca29a9700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.249 INFO:tasks.ceph.mon.c.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.249 INFO:tasks.ceph.mon.c.smithi071.stderr: -1> 2019-10-13T05:35:36.191+0000 7f2ca29a9700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2ca29a9700 time 2019-10-13T05:35:36.192432+0000 2019-10-13T05:35:36.249 INFO:tasks.ceph.mon.c.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.249 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2caf186008] 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.250 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.251 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.251 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.251 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.251 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.251 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: 0> 2019-10-13T05:35:36.195+0000 7f2ca29a9700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.252 INFO:tasks.ceph.mon.c.smithi071.stderr: in thread 7f2ca29a9700 thread_name:ms_dispatch 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (()+0x12890) [0x7f2cadff3890] 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f2cad0ebe97] 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (abort()+0x141) [0x7f2cad0ed801] 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2caf1860d9] 2019-10-13T05:35:36.253 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.254 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.254 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.254 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.255 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.256 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.256 INFO:tasks.ceph.mon.c.smithi071.stderr: 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.256 INFO:tasks.ceph.mon.c.smithi071.stderr: 17: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.256 INFO:tasks.ceph.mon.c.smithi071.stderr: 18: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.256 INFO:tasks.ceph.mon.c.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr: -9999> 2019-10-13T05:35:36.187+0000 7f2ca29a9700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr: -9998> 2019-10-13T05:35:36.191+0000 7f2ca29a9700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2ca29a9700 time 2019-10-13T05:35:36.192432+0000 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.257 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2caf186008] 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.258 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.259 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: -9997> 2019-10-13T05:35:36.195+0000 7f2ca29a9700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: in thread 7f2ca29a9700 thread_name:ms_dispatch 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.260 INFO:tasks.ceph.mon.c.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 1: (()+0x12890) [0x7f2cadff3890] 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f2cad0ebe97] 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 3: (abort()+0x141) [0x7f2cad0ed801] 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2caf1860d9] 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560f24a7a760] 2019-10-13T05:35:36.261 INFO:tasks.ceph.mon.c.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x560f24c28a61] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x560f24b9ff43] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x560f24a8605c] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x560f24b8e14f] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 10: (Paxos::handle_commit(boost::intrusive_ptr)+0x153) [0x560f24b93873] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 11: (Paxos::dispatch(boost::intrusive_ptr)+0x433) [0x560f24b998d3] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 12: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560f24abbb1d] 2019-10-13T05:35:36.262 INFO:tasks.ceph.mon.c.smithi071.stderr: 13: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560f24abc06a] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560f24aeb8c8] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 15: (DispatchQueue::entry()+0x11ea) [0x7f2caf368caa] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2caf4053cd] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 17: (()+0x76db) [0x7f2cadfe86db] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 18: (clone()+0x3f) [0x7f2cad1ce88f] 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.263 INFO:tasks.ceph.mon.c.smithi071.stderr: 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr: -2> 2019-10-13T05:35:36.187+0000 7f0e63bfa700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr: -1> 2019-10-13T05:35:36.195+0000 7f0e63bfa700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f0e63bfa700 time 2019-10-13T05:35:36.191376+0000 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.264 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.266 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f0e71bda008] 2019-10-13T05:35:36.266 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.266 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.267 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.268 INFO:tasks.ceph.mon.b.smithi071.stderr: 0> 2019-10-13T05:35:36.199+0000 7f0e63bfa700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.269 INFO:tasks.ceph.mon.b.smithi071.stderr: in thread 7f0e63bfa700 thread_name:fn_monstore 2019-10-13T05:35:36.269 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.269 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.269 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (()+0x12890) [0x7f0e70a47890] 2019-10-13T05:35:36.269 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f0e6fb3fe97] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (abort()+0x141) [0x7f0e6fb41801] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f0e71bda0d9] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.270 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.271 INFO:tasks.ceph.mon.b.smithi071.stderr: 15: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: 16: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: 17: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: -9999> 2019-10-13T05:35:36.187+0000 7f0e63bfa700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr:Put( Prefix = o key = 'dmap'0x0066756c'l_latest' Value size = 8) 2019-10-13T05:35:36.272 INFO:tasks.ceph.mon.b.smithi071.stderr: -9998> 2019-10-13T05:35:36.195+0000 7f0e63bfa700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f0e63bfa700 time 2019-10-13T05:35:36.191376+0000 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f0e71bda008] 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.273 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.274 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.275 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.275 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.275 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.275 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: -9997> 2019-10-13T05:35:36.199+0000 7f0e63bfa700 -1 *** Caught signal (Aborted) ** 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: in thread 7f0e63bfa700 thread_name:fn_monstore 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: 1: (()+0x12890) [0x7f0e70a47890] 2019-10-13T05:35:36.276 INFO:tasks.ceph.mon.b.smithi071.stderr: 2: (gsignal()+0xc7) [0x7f0e6fb3fe97] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 3: (abort()+0x141) [0x7f0e6fb41801] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f0e71bda0d9] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x55a2b5a0f760] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 6: (OSDMonitor::update_from_paxos(bool*)+0x1b11) [0x55a2b5bbda61] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 7: (PaxosService::refresh(bool*)+0x273) [0x55a2b5b34f43] 2019-10-13T05:35:36.277 INFO:tasks.ceph.mon.b.smithi071.stderr: 8: (Monitor::refresh_from_paxos(bool*)+0x10c) [0x55a2b5a1b05c] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 9: (Paxos::do_refresh()+0x4f) [0x55a2b5b2314f] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 10: (Paxos::commit_finish()+0x445) [0x55a2b5b2ef55] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 11: (C_Committed::finish(int)+0x41) [0x55a2b5b338e1] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 12: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 13: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55a2b5b3363f] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 14: (Context::complete(int)+0x9) [0x55a2b5a56869] 2019-10-13T05:35:36.278 INFO:tasks.ceph.mon.b.smithi071.stderr: 15: (Finisher::finisher_thread_entry()+0x195) [0x7f0e71c2e355] 2019-10-13T05:35:36.279 INFO:tasks.ceph.mon.b.smithi071.stderr: 16: (()+0x76db) [0x7f0e70a3c6db] 2019-10-13T05:35:36.279 INFO:tasks.ceph.mon.b.smithi071.stderr: 17: (clone()+0x3f) [0x7f0e6fc2288f] 2019-10-13T05:35:36.279 INFO:tasks.ceph.mon.b.smithi071.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T05:35:36.279 INFO:tasks.ceph.mon.b.smithi071.stderr: 2019-10-13T05:35:36.475 INFO:tasks.ceph.mon.c.smithi071.stderr:daemon-helper: command crashed with signal 6 2019-10-13T05:35:36.679 INFO:tasks.ceph.mon.b.smithi071.stderr:daemon-helper: command crashed with signal 6 2019-10-13T05:35:40.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2019-10-13T05:35:40.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2019-10-13T05:35:40.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2019-10-13T05:35:40.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2019-10-13T05:35:40.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2019-10-13T05:35:40.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2019-10-13T05:35:44.747 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:35:44.747 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:44.751 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:35:44.751 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:46.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2019-10-13T05:35:46.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2019-10-13T05:35:46.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2019-10-13T05:35:46.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2019-10-13T05:35:46.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2019-10-13T05:35:46.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2019-10-13T05:35:52.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~12s 2019-10-13T05:35:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2019-10-13T05:35:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~12s 2019-10-13T05:35:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~18s 2019-10-13T05:35:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~12s 2019-10-13T05:35:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~12s 2019-10-13T05:35:58.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~18s 2019-10-13T05:35:58.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~18s 2019-10-13T05:35:58.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~18s 2019-10-13T05:35:58.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~24s 2019-10-13T05:35:58.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~18s 2019-10-13T05:35:58.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~18s 2019-10-13T05:36:04.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~24s 2019-10-13T05:36:04.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~24s 2019-10-13T05:36:04.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~24s 2019-10-13T05:36:04.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2019-10-13T05:36:04.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~24s 2019-10-13T05:36:04.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~24s 2019-10-13T05:36:10.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2019-10-13T05:36:10.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2019-10-13T05:36:10.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2019-10-13T05:36:10.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~36s 2019-10-13T05:36:10.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2019-10-13T05:36:10.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2019-10-13T05:36:14.795 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:36:14.795 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:14.802 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:36:14.802 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:16.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2019-10-13T05:36:16.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2019-10-13T05:36:16.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2019-10-13T05:36:16.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~42s 2019-10-13T05:36:16.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2019-10-13T05:36:16.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2019-10-13T05:36:21.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~41s 2019-10-13T05:36:21.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~41s 2019-10-13T05:36:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~41s 2019-10-13T05:36:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~48s 2019-10-13T05:36:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~41s 2019-10-13T05:36:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~41s 2019-10-13T05:36:27.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~47s 2019-10-13T05:36:27.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~47s 2019-10-13T05:36:27.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~47s 2019-10-13T05:36:27.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~54s 2019-10-13T05:36:27.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~47s 2019-10-13T05:36:27.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~47s 2019-10-13T05:36:33.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~53s 2019-10-13T05:36:33.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~53s 2019-10-13T05:36:33.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~53s 2019-10-13T05:36:33.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2019-10-13T05:36:33.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~53s 2019-10-13T05:36:33.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~53s 2019-10-13T05:36:39.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~59s 2019-10-13T05:36:39.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~59s 2019-10-13T05:36:39.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~59s 2019-10-13T05:36:39.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~66s 2019-10-13T05:36:39.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~59s 2019-10-13T05:36:39.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~59s 2019-10-13T05:36:44.918 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:36:44.919 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:44.923 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:36:44.923 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:45.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2019-10-13T05:36:45.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2019-10-13T05:36:45.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2019-10-13T05:36:45.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~71s 2019-10-13T05:36:45.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2019-10-13T05:36:45.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2019-10-13T05:36:51.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~71s 2019-10-13T05:36:51.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~71s 2019-10-13T05:36:51.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~71s 2019-10-13T05:36:51.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~77s 2019-10-13T05:36:51.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~71s 2019-10-13T05:36:51.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~71s 2019-10-13T05:36:57.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~77s 2019-10-13T05:36:57.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~77s 2019-10-13T05:36:57.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~77s 2019-10-13T05:36:57.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~83s 2019-10-13T05:36:57.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~77s 2019-10-13T05:36:57.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~77s 2019-10-13T05:37:03.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~83s 2019-10-13T05:37:03.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~83s 2019-10-13T05:37:03.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~83s 2019-10-13T05:37:03.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~89s 2019-10-13T05:37:03.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~83s 2019-10-13T05:37:03.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~83s 2019-10-13T05:37:09.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~89s 2019-10-13T05:37:09.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~89s 2019-10-13T05:37:09.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~89s 2019-10-13T05:37:09.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2019-10-13T05:37:09.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~89s 2019-10-13T05:37:09.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~89s 2019-10-13T05:37:15.008 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:37:15.008 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:15.011 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:37:15.012 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:15.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2019-10-13T05:37:15.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2019-10-13T05:37:15.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2019-10-13T05:37:15.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~101s 2019-10-13T05:37:15.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2019-10-13T05:37:15.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~101s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~101s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~101s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~107s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~101s 2019-10-13T05:37:21.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~101s 2019-10-13T05:37:26.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~106s 2019-10-13T05:37:26.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~106s 2019-10-13T05:37:26.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~106s 2019-10-13T05:37:26.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~113s 2019-10-13T05:37:26.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~106s 2019-10-13T05:37:26.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~106s 2019-10-13T05:37:32.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~112s 2019-10-13T05:37:32.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~112s 2019-10-13T05:37:32.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~112s 2019-10-13T05:37:32.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~119s 2019-10-13T05:37:32.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~112s 2019-10-13T05:37:32.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~112s 2019-10-13T05:37:37.891 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:37:37.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:37:38.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~118s 2019-10-13T05:37:38.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~118s 2019-10-13T05:37:38.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~118s 2019-10-13T05:37:38.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~125s 2019-10-13T05:37:38.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~118s 2019-10-13T05:37:38.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~118s 2019-10-13T05:37:42.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:37:42.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:37:44.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~124s 2019-10-13T05:37:44.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~124s 2019-10-13T05:37:44.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~124s 2019-10-13T05:37:44.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~131s 2019-10-13T05:37:44.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~124s 2019-10-13T05:37:44.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~124s 2019-10-13T05:37:45.133 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:37:45.133 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:45.137 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:37:45.137 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:47.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:37:47.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:37:50.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2019-10-13T05:37:50.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2019-10-13T05:37:50.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2019-10-13T05:37:50.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~137s 2019-10-13T05:37:50.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2019-10-13T05:37:50.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2019-10-13T05:37:52.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:37:52.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:37:56.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~136s 2019-10-13T05:37:56.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~136s 2019-10-13T05:37:56.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~136s 2019-10-13T05:37:56.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~142s 2019-10-13T05:37:56.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~136s 2019-10-13T05:37:56.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~136s 2019-10-13T05:37:57.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:37:57.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:02.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~142s 2019-10-13T05:38:02.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~142s 2019-10-13T05:38:02.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~142s 2019-10-13T05:38:02.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~148s 2019-10-13T05:38:02.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~142s 2019-10-13T05:38:02.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~142s 2019-10-13T05:38:02.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:02.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:07.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:07.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:08.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~148s 2019-10-13T05:38:08.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~148s 2019-10-13T05:38:08.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~148s 2019-10-13T05:38:08.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~154s 2019-10-13T05:38:08.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~148s 2019-10-13T05:38:08.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~148s 2019-10-13T05:38:12.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:12.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~154s 2019-10-13T05:38:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~154s 2019-10-13T05:38:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~154s 2019-10-13T05:38:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2019-10-13T05:38:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~154s 2019-10-13T05:38:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~154s 2019-10-13T05:38:15.256 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:38:15.257 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:15.264 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:38:15.264 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:17.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:17.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:20.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2019-10-13T05:38:20.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2019-10-13T05:38:20.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2019-10-13T05:38:20.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~166s 2019-10-13T05:38:20.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2019-10-13T05:38:20.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2019-10-13T05:38:22.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:22.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:26.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~166s 2019-10-13T05:38:26.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~166s 2019-10-13T05:38:26.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~166s 2019-10-13T05:38:26.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~172s 2019-10-13T05:38:26.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~166s 2019-10-13T05:38:26.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~166s 2019-10-13T05:38:27.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:27.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:31.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~171s 2019-10-13T05:38:31.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~171s 2019-10-13T05:38:31.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~171s 2019-10-13T05:38:31.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~178s 2019-10-13T05:38:31.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~171s 2019-10-13T05:38:31.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~171s 2019-10-13T05:38:32.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:32.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~177s 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~177s 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~177s 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~184s 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~177s 2019-10-13T05:38:37.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~177s 2019-10-13T05:38:37.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:37.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:42.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:42.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~183s 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~183s 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~183s 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~190s 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~183s 2019-10-13T05:38:43.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~183s 2019-10-13T05:38:45.362 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:38:45.362 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:45.369 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:38:45.370 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:47.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:47.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~189s 2019-10-13T05:38:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~189s 2019-10-13T05:38:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~189s 2019-10-13T05:38:49.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~196s 2019-10-13T05:38:49.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~189s 2019-10-13T05:38:49.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~189s 2019-10-13T05:38:52.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:52.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:38:55.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2019-10-13T05:38:55.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2019-10-13T05:38:55.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2019-10-13T05:38:55.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~202s 2019-10-13T05:38:55.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~195s 2019-10-13T05:38:55.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~195s 2019-10-13T05:38:57.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:38:57.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~201s 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~201s 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~201s 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~207s 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~201s 2019-10-13T05:39:01.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~201s 2019-10-13T05:39:02.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:02.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~207s 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~207s 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~207s 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~213s 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~207s 2019-10-13T05:39:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~207s 2019-10-13T05:39:07.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:07.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:12.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:12.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:13.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~213s 2019-10-13T05:39:13.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~213s 2019-10-13T05:39:13.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~213s 2019-10-13T05:39:13.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~219s 2019-10-13T05:39:13.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~213s 2019-10-13T05:39:13.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~213s 2019-10-13T05:39:15.505 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:39:15.505 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:15.509 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:39:15.509 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:17.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:17.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:19.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~219s 2019-10-13T05:39:19.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~219s 2019-10-13T05:39:19.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~219s 2019-10-13T05:39:19.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2019-10-13T05:39:19.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~219s 2019-10-13T05:39:19.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~219s 2019-10-13T05:39:22.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:22.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:25.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2019-10-13T05:39:25.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2019-10-13T05:39:25.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2019-10-13T05:39:25.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~231s 2019-10-13T05:39:25.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~225s 2019-10-13T05:39:25.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~225s 2019-10-13T05:39:27.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:27.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 424 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:31.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~231s 2019-10-13T05:39:31.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~231s 2019-10-13T05:39:31.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~231s 2019-10-13T05:39:31.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~237s 2019-10-13T05:39:31.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~231s 2019-10-13T05:39:31.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~231s 2019-10-13T05:39:32.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:32.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:36.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~236s 2019-10-13T05:39:36.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~236s 2019-10-13T05:39:36.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~236s 2019-10-13T05:39:36.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~243s 2019-10-13T05:39:36.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~236s 2019-10-13T05:39:36.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~236s 2019-10-13T05:39:37.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:37.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:42.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~242s 2019-10-13T05:39:42.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~242s 2019-10-13T05:39:42.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~242s 2019-10-13T05:39:42.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~249s 2019-10-13T05:39:42.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~242s 2019-10-13T05:39:42.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~242s 2019-10-13T05:39:42.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:42.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:45.619 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:39:45.619 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:45.623 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:39:45.624 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:47.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:47.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 503 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~248s 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~248s 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~248s 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~255s 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~248s 2019-10-13T05:39:48.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~248s 2019-10-13T05:39:52.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:52.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~254s 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~254s 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~254s 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~261s 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~254s 2019-10-13T05:39:54.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~254s 2019-10-13T05:39:57.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:39:57.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:00.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~260s 2019-10-13T05:40:00.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~260s 2019-10-13T05:40:00.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~260s 2019-10-13T05:40:00.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~267s 2019-10-13T05:40:00.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~260s 2019-10-13T05:40:00.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~260s 2019-10-13T05:40:02.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:02.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~266s 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~266s 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~266s 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~272s 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~266s 2019-10-13T05:40:06.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~266s 2019-10-13T05:40:07.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:07.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~272s 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~272s 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~272s 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~278s 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~272s 2019-10-13T05:40:12.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~272s 2019-10-13T05:40:12.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:12.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:15.767 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:40:15.768 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:15.771 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:40:15.771 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:17.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:17.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:18.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~278s 2019-10-13T05:40:18.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~278s 2019-10-13T05:40:18.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~278s 2019-10-13T05:40:18.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~284s 2019-10-13T05:40:18.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~278s 2019-10-13T05:40:18.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~278s 2019-10-13T05:40:22.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:22.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:24.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~284s 2019-10-13T05:40:24.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~284s 2019-10-13T05:40:24.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~284s 2019-10-13T05:40:24.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2019-10-13T05:40:24.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~284s 2019-10-13T05:40:24.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~284s 2019-10-13T05:40:27.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:27.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:30.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2019-10-13T05:40:30.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2019-10-13T05:40:30.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2019-10-13T05:40:30.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~296s 2019-10-13T05:40:30.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2019-10-13T05:40:30.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2019-10-13T05:40:32.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:32.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:36.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~296s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~296s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~296s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~302s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~296s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~296s 2019-10-13T05:40:36.073 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-10-13T05:40:36.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T05:40:36.074 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:40:36.074 INFO:teuthology.orchestra.run.smithi129:> stat --file-system '--printf=%T 2019-10-13T05:40:36.074 INFO:teuthology.orchestra.run.smithi129:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T05:40:37.919 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:37.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:42.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:42.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:45.823 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:40:45.824 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:45.827 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:40:45.827 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:47.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:47.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:52.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:52.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:40:57.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:40:57.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:02.923 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:02.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:07.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:07.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:12.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:12.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:15.872 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:41:15.872 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:15.876 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:41:15.876 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:17.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:17.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:22.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:22.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:27.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:27.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:32.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:32.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:37.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:37.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:42.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:42.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:45.971 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:41:45.972 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:45.975 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:41:45.975 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:47.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:47.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 983 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:52.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:52.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:41:57.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:41:57.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:02.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:02.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:07.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:07.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1063 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:12.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:12.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:16.026 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:42:16.027 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:16.030 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:42:16.030 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:17.936 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:17.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:22.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:22.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:27.939 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:27.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:32.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:32.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:37.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:37.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:42.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:42.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1205 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:46.158 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:42:46.158 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:46.162 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:42:46.162 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:47.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:47.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:52.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:52.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:42:57.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:42:57.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:02.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:02.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:07.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:07.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:12.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:12.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:16.298 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:43:16.298 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:16.302 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:43:16.302 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:17.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:17.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:22.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:22.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:27.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:27.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:32.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:32.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:37.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:37.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:42.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:42.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:46.391 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:43:46.391 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:46.395 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:43:46.395 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:47.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:47.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:52.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:52.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:43:57.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:43:57.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:02.957 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:02.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:07.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:07.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1561 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:12.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:12.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:16.503 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:44:16.503 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:16.507 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:44:16.507 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:17.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:17.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:22.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:22.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:27.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:27.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:32.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:32.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:37.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:37.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:42.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:42.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:46.643 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:44:46.644 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:46.647 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:44:46.648 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:47.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:47.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:52.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:52.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:44:57.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:44:57.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1771 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:02.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:02.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:07.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:07.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:12.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:12.964+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:16.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:45:16.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:16.851 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:45:16.851 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:17.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:17.964+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:22.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:22.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:27.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:27.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:32.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:32.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:37.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:37.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:42.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:42.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:46.905 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:45:46.905 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:46.909 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:45:46.909 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:47.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:47.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 1980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:52.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:52.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:45:57.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:45:57.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:02.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:02.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:07.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:07.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:12.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:12.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:17.023 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:46:17.023 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:17.026 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:46:17.026 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:17.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:17.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:22.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:22.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:27.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:27.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:32.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:32.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:37.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:37.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:42.988 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:42.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2211 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:47.208 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:46:47.208 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:47.212 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:46:47.212 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:47.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:47.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:52.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:52.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2254 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:46:57.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:46:57.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:02.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:02.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:07.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:07.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:12.993 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:12.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:17.320 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:47:17.320 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:17.324 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:47:17.324 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:17.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:17.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:22.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:22.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:27.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:27.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:32.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:32.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:37.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:37.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:42.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:42.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:47.439 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:47:47.439 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:47.445 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:47:47.445 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:47.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:47.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:53.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:52.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:47:58.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:47:57.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:03.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:02.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:08.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:07.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:13.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:12.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:17.567 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:48:17.567 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:17.571 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:48:17.571 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:18.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:17.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:23.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:22.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2629 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:28.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:28.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:33.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:33.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:38.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:38.004+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:43.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:43.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:47.730 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:48:47.731 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:47.736 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:48:47.736 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:48.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:48.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:53.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:53.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:48:58.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:48:58.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:03.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:03.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2797 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:08.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:08.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:13.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:13.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:17.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:49:17.848 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:17.852 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:49:17.852 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:18.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:18.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:23.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:23.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2881 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:28.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:28.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:33.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:33.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:38.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:38.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:43.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:43.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:47.959 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:49:47.959 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:47.963 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:49:47.963 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:48.021 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:48.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 2986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:53.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:53.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:49:58.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:49:58.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:03.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:03.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:08.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:08.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:13.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:13.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:18.027 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:18.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:18.209 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:50:18.209 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:18.213 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:50:18.213 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:23.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:23.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3132 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:28.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:28.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:33.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:33.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:38.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:38.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:43.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:43.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:48.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:48.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:48.339 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:50:48.339 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:48.343 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:50:48.343 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:53.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:53.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:50:58.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:50:58.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:03.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:03.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:08.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:08.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:13.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:13.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:18.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:18.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:18.476 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:51:18.476 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:18.493 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:51:18.493 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:23.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:23.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3383 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:28.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:28.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:33.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:33.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:38.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:38.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:43.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:43.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:48.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:48.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:48.601 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:51:48.601 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:48.605 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:51:48.605 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:53.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:53.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:51:58.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:51:58.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:03.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:03.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:08.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:08.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:13.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:13.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:18.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:18.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:18.722 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:52:18.723 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:18.726 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:52:18.726 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:23.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:23.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:28.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:28.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:33.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:33.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:38.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:38.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:43.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:43.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:48.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:48.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:48.851 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:52:48.851 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:48.855 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:52:48.855 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:53.056 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:53.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:52:58.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:52:58.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:03.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:03.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:08.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:08.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:13.060 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:13.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:18.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:18.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:19.025 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:53:19.025 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:19.029 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:53:19.029 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:23.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:23.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:28.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:28.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:33.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:33.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:38.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:38.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3948 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:43.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:43.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:48.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:48.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 3991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:49.140 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:53:49.140 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:49.144 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:53:49.144 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:53.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:53.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:53:58.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:53:58.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:03.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:03.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4054 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:08.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:08.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:13.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:13.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:18.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:18.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:19.251 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:54:19.252 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:19.255 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:54:19.255 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:23.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:23.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:28.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:28.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:33.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:33.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:38.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:38.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:43.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:43.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4221 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:48.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:48.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:49.370 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:54:49.371 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:49.374 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:54:49.375 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:53.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:53.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4264 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:54:58.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:54:58.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:03.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:03.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:08.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:08.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:13.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:13.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:18.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:18.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:19.461 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:55:19.461 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:19.465 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:55:19.465 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:23.087 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:23.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4388 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:28.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:28.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:33.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:33.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4431 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:36.084 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T05:55:36.085 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T05:55:38.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:38.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:43.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:43.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4472 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:48.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:48.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:49.551 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:55:49.551 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:49.555 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:55:49.555 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:53.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:53.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:55:58.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:55:58.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:03.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:03.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:08.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:08.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:13.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:13.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:18.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:18.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:19.683 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:56:19.683 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:19.686 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:56:19.687 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:23.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:23.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4640 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:28.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:28.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4662 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:33.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:33.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:38.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:38.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:43.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:43.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:48.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:48.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:49.807 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:56:49.808 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:49.811 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:56:49.812 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:53.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:53.096+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:56:58.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:56:58.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:03.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:03.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:08.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:08.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:13.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:13.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:18.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:18.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:19.904 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:57:19.905 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:19.909 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:57:19.909 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:23.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:23.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:28.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:28.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4913 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:33.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:33.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:38.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:38.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:43.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:43.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:48.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:48.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 4997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:50.014 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:57:50.014 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:50.025 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:57:50.025 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:53.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:53.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:57:58.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:57:58.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:03.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:03.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:08.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:08.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:13.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:13.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:18.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:18.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:20.142 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:58:20.142 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:20.145 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:58:20.146 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:23.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:23.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:28.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:28.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:33.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:33.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:38.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:38.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:43.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:43.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:48.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:48.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:50.246 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:58:50.246 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:50.251 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:58:50.251 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:53.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:53.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:58:58.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:58:58.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:03.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:03.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:08.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:08.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5333 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:13.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:13.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:18.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:18.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:20.382 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:59:20.382 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:20.393 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:59:20.393 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:23.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:23.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:28.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:28.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:33.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:33.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:38.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:38.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:43.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:43.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:48.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:48.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:50.462 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T05:59:50.463 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:50.466 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T05:59:50.466 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:53.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:53.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T05:59:58.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T05:59:58.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:03.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:03.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:08.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:08.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:13.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:13.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:18.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:18.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:20.519 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:00:20.519 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:20.523 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:00:20.523 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:23.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:23.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:28.156 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:28.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:33.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:33.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:38.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:38.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:43.156 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:43.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:48.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:48.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:50.627 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:00:50.627 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:50.630 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:00:50.630 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:53.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:53.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:00:58.160 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:00:58.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:03.161 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:03.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:08.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:08.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:13.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:13.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:18.164 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:18.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5878 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:20.716 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:01:20.717 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:20.720 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:01:20.720 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:23.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:23.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:28.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:28.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:33.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:33.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:38.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:38.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:43.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:43.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 5985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:48.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:48.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:51.014 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:01:51.014 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:51.018 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:01:51.018 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:53.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:53.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:01:58.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:01:58.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:03.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:03.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:08.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:08.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:13.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:13.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:18.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:18.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:21.063 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:02:21.063 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:21.067 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:02:21.067 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:23.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:23.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:28.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:28.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:33.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:33.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:38.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:38.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:43.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:43.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6241 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:48.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:48.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6266 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:51.193 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:02:51.194 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:51.198 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:02:51.198 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:53.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:53.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:02:58.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:02:58.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:03.190 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:03.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:08.191 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:08.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:13.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:13.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:18.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:18.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:21.326 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:03:21.326 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:21.330 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:03:21.330 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:23.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:23.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:28.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:28.187+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:33.197 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:33.191+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:38.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:38.191+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:43.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:43.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:48.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:48.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:51.433 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:03:51.433 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:51.437 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:03:51.437 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:53.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:53.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:03:58.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:03:58.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:03.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:03.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:08.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:08.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:13.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:13.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6629 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:18.208 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:18.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:21.581 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:04:21.581 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:21.585 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:04:21.585 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:23.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:23.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:28.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:28.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:33.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:33.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6714 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:38.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:38.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:43.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:43.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:48.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:48.210+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6780 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:51.679 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:04:51.679 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:51.683 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:04:51.683 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:53.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:53.210+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:04:58.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:04:58.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:03.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:03.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:08.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:08.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:13.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:13.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:18.223 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:18.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:21.910 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:05:21.910 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:21.914 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:05:21.914 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:23.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:23.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:28.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:28.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:33.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:33.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:38.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:38.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 6994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:43.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:43.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:48.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:48.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:52.118 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:05:52.118 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:52.122 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:05:52.122 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:53.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:53.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7057 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:05:58.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:05:58.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:03.234 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:03.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:08.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:08.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:13.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:13.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:18.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:18.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:22.221 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:06:22.221 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:22.225 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:06:22.225 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:23.239 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:23.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:28.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:28.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:33.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:33.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:38.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:38.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7251 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:43.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:43.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:48.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:48.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:52.340 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:06:52.340 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:52.344 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:06:52.344 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:53.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:53.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:06:58.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:06:58.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:03.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:03.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:08.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:08.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:13.252 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:13.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7401 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:18.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:18.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:22.403 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:07:22.403 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:22.407 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:07:22.407 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:23.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:23.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:28.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:28.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:33.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:33.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:38.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:38.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7508 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:43.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:43.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:48.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:48.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:52.501 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:07:52.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:52.505 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:07:52.505 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:53.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:53.253+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:07:58.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:07:58.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7594 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:03.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:03.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:08.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:08.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:13.267 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:13.261+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:18.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:18.261+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:22.631 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:08:22.631 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:22.638 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:08:22.638 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:23.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:23.261+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7700 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:28.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:28.265+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:33.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:33.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7744 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:38.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:38.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:43.275 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:43.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:48.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:48.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7810 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:52.752 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:08:52.752 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:52.756 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:08:52.756 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:53.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:53.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:08:58.280 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:08:58.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:03.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:03.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:08.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:08.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:13.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:13.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7915 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:18.285 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:18.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:22.871 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:09:22.871 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:22.875 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:09:22.875 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:23.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:23.279+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7957 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:28.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:28.279+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 7982 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:33.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:33.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:38.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:38.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:43.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:43.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:48.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:48.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:52.993 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:09:52.993 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:52.997 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:09:52.997 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:53.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:53.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:09:58.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:09:58.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:03.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:03.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:08.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:08.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:13.300 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:13.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:18.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:18.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:23.223 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:10:23.224 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:23.228 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:10:23.228 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:23.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:23.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:28.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:28.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:33.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:33.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:38.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:38.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:43.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:43.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8301 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:48.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:48.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:53.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:53.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:10:53.556 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:10:53.556 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:53.563 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:10:53.564 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:58.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:10:58.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:03.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:03.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:08.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:08.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:13.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:13.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8431 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:18.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:18.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:23.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:23.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:23.684 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:11:23.684 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:23.692 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:11:23.692 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:28.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:28.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8496 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:33.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:33.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:38.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:38.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8538 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:43.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:43.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:48.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:48.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:53.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:53.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:11:53.738 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:11:53.738 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:53.742 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:11:53.742 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:58.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:11:58.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:03.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:03.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:08.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:08.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:13.332 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:13.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:18.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:18.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8710 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:23.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:23.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:23.889 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:12:23.889 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:23.893 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:12:23.893 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:28.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:28.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:33.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:33.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:38.339 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:38.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:43.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:43.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8815 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:48.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:48.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:53.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:53.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:12:54.007 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:12:54.007 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:54.012 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:12:54.012 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:58.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:12:58.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:03.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:03.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:08.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:08.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:13.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:13.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:18.351 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:18.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:23.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:23.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:24.306 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:13:24.306 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:24.310 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:13:24.310 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:28.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:28.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:33.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:33.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:38.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:38.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:43.359 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:43.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:48.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:48.353+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:53.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:53.353+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9117 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:13:54.347 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:13:54.347 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:54.351 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:13:54.351 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:58.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:13:58.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9138 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:03.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:03.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:08.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:08.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:13.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:13.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:18.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:18.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9224 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:23.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:23.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:24.450 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:14:24.450 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:24.454 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:14:24.454 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:28.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:28.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:33.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:33.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:38.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:38.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:43.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:43.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:48.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:48.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:53.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:53.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:14:54.561 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:14:54.561 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:54.565 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:14:54.565 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:58.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:14:58.376+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:03.382 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:03.376+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:08.384 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:08.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9440 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:13.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:13.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:18.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:18.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:23.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:23.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:24.689 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:15:24.689 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:24.693 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:15:24.693 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:28.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:28.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:33.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:33.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:38.393 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:38.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:43.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:43.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:48.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:48.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:53.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:53.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:15:55.050 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:15:55.050 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:55.054 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:15:55.054 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:58.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:15:58.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:03.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:03.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:08.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:08.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:13.403 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:13.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:18.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:18.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:23.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:23.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:25.145 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:16:25.145 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:25.149 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:16:25.149 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:28.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:28.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:33.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:33.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:38.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:38.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:43.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:43.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:48.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:48.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:53.415 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:53.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:16:55.190 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:16:55.190 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:55.194 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:16:55.194 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:58.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:16:58.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:03.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:03.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:08.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:08.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:13.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:13.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:18.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:18.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 9997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:23.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:23.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:25.301 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:17:25.301 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:25.305 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:17:25.305 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:28.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:28.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:33.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:33.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:38.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:38.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:43.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:43.424+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:48.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:48.424+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:53.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:53.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:17:55.450 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:17:55.450 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:55.455 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:17:55.455 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:58.435 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:17:58.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10168 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:03.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:03.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:08.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:08.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:13.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:13.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:18.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:18.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10254 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:23.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:23.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:25.562 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:18:25.563 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:25.566 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:18:25.567 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:28.443 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:28.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:33.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:33.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:38.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:38.443+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:43.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:43.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:48.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:48.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:53.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:53.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:18:55.712 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:18:55.712 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:55.715 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:18:55.716 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:58.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:18:58.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:03.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:03.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10447 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:08.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:08.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:13.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:13.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:18.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:18.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:23.461 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:23.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:25.852 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:19:25.852 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:25.856 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:19:25.856 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:28.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:28.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:33.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:33.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:38.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:38.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:43.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:43.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:48.469 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:48.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:53.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:53.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:19:55.974 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:19:55.974 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:55.986 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:19:55.986 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:58.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:19:58.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:03.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:03.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:08.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:08.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:13.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:13.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10747 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:18.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:18.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:23.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:23.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:26.110 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:20:26.110 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:26.113 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:20:26.114 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:28.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:28.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:33.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:33.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:38.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:38.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:43.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:43.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:48.487 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:48.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:53.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:53.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:20:56.421 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:20:56.422 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:56.425 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:20:56.426 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:58.490 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:20:58.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:03.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:03.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:08.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:08.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 10984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:13.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:13.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:18.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:18.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:23.499 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:23.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:26.549 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:21:26.549 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:26.553 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:21:26.553 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:28.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:28.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:33.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:33.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:38.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:38.498+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11112 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:43.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:43.498+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:48.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:48.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:53.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:53.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11176 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:21:56.626 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:21:56.626 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:56.630 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:21:56.630 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:58.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:21:58.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:03.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:03.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:08.513 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:08.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:13.515 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:13.510+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:18.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:18.510+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:23.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:23.509+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:26.757 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:22:26.757 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:26.763 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:22:26.763 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:28.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:28.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:33.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:33.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:38.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:38.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:43.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:43.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:48.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:48.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:53.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:53.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11433 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:22:56.865 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:22:56.865 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:56.869 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:22:56.869 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:58.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:22:58.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:03.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:03.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:08.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:08.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:13.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:13.528+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:18.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:18.528+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:23.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:23.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:26.986 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:23:26.986 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:26.990 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:23:26.990 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:28.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:28.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:33.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:33.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:38.543 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:38.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:43.545 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:43.540+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11648 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:48.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:48.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:53.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:53.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:23:57.143 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:23:57.143 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:57.147 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:23:57.147 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:58.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:23:58.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:03.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:03.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:08.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:08.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:13.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:13.547+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:18.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:18.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:23.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:23.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:27.200 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:24:27.200 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:27.204 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:24:27.204 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:28.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:28.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:33.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:33.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:38.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:38.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:43.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:43.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:48.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:48.562+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11928 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:53.568 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:53.562+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:24:57.311 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:24:57.312 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:57.315 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:24:57.315 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:58.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:24:58.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:03.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:03.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 11992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:08.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:08.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12014 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:13.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:13.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:18.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:18.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:23.579 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:23.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:27.471 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:25:27.472 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:27.475 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:25:27.475 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:28.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:28.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:33.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:33.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12120 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:38.584 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:38.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:43.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:43.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:48.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:48.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:53.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:53.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:25:57.672 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:25:57.672 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:57.676 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:25:57.676 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:58.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:25:58.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:03.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:03.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:08.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:08.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:13.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:13.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:18.597 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:18.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:23.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:23.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12334 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:27.792 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:26:27.792 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:27.796 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:26:27.796 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:28.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:28.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:33.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:33.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:38.606 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:38.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:43.606 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:43.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12420 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:48.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:48.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:53.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:53.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:26:57.919 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:26:57.919 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:57.923 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:26:57.924 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:58.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:26:58.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:03.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:03.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:08.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:08.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:13.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:13.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12549 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:18.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:18.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:23.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:23.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:28.066 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:27:28.066 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:28.070 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:27:28.070 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:28.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:28.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:33.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:33.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:38.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:38.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:43.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:43.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:48.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:48.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:53.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:53.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:27:58.115 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:27:58.115 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:58.125 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:27:58.125 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:58.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:27:58.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:03.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:03.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:08.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:08.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:13.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:13.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:18.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:18.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12828 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:23.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:23.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:28.255 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:28:28.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:28.259 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:28:28.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:28.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:28.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:33.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:33.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:38.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:38.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12914 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:43.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:43.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:48.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:48.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:53.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:53.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 12978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:28:58.365 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:28:58.365 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:58.369 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:28:58.369 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:58.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:28:58.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:03.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:03.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:08.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:08.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:13.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:13.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:18.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:18.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:23.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:23.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:28.460 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:29:28.460 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:28.464 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:29:28.464 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:28.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:28.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:33.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:33.659+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:38.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:38.659+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:43.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:43.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:48.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:48.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:53.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:53.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:29:58.624 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:29:58.624 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:58.628 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:29:58.628 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:58.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:29:58.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:03.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:03.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:08.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:08.671+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:13.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:13.675+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:18.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:18.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:23.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:23.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:28.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:28.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13388 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:28.989 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:30:28.990 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:28.994 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:30:28.994 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:33.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:33.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:38.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:38.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:43.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:43.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:48.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:48.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:53.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:53.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:58.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:30:58.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:30:59.105 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:30:59.105 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:59.109 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:30:59.109 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:03.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:03.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:08.699 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:08.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:13.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:13.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:18.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:18.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:23.705 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:23.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:28.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:28.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:29.220 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:31:29.220 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:29.224 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:31:29.224 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:33.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:33.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:38.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:38.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:43.713 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:43.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:48.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:48.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:53.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:53.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:58.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:31:58.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:31:59.351 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:31:59.351 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:59.355 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:31:59.355 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:03.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:03.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13794 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:08.722 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:08.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13816 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:13.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:13.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:18.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:18.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:23.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:23.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:28.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:28.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:29.459 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:32:29.459 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:29.463 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:32:29.463 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:33.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:33.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:38.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:38.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:43.735 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:43.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:48.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:48.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 13988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:53.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:53.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:58.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:32:58.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:32:59.781 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:32:59.781 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:59.785 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:32:59.786 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:03.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:03.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:08.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:08.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:13.747 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:13.742+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14093 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:18.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:18.742+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:23.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:23.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:28.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:28.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:29.830 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:33:29.830 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:29.834 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:33:29.834 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:33.754 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:33.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:38.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:38.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:43.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:43.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:48.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:48.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14246 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:53.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:53.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:58.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:33:58.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:33:59.926 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:33:59.927 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:59.930 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:33:59.930 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:03.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:03.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:08.767 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:08.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:13.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:13.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:18.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:18.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:23.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:23.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:28.774 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:28.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:30.004 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:34:30.005 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:30.008 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:34:30.008 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:33.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:33.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:38.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:38.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:43.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:43.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:48.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:48.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:53.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:53.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:34:58.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:34:58.780+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:00.125 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:35:00.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:00.129 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:35:00.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:03.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:03.780+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:08.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:08.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:13.792 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:13.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:18.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:18.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:23.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:23.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:28.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:28.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:30.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:35:30.263 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:30.269 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:35:30.269 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:33.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:33.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:38.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:38.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:43.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:43.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:48.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:48.799+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:53.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:53.799+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14780 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:35:58.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:35:58.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:00.580 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:36:00.580 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:00.585 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:36:00.585 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:03.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:03.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:08.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:08.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:13.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:13.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:18.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:18.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:23.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:23.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:28.820 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:28.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:30.692 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:36:30.692 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:30.695 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:36:30.696 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:33.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:33.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:38.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:38.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:43.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:43.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 14994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:48.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:48.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:53.830 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:53.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:36:58.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:36:58.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:00.810 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:37:00.811 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:00.814 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:37:00.815 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:03.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:03.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:08.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:08.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:13.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:13.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:18.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:18.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15146 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:23.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:23.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:28.844 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:28.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:30.938 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:37:30.938 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:30.942 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:37:30.942 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:33.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:33.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:38.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:38.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:43.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:43.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:48.851 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:48.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:53.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:53.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:37:58.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:37:58.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:00.990 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:38:00.991 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:00.995 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:38:00.995 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:03.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:03.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:08.859 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:08.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:13.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:13.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:18.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:18.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:23.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:23.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15424 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:28.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:28.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:31.114 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:38:31.114 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:31.118 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:38:31.119 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:33.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:33.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:38.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:38.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:43.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:43.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:48.875 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:48.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:53.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:53.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15553 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:38:58.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:38:58.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:01.228 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:39:01.228 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:01.232 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:39:01.232 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:03.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:03.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:08.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:08.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:13.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:13.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:18.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:18.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15660 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:23.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:23.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:28.891 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:28.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:31.367 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:39:31.367 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:31.371 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:39:31.371 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:33.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:33.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:38.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:38.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:43.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:43.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:48.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:48.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:53.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:53.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15810 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:39:58.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:39:58.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15832 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:01.432 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:40:01.432 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:01.436 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:40:01.436 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:03.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:03.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:08.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:08.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:13.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:13.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:18.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:18.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:23.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:23.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:28.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:28.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:31.648 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:40:31.648 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:31.652 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:40:31.652 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:33.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:33.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 15981 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:38.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:38.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:43.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:43.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:48.923 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:48.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:53.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:53.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:40:58.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:40:58.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16090 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:01.868 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:41:01.868 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:01.872 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:41:01.872 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:03.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:03.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:08.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:08.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:13.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:13.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:18.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:18.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:23.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:23.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:28.939 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:28.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:32.002 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:41:32.003 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:32.009 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:41:32.009 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:33.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:33.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:38.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:38.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16261 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:43.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:43.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:48.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:48.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:53.949 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:53.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:41:58.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:41:58.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:02.114 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:42:02.114 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:02.118 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:42:02.118 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:03.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:03.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:08.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:08.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:13.957 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:13.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:18.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:18.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16432 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:23.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:23.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:28.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:28.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:32.224 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:42:32.225 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:32.228 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:42:32.229 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:33.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:33.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:38.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:38.962+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16518 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:43.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:43.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:48.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:48.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:53.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:53.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:42:58.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:42:58.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:02.308 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:43:02.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:02.312 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:43:02.313 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:03.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:03.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:08.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:08.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16648 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:13.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:13.977+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:18.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:18.977+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:23.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:23.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:28.988 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:28.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:32.435 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:43:32.435 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:32.439 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:43:32.439 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:33.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:33.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:38.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:38.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16776 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:43.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:43.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:48.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:48.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:53.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:53.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:43:59.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:43:58.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16862 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:02.584 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:44:02.584 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:02.588 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:44:02.589 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:04.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:03.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:09.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:08.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:14.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:13.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16925 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:19.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:19.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16948 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:24.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:24.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:29.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:29.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 16992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:32.640 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:44:32.640 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:32.644 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:44:32.644 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:34.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:34.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:39.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:39.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:44.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:44.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17054 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:49.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:49.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:54.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:54.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:44:59.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:44:59.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:02.775 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:45:02.775 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:02.778 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:45:02.779 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:04.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:04.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:09.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:09.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:14.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:14.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:19.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:19.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17205 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:24.038 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:24.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:29.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:29.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:32.876 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:45:32.876 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:32.880 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:45:32.880 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:34.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:34.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:39.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:39.038+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:44.046 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:44.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:49.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:49.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17334 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:54.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:54.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:45:59.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:45:59.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17376 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:02.943 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:46:02.943 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:02.946 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:46:02.946 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:04.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:04.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17398 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:09.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:09.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17420 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:14.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:14.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:19.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:19.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:24.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:24.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:29.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:29.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:33.075 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:46:33.075 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:33.078 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:46:33.079 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:34.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:34.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:39.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:39.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:44.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:44.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:49.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:49.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:54.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:54.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:46:59.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:46:59.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:03.193 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:47:03.193 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:03.197 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:47:03.197 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:04.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:04.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:09.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:09.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:14.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:14.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:19.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:19.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:24.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:24.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:29.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:29.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:33.382 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:47:33.382 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:33.386 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:47:33.386 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:34.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:34.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:39.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:39.091+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17806 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:44.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:44.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:49.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:49.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:54.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:54.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:47:59.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:47:59.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:03.488 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:48:03.488 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:03.493 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:48:03.493 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:04.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:04.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:09.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:09.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:14.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:14.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:19.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:19.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:24.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:24.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 17998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:29.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:29.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:33.591 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:48:33.591 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:33.595 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:48:33.595 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:34.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:34.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:39.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:39.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18063 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:44.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:44.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18084 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:49.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:49.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:54.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:54.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:48:59.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:48:59.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:03.644 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:49:03.644 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:03.648 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:49:03.648 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:04.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:04.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:09.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:09.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:14.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:14.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:19.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:19.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:24.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:24.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:29.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:29.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:33.738 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:49:33.738 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:33.742 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:49:33.742 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:34.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:34.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:39.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:39.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:44.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:44.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:49.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:49.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:54.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:54.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:49:59.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:49:59.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:03.841 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:50:03.841 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:03.845 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:50:03.845 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:04.161 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:04.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:09.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:09.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:14.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:14.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:19.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:19.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18492 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:24.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:24.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:29.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:29.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:34.012 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:50:34.012 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:34.016 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:50:34.016 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:34.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:34.171+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:39.178 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:39.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:44.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:44.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:49.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:49.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:54.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:54.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:50:59.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:50:59.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18664 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:04.144 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:51:04.144 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:04.148 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:51:04.148 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:04.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:04.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:09.191 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:09.186+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:14.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:14.186+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:19.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:19.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:24.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:24.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18771 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:29.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:29.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18794 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:34.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:34.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:34.274 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:51:34.274 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:34.278 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:51:34.279 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:39.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:39.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:44.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:44.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:49.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:49.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:54.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:54.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:51:59.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:51:59.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:04.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:04.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:04.383 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:52:04.383 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:04.387 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:52:04.387 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:09.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:09.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:14.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:14.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 18985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:19.223 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:19.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:24.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:24.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:29.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:29.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:34.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:34.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:34.505 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:52:34.505 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:34.509 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:52:34.509 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:39.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:39.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:44.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:44.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:49.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:49.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:54.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:54.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:52:59.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:52:59.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:04.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:04.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:04.632 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:53:04.632 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:04.636 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:53:04.636 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:09.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:09.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:14.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:14.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:19.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:19.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19264 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:24.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:24.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:29.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:29.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:34.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:34.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:34.684 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:53:34.685 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:34.688 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:53:34.689 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:39.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:39.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:44.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:44.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:49.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:49.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:54.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:54.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:53:59.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:53:59.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:04.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:04.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:04.741 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:54:04.741 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:04.745 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:54:04.745 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:09.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:09.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:14.275 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:14.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:19.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:19.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:24.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:24.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:29.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:29.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:34.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:34.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:34.866 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:54:34.867 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:34.870 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:54:34.870 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:39.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:39.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:44.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:44.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19629 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:49.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:49.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:54.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:54.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:54:59.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:54:59.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:04.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:04.293+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:04.986 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:55:04.986 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:04.989 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:55:04.990 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:09.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:09.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:14.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:14.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:19.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:19.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:24.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:24.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:29.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:29.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:34.313 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:34.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:35.114 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:55:35.114 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:35.118 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:55:35.118 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:39.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:39.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:44.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:44.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:49.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:49.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:54.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:54.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:55:59.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:55:59.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:04.328 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:04.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:05.183 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:56:05.183 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:05.187 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:56:05.187 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:09.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:09.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 19994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:14.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:14.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:19.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:19.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:24.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:24.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:29.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:29.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:34.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:34.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:35.294 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:56:35.294 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:35.298 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:56:35.298 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:39.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:39.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:44.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:44.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:49.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:49.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:54.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:54.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:56:59.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:56:59.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:04.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:04.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:05.421 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:57:05.422 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:05.425 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:57:05.425 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:09.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:09.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:14.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:14.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:19.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:19.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:24.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:24.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:29.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:29.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:34.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:34.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:35.533 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:57:35.534 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:35.538 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:57:35.538 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:39.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:39.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:44.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:44.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20401 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:49.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:49.373+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20424 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:54.382 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:54.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:57:59.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:57:59.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:04.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:04.380+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:05.652 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:58:05.652 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:05.656 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:58:05.657 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:09.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:09.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:14.393 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:14.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:19.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:19.388+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:24.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:24.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:29.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:29.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:34.402 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:34.396+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:35.755 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:58:35.756 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:35.759 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:58:35.760 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:39.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:39.400+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:44.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:44.400+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:49.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:49.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:54.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:54.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:58:59.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:58:59.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:04.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:04.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:06.006 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:59:06.007 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:06.010 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:59:06.011 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:09.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:09.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:14.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:14.415+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:19.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:19.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20809 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:24.426 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:24.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:29.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:29.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:34.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:34.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:36.146 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T06:59:36.146 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:36.150 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T06:59:36.150 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:39.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:39.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:44.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:44.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:49.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:49.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:54.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:54.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T06:59:59.443 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T06:59:59.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 20980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:04.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:04.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:06.245 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:00:06.245 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:06.249 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:00:06.249 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:09.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:09.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:14.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:14.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:19.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:19.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:24.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:24.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21089 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:29.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:29.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:34.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:34.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:36.380 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:00:36.381 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:36.384 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:00:36.385 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:39.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:39.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:44.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:44.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:49.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:49.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:54.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:54.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:00:59.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:00:59.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:04.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:04.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:06.612 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:01:06.612 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:06.616 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:01:06.617 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:09.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:09.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:14.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:14.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21301 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:19.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:19.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:24.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:24.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:29.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:29.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:34.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:34.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:36.730 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:01:36.731 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:36.734 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:01:36.735 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:39.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:39.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:44.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:44.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21431 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:49.499 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:49.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:54.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:54.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:01:59.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:01:59.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21496 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:04.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:04.499+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:06.835 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:02:06.836 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:06.839 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:02:06.839 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:09.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:09.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:14.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:14.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:19.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:19.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:24.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:24.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:29.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:29.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:34.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:34.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:36.960 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:02:36.960 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:36.964 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:02:36.964 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:39.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:39.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:44.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:44.522+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:49.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:49.522+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:54.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:54.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:02:59.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:02:59.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:04.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:04.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:07.082 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:03:07.082 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:07.086 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:03:07.086 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:09.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:09.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:14.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:14.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:19.543 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:19.537+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:24.545 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:24.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:29.548 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:29.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:34.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:34.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:37.222 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:03:37.222 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:37.226 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:03:37.226 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:39.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:39.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21925 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:44.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:44.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:49.559 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:49.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:54.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:54.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 21989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:03:59.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:03:59.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:04.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:04.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:07.336 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:04:07.336 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:07.339 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:04:07.340 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:09.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:09.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22054 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:14.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:14.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:19.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:19.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:24.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:24.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:29.579 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:29.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:34.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:34.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:37.443 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:04:37.443 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:37.447 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:04:37.448 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:39.584 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:39.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:44.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:44.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:49.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:49.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:54.592 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:54.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:04:59.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:04:59.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:04.597 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:04.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:07.582 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:05:07.582 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:07.586 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:05:07.586 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:09.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:09.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:14.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:14.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:19.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:19.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:24.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:24.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:29.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:29.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22398 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:34.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:34.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:37.940 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:05:37.940 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:37.944 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:05:37.944 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:39.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:39.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:44.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:44.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:49.621 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:49.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:54.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:54.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22503 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:05:59.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:05:59.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:04.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:04.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:08.063 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:06:08.063 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:08.067 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:06:08.067 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:09.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:09.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:14.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:14.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:19.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:19.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:24.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:24.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:29.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:29.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:34.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:34.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:38.192 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:06:38.192 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:38.197 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:06:38.197 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:39.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:39.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:44.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:44.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:49.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:49.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:54.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:54.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:06:59.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:06:59.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:04.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:04.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:08.311 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:07:08.312 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:08.316 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:07:08.316 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:09.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:09.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:14.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:14.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22847 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:19.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:19.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:24.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:24.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:29.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:29.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:34.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:34.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:38.439 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:07:38.439 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:38.443 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:07:38.443 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:39.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:39.671+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:44.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:44.675+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:49.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:49.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 22998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:54.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:54.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:07:59.691 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:07:59.683+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:04.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:04.687+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:08.517 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:08:08.517 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:08.527 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:08:08.528 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:09.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:09.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23084 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:14.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:14.695+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:19.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:19.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:24.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:24.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:29.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:29.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:34.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:34.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:38.664 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:08:38.664 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:38.668 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:08:38.668 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:39.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:39.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:44.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:44.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:49.717 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:49.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:54.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:54.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:08:59.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:08:59.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:04.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:04.717+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:08.764 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:09:08.765 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:08.768 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:09:08.769 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:09.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:09.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:14.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:14.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:19.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:19.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:24.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:24.729+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:29.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:29.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:34.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:34.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23447 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:38.900 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:09:38.900 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:38.904 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:09:38.905 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:39.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:39.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:44.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:44.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:49.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:49.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:54.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:54.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23533 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:09:59.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:09:59.748+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:04.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:04.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:09.019 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:10:09.020 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:09.025 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:10:09.025 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:09.762 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:09.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:14.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:14.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:19.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:19.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:24.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:24.764+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:29.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:29.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:34.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:34.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:39.130 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:10:39.131 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:39.134 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:10:39.134 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:39.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:39.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:44.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:44.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23749 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:49.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:49.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:54.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:54.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:10:59.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:10:59.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:04.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:04.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:09.187 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:11:09.187 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:09.191 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:11:09.191 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:09.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:09.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:14.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:14.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:19.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:19.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:24.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:24.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23920 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:29.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:29.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:34.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:34.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:39.304 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:11:39.304 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:39.308 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:11:39.308 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:39.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:39.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 23985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:44.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:44.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:49.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:49.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:54.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:54.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24047 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:11:59.820 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:11:59.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:04.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:04.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:09.406 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:12:09.406 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:09.410 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:12:09.410 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:09.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:09.821+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:14.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:14.821+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:19.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:19.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:24.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:24.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:29.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:29.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:34.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:34.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:39.556 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:12:39.557 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:39.560 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:12:39.560 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:39.844 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:39.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:44.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:44.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:49.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:49.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:54.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:54.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:12:59.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:12:59.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:04.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:04.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:09.687 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:13:09.687 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:09.693 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:13:09.693 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:09.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:09.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:14.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:14.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:19.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:19.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:24.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:24.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:29.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:29.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:34.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:34.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:39.765 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:13:39.765 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:39.768 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:13:39.769 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:39.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:39.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:44.876 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:44.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:49.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:49.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:54.882 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:54.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:13:59.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:13:59.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:04.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:04.883+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:09.885 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:14:09.885 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:09.889 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:14:09.889 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:09.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:09.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24628 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:14.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:14.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:19.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:19.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:24.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:24.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:29.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:29.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24714 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:34.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:34.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:39.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:39.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:40.187 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:14:40.187 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:40.191 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:14:40.191 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:44.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:44.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24778 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:49.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:49.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:54.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:54.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:14:59.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:14:59.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:04.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:04.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:09.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:09.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:10.306 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:15:10.306 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:10.310 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:15:10.310 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:14.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:14.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:19.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:19.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:24.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:24.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:29.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:29.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:34.938 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:34.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 24991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:39.939 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:39.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:40.427 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:15:40.427 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:40.431 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:15:40.431 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:44.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:44.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:49.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:49.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:54.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:54.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:15:59.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:15:59.944+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:04.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:04.944+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:09.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:09.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:10.482 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:16:10.482 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:10.486 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:16:10.486 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:14.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:14.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:19.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:19.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:24.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:24.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:29.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:29.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:34.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:34.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:39.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:39.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:40.594 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:16:40.595 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:40.598 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:16:40.598 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:44.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:44.967+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:49.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:49.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:54.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:54.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:16:59.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:16:59.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:04.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:04.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25379 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:09.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:09.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25401 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:10.742 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:17:10.743 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:10.746 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:17:10.746 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:14.993 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:14.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:19.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:19.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:24.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:24.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:30.001 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:29.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:35.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:34.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:40.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:40.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:40.873 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:17:40.873 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:40.877 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:17:40.877 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:45.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:45.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:50.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:50.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:17:55.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:17:55.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:00.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:00.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:05.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:05.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:10.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:10.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:11.014 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:18:11.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:11.018 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:18:11.018 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:15.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:15.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:20.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:20.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:25.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:25.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:30.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:30.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25744 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:35.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:35.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:40.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:40.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:41.129 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:18:41.129 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:41.133 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:18:41.133 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:45.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:45.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:50.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:50.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:18:55.051 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:18:55.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:00.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:00.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:05.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:05.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:10.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:10.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:11.232 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:19:11.232 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:11.236 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:19:11.237 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:15.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:15.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:20.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:20.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:25.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:25.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 25979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:30.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:30.063+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:35.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:35.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:40.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:40.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:41.343 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:19:41.343 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:41.347 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:19:41.347 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:45.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:45.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26065 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:50.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:50.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:19:55.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:19:55.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:00.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:00.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:05.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:05.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:10.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:10.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:11.450 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:20:11.450 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:11.455 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:20:11.455 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:15.096 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:15.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:20.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:20.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:25.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:25.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:30.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:30.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:35.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:35.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:40.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:40.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:41.531 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:20:41.531 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:41.535 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:20:41.535 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:45.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:45.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:50.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:50.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:20:55.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:20:55.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:00.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:00.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:05.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:05.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:10.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:10.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:11.593 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:21:11.593 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:11.597 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:21:11.597 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:15.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:15.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:20.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:20.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:25.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:25.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:30.140 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:30.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:35.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:35.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26537 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:40.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:40.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:41.721 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:21:41.721 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:41.725 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:21:41.725 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:45.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:45.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:50.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:50.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:21:55.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:21:55.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:00.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:00.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:05.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:05.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:10.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:10.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:11.849 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:22:12.102 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:12.106 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:22:12.107 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:15.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:15.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:20.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:20.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:25.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:25.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:30.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:30.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:35.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:35.171+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26793 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:40.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:40.171+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:42.146 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:22:42.146 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:42.150 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:22:42.150 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:45.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:45.179+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:50.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:50.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:22:55.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:22:55.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:00.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:00.186+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:05.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:05.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:10.197 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:10.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26946 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:12.279 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:23:12.279 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:12.283 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:23:12.283 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:15.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:15.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:20.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:20.198+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 26989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:25.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:25.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:30.208 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:30.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:35.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:35.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:40.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:40.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:42.382 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:23:42.383 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:42.386 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:23:42.387 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:45.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:45.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:50.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:50.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:23:55.223 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:23:55.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:00.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:00.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:05.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:05.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:10.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:10.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:12.492 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:24:12.492 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:12.496 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:24:12.496 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:15.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:15.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:20.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:20.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:25.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:25.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:30.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:30.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:35.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:35.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:40.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:40.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:42.613 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:24:42.613 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:42.616 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:24:42.616 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:45.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:45.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:50.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:50.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:24:55.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:24:55.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:00.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:00.256+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:05.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:05.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:10.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:10.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:12.769 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:25:12.769 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:12.773 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:25:12.773 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:15.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:15.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:20.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:20.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:25.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:25.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:30.280 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:30.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:35.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:35.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:40.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:40.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:43.008 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:25:43.009 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:43.013 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:25:43.013 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:45.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:45.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:50.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:50.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:25:55.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:25:55.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:00.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:00.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 27984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:05.302 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:05.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28017 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:10.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:10.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28049 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:13.124 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:26:13.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:13.128 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:26:13.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:15.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:15.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:20.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:20.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:25.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:25.310+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:30.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:30.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:35.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:35.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:40.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:40.317+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:43.224 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:26:43.224 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:43.228 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:26:43.228 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:45.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:45.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:50.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:50.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:26:55.332 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:26:55.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:00.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:00.329+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:05.339 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:05.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:10.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:10.337+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:13.361 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:27:13.361 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:13.365 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:27:13.365 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:15.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:15.337+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:20.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:20.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:25.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:25.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:30.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:30.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28576 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:35.357 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:35.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:40.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:40.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28643 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:43.472 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:27:43.472 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:43.476 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:27:43.476 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:45.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:45.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28676 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:50.369 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:50.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:27:55.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:27:55.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:00.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:00.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:05.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:05.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:10.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:10.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:13.606 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:28:13.606 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:13.610 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:28:13.610 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:15.384 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:15.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:20.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:20.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:25.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:25.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:30.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:30.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 28971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:35.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:35.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:40.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:40.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:43.662 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:28:43.662 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:43.666 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:28:43.666 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:45.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:45.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:50.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:50.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:28:55.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:28:55.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:00.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:00.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:05.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:05.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:10.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:10.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:13.794 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:29:13.794 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:13.798 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:29:13.798 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:15.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:15.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:20.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:20.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:25.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:25.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:30.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:30.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29367 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:35.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:35.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:40.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:40.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:43.908 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:29:43.908 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:43.912 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:29:43.912 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:45.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:45.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:50.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:50.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:29:55.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:29:55.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:00.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:00.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:05.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:05.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:10.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:10.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:14.039 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:30:14.039 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:14.043 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:30:14.043 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:15.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:15.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:20.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:20.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:25.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:25.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:30.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:30.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:35.472 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:35.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:40.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:40.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:44.146 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:30:44.146 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:44.150 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:30:44.150 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:45.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:45.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29862 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:50.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:50.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:30:55.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:30:55.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:00.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:00.479+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:05.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:05.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 29994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:10.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:10.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:14.269 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:31:14.270 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:14.273 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:31:14.273 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:15.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:15.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:20.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:20.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30093 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:25.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:25.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:30.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:30.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:35.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:35.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30191 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:40.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:40.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30224 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:44.416 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:31:44.417 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:44.420 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:31:44.421 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:45.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:45.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:50.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:50.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:31:55.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:31:55.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:00.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:00.522+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:05.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:05.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:10.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:10.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:14.632 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:32:14.633 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:14.636 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:32:14.636 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:15.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:15.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:20.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:20.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:25.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:25.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:30.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:30.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30553 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:35.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:35.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:40.554 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:40.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:44.680 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:32:44.680 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:44.684 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:32:44.684 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:45.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:45.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:50.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:50.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:32:55.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:32:55.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:00.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:00.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:05.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:05.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:10.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:10.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:14.779 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:33:14.779 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:14.783 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:33:14.783 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:15.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:15.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:20.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:20.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:25.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:25.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:30.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:30.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:35.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:35.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 30981 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:40.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:40.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:44.906 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:33:44.906 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:44.910 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:33:44.910 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:45.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:45.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:50.597 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:50.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:33:55.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:33:55.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:00.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:00.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:05.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:05.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:10.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:10.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:15.017 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:34:15.017 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:15.021 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:34:15.022 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:15.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:15.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:20.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:20.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:25.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:25.615+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:30.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:30.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:35.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:35.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:40.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:40.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:45.164 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:34:45.164 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:45.168 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:34:45.168 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:45.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:45.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:50.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:50.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:34:55.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:34:55.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:00.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:00.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:05.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:05.642+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31576 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:10.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:10.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:15.272 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:35:15.272 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:15.276 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:35:15.276 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:15.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:15.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:20.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:20.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:25.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:25.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:30.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:30.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:35.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:35.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:40.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:40.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31806 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:45.401 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:35:45.401 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:45.405 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:35:45.405 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:45.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:45.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:50.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:50.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:35:55.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:35:55.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:00.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:00.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:05.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:05.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 31972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:10.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:10.684+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:15.541 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:36:15.542 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:15.546 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:36:15.546 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:15.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:15.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:20.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:20.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:25.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:25.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:30.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:30.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:35.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:35.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:40.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:40.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:45.672 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:36:45.672 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:45.676 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:36:45.676 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:45.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:45.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:50.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:50.711+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32266 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:36:55.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:36:55.715+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:00.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:00.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:05.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:05.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:10.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:10.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:15.735 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:15.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:15.771 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:37:15.771 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:15.775 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:37:15.775 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:20.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:20.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:25.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:25.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:30.747 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:30.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:35.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:35.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:40.753 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:40.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:45.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:45.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:45.827 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:37:45.827 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:45.831 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:37:45.831 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:50.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:50.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:37:55.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:37:55.758+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32682 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:00.767 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:00.762+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:05.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:05.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:10.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:10.770+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:15.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:15.774+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:16.124 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:38:16.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:16.128 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:38:16.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:20.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:20.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:25.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:25.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:30.787 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:30.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:35.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:35.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:40.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:40.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:45.797 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:45.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 32997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:46.309 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:38:46.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:46.313 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:38:46.313 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:50.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:50.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:38:55.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:38:55.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:00.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:00.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:05.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:05.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:10.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:10.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33154 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:15.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:15.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:16.381 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:39:16.381 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:16.385 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:39:16.385 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:20.820 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:20.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:25.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:25.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:30.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:30.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:35.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:35.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:40.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:40.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:45.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:45.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:46.497 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:39:46.497 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:46.501 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:39:46.501 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:50.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:50.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:39:55.843 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:39:55.835+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:00.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:00.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:05.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:05.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:10.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:10.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:15.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:15.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:16.625 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:40:16.625 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:16.629 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:40:16.629 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:20.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:20.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:25.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:25.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:30.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:30.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:35.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:35.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:40.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:40.870+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:45.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:45.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:46.955 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:40:46.956 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:46.959 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:40:46.960 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:50.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:50.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:40:55.885 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:40:55.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:00.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:00.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33848 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:05.891 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:05.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:10.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:10.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33910 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:15.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:15.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:17.089 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:41:17.090 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:17.094 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:41:17.094 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:20.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:20.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 33974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:25.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:25.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:30.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:30.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:35.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:35.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:40.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:40.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:45.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:45.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:47.225 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:41:47.225 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:47.237 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:41:47.237 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:50.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:50.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:41:55.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:41:55.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:00.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:00.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:05.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:05.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:10.938 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:10.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:15.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:15.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:17.479 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:42:17.479 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:17.483 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:42:17.483 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:20.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:20.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:25.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:25.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:30.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:30.944+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:35.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:35.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:40.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:40.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:45.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:45.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:47.573 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:42:47.573 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:47.577 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:42:47.577 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:50.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:50.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34538 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:42:55.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:42:55.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:00.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:00.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:05.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:05.967+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:10.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:10.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:15.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:15.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:17.701 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:43:17.701 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:17.705 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:43:17.705 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:20.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:20.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34726 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:25.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:25.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:30.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:30.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:35.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:35.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:41.001 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:40.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:46.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:46.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:47.826 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:43:47.827 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:47.830 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:43:47.830 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:51.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:51.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:43:56.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:43:56.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:01.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:01.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 34974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:06.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:06.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:11.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:11.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:16.027 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:16.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:17.975 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:44:17.975 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:17.979 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:44:17.979 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:21.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:21.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:26.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:26.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:31.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:31.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:36.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:36.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:41.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:41.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:46.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:46.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:48.077 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:44:48.077 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:48.080 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:44:48.081 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:51.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:51.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:44:56.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:44:56.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:01.060 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:01.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:06.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:06.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:11.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:11.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:16.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:16.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:18.237 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:45:18.238 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:18.242 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:45:18.242 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:21.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:21.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:26.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:26.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:31.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:31.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:36.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:36.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:41.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:41.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:46.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:46.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:48.377 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:45:48.377 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:48.381 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:45:48.381 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:51.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:51.091+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:45:56.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:45:56.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:01.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:01.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:06.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:06.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:11.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:11.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:16.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:16.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:18.719 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:46:18.719 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:18.723 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:46:18.723 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:21.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:21.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:26.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:26.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:31.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:31.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:36.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:36.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:41.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:41.134+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:46.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:46.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 35994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:48.846 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:46:48.846 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:48.850 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:46:48.850 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:51.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:51.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36025 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:46:56.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:46:56.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:01.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:01.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:06.156 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:06.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36117 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:11.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:11.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:16.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:16.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:18.886 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:47:18.886 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:18.890 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:47:18.890 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:21.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:21.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:26.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:26.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36241 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:31.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:31.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:36.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:36.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:41.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:41.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36336 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:46.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:46.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:49.027 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:47:49.027 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:49.035 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:47:49.035 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:51.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:51.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36397 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:47:56.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:47:56.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:01.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:01.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:06.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:06.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:11.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:11.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:16.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:16.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:19.129 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:48:19.129 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:19.134 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:48:19.134 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:21.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:21.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:26.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:26.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:31.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:31.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:36.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:36.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:41.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:41.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:46.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:46.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:49.274 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:48:49.274 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:49.278 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:48:49.278 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:51.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:51.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:48:56.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:48:56.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:01.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:01.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36832 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:06.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:06.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36862 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:11.248 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:11.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:16.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:16.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36925 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:19.376 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:49:19.377 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:19.380 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:49:19.381 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:21.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:21.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:26.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:26.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 36988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:31.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:31.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:36.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:36.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:41.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:41.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:46.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:46.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:49.444 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:49:49.444 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:49.449 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:49:49.449 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:51.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:51.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:49:56.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:49:56.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:01.285 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:01.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:06.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:06.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:11.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:11.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:16.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:16.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:19.500 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:50:19.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:19.504 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:50:19.504 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:21.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:21.293+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:26.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:26.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:31.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:31.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37397 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:36.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:36.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:41.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:41.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:46.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:46.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:49.558 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:50:49.558 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:49.562 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:50:49.562 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:51.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:51.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:50:56.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:50:56.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:01.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:01.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:06.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:06.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:11.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:11.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:16.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:16.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:19.613 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:51:19.613 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:19.617 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:51:19.617 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:21.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:21.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:26.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:26.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:31.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:31.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:36.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:36.347+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:41.359 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:41.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:46.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:46.355+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:49.747 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:51:49.747 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:49.751 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:51:49.751 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:51.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:51.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:51:56.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:51:56.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:01.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:01.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:06.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:06.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 37994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:11.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:11.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:16.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:16.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:19.845 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:52:19.845 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:19.849 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:52:19.849 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:21.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:21.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:26.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:26.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38120 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:31.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:31.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:36.402 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:36.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:41.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:41.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:46.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:46.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38246 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:49.893 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:52:49.893 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:49.897 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:52:49.897 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:51.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:51.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:52:56.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:52:56.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:01.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:01.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:06.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:06.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:11.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:11.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:16.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:16.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:20.033 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:53:20.033 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:20.036 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:53:20.037 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:21.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:21.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:26.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:26.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:31.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:31.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:36.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:36.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:41.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:41.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:46.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:46.448+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:50.097 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:53:50.098 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:50.101 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:53:50.101 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:51.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:51.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:53:56.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:53:56.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:01.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:01.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:06.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:06.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:11.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:11.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:16.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:16.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:20.153 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:54:20.154 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:20.157 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:54:20.158 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:21.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:21.475+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:26.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:26.479+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:31.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:31.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:36.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:36.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:41.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:41.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 38971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:46.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:46.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:50.291 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:54:50.291 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:50.295 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:54:50.295 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:51.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:51.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:54:56.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:54:56.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39065 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:01.515 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:01.510+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:06.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:06.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:11.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:11.522+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:16.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:16.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:20.375 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:55:20.376 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:20.379 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:55:20.379 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:21.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:21.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:26.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:26.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:31.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:31.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:36.543 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:36.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:41.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:41.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:46.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:46.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:50.457 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:55:50.458 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:50.461 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:55:50.462 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:51.554 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:51.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:55:56.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:55:56.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39442 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:01.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:01.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:06.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:06.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:11.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:11.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:16.573 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:16.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:20.579 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:56:20.580 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:20.583 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:56:20.584 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:21.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:21.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:26.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:26.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:31.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:31.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39662 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:36.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:36.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:41.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:41.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:46.597 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:46.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:50.670 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:56:50.670 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:50.674 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:56:50.674 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:51.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:51.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:56:56.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:56:56.600+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:01.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:01.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:06.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:06.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39881 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:11.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:11.612+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:16.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:16.615+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:20.726 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:57:20.726 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:20.730 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:57:20.730 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:21.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:21.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 39976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:26.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:26.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:31.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:31.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:36.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:36.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:41.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:41.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:46.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:46.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:50.782 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:57:50.782 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:50.786 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:57:50.786 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:51.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:51.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:57:56.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:57:56.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:01.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:01.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:06.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:06.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:11.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:11.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:16.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:16.662+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:20.839 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:58:20.840 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:20.843 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:58:20.844 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:21.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:21.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:26.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:26.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:31.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:31.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:36.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:36.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:41.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:41.681+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:46.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:46.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:50.895 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:58:50.896 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:50.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:58:50.899 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:51.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:51.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:58:56.699 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:58:56.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:01.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:01.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40606 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:06.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:06.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:11.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:11.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:16.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:16.709+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:21.044 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:59:21.045 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:21.048 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:59:21.048 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:21.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:21.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:26.723 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:26.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:31.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:31.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:36.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:36.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:41.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:41.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:46.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:46.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:51.109 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T07:59:51.109 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:51.113 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T07:59:51.113 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:51.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:51.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40920 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T07:59:56.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T07:59:56.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:01.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:01.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 40983 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:06.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:06.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:11.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:11.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:16.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:16.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:21.216 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:00:21.216 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:21.220 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:00:21.220 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:21.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:21.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:26.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:26.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:31.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:31.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:36.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:36.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:41.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:41.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:46.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:46.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41270 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:51.288 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:00:51.289 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:51.292 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:00:51.293 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:51.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:51.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:00:56.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:00:56.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41336 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:01.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:01.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41369 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:06.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:06.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:11.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:11.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:16.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:16.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:21.404 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:01:21.404 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:21.407 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:01:21.408 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:21.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:21.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:26.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:26.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:31.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:31.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:36.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:36.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:41.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:41.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:46.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:46.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41662 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:51.511 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:01:51.511 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:51.515 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:01:51.515 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:51.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:51.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:01:56.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:01:56.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:01.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:01.849+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:06.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:06.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41794 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:11.862 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:11.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:16.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:16.861+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:21.659 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:02:21.659 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:21.663 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:02:21.663 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:21.870 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:21.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:26.875 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:26.868+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41925 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:31.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:31.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:36.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:36.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 41992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:41.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:41.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:46.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:46.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42057 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:51.787 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:02:51.787 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:51.791 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:02:51.791 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:51.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:51.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:02:56.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:02:56.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:01.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:01.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:06.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:06.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:11.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:11.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:16.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:16.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:21.846 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:03:21.846 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:21.850 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:03:21.850 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:21.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:21.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:26.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:26.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:31.932 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:31.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:36.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:36.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42388 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:41.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:41.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:46.949 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:46.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:51.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:51.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:03:51.993 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:03:51.994 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:51.998 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:03:51.998 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:56.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:03:56.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:01.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:01.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:06.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:06.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:11.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:11.962+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:16.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:16.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:21.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:21.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:22.042 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:04:22.043 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:22.046 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:04:22.046 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:26.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:26.974+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:31.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:31.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:36.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:36.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:41.993 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:41.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:46.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:46.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:52.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:51.997+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:04:52.174 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:04:52.174 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:52.178 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:04:52.178 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:57.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:04:57.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:02.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:02.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:07.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:07.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 42981 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:12.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:12.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43017 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:17.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:17.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:22.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:22.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43084 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:22.324 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:05:22.325 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:22.328 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:05:22.329 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:27.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:27.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:32.038 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:32.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:37.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:37.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:42.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:42.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:47.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:47.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:52.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:52.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:05:52.445 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:05:52.445 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:52.449 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:05:52.449 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:57.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:05:57.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:02.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:02.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:07.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:07.063+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:12.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:12.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:17.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:17.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:22.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:22.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:22.560 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:06:22.560 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:22.564 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:06:22.564 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:27.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:27.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:32.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:32.091+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43553 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:37.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:37.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:42.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:42.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:47.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:47.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:52.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:52.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:06:52.633 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:06:52.633 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:52.637 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:06:52.638 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:57.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:06:57.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:02.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:02.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:07.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:07.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:12.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:12.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:17.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:17.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:22.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:22.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:22.762 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:07:22.762 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:22.766 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:07:22.766 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:27.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:27.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:32.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:32.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:37.147 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:37.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 43985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:42.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:42.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:47.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:47.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:52.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:52.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:07:52.887 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:07:52.887 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:52.891 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:07:52.891 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:57.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:07:57.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:02.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:02.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44154 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:07.172 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:07.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:12.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:12.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44221 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:17.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:17.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:22.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:22.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:23.027 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:08:23.027 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:23.031 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:08:23.031 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:27.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:27.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:32.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:32.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:37.197 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:37.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:42.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:42.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:47.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:47.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:52.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:52.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:08:53.066 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:08:53.066 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:53.069 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:08:53.069 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:57.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:08:57.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:02.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:02.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:07.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:07.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:12.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:12.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:17.234 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:17.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:22.239 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:22.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:23.190 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:09:23.190 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:23.193 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:09:23.194 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:27.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:27.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:32.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:32.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:37.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:37.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:42.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:42.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:47.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:47.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:52.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:52.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:09:53.318 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:09:53.318 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:53.322 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:09:53.322 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:57.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:09:57.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:02.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:02.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:07.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:07.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 44987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:12.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:12.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45025 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:17.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:17.286+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:22.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:22.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:23.435 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:10:23.435 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:23.439 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:10:23.439 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:27.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:27.293+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:32.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:32.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:37.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:37.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:42.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:42.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:47.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:47.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:52.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:52.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:10:53.532 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:10:53.532 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:53.536 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:10:53.536 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:57.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:10:57.317+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:02.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:02.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:07.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:07.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:12.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:12.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:17.339 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:17.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:22.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:22.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:23.590 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:11:23.590 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:23.594 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:11:23.594 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:27.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:27.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:32.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:32.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:37.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:37.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45591 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:42.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:42.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:47.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:47.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:52.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:52.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:11:53.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:11:53.638 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:53.642 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:11:53.642 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:57.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:11:57.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:02.384 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:02.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:07.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:07.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:12.393 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:12.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45827 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:17.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:17.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:22.403 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:22.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:23.774 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:12:23.774 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:23.778 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:12:23.778 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:27.408 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:27.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45925 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:32.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:32.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:37.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:37.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 45992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:42.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:42.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:47.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:47.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:52.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:52.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:12:53.876 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:12:53.876 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:53.879 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:12:53.880 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:57.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:12:57.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:02.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:02.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:07.443 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:07.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:12.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:12.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:17.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:17.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:22.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:22.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:23.962 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:13:23.963 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:23.966 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:13:23.966 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:27.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:27.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:32.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:32.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:37.469 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:37.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:42.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:42.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:47.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:47.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:52.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:52.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:13:54.111 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:13:54.112 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:54.115 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:13:54.115 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:57.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:13:57.481+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:02.490 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:02.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:07.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:07.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:12.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:12.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:17.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:17.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:22.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:22.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:24.252 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:14:24.252 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:24.255 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:14:24.256 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:27.513 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:27.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:32.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:32.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:37.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:37.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:42.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:42.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:47.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:47.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:52.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:52.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:14:54.354 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:14:54.355 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:54.358 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:14:54.358 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:57.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:14:57.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:02.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:02.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:07.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:07.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 46995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:12.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:12.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:17.559 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:17.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:22.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:22.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:24.493 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:15:24.493 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:24.497 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:15:24.497 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:27.568 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:27.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:32.573 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:32.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:37.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:37.574+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:42.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:42.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:47.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:47.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:52.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:52.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:15:54.760 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:15:54.760 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:54.764 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:15:54.764 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:57.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:15:57.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:02.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:02.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47360 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:07.606 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:07.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:12.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:12.605+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:17.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:17.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:22.621 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:22.617+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47492 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:25.129 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:16:25.129 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:25.133 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:16:25.133 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:27.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:27.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:32.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:32.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:37.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:37.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:42.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:42.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:47.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:47.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:52.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:52.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:16:55.175 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:16:55.175 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:55.179 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:16:55.179 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:57.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:16:57.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:02.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:02.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:07.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:07.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:12.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:12.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:17.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:17.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:22.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:22.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:25.293 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:17:25.293 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:25.296 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:17:25.297 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:27.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:27.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:32.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:32.684+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:37.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:37.687+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 47985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:42.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:42.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:47.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:47.695+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:52.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:52.699+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:17:55.413 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:17:55.413 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:55.417 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:17:55.417 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:57.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:17:57.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48117 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:02.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:02.711+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:07.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:07.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:12.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:12.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:17.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:17.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:22.734 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:22.730+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:25.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:18:25.536 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:25.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:18:25.540 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:27.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:27.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:32.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:32.738+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:37.747 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:37.742+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:42.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:42.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48415 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:47.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:47.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:52.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:52.758+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:18:55.667 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:18:55.667 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:55.671 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:18:55.671 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:57.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:18:57.762+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:02.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:02.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:07.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:07.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:12.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:12.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:17.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:17.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:22.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:22.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:25.790 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:19:25.790 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:25.794 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:19:25.794 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:27.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:27.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:32.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:32.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:37.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:37.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:42.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:42.805+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48810 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:47.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:47.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:52.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:52.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:19:55.900 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:19:55.900 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:55.904 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:19:55.904 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:57.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:19:57.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:02.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:02.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:07.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:07.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 48974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:12.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:12.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:17.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:17.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:22.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:22.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:26.027 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:20:26.028 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:26.031 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:20:26.032 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:27.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:27.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:32.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:32.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:37.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:37.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:42.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:42.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:47.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:47.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:52.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:52.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:20:56.164 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:20:56.165 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:56.168 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:20:56.168 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:57.885 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:20:57.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:02.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:02.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:07.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:07.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49369 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:12.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:12.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:17.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:17.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:18.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:18.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:18.447348+0000) 2019-10-13T08:21:19.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:19.432+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:19.433442+0000) 2019-10-13T08:21:19.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:19.868+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:19.870086+0000) 2019-10-13T08:21:21.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:21.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.688592+0000) 2019-10-13T08:21:21.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:21.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.734510+0000) 2019-10-13T08:21:21.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:21.926+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.928835+0000) 2019-10-13T08:21:22.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:22.902+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.904731+0000) 2019-10-13T08:21:22.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:22.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:22.906 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:22.902+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.905950+0000) 2019-10-13T08:21:22.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:22.902+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.906223+0000) 2019-10-13T08:21:22.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:22.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.906529+0000) 2019-10-13T08:21:22.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:22.904+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.907644+0000) 2019-10-13T08:21:22.909 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:22.906+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.908281+0000) 2019-10-13T08:21:22.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:22.902+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.907140+0000) 2019-10-13T08:21:22.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:22.908+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.911583+0000) 2019-10-13T08:21:24.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:24.954+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:24.957741+0000) 2019-10-13T08:21:26.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:26.178+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.179723+0000) 2019-10-13T08:21:26.325 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:21:26.326 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:26.329 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:21:26.329 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:27.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:27.906+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.909426+0000) 2019-10-13T08:21:27.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:27.906+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.909779+0000) 2019-10-13T08:21:27.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:27.906+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.910154+0000) 2019-10-13T08:21:27.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:27.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:27.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:27.910+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.911603+0000) 2019-10-13T08:21:27.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:27.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.912148+0000) 2019-10-13T08:21:27.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:27.908+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.913287+0000) 2019-10-13T08:21:27.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:27.910+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.913460+0000) 2019-10-13T08:21:27.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:27.916+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.919108+0000) 2019-10-13T08:21:28.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:28.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:28.447545+0000) 2019-10-13T08:21:29.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:29.428+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:29.433641+0000) 2019-10-13T08:21:29.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:29.872+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:29.876761+0000) 2019-10-13T08:21:31.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:31.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.688790+0000) 2019-10-13T08:21:31.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:31.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.734733+0000) 2019-10-13T08:21:31.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:31.926+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.929067+0000) 2019-10-13T08:21:32.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:32.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.913697+0000) 2019-10-13T08:21:32.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:32.910+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.914781+0000) 2019-10-13T08:21:32.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:32.910+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49537 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:32.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:32.910+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.915101+0000) 2019-10-13T08:21:32.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:32.914+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.915996+0000) 2019-10-13T08:21:32.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:32.914+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.916509+0000) 2019-10-13T08:21:32.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:32.914+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.917815+0000) 2019-10-13T08:21:32.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:32.912+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.918034+0000) 2019-10-13T08:21:32.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:32.916+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.921474+0000) 2019-10-13T08:21:34.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:34.954+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:34.957974+0000) 2019-10-13T08:21:36.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:36.178+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.179918+0000) 2019-10-13T08:21:37.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:37.914+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.918017+0000) 2019-10-13T08:21:37.920 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:37.914+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.919089+0000) 2019-10-13T08:21:37.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:37.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:37.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:37.918+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.920030+0000) 2019-10-13T08:21:37.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:37.918+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.920490+0000) 2019-10-13T08:21:37.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:37.918+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.920807+0000) 2019-10-13T08:21:37.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:37.920+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.922402+0000) 2019-10-13T08:21:37.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:37.918+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.923229+0000) 2019-10-13T08:21:37.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:37.920+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.925569+0000) 2019-10-13T08:21:38.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:38.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:38.447737+0000) 2019-10-13T08:21:39.434 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:39.428+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:39.433836+0000) 2019-10-13T08:21:39.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:39.888+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:39.888528+0000) 2019-10-13T08:21:41.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:41.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.688993+0000) 2019-10-13T08:21:41.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:41.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.734932+0000) 2019-10-13T08:21:41.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:41.926+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.929268+0000) 2019-10-13T08:21:42.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:42.918+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.922793+0000) 2019-10-13T08:21:42.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:42.918+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.923157+0000) 2019-10-13T08:21:42.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:42.922+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.923943+0000) 2019-10-13T08:21:42.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:42.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:42.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:42.922+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.924672+0000) 2019-10-13T08:21:42.925 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:42.922+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.924941+0000) 2019-10-13T08:21:42.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:42.920+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.925119+0000) 2019-10-13T08:21:42.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:42.922+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.925498+0000) 2019-10-13T08:21:42.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:42.928+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.932571+0000) 2019-10-13T08:21:44.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:44.954+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:44.958204+0000) 2019-10-13T08:21:46.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:46.178+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.180105+0000) 2019-10-13T08:21:47.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:47.924+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.926795+0000) 2019-10-13T08:21:47.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:47.926+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.928221+0000) 2019-10-13T08:21:47.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:47.926+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.928485+0000) 2019-10-13T08:21:47.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:47.926+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.929523+0000) 2019-10-13T08:21:47.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:47.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:47.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:47.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.930538+0000) 2019-10-13T08:21:47.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:47.926+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.930894+0000) 2019-10-13T08:21:47.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:47.930+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.931454+0000) 2019-10-13T08:21:47.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:47.932+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.935846+0000) 2019-10-13T08:21:48.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:48.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:48.447911+0000) 2019-10-13T08:21:49.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:49.432+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:49.434027+0000) 2019-10-13T08:21:49.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:49.892+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:49.895083+0000) 2019-10-13T08:21:51.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:51.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.689221+0000) 2019-10-13T08:21:51.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:51.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.735137+0000) 2019-10-13T08:21:51.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:51.926+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.929487+0000) 2019-10-13T08:21:52.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:52.930+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.932653+0000) 2019-10-13T08:21:52.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:52.928+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.933593+0000) 2019-10-13T08:21:52.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:52.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:52.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:52.930+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.934679+0000) 2019-10-13T08:21:52.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:52.930+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.934879+0000) 2019-10-13T08:21:52.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:52.930+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.935175+0000) 2019-10-13T08:21:52.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:52.934+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.935643+0000) 2019-10-13T08:21:52.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:52.934+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.937193+0000) 2019-10-13T08:21:52.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:52.940+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.942783+0000) 2019-10-13T08:21:54.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:54.954+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:54.958434+0000) 2019-10-13T08:21:56.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:56.178+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.180298+0000) 2019-10-13T08:21:56.463 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:21:56.463 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:56.467 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:21:56.467 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:57.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:57.931+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.937611+0000) 2019-10-13T08:21:57.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:21:57.934+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.938529+0000) 2019-10-13T08:21:57.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:21:57.934+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.939019+0000) 2019-10-13T08:21:57.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:21:57.934+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.939173+0000) 2019-10-13T08:21:57.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:21:57.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:21:57.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:57.938+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.940013+0000) 2019-10-13T08:21:57.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:21:57.938+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.940448+0000) 2019-10-13T08:21:57.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:21:57.938+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.940807+0000) 2019-10-13T08:21:57.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:57.939+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.944533+0000) 2019-10-13T08:21:58.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:21:58.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:58.448087+0000) 2019-10-13T08:21:59.434 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:21:59.431+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:59.434223+0000) 2019-10-13T08:21:59.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:21:59.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:59.901485+0000) 2019-10-13T08:22:01.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:01.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.689478+0000) 2019-10-13T08:22:01.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:01.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.735361+0000) 2019-10-13T08:22:01.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:01.925+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.929695+0000) 2019-10-13T08:22:02.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:02.941+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.943474+0000) 2019-10-13T08:22:02.944 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:02.941+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.943820+0000) 2019-10-13T08:22:02.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:02.941+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:02.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:02.939+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.944018+0000) 2019-10-13T08:22:02.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:02.941+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.944470+0000) 2019-10-13T08:22:02.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:02.941+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.945136+0000) 2019-10-13T08:22:02.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:02.941+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.945628+0000) 2019-10-13T08:22:02.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:02.941+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.945850+0000) 2019-10-13T08:22:02.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:02.943+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.948300+0000) 2019-10-13T08:22:04.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:04.953+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:04.958694+0000) 2019-10-13T08:22:06.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:06.177+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.180492+0000) 2019-10-13T08:22:07.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:07.945+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.947365+0000) 2019-10-13T08:22:07.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:07.943+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.947814+0000) 2019-10-13T08:22:07.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:07.945+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.948312+0000) 2019-10-13T08:22:07.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:07.945+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.948534+0000) 2019-10-13T08:22:07.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:07.945+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.949196+0000) 2019-10-13T08:22:07.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:07.945+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:07.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:07.945+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.949635+0000) 2019-10-13T08:22:07.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:07.945+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.949986+0000) 2019-10-13T08:22:07.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:07.955+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.959105+0000) 2019-10-13T08:22:08.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:08.445+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:08.448270+0000) 2019-10-13T08:22:09.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:09.431+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:09.434419+0000) 2019-10-13T08:22:09.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:09.903+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:09.907861+0000) 2019-10-13T08:22:11.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:11.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.689706+0000) 2019-10-13T08:22:11.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:11.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.735586+0000) 2019-10-13T08:22:11.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:11.929+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.929893+0000) 2019-10-13T08:22:12.953 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:12.949+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.952658+0000) 2019-10-13T08:22:12.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:12.949+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.953930+0000) 2019-10-13T08:22:12.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:12.949+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.953916+0000) 2019-10-13T08:22:12.954 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:12.949+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.954275+0000) 2019-10-13T08:22:12.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:12.949+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.954693+0000) 2019-10-13T08:22:12.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:12.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:12.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:12.951+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.955325+0000) 2019-10-13T08:22:12.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:12.953+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.956271+0000) 2019-10-13T08:22:12.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:12.955+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.961590+0000) 2019-10-13T08:22:14.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:14.957+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:14.958893+0000) 2019-10-13T08:22:16.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:16.177+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.180693+0000) 2019-10-13T08:22:17.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:17.953+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.958154+0000) 2019-10-13T08:22:17.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:17.953+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.958446+0000) 2019-10-13T08:22:17.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:17.953+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.958910+0000) 2019-10-13T08:22:17.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:17.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:17.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:17.957+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.959333+0000) 2019-10-13T08:22:17.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:17.957+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.960161+0000) 2019-10-13T08:22:17.961 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:17.957+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.961291+0000) 2019-10-13T08:22:17.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:17.955+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.961565+0000) 2019-10-13T08:22:17.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:17.959+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.965773+0000) 2019-10-13T08:22:18.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:18.445+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:18.448478+0000) 2019-10-13T08:22:19.434 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:19.431+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:19.434632+0000) 2019-10-13T08:22:19.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:19.911+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:19.914330+0000) 2019-10-13T08:22:21.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:21.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.689916+0000) 2019-10-13T08:22:21.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:21.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.735784+0000) 2019-10-13T08:22:21.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:21.929+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.930089+0000) 2019-10-13T08:22:22.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:22.957+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.962478+0000) 2019-10-13T08:22:22.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:22.957+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.962996+0000) 2019-10-13T08:22:22.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:22.961+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.963586+0000) 2019-10-13T08:22:22.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:22.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:22.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:22.961+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.964031+0000) 2019-10-13T08:22:22.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:22.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.964472+0000) 2019-10-13T08:22:22.965 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:22.961+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.964566+0000) 2019-10-13T08:22:22.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:22.961+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.965053+0000) 2019-10-13T08:22:22.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:22.967+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.971018+0000) 2019-10-13T08:22:24.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:24.957+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:24.959096+0000) 2019-10-13T08:22:26.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:26.177+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.180888+0000) 2019-10-13T08:22:26.612 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:22:26.612 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:26.616 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:22:26.616 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:27.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:27.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.966539+0000) 2019-10-13T08:22:27.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:27.965+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.967372+0000) 2019-10-13T08:22:27.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:27.965+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.967926+0000) 2019-10-13T08:22:27.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:27.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:27.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:27.965+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.968732+0000) 2019-10-13T08:22:27.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:27.963+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.969394+0000) 2019-10-13T08:22:27.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:27.965+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.969063+0000) 2019-10-13T08:22:27.971 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:27.965+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.970921+0000) 2019-10-13T08:22:27.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:27.971+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.974778+0000) 2019-10-13T08:22:28.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:28.445+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:28.448650+0000) 2019-10-13T08:22:29.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:29.431+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:29.434846+0000) 2019-10-13T08:22:29.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:29.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:29.920651+0000) 2019-10-13T08:22:31.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:31.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.690150+0000) 2019-10-13T08:22:31.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:31.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.735993+0000) 2019-10-13T08:22:31.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:31.929+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.930311+0000) 2019-10-13T08:22:32.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:32.969+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.971872+0000) 2019-10-13T08:22:32.973 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:32.969+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.972362+0000) 2019-10-13T08:22:32.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:32.969+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.972481+0000) 2019-10-13T08:22:32.973 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:32.969+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.972936+0000) 2019-10-13T08:22:32.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:32.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:32.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:32.969+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.973801+0000) 2019-10-13T08:22:32.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:32.969+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.974293+0000) 2019-10-13T08:22:32.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:32.971+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.976495+0000) 2019-10-13T08:22:32.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:32.975+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.979499+0000) 2019-10-13T08:22:34.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:34.957+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:34.959293+0000) 2019-10-13T08:22:36.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:36.177+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.181082+0000) 2019-10-13T08:22:37.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:37.973+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.976328+0000) 2019-10-13T08:22:37.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:37.973+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.976844+0000) 2019-10-13T08:22:37.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:37.973+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.977608+0000) 2019-10-13T08:22:37.978 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:37.973+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.978170+0000) 2019-10-13T08:22:37.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:37.973+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.978545+0000) 2019-10-13T08:22:37.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:37.975+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.979310+0000) 2019-10-13T08:22:37.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:37.977+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:37.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:37.977+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.979819+0000) 2019-10-13T08:22:37.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:37.979+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.985256+0000) 2019-10-13T08:22:38.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:38.445+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:38.448848+0000) 2019-10-13T08:22:39.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:39.431+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:39.435063+0000) 2019-10-13T08:22:39.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:39.931+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:39.931898+0000) 2019-10-13T08:22:41.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:41.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.690357+0000) 2019-10-13T08:22:41.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:41.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.736226+0000) 2019-10-13T08:22:41.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:41.925+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.930537+0000) 2019-10-13T08:22:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 124 2019-10-13T08:22:42.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:42.979+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.982771+0000) 2019-10-13T08:22:42.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:42.980+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.983711+0000) 2019-10-13T08:22:42.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:42.980+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.983364+0000) 2019-10-13T08:22:42.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:42.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 49996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:42.985 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:42.980+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.984640+0000) 2019-10-13T08:22:42.985 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:42.980+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.984897+0000) 2019-10-13T08:22:42.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:42.980+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.985520+0000) 2019-10-13T08:22:42.986 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:42.980+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.985836+0000) 2019-10-13T08:22:42.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:42.987+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.991259+0000) 2019-10-13T08:22:44.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:44.956+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:44.959548+0000) 2019-10-13T08:22:46.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:46.180+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.181274+0000) 2019-10-13T08:22:47.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:47.984+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.987753+0000) 2019-10-13T08:22:47.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:47.983+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.988118+0000) 2019-10-13T08:22:47.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:47.984+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.988827+0000) 2019-10-13T08:22:47.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:47.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:47.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:47.984+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.989136+0000) 2019-10-13T08:22:47.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:47.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.990307+0000) 2019-10-13T08:22:47.991 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:47.984+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.990556+0000) 2019-10-13T08:22:47.992 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:47.988+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.991345+0000) 2019-10-13T08:22:47.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:47.995+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.998744+0000) 2019-10-13T08:22:48.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:48.444+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:48.449030+0000) 2019-10-13T08:22:49.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:49.435+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:49.435280+0000) 2019-10-13T08:22:49.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:49.935+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:49.938175+0000) 2019-10-13T08:22:51.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:51.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.690573+0000) 2019-10-13T08:22:51.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:51.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.736432+0000) 2019-10-13T08:22:51.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:51.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.930741+0000) 2019-10-13T08:22:52.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:52.988+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.992324+0000) 2019-10-13T08:22:52.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:52.988+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.992725+0000) 2019-10-13T08:22:52.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:52.988+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.993103+0000) 2019-10-13T08:22:52.993 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:52.988+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.993694+0000) 2019-10-13T08:22:52.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:52.988+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.994723+0000) 2019-10-13T08:22:52.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:52.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:52.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:52.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.994953+0000) 2019-10-13T08:22:52.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:52.992+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.995442+0000) 2019-10-13T08:22:53.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:52.994+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:53.000918+0000) 2019-10-13T08:22:54.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:54.956+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:54.959720+0000) 2019-10-13T08:22:56.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:56.180+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.181496+0000) 2019-10-13T08:22:56.686 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:22:56.686 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:56.693 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:22:56.693 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:57.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:57.992+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.997964+0000) 2019-10-13T08:22:57.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:57.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.998114+0000) 2019-10-13T08:22:57.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:22:57.992+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.998842+0000) 2019-10-13T08:22:57.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:22:57.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:22:57.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:22:57.992+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.999250+0000) 2019-10-13T08:22:58.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:22:57.996+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.000021+0000) 2019-10-13T08:22:58.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:22:57.996+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.999700+0000) 2019-10-13T08:22:58.001 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:22:57.996+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.001974+0000) 2019-10-13T08:22:58.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:58.002+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.007694+0000) 2019-10-13T08:22:58.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:22:58.444+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.449223+0000) 2019-10-13T08:22:59.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:22:59.434+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:59.435506+0000) 2019-10-13T08:22:59.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:22:59.938+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:59.944408+0000) 2019-10-13T08:23:01.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:01.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.690776+0000) 2019-10-13T08:23:01.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:01.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.736665+0000) 2019-10-13T08:23:01.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:01.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.930901+0000) 2019-10-13T08:23:03.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:02.996+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.002636+0000) 2019-10-13T08:23:03.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:02.996+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.002875+0000) 2019-10-13T08:23:03.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:03.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:03.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:02.998+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.004060+0000) 2019-10-13T08:23:03.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:03.000+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.005289+0000) 2019-10-13T08:23:03.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:03.000+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.005832+0000) 2019-10-13T08:23:03.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:03.004+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.007862+0000) 2019-10-13T08:23:03.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:03.004+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.008236+0000) 2019-10-13T08:23:03.010 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:03.006+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:03.010152+0000) 2019-10-13T08:23:04.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:04.956+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:04.959933+0000) 2019-10-13T08:23:06.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:06.176+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.181690+0000) 2019-10-13T08:23:08.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:08.004+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.008408+0000) 2019-10-13T08:23:08.009 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:08.004+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.009016+0000) 2019-10-13T08:23:08.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:08.004+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.009443+0000) 2019-10-13T08:23:08.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:08.004+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.009829+0000) 2019-10-13T08:23:08.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:08.002+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.009993+0000) 2019-10-13T08:23:08.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:08.004+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.010373+0000) 2019-10-13T08:23:08.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:08.004+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:08.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:08.004+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.010804+0000) 2019-10-13T08:23:08.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:08.010+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.014583+0000) 2019-10-13T08:23:08.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:08.444+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.449403+0000) 2019-10-13T08:23:09.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:09.430+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:09.435701+0000) 2019-10-13T08:23:09.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:09.950+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:09.950788+0000) 2019-10-13T08:23:11.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:11.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.690981+0000) 2019-10-13T08:23:11.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:11.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.736897+0000) 2019-10-13T08:23:11.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:11.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.931165+0000) 2019-10-13T08:23:13.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:13.008+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.014275+0000) 2019-10-13T08:23:13.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:13.010+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.014713+0000) 2019-10-13T08:23:13.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:13.008+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.014664+0000) 2019-10-13T08:23:13.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:13.008+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.015233+0000) 2019-10-13T08:23:13.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:13.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:13.017 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:13.012+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.016158+0000) 2019-10-13T08:23:13.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:13.012+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.016474+0000) 2019-10-13T08:23:13.018 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:13.012+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.018123+0000) 2019-10-13T08:23:13.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:13.018+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:13.023888+0000) 2019-10-13T08:23:14.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:14.956+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:14.960172+0000) 2019-10-13T08:23:16.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:16.180+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.181919+0000) 2019-10-13T08:23:18.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:18.012+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.018925+0000) 2019-10-13T08:23:18.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:18.016+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.019622+0000) 2019-10-13T08:23:18.020 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:18.016+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.020237+0000) 2019-10-13T08:23:18.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:18.016+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:18.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:18.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.020665+0000) 2019-10-13T08:23:18.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:18.016+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.021124+0000) 2019-10-13T08:23:18.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:18.016+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.022589+0000) 2019-10-13T08:23:18.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:18.016+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.023105+0000) 2019-10-13T08:23:18.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:18.022+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.026543+0000) 2019-10-13T08:23:18.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:18.448+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.449579+0000) 2019-10-13T08:23:19.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:19.434+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:19.435893+0000) 2019-10-13T08:23:19.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:19.958+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:19.961331+0000) 2019-10-13T08:23:21.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:21.684+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.691220+0000) 2019-10-13T08:23:21.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:21.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.737104+0000) 2019-10-13T08:23:21.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:21.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.931384+0000) 2019-10-13T08:23:23.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:23.020+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.024637+0000) 2019-10-13T08:23:23.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:23.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:23.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:23.020+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.025800+0000) 2019-10-13T08:23:23.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:23.020+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.026031+0000) 2019-10-13T08:23:23.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:23.020+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.026431+0000) 2019-10-13T08:23:23.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:23.022+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.026678+0000) 2019-10-13T08:23:23.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:23.020+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.027161+0000) 2019-10-13T08:23:23.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:23.024+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.027460+0000) 2019-10-13T08:23:23.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:23.026+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:23.032059+0000) 2019-10-13T08:23:24.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:24.960+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:24.960371+0000) 2019-10-13T08:23:26.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:26.179+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.182075+0000) 2019-10-13T08:23:26.861 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:23:26.862 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:26.865 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:23:26.866 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:28.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:28.023+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.029740+0000) 2019-10-13T08:23:28.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:28.023+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.029865+0000) 2019-10-13T08:23:28.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:28.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:28.032 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:28.023+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.030489+0000) 2019-10-13T08:23:28.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:28.023+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.031168+0000) 2019-10-13T08:23:28.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:28.027+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.032501+0000) 2019-10-13T08:23:28.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:28.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.031454+0000) 2019-10-13T08:23:28.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:28.026+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.033799+0000) 2019-10-13T08:23:28.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:28.034+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.039468+0000) 2019-10-13T08:23:28.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:28.447+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.449759+0000) 2019-10-13T08:23:29.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:29.434+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:29.436112+0000) 2019-10-13T08:23:29.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:29.966+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:29.967762+0000) 2019-10-13T08:23:31.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:31.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.691451+0000) 2019-10-13T08:23:31.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:31.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.737326+0000) 2019-10-13T08:23:31.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:31.927+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.931585+0000) 2019-10-13T08:23:33.033 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:33.027+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.033267+0000) 2019-10-13T08:23:33.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:33.030+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.034099+0000) 2019-10-13T08:23:33.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:33.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.035063+0000) 2019-10-13T08:23:33.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:33.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:33.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:33.031+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.035901+0000) 2019-10-13T08:23:33.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:33.031+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.036698+0000) 2019-10-13T08:23:33.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:33.031+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.036998+0000) 2019-10-13T08:23:33.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:33.031+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.036042+0000) 2019-10-13T08:23:33.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:33.034+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:33.041080+0000) 2019-10-13T08:23:34.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:34.959+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:34.960586+0000) 2019-10-13T08:23:36.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:36.179+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.182301+0000) 2019-10-13T08:23:38.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:38.035+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.038887+0000) 2019-10-13T08:23:38.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:38.035+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.040163+0000) 2019-10-13T08:23:38.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:38.035+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.040139+0000) 2019-10-13T08:23:38.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:38.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:38.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:38.034+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.040433+0000) 2019-10-13T08:23:38.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:38.035+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.041699+0000) 2019-10-13T08:23:38.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:38.039+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.043799+0000) 2019-10-13T08:23:38.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:38.035+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.042948+0000) 2019-10-13T08:23:38.047 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:38.042+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.047554+0000) 2019-10-13T08:23:38.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:38.443+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.449957+0000) 2019-10-13T08:23:39.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:39.434+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:39.436333+0000) 2019-10-13T08:23:39.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:39.970+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:39.974107+0000) 2019-10-13T08:23:41.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:41.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.691680+0000) 2019-10-13T08:23:41.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:41.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.737521+0000) 2019-10-13T08:23:41.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:41.927+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.931794+0000) 2019-10-13T08:23:42.375 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T08:23:43.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:43.039+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.044480+0000) 2019-10-13T08:23:43.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:43.039+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.044889+0000) 2019-10-13T08:23:43.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:43.039+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.045141+0000) 2019-10-13T08:23:43.046 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:43.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:43.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:43.039+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.045565+0000) 2019-10-13T08:23:43.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:43.039+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.046964+0000) 2019-10-13T08:23:43.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:43.046+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.046985+0000) 2019-10-13T08:23:43.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:43.043+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.048203+0000) 2019-10-13T08:23:43.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:43.050+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:43.053267+0000) 2019-10-13T08:23:44.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:44.955+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:44.960789+0000) 2019-10-13T08:23:46.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:46.175+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.182495+0000) 2019-10-13T08:23:48.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:48.043+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.048467+0000) 2019-10-13T08:23:48.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:48.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:48.050 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:48.043+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.049816+0000) 2019-10-13T08:23:48.050 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:48.043+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.050260+0000) 2019-10-13T08:23:48.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:48.049+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.051220+0000) 2019-10-13T08:23:48.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:48.047+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.052045+0000) 2019-10-13T08:23:48.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:48.047+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.052485+0000) 2019-10-13T08:23:48.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:48.047+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.054020+0000) 2019-10-13T08:23:48.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:48.057+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.061282+0000) 2019-10-13T08:23:48.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:48.447+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.450137+0000) 2019-10-13T08:23:49.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:49.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:49.436525+0000) 2019-10-13T08:23:49.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:49.977+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:49.980444+0000) 2019-10-13T08:23:51.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:51.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.691916+0000) 2019-10-13T08:23:51.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:51.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.737750+0000) 2019-10-13T08:23:51.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:51.931+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.931990+0000) 2019-10-13T08:23:53.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:53.049+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.053749+0000) 2019-10-13T08:23:53.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:53.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:53.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:53.047+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.054228+0000) 2019-10-13T08:23:53.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:53.047+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.054996+0000) 2019-10-13T08:23:53.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:53.051+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.055633+0000) 2019-10-13T08:23:53.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:53.051+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.056151+0000) 2019-10-13T08:23:53.056 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:53.051+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.056721+0000) 2019-10-13T08:23:53.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:53.051+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.057360+0000) 2019-10-13T08:23:53.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:53.057+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:53.060662+0000) 2019-10-13T08:23:54.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:54.959+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:54.960963+0000) 2019-10-13T08:23:56.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:56.179+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.182680+0000) 2019-10-13T08:23:56.956 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:23:56.956 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:56.960 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:23:56.960 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:58.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:23:58.055+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.057798+0000) 2019-10-13T08:23:58.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:23:58.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:23:58.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:23:58.055+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.058640+0000) 2019-10-13T08:23:58.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:58.057+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.058826+0000) 2019-10-13T08:23:58.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:23:58.059+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.059290+0000) 2019-10-13T08:23:58.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:23:58.059+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.060730+0000) 2019-10-13T08:23:58.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:23:58.059+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.061318+0000) 2019-10-13T08:23:58.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:58.059+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.061848+0000) 2019-10-13T08:23:58.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:58.065+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.066835+0000) 2019-10-13T08:23:58.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:23:58.447+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.450320+0000) 2019-10-13T08:23:59.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:23:59.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:59.436723+0000) 2019-10-13T08:23:59.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:23:59.985+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:59.987025+0000) 2019-10-13T08:24:01.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:01.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.692150+0000) 2019-10-13T08:24:01.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:01.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.737951+0000) 2019-10-13T08:24:01.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:01.931+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.932184+0000) 2019-10-13T08:24:03.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:03.059+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.061594+0000) 2019-10-13T08:24:03.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:03.059+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.062724+0000) 2019-10-13T08:24:03.063 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:03.059+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.062731+0000) 2019-10-13T08:24:03.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:03.061+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.062967+0000) 2019-10-13T08:24:03.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:03.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:03.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:03.063+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.063741+0000) 2019-10-13T08:24:03.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:03.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.066604+0000) 2019-10-13T08:24:03.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:03.063+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.067230+0000) 2019-10-13T08:24:03.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:03.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:03.071476+0000) 2019-10-13T08:24:04.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:04.959+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:04.961158+0000) 2019-10-13T08:24:06.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:06.179+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.182840+0000) 2019-10-13T08:24:08.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:08.063+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.066027+0000) 2019-10-13T08:24:08.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:08.063+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.066761+0000) 2019-10-13T08:24:08.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:08.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:08.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:08.067+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.068463+0000) 2019-10-13T08:24:08.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:08.067+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.069030+0000) 2019-10-13T08:24:08.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:08.069+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.071210+0000) 2019-10-13T08:24:08.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:08.071+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.071644+0000) 2019-10-13T08:24:08.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:08.071+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.072414+0000) 2019-10-13T08:24:08.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:08.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.073588+0000) 2019-10-13T08:24:08.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:08.447+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.450522+0000) 2019-10-13T08:24:09.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:09.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:09.436921+0000) 2019-10-13T08:24:09.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:09.989+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:09.993405+0000) 2019-10-13T08:24:11.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:11.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.692348+0000) 2019-10-13T08:24:11.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:11.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.738149+0000) 2019-10-13T08:24:11.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:11.930+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.932395+0000) 2019-10-13T08:24:13.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:13.066+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.070819+0000) 2019-10-13T08:24:13.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:13.070+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.072661+0000) 2019-10-13T08:24:13.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:13.070+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.073033+0000) 2019-10-13T08:24:13.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:13.069+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.073478+0000) 2019-10-13T08:24:13.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:13.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:13.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:13.070+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.074348+0000) 2019-10-13T08:24:13.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:13.070+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.074702+0000) 2019-10-13T08:24:13.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:13.074+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.076268+0000) 2019-10-13T08:24:13.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:13.077+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:13.079374+0000) 2019-10-13T08:24:14.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:14.958+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:14.961358+0000) 2019-10-13T08:24:16.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:16.178+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.183033+0000) 2019-10-13T08:24:18.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:18.074+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.076455+0000) 2019-10-13T08:24:18.077 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:18.074+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.077175+0000) 2019-10-13T08:24:18.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:18.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:18.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:18.074+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.077987+0000) 2019-10-13T08:24:18.079 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:18.074+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.078625+0000) 2019-10-13T08:24:18.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:18.077+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.078205+0000) 2019-10-13T08:24:18.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:18.078+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.079698+0000) 2019-10-13T08:24:18.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:18.078+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.079697+0000) 2019-10-13T08:24:18.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:18.085+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.086319+0000) 2019-10-13T08:24:18.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:18.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.450723+0000) 2019-10-13T08:24:19.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:19.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:19.437159+0000) 2019-10-13T08:24:20.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:19.997+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:19.999763+0000) 2019-10-13T08:24:21.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:21.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.692542+0000) 2019-10-13T08:24:21.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:21.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.738346+0000) 2019-10-13T08:24:21.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:21.930+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.932640+0000) 2019-10-13T08:24:23.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:23.078+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.081139+0000) 2019-10-13T08:24:23.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:23.078+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.081929+0000) 2019-10-13T08:24:23.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:23.078+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.082556+0000) 2019-10-13T08:24:23.083 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:23.078+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.082712+0000) 2019-10-13T08:24:23.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:23.081+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.083223+0000) 2019-10-13T08:24:23.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:23.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:23.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:23.082+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.083940+0000) 2019-10-13T08:24:23.084 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:23.082+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.084423+0000) 2019-10-13T08:24:23.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:23.089+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:23.090764+0000) 2019-10-13T08:24:24.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:24.958+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:24.961538+0000) 2019-10-13T08:24:26.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:26.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.183234+0000) 2019-10-13T08:24:27.059 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:24:27.059 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:27.063 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:24:27.064 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:28.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:28.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.086437+0000) 2019-10-13T08:24:28.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:28.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.087613+0000) 2019-10-13T08:24:28.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:28.086+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.087953+0000) 2019-10-13T08:24:28.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:28.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:28.088 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:28.086+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.088702+0000) 2019-10-13T08:24:28.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:28.085+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.089006+0000) 2019-10-13T08:24:28.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:28.086+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.089566+0000) 2019-10-13T08:24:28.092 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:28.090+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.092177+0000) 2019-10-13T08:24:28.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:28.093+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.098056+0000) 2019-10-13T08:24:28.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:28.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.450905+0000) 2019-10-13T08:24:29.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:29.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:29.437385+0000) 2019-10-13T08:24:30.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:30.009+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:30.010743+0000) 2019-10-13T08:24:31.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:31.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692737+0000) 2019-10-13T08:24:31.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:31.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.738544+0000) 2019-10-13T08:24:31.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:31.930+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.932839+0000) 2019-10-13T08:24:33.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:33.090+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.091778+0000) 2019-10-13T08:24:33.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:33.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.092517+0000) 2019-10-13T08:24:33.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:33.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:33.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:33.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.092935+0000) 2019-10-13T08:24:33.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:33.090+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.093489+0000) 2019-10-13T08:24:33.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:33.090+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.093655+0000) 2019-10-13T08:24:33.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:33.093+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.095161+0000) 2019-10-13T08:24:33.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:33.094+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.096035+0000) 2019-10-13T08:24:33.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:33.101+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:33.102596+0000) 2019-10-13T08:24:34.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:34.958+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:34.961707+0000) 2019-10-13T08:24:36.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:36.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.183498+0000) 2019-10-13T08:24:38.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:38.094+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.096514+0000) 2019-10-13T08:24:38.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:38.094+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.096933+0000) 2019-10-13T08:24:38.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:38.094+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.097361+0000) 2019-10-13T08:24:38.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:38.094+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.098182+0000) 2019-10-13T08:24:38.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:38.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:38.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:38.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.098792+0000) 2019-10-13T08:24:38.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:38.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.100170+0000) 2019-10-13T08:24:38.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:38.097+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.101489+0000) 2019-10-13T08:24:38.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:38.101+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.104064+0000) 2019-10-13T08:24:38.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:38.446+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.451088+0000) 2019-10-13T08:24:39.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:39.433+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:39.437615+0000) 2019-10-13T08:24:40.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:40.013+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:40.016024+0000) 2019-10-13T08:24:41.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:41.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.692940+0000) 2019-10-13T08:24:41.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:41.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.738744+0000) 2019-10-13T08:24:41.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:41.930+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.933062+0000) 2019-10-13T08:24:42.376 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T08:24:42.376 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2019-10-13T08:24:42.376 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:24:42.376 INFO:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-10-13T08:24:43.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:43.096+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.101363+0000) 2019-10-13T08:24:43.102 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:43.098+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.102173+0000) 2019-10-13T08:24:43.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:43.098+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.102747+0000) 2019-10-13T08:24:43.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:43.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.102659+0000) 2019-10-13T08:24:43.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:43.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.103597+0000) 2019-10-13T08:24:43.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:43.102+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.103837+0000) 2019-10-13T08:24:43.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:43.102+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.104259+0000) 2019-10-13T08:24:43.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:43.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:43.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:43.108+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:43.111416+0000) 2019-10-13T08:24:43.238 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi129 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2019-10-13T08:24:43.239 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2019-10-13T08:24:43.267 DEBUG:tasks.check_counter:Getting stats from a 2019-10-13T08:24:43.267 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:24:43.268 INFO:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump 2019-10-13T08:24:43.392 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2019-10-13T08:24:43.392 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T08:24:43.392 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2035, 2019-10-13T08:24:43.393 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 3885, 2019-10-13T08:24:43.393 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 22501571, 2019-10-13T08:24:43.393 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 660290, 2019-10-13T08:24:43.393 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2030, 2019-10-13T08:24:43.394 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T08:24:43.394 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 1.781096954, 2019-10-13T08:24:43.394 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.116890051, 2019-10-13T08:24:43.394 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 1.906047085, 2019-10-13T08:24:43.395 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T08:24:43.395 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.395 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 2537, 2019-10-13T08:24:43.395 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 4.213075599, 2019-10-13T08:24:43.395 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001660652 2019-10-13T08:24:43.396 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.396 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.396 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2019-10-13T08:24:43.396 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.397 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.397 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T08:24:43.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.397 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T08:24:43.397 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 9082, 2019-10-13T08:24:43.398 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 18160, 2019-10-13T08:24:43.398 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 22877082, 2019-10-13T08:24:43.398 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 440798866, 2019-10-13T08:24:43.398 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 9099, 2019-10-13T08:24:43.399 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T08:24:43.399 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 6.755331972, 2019-10-13T08:24:43.399 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.925900710, 2019-10-13T08:24:43.399 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 5.694346722, 2019-10-13T08:24:43.399 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T08:24:43.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.400 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 16800, 2019-10-13T08:24:43.400 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 12.084271899, 2019-10-13T08:24:43.400 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000719301 2019-10-13T08:24:43.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2019-10-13T08:24:43.401 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.402 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T08:24:43.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.402 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T08:24:43.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 5604, 2019-10-13T08:24:43.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 10785, 2019-10-13T08:24:43.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 22803239, 2019-10-13T08:24:43.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 209586296, 2019-10-13T08:24:43.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5380, 2019-10-13T08:24:43.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T08:24:43.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 4.171197049, 2019-10-13T08:24:43.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.519120755, 2019-10-13T08:24:43.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 3.715571338, 2019-10-13T08:24:43.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.007853610, 2019-10-13T08:24:43.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 9437, 2019-10-13T08:24:43.405 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 7.989613865, 2019-10-13T08:24:43.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000846626 2019-10-13T08:24:43.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.406 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.406 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2019-10-13T08:24:43.406 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.406 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.407 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T08:24:43.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.407 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2019-10-13T08:24:43.407 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2019-10-13T08:24:43.408 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2019-10-13T08:24:43.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.408 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2019-10-13T08:24:43.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2019-10-13T08:24:43.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2019-10-13T08:24:43.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T08:24:43.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2019-10-13T08:24:43.410 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T08:24:43.410 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2019-10-13T08:24:43.410 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T08:24:43.410 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T08:24:43.410 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T08:24:43.411 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2019-10-13T08:24:43.411 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T08:24:43.411 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2019-10-13T08:24:43.411 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2019-10-13T08:24:43.412 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2019-10-13T08:24:43.412 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T08:24:43.412 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T08:24:43.412 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2019-10-13T08:24:43.412 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2019-10-13T08:24:43.413 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2019-10-13T08:24:43.413 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2019-10-13T08:24:43.413 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 78663, 2019-10-13T08:24:43.413 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 43, 2019-10-13T08:24:43.414 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2019-10-13T08:24:43.414 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2019-10-13T08:24:43.414 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2019-10-13T08:24:43.414 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2019-10-13T08:24:43.415 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2019-10-13T08:24:43.415 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2019-10-13T08:24:43.415 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2019-10-13T08:24:43.415 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2019-10-13T08:24:43.415 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2019-10-13T08:24:43.416 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2019-10-13T08:24:43.416 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T08:24:43.416 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2019-10-13T08:24:43.417 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2019-10-13T08:24:43.417 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2019-10-13T08:24:43.417 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 0, 2019-10-13T08:24:43.417 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 0, 2019-10-13T08:24:43.417 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2019-10-13T08:24:43.418 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2019-10-13T08:24:43.418 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2019-10-13T08:24:43.418 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2019-10-13T08:24:43.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2019-10-13T08:24:43.419 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T08:24:43.419 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2019-10-13T08:24:43.419 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2019-10-13T08:24:43.419 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2019-10-13T08:24:43.419 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 16718, 2019-10-13T08:24:43.420 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 66943768, 2019-10-13T08:24:43.420 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.420 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 16718, 2019-10-13T08:24:43.421 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2019-10-13T08:24:43.421 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2019-10-13T08:24:43.421 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 16718, 2019-10-13T08:24:43.421 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 66943768, 2019-10-13T08:24:43.421 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2019-10-13T08:24:43.422 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2019-10-13T08:24:43.422 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.422 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.422 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T08:24:43.423 INFO:teuthology.orchestra.run.smithi129.stdout: } 2019-10-13T08:24:43.423 INFO:teuthology.orchestra.run.smithi129.stdout:} 2019-10-13T08:24:43.425 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2019-10-13T08:24:43.425 DEBUG:tasks.check_counter:Getting stats from b 2019-10-13T08:24:43.425 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:24:43.426 INFO:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok perf dump 2019-10-13T08:24:43.813 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2019-10-13T08:24:43.813 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T08:24:43.813 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 7112, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 11901, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 35744429, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 6581473, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 3131, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 0, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 2.888717500, 2019-10-13T08:24:43.814 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.394193313, 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 2.725999456, 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.041645588, 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 9824, 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 7.194518675, 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000732341 2019-10-13T08:24:43.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 7653, 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.002536313, 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000331 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.816 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 3328, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 7945, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 30833730, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 8571338, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 2792, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 1, 2019-10-13T08:24:43.817 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 2.339287444, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.330863661, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 2.127118334, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.020962656, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 6104, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 6.324362271, 2019-10-13T08:24:43.818 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001036101 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.820 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T08:24:43.820 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 819, 2019-10-13T08:24:43.820 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 1158, 2019-10-13T08:24:43.820 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 2074760, 2019-10-13T08:24:43.820 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 4887491, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 222, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 1, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.272491306, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.071661686, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.163624786, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.026275045, 2019-10-13T08:24:43.821 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1021, 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.401393472, 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000393137 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.822 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: "cct": { 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: "total_workers": 1, 2019-10-13T08:24:43.823 INFO:teuthology.orchestra.run.smithi071.stdout: "unhealthy_workers": 0 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-MDSRank": { 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2683, 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.895651590, 2019-10-13T08:24:43.824 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000706541 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-PurgeQueue": { 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 327, 2019-10-13T08:24:43.825 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.040321746, 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000123308 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: "mds": { 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: "request": 787, 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: "reply": 787, 2019-10-13T08:24:43.826 INFO:teuthology.orchestra.run.smithi071.stdout: "reply_latency": { 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 787, 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 2.556636843, 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.003248585 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "forward": 0, 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch": 0, 2019-10-13T08:24:43.827 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_commit": 21, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_split": 1, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_merge": 0, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "inode_max": 2147483647, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes": 232, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_top": 0, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_bottom": 232, 2019-10-13T08:24:43.828 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pin_tail": 0, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pinned": 232, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_expired": 0, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_with_caps": 223, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": 223, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "subtrees": 2, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse": 1265, 2019-10-13T08:24:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_hit": 882, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_forward": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_discover": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_dir_fetch": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_remote_ino": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_lock": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "load_cent": 1323, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "q": 0, 2019-10-13T08:24:43.830 INFO:teuthology.orchestra.run.smithi071.stdout: "exported": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "exported_inodes": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "imported": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "imported_inodes": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_dir_fetch": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_backtrace_fetch": 0, 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_peer_discover": 0 2019-10-13T08:24:43.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache": { 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays": 0, 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_delayed": 0, 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_enqueuing": 0, 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_created": 233, 2019-10-13T08:24:43.832 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_enqueued": 233, 2019-10-13T08:24:43.839 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_reintegrated": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_migrated": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_processing": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_enqueued": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_prioritized": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_started": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_completed": 0, 2019-10-13T08:24:43.840 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_enqueue_scrub": 0, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_exportdir": 0, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_flush": 0, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragmentdir": 1, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragstats": 0, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_inodestats": 0 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.841 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log": { 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "evadd": 2819, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "evex": 0, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "evtrm": 0, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "ev": 2819, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "evexg": 0, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "evexd": 0, 2019-10-13T08:24:43.842 INFO:teuthology.orchestra.run.smithi071.stdout: "segadd": 3, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "segex": 0, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "segtrm": 0, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "seg": 3, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "segexg": 0, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "segexd": 0, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "expos": 4194304, 2019-10-13T08:24:43.843 INFO:teuthology.orchestra.run.smithi071.stdout: "wrpos": 10922258, 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: "rdpos": 0, 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: "jlat": { 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1273, 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 4.388548694, 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.003447406 2019-10-13T08:24:43.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "replayed": 0 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mem": { 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "ino": 235, 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "ino+": 468, 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "ino-": 233, 2019-10-13T08:24:43.845 INFO:teuthology.orchestra.run.smithi071.stdout: "dir": 27, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "dir+": 34, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "dir-": 7, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "dn": 271, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "dn+": 532, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "dn-": 261, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "cap": 223, 2019-10-13T08:24:43.846 INFO:teuthology.orchestra.run.smithi071.stdout: "cap+": 456, 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "cap-": 233, 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "rss": 51500, 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "heap": 314020 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_server": { 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_client_request": 860, 2019-10-13T08:24:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_server_request": 0, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_request": 787, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_session": 44, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_slave_request": 0, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "req_create_latency": { 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 437, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.207284539, 2019-10-13T08:24:43.848 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.002762664 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getattr_latency": { 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 7, 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.005542120, 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000791731 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.849 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getfilelock_latency": { 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: "req_link_latency": { 2019-10-13T08:24:43.850 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookup_latency": { 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1, 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000653811, 2019-10-13T08:24:43.851 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000653811 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookuphash_latency": { 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.852 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupino_latency": { 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupname_latency": { 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.853 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupparent_latency": { 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.854 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupsnap_latency": { 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lssnap_latency": { 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.855 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mkdir_latency": { 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 14, 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.040747526, 2019-10-13T08:24:43.856 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.002910537 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mknod_latency": { 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mksnap_latency": { 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "req_open_latency": { 2019-10-13T08:24:43.858 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 44, 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.043487528, 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000988352 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "req_readdir_latency": { 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 43, 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.502201265, 2019-10-13T08:24:43.859 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.011679099 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rename_latency": { 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 22, 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.093837074, 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.004265321 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.860 INFO:teuthology.orchestra.run.smithi071.stdout: "req_renamesnap_latency": { 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmdir_latency": { 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 6, 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.130376914, 2019-10-13T08:24:43.861 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.021729485 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmsnap_latency": { 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.862 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmxattr_latency": { 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setattr_latency": { 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1, 2019-10-13T08:24:43.863 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000915044, 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000915044 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setdirlayout_latency": { 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setfilelock_latency": { 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setlayout_latency": { 2019-10-13T08:24:43.865 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setxattr_latency": { 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.866 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.867 INFO:teuthology.orchestra.run.smithi071.stdout: "req_symlink_latency": { 2019-10-13T08:24:43.867 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4, 2019-10-13T08:24:43.867 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.008674730, 2019-10-13T08:24:43.867 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.002168682 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: "req_unlink_latency": { 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 208, 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.522916292, 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.002514020 2019-10-13T08:24:43.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_revoke_eviction": 0 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessions": { 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "session_count": 1, 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "session_add": 1, 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "session_remove": 0, 2019-10-13T08:24:43.869 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_open": 1, 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_stale": 0, 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "total_load": 13, 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "average_load": 13, 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "avg_session_uptime": 836 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool": { 2019-10-13T08:24:43.870 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_bytes": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_items": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_items": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_items": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T08:24:43.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_items": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_items": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_bytes": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_items": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T08:24:43.872 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_bytes": 0, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_items": 0, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_bytes": 0, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_items": 0, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_bytes": 124518, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_items": 598, 2019-10-13T08:24:43.873 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_bytes": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_items": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bytes": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_items": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_bytes": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_items": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_bytes": 0, 2019-10-13T08:24:43.874 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_items": 0, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_bytes": 5048, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_items": 75, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_items": 0, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_bytes": 0, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_items": 0, 2019-10-13T08:24:43.875 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_bytes": 731707, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_items": 4945, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_bytes": 0, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_items": 0, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_bytes": 0, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_items": 0 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.876 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter": { 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_active": 1, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_laggy": 0, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send": 1820, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send_bytes": 6897128, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_resend": 31, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op_reply": 1819, 2019-10-13T08:24:43.877 INFO:teuthology.orchestra.run.smithi071.stdout: "op": 1820, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "op_r": 23, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "op_w": 1797, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "op_rmw": 0, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "op_pg": 0, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_stat": 0, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_create": 11, 2019-10-13T08:24:43.878 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_read": 23, 2019-10-13T08:24:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_write": 1296, 2019-10-13T08:24:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writefull": 71, 2019-10-13T08:24:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writesame": 0, 2019-10-13T08:24:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_append": 0, 2019-10-13T08:24:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_zero": 0, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_truncate": 1, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_delete": 246, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_mapext": 0, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_sparse_read": 0, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_clonerange": 0, 2019-10-13T08:24:43.880 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_getxattr": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_setxattr": 22, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_cmpxattr": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_rmxattr": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_resetxattrs": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_call": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_watch": 0, 2019-10-13T08:24:43.881 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_notify": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_src_cmpxattr": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls_filter": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_other": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_active": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_send": 0, 2019-10-13T08:24:43.882 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_resend": 0, 2019-10-13T08:24:43.883 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_ping": 0, 2019-10-13T08:24:43.883 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_active": 0, 2019-10-13T08:24:43.883 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_send": 0, 2019-10-13T08:24:43.883 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_resend": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_active": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_send": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_resend": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_active": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_send": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_resend": 0, 2019-10-13T08:24:43.884 INFO:teuthology.orchestra.run.smithi071.stdout: "command_active": 0, 2019-10-13T08:24:43.885 INFO:teuthology.orchestra.run.smithi071.stdout: "command_send": 0, 2019-10-13T08:24:43.885 INFO:teuthology.orchestra.run.smithi071.stdout: "command_resend": 0, 2019-10-13T08:24:43.885 INFO:teuthology.orchestra.run.smithi071.stdout: "map_epoch": 24, 2019-10-13T08:24:43.885 INFO:teuthology.orchestra.run.smithi071.stdout: "map_full": 0, 2019-10-13T08:24:43.885 INFO:teuthology.orchestra.run.smithi071.stdout: "map_inc": 3, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_sessions": 3, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_open": 44, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_close": 41, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_laggy": 0, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_wr": 300, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_rd": 0, 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_del": 115 2019-10-13T08:24:43.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "purge_queue": { 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops": 0, 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing": 0, 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed": 233, 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_item_in_journal": 0 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.887 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 11254, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 67818917, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.888 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 11254, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 11254, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 67818917, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.889 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_bytes": { 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 90, 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2019-10-13T08:24:43.890 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 1820, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 6920661, 2019-10-13T08:24:43.891 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1572, 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 6920571, 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.892 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_ops": { 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 1024, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.893 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 1820, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 1820, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1819, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 1819, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.894 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle": { 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2019-10-13T08:24:43.895 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 233, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 23533, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 233, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2019-10-13T08:24:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 23, 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 23533, 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2019-10-13T08:24:43.899 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle-0x55fec85cd7c0": { 2019-10-13T08:24:43.899 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2019-10-13T08:24:43.899 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2019-10-13T08:24:43.899 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2019-10-13T08:24:43.899 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 2819, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 6727954, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 2819, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1273, 2019-10-13T08:24:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 6727954, 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout: } 2019-10-13T08:24:43.901 INFO:teuthology.orchestra.run.smithi071.stdout:} 2019-10-13T08:24:43.903 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2019-10-13T08:24:43.903 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2019-10-13T08:24:43.987 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2019-10-13T08:24:43.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T08:24:43.988 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:24:43.988 INFO:teuthology.orchestra.run.smithi129:> stat --file-system '--printf=%T 2019-10-13T08:24:43.988 INFO:teuthology.orchestra.run.smithi129:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T08:24:44.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:44.958+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:44.961911+0000) 2019-10-13T08:24:46.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:46.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.183697+0000) 2019-10-13T08:24:48.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:48.106+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.107847+0000) 2019-10-13T08:24:48.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:48.106+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.108234+0000) 2019-10-13T08:24:48.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:48.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:48.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:48.106+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.109059+0000) 2019-10-13T08:24:48.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:48.110+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.111294+0000) 2019-10-13T08:24:48.111 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:48.110+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.111655+0000) 2019-10-13T08:24:48.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:48.108+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.112012+0000) 2019-10-13T08:24:48.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:48.110+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.112949+0000) 2019-10-13T08:24:48.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:48.120+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.123087+0000) 2019-10-13T08:24:48.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:48.450+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.451349+0000) 2019-10-13T08:24:49.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:49.432+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:49.437840+0000) 2019-10-13T08:24:50.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:50.016+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:50.019779+0000) 2019-10-13T08:24:51.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:51.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.693138+0000) 2019-10-13T08:24:51.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:51.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.738957+0000) 2019-10-13T08:24:51.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:51.930+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.933264+0000) 2019-10-13T08:24:53.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:53.109+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.112164+0000) 2019-10-13T08:24:53.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:53.109+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.112509+0000) 2019-10-13T08:24:53.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:53.108+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.113067+0000) 2019-10-13T08:24:53.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:53.109+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.112885+0000) 2019-10-13T08:24:53.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:53.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:53.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:53.109+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.113861+0000) 2019-10-13T08:24:53.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:53.113+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.116492+0000) 2019-10-13T08:24:53.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:53.113+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.115868+0000) 2019-10-13T08:24:53.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:53.116+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:53.120478+0000) 2019-10-13T08:24:54.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:54.957+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:54.962111+0000) 2019-10-13T08:24:56.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:56.181+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.183895+0000) 2019-10-13T08:24:57.195 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:24:57.195 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:57.199 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:24:57.199 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:58.117 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:58.113+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.116546+0000) 2019-10-13T08:24:58.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:24:58.113+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.116827+0000) 2019-10-13T08:24:58.118 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:24:58.113+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.118394+0000) 2019-10-13T08:24:58.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:24:58.113+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.118469+0000) 2019-10-13T08:24:58.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:24:58.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:24:58.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:58.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.118598+0000) 2019-10-13T08:24:58.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:24:58.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.118871+0000) 2019-10-13T08:24:58.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:24:58.113+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.119270+0000) 2019-10-13T08:24:58.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:24:58.120+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.123346+0000) 2019-10-13T08:24:58.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:24:58.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.451531+0000) 2019-10-13T08:24:59.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:24:59.432+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:59.438065+0000) 2019-10-13T08:25:00.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:00.020+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:00.024543+0000) 2019-10-13T08:25:01.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:01.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.693359+0000) 2019-10-13T08:25:01.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:01.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.739165+0000) 2019-10-13T08:25:01.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:01.929+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.933457+0000) 2019-10-13T08:25:03.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:03.117+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.121633+0000) 2019-10-13T08:25:03.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:03.117+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.122425+0000) 2019-10-13T08:25:03.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:03.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:03.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:03.120+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.123774+0000) 2019-10-13T08:25:03.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:03.121+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.124479+0000) 2019-10-13T08:25:03.124 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:03.121+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.124785+0000) 2019-10-13T08:25:03.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:03.121+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.125283+0000) 2019-10-13T08:25:03.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:03.125+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.128044+0000) 2019-10-13T08:25:03.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:03.128+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:03.130376+0000) 2019-10-13T08:25:04.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:04.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:04.962314+0000) 2019-10-13T08:25:06.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:06.181+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.184101+0000) 2019-10-13T08:25:08.127 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:08.121+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.127034+0000) 2019-10-13T08:25:08.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:08.125+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.127705+0000) 2019-10-13T08:25:08.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:08.125+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.128545+0000) 2019-10-13T08:25:08.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:08.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:08.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:08.125+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.129006+0000) 2019-10-13T08:25:08.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:08.124+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.129259+0000) 2019-10-13T08:25:08.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:08.125+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.129782+0000) 2019-10-13T08:25:08.132 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:08.129+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.132056+0000) 2019-10-13T08:25:08.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:08.132+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.135521+0000) 2019-10-13T08:25:08.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:08.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.451713+0000) 2019-10-13T08:25:09.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:09.436+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:09.438252+0000) 2019-10-13T08:25:10.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:10.028+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:10.033050+0000) 2019-10-13T08:25:11.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:11.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.693563+0000) 2019-10-13T08:25:11.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:11.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.739379+0000) 2019-10-13T08:25:11.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:11.933+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.933694+0000) 2019-10-13T08:25:13.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:13.129+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.131483+0000) 2019-10-13T08:25:13.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:13.129+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.132165+0000) 2019-10-13T08:25:13.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:13.129+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.132672+0000) 2019-10-13T08:25:13.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:13.129+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.133951+0000) 2019-10-13T08:25:13.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:13.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 50986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:13.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:13.128+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.134071+0000) 2019-10-13T08:25:13.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:13.129+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.134638+0000) 2019-10-13T08:25:13.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:13.133+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.135876+0000) 2019-10-13T08:25:13.141 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:13.136+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:13.141367+0000) 2019-10-13T08:25:14.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:14.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:14.962526+0000) 2019-10-13T08:25:16.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:16.181+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.184293+0000) 2019-10-13T08:25:18.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:18.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.137138+0000) 2019-10-13T08:25:18.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:18.133+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.137892+0000) 2019-10-13T08:25:18.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:18.133+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.138327+0000) 2019-10-13T08:25:18.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:18.133+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.138976+0000) 2019-10-13T08:25:18.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:18.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:18.140 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:18.137+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.139915+0000) 2019-10-13T08:25:18.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:18.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.140218+0000) 2019-10-13T08:25:18.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:18.137+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.140810+0000) 2019-10-13T08:25:18.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:18.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.147152+0000) 2019-10-13T08:25:18.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:18.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.451892+0000) 2019-10-13T08:25:19.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:19.436+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:19.438439+0000) 2019-10-13T08:25:20.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:20.036+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:20.039554+0000) 2019-10-13T08:25:21.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:21.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.693796+0000) 2019-10-13T08:25:21.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:21.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.739575+0000) 2019-10-13T08:25:21.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:21.933+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.933927+0000) 2019-10-13T08:25:23.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:23.137+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.142975+0000) 2019-10-13T08:25:23.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:23.141+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.144346+0000) 2019-10-13T08:25:23.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:23.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:23.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:23.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.144585+0000) 2019-10-13T08:25:23.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:23.141+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.144844+0000) 2019-10-13T08:25:23.146 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:23.140+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.143798+0000) 2019-10-13T08:25:23.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:23.141+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.145688+0000) 2019-10-13T08:25:23.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:23.141+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.145914+0000) 2019-10-13T08:25:23.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:23.148+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:23.151525+0000) 2019-10-13T08:25:24.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:24.957+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:24.962723+0000) 2019-10-13T08:25:26.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:26.181+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.184522+0000) 2019-10-13T08:25:27.764 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:25:27.764 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:27.768 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:25:27.768 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:28.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:28.145+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.147709+0000) 2019-10-13T08:25:28.149 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:28.145+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.148145+0000) 2019-10-13T08:25:28.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:28.145+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.148763+0000) 2019-10-13T08:25:28.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:28.144+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.149119+0000) 2019-10-13T08:25:28.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:28.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:28.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:28.145+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.149735+0000) 2019-10-13T08:25:28.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:28.145+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.150614+0000) 2019-10-13T08:25:28.152 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:28.145+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.150954+0000) 2019-10-13T08:25:28.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:28.152+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.155505+0000) 2019-10-13T08:25:28.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:28.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.452105+0000) 2019-10-13T08:25:29.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:29.436+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:29.438662+0000) 2019-10-13T08:25:30.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:30.040+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:30.045918+0000) 2019-10-13T08:25:31.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:31.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.693993+0000) 2019-10-13T08:25:31.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:31.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.739799+0000) 2019-10-13T08:25:31.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:31.929+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.934155+0000) 2019-10-13T08:25:33.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:33.149+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.152857+0000) 2019-10-13T08:25:33.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:33.148+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.153296+0000) 2019-10-13T08:25:33.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:33.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:33.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:33.149+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.154130+0000) 2019-10-13T08:25:33.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:33.149+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.154249+0000) 2019-10-13T08:25:33.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:33.149+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.154663+0000) 2019-10-13T08:25:33.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:33.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.155382+0000) 2019-10-13T08:25:33.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:33.153+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.157541+0000) 2019-10-13T08:25:33.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:33.156+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:33.161179+0000) 2019-10-13T08:25:34.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:34.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:34.962924+0000) 2019-10-13T08:25:36.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:36.180+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.184715+0000) 2019-10-13T08:25:38.158 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:38.152+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.157483+0000) 2019-10-13T08:25:38.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:38.152+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.157749+0000) 2019-10-13T08:25:38.159 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:38.155+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.158176+0000) 2019-10-13T08:25:38.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:38.152+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.158608+0000) 2019-10-13T08:25:38.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:38.152+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.159280+0000) 2019-10-13T08:25:38.160 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:38.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:38.160 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:38.156+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.160342+0000) 2019-10-13T08:25:38.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:38.156+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.161677+0000) 2019-10-13T08:25:38.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:38.159+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.164857+0000) 2019-10-13T08:25:38.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:38.448+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.452284+0000) 2019-10-13T08:25:39.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:39.435+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:39.438886+0000) 2019-10-13T08:25:40.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:40.047+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:40.052230+0000) 2019-10-13T08:25:41.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:41.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.694194+0000) 2019-10-13T08:25:41.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:41.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.740031+0000) 2019-10-13T08:25:41.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:41.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.934384+0000) 2019-10-13T08:25:43.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:43.156+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.162269+0000) 2019-10-13T08:25:43.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:43.156+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.162614+0000) 2019-10-13T08:25:43.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:43.160+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.163398+0000) 2019-10-13T08:25:43.164 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:43.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:43.164 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:43.160+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.164605+0000) 2019-10-13T08:25:43.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:43.160+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.165177+0000) 2019-10-13T08:25:43.165 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:43.159+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.165352+0000) 2019-10-13T08:25:43.166 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:43.160+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.166077+0000) 2019-10-13T08:25:43.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:43.167+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:43.171538+0000) 2019-10-13T08:25:44.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:44.956+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:44.963156+0000) 2019-10-13T08:25:46.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:46.180+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.184910+0000) 2019-10-13T08:25:48.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:48.164+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.167637+0000) 2019-10-13T08:25:48.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:48.164+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.168104+0000) 2019-10-13T08:25:48.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:48.164+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.168255+0000) 2019-10-13T08:25:48.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:48.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.169123+0000) 2019-10-13T08:25:48.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:48.164+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.169672+0000) 2019-10-13T08:25:48.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:48.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:48.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:48.164+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.170393+0000) 2019-10-13T08:25:48.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:48.164+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.170125+0000) 2019-10-13T08:25:48.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:48.171+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.177212+0000) 2019-10-13T08:25:48.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:48.448+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.452460+0000) 2019-10-13T08:25:49.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:49.435+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:49.439082+0000) 2019-10-13T08:25:50.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:50.051+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:50.057679+0000) 2019-10-13T08:25:51.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:51.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.694394+0000) 2019-10-13T08:25:51.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:51.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.740244+0000) 2019-10-13T08:25:51.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:51.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.934619+0000) 2019-10-13T08:25:53.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:53.168+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.172800+0000) 2019-10-13T08:25:53.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:53.168+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.173332+0000) 2019-10-13T08:25:53.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:53.168+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.173764+0000) 2019-10-13T08:25:53.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:53.168+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.174317+0000) 2019-10-13T08:25:53.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:53.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:53.175 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:53.171+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.175050+0000) 2019-10-13T08:25:53.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:53.168+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.174995+0000) 2019-10-13T08:25:53.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:53.172+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.175703+0000) 2019-10-13T08:25:53.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:53.179+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:53.184109+0000) 2019-10-13T08:25:54.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:54.960+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:54.963386+0000) 2019-10-13T08:25:56.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:56.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.185106+0000) 2019-10-13T08:25:57.931 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:25:57.931 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:57.935 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:25:57.935 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:58.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:25:58.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.178171+0000) 2019-10-13T08:25:58.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:25:58.172+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.178327+0000) 2019-10-13T08:25:58.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:25:58.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:25:58.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:25:58.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.179690+0000) 2019-10-13T08:25:58.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:25:58.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.180135+0000) 2019-10-13T08:25:58.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:58.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.180473+0000) 2019-10-13T08:25:58.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:58.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.182382+0000) 2019-10-13T08:25:58.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:25:58.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.182702+0000) 2019-10-13T08:25:58.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:25:58.179+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.183647+0000) 2019-10-13T08:25:58.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:25:58.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.452637+0000) 2019-10-13T08:25:59.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:25:59.435+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:59.439290+0000) 2019-10-13T08:26:00.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:00.059+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:00.062251+0000) 2019-10-13T08:26:01.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:01.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.694599+0000) 2019-10-13T08:26:01.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:01.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.740446+0000) 2019-10-13T08:26:01.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:01.928+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.934831+0000) 2019-10-13T08:26:03.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:03.176+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.182695+0000) 2019-10-13T08:26:03.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:03.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.183194+0000) 2019-10-13T08:26:03.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:03.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.183561+0000) 2019-10-13T08:26:03.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:03.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:03.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:03.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.184808+0000) 2019-10-13T08:26:03.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:03.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.184910+0000) 2019-10-13T08:26:03.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:03.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.183999+0000) 2019-10-13T08:26:03.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:03.180+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.186580+0000) 2019-10-13T08:26:03.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:03.187+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:03.192946+0000) 2019-10-13T08:26:04.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:04.960+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:04.963591+0000) 2019-10-13T08:26:06.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:06.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.185301+0000) 2019-10-13T08:26:08.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:08.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.186864+0000) 2019-10-13T08:26:08.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:08.180+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.187272+0000) 2019-10-13T08:26:08.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:08.184+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.188135+0000) 2019-10-13T08:26:08.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:08.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.188619+0000) 2019-10-13T08:26:08.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:08.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:08.189 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:08.184+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.189208+0000) 2019-10-13T08:26:08.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:08.184+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.189732+0000) 2019-10-13T08:26:08.190 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:08.184+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.190210+0000) 2019-10-13T08:26:08.198 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:08.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.198525+0000) 2019-10-13T08:26:08.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:08.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.452850+0000) 2019-10-13T08:26:09.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:09.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:09.439499+0000) 2019-10-13T08:26:10.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:10.063+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:10.068632+0000) 2019-10-13T08:26:11.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:11.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.694796+0000) 2019-10-13T08:26:11.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:11.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.740649+0000) 2019-10-13T08:26:11.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:11.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.935026+0000) 2019-10-13T08:26:13.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:13.188+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.191996+0000) 2019-10-13T08:26:13.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:13.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.192396+0000) 2019-10-13T08:26:13.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:13.188+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.193062+0000) 2019-10-13T08:26:13.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:13.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:13.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:13.191+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.194258+0000) 2019-10-13T08:26:13.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:13.188+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.194228+0000) 2019-10-13T08:26:13.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:13.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.194653+0000) 2019-10-13T08:26:13.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:13.188+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.194978+0000) 2019-10-13T08:26:13.202 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:13.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:13.202340+0000) 2019-10-13T08:26:14.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:14.960+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:14.963792+0000) 2019-10-13T08:26:16.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:16.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.185455+0000) 2019-10-13T08:26:18.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:18.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.197060+0000) 2019-10-13T08:26:18.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:18.192+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.198325+0000) 2019-10-13T08:26:18.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:18.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:18.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:18.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.197334+0000) 2019-10-13T08:26:18.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:18.195+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.198459+0000) 2019-10-13T08:26:18.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:18.196+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.199633+0000) 2019-10-13T08:26:18.201 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:18.196+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.199546+0000) 2019-10-13T08:26:18.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:18.196+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.199866+0000) 2019-10-13T08:26:18.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:18.203+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.207508+0000) 2019-10-13T08:26:18.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:18.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.453023+0000) 2019-10-13T08:26:19.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:19.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:19.439751+0000) 2019-10-13T08:26:20.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:20.071+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:20.075026+0000) 2019-10-13T08:26:21.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:21.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.694997+0000) 2019-10-13T08:26:21.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:21.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.740857+0000) 2019-10-13T08:26:21.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:21.931+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.935241+0000) 2019-10-13T08:26:23.201 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:23.195+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.201389+0000) 2019-10-13T08:26:23.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:23.195+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.202507+0000) 2019-10-13T08:26:23.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:23.195+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.202832+0000) 2019-10-13T08:26:23.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:23.199+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.203698+0000) 2019-10-13T08:26:23.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:23.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:23.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:23.199+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.204171+0000) 2019-10-13T08:26:23.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:23.199+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.204851+0000) 2019-10-13T08:26:23.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:23.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.205066+0000) 2019-10-13T08:26:23.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:23.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:23.211825+0000) 2019-10-13T08:26:24.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:24.959+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:24.963991+0000) 2019-10-13T08:26:26.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:26.183+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.185717+0000) 2019-10-13T08:26:28.028 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:26:28.028 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:28.032 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:26:28.032 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:28.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:28.199+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.207263+0000) 2019-10-13T08:26:28.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:28.203+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.207786+0000) 2019-10-13T08:26:28.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:28.203+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.208341+0000) 2019-10-13T08:26:28.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:28.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:28.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:28.203+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.208964+0000) 2019-10-13T08:26:28.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:28.203+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.209828+0000) 2019-10-13T08:26:28.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:28.203+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.210542+0000) 2019-10-13T08:26:28.211 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:28.203+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.211001+0000) 2019-10-13T08:26:28.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:28.215+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.218744+0000) 2019-10-13T08:26:28.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:28.447+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.453206+0000) 2019-10-13T08:26:29.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:29.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:29.439970+0000) 2019-10-13T08:26:30.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:30.083+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:30.086752+0000) 2019-10-13T08:26:31.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:31.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.695193+0000) 2019-10-13T08:26:31.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:31.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.741063+0000) 2019-10-13T08:26:31.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:31.931+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.935466+0000) 2019-10-13T08:26:33.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:33.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.211849+0000) 2019-10-13T08:26:33.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:33.207+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.212266+0000) 2019-10-13T08:26:33.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:33.206+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.212892+0000) 2019-10-13T08:26:33.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:33.207+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:33.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:33.207+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.213543+0000) 2019-10-13T08:26:33.214 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:33.207+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.214344+0000) 2019-10-13T08:26:33.215 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:33.207+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.214676+0000) 2019-10-13T08:26:33.216 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:33.211+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.216057+0000) 2019-10-13T08:26:33.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:33.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:33.222193+0000) 2019-10-13T08:26:34.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:34.963+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:34.964219+0000) 2019-10-13T08:26:36.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:36.179+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.185903+0000) 2019-10-13T08:26:38.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:38.211+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.217367+0000) 2019-10-13T08:26:38.218 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:38.210+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.217906+0000) 2019-10-13T08:26:38.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:38.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:38.219 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:38.211+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.218508+0000) 2019-10-13T08:26:38.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:38.211+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.218816+0000) 2019-10-13T08:26:38.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:38.211+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.219082+0000) 2019-10-13T08:26:38.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:38.215+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.219649+0000) 2019-10-13T08:26:38.220 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:38.215+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.220198+0000) 2019-10-13T08:26:38.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:38.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.223172+0000) 2019-10-13T08:26:38.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:38.451+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.453445+0000) 2019-10-13T08:26:39.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:39.438+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:39.440165+0000) 2019-10-13T08:26:40.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:40.094+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:40.098463+0000) 2019-10-13T08:26:41.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:41.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.695408+0000) 2019-10-13T08:26:41.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:41.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.741295+0000) 2019-10-13T08:26:41.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:41.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.935670+0000) 2019-10-13T08:26:43.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:43.215+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.221140+0000) 2019-10-13T08:26:43.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:43.215+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.221651+0000) 2019-10-13T08:26:43.222 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:43.215+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.221890+0000) 2019-10-13T08:26:43.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:43.215+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.222312+0000) 2019-10-13T08:26:43.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:43.215+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.223098+0000) 2019-10-13T08:26:43.223 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:43.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:43.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:43.218+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.223602+0000) 2019-10-13T08:26:43.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:43.219+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.224524+0000) 2019-10-13T08:26:43.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:43.226+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:43.230715+0000) 2019-10-13T08:26:44.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:44.963+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:44.964417+0000) 2019-10-13T08:26:46.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:46.183+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.186096+0000) 2019-10-13T08:26:48.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:48.223+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.226896+0000) 2019-10-13T08:26:48.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:48.223+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.227771+0000) 2019-10-13T08:26:48.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:48.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:48.228 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:48.223+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.228426+0000) 2019-10-13T08:26:48.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:48.223+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.228872+0000) 2019-10-13T08:26:48.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:48.223+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.229457+0000) 2019-10-13T08:26:48.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:48.223+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.229661+0000) 2019-10-13T08:26:48.230 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:48.226+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.230380+0000) 2019-10-13T08:26:48.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:48.226+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.232848+0000) 2019-10-13T08:26:48.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:48.451+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.453622+0000) 2019-10-13T08:26:49.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:49.438+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:49.440392+0000) 2019-10-13T08:26:50.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:50.098+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:50.105005+0000) 2019-10-13T08:26:51.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:51.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.695590+0000) 2019-10-13T08:26:51.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:51.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.741544+0000) 2019-10-13T08:26:51.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:51.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.935894+0000) 2019-10-13T08:26:53.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:53.227+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.231024+0000) 2019-10-13T08:26:53.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:53.231+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.232154+0000) 2019-10-13T08:26:53.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:53.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.232547+0000) 2019-10-13T08:26:53.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:53.231+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.233122+0000) 2019-10-13T08:26:53.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:53.231+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.233632+0000) 2019-10-13T08:26:53.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:53.226+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.233899+0000) 2019-10-13T08:26:53.234 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:53.231+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.234276+0000) 2019-10-13T08:26:53.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:53.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51647 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:53.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:53.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:53.242015+0000) 2019-10-13T08:26:54.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:54.963+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:54.964618+0000) 2019-10-13T08:26:56.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:56.183+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.186287+0000) 2019-10-13T08:26:58.160 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:26:58.160 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:58.164 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:26:58.164 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:58.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:58.234+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.238397+0000) 2019-10-13T08:26:58.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:26:58.235+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.237820+0000) 2019-10-13T08:26:58.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:26:58.235+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.238111+0000) 2019-10-13T08:26:58.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:58.235+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.238784+0000) 2019-10-13T08:26:58.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:26:58.235+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.239001+0000) 2019-10-13T08:26:58.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:26:58.239+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.239274+0000) 2019-10-13T08:26:58.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:26:58.235+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.237929+0000) 2019-10-13T08:26:58.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:26:58.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:26:58.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:26:58.238+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.245967+0000) 2019-10-13T08:26:58.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:26:58.451+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.453807+0000) 2019-10-13T08:26:59.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:26:59.438+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:59.440584+0000) 2019-10-13T08:27:00.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:00.102+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:00.109515+0000) 2019-10-13T08:27:01.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:01.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.695819+0000) 2019-10-13T08:27:01.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:01.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.741769+0000) 2019-10-13T08:27:01.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:01.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.936110+0000) 2019-10-13T08:27:03.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:03.238+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.242975+0000) 2019-10-13T08:27:03.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:03.242+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.243660+0000) 2019-10-13T08:27:03.244 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:03.242+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.244332+0000) 2019-10-13T08:27:03.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:03.238+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.244005+0000) 2019-10-13T08:27:03.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:03.242+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.245035+0000) 2019-10-13T08:27:03.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:03.242+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51714 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:03.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:03.242+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.245634+0000) 2019-10-13T08:27:03.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:03.242+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.246017+0000) 2019-10-13T08:27:03.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:03.242+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:03.248759+0000) 2019-10-13T08:27:04.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:04.962+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:04.964851+0000) 2019-10-13T08:27:06.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:06.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.186475+0000) 2019-10-13T08:27:08.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:08.242+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.248459+0000) 2019-10-13T08:27:08.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:08.246+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.248840+0000) 2019-10-13T08:27:08.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:08.246+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.249096+0000) 2019-10-13T08:27:08.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:08.246+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.249572+0000) 2019-10-13T08:27:08.250 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:08.246+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.249769+0000) 2019-10-13T08:27:08.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:08.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:08.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:08.246+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.249803+0000) 2019-10-13T08:27:08.251 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:08.246+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.250372+0000) 2019-10-13T08:27:08.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:08.250+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.256621+0000) 2019-10-13T08:27:08.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:08.450+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.453986+0000) 2019-10-13T08:27:09.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:09.438+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:09.440775+0000) 2019-10-13T08:27:10.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:10.110+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:10.116024+0000) 2019-10-13T08:27:11.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:11.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.696023+0000) 2019-10-13T08:27:11.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:11.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.741966+0000) 2019-10-13T08:27:11.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:11.934+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.936310+0000) 2019-10-13T08:27:13.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:13.250+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.252386+0000) 2019-10-13T08:27:13.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:13.250+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.254002+0000) 2019-10-13T08:27:13.255 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:13.250+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.254271+0000) 2019-10-13T08:27:13.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:13.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:13.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:13.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.254987+0000) 2019-10-13T08:27:13.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:13.254+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.255511+0000) 2019-10-13T08:27:13.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:13.254+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.255541+0000) 2019-10-13T08:27:13.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:13.254+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.256508+0000) 2019-10-13T08:27:13.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:13.254+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:13.260219+0000) 2019-10-13T08:27:14.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:14.962+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:14.965022+0000) 2019-10-13T08:27:16.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:16.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.186691+0000) 2019-10-13T08:27:18.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:18.254+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.257137+0000) 2019-10-13T08:27:18.258 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:18.254+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.257462+0000) 2019-10-13T08:27:18.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:18.254+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.257585+0000) 2019-10-13T08:27:18.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:18.254+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.258349+0000) 2019-10-13T08:27:18.259 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:18.254+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.258675+0000) 2019-10-13T08:27:18.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:18.254+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.258919+0000) 2019-10-13T08:27:18.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:18.254+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.259143+0000) 2019-10-13T08:27:18.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:18.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51816 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:18.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:18.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.265476+0000) 2019-10-13T08:27:18.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:18.450+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.454168+0000) 2019-10-13T08:27:19.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:19.438+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:19.440966+0000) 2019-10-13T08:27:20.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:20.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:20.122571+0000) 2019-10-13T08:27:21.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:21.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.696225+0000) 2019-10-13T08:27:21.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:21.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.742166+0000) 2019-10-13T08:27:21.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:21.934+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.936509+0000) 2019-10-13T08:27:23.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:23.258+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.262283+0000) 2019-10-13T08:27:23.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:23.258+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.262811+0000) 2019-10-13T08:27:23.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:23.258+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.263017+0000) 2019-10-13T08:27:23.264 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:23.262+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.263907+0000) 2019-10-13T08:27:23.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:23.262+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.264191+0000) 2019-10-13T08:27:23.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:23.262+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.264198+0000) 2019-10-13T08:27:23.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:23.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:23.265 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:23.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.264808+0000) 2019-10-13T08:27:23.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:23.270+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:23.271172+0000) 2019-10-13T08:27:24.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:24.962+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:24.965249+0000) 2019-10-13T08:27:26.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:26.186+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.186881+0000) 2019-10-13T08:27:28.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:28.262+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.267055+0000) 2019-10-13T08:27:28.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:28.262+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.267210+0000) 2019-10-13T08:27:28.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:28.265+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.268811+0000) 2019-10-13T08:27:28.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:28.265+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.269845+0000) 2019-10-13T08:27:28.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:28.266+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.268427+0000) 2019-10-13T08:27:28.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:28.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:28.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:28.266+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.269236+0000) 2019-10-13T08:27:28.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:28.266+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.270031+0000) 2019-10-13T08:27:28.273 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:28.266+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.271169+0000) 2019-10-13T08:27:28.309 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:27:28.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:28.313 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:27:28.313 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:28.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:28.450+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.454348+0000) 2019-10-13T08:27:29.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:29.437+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:29.441178+0000) 2019-10-13T08:27:30.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:30.125+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:30.128879+0000) 2019-10-13T08:27:31.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:31.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.696436+0000) 2019-10-13T08:27:31.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:31.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.742407+0000) 2019-10-13T08:27:31.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:31.934+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.936738+0000) 2019-10-13T08:27:33.273 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:33.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.272696+0000) 2019-10-13T08:27:33.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:33.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.273633+0000) 2019-10-13T08:27:33.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:33.270+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.273871+0000) 2019-10-13T08:27:33.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:33.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:33.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:33.270+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.274440+0000) 2019-10-13T08:27:33.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:33.270+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.274865+0000) 2019-10-13T08:27:33.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:33.274+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.275636+0000) 2019-10-13T08:27:33.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:33.274+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.275892+0000) 2019-10-13T08:27:33.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:33.277+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:33.280299+0000) 2019-10-13T08:27:34.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:34.962+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:34.965476+0000) 2019-10-13T08:27:36.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:36.182+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.187031+0000) 2019-10-13T08:27:38.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:38.277+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.278907+0000) 2019-10-13T08:27:38.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:38.274+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.278635+0000) 2019-10-13T08:27:38.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:38.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:38.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:38.278+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.279346+0000) 2019-10-13T08:27:38.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:38.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.279515+0000) 2019-10-13T08:27:38.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:38.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.280147+0000) 2019-10-13T08:27:38.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:38.278+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.280547+0000) 2019-10-13T08:27:38.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:38.278+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.280893+0000) 2019-10-13T08:27:38.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:38.281+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.284279+0000) 2019-10-13T08:27:38.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:38.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.454539+0000) 2019-10-13T08:27:39.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:39.437+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:39.441379+0000) 2019-10-13T08:27:40.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:40.133+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:40.135328+0000) 2019-10-13T08:27:41.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:41.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.696635+0000) 2019-10-13T08:27:41.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:41.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.742615+0000) 2019-10-13T08:27:41.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:41.934+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.936938+0000) 2019-10-13T08:27:43.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:43.278+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.282957+0000) 2019-10-13T08:27:43.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:43.281+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.283112+0000) 2019-10-13T08:27:43.284 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:43.282+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.283528+0000) 2019-10-13T08:27:43.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:43.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 51986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:43.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:43.282+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.284420+0000) 2019-10-13T08:27:43.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:43.282+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.284548+0000) 2019-10-13T08:27:43.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:43.282+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.285049+0000) 2019-10-13T08:27:43.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:43.282+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.285591+0000) 2019-10-13T08:27:43.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:43.289+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:43.291943+0000) 2019-10-13T08:27:44.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:44.962+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:44.965675+0000) 2019-10-13T08:27:46.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:46.181+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.187236+0000) 2019-10-13T08:27:48.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:48.285+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.287271+0000) 2019-10-13T08:27:48.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:48.285+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.287798+0000) 2019-10-13T08:27:48.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:48.285+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.288242+0000) 2019-10-13T08:27:48.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:48.285+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.288708+0000) 2019-10-13T08:27:48.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:48.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52019 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:48.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:48.285+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.289639+0000) 2019-10-13T08:27:48.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:48.285+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.290196+0000) 2019-10-13T08:27:48.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:48.289+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.291654+0000) 2019-10-13T08:27:48.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:48.293+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.295475+0000) 2019-10-13T08:27:48.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:48.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.454776+0000) 2019-10-13T08:27:49.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:49.437+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:49.441573+0000) 2019-10-13T08:27:50.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:50.141+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:50.141684+0000) 2019-10-13T08:27:51.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:51.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.696846+0000) 2019-10-13T08:27:51.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:51.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.742828+0000) 2019-10-13T08:27:51.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:51.933+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.937138+0000) 2019-10-13T08:27:53.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:53.289+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.292141+0000) 2019-10-13T08:27:53.293 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:53.289+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.293064+0000) 2019-10-13T08:27:53.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:53.289+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.293583+0000) 2019-10-13T08:27:53.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:53.289+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.294017+0000) 2019-10-13T08:27:53.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:53.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:53.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:53.293+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.294216+0000) 2019-10-13T08:27:53.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:53.289+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.294882+0000) 2019-10-13T08:27:53.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:53.293+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.295824+0000) 2019-10-13T08:27:53.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:53.297+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:53.301887+0000) 2019-10-13T08:27:54.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:54.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:54.965909+0000) 2019-10-13T08:27:56.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:56.185+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.187490+0000) 2019-10-13T08:27:58.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:58.293+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.297006+0000) 2019-10-13T08:27:58.298 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:58.293+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.297955+0000) 2019-10-13T08:27:58.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:27:58.293+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.298743+0000) 2019-10-13T08:27:58.299 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:27:58.293+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.299268+0000) 2019-10-13T08:27:58.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:27:58.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:27:58.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:27:58.297+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.299884+0000) 2019-10-13T08:27:58.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:27:58.297+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.300503+0000) 2019-10-13T08:27:58.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:27:58.297+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.301143+0000) 2019-10-13T08:27:58.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:27:58.301+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.306085+0000) 2019-10-13T08:27:58.390 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:27:58.390 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:58.394 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:27:58.395 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:58.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:27:58.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.454953+0000) 2019-10-13T08:27:59.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:27:59.441+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:59.441768+0000) 2019-10-13T08:28:00.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:00.145+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:00.147916+0000) 2019-10-13T08:28:01.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:01.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:01.697087+0000) 2019-10-13T08:28:01.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:01.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:01.743030+0000) 2019-10-13T08:28:01.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:01.933+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:01.937356+0000) 2019-10-13T08:28:03.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:03.297+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.302967+0000) 2019-10-13T08:28:03.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:03.301+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.303631+0000) 2019-10-13T08:28:03.304 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:03.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:03.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:03.301+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.304095+0000) 2019-10-13T08:28:03.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:03.301+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.304316+0000) 2019-10-13T08:28:03.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:03.301+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.304933+0000) 2019-10-13T08:28:03.306 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:03.301+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.306160+0000) 2019-10-13T08:28:03.307 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:03.301+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.307247+0000) 2019-10-13T08:28:03.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:03.313+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:03.314227+0000) 2019-10-13T08:28:04.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:04.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:04.966099+0000) 2019-10-13T08:28:06.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:06.185+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:06.187681+0000) 2019-10-13T08:28:08.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:08.301+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.307205+0000) 2019-10-13T08:28:08.308 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:08.305+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.308527+0000) 2019-10-13T08:28:08.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:08.305+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.308619+0000) 2019-10-13T08:28:08.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:08.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:08.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:08.305+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.310879+0000) 2019-10-13T08:28:08.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:08.309+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.311439+0000) 2019-10-13T08:28:08.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:08.309+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.312813+0000) 2019-10-13T08:28:08.313 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:08.309+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.312977+0000) 2019-10-13T08:28:08.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:08.317+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.318505+0000) 2019-10-13T08:28:08.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:08.449+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.455135+0000) 2019-10-13T08:28:09.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:09.437+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:09.441965+0000) 2019-10-13T08:28:10.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:10.153+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:10.154272+0000) 2019-10-13T08:28:11.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:11.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:11.697296+0000) 2019-10-13T08:28:11.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:11.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:11.743234+0000) 2019-10-13T08:28:11.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:11.937+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:11.937553+0000) 2019-10-13T08:28:13.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:13.309+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.312195+0000) 2019-10-13T08:28:13.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:13.309+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.312703+0000) 2019-10-13T08:28:13.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:13.309+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.313234+0000) 2019-10-13T08:28:13.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:13.309+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.313476+0000) 2019-10-13T08:28:13.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:13.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52191 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:13.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:13.309+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.314452+0000) 2019-10-13T08:28:13.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:13.313+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.314970+0000) 2019-10-13T08:28:13.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:13.313+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.315487+0000) 2019-10-13T08:28:13.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:13.317+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:13.320028+0000) 2019-10-13T08:28:14.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:14.965+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:14.966317+0000) 2019-10-13T08:28:16.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:16.185+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:16.187891+0000) 2019-10-13T08:28:18.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:18.313+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.317398+0000) 2019-10-13T08:28:18.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:18.313+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.317842+0000) 2019-10-13T08:28:18.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:18.313+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.318337+0000) 2019-10-13T08:28:18.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:18.313+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.318643+0000) 2019-10-13T08:28:18.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:18.317+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:18.320 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:18.317+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.320538+0000) 2019-10-13T08:28:18.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:18.317+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.321333+0000) 2019-10-13T08:28:18.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:18.317+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.321484+0000) 2019-10-13T08:28:18.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:18.325+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.326463+0000) 2019-10-13T08:28:18.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:18.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.455350+0000) 2019-10-13T08:28:19.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:19.441+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:19.442183+0000) 2019-10-13T08:28:20.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:20.157+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:20.160729+0000) 2019-10-13T08:28:21.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:21.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:21.697527+0000) 2019-10-13T08:28:21.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:21.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:21.743477+0000) 2019-10-13T08:28:21.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:21.933+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:21.937762+0000) 2019-10-13T08:28:23.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:23.321+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.323378+0000) 2019-10-13T08:28:23.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:23.321+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.323817+0000) 2019-10-13T08:28:23.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:23.321+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.324602+0000) 2019-10-13T08:28:23.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:23.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:23.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:23.321+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.325699+0000) 2019-10-13T08:28:23.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:23.321+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.326013+0000) 2019-10-13T08:28:23.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:23.321+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.326612+0000) 2019-10-13T08:28:23.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:23.325+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.327029+0000) 2019-10-13T08:28:23.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:23.329+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:23.333988+0000) 2019-10-13T08:28:24.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:24.961+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:24.966549+0000) 2019-10-13T08:28:26.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:26.185+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:26.188079+0000) 2019-10-13T08:28:28.329 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:28.325+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.328147+0000) 2019-10-13T08:28:28.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:28.325+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.328410+0000) 2019-10-13T08:28:28.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:28.325+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.328794+0000) 2019-10-13T08:28:28.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:28.325+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.330067+0000) 2019-10-13T08:28:28.331 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:28.325+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.330415+0000) 2019-10-13T08:28:28.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:28.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:28.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:28.328+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.330652+0000) 2019-10-13T08:28:28.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:28.329+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.332248+0000) 2019-10-13T08:28:28.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:28.336+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.339450+0000) 2019-10-13T08:28:28.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:28.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.455484+0000) 2019-10-13T08:28:28.738 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:28:28.739 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:28.743 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:28:28.743 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:29.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:29.440+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:29.442410+0000) 2019-10-13T08:28:30.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:30.164+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:30.167275+0000) 2019-10-13T08:28:31.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:31.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:31.697732+0000) 2019-10-13T08:28:31.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:31.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:31.743692+0000) 2019-10-13T08:28:31.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:31.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:31.938007+0000) 2019-10-13T08:28:33.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:33.328+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.334177+0000) 2019-10-13T08:28:33.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:33.328+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.335018+0000) 2019-10-13T08:28:33.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:33.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:33.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:33.332+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.335435+0000) 2019-10-13T08:28:33.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:33.332+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.335939+0000) 2019-10-13T08:28:33.336 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:33.332+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.335788+0000) 2019-10-13T08:28:33.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:33.332+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.336819+0000) 2019-10-13T08:28:33.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:33.332+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.337929+0000) 2019-10-13T08:28:33.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:33.340+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:33.343491+0000) 2019-10-13T08:28:34.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:34.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:34.966774+0000) 2019-10-13T08:28:36.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:36.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:36.188290+0000) 2019-10-13T08:28:38.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:38.336+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.338641+0000) 2019-10-13T08:28:38.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:38.332+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.339189+0000) 2019-10-13T08:28:38.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:38.336+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.340166+0000) 2019-10-13T08:28:38.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:38.336+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.340833+0000) 2019-10-13T08:28:38.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:38.336+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.340828+0000) 2019-10-13T08:28:38.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:38.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:38.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:38.336+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.341463+0000) 2019-10-13T08:28:38.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:38.336+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.342188+0000) 2019-10-13T08:28:38.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:38.340+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.344947+0000) 2019-10-13T08:28:38.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:38.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.455671+0000) 2019-10-13T08:28:39.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:39.440+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:39.442668+0000) 2019-10-13T08:28:40.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:40.168+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:40.173651+0000) 2019-10-13T08:28:41.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:41.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:41.697966+0000) 2019-10-13T08:28:41.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:41.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:41.743858+0000) 2019-10-13T08:28:41.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:41.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:41.938203+0000) 2019-10-13T08:28:43.344 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:43.340+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.343678+0000) 2019-10-13T08:28:43.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:43.340+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.344918+0000) 2019-10-13T08:28:43.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:43.340+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.345425+0000) 2019-10-13T08:28:43.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:43.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:43.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:43.340+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.345963+0000) 2019-10-13T08:28:43.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:43.340+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.346527+0000) 2019-10-13T08:28:43.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:43.344+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.347456+0000) 2019-10-13T08:28:43.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:43.344+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.348120+0000) 2019-10-13T08:28:43.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:43.352+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:43.354568+0000) 2019-10-13T08:28:44.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:44.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:44.966970+0000) 2019-10-13T08:28:46.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:46.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:46.188491+0000) 2019-10-13T08:28:48.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:48.344+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.349263+0000) 2019-10-13T08:28:48.350 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:48.344+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.349437+0000) 2019-10-13T08:28:48.350 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:48.348+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.350125+0000) 2019-10-13T08:28:48.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:48.344+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.350705+0000) 2019-10-13T08:28:48.351 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:48.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:48.351 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:48.344+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.350985+0000) 2019-10-13T08:28:48.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:48.348+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.352981+0000) 2019-10-13T08:28:48.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:48.348+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.352306+0000) 2019-10-13T08:28:48.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:48.356+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.358589+0000) 2019-10-13T08:28:48.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:48.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.455873+0000) 2019-10-13T08:28:49.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:49.440+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:49.442888+0000) 2019-10-13T08:28:50.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:50.180+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:50.183819+0000) 2019-10-13T08:28:51.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:51.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:51.698166+0000) 2019-10-13T08:28:51.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:51.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:51.744022+0000) 2019-10-13T08:28:51.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:51.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:51.938397+0000) 2019-10-13T08:28:53.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:53.352+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.354306+0000) 2019-10-13T08:28:53.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:53.348+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.354782+0000) 2019-10-13T08:28:53.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:53.348+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.355148+0000) 2019-10-13T08:28:53.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:53.352+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.355588+0000) 2019-10-13T08:28:53.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:53.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:53.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:53.352+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.356111+0000) 2019-10-13T08:28:53.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:53.352+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.356754+0000) 2019-10-13T08:28:53.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:53.352+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.356923+0000) 2019-10-13T08:28:53.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:53.360+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:53.363741+0000) 2019-10-13T08:28:54.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:54.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:54.967124+0000) 2019-10-13T08:28:56.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:56.188+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:56.188685+0000) 2019-10-13T08:28:58.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:58.356+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.359373+0000) 2019-10-13T08:28:58.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:28:58.356+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.359762+0000) 2019-10-13T08:28:58.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:28:58.356+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.360740+0000) 2019-10-13T08:28:58.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:28:58.356+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.361065+0000) 2019-10-13T08:28:58.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:28:58.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:28:58.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:58.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.361082+0000) 2019-10-13T08:28:58.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:28:58.356+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.362901+0000) 2019-10-13T08:28:58.364 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:28:58.360+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.363532+0000) 2019-10-13T08:28:58.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:28:58.368+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.370283+0000) 2019-10-13T08:28:58.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:28:58.452+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.456072+0000) 2019-10-13T08:28:58.786 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:28:58.786 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:58.790 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:28:58.790 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:59.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:28:59.440+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:59.443125+0000) 2019-10-13T08:29:00.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:00.188+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:00.190009+0000) 2019-10-13T08:29:01.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:01.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:01.698407+0000) 2019-10-13T08:29:01.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:01.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:01.744256+0000) 2019-10-13T08:29:01.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:01.936+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:01.938626+0000) 2019-10-13T08:29:03.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:03.360+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.363621+0000) 2019-10-13T08:29:03.365 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:03.360+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.364950+0000) 2019-10-13T08:29:03.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:03.360+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.365386+0000) 2019-10-13T08:29:03.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:03.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:03.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:03.360+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.365756+0000) 2019-10-13T08:29:03.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:03.360+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.366449+0000) 2019-10-13T08:29:03.368 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:03.360+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.366739+0000) 2019-10-13T08:29:03.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:03.364+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.367449+0000) 2019-10-13T08:29:03.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:03.368+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:03.371662+0000) 2019-10-13T08:29:04.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:04.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:04.967346+0000) 2019-10-13T08:29:06.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:06.184+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:06.188899+0000) 2019-10-13T08:29:08.369 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:08.364+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.368703+0000) 2019-10-13T08:29:08.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:08.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:08.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:08.364+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.369954+0000) 2019-10-13T08:29:08.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:08.364+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.369566+0000) 2019-10-13T08:29:08.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:08.364+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.370630+0000) 2019-10-13T08:29:08.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:08.368+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.371527+0000) 2019-10-13T08:29:08.372 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:08.368+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.372201+0000) 2019-10-13T08:29:08.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:08.368+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.372941+0000) 2019-10-13T08:29:08.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:08.376+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.380646+0000) 2019-10-13T08:29:08.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:08.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.456299+0000) 2019-10-13T08:29:09.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:09.440+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:09.443322+0000) 2019-10-13T08:29:10.202 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:10.200+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:10.201554+0000) 2019-10-13T08:29:11.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:11.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:11.698606+0000) 2019-10-13T08:29:11.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:11.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:11.744495+0000) 2019-10-13T08:29:11.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:11.932+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:11.938817+0000) 2019-10-13T08:29:13.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:13.367+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.373982+0000) 2019-10-13T08:29:13.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:13.368+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.373995+0000) 2019-10-13T08:29:13.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:13.367+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.374543+0000) 2019-10-13T08:29:13.375 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:13.367+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.374773+0000) 2019-10-13T08:29:13.376 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:13.367+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.375202+0000) 2019-10-13T08:29:13.376 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:13.371+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.376064+0000) 2019-10-13T08:29:13.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:13.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:13.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:13.371+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.376833+0000) 2019-10-13T08:29:13.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:13.380+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.384470+0000) 2019-10-13T08:29:14.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:14.963+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:14.967560+0000) 2019-10-13T08:29:16.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:16.187+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:16.189109+0000) 2019-10-13T08:29:18.378 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:18.375+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.378382+0000) 2019-10-13T08:29:18.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:18.375+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.379458+0000) 2019-10-13T08:29:18.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:18.376+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.380111+0000) 2019-10-13T08:29:18.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:18.375+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.380481+0000) 2019-10-13T08:29:18.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:18.375+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.381490+0000) 2019-10-13T08:29:18.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:18.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.381705+0000) 2019-10-13T08:29:18.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:18.375+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.382480+0000) 2019-10-13T08:29:18.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:18.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:18.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:18.384+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.388248+0000) 2019-10-13T08:29:18.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:18.455+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.456529+0000) 2019-10-13T08:29:19.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:19.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:19.443548+0000) 2019-10-13T08:29:20.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:20.203+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:20.207899+0000) 2019-10-13T08:29:21.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:21.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:21.698846+0000) 2019-10-13T08:29:21.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:21.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:21.744697+0000) 2019-10-13T08:29:21.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:21.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:21.939013+0000) 2019-10-13T08:29:23.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:23.379+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.384576+0000) 2019-10-13T08:29:23.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:23.379+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.384731+0000) 2019-10-13T08:29:23.386 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:23.379+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.385870+0000) 2019-10-13T08:29:23.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:23.379+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.386264+0000) 2019-10-13T08:29:23.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:23.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:23.387 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:23.379+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.386475+0000) 2019-10-13T08:29:23.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:23.379+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.387193+0000) 2019-10-13T08:29:23.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:23.379+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.386782+0000) 2019-10-13T08:29:23.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:23.387+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.391840+0000) 2019-10-13T08:29:24.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:24.967+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:24.967814+0000) 2019-10-13T08:29:26.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:26.187+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:26.189321+0000) 2019-10-13T08:29:28.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:28.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.389959+0000) 2019-10-13T08:29:28.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:28.383+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.390074+0000) 2019-10-13T08:29:28.391 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:28.383+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.390816+0000) 2019-10-13T08:29:28.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:28.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.391067+0000) 2019-10-13T08:29:28.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:28.387+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.391336+0000) 2019-10-13T08:29:28.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:28.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:28.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:28.387+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.392858+0000) 2019-10-13T08:29:28.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:28.391+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.396097+0000) 2019-10-13T08:29:28.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:28.395+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.399057+0000) 2019-10-13T08:29:28.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:28.455+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.456718+0000) 2019-10-13T08:29:28.887 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:29:28.887 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:28.891 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:29:28.891 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:29.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:29.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:29.443775+0000) 2019-10-13T08:29:30.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:30.215+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:30.219114+0000) 2019-10-13T08:29:31.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:31.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:31.699081+0000) 2019-10-13T08:29:31.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:31.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:31.744933+0000) 2019-10-13T08:29:31.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:31.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:31.939248+0000) 2019-10-13T08:29:33.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:33.391+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.394230+0000) 2019-10-13T08:29:33.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:33.391+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.395204+0000) 2019-10-13T08:29:33.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:33.391+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.394939+0000) 2019-10-13T08:29:33.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:33.391+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.396208+0000) 2019-10-13T08:29:33.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:33.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:33.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:33.391+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.397066+0000) 2019-10-13T08:29:33.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:33.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.397491+0000) 2019-10-13T08:29:33.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:33.395+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.400872+0000) 2019-10-13T08:29:33.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:33.399+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.405280+0000) 2019-10-13T08:29:34.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:34.967+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:34.968014+0000) 2019-10-13T08:29:36.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:36.183+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:36.189550+0000) 2019-10-13T08:29:38.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:38.395+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.400301+0000) 2019-10-13T08:29:38.401 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:38.395+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.400554+0000) 2019-10-13T08:29:38.402 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:38.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:38.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:38.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.401433+0000) 2019-10-13T08:29:38.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:38.395+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.401172+0000) 2019-10-13T08:29:38.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:38.395+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.401647+0000) 2019-10-13T08:29:38.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:38.395+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.402210+0000) 2019-10-13T08:29:38.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:38.399+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.402477+0000) 2019-10-13T08:29:38.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:38.403+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.409757+0000) 2019-10-13T08:29:38.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:38.455+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.456964+0000) 2019-10-13T08:29:39.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:39.439+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:39.443999+0000) 2019-10-13T08:29:40.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:40.219+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:40.225495+0000) 2019-10-13T08:29:41.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:41.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:41.699279+0000) 2019-10-13T08:29:41.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:41.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:41.745136+0000) 2019-10-13T08:29:41.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:41.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:41.939508+0000) 2019-10-13T08:29:43.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:43.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.405047+0000) 2019-10-13T08:29:43.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:43.403+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.405277+0000) 2019-10-13T08:29:43.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:43.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:43.406 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:43.399+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.406107+0000) 2019-10-13T08:29:43.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:43.403+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.406662+0000) 2019-10-13T08:29:43.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:43.407+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.407753+0000) 2019-10-13T08:29:43.408 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:43.407+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.408317+0000) 2019-10-13T08:29:43.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:43.407+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.408937+0000) 2019-10-13T08:29:43.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:43.407+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.412282+0000) 2019-10-13T08:29:44.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:44.967+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:44.968239+0000) 2019-10-13T08:29:46.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:46.187+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:46.189775+0000) 2019-10-13T08:29:48.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:48.407+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.410377+0000) 2019-10-13T08:29:48.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:48.407+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.410703+0000) 2019-10-13T08:29:48.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:48.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:48.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:48.411+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.411572+0000) 2019-10-13T08:29:48.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:48.407+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.411022+0000) 2019-10-13T08:29:48.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:48.411+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.411890+0000) 2019-10-13T08:29:48.413 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:48.407+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.412367+0000) 2019-10-13T08:29:48.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:48.411+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.414914+0000) 2019-10-13T08:29:48.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:48.419+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.422812+0000) 2019-10-13T08:29:48.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:48.455+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.457197+0000) 2019-10-13T08:29:49.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:49.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:49.444237+0000) 2019-10-13T08:29:50.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:50.231+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:50.231913+0000) 2019-10-13T08:29:51.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:51.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:51.699513+0000) 2019-10-13T08:29:51.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:51.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:51.745331+0000) 2019-10-13T08:29:51.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:51.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:51.939704+0000) 2019-10-13T08:29:53.414 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:53.411+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.413898+0000) 2019-10-13T08:29:53.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:53.411+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.414262+0000) 2019-10-13T08:29:53.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:53.411+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.415220+0000) 2019-10-13T08:29:53.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:53.411+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.415103+0000) 2019-10-13T08:29:53.416 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:53.415+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.416012+0000) 2019-10-13T08:29:53.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:53.415+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:53.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:53.415+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.417232+0000) 2019-10-13T08:29:53.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:53.411+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.418002+0000) 2019-10-13T08:29:53.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:53.419+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.424527+0000) 2019-10-13T08:29:54.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:54.967+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:54.968441+0000) 2019-10-13T08:29:56.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:56.186+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:56.189972+0000) 2019-10-13T08:29:58.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:58.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.420252+0000) 2019-10-13T08:29:58.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:29:58.418+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.420106+0000) 2019-10-13T08:29:58.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:29:58.418+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.420624+0000) 2019-10-13T08:29:58.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:29:58.418+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.421517+0000) 2019-10-13T08:29:58.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:29:58.418+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.421958+0000) 2019-10-13T08:29:58.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:29:58.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:29:58.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:58.419+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.422999+0000) 2019-10-13T08:29:58.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:29:58.422+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.424214+0000) 2019-10-13T08:29:58.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:29:58.423+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.428979+0000) 2019-10-13T08:29:58.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:29:58.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.457357+0000) 2019-10-13T08:29:59.008 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:29:59.008 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:59.012 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:29:59.012 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:59.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:29:59.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:59.444464+0000) 2019-10-13T08:30:00.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:00.235+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:00.238319+0000) 2019-10-13T08:30:01.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:01.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:01.699715+0000) 2019-10-13T08:30:01.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:01.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:01.745526+0000) 2019-10-13T08:30:01.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:01.938+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:01.939896+0000) 2019-10-13T08:30:03.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:03.422+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.425532+0000) 2019-10-13T08:30:03.426 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:03.422+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.425865+0000) 2019-10-13T08:30:03.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:03.422+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.426564+0000) 2019-10-13T08:30:03.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:03.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:03.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:03.422+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.426655+0000) 2019-10-13T08:30:03.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:03.422+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.426973+0000) 2019-10-13T08:30:03.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:03.423+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.427598+0000) 2019-10-13T08:30:03.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:03.426+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.428702+0000) 2019-10-13T08:30:03.435 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:03.431+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.435209+0000) 2019-10-13T08:30:04.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:04.966+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:04.968673+0000) 2019-10-13T08:30:06.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:06.186+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:06.190165+0000) 2019-10-13T08:30:08.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:08.426+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.430639+0000) 2019-10-13T08:30:08.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:08.426+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.430899+0000) 2019-10-13T08:30:08.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:08.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.431193+0000) 2019-10-13T08:30:08.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:08.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 52970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:08.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:08.427+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.431573+0000) 2019-10-13T08:30:08.433 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:08.430+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.432179+0000) 2019-10-13T08:30:08.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:08.430+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.432641+0000) 2019-10-13T08:30:08.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:08.430+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.432959+0000) 2019-10-13T08:30:08.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:08.431+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.437362+0000) 2019-10-13T08:30:08.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:08.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.457527+0000) 2019-10-13T08:30:09.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:09.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:09.444666+0000) 2019-10-13T08:30:10.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:10.243+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:10.244819+0000) 2019-10-13T08:30:11.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:11.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:11.699917+0000) 2019-10-13T08:30:11.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:11.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:11.745728+0000) 2019-10-13T08:30:11.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:11.938+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:11.940103+0000) 2019-10-13T08:30:13.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:13.430+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.434718+0000) 2019-10-13T08:30:13.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:13.434+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.435288+0000) 2019-10-13T08:30:13.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:13.431+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.435736+0000) 2019-10-13T08:30:13.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:13.434+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.436257+0000) 2019-10-13T08:30:13.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:13.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:13.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:13.434+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.437127+0000) 2019-10-13T08:30:13.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:13.434+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.437188+0000) 2019-10-13T08:30:13.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:13.434+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.438161+0000) 2019-10-13T08:30:13.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:13.439+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.444609+0000) 2019-10-13T08:30:14.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:14.966+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:14.968872+0000) 2019-10-13T08:30:16.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:16.186+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:16.190402+0000) 2019-10-13T08:30:18.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:18.438+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.439672+0000) 2019-10-13T08:30:18.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:18.438+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.440247+0000) 2019-10-13T08:30:18.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:18.438+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.440403+0000) 2019-10-13T08:30:18.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:18.438+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.440768+0000) 2019-10-13T08:30:18.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:18.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:18.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:18.438+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.442723+0000) 2019-10-13T08:30:18.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:18.438+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.443023+0000) 2019-10-13T08:30:18.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:18.442+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.444648+0000) 2019-10-13T08:30:18.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:18.442+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.448186+0000) 2019-10-13T08:30:18.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:18.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.457685+0000) 2019-10-13T08:30:19.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:19.442+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:19.444862+0000) 2019-10-13T08:30:20.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:20.250+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:20.251262+0000) 2019-10-13T08:30:21.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:21.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:21.700109+0000) 2019-10-13T08:30:21.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:21.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:21.745886+0000) 2019-10-13T08:30:21.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:21.938+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:21.940337+0000) 2019-10-13T08:30:23.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:23.442+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.446000+0000) 2019-10-13T08:30:23.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:23.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:23.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:23.442+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.447013+0000) 2019-10-13T08:30:23.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:23.446+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.447567+0000) 2019-10-13T08:30:23.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:23.442+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.447202+0000) 2019-10-13T08:30:23.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:23.446+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.448077+0000) 2019-10-13T08:30:23.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:23.442+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.447786+0000) 2019-10-13T08:30:23.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:23.446+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.449102+0000) 2019-10-13T08:30:23.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:23.450+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.454733+0000) 2019-10-13T08:30:24.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:24.966+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:24.969073+0000) 2019-10-13T08:30:26.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:26.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:26.190614+0000) 2019-10-13T08:30:28.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:28.446+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.450688+0000) 2019-10-13T08:30:28.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:28.450+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.451339+0000) 2019-10-13T08:30:28.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:28.450+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.452020+0000) 2019-10-13T08:30:28.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:28.450+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.452308+0000) 2019-10-13T08:30:28.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:28.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:28.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:28.450+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.452778+0000) 2019-10-13T08:30:28.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:28.450+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.454228+0000) 2019-10-13T08:30:28.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:28.450+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.453127+0000) 2019-10-13T08:30:28.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:28.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.457804+0000) 2019-10-13T08:30:28.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:28.458+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.462689+0000) 2019-10-13T08:30:29.118 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:30:29.118 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:29.122 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:30:29.122 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:29.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:29.442+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:29.445117+0000) 2019-10-13T08:30:30.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:30.250+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:30.257736+0000) 2019-10-13T08:30:31.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:31.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:31.700312+0000) 2019-10-13T08:30:31.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:31.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:31.746086+0000) 2019-10-13T08:30:31.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:31.938+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:31.940572+0000) 2019-10-13T08:30:33.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:33.454+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.455793+0000) 2019-10-13T08:30:33.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:33.454+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.456050+0000) 2019-10-13T08:30:33.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:33.454+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.456318+0000) 2019-10-13T08:30:33.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:33.454+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.457295+0000) 2019-10-13T08:30:33.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:33.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:33.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:33.454+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.458426+0000) 2019-10-13T08:30:33.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:33.454+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.459215+0000) 2019-10-13T08:30:33.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:33.454+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.459592+0000) 2019-10-13T08:30:33.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:33.458+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.464728+0000) 2019-10-13T08:30:34.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:34.966+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:34.969270+0000) 2019-10-13T08:30:36.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:36.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:36.190807+0000) 2019-10-13T08:30:38.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:38.454+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.458004+0000) 2019-10-13T08:30:38.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:38.458+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.461200+0000) 2019-10-13T08:30:38.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:38.458+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.461563+0000) 2019-10-13T08:30:38.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:38.458+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.462529+0000) 2019-10-13T08:30:38.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:38.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:38.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:38.462+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.463516+0000) 2019-10-13T08:30:38.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:38.462+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.463436+0000) 2019-10-13T08:30:38.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:38.462+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.464074+0000) 2019-10-13T08:30:38.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:38.458+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.465313+0000) 2019-10-13T08:30:38.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:38.462+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.468960+0000) 2019-10-13T08:30:39.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:39.442+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:39.445339+0000) 2019-10-13T08:30:40.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:40.262+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:40.264104+0000) 2019-10-13T08:30:41.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:41.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:41.700512+0000) 2019-10-13T08:30:41.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:41.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:41.746289+0000) 2019-10-13T08:30:41.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:41.937+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:41.940775+0000) 2019-10-13T08:30:43.466 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:43.461+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.466177+0000) 2019-10-13T08:30:43.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:43.461+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.466977+0000) 2019-10-13T08:30:43.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:43.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.467607+0000) 2019-10-13T08:30:43.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:43.465+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.468617+0000) 2019-10-13T08:30:43.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:43.465+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.468859+0000) 2019-10-13T08:30:43.469 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:43.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:43.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:43.465+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.469622+0000) 2019-10-13T08:30:43.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:43.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.470162+0000) 2019-10-13T08:30:43.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:43.470+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.476559+0000) 2019-10-13T08:30:44.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:44.965+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:44.969501+0000) 2019-10-13T08:30:46.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:46.189+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:46.191001+0000) 2019-10-13T08:30:48.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:48.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.458218+0000) 2019-10-13T08:30:48.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:48.469+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.472805+0000) 2019-10-13T08:30:48.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:48.469+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.473144+0000) 2019-10-13T08:30:48.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:48.469+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.474178+0000) 2019-10-13T08:30:48.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:48.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:48.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:48.469+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.474614+0000) 2019-10-13T08:30:48.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:48.469+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.475052+0000) 2019-10-13T08:30:48.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:48.473+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.475713+0000) 2019-10-13T08:30:48.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:48.470+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.476240+0000) 2019-10-13T08:30:48.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:48.474+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.478801+0000) 2019-10-13T08:30:49.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:49.442+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:49.445534+0000) 2019-10-13T08:30:50.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:50.270+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:50.270487+0000) 2019-10-13T08:30:51.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:51.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:51.700713+0000) 2019-10-13T08:30:51.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:51.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:51.746483+0000) 2019-10-13T08:30:51.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:51.937+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:51.940975+0000) 2019-10-13T08:30:53.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:53.473+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.477920+0000) 2019-10-13T08:30:53.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:53.473+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.478665+0000) 2019-10-13T08:30:53.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:53.474+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.478864+0000) 2019-10-13T08:30:53.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:53.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:53.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:53.477+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.479737+0000) 2019-10-13T08:30:53.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:53.477+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.481986+0000) 2019-10-13T08:30:53.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:53.477+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.482822+0000) 2019-10-13T08:30:53.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:53.477+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.483266+0000) 2019-10-13T08:30:53.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:53.486+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.490538+0000) 2019-10-13T08:30:54.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:54.969+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:54.969704+0000) 2019-10-13T08:30:56.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:56.185+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:56.191189+0000) 2019-10-13T08:30:58.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:58.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.458417+0000) 2019-10-13T08:30:58.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:30:58.477+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.482947+0000) 2019-10-13T08:30:58.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:30:58.481+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.483345+0000) 2019-10-13T08:30:58.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:30:58.481+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:30:58.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:30:58.481+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.484256+0000) 2019-10-13T08:30:58.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:30:58.481+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.484489+0000) 2019-10-13T08:30:58.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:30:58.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.485984+0000) 2019-10-13T08:30:58.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:30:58.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.485259+0000) 2019-10-13T08:30:58.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:58.482+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.486409+0000) 2019-10-13T08:30:58.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:30:58.482+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.488937+0000) 2019-10-13T08:30:59.262 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:30:59.263 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:59.266 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:30:59.267 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:59.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:30:59.442+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:59.445741+0000) 2019-10-13T08:31:00.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:00.274+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:00.276839+0000) 2019-10-13T08:31:01.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:01.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:01.700868+0000) 2019-10-13T08:31:01.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:01.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:01.746680+0000) 2019-10-13T08:31:01.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:01.937+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:01.941147+0000) 2019-10-13T08:31:03.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:03.485+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.487293+0000) 2019-10-13T08:31:03.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:03.485+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.487661+0000) 2019-10-13T08:31:03.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:03.485+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.487731+0000) 2019-10-13T08:31:03.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:03.485+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.488175+0000) 2019-10-13T08:31:03.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:03.485+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.488419+0000) 2019-10-13T08:31:03.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:03.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:03.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:03.486+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.488982+0000) 2019-10-13T08:31:03.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:03.485+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.489140+0000) 2019-10-13T08:31:03.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:03.494+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.496901+0000) 2019-10-13T08:31:04.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:04.965+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:04.969948+0000) 2019-10-13T08:31:06.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:06.189+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:06.191377+0000) 2019-10-13T08:31:08.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:08.453+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.458613+0000) 2019-10-13T08:31:08.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:08.489+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.492614+0000) 2019-10-13T08:31:08.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:08.489+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.492975+0000) 2019-10-13T08:31:08.493 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:08.489+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.493335+0000) 2019-10-13T08:31:08.494 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:08.489+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.493643+0000) 2019-10-13T08:31:08.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:08.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53379 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:08.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:08.489+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.494808+0000) 2019-10-13T08:31:08.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:08.494+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.495147+0000) 2019-10-13T08:31:08.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:08.493+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.495807+0000) 2019-10-13T08:31:08.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:08.498+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.501723+0000) 2019-10-13T08:31:09.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:09.441+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:09.445938+0000) 2019-10-13T08:31:10.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:10.281+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:10.283342+0000) 2019-10-13T08:31:11.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:11.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:11.701094+0000) 2019-10-13T08:31:11.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:11.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:11.746931+0000) 2019-10-13T08:31:11.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:11.937+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:11.941334+0000) 2019-10-13T08:31:13.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:13.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.498085+0000) 2019-10-13T08:31:13.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:13.497+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.498463+0000) 2019-10-13T08:31:13.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:13.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.498950+0000) 2019-10-13T08:31:13.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:13.497+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.499710+0000) 2019-10-13T08:31:13.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:13.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:13.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:13.497+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.499969+0000) 2019-10-13T08:31:13.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:13.497+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.500244+0000) 2019-10-13T08:31:13.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:13.497+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.501092+0000) 2019-10-13T08:31:13.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:13.505+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.508130+0000) 2019-10-13T08:31:14.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:14.965+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:14.970172+0000) 2019-10-13T08:31:16.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:16.189+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:16.191628+0000) 2019-10-13T08:31:18.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:18.457+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.458813+0000) 2019-10-13T08:31:18.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:18.501+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.503455+0000) 2019-10-13T08:31:18.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:18.501+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.504129+0000) 2019-10-13T08:31:18.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:18.501+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.504386+0000) 2019-10-13T08:31:18.505 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:18.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:18.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:18.501+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.504858+0000) 2019-10-13T08:31:18.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:18.501+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.505387+0000) 2019-10-13T08:31:18.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:18.501+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.505634+0000) 2019-10-13T08:31:18.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:18.501+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.506036+0000) 2019-10-13T08:31:18.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:18.513+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.514948+0000) 2019-10-13T08:31:19.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:19.445+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:19.446188+0000) 2019-10-13T08:31:20.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:20.285+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:20.289691+0000) 2019-10-13T08:31:21.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:21.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:21.701317+0000) 2019-10-13T08:31:21.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:21.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:21.747088+0000) 2019-10-13T08:31:21.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:21.941+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:21.941523+0000) 2019-10-13T08:31:23.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:23.505+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.508330+0000) 2019-10-13T08:31:23.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:23.504+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.508216+0000) 2019-10-13T08:31:23.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:23.504+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.508799+0000) 2019-10-13T08:31:23.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:23.504+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.509202+0000) 2019-10-13T08:31:23.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:23.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:23.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:23.504+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.509567+0000) 2019-10-13T08:31:23.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:23.504+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.510176+0000) 2019-10-13T08:31:23.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:23.504+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.511045+0000) 2019-10-13T08:31:23.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:23.513+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.516993+0000) 2019-10-13T08:31:24.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:24.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:24.970369+0000) 2019-10-13T08:31:26.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:26.188+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:26.191835+0000) 2019-10-13T08:31:28.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:28.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.459031+0000) 2019-10-13T08:31:28.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:28.508+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.513034+0000) 2019-10-13T08:31:28.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:28.508+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.513422+0000) 2019-10-13T08:31:28.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:28.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.513552+0000) 2019-10-13T08:31:28.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:28.508+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.514131+0000) 2019-10-13T08:31:28.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:28.508+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.514598+0000) 2019-10-13T08:31:28.515 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:28.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:28.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:28.508+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.515249+0000) 2019-10-13T08:31:28.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:28.512+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.515939+0000) 2019-10-13T08:31:28.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:28.517+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.521420+0000) 2019-10-13T08:31:29.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:29.445+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:29.446430+0000) 2019-10-13T08:31:29.565 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:31:29.565 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:29.569 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:31:29.569 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:30.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:30.293+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:30.296174+0000) 2019-10-13T08:31:31.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:31.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:31.701522+0000) 2019-10-13T08:31:31.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:31.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:31.747330+0000) 2019-10-13T08:31:31.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:31.940+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:31.941737+0000) 2019-10-13T08:31:33.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:33.512+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.517990+0000) 2019-10-13T08:31:33.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:33.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53549 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:33.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:33.516+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.519581+0000) 2019-10-13T08:31:33.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:33.516+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.519775+0000) 2019-10-13T08:31:33.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:33.517+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.520406+0000) 2019-10-13T08:31:33.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:33.516+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.520131+0000) 2019-10-13T08:31:33.522 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:33.516+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.521059+0000) 2019-10-13T08:31:33.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:33.516+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.521418+0000) 2019-10-13T08:31:33.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:33.521+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.525034+0000) 2019-10-13T08:31:34.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:34.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:34.970567+0000) 2019-10-13T08:31:36.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:36.188+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:36.192040+0000) 2019-10-13T08:31:38.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:38.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.459236+0000) 2019-10-13T08:31:38.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:38.516+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.522805+0000) 2019-10-13T08:31:38.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:38.516+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.523230+0000) 2019-10-13T08:31:38.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:38.520+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.523806+0000) 2019-10-13T08:31:38.524 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:38.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.524323+0000) 2019-10-13T08:31:38.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:38.521+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.524618+0000) 2019-10-13T08:31:38.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:38.520+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.524999+0000) 2019-10-13T08:31:38.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:38.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:38.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:38.520+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.525768+0000) 2019-10-13T08:31:38.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:38.529+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.531841+0000) 2019-10-13T08:31:39.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:39.445+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:39.446586+0000) 2019-10-13T08:31:40.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:40.301+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:40.302542+0000) 2019-10-13T08:31:41.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:41.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:41.701746+0000) 2019-10-13T08:31:41.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:41.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:41.747577+0000) 2019-10-13T08:31:41.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:41.936+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:41.941965+0000) 2019-10-13T08:31:43.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:43.524+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.529498+0000) 2019-10-13T08:31:43.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:43.524+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.529218+0000) 2019-10-13T08:31:43.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:43.524+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.530203+0000) 2019-10-13T08:31:43.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:43.524+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.530261+0000) 2019-10-13T08:31:43.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:43.529+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.531110+0000) 2019-10-13T08:31:43.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:43.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53616 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:43.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:43.528+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.531429+0000) 2019-10-13T08:31:43.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:43.528+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.532598+0000) 2019-10-13T08:31:43.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:43.533+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.534546+0000) 2019-10-13T08:31:44.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:44.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:44.970780+0000) 2019-10-13T08:31:46.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:46.188+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:46.192226+0000) 2019-10-13T08:31:48.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:48.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.459479+0000) 2019-10-13T08:31:48.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:48.528+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.533159+0000) 2019-10-13T08:31:48.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:48.528+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.535153+0000) 2019-10-13T08:31:48.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:48.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:48.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:48.532+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.536277+0000) 2019-10-13T08:31:48.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:48.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.536303+0000) 2019-10-13T08:31:48.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:48.532+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.536697+0000) 2019-10-13T08:31:48.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:48.532+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.537165+0000) 2019-10-13T08:31:48.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:48.532+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.538182+0000) 2019-10-13T08:31:48.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:48.541+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.544877+0000) 2019-10-13T08:31:49.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:49.445+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:49.446774+0000) 2019-10-13T08:31:50.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:50.305+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:50.309180+0000) 2019-10-13T08:31:51.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:51.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:51.701969+0000) 2019-10-13T08:31:51.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:51.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:51.747814+0000) 2019-10-13T08:31:51.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:51.936+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:51.942160+0000) 2019-10-13T08:31:53.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:53.536+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.539557+0000) 2019-10-13T08:31:53.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:53.536+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.539739+0000) 2019-10-13T08:31:53.541 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:53.536+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.539989+0000) 2019-10-13T08:31:53.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:53.536+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.540425+0000) 2019-10-13T08:31:53.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:53.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:53.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:53.536+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.540797+0000) 2019-10-13T08:31:53.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:53.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.542475+0000) 2019-10-13T08:31:53.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:53.536+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.543056+0000) 2019-10-13T08:31:53.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:53.545+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.547465+0000) 2019-10-13T08:31:54.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:54.964+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:54.970963+0000) 2019-10-13T08:31:56.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:56.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:56.192457+0000) 2019-10-13T08:31:58.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:58.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.459681+0000) 2019-10-13T08:31:58.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:31:58.540+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.543753+0000) 2019-10-13T08:31:58.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:31:58.540+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.544460+0000) 2019-10-13T08:31:58.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:31:58.540+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.545071+0000) 2019-10-13T08:31:58.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:31:58.540+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.546039+0000) 2019-10-13T08:31:58.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:31:58.540+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:31:58.547 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:31:58.540+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.546826+0000) 2019-10-13T08:31:58.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:31:58.540+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.547118+0000) 2019-10-13T08:31:58.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:58.545+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.547390+0000) 2019-10-13T08:31:58.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:31:58.549+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.552140+0000) 2019-10-13T08:31:59.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:31:59.445+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:59.446977+0000) 2019-10-13T08:31:59.669 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:31:59.669 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:59.672 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:31:59.672 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:00.316 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:00.313+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:00.315577+0000) 2019-10-13T08:32:01.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:01.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:01.702172+0000) 2019-10-13T08:32:01.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:01.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:01.748042+0000) 2019-10-13T08:32:01.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:01.940+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:01.942359+0000) 2019-10-13T08:32:03.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:03.544+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.549907+0000) 2019-10-13T08:32:03.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:03.544+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.550257+0000) 2019-10-13T08:32:03.551 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:03.544+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.550580+0000) 2019-10-13T08:32:03.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:03.544+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.550920+0000) 2019-10-13T08:32:03.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:03.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:03.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:03.548+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.552037+0000) 2019-10-13T08:32:03.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:03.548+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.552629+0000) 2019-10-13T08:32:03.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:03.549+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.553448+0000) 2019-10-13T08:32:03.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:03.557+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.561375+0000) 2019-10-13T08:32:04.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:04.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:04.971162+0000) 2019-10-13T08:32:06.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:06.187+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:06.192659+0000) 2019-10-13T08:32:08.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:08.455+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.459896+0000) 2019-10-13T08:32:08.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:08.551+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.555448+0000) 2019-10-13T08:32:08.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:08.551+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.555862+0000) 2019-10-13T08:32:08.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:08.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.556239+0000) 2019-10-13T08:32:08.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:08.552+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.556511+0000) 2019-10-13T08:32:08.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:08.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:08.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:08.551+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.558153+0000) 2019-10-13T08:32:08.559 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:08.551+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.558234+0000) 2019-10-13T08:32:08.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:08.551+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.558866+0000) 2019-10-13T08:32:08.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:08.560+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.564570+0000) 2019-10-13T08:32:09.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:09.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:09.447153+0000) 2019-10-13T08:32:10.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:10.320+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:10.322199+0000) 2019-10-13T08:32:11.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:11.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:11.702375+0000) 2019-10-13T08:32:11.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:11.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:11.748275+0000) 2019-10-13T08:32:11.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:11.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:11.942561+0000) 2019-10-13T08:32:13.563 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:13.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.561671+0000) 2019-10-13T08:32:13.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:13.555+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.561051+0000) 2019-10-13T08:32:13.564 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:13.555+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.562074+0000) 2019-10-13T08:32:13.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:13.555+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.562789+0000) 2019-10-13T08:32:13.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:13.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:13.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:13.555+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.563205+0000) 2019-10-13T08:32:13.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:13.559+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.563541+0000) 2019-10-13T08:32:13.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:13.559+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.564241+0000) 2019-10-13T08:32:13.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:13.564+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.567992+0000) 2019-10-13T08:32:14.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:14.967+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:14.971390+0000) 2019-10-13T08:32:16.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:16.187+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:16.192851+0000) 2019-10-13T08:32:18.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:18.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.460111+0000) 2019-10-13T08:32:18.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:18.559+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.566095+0000) 2019-10-13T08:32:18.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:18.564+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.567212+0000) 2019-10-13T08:32:18.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:18.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:18.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:18.563+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.567840+0000) 2019-10-13T08:32:18.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:18.563+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.568161+0000) 2019-10-13T08:32:18.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:18.563+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.568544+0000) 2019-10-13T08:32:18.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:18.563+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.568769+0000) 2019-10-13T08:32:18.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:18.563+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.570069+0000) 2019-10-13T08:32:18.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:18.572+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.577834+0000) 2019-10-13T08:32:19.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:19.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:19.447330+0000) 2019-10-13T08:32:20.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:20.324+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:20.328611+0000) 2019-10-13T08:32:21.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:21.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:21.702611+0000) 2019-10-13T08:32:21.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:21.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:21.748494+0000) 2019-10-13T08:32:21.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:21.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:21.942789+0000) 2019-10-13T08:32:23.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:23.567+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.571521+0000) 2019-10-13T08:32:23.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:23.568+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.571783+0000) 2019-10-13T08:32:23.574 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:23.567+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.572606+0000) 2019-10-13T08:32:23.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:23.567+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.573064+0000) 2019-10-13T08:32:23.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:23.567+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.573391+0000) 2019-10-13T08:32:23.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:23.567+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.574006+0000) 2019-10-13T08:32:23.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:23.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:23.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:23.567+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.574861+0000) 2019-10-13T08:32:23.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:23.576+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.580831+0000) 2019-10-13T08:32:24.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:24.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:24.971622+0000) 2019-10-13T08:32:26.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:26.191+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:26.193068+0000) 2019-10-13T08:32:28.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:28.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.460315+0000) 2019-10-13T08:32:28.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:28.572+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.575926+0000) 2019-10-13T08:32:28.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:28.571+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.577068+0000) 2019-10-13T08:32:28.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:28.580+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.583792+0000) 2019-10-13T08:32:28.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:28.571+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.577646+0000) 2019-10-13T08:32:28.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:28.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:28.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:28.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.578085+0000) 2019-10-13T08:32:28.613 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:28.571+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.578518+0000) 2019-10-13T08:32:28.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:28.571+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.579090+0000) 2019-10-13T08:32:28.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:28.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.579546+0000) 2019-10-13T08:32:29.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:29.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:29.447507+0000) 2019-10-13T08:32:29.798 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:32:29.798 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:29.802 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:32:29.802 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:30.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:30.332+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:30.335125+0000) 2019-10-13T08:32:31.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:31.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:31.702815+0000) 2019-10-13T08:32:31.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:31.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:31.748696+0000) 2019-10-13T08:32:31.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:31.935+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:31.942939+0000) 2019-10-13T08:32:33.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:33.575+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.581600+0000) 2019-10-13T08:32:33.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:33.575+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.582008+0000) 2019-10-13T08:32:33.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:33.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:33.584 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:33.579+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.584558+0000) 2019-10-13T08:32:33.585 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:33.580+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.584653+0000) 2019-10-13T08:32:33.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:33.579+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.584966+0000) 2019-10-13T08:32:33.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:33.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.585293+0000) 2019-10-13T08:32:33.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:33.579+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.586482+0000) 2019-10-13T08:32:33.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:33.588+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.590732+0000) 2019-10-13T08:32:34.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:34.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:34.971828+0000) 2019-10-13T08:32:36.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:36.191+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:36.193260+0000) 2019-10-13T08:32:38.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:38.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.460515+0000) 2019-10-13T08:32:38.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:38.583+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.586417+0000) 2019-10-13T08:32:38.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:38.583+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.586752+0000) 2019-10-13T08:32:38.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:38.587+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.587614+0000) 2019-10-13T08:32:38.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:38.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 53987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:38.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:38.587+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.588332+0000) 2019-10-13T08:32:38.589 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:38.584+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.588937+0000) 2019-10-13T08:32:38.590 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:38.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.589562+0000) 2019-10-13T08:32:38.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:38.587+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.589709+0000) 2019-10-13T08:32:38.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:38.592+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.597569+0000) 2019-10-13T08:32:39.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:39.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:39.447676+0000) 2019-10-13T08:32:40.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:40.336+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:40.341485+0000) 2019-10-13T08:32:41.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:41.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:41.703025+0000) 2019-10-13T08:32:41.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:41.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:41.748905+0000) 2019-10-13T08:32:41.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:41.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:41.943151+0000) 2019-10-13T08:32:43.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:43.587+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.590534+0000) 2019-10-13T08:32:43.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:43.591+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.591732+0000) 2019-10-13T08:32:43.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:43.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.592087+0000) 2019-10-13T08:32:43.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:43.591+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.592680+0000) 2019-10-13T08:32:43.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:43.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:43.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:43.591+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.592797+0000) 2019-10-13T08:32:43.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:43.591+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.593673+0000) 2019-10-13T08:32:43.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:43.591+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.594739+0000) 2019-10-13T08:32:43.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:43.600+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.603370+0000) 2019-10-13T08:32:44.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:44.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:44.972048+0000) 2019-10-13T08:32:46.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:46.191+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:46.193485+0000) 2019-10-13T08:32:48.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:48.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.460718+0000) 2019-10-13T08:32:48.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:48.595+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.596714+0000) 2019-10-13T08:32:48.598 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:48.595+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.597380+0000) 2019-10-13T08:32:48.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:48.595+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.597539+0000) 2019-10-13T08:32:48.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:48.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:48.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:48.595+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.597767+0000) 2019-10-13T08:32:48.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:48.595+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.598336+0000) 2019-10-13T08:32:48.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:48.596+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.599164+0000) 2019-10-13T08:32:48.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:48.599+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.600761+0000) 2019-10-13T08:32:48.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:48.608+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.610513+0000) 2019-10-13T08:32:49.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:49.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:49.447852+0000) 2019-10-13T08:32:50.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:50.344+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:50.348271+0000) 2019-10-13T08:32:51.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:51.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:51.703229+0000) 2019-10-13T08:32:51.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:51.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:51.749134+0000) 2019-10-13T08:32:51.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:51.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:51.943348+0000) 2019-10-13T08:32:53.602 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:53.598+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.601564+0000) 2019-10-13T08:32:53.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:53.598+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.601839+0000) 2019-10-13T08:32:53.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:53.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:53.603 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:53.600+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.602918+0000) 2019-10-13T08:32:53.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:53.602+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.603423+0000) 2019-10-13T08:32:53.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:53.602+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.603978+0000) 2019-10-13T08:32:53.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:53.602+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.604426+0000) 2019-10-13T08:32:53.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:53.602+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.605015+0000) 2019-10-13T08:32:53.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:53.604+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.609181+0000) 2019-10-13T08:32:54.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:54.970+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:54.972253+0000) 2019-10-13T08:32:56.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:56.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:56.193675+0000) 2019-10-13T08:32:58.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:58.458+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.460908+0000) 2019-10-13T08:32:58.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:32:58.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.606437+0000) 2019-10-13T08:32:58.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:32:58.602+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.606732+0000) 2019-10-13T08:32:58.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:32:58.606+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.607413+0000) 2019-10-13T08:32:58.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:32:58.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:32:58.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:58.604+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.608722+0000) 2019-10-13T08:32:58.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:32:58.606+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.609226+0000) 2019-10-13T08:32:58.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:32:58.606+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.610246+0000) 2019-10-13T08:32:58.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:32:58.610+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.611784+0000) 2019-10-13T08:32:58.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:32:58.612+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.615711+0000) 2019-10-13T08:32:59.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:32:59.444+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:59.448030+0000) 2019-10-13T08:33:00.133 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:33:00.134 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:00.137 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:33:00.137 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:00.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:00.351+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:00.354662+0000) 2019-10-13T08:33:01.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:01.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:01.703426+0000) 2019-10-13T08:33:01.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:01.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:01.749341+0000) 2019-10-13T08:33:01.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:01.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:01.943541+0000) 2019-10-13T08:33:03.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:03.610+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.612141+0000) 2019-10-13T08:33:03.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:03.610+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.612821+0000) 2019-10-13T08:33:03.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:03.610+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.613192+0000) 2019-10-13T08:33:03.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:03.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:03.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:03.610+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.613938+0000) 2019-10-13T08:33:03.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:03.610+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.613993+0000) 2019-10-13T08:33:03.615 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:03.611+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.614396+0000) 2019-10-13T08:33:03.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:03.610+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.615081+0000) 2019-10-13T08:33:03.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:03.615+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.618654+0000) 2019-10-13T08:33:04.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:04.970+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:04.972488+0000) 2019-10-13T08:33:06.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:06.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:06.193874+0000) 2019-10-13T08:33:08.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:08.458+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.461110+0000) 2019-10-13T08:33:08.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:08.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.617425+0000) 2019-10-13T08:33:08.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:08.614+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.618279+0000) 2019-10-13T08:33:08.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:08.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.618508+0000) 2019-10-13T08:33:08.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:08.614+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.618958+0000) 2019-10-13T08:33:08.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:08.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:08.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:08.618+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.619412+0000) 2019-10-13T08:33:08.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:08.618+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.620883+0000) 2019-10-13T08:33:08.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:08.615+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.621019+0000) 2019-10-13T08:33:08.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:08.619+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.623752+0000) 2019-10-13T08:33:09.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:09.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:09.448221+0000) 2019-10-13T08:33:10.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:10.355+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:10.361113+0000) 2019-10-13T08:33:11.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:11.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:11.703648+0000) 2019-10-13T08:33:11.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:11.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:11.749537+0000) 2019-10-13T08:33:11.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:11.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:11.943739+0000) 2019-10-13T08:33:13.623 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:13.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.622327+0000) 2019-10-13T08:33:13.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:13.618+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.622799+0000) 2019-10-13T08:33:13.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:13.618+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.623185+0000) 2019-10-13T08:33:13.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:13.622+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.623670+0000) 2019-10-13T08:33:13.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:13.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:13.628 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:13.622+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.625606+0000) 2019-10-13T08:33:13.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:13.622+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.626010+0000) 2019-10-13T08:33:13.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:13.622+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.626472+0000) 2019-10-13T08:33:13.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:13.627+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.632301+0000) 2019-10-13T08:33:14.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:14.970+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:14.972691+0000) 2019-10-13T08:33:16.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:16.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:16.194060+0000) 2019-10-13T08:33:18.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:18.458+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.461343+0000) 2019-10-13T08:33:18.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:18.626+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.627539+0000) 2019-10-13T08:33:18.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:18.626+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.628172+0000) 2019-10-13T08:33:18.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:18.626+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.629177+0000) 2019-10-13T08:33:18.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:18.626+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.629710+0000) 2019-10-13T08:33:18.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:18.626+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.629921+0000) 2019-10-13T08:33:18.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:18.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:18.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:18.627+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.630268+0000) 2019-10-13T08:33:18.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:18.626+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.630957+0000) 2019-10-13T08:33:18.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:18.635+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.638339+0000) 2019-10-13T08:33:19.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:19.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:19.448398+0000) 2019-10-13T08:33:20.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:20.363+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:20.367450+0000) 2019-10-13T08:33:21.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:21.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:21.703881+0000) 2019-10-13T08:33:21.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:21.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:21.749734+0000) 2019-10-13T08:33:21.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:21.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:21.943936+0000) 2019-10-13T08:33:23.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:23.630+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.632401+0000) 2019-10-13T08:33:23.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:23.630+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.634022+0000) 2019-10-13T08:33:23.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:23.630+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.634628+0000) 2019-10-13T08:33:23.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:23.630+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.634867+0000) 2019-10-13T08:33:23.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:23.630+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.635279+0000) 2019-10-13T08:33:23.636 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:23.631+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.635389+0000) 2019-10-13T08:33:23.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:23.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:23.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:23.634+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.637114+0000) 2019-10-13T08:33:23.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:23.639+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.644329+0000) 2019-10-13T08:33:24.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:24.970+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:24.972908+0000) 2019-10-13T08:33:26.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:26.190+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:26.194311+0000) 2019-10-13T08:33:28.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:28.458+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.461542+0000) 2019-10-13T08:33:28.639 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:28.634+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.639213+0000) 2019-10-13T08:33:28.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:28.638+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.639549+0000) 2019-10-13T08:33:28.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:28.638+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.640113+0000) 2019-10-13T08:33:28.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:28.638+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.640469+0000) 2019-10-13T08:33:28.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:28.638+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.640782+0000) 2019-10-13T08:33:28.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:28.635+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.641526+0000) 2019-10-13T08:33:28.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:28.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:28.643 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:28.638+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.642381+0000) 2019-10-13T08:33:28.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:28.643+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.646240+0000) 2019-10-13T08:33:29.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:29.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:29.448588+0000) 2019-10-13T08:33:30.206 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:33:30.206 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:30.210 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:33:30.210 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:30.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:30.371+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:30.374853+0000) 2019-10-13T08:33:31.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:31.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:31.704113+0000) 2019-10-13T08:33:31.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:31.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:31.749938+0000) 2019-10-13T08:33:31.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:31.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:31.944136+0000) 2019-10-13T08:33:33.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:33.639+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.644746+0000) 2019-10-13T08:33:33.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:33.641+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.646318+0000) 2019-10-13T08:33:33.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:33.641+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.646628+0000) 2019-10-13T08:33:33.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:33.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:33.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:33.641+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.647114+0000) 2019-10-13T08:33:33.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:33.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.647658+0000) 2019-10-13T08:33:33.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:33.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.648045+0000) 2019-10-13T08:33:33.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:33.645+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.649034+0000) 2019-10-13T08:33:33.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:33.647+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.651720+0000) 2019-10-13T08:33:34.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:34.969+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:34.973110+0000) 2019-10-13T08:33:36.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:36.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:36.194504+0000) 2019-10-13T08:33:38.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:38.457+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.461745+0000) 2019-10-13T08:33:38.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:38.645+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.650358+0000) 2019-10-13T08:33:38.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:38.647+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.650752+0000) 2019-10-13T08:33:38.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:38.649+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.651457+0000) 2019-10-13T08:33:38.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:38.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.651796+0000) 2019-10-13T08:33:38.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:38.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:38.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:38.649+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.652855+0000) 2019-10-13T08:33:38.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:38.649+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.653392+0000) 2019-10-13T08:33:38.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:38.649+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.654342+0000) 2019-10-13T08:33:38.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:38.651+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.658104+0000) 2019-10-13T08:33:39.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:39.447+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:39.448789+0000) 2019-10-13T08:33:40.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:40.375+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:40.381402+0000) 2019-10-13T08:33:41.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:41.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:41.704312+0000) 2019-10-13T08:33:41.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:41.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:41.750166+0000) 2019-10-13T08:33:41.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:41.941+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:41.944356+0000) 2019-10-13T08:33:43.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:43.649+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.655166+0000) 2019-10-13T08:33:43.657 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:43.653+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.656716+0000) 2019-10-13T08:33:43.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:43.653+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.656966+0000) 2019-10-13T08:33:43.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:43.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:43.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:43.653+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.658007+0000) 2019-10-13T08:33:43.658 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:43.653+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.657725+0000) 2019-10-13T08:33:43.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:43.655+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.658322+0000) 2019-10-13T08:33:43.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:43.657+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.659467+0000) 2019-10-13T08:33:43.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:43.659+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.664299+0000) 2019-10-13T08:33:44.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:44.969+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:44.973314+0000) 2019-10-13T08:33:46.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:46.189+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:46.194708+0000) 2019-10-13T08:33:48.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:48.457+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.461972+0000) 2019-10-13T08:33:48.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:48.657+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.660511+0000) 2019-10-13T08:33:48.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:48.657+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.662038+0000) 2019-10-13T08:33:48.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:48.657+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.662617+0000) 2019-10-13T08:33:48.663 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:48.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.663253+0000) 2019-10-13T08:33:48.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:48.659+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.663504+0000) 2019-10-13T08:33:48.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:48.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:48.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:48.661+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.663380+0000) 2019-10-13T08:33:48.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:48.661+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.664156+0000) 2019-10-13T08:33:48.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:48.663+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.668358+0000) 2019-10-13T08:33:49.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:49.443+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:49.448963+0000) 2019-10-13T08:33:50.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:50.387+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:50.387426+0000) 2019-10-13T08:33:51.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:51.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:51.704518+0000) 2019-10-13T08:33:51.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:51.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:51.750366+0000) 2019-10-13T08:33:51.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:51.941+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:51.944555+0000) 2019-10-13T08:33:53.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:53.661+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.666696+0000) 2019-10-13T08:33:53.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:53.661+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.667207+0000) 2019-10-13T08:33:53.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:53.665+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.667605+0000) 2019-10-13T08:33:53.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:53.665+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.667949+0000) 2019-10-13T08:33:53.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:53.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:53.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:53.665+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.668722+0000) 2019-10-13T08:33:53.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:53.665+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.668779+0000) 2019-10-13T08:33:53.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:53.663+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.670038+0000) 2019-10-13T08:33:53.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:53.671+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.676035+0000) 2019-10-13T08:33:54.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:54.973+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:54.973521+0000) 2019-10-13T08:33:56.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:56.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:56.194905+0000) 2019-10-13T08:33:58.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:58.457+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.462169+0000) 2019-10-13T08:33:58.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:33:58.665+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.671218+0000) 2019-10-13T08:33:58.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:33:58.669+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.671655+0000) 2019-10-13T08:33:58.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:33:58.669+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.672483+0000) 2019-10-13T08:33:58.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:33:58.669+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.672752+0000) 2019-10-13T08:33:58.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:33:58.669+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.673003+0000) 2019-10-13T08:33:58.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:58.666+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.673552+0000) 2019-10-13T08:33:58.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:33:58.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:33:58.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:33:58.669+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.674609+0000) 2019-10-13T08:33:58.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:33:58.674+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.680234+0000) 2019-10-13T08:33:59.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:33:59.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:59.449203+0000) 2019-10-13T08:34:00.346 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:34:00.346 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:00.349 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:34:00.350 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:00.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:00.386+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:00.393972+0000) 2019-10-13T08:34:01.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:01.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:01.704720+0000) 2019-10-13T08:34:01.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:01.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:01.750597+0000) 2019-10-13T08:34:01.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:01.941+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:01.944760+0000) 2019-10-13T08:34:03.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:03.673+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.676390+0000) 2019-10-13T08:34:03.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:03.673+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.677171+0000) 2019-10-13T08:34:03.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:03.673+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.677707+0000) 2019-10-13T08:34:03.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:03.673+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.678113+0000) 2019-10-13T08:34:03.678 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:03.673+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.678589+0000) 2019-10-13T08:34:03.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:03.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:03.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:03.674+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.679428+0000) 2019-10-13T08:34:03.680 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:03.677+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.680026+0000) 2019-10-13T08:34:03.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:03.682+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.687660+0000) 2019-10-13T08:34:04.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:04.973+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:04.973724+0000) 2019-10-13T08:34:06.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:06.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:06.195129+0000) 2019-10-13T08:34:08.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:08.461+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.462397+0000) 2019-10-13T08:34:08.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:08.677+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.682430+0000) 2019-10-13T08:34:08.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:08.677+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.682776+0000) 2019-10-13T08:34:08.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:08.678+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.683071+0000) 2019-10-13T08:34:08.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:08.681+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.683624+0000) 2019-10-13T08:34:08.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:08.681+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.683737+0000) 2019-10-13T08:34:08.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:08.681+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:08.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:08.681+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.685054+0000) 2019-10-13T08:34:08.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:08.681+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.685738+0000) 2019-10-13T08:34:08.691 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:08.686+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.691448+0000) 2019-10-13T08:34:09.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:09.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:09.449374+0000) 2019-10-13T08:34:10.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:10.402+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:10.404286+0000) 2019-10-13T08:34:11.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:11.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:11.704925+0000) 2019-10-13T08:34:11.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:11.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:11.750822+0000) 2019-10-13T08:34:11.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:11.941+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:11.944997+0000) 2019-10-13T08:34:13.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:13.685+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.687919+0000) 2019-10-13T08:34:13.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:13.682+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.688146+0000) 2019-10-13T08:34:13.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:13.685+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.688686+0000) 2019-10-13T08:34:13.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:13.685+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.689256+0000) 2019-10-13T08:34:13.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:13.685+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.689340+0000) 2019-10-13T08:34:13.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:13.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:13.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:13.685+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.690400+0000) 2019-10-13T08:34:13.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:13.685+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.690946+0000) 2019-10-13T08:34:13.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:13.690+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.697382+0000) 2019-10-13T08:34:14.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:14.969+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:14.973946+0000) 2019-10-13T08:34:16.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:16.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:16.195342+0000) 2019-10-13T08:34:18.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:18.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.462601+0000) 2019-10-13T08:34:18.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:18.688+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.693685+0000) 2019-10-13T08:34:18.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:18.688+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.694517+0000) 2019-10-13T08:34:18.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:18.690+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.694813+0000) 2019-10-13T08:34:18.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:18.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:18.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:18.688+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.695277+0000) 2019-10-13T08:34:18.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:18.692+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.695806+0000) 2019-10-13T08:34:18.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:18.692+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.696392+0000) 2019-10-13T08:34:18.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:18.692+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.696751+0000) 2019-10-13T08:34:18.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:18.698+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.703502+0000) 2019-10-13T08:34:19.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:19.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:19.449551+0000) 2019-10-13T08:34:20.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:20.410+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:20.410731+0000) 2019-10-13T08:34:21.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:21.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:21.705126+0000) 2019-10-13T08:34:21.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:21.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:21.751052+0000) 2019-10-13T08:34:21.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:21.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:21.945241+0000) 2019-10-13T08:34:23.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:23.692+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.698472+0000) 2019-10-13T08:34:23.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:23.692+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.699225+0000) 2019-10-13T08:34:23.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:23.696+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.699741+0000) 2019-10-13T08:34:23.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:23.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.699252+0000) 2019-10-13T08:34:23.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:23.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54700 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:23.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:23.694+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.701355+0000) 2019-10-13T08:34:23.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:23.696+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.701796+0000) 2019-10-13T08:34:23.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:23.696+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.702672+0000) 2019-10-13T08:34:23.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:23.702+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.706511+0000) 2019-10-13T08:34:24.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:24.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:24.974143+0000) 2019-10-13T08:34:26.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:26.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:26.195546+0000) 2019-10-13T08:34:28.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:28.456+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.462800+0000) 2019-10-13T08:34:28.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:28.700+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.703624+0000) 2019-10-13T08:34:28.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:28.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.703894+0000) 2019-10-13T08:34:28.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:28.700+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.704241+0000) 2019-10-13T08:34:28.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:28.700+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.705009+0000) 2019-10-13T08:34:28.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:28.700+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.705316+0000) 2019-10-13T08:34:28.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:28.698+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.705727+0000) 2019-10-13T08:34:28.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:28.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:28.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:28.704+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.707646+0000) 2019-10-13T08:34:28.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:28.706+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.711594+0000) 2019-10-13T08:34:29.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:29.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:29.449727+0000) 2019-10-13T08:34:30.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:30.414+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:30.417281+0000) 2019-10-13T08:34:30.435 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:34:30.435 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:30.439 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:34:30.439 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:31.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:31.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:31.705387+0000) 2019-10-13T08:34:31.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:31.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:31.751245+0000) 2019-10-13T08:34:31.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:31.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:31.945439+0000) 2019-10-13T08:34:33.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:33.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.709480+0000) 2019-10-13T08:34:33.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:33.704+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.709857+0000) 2019-10-13T08:34:33.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:33.704+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.710390+0000) 2019-10-13T08:34:33.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:33.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:33.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:33.704+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.710657+0000) 2019-10-13T08:34:33.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:33.708+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.711953+0000) 2019-10-13T08:34:33.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:33.708+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.712065+0000) 2019-10-13T08:34:33.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:33.706+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.712617+0000) 2019-10-13T08:34:33.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:33.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.717661+0000) 2019-10-13T08:34:34.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:34.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:34.974338+0000) 2019-10-13T08:34:36.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:36.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:36.195742+0000) 2019-10-13T08:34:38.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:38.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.463004+0000) 2019-10-13T08:34:38.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:38.708+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.714327+0000) 2019-10-13T08:34:38.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:38.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.714701+0000) 2019-10-13T08:34:38.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:38.708+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.714970+0000) 2019-10-13T08:34:38.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:38.710+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.715807+0000) 2019-10-13T08:34:38.717 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:38.712+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.716334+0000) 2019-10-13T08:34:38.717 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:38.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:38.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:38.712+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.717640+0000) 2019-10-13T08:34:38.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:38.712+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.718446+0000) 2019-10-13T08:34:38.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:38.718+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.725273+0000) 2019-10-13T08:34:39.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:39.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:39.449931+0000) 2019-10-13T08:34:40.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:40.426+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:40.428938+0000) 2019-10-13T08:34:41.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:41.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:41.705591+0000) 2019-10-13T08:34:41.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:41.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:41.751501+0000) 2019-10-13T08:34:41.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:41.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:41.945666+0000) 2019-10-13T08:34:43.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:43.716+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.720087+0000) 2019-10-13T08:34:43.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:43.716+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.720780+0000) 2019-10-13T08:34:43.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:43.716+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.721165+0000) 2019-10-13T08:34:43.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:43.716+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.721889+0000) 2019-10-13T08:34:43.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:43.722+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.722182+0000) 2019-10-13T08:34:43.723 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:43.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:43.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:43.720+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.723599+0000) 2019-10-13T08:34:43.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:43.716+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.722995+0000) 2019-10-13T08:34:43.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:43.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.729952+0000) 2019-10-13T08:34:44.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:44.972+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:44.974540+0000) 2019-10-13T08:34:46.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:46.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:46.195953+0000) 2019-10-13T08:34:48.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:48.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.463206+0000) 2019-10-13T08:34:48.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:48.722+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.725353+0000) 2019-10-13T08:34:48.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:48.720+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.726637+0000) 2019-10-13T08:34:48.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:48.720+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.727178+0000) 2019-10-13T08:34:48.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:48.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:48.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:48.724+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.727728+0000) 2019-10-13T08:34:48.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:48.724+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.727424+0000) 2019-10-13T08:34:48.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:48.724+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.728898+0000) 2019-10-13T08:34:48.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:48.724+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.728971+0000) 2019-10-13T08:34:48.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:48.730+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.733261+0000) 2019-10-13T08:34:49.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:49.450+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:49.450120+0000) 2019-10-13T08:34:50.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:50.434+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:50.438071+0000) 2019-10-13T08:34:51.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:51.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:51.705793+0000) 2019-10-13T08:34:51.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:51.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:51.751706+0000) 2019-10-13T08:34:51.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:51.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:51.945861+0000) 2019-10-13T08:34:53.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:53.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.730525+0000) 2019-10-13T08:34:53.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:53.724+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.731000+0000) 2019-10-13T08:34:53.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:53.728+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.731516+0000) 2019-10-13T08:34:53.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:53.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54905 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:53.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:53.728+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.731926+0000) 2019-10-13T08:34:53.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:53.728+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.732504+0000) 2019-10-13T08:34:53.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:53.728+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.733623+0000) 2019-10-13T08:34:53.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:53.729+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.732666+0000) 2019-10-13T08:34:53.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:53.741+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.742460+0000) 2019-10-13T08:34:54.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:54.968+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:54.974771+0000) 2019-10-13T08:34:56.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:56.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:56.196158+0000) 2019-10-13T08:34:58.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:58.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.463403+0000) 2019-10-13T08:34:58.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:34:58.732+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.735753+0000) 2019-10-13T08:34:58.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:34:58.732+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.735417+0000) 2019-10-13T08:34:58.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:34:58.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.736814+0000) 2019-10-13T08:34:58.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:34:58.732+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.737373+0000) 2019-10-13T08:34:58.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:34:58.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:34:58.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:34:58.732+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.737960+0000) 2019-10-13T08:34:58.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:34:58.732+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.738454+0000) 2019-10-13T08:34:58.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:58.737+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.738913+0000) 2019-10-13T08:34:58.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:34:58.741+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.743301+0000) 2019-10-13T08:34:59.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:34:59.449+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:59.450326+0000) 2019-10-13T08:35:00.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:00.441+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:00.444689+0000) 2019-10-13T08:35:00.583 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:35:00.583 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:00.587 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:35:00.587 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:01.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:01.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:01.706027+0000) 2019-10-13T08:35:01.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:01.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:01.751921+0000) 2019-10-13T08:35:01.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:01.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:01.946056+0000) 2019-10-13T08:35:03.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:03.735+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.741266+0000) 2019-10-13T08:35:03.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:03.737+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.741702+0000) 2019-10-13T08:35:03.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:03.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.742145+0000) 2019-10-13T08:35:03.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:03.735+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.742447+0000) 2019-10-13T08:35:03.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:03.735+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.743017+0000) 2019-10-13T08:35:03.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:03.735+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.743145+0000) 2019-10-13T08:35:03.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:03.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 54972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:03.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:03.739+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.743606+0000) 2019-10-13T08:35:03.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:03.749+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.751817+0000) 2019-10-13T08:35:04.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:04.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:04.974990+0000) 2019-10-13T08:35:06.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:06.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:06.196358+0000) 2019-10-13T08:35:08.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:08.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.463609+0000) 2019-10-13T08:35:08.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:08.739+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.746088+0000) 2019-10-13T08:35:08.747 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:08.739+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.746628+0000) 2019-10-13T08:35:08.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:08.739+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.747209+0000) 2019-10-13T08:35:08.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:08.743+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.747741+0000) 2019-10-13T08:35:08.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:08.743+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.748166+0000) 2019-10-13T08:35:08.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:08.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:08.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:08.745+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.748860+0000) 2019-10-13T08:35:08.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:08.743+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.750033+0000) 2019-10-13T08:35:08.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:08.753+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.756920+0000) 2019-10-13T08:35:09.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:09.449+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:09.450532+0000) 2019-10-13T08:35:10.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:10.449+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:10.451713+0000) 2019-10-13T08:35:11.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:11.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:11.706231+0000) 2019-10-13T08:35:11.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:11.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:11.752133+0000) 2019-10-13T08:35:11.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:11.939+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:11.946279+0000) 2019-10-13T08:35:13.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:13.753+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.754465+0000) 2019-10-13T08:35:13.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:13.747+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.751375+0000) 2019-10-13T08:35:13.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:13.747+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.753087+0000) 2019-10-13T08:35:13.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:13.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:13.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:13.747+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.753508+0000) 2019-10-13T08:35:13.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:13.747+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.754162+0000) 2019-10-13T08:35:13.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:13.747+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.753733+0000) 2019-10-13T08:35:13.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:13.747+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.755177+0000) 2019-10-13T08:35:13.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:13.757+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.759182+0000) 2019-10-13T08:35:14.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:14.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:14.975189+0000) 2019-10-13T08:35:16.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:16.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:16.196580+0000) 2019-10-13T08:35:18.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:18.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.463830+0000) 2019-10-13T08:35:18.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:18.751+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.756722+0000) 2019-10-13T08:35:18.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:18.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.757068+0000) 2019-10-13T08:35:18.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:18.753+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.757408+0000) 2019-10-13T08:35:18.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:18.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:18.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:18.751+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.758325+0000) 2019-10-13T08:35:18.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:18.751+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.759045+0000) 2019-10-13T08:35:18.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:18.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.759698+0000) 2019-10-13T08:35:18.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:18.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.760188+0000) 2019-10-13T08:35:18.766 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:18.761+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.765640+0000) 2019-10-13T08:35:19.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:19.449+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:19.450736+0000) 2019-10-13T08:35:20.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:20.457+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:20.458076+0000) 2019-10-13T08:35:21.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:21.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:21.706469+0000) 2019-10-13T08:35:21.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:21.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:21.752336+0000) 2019-10-13T08:35:21.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:21.943+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:21.946474+0000) 2019-10-13T08:35:23.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:23.755+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.761464+0000) 2019-10-13T08:35:23.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:23.755+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.762037+0000) 2019-10-13T08:35:23.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:23.759+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.763644+0000) 2019-10-13T08:35:23.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:23.759+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.763932+0000) 2019-10-13T08:35:23.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:23.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55108 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:23.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:23.761+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.764623+0000) 2019-10-13T08:35:23.766 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:23.759+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.765326+0000) 2019-10-13T08:35:23.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:23.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.766310+0000) 2019-10-13T08:35:23.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:23.765+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.769404+0000) 2019-10-13T08:35:24.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:24.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:24.975346+0000) 2019-10-13T08:35:26.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:26.191+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:26.196776+0000) 2019-10-13T08:35:28.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:28.463+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.464030+0000) 2019-10-13T08:35:28.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:28.763+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.768071+0000) 2019-10-13T08:35:28.769 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:28.765+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.768230+0000) 2019-10-13T08:35:28.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:28.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:28.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:28.763+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.769372+0000) 2019-10-13T08:35:28.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:28.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.769920+0000) 2019-10-13T08:35:28.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:28.763+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.770815+0000) 2019-10-13T08:35:28.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:28.767+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.771564+0000) 2019-10-13T08:35:28.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:28.767+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.772990+0000) 2019-10-13T08:35:28.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:28.773+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.777350+0000) 2019-10-13T08:35:29.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:29.449+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:29.450936+0000) 2019-10-13T08:35:30.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:30.461+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:30.464609+0000) 2019-10-13T08:35:30.698 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:35:30.699 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:30.703 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:35:30.703 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:31.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:31.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:31.706702+0000) 2019-10-13T08:35:31.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:31.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:31.752558+0000) 2019-10-13T08:35:31.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:31.943+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:31.946705+0000) 2019-10-13T08:35:33.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:33.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.772746+0000) 2019-10-13T08:35:33.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:33.771+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.773173+0000) 2019-10-13T08:35:33.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:33.771+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.773269+0000) 2019-10-13T08:35:33.775 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:33.773+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.774839+0000) 2019-10-13T08:35:33.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:33.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:33.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:33.775+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.775499+0000) 2019-10-13T08:35:33.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:33.775+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.775909+0000) 2019-10-13T08:35:33.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:33.775+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.776312+0000) 2019-10-13T08:35:33.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:33.781+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.783026+0000) 2019-10-13T08:35:34.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:34.975+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:34.975588+0000) 2019-10-13T08:35:36.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:36.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:36.196992+0000) 2019-10-13T08:35:38.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:38.463+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.464233+0000) 2019-10-13T08:35:38.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:38.775+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.779024+0000) 2019-10-13T08:35:38.780 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:38.777+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.779240+0000) 2019-10-13T08:35:38.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:38.779+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.779786+0000) 2019-10-13T08:35:38.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:38.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:38.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:38.779+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.780035+0000) 2019-10-13T08:35:38.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:38.779+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.780642+0000) 2019-10-13T08:35:38.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:38.779+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.781263+0000) 2019-10-13T08:35:38.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:38.779+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.781799+0000) 2019-10-13T08:35:38.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:38.785+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.788938+0000) 2019-10-13T08:35:39.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:39.449+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:39.451105+0000) 2019-10-13T08:35:40.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:40.469+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:40.471066+0000) 2019-10-13T08:35:41.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:41.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:41.706966+0000) 2019-10-13T08:35:41.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:41.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:41.752767+0000) 2019-10-13T08:35:41.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:41.943+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:41.946911+0000) 2019-10-13T08:35:43.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:43.782+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.783991+0000) 2019-10-13T08:35:43.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:43.782+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.784253+0000) 2019-10-13T08:35:43.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:43.782+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.784875+0000) 2019-10-13T08:35:43.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:43.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:43.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:43.782+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.785356+0000) 2019-10-13T08:35:43.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:43.781+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.785646+0000) 2019-10-13T08:35:43.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:43.782+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.786547+0000) 2019-10-13T08:35:43.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:43.786+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.787964+0000) 2019-10-13T08:35:43.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:43.793+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.794332+0000) 2019-10-13T08:35:44.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:44.974+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:44.975793+0000) 2019-10-13T08:35:46.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:46.194+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:46.197233+0000) 2019-10-13T08:35:48.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:48.462+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.464432+0000) 2019-10-13T08:35:48.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:48.786+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.788235+0000) 2019-10-13T08:35:48.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:48.786+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.788747+0000) 2019-10-13T08:35:48.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:48.786+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.789006+0000) 2019-10-13T08:35:48.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:48.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:48.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:48.786+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.790704+0000) 2019-10-13T08:35:48.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:48.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.791198+0000) 2019-10-13T08:35:48.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:48.790+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.791778+0000) 2019-10-13T08:35:48.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:48.788+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.792357+0000) 2019-10-13T08:35:48.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:48.796+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.798385+0000) 2019-10-13T08:35:49.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:49.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:49.451280+0000) 2019-10-13T08:35:50.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:50.472+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:50.477530+0000) 2019-10-13T08:35:51.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:51.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:51.707166+0000) 2019-10-13T08:35:51.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:51.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:51.752987+0000) 2019-10-13T08:35:51.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:51.942+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:51.947111+0000) 2019-10-13T08:35:53.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:53.790+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.794577+0000) 2019-10-13T08:35:53.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:53.790+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.795103+0000) 2019-10-13T08:35:53.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:53.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:53.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:53.794+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.795408+0000) 2019-10-13T08:35:53.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:53.794+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.795777+0000) 2019-10-13T08:35:53.797 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:53.792+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.796106+0000) 2019-10-13T08:35:53.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:53.794+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.796849+0000) 2019-10-13T08:35:53.798 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:53.794+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.797489+0000) 2019-10-13T08:35:53.801 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:53.796+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.801080+0000) 2019-10-13T08:35:54.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:54.974+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:54.975997+0000) 2019-10-13T08:35:56.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:56.194+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:56.197424+0000) 2019-10-13T08:35:58.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:58.462+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.464649+0000) 2019-10-13T08:35:58.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:35:58.794+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.798282+0000) 2019-10-13T08:35:58.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:35:58.798+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.799437+0000) 2019-10-13T08:35:58.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:35:58.798+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.800104+0000) 2019-10-13T08:35:58.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:35:58.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:35:58.801 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:35:58.798+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.801211+0000) 2019-10-13T08:35:58.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:58.796+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.801918+0000) 2019-10-13T08:35:58.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:35:58.798+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.802297+0000) 2019-10-13T08:35:58.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:35:58.798+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.803136+0000) 2019-10-13T08:35:58.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:35:58.804+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.808582+0000) 2019-10-13T08:35:59.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:35:59.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:59.451454+0000) 2019-10-13T08:36:00.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:00.480+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:00.483990+0000) 2019-10-13T08:36:00.846 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:36:00.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:00.851 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:36:00.851 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:01.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:01.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:01.707375+0000) 2019-10-13T08:36:01.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:01.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:01.753216+0000) 2019-10-13T08:36:01.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:01.946+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:01.947344+0000) 2019-10-13T08:36:03.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:03.798+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.803190+0000) 2019-10-13T08:36:03.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:03.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.804086+0000) 2019-10-13T08:36:03.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:03.802+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.805091+0000) 2019-10-13T08:36:03.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:03.802+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.805483+0000) 2019-10-13T08:36:03.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:03.800+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.806070+0000) 2019-10-13T08:36:03.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:03.802+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.805906+0000) 2019-10-13T08:36:03.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:03.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:03.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:03.806+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.807298+0000) 2019-10-13T08:36:03.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:03.808+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.811138+0000) 2019-10-13T08:36:04.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:04.974+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:04.976202+0000) 2019-10-13T08:36:06.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:06.194+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:06.197642+0000) 2019-10-13T08:36:08.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:08.462+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.464884+0000) 2019-10-13T08:36:08.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:08.806+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.809333+0000) 2019-10-13T08:36:08.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:08.806+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.809988+0000) 2019-10-13T08:36:08.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:08.806+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.810810+0000) 2019-10-13T08:36:08.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:08.806+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.811070+0000) 2019-10-13T08:36:08.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:08.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:08.812 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:08.808+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.811316+0000) 2019-10-13T08:36:08.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:08.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.811653+0000) 2019-10-13T08:36:08.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:08.810+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.812250+0000) 2019-10-13T08:36:08.818 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:08.816+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.818278+0000) 2019-10-13T08:36:09.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:09.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:09.451628+0000) 2019-10-13T08:36:10.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:10.488+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:10.490469+0000) 2019-10-13T08:36:11.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:11.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:11.707563+0000) 2019-10-13T08:36:11.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:11.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:11.753431+0000) 2019-10-13T08:36:11.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:11.946+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:11.947541+0000) 2019-10-13T08:36:13.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:13.810+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.814078+0000) 2019-10-13T08:36:13.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:13.814+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.815547+0000) 2019-10-13T08:36:13.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:13.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:13.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:13.814+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.816827+0000) 2019-10-13T08:36:13.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:13.814+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.817156+0000) 2019-10-13T08:36:13.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:13.814+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.817626+0000) 2019-10-13T08:36:13.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:13.816+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.818112+0000) 2019-10-13T08:36:13.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:13.814+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.818844+0000) 2019-10-13T08:36:13.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:13.820+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.822555+0000) 2019-10-13T08:36:14.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:14.974+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:14.976401+0000) 2019-10-13T08:36:16.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:16.194+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:16.197896+0000) 2019-10-13T08:36:18.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:18.462+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.465145+0000) 2019-10-13T08:36:18.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:18.818+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.820308+0000) 2019-10-13T08:36:18.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:18.818+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.820625+0000) 2019-10-13T08:36:18.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:18.818+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.821313+0000) 2019-10-13T08:36:18.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:18.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55481 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:18.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:18.818+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.821731+0000) 2019-10-13T08:36:18.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:18.818+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.822345+0000) 2019-10-13T08:36:18.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:18.820+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.822741+0000) 2019-10-13T08:36:18.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:18.822+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.823525+0000) 2019-10-13T08:36:18.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:18.820+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.826042+0000) 2019-10-13T08:36:19.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:19.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:19.451806+0000) 2019-10-13T08:36:20.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:20.492+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:20.496988+0000) 2019-10-13T08:36:21.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:21.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:21.707765+0000) 2019-10-13T08:36:21.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:21.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:21.753631+0000) 2019-10-13T08:36:21.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:21.946+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:21.947740+0000) 2019-10-13T08:36:23.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:23.822+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.824569+0000) 2019-10-13T08:36:23.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:23.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.825534+0000) 2019-10-13T08:36:23.826 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:23.820+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.825954+0000) 2019-10-13T08:36:23.826 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:23.822+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.826384+0000) 2019-10-13T08:36:23.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:23.822+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.826717+0000) 2019-10-13T08:36:23.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:23.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:23.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:23.826+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.828029+0000) 2019-10-13T08:36:23.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:23.826+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.828567+0000) 2019-10-13T08:36:23.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:23.828+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.833785+0000) 2019-10-13T08:36:24.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:24.974+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:24.976626+0000) 2019-10-13T08:36:26.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:26.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:26.198086+0000) 2019-10-13T08:36:28.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:28.461+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.465329+0000) 2019-10-13T08:36:28.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:28.825+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.830046+0000) 2019-10-13T08:36:28.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:28.825+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.830549+0000) 2019-10-13T08:36:28.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:28.825+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.830285+0000) 2019-10-13T08:36:28.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:28.825+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.831264+0000) 2019-10-13T08:36:28.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:28.829+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.831698+0000) 2019-10-13T08:36:28.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:28.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:28.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:28.829+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.832222+0000) 2019-10-13T08:36:28.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:28.828+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.832976+0000) 2019-10-13T08:36:28.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:28.836+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.839640+0000) 2019-10-13T08:36:29.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:29.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:29.452031+0000) 2019-10-13T08:36:30.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:30.500+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:30.506058+0000) 2019-10-13T08:36:30.986 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:36:30.986 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:30.990 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:36:30.990 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:31.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:31.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:31.707992+0000) 2019-10-13T08:36:31.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:31.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:31.753855+0000) 2019-10-13T08:36:31.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:31.945+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:31.947965+0000) 2019-10-13T08:36:33.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:33.833+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.835913+0000) 2019-10-13T08:36:33.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:33.833+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.836196+0000) 2019-10-13T08:36:33.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:33.833+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.836479+0000) 2019-10-13T08:36:33.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:33.832+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.836776+0000) 2019-10-13T08:36:33.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:33.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:33.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:33.833+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.837937+0000) 2019-10-13T08:36:33.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:33.833+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.838778+0000) 2019-10-13T08:36:33.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:33.837+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.840092+0000) 2019-10-13T08:36:33.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:33.840+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.844047+0000) 2019-10-13T08:36:34.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:34.973+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:34.976828+0000) 2019-10-13T08:36:36.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:36.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:36.198304+0000) 2019-10-13T08:36:38.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:38.461+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.465524+0000) 2019-10-13T08:36:38.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:38.837+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.840550+0000) 2019-10-13T08:36:38.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:38.837+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.841275+0000) 2019-10-13T08:36:38.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:38.837+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.841845+0000) 2019-10-13T08:36:38.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:38.837+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.842386+0000) 2019-10-13T08:36:38.843 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:38.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:38.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:38.840+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.843396+0000) 2019-10-13T08:36:38.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:38.841+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.844314+0000) 2019-10-13T08:36:38.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:38.841+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.845154+0000) 2019-10-13T08:36:38.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:38.848+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.850331+0000) 2019-10-13T08:36:39.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:39.448+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:39.452201+0000) 2019-10-13T08:36:40.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:40.508+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:40.512685+0000) 2019-10-13T08:36:41.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:41.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:41.708194+0000) 2019-10-13T08:36:41.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:41.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:41.754099+0000) 2019-10-13T08:36:41.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:41.945+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:41.948165+0000) 2019-10-13T08:36:43.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:43.843+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.846228+0000) 2019-10-13T08:36:43.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:43.841+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.846550+0000) 2019-10-13T08:36:43.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:43.841+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.846961+0000) 2019-10-13T08:36:43.848 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:43.841+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.847204+0000) 2019-10-13T08:36:43.848 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:43.845+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.847667+0000) 2019-10-13T08:36:43.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:43.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:43.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:43.845+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.848042+0000) 2019-10-13T08:36:43.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:43.845+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.849337+0000) 2019-10-13T08:36:43.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:43.851+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.856528+0000) 2019-10-13T08:36:44.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:44.973+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:44.977030+0000) 2019-10-13T08:36:46.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:46.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:46.198492+0000) 2019-10-13T08:36:48.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:48.461+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.465750+0000) 2019-10-13T08:36:48.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:48.845+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.850996+0000) 2019-10-13T08:36:48.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:48.849+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.851644+0000) 2019-10-13T08:36:48.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:48.849+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:48.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:48.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.852787+0000) 2019-10-13T08:36:48.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:48.849+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.853375+0000) 2019-10-13T08:36:48.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:48.849+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.853522+0000) 2019-10-13T08:36:48.856 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:48.849+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.854270+0000) 2019-10-13T08:36:48.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:48.849+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.853852+0000) 2019-10-13T08:36:48.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:48.855+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.858624+0000) 2019-10-13T08:36:49.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:49.451+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:49.452389+0000) 2019-10-13T08:36:50.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:50.515+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:50.519097+0000) 2019-10-13T08:36:51.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:51.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:51.708397+0000) 2019-10-13T08:36:51.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:51.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:51.754264+0000) 2019-10-13T08:36:51.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:51.945+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:51.948372+0000) 2019-10-13T08:36:53.857 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:53.853+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.856532+0000) 2019-10-13T08:36:53.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:53.853+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.857220+0000) 2019-10-13T08:36:53.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:53.853+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.857628+0000) 2019-10-13T08:36:53.859 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:53.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:53.859 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:53.853+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.857865+0000) 2019-10-13T08:36:53.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:53.853+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.858795+0000) 2019-10-13T08:36:53.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:53.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.860688+0000) 2019-10-13T08:36:53.862 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:53.857+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.861842+0000) 2019-10-13T08:36:53.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:53.859+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.865759+0000) 2019-10-13T08:36:54.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:54.973+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:54.977238+0000) 2019-10-13T08:36:56.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:56.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:56.198684+0000) 2019-10-13T08:36:58.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:58.465+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.465936+0000) 2019-10-13T08:36:58.862 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:36:58.857+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.861172+0000) 2019-10-13T08:36:58.863 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:36:58.857+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.861915+0000) 2019-10-13T08:36:58.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:36:58.857+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.862349+0000) 2019-10-13T08:36:58.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:58.859+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.863108+0000) 2019-10-13T08:36:58.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:36:58.857+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.862793+0000) 2019-10-13T08:36:58.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:36:58.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:36:58.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:36:58.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.863773+0000) 2019-10-13T08:36:58.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:36:58.861+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.864603+0000) 2019-10-13T08:36:58.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:36:58.867+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.871480+0000) 2019-10-13T08:36:59.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:36:59.451+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:59.452576+0000) 2019-10-13T08:37:00.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:00.519+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:00.525512+0000) 2019-10-13T08:37:01.193 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:37:01.193 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:01.197 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:37:01.197 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:01.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:01.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:01.708604+0000) 2019-10-13T08:37:01.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:01.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:01.754492+0000) 2019-10-13T08:37:01.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:01.945+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:01.948582+0000) 2019-10-13T08:37:03.867 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:03.861+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.866492+0000) 2019-10-13T08:37:03.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:03.861+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.866901+0000) 2019-10-13T08:37:03.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:03.865+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.867554+0000) 2019-10-13T08:37:03.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:03.865+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:03.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:03.865+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.867785+0000) 2019-10-13T08:37:03.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:03.865+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.868620+0000) 2019-10-13T08:37:03.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:03.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.869021+0000) 2019-10-13T08:37:03.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:03.865+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.870271+0000) 2019-10-13T08:37:03.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:03.871+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.875003+0000) 2019-10-13T08:37:04.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:04.977+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:04.977484+0000) 2019-10-13T08:37:06.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:06.193+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:06.198877+0000) 2019-10-13T08:37:08.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:08.465+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.466135+0000) 2019-10-13T08:37:08.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:08.865+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.870961+0000) 2019-10-13T08:37:08.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:08.869+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.871950+0000) 2019-10-13T08:37:08.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:08.869+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.872096+0000) 2019-10-13T08:37:08.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:08.869+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.872537+0000) 2019-10-13T08:37:08.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:08.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:08.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:08.869+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.873255+0000) 2019-10-13T08:37:08.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:08.869+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.873934+0000) 2019-10-13T08:37:08.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:08.871+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.874556+0000) 2019-10-13T08:37:08.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:08.879+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.882326+0000) 2019-10-13T08:37:09.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:09.451+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:09.452749+0000) 2019-10-13T08:37:10.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:10.527+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:10.531926+0000) 2019-10-13T08:37:11.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:11.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:11.708795+0000) 2019-10-13T08:37:11.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:11.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:11.754704+0000) 2019-10-13T08:37:11.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:11.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:11.948777+0000) 2019-10-13T08:37:13.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:13.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.876762+0000) 2019-10-13T08:37:13.880 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:13.872+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.877459+0000) 2019-10-13T08:37:13.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:13.872+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.877861+0000) 2019-10-13T08:37:13.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:13.872+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.877277+0000) 2019-10-13T08:37:13.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:13.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:13.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:13.872+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.878450+0000) 2019-10-13T08:37:13.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:13.872+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.879070+0000) 2019-10-13T08:37:13.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:13.875+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.880041+0000) 2019-10-13T08:37:13.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:13.879+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.884250+0000) 2019-10-13T08:37:14.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:14.976+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:14.977686+0000) 2019-10-13T08:37:16.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:16.192+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:16.199127+0000) 2019-10-13T08:37:18.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:18.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.466343+0000) 2019-10-13T08:37:18.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:18.876+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.881387+0000) 2019-10-13T08:37:18.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:18.876+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.882301+0000) 2019-10-13T08:37:18.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:18.876+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.882856+0000) 2019-10-13T08:37:18.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:18.876+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.883221+0000) 2019-10-13T08:37:18.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:18.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.883793+0000) 2019-10-13T08:37:18.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:18.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:18.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:18.879+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.884355+0000) 2019-10-13T08:37:18.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:18.880+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.884657+0000) 2019-10-13T08:37:18.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:18.887+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.891408+0000) 2019-10-13T08:37:19.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:19.451+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:19.452922+0000) 2019-10-13T08:37:20.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:20.539+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:20.541420+0000) 2019-10-13T08:37:21.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:21.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:21.709025+0000) 2019-10-13T08:37:21.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:21.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:21.754885+0000) 2019-10-13T08:37:21.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:21.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:21.948970+0000) 2019-10-13T08:37:23.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:23.880+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.886933+0000) 2019-10-13T08:37:23.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:23.884+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.888542+0000) 2019-10-13T08:37:23.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:23.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.889094+0000) 2019-10-13T08:37:23.890 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:23.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:23.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:23.884+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.889506+0000) 2019-10-13T08:37:23.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:23.883+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.889760+0000) 2019-10-13T08:37:23.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:23.884+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.890252+0000) 2019-10-13T08:37:23.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:23.884+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.891018+0000) 2019-10-13T08:37:23.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:23.891+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.895449+0000) 2019-10-13T08:37:24.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:24.976+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:24.977926+0000) 2019-10-13T08:37:26.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:26.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:26.199377+0000) 2019-10-13T08:37:28.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:28.464+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.466558+0000) 2019-10-13T08:37:28.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:28.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.892668+0000) 2019-10-13T08:37:28.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:28.888+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.893002+0000) 2019-10-13T08:37:28.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:28.888+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.893788+0000) 2019-10-13T08:37:28.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:28.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:28.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:28.888+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.894221+0000) 2019-10-13T08:37:28.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:28.888+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.894735+0000) 2019-10-13T08:37:28.896 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:28.892+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.895475+0000) 2019-10-13T08:37:28.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:28.892+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.896218+0000) 2019-10-13T08:37:28.902 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:28.895+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.902110+0000) 2019-10-13T08:37:29.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:29.451+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:29.453099+0000) 2019-10-13T08:37:30.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:30.547+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:30.547848+0000) 2019-10-13T08:37:31.300 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:37:31.300 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:31.304 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:37:31.304 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:31.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:31.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:31.709244+0000) 2019-10-13T08:37:31.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:31.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:31.755111+0000) 2019-10-13T08:37:31.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:31.948+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:31.949190+0000) 2019-10-13T08:37:33.898 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:33.892+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.897937+0000) 2019-10-13T08:37:33.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:33.892+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.899108+0000) 2019-10-13T08:37:33.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:33.892+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.899269+0000) 2019-10-13T08:37:33.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:33.896+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.899787+0000) 2019-10-13T08:37:33.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:33.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 55992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:33.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:33.895+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.899909+0000) 2019-10-13T08:37:33.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:33.896+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.900457+0000) 2019-10-13T08:37:33.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:33.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.900956+0000) 2019-10-13T08:37:33.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:33.899+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.905047+0000) 2019-10-13T08:37:34.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:34.976+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:34.978156+0000) 2019-10-13T08:37:36.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:36.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:36.199569+0000) 2019-10-13T08:37:38.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:38.464+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.466781+0000) 2019-10-13T08:37:38.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:38.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.903117+0000) 2019-10-13T08:37:38.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:38.898+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.904399+0000) 2019-10-13T08:37:38.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:38.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56025 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:38.905 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:38.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.904857+0000) 2019-10-13T08:37:38.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:38.900+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.905072+0000) 2019-10-13T08:37:38.906 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:38.900+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.905316+0000) 2019-10-13T08:37:38.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:38.900+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.906653+0000) 2019-10-13T08:37:38.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:38.900+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.906926+0000) 2019-10-13T08:37:38.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:38.910+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.915190+0000) 2019-10-13T08:37:39.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:39.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:39.453326+0000) 2019-10-13T08:37:40.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:40.554+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:40.558908+0000) 2019-10-13T08:37:41.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:41.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:41.709450+0000) 2019-10-13T08:37:41.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:41.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:41.755346+0000) 2019-10-13T08:37:41.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:41.944+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:41.949399+0000) 2019-10-13T08:37:43.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:43.904+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.908242+0000) 2019-10-13T08:37:43.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:43.904+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.909173+0000) 2019-10-13T08:37:43.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:43.904+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.909647+0000) 2019-10-13T08:37:43.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:43.902+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.910063+0000) 2019-10-13T08:37:43.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:43.904+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.910497+0000) 2019-10-13T08:37:43.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:43.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:43.911 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:43.904+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.910879+0000) 2019-10-13T08:37:43.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:43.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.912346+0000) 2019-10-13T08:37:43.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:43.914+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.918643+0000) 2019-10-13T08:37:44.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:44.976+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:44.978358+0000) 2019-10-13T08:37:46.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:46.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:46.199761+0000) 2019-10-13T08:37:48.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:48.460+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.466980+0000) 2019-10-13T08:37:48.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:48.908+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.914402+0000) 2019-10-13T08:37:48.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:48.908+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.915191+0000) 2019-10-13T08:37:48.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:48.910+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.915472+0000) 2019-10-13T08:37:48.916 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:48.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:48.917 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:48.912+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.916331+0000) 2019-10-13T08:37:48.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:48.912+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.916862+0000) 2019-10-13T08:37:48.918 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:48.912+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.917307+0000) 2019-10-13T08:37:48.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:48.912+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.917385+0000) 2019-10-13T08:37:48.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:48.918+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.922398+0000) 2019-10-13T08:37:49.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:49.450+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:49.453526+0000) 2019-10-13T08:37:50.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:50.562+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:50.565350+0000) 2019-10-13T08:37:51.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:51.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:51.709653+0000) 2019-10-13T08:37:51.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:51.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:51.755579+0000) 2019-10-13T08:37:51.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:51.948+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:51.949617+0000) 2019-10-13T08:37:53.920 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:53.911+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.918977+0000) 2019-10-13T08:37:53.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:53.915+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.919673+0000) 2019-10-13T08:37:53.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:53.914+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.920058+0000) 2019-10-13T08:37:53.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:53.915+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.920757+0000) 2019-10-13T08:37:53.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:53.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:53.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:53.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.921509+0000) 2019-10-13T08:37:53.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:53.915+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.921718+0000) 2019-10-13T08:37:53.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:53.915+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.922189+0000) 2019-10-13T08:37:53.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:53.922+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.929089+0000) 2019-10-13T08:37:54.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:54.975+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:54.978585+0000) 2019-10-13T08:37:56.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:56.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:56.199987+0000) 2019-10-13T08:37:58.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:58.459+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.467177+0000) 2019-10-13T08:37:58.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:37:58.919+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.923946+0000) 2019-10-13T08:37:58.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:37:58.919+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.924726+0000) 2019-10-13T08:37:58.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:58.918+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.924902+0000) 2019-10-13T08:37:58.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:37:58.919+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.925449+0000) 2019-10-13T08:37:58.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:37:58.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:37:58.926 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:37:58.919+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.925648+0000) 2019-10-13T08:37:58.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:37:58.919+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.926264+0000) 2019-10-13T08:37:58.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:37:58.919+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.926833+0000) 2019-10-13T08:37:58.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:37:58.930+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.935604+0000) 2019-10-13T08:37:59.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:37:59.450+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:59.453700+0000) 2019-10-13T08:38:00.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:00.566+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:00.571858+0000) 2019-10-13T08:38:01.428 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:38:01.429 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:01.432 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:38:01.432 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:01.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:01.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:01.709861+0000) 2019-10-13T08:38:01.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:01.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:01.755780+0000) 2019-10-13T08:38:01.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:01.943+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:01.949808+0000) 2019-10-13T08:38:03.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:03.923+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.928658+0000) 2019-10-13T08:38:03.929 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:03.923+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.929137+0000) 2019-10-13T08:38:03.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:03.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.929600+0000) 2019-10-13T08:38:03.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:03.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:03.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:03.923+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.930811+0000) 2019-10-13T08:38:03.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:03.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.931758+0000) 2019-10-13T08:38:03.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:03.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.932187+0000) 2019-10-13T08:38:03.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:03.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.931724+0000) 2019-10-13T08:38:03.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:03.930+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.937260+0000) 2019-10-13T08:38:04.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:04.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:04.978813+0000) 2019-10-13T08:38:06.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:06.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:06.200252+0000) 2019-10-13T08:38:08.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:08.463+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.467382+0000) 2019-10-13T08:38:08.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:08.927+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.934189+0000) 2019-10-13T08:38:08.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:08.927+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.934697+0000) 2019-10-13T08:38:08.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:08.927+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.935023+0000) 2019-10-13T08:38:08.936 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:08.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:08.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:08.931+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.936096+0000) 2019-10-13T08:38:08.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:08.931+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.936695+0000) 2019-10-13T08:38:08.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:08.930+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.937247+0000) 2019-10-13T08:38:08.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:08.931+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.939033+0000) 2019-10-13T08:38:08.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:08.938+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.945936+0000) 2019-10-13T08:38:09.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:09.450+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:09.453853+0000) 2019-10-13T08:38:10.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:10.574+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:10.578394+0000) 2019-10-13T08:38:11.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:11.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:11.710059+0000) 2019-10-13T08:38:11.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:11.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:11.756004+0000) 2019-10-13T08:38:11.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:11.947+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:11.950028+0000) 2019-10-13T08:38:13.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:13.935+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.939250+0000) 2019-10-13T08:38:13.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:13.935+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.940084+0000) 2019-10-13T08:38:13.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:13.934+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.940477+0000) 2019-10-13T08:38:13.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:13.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:13.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:13.935+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.940953+0000) 2019-10-13T08:38:13.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:13.935+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.941751+0000) 2019-10-13T08:38:13.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:13.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.941966+0000) 2019-10-13T08:38:13.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:13.935+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.942664+0000) 2019-10-13T08:38:13.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:13.946+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.950410+0000) 2019-10-13T08:38:14.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:14.971+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:14.979043+0000) 2019-10-13T08:38:16.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:16.195+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:16.200445+0000) 2019-10-13T08:38:18.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:18.467+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.467591+0000) 2019-10-13T08:38:18.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:18.939+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.945282+0000) 2019-10-13T08:38:18.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:18.939+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.945686+0000) 2019-10-13T08:38:18.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:18.938+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.945985+0000) 2019-10-13T08:38:18.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:18.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56297 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:18.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:18.939+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.946419+0000) 2019-10-13T08:38:18.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:18.939+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.946701+0000) 2019-10-13T08:38:18.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:18.943+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.947593+0000) 2019-10-13T08:38:18.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:18.943+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.947678+0000) 2019-10-13T08:38:18.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:18.946+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.951500+0000) 2019-10-13T08:38:19.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:19.446+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:19.454021+0000) 2019-10-13T08:38:20.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:20.582+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:20.584915+0000) 2019-10-13T08:38:21.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:21.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:21.710264+0000) 2019-10-13T08:38:21.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:21.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:21.756238+0000) 2019-10-13T08:38:21.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:21.947+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:21.950227+0000) 2019-10-13T08:38:23.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:23.947+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.949842+0000) 2019-10-13T08:38:23.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:23.947+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.950047+0000) 2019-10-13T08:38:23.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:23.947+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.950714+0000) 2019-10-13T08:38:23.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:23.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:23.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:23.947+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.951104+0000) 2019-10-13T08:38:23.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:23.946+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.951387+0000) 2019-10-13T08:38:23.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:23.951+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.952485+0000) 2019-10-13T08:38:23.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:23.951+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.952801+0000) 2019-10-13T08:38:23.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:23.954+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.958488+0000) 2019-10-13T08:38:24.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:24.975+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:24.979238+0000) 2019-10-13T08:38:26.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:26.199+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:26.200669+0000) 2019-10-13T08:38:28.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:28.467+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.467798+0000) 2019-10-13T08:38:28.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:28.951+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.953789+0000) 2019-10-13T08:38:28.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:28.951+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.954630+0000) 2019-10-13T08:38:28.955 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:28.951+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.955252+0000) 2019-10-13T08:38:28.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:28.955+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.955674+0000) 2019-10-13T08:38:28.956 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:28.955+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.956194+0000) 2019-10-13T08:38:28.957 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:28.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:28.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:28.954+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.957446+0000) 2019-10-13T08:38:28.959 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:28.955+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.958159+0000) 2019-10-13T08:38:28.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:28.962+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.963075+0000) 2019-10-13T08:38:29.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:29.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:29.454197+0000) 2019-10-13T08:38:30.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:30.590+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:30.591535+0000) 2019-10-13T08:38:31.539 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:38:31.539 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:31.543 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:38:31.544 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:31.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:31.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:31.710445+0000) 2019-10-13T08:38:31.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:31.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:31.756469+0000) 2019-10-13T08:38:31.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:31.947+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:31.950450+0000) 2019-10-13T08:38:33.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:33.959+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.959495+0000) 2019-10-13T08:38:33.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:33.959+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.959592+0000) 2019-10-13T08:38:33.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:33.959+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.960219+0000) 2019-10-13T08:38:33.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:33.959+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.960498+0000) 2019-10-13T08:38:33.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:33.957+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.960942+0000) 2019-10-13T08:38:33.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:33.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.961455+0000) 2019-10-13T08:38:33.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:33.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56399 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:33.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:33.959+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.963127+0000) 2019-10-13T08:38:33.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:33.965+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.969115+0000) 2019-10-13T08:38:34.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:34.979+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:34.979498+0000) 2019-10-13T08:38:36.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:36.199+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:36.200866+0000) 2019-10-13T08:38:38.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:38.466+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.467997+0000) 2019-10-13T08:38:38.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:38.962+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.965585+0000) 2019-10-13T08:38:38.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:38.962+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.966013+0000) 2019-10-13T08:38:38.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:38.962+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.966377+0000) 2019-10-13T08:38:38.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:38.962+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56433 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:38.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:38.966+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.967638+0000) 2019-10-13T08:38:38.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:38.966+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.968048+0000) 2019-10-13T08:38:38.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:38.966+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.968968+0000) 2019-10-13T08:38:38.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:38.965+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.969303+0000) 2019-10-13T08:38:38.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:38.973+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.975173+0000) 2019-10-13T08:38:39.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:39.453+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:39.454367+0000) 2019-10-13T08:38:40.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:40.601+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:40.602818+0000) 2019-10-13T08:38:41.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:41.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:41.710628+0000) 2019-10-13T08:38:41.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:41.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:41.756714+0000) 2019-10-13T08:38:41.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:41.946+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:41.950688+0000) 2019-10-13T08:38:43.972 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:43.966+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.970965+0000) 2019-10-13T08:38:43.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:43.966+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.971276+0000) 2019-10-13T08:38:43.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:43.970+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.971958+0000) 2019-10-13T08:38:43.973 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:43.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:43.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:43.970+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.972731+0000) 2019-10-13T08:38:43.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:43.969+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.973855+0000) 2019-10-13T08:38:43.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:43.970+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.973239+0000) 2019-10-13T08:38:43.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:43.970+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.973031+0000) 2019-10-13T08:38:43.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:43.977+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.978319+0000) 2019-10-13T08:38:44.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:44.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:44.979726+0000) 2019-10-13T08:38:46.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:46.198+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:46.201062+0000) 2019-10-13T08:38:48.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:48.466+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.468228+0000) 2019-10-13T08:38:48.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:48.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.976109+0000) 2019-10-13T08:38:48.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:48.974+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.976829+0000) 2019-10-13T08:38:48.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:48.974+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.977822+0000) 2019-10-13T08:38:48.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:48.974+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.978249+0000) 2019-10-13T08:38:48.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:48.974+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:48.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:48.974+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.978680+0000) 2019-10-13T08:38:48.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:48.974+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.978903+0000) 2019-10-13T08:38:48.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:48.977+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.979150+0000) 2019-10-13T08:38:48.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:48.981+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.983605+0000) 2019-10-13T08:38:49.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:49.453+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:49.454540+0000) 2019-10-13T08:38:50.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:50.605+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:50.609198+0000) 2019-10-13T08:38:51.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:51.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:51.710817+0000) 2019-10-13T08:38:51.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:51.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:51.756925+0000) 2019-10-13T08:38:51.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:51.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:51.950883+0000) 2019-10-13T08:38:53.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:53.978+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.982000+0000) 2019-10-13T08:38:53.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:53.978+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.982943+0000) 2019-10-13T08:38:53.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:53.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:53.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:53.982+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.983613+0000) 2019-10-13T08:38:53.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:53.982+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.983607+0000) 2019-10-13T08:38:53.985 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:53.982+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.984163+0000) 2019-10-13T08:38:53.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:53.982+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.984523+0000) 2019-10-13T08:38:53.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:53.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.985512+0000) 2019-10-13T08:38:53.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:53.989+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.993101+0000) 2019-10-13T08:38:54.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:54.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:54.979949+0000) 2019-10-13T08:38:56.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:56.198+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:56.201291+0000) 2019-10-13T08:38:58.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:58.466+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.468425+0000) 2019-10-13T08:38:58.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:38:58.982+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.986627+0000) 2019-10-13T08:38:58.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:38:58.986+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.988032+0000) 2019-10-13T08:38:58.989 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:38:58.986+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.988450+0000) 2019-10-13T08:38:58.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:38:58.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:38:58.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:58.985+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.988959+0000) 2019-10-13T08:38:58.990 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:38:58.986+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.989353+0000) 2019-10-13T08:38:58.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:38:58.986+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.990358+0000) 2019-10-13T08:38:58.991 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:38:58.986+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.991058+0000) 2019-10-13T08:38:58.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:38:58.993+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.996052+0000) 2019-10-13T08:38:59.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:38:59.453+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:59.454744+0000) 2019-10-13T08:39:00.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:00.613+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:00.615641+0000) 2019-10-13T08:39:01.662 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:39:01.662 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:01.665 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:39:01.665 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:01.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:01.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:01.711016+0000) 2019-10-13T08:39:01.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:01.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:01.757131+0000) 2019-10-13T08:39:01.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:01.946+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:01.951074+0000) 2019-10-13T08:39:03.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:03.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.991737+0000) 2019-10-13T08:39:03.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:03.990+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.992178+0000) 2019-10-13T08:39:03.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:03.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.992899+0000) 2019-10-13T08:39:03.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:03.989+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.993280+0000) 2019-10-13T08:39:03.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:03.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:03.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:03.990+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.994923+0000) 2019-10-13T08:39:03.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:03.990+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.994645+0000) 2019-10-13T08:39:03.998 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:03.994+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.997642+0000) 2019-10-13T08:39:04.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:03.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:04.000280+0000) 2019-10-13T08:39:04.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:04.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:04.980169+0000) 2019-10-13T08:39:06.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:06.198+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:06.201430+0000) 2019-10-13T08:39:08.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:08.466+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.468631+0000) 2019-10-13T08:39:08.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:08.994+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.996267+0000) 2019-10-13T08:39:08.998 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:08.994+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.997597+0000) 2019-10-13T08:39:08.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:08.994+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.998412+0000) 2019-10-13T08:39:08.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:08.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:08.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:08.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.999038+0000) 2019-10-13T08:39:09.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:08.998+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.999564+0000) 2019-10-13T08:39:09.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:08.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:09.000115+0000) 2019-10-13T08:39:09.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:08.998+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.999785+0000) 2019-10-13T08:39:09.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:09.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:09.006479+0000) 2019-10-13T08:39:09.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:09.453+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:09.454957+0000) 2019-10-13T08:39:10.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:10.621+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:10.622206+0000) 2019-10-13T08:39:11.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:11.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:11.711229+0000) 2019-10-13T08:39:11.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:11.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:11.757338+0000) 2019-10-13T08:39:11.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:11.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:11.951277+0000) 2019-10-13T08:39:14.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:13.998+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.001663+0000) 2019-10-13T08:39:14.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:13.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.002904+0000) 2019-10-13T08:39:14.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:14.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:14.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:14.001+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.003647+0000) 2019-10-13T08:39:14.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:14.002+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.004506+0000) 2019-10-13T08:39:14.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:14.002+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.004210+0000) 2019-10-13T08:39:14.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:14.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.005153+0000) 2019-10-13T08:39:14.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:14.002+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.006821+0000) 2019-10-13T08:39:14.010 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:14.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.009676+0000) 2019-10-13T08:39:14.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:14.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.980363+0000) 2019-10-13T08:39:16.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:16.198+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:16.201605+0000) 2019-10-13T08:39:18.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:18.466+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.468835+0000) 2019-10-13T08:39:19.008 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:19.006+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.007390+0000) 2019-10-13T08:39:19.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:19.006+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.007810+0000) 2019-10-13T08:39:19.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:19.006+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.008082+0000) 2019-10-13T08:39:19.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:19.006+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.008793+0000) 2019-10-13T08:39:19.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:19.006+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.009080+0000) 2019-10-13T08:39:19.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:19.005+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.009300+0000) 2019-10-13T08:39:19.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:19.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:19.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:19.006+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.009964+0000) 2019-10-13T08:39:19.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:19.013+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.016390+0000) 2019-10-13T08:39:19.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:19.453+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:19.455133+0000) 2019-10-13T08:39:20.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:20.625+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:20.628684+0000) 2019-10-13T08:39:21.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:21.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:21.711511+0000) 2019-10-13T08:39:21.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:21.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:21.757545+0000) 2019-10-13T08:39:21.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:21.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:21.951499+0000) 2019-10-13T08:39:24.013 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:24.009+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.012798+0000) 2019-10-13T08:39:24.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:24.009+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.013084+0000) 2019-10-13T08:39:24.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:24.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.013936+0000) 2019-10-13T08:39:24.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:24.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:24.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:24.009+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.014638+0000) 2019-10-13T08:39:24.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:24.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.015508+0000) 2019-10-13T08:39:24.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:24.013+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.015709+0000) 2019-10-13T08:39:24.018 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:24.013+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.017926+0000) 2019-10-13T08:39:24.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:24.017+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.020679+0000) 2019-10-13T08:39:24.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:24.977+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.980542+0000) 2019-10-13T08:39:26.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:26.197+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:26.201776+0000) 2019-10-13T08:39:28.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:28.465+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.469039+0000) 2019-10-13T08:39:29.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:29.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.018825+0000) 2019-10-13T08:39:29.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:29.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.018921+0000) 2019-10-13T08:39:29.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:29.017+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.019425+0000) 2019-10-13T08:39:29.021 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:29.017+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.020017+0000) 2019-10-13T08:39:29.021 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:29.017+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.020301+0000) 2019-10-13T08:39:29.021 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:29.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:29.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:29.017+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.020566+0000) 2019-10-13T08:39:29.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:29.016+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.021101+0000) 2019-10-13T08:39:29.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:29.024+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.026775+0000) 2019-10-13T08:39:29.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:29.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:29.455374+0000) 2019-10-13T08:39:30.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:30.632+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:30.635234+0000) 2019-10-13T08:39:31.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:31.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:31.711719+0000) 2019-10-13T08:39:31.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:31.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:31.757742+0000) 2019-10-13T08:39:31.793 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:39:31.793 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:31.797 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:39:31.797 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:31.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:31.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:31.951721+0000) 2019-10-13T08:39:34.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:34.021+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.024127+0000) 2019-10-13T08:39:34.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:34.021+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.024808+0000) 2019-10-13T08:39:34.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:34.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:34.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:34.021+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.025226+0000) 2019-10-13T08:39:34.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:34.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.025584+0000) 2019-10-13T08:39:34.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:34.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.025995+0000) 2019-10-13T08:39:34.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:34.021+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.026030+0000) 2019-10-13T08:39:34.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:34.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.027241+0000) 2019-10-13T08:39:34.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:34.028+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.033531+0000) 2019-10-13T08:39:34.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:34.977+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.980746+0000) 2019-10-13T08:39:36.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:36.197+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:36.201971+0000) 2019-10-13T08:39:38.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:38.465+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.469271+0000) 2019-10-13T08:39:39.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:39.025+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.028680+0000) 2019-10-13T08:39:39.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:39.025+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.029053+0000) 2019-10-13T08:39:39.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:39.025+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.029441+0000) 2019-10-13T08:39:39.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:39.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:39.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:39.025+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.030727+0000) 2019-10-13T08:39:39.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:39.025+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.030826+0000) 2019-10-13T08:39:39.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:39.028+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.031333+0000) 2019-10-13T08:39:39.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:39.029+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.032096+0000) 2019-10-13T08:39:39.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:39.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.038669+0000) 2019-10-13T08:39:39.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:39.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:39.455542+0000) 2019-10-13T08:39:40.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:40.636+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:40.641667+0000) 2019-10-13T08:39:41.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:41.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:41.711926+0000) 2019-10-13T08:39:41.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:41.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:41.757992+0000) 2019-10-13T08:39:41.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:41.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:41.951920+0000) 2019-10-13T08:39:43.995 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T08:39:43.996 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T08:39:44.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:44.029+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.033504+0000) 2019-10-13T08:39:44.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:44.029+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.033852+0000) 2019-10-13T08:39:44.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:44.032+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.034821+0000) 2019-10-13T08:39:44.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:44.033+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.035426+0000) 2019-10-13T08:39:44.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:44.033+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.035720+0000) 2019-10-13T08:39:44.036 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:44.033+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.035896+0000) 2019-10-13T08:39:44.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:44.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:44.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:44.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.037372+0000) 2019-10-13T08:39:44.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:44.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.040992+0000) 2019-10-13T08:39:44.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:44.977+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.980922+0000) 2019-10-13T08:39:46.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:46.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:46.202186+0000) 2019-10-13T08:39:48.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:48.469+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.469495+0000) 2019-10-13T08:39:49.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:49.033+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.039124+0000) 2019-10-13T08:39:49.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:49.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.039507+0000) 2019-10-13T08:39:49.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:49.037+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.040255+0000) 2019-10-13T08:39:49.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:49.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.041396+0000) 2019-10-13T08:39:49.047 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:49.037+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.041298+0000) 2019-10-13T08:39:49.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:49.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:49.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:49.037+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.042318+0000) 2019-10-13T08:39:49.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:49.037+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.042163+0000) 2019-10-13T08:39:49.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:49.044+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.049520+0000) 2019-10-13T08:39:49.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:49.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:49.455743+0000) 2019-10-13T08:39:50.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:50.644+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:50.648133+0000) 2019-10-13T08:39:51.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:51.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:51.712164+0000) 2019-10-13T08:39:51.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:51.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:51.758189+0000) 2019-10-13T08:39:51.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:51.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:51.952140+0000) 2019-10-13T08:39:54.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:54.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.044733+0000) 2019-10-13T08:39:54.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:54.040+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.045217+0000) 2019-10-13T08:39:54.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:54.041+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.046044+0000) 2019-10-13T08:39:54.047 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:54.041+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.046668+0000) 2019-10-13T08:39:54.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:54.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:54.047 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:54.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.047365+0000) 2019-10-13T08:39:54.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:54.045+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.047777+0000) 2019-10-13T08:39:54.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:54.045+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.048229+0000) 2019-10-13T08:39:54.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:54.048+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.053624+0000) 2019-10-13T08:39:54.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:54.977+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.981119+0000) 2019-10-13T08:39:56.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:56.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:56.202355+0000) 2019-10-13T08:39:58.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:58.465+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.469696+0000) 2019-10-13T08:39:59.050 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:39:59.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.050133+0000) 2019-10-13T08:39:59.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:59.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.050340+0000) 2019-10-13T08:39:59.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:39:59.045+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.051053+0000) 2019-10-13T08:39:59.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:39:59.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.051432+0000) 2019-10-13T08:39:59.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:39:59.049+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.051845+0000) 2019-10-13T08:39:59.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:39:59.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 56975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:39:59.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:39:59.049+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.052775+0000) 2019-10-13T08:39:59.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:39:59.049+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.053277+0000) 2019-10-13T08:39:59.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:39:59.056+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.059644+0000) 2019-10-13T08:39:59.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:39:59.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:59.455914+0000) 2019-10-13T08:40:00.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:00.652+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:00.654868+0000) 2019-10-13T08:40:01.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:01.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:01.712372+0000) 2019-10-13T08:40:01.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:01.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:01.758394+0000) 2019-10-13T08:40:01.904 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:40:01.905 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:01.908 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:40:01.909 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:01.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:01.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:01.952353+0000) 2019-10-13T08:40:04.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:04.052+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.055971+0000) 2019-10-13T08:40:04.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:04.052+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.056748+0000) 2019-10-13T08:40:04.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:04.052+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.057099+0000) 2019-10-13T08:40:04.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:04.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:04.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:04.052+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.057988+0000) 2019-10-13T08:40:04.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:04.052+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.058418+0000) 2019-10-13T08:40:04.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:04.052+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.058589+0000) 2019-10-13T08:40:04.060 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:04.056+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.059638+0000) 2019-10-13T08:40:04.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:04.064+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.068453+0000) 2019-10-13T08:40:04.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:04.980+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.981303+0000) 2019-10-13T08:40:06.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:06.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:06.202569+0000) 2019-10-13T08:40:08.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:08.468+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.469893+0000) 2019-10-13T08:40:09.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:09.056+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.061649+0000) 2019-10-13T08:40:09.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:09.056+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.062317+0000) 2019-10-13T08:40:09.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:09.056+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.062677+0000) 2019-10-13T08:40:09.063 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:09.056+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.062232+0000) 2019-10-13T08:40:09.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:09.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:09.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:09.060+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.063553+0000) 2019-10-13T08:40:09.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:09.060+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.065026+0000) 2019-10-13T08:40:09.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:09.060+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.065599+0000) 2019-10-13T08:40:09.072 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:09.068+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.072357+0000) 2019-10-13T08:40:09.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:09.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:09.456081+0000) 2019-10-13T08:40:10.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:10.660+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:10.661268+0000) 2019-10-13T08:40:11.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:11.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:11.712594+0000) 2019-10-13T08:40:11.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:11.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:11.758629+0000) 2019-10-13T08:40:11.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:11.948+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:11.952597+0000) 2019-10-13T08:40:14.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:14.060+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.066607+0000) 2019-10-13T08:40:14.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:14.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.066996+0000) 2019-10-13T08:40:14.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:14.064+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.067613+0000) 2019-10-13T08:40:14.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:14.064+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.068551+0000) 2019-10-13T08:40:14.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:14.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:14.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:14.064+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.069050+0000) 2019-10-13T08:40:14.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:14.064+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.069590+0000) 2019-10-13T08:40:14.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:14.064+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.070039+0000) 2019-10-13T08:40:14.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:14.068+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.073444+0000) 2019-10-13T08:40:14.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:14.980+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.981478+0000) 2019-10-13T08:40:16.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:16.200+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:16.202763+0000) 2019-10-13T08:40:18.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:18.464+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.470091+0000) 2019-10-13T08:40:19.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:19.068+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.071919+0000) 2019-10-13T08:40:19.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:19.068+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.072395+0000) 2019-10-13T08:40:19.073 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:19.068+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.072855+0000) 2019-10-13T08:40:19.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:19.068+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.073084+0000) 2019-10-13T08:40:19.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:19.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:19.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:19.068+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.073473+0000) 2019-10-13T08:40:19.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:19.068+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.073900+0000) 2019-10-13T08:40:19.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:19.072+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.075482+0000) 2019-10-13T08:40:19.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:19.080+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.082469+0000) 2019-10-13T08:40:19.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:19.452+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:19.456291+0000) 2019-10-13T08:40:20.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:20.664+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:20.667871+0000) 2019-10-13T08:40:21.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:21.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:21.712800+0000) 2019-10-13T08:40:21.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:21.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:21.758836+0000) 2019-10-13T08:40:21.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:21.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:21.952793+0000) 2019-10-13T08:40:24.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:24.072+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.077057+0000) 2019-10-13T08:40:24.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:24.072+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.077398+0000) 2019-10-13T08:40:24.078 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:24.072+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.077973+0000) 2019-10-13T08:40:24.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:24.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:24.079 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:24.072+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.079117+0000) 2019-10-13T08:40:24.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:24.076+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.079670+0000) 2019-10-13T08:40:24.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:24.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.080213+0000) 2019-10-13T08:40:24.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:24.075+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.080549+0000) 2019-10-13T08:40:24.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:24.083+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.089318+0000) 2019-10-13T08:40:24.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:24.976+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.981653+0000) 2019-10-13T08:40:26.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:26.200+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:26.202962+0000) 2019-10-13T08:40:28.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:28.464+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.470285+0000) 2019-10-13T08:40:29.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:29.076+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.082591+0000) 2019-10-13T08:40:29.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:29.079+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.083135+0000) 2019-10-13T08:40:29.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:29.080+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.083481+0000) 2019-10-13T08:40:29.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:29.080+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.084133+0000) 2019-10-13T08:40:29.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:29.080+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:29.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:29.080+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.085386+0000) 2019-10-13T08:40:29.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:29.080+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.086029+0000) 2019-10-13T08:40:29.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:29.080+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.084806+0000) 2019-10-13T08:40:29.093 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:29.087+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.093032+0000) 2019-10-13T08:40:29.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:29.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:29.456463+0000) 2019-10-13T08:40:30.674 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:30.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:30.674193+0000) 2019-10-13T08:40:31.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:31.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:31.713025+0000) 2019-10-13T08:40:31.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:31.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:31.759033+0000) 2019-10-13T08:40:31.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:31.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:31.953013+0000) 2019-10-13T08:40:32.035 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:40:32.035 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:32.040 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:40:32.040 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:34.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:34.084+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.087268+0000) 2019-10-13T08:40:34.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:34.084+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.087801+0000) 2019-10-13T08:40:34.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:34.084+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.088241+0000) 2019-10-13T08:40:34.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:34.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:34.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:34.084+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.089521+0000) 2019-10-13T08:40:34.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:34.087+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.090619+0000) 2019-10-13T08:40:34.092 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:34.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.091506+0000) 2019-10-13T08:40:34.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:34.088+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.092107+0000) 2019-10-13T08:40:34.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:34.091+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.095504+0000) 2019-10-13T08:40:34.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:34.980+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.981835+0000) 2019-10-13T08:40:36.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:36.196+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:36.203167+0000) 2019-10-13T08:40:38.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:38.464+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.470489+0000) 2019-10-13T08:40:39.093 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:39.088+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.092637+0000) 2019-10-13T08:40:39.093 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:39.088+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.092995+0000) 2019-10-13T08:40:39.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:39.088+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.093597+0000) 2019-10-13T08:40:39.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:39.088+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.094004+0000) 2019-10-13T08:40:39.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:39.088+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:39.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:39.088+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.094451+0000) 2019-10-13T08:40:39.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:39.091+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.097614+0000) 2019-10-13T08:40:39.098 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:39.092+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.098082+0000) 2019-10-13T08:40:39.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:39.099+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.103239+0000) 2019-10-13T08:40:39.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:39.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:39.456659+0000) 2019-10-13T08:40:40.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:40.679+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:40.680725+0000) 2019-10-13T08:40:41.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:41.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:41.713228+0000) 2019-10-13T08:40:41.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:41.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:41.759196+0000) 2019-10-13T08:40:41.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:41.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:41.953208+0000) 2019-10-13T08:40:44.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:44.092+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.097944+0000) 2019-10-13T08:40:44.099 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:44.092+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.098445+0000) 2019-10-13T08:40:44.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:44.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.098603+0000) 2019-10-13T08:40:44.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:44.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.099117+0000) 2019-10-13T08:40:44.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:44.096+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:44.101 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:44.096+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.100703+0000) 2019-10-13T08:40:44.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:44.095+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.100840+0000) 2019-10-13T08:40:44.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:44.096+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.100325+0000) 2019-10-13T08:40:44.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:44.099+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.105868+0000) 2019-10-13T08:40:44.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:44.980+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.982011+0000) 2019-10-13T08:40:46.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:46.200+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:46.203362+0000) 2019-10-13T08:40:48.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:48.467+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.470686+0000) 2019-10-13T08:40:49.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:49.095+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.102948+0000) 2019-10-13T08:40:49.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:49.099+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.103462+0000) 2019-10-13T08:40:49.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:49.099+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.103915+0000) 2019-10-13T08:40:49.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:49.099+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.104476+0000) 2019-10-13T08:40:49.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:49.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:49.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:49.099+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.105415+0000) 2019-10-13T08:40:49.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:49.099+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.105988+0000) 2019-10-13T08:40:49.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:49.099+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.106406+0000) 2019-10-13T08:40:49.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:49.107+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.111201+0000) 2019-10-13T08:40:49.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:49.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:49.456832+0000) 2019-10-13T08:40:50.687 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:50.683+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:50.686937+0000) 2019-10-13T08:40:51.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:51.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:51.713431+0000) 2019-10-13T08:40:51.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:51.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:51.759422+0000) 2019-10-13T08:40:51.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:51.951+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:51.953420+0000) 2019-10-13T08:40:54.108 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:54.103+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.108011+0000) 2019-10-13T08:40:54.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:54.103+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.108395+0000) 2019-10-13T08:40:54.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:54.103+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.108926+0000) 2019-10-13T08:40:54.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:54.103+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.109281+0000) 2019-10-13T08:40:54.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:54.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:54.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:54.103+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.109956+0000) 2019-10-13T08:40:54.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:54.103+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.110610+0000) 2019-10-13T08:40:54.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:54.107+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.111344+0000) 2019-10-13T08:40:54.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:54.111+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.117590+0000) 2019-10-13T08:40:54.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:54.975+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.982185+0000) 2019-10-13T08:40:56.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:56.199+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:56.203562+0000) 2019-10-13T08:40:58.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:58.467+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.470886+0000) 2019-10-13T08:40:59.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:59.107+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.113060+0000) 2019-10-13T08:40:59.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:40:59.107+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.113579+0000) 2019-10-13T08:40:59.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:40:59.107+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.113827+0000) 2019-10-13T08:40:59.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:40:59.107+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.114778+0000) 2019-10-13T08:40:59.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:40:59.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:40:59.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:40:59.111+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.115467+0000) 2019-10-13T08:40:59.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:40:59.111+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.116005+0000) 2019-10-13T08:40:59.117 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:40:59.111+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.116293+0000) 2019-10-13T08:40:59.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:40:59.119+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.122632+0000) 2019-10-13T08:40:59.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:40:59.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:59.457004+0000) 2019-10-13T08:41:00.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:00.687+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:00.693428+0000) 2019-10-13T08:41:01.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:01.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:01.713631+0000) 2019-10-13T08:41:01.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:01.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:01.759623+0000) 2019-10-13T08:41:01.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:01.951+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:01.953618+0000) 2019-10-13T08:41:02.255 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:41:02.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:02.259 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:41:02.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:04.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:04.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.118529+0000) 2019-10-13T08:41:04.120 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:04.115+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.119387+0000) 2019-10-13T08:41:04.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:04.111+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.119053+0000) 2019-10-13T08:41:04.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:04.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.119709+0000) 2019-10-13T08:41:04.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:04.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.120030+0000) 2019-10-13T08:41:04.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:04.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:04.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:04.115+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.120226+0000) 2019-10-13T08:41:04.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:04.115+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.120878+0000) 2019-10-13T08:41:04.127 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:04.123+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.127319+0000) 2019-10-13T08:41:04.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:04.979+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.982367+0000) 2019-10-13T08:41:06.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:06.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:06.203761+0000) 2019-10-13T08:41:08.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:08.467+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.471105+0000) 2019-10-13T08:41:09.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:09.119+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.123468+0000) 2019-10-13T08:41:09.124 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:09.119+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.123893+0000) 2019-10-13T08:41:09.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:09.119+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.124395+0000) 2019-10-13T08:41:09.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:09.119+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.124728+0000) 2019-10-13T08:41:09.126 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:09.119+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.125612+0000) 2019-10-13T08:41:09.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:09.119+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.125869+0000) 2019-10-13T08:41:09.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:09.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:09.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:09.119+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.127172+0000) 2019-10-13T08:41:09.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:09.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.131527+0000) 2019-10-13T08:41:09.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:09.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:09.457208+0000) 2019-10-13T08:41:10.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:10.699+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:10.699507+0000) 2019-10-13T08:41:11.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:11.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:11.713830+0000) 2019-10-13T08:41:11.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:11.759+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:11.759819+0000) 2019-10-13T08:41:11.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:11.951+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:11.953819+0000) 2019-10-13T08:41:14.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:14.123+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.129881+0000) 2019-10-13T08:41:14.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:14.123+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.130839+0000) 2019-10-13T08:41:14.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:14.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:14.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:14.127+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.131485+0000) 2019-10-13T08:41:14.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:14.127+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.131759+0000) 2019-10-13T08:41:14.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:14.127+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.132281+0000) 2019-10-13T08:41:14.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:14.127+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.132575+0000) 2019-10-13T08:41:14.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:14.127+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.133424+0000) 2019-10-13T08:41:14.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:14.131+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.135173+0000) 2019-10-13T08:41:14.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:14.979+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.982539+0000) 2019-10-13T08:41:16.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:16.199+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:16.203947+0000) 2019-10-13T08:41:18.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:18.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.471333+0000) 2019-10-13T08:41:19.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:19.131+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.133873+0000) 2019-10-13T08:41:19.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:19.131+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.134709+0000) 2019-10-13T08:41:19.135 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:19.131+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.135190+0000) 2019-10-13T08:41:19.136 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:19.135+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.135492+0000) 2019-10-13T08:41:19.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:19.135+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.136005+0000) 2019-10-13T08:41:19.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:19.130+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.136221+0000) 2019-10-13T08:41:19.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:19.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:19.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:19.135+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.137888+0000) 2019-10-13T08:41:19.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:19.142+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.147265+0000) 2019-10-13T08:41:19.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:19.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:19.457377+0000) 2019-10-13T08:41:20.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:20.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:20.705890+0000) 2019-10-13T08:41:21.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:21.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:21.714032+0000) 2019-10-13T08:41:21.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:21.759+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:21.760039+0000) 2019-10-13T08:41:21.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:21.951+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:21.954055+0000) 2019-10-13T08:41:24.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:24.139+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.139378+0000) 2019-10-13T08:41:24.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:24.139+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.139986+0000) 2019-10-13T08:41:24.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:24.139+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.141073+0000) 2019-10-13T08:41:24.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:24.134+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.141745+0000) 2019-10-13T08:41:24.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:24.139+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.142121+0000) 2019-10-13T08:41:24.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:24.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:24.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:24.139+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.142671+0000) 2019-10-13T08:41:24.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:24.139+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.143150+0000) 2019-10-13T08:41:24.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:24.142+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.148680+0000) 2019-10-13T08:41:24.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:24.979+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.982717+0000) 2019-10-13T08:41:26.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:26.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:26.204142+0000) 2019-10-13T08:41:28.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:28.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.471536+0000) 2019-10-13T08:41:29.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:29.143+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.145026+0000) 2019-10-13T08:41:29.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:29.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.145492+0000) 2019-10-13T08:41:29.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:29.143+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.145940+0000) 2019-10-13T08:41:29.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:29.143+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.146499+0000) 2019-10-13T08:41:29.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:29.143+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.146957+0000) 2019-10-13T08:41:29.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:29.142+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.147369+0000) 2019-10-13T08:41:29.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:29.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:29.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:29.147+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.147903+0000) 2019-10-13T08:41:29.153 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:29.146+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.152846+0000) 2019-10-13T08:41:29.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:29.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:29.457548+0000) 2019-10-13T08:41:30.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:30.710+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:30.712080+0000) 2019-10-13T08:41:31.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:31.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:31.714229+0000) 2019-10-13T08:41:31.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:31.758+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:31.760241+0000) 2019-10-13T08:41:31.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:31.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:31.954278+0000) 2019-10-13T08:41:32.397 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:41:32.397 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:32.401 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:41:32.401 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:34.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:34.150+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.151628+0000) 2019-10-13T08:41:34.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:34.150+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.151730+0000) 2019-10-13T08:41:34.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:34.150+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.152035+0000) 2019-10-13T08:41:34.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:34.150+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.152594+0000) 2019-10-13T08:41:34.154 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:34.146+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.153635+0000) 2019-10-13T08:41:34.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:34.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:34.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:34.150+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.153908+0000) 2019-10-13T08:41:34.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:34.154+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.155296+0000) 2019-10-13T08:41:34.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:34.154+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.160700+0000) 2019-10-13T08:41:34.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:34.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.982928+0000) 2019-10-13T08:41:36.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:36.202+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:36.204340+0000) 2019-10-13T08:41:38.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:38.470+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.471698+0000) 2019-10-13T08:41:39.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:39.154+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.156943+0000) 2019-10-13T08:41:39.158 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:39.154+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.158043+0000) 2019-10-13T08:41:39.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:39.154+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.158284+0000) 2019-10-13T08:41:39.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:39.154+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:39.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:39.154+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.158974+0000) 2019-10-13T08:41:39.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:39.154+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.158967+0000) 2019-10-13T08:41:39.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:39.158+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.159527+0000) 2019-10-13T08:41:39.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:39.158+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.160429+0000) 2019-10-13T08:41:39.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:39.158+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.164166+0000) 2019-10-13T08:41:39.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:39.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:39.457718+0000) 2019-10-13T08:41:40.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:40.714+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:40.718534+0000) 2019-10-13T08:41:41.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:41.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:41.714431+0000) 2019-10-13T08:41:41.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:41.758+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:41.760449+0000) 2019-10-13T08:41:41.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:41.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:41.954475+0000) 2019-10-13T08:41:44.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:44.158+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.161737+0000) 2019-10-13T08:41:44.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:44.158+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.161973+0000) 2019-10-13T08:41:44.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:44.158+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.163084+0000) 2019-10-13T08:41:44.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:44.162+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.163759+0000) 2019-10-13T08:41:44.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:44.162+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.163876+0000) 2019-10-13T08:41:44.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:44.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:44.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:44.162+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.164769+0000) 2019-10-13T08:41:44.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:44.162+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.165532+0000) 2019-10-13T08:41:44.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:44.162+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.169685+0000) 2019-10-13T08:41:44.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:44.978+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.983100+0000) 2019-10-13T08:41:46.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:46.202+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:46.204533+0000) 2019-10-13T08:41:48.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:48.470+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.471907+0000) 2019-10-13T08:41:49.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:49.162+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.167150+0000) 2019-10-13T08:41:49.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:49.166+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.167828+0000) 2019-10-13T08:41:49.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:49.166+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.168782+0000) 2019-10-13T08:41:49.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:49.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:49.170 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:49.166+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.169272+0000) 2019-10-13T08:41:49.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:49.162+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.169878+0000) 2019-10-13T08:41:49.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:49.166+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.169547+0000) 2019-10-13T08:41:49.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:49.166+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.170412+0000) 2019-10-13T08:41:49.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:49.166+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.173214+0000) 2019-10-13T08:41:49.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:49.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:49.457892+0000) 2019-10-13T08:41:50.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:50.722+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:50.724907+0000) 2019-10-13T08:41:51.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:51.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:51.714666+0000) 2019-10-13T08:41:51.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:51.758+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:51.760692+0000) 2019-10-13T08:41:51.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:51.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:51.954682+0000) 2019-10-13T08:41:54.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:54.170+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.172535+0000) 2019-10-13T08:41:54.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:54.166+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.173342+0000) 2019-10-13T08:41:54.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:54.170+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.174141+0000) 2019-10-13T08:41:54.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:54.170+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.174265+0000) 2019-10-13T08:41:54.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:54.170+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.174607+0000) 2019-10-13T08:41:54.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:54.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:54.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:54.174+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.175736+0000) 2019-10-13T08:41:54.180 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:54.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.176966+0000) 2019-10-13T08:41:54.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:54.178+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.182746+0000) 2019-10-13T08:41:54.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:54.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.983295+0000) 2019-10-13T08:41:56.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:56.202+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:56.204761+0000) 2019-10-13T08:41:58.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:58.470+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.472134+0000) 2019-10-13T08:41:59.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:41:59.174+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.178151+0000) 2019-10-13T08:41:59.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:41:59.174+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.178450+0000) 2019-10-13T08:41:59.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:59.174+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.179338+0000) 2019-10-13T08:41:59.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:41:59.178+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.179882+0000) 2019-10-13T08:41:59.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:41:59.178+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.181173+0000) 2019-10-13T08:41:59.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:41:59.178+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.181355+0000) 2019-10-13T08:41:59.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:41:59.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:41:59.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:41:59.178+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.181776+0000) 2019-10-13T08:41:59.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:41:59.182+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.186811+0000) 2019-10-13T08:41:59.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:41:59.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:59.458082+0000) 2019-10-13T08:42:00.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:00.734+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:00.736550+0000) 2019-10-13T08:42:01.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:01.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:01.715052+0000) 2019-10-13T08:42:01.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:01.758+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:01.760902+0000) 2019-10-13T08:42:01.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:01.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:01.954883+0000) 2019-10-13T08:42:02.443 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:42:02.443 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:02.447 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:42:02.447 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:04.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:04.182+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.184730+0000) 2019-10-13T08:42:04.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:04.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.185142+0000) 2019-10-13T08:42:04.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:04.182+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.186122+0000) 2019-10-13T08:42:04.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:04.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:04.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:04.182+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.186669+0000) 2019-10-13T08:42:04.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:04.182+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.187258+0000) 2019-10-13T08:42:04.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:04.182+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.187031+0000) 2019-10-13T08:42:04.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:04.186+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.187657+0000) 2019-10-13T08:42:04.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:04.186+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.190458+0000) 2019-10-13T08:42:04.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:04.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.983542+0000) 2019-10-13T08:42:06.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:06.202+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:06.204979+0000) 2019-10-13T08:42:08.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:08.470+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.472336+0000) 2019-10-13T08:42:09.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:09.186+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.189895+0000) 2019-10-13T08:42:09.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:09.186+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.190903+0000) 2019-10-13T08:42:09.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:09.186+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.191203+0000) 2019-10-13T08:42:09.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:09.190+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.191725+0000) 2019-10-13T08:42:09.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:09.190+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.192113+0000) 2019-10-13T08:42:09.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:09.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:09.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:09.190+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.192461+0000) 2019-10-13T08:42:09.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:09.190+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.194611+0000) 2019-10-13T08:42:09.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:09.198+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.199015+0000) 2019-10-13T08:42:09.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:09.458+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:09.458253+0000) 2019-10-13T08:42:10.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:10.742+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:10.742851+0000) 2019-10-13T08:42:11.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:11.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:11.715282+0000) 2019-10-13T08:42:11.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:11.758+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:11.761094+0000) 2019-10-13T08:42:11.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:11.950+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:11.955076+0000) 2019-10-13T08:42:14.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:14.193+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.196059+0000) 2019-10-13T08:42:14.197 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:14.193+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.196401+0000) 2019-10-13T08:42:14.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:14.193+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.197268+0000) 2019-10-13T08:42:14.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:14.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:14.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:14.193+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.198409+0000) 2019-10-13T08:42:14.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:14.193+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.198152+0000) 2019-10-13T08:42:14.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:14.193+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.199103+0000) 2019-10-13T08:42:14.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:14.197+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.199600+0000) 2019-10-13T08:42:14.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:14.201+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.202625+0000) 2019-10-13T08:42:14.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:14.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.983742+0000) 2019-10-13T08:42:16.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:16.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:16.205171+0000) 2019-10-13T08:42:18.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:18.469+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.472535+0000) 2019-10-13T08:42:19.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:19.197+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.200813+0000) 2019-10-13T08:42:19.202 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:19.197+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.201635+0000) 2019-10-13T08:42:19.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:19.197+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.201751+0000) 2019-10-13T08:42:19.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:19.197+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.202281+0000) 2019-10-13T08:42:19.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:19.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:19.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:19.197+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.202878+0000) 2019-10-13T08:42:19.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:19.197+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.203186+0000) 2019-10-13T08:42:19.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:19.201+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.203618+0000) 2019-10-13T08:42:19.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:19.209+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.210672+0000) 2019-10-13T08:42:19.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:19.457+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:19.458454+0000) 2019-10-13T08:42:20.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:20.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:20.749214+0000) 2019-10-13T08:42:21.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:21.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:21.715526+0000) 2019-10-13T08:42:21.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:21.757+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:21.761299+0000) 2019-10-13T08:42:21.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:21.949+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:21.955282+0000) 2019-10-13T08:42:24.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:24.201+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.206697+0000) 2019-10-13T08:42:24.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:24.201+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.206505+0000) 2019-10-13T08:42:24.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:24.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.207141+0000) 2019-10-13T08:42:24.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:24.205+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.207693+0000) 2019-10-13T08:42:24.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:24.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:24.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:24.205+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.207939+0000) 2019-10-13T08:42:24.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:24.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.208734+0000) 2019-10-13T08:42:24.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:24.205+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.209257+0000) 2019-10-13T08:42:24.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:24.213+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.214570+0000) 2019-10-13T08:42:24.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:24.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.983917+0000) 2019-10-13T08:42:26.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:26.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:26.205364+0000) 2019-10-13T08:42:28.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:28.469+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.472737+0000) 2019-10-13T08:42:29.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:29.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.210857+0000) 2019-10-13T08:42:29.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:29.209+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.211996+0000) 2019-10-13T08:42:29.213 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:29.209+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.212519+0000) 2019-10-13T08:42:29.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:29.209+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.212829+0000) 2019-10-13T08:42:29.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:29.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 57986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:29.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:29.209+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.213373+0000) 2019-10-13T08:42:29.214 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:29.209+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.213966+0000) 2019-10-13T08:42:29.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:29.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.214716+0000) 2019-10-13T08:42:29.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:29.221+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.222582+0000) 2019-10-13T08:42:29.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:29.457+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:29.458626+0000) 2019-10-13T08:42:30.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:30.757+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:30.760914+0000) 2019-10-13T08:42:31.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:31.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:31.715758+0000) 2019-10-13T08:42:31.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:31.757+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:31.761536+0000) 2019-10-13T08:42:31.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:31.953+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:31.955467+0000) 2019-10-13T08:42:32.570 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:42:32.570 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:32.573 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:42:32.574 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:34.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:34.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.216695+0000) 2019-10-13T08:42:34.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:34.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.217191+0000) 2019-10-13T08:42:34.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:34.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.218374+0000) 2019-10-13T08:42:34.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:34.213+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.218575+0000) 2019-10-13T08:42:34.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:34.213+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.219076+0000) 2019-10-13T08:42:34.220 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:34.213+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.219168+0000) 2019-10-13T08:42:34.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:34.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58019 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:34.221 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:34.217+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.219767+0000) 2019-10-13T08:42:34.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:34.221+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.224004+0000) 2019-10-13T08:42:34.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:34.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.984045+0000) 2019-10-13T08:42:36.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:36.205+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:36.205560+0000) 2019-10-13T08:42:38.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:38.469+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.472932+0000) 2019-10-13T08:42:39.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:39.217+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.222203+0000) 2019-10-13T08:42:39.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:39.217+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.222626+0000) 2019-10-13T08:42:39.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:39.221+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.222938+0000) 2019-10-13T08:42:39.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:39.221+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.223870+0000) 2019-10-13T08:42:39.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:39.221+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.223996+0000) 2019-10-13T08:42:39.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:39.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:39.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:39.221+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.224396+0000) 2019-10-13T08:42:39.226 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:39.221+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.225770+0000) 2019-10-13T08:42:39.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:39.229+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.231195+0000) 2019-10-13T08:42:39.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:39.457+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:39.458797+0000) 2019-10-13T08:42:40.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:40.765+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:40.767245+0000) 2019-10-13T08:42:41.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:41.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:41.715979+0000) 2019-10-13T08:42:41.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:41.757+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:41.761753+0000) 2019-10-13T08:42:41.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:41.953+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:41.955613+0000) 2019-10-13T08:42:44.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:44.225+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.228619+0000) 2019-10-13T08:42:44.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:44.225+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.229423+0000) 2019-10-13T08:42:44.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:44.225+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.229958+0000) 2019-10-13T08:42:44.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:44.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:44.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:44.229+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.230432+0000) 2019-10-13T08:42:44.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:44.225+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.230316+0000) 2019-10-13T08:42:44.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:44.229+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.231583+0000) 2019-10-13T08:42:44.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:44.229+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.232474+0000) 2019-10-13T08:42:44.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:44.233+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.235415+0000) 2019-10-13T08:42:44.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:44.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.984241+0000) 2019-10-13T08:42:46.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:46.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:46.205771+0000) 2019-10-13T08:42:48.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:48.469+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.473127+0000) 2019-10-13T08:42:49.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:49.229+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.233382+0000) 2019-10-13T08:42:49.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:49.229+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.234207+0000) 2019-10-13T08:42:49.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:49.229+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.234745+0000) 2019-10-13T08:42:49.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:49.229+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.235266+0000) 2019-10-13T08:42:49.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:49.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:49.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:49.233+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.235944+0000) 2019-10-13T08:42:49.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:49.233+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.236289+0000) 2019-10-13T08:42:49.240 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:49.233+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.237644+0000) 2019-10-13T08:42:49.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:49.241+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.244154+0000) 2019-10-13T08:42:49.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:49.457+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:49.458991+0000) 2019-10-13T08:42:50.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:50.769+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:50.773777+0000) 2019-10-13T08:42:51.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:51.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:51.716170+0000) 2019-10-13T08:42:51.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:51.757+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:51.761972+0000) 2019-10-13T08:42:51.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:51.953+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:51.955854+0000) 2019-10-13T08:42:54.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:54.233+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.239032+0000) 2019-10-13T08:42:54.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:54.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.239650+0000) 2019-10-13T08:42:54.241 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:54.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.240310+0000) 2019-10-13T08:42:54.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:54.237+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.240524+0000) 2019-10-13T08:42:54.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:54.237+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.241033+0000) 2019-10-13T08:42:54.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:54.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:54.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:54.237+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.242005+0000) 2019-10-13T08:42:54.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:54.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.242308+0000) 2019-10-13T08:42:54.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:54.245+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.248765+0000) 2019-10-13T08:42:54.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:54.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.984418+0000) 2019-10-13T08:42:56.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:56.201+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:56.206015+0000) 2019-10-13T08:42:58.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:58.468+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.473360+0000) 2019-10-13T08:42:59.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:42:59.240+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.244567+0000) 2019-10-13T08:42:59.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:42:59.240+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.245522+0000) 2019-10-13T08:42:59.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:42:59.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.245833+0000) 2019-10-13T08:42:59.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:59.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.246101+0000) 2019-10-13T08:42:59.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:42:59.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:42:59.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:42:59.240+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.246798+0000) 2019-10-13T08:42:59.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:42:59.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.247137+0000) 2019-10-13T08:42:59.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:42:59.244+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.249483+0000) 2019-10-13T08:42:59.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:42:59.249+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.251460+0000) 2019-10-13T08:42:59.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:42:59.457+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:59.459200+0000) 2019-10-13T08:43:00.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:00.785+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:00.785483+0000) 2019-10-13T08:43:01.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:01.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:01.716372+0000) 2019-10-13T08:43:01.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:01.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:01.762216+0000) 2019-10-13T08:43:01.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:01.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:01.956052+0000) 2019-10-13T08:43:02.624 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:43:02.625 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:02.628 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:43:02.629 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:04.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:04.244+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.249970+0000) 2019-10-13T08:43:04.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:04.244+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.250693+0000) 2019-10-13T08:43:04.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:04.249+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.250946+0000) 2019-10-13T08:43:04.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:04.248+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.251752+0000) 2019-10-13T08:43:04.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:04.248+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.252203+0000) 2019-10-13T08:43:04.253 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:04.248+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.252687+0000) 2019-10-13T08:43:04.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:04.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58220 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:04.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:04.248+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.253128+0000) 2019-10-13T08:43:04.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:04.257+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.259462+0000) 2019-10-13T08:43:04.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:04.980+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.984593+0000) 2019-10-13T08:43:06.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:06.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:06.206217+0000) 2019-10-13T08:43:08.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:08.468+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.473556+0000) 2019-10-13T08:43:09.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:09.252+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.256367+0000) 2019-10-13T08:43:09.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:09.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.256755+0000) 2019-10-13T08:43:09.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:09.252+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.257387+0000) 2019-10-13T08:43:09.259 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:09.252+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.257803+0000) 2019-10-13T08:43:09.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:09.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.258139+0000) 2019-10-13T08:43:09.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:09.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:09.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:09.252+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.259094+0000) 2019-10-13T08:43:09.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:09.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.259720+0000) 2019-10-13T08:43:09.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:09.260+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.265299+0000) 2019-10-13T08:43:09.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:09.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:09.459383+0000) 2019-10-13T08:43:10.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:10.792+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:10.797264+0000) 2019-10-13T08:43:11.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:11.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:11.716567+0000) 2019-10-13T08:43:11.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:11.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:11.762415+0000) 2019-10-13T08:43:11.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:11.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:11.956271+0000) 2019-10-13T08:43:14.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:14.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.262827+0000) 2019-10-13T08:43:14.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:14.260+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.263128+0000) 2019-10-13T08:43:14.264 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:14.260+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.263583+0000) 2019-10-13T08:43:14.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:14.260+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.263936+0000) 2019-10-13T08:43:14.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:14.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:14.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:14.260+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.264507+0000) 2019-10-13T08:43:14.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:14.260+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.265599+0000) 2019-10-13T08:43:14.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:14.260+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.265468+0000) 2019-10-13T08:43:14.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:14.268+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.272129+0000) 2019-10-13T08:43:14.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:14.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.984806+0000) 2019-10-13T08:43:16.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:16.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:16.206361+0000) 2019-10-13T08:43:18.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:18.472+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.473800+0000) 2019-10-13T08:43:19.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:19.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.267408+0000) 2019-10-13T08:43:19.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:19.264+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.268357+0000) 2019-10-13T08:43:19.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:19.264+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.268886+0000) 2019-10-13T08:43:19.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:19.268+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.270216+0000) 2019-10-13T08:43:19.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:19.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:19.271 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:19.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.270795+0000) 2019-10-13T08:43:19.271 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:19.264+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.271176+0000) 2019-10-13T08:43:19.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:19.268+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.271635+0000) 2019-10-13T08:43:19.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:19.272+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.276759+0000) 2019-10-13T08:43:19.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:19.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:19.459615+0000) 2019-10-13T08:43:20.804 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:20.800+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:20.803562+0000) 2019-10-13T08:43:21.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:21.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:21.716769+0000) 2019-10-13T08:43:21.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:21.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:21.762613+0000) 2019-10-13T08:43:21.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:21.952+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:21.956483+0000) 2019-10-13T08:43:24.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:24.268+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.274178+0000) 2019-10-13T08:43:24.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:24.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.274442+0000) 2019-10-13T08:43:24.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:24.272+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.274862+0000) 2019-10-13T08:43:24.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:24.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:24.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:24.272+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.276016+0000) 2019-10-13T08:43:24.277 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:24.272+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.276587+0000) 2019-10-13T08:43:24.278 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:24.272+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.277439+0000) 2019-10-13T08:43:24.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:24.272+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.277977+0000) 2019-10-13T08:43:24.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:24.280+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.282129+0000) 2019-10-13T08:43:24.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:24.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.984978+0000) 2019-10-13T08:43:26.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:26.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:26.206580+0000) 2019-10-13T08:43:28.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:28.468+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.474030+0000) 2019-10-13T08:43:29.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:29.276+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.279599+0000) 2019-10-13T08:43:29.281 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:29.276+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.279878+0000) 2019-10-13T08:43:29.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:29.276+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.280349+0000) 2019-10-13T08:43:29.281 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:29.276+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.280522+0000) 2019-10-13T08:43:29.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:29.276+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.281283+0000) 2019-10-13T08:43:29.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:29.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:29.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:29.276+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.281929+0000) 2019-10-13T08:43:29.284 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:29.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.283259+0000) 2019-10-13T08:43:29.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:29.284+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.288673+0000) 2019-10-13T08:43:29.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:29.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:29.459785+0000) 2019-10-13T08:43:30.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:30.804+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:30.809820+0000) 2019-10-13T08:43:31.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:31.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:31.716992+0000) 2019-10-13T08:43:31.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:31.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:31.762803+0000) 2019-10-13T08:43:31.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:31.956+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:31.956684+0000) 2019-10-13T08:43:32.680 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:43:32.681 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:32.684 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:43:32.684 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:34.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:34.280+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.284819+0000) 2019-10-13T08:43:34.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:34.280+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.285249+0000) 2019-10-13T08:43:34.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:34.280+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.285733+0000) 2019-10-13T08:43:34.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:34.280+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:34.287 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:34.280+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.286839+0000) 2019-10-13T08:43:34.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:34.284+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.287582+0000) 2019-10-13T08:43:34.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:34.284+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.288671+0000) 2019-10-13T08:43:34.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:34.284+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.289976+0000) 2019-10-13T08:43:34.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:34.292+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.295733+0000) 2019-10-13T08:43:34.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:34.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.985197+0000) 2019-10-13T08:43:36.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:36.200+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:36.206774+0000) 2019-10-13T08:43:38.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:38.472+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.474234+0000) 2019-10-13T08:43:39.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:39.284+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.289896+0000) 2019-10-13T08:43:39.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:39.284+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.290287+0000) 2019-10-13T08:43:39.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:39.288+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.291366+0000) 2019-10-13T08:43:39.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:39.288+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.291716+0000) 2019-10-13T08:43:39.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:39.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:39.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:39.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.292675+0000) 2019-10-13T08:43:39.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:39.288+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.292827+0000) 2019-10-13T08:43:39.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:39.288+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.293146+0000) 2019-10-13T08:43:39.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:39.292+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.297265+0000) 2019-10-13T08:43:39.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:39.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:39.459986+0000) 2019-10-13T08:43:40.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:40.812+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:40.816274+0000) 2019-10-13T08:43:41.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:41.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:41.717209+0000) 2019-10-13T08:43:41.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:41.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:41.763001+0000) 2019-10-13T08:43:41.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:41.955+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:41.956901+0000) 2019-10-13T08:43:44.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:44.291+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.295158+0000) 2019-10-13T08:43:44.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:44.291+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.296447+0000) 2019-10-13T08:43:44.297 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:44.291+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.296853+0000) 2019-10-13T08:43:44.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:44.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.297232+0000) 2019-10-13T08:43:44.298 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:44.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.297458+0000) 2019-10-13T08:43:44.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:44.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:44.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:44.296+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.298576+0000) 2019-10-13T08:43:44.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:44.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.299459+0000) 2019-10-13T08:43:44.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:44.300+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.303206+0000) 2019-10-13T08:43:44.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:44.983+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.985343+0000) 2019-10-13T08:43:46.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:46.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:46.206969+0000) 2019-10-13T08:43:48.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:48.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.474429+0000) 2019-10-13T08:43:49.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:49.295+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.300828+0000) 2019-10-13T08:43:49.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:49.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.301572+0000) 2019-10-13T08:43:49.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:49.295+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.301908+0000) 2019-10-13T08:43:49.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:49.295+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.302264+0000) 2019-10-13T08:43:49.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:49.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:49.304 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:49.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.303200+0000) 2019-10-13T08:43:49.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:49.300+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.304163+0000) 2019-10-13T08:43:49.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:49.299+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.304543+0000) 2019-10-13T08:43:49.311 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:49.304+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.310099+0000) 2019-10-13T08:43:49.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:49.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:49.460177+0000) 2019-10-13T08:43:50.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:50.820+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:50.822758+0000) 2019-10-13T08:43:51.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:51.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:51.717413+0000) 2019-10-13T08:43:51.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:51.755+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:51.763197+0000) 2019-10-13T08:43:51.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:51.955+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:51.957104+0000) 2019-10-13T08:43:54.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:54.299+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.305598+0000) 2019-10-13T08:43:54.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:54.303+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.307554+0000) 2019-10-13T08:43:54.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:54.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.307721+0000) 2019-10-13T08:43:54.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:54.304+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.307976+0000) 2019-10-13T08:43:54.309 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:54.303+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.308570+0000) 2019-10-13T08:43:54.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:54.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:54.310 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:54.303+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.309229+0000) 2019-10-13T08:43:54.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:54.303+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.309990+0000) 2019-10-13T08:43:54.316 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:54.312+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.315405+0000) 2019-10-13T08:43:54.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:54.979+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.985525+0000) 2019-10-13T08:43:56.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:56.199+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:56.207172+0000) 2019-10-13T08:43:58.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:58.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.474622+0000) 2019-10-13T08:43:59.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:43:59.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.311717+0000) 2019-10-13T08:43:59.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:43:59.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.312262+0000) 2019-10-13T08:43:59.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:43:59.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.313346+0000) 2019-10-13T08:43:59.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:43:59.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:43:59.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:59.312+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.314495+0000) 2019-10-13T08:43:59.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:43:59.307+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.315139+0000) 2019-10-13T08:43:59.316 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:43:59.307+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.314879+0000) 2019-10-13T08:43:59.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:43:59.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.315571+0000) 2019-10-13T08:43:59.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:43:59.316+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.321331+0000) 2019-10-13T08:43:59.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:43:59.456+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:59.460346+0000) 2019-10-13T08:44:00.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:00.824+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:00.829203+0000) 2019-10-13T08:44:01.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:01.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:01.717613+0000) 2019-10-13T08:44:01.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:01.759+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:01.763395+0000) 2019-10-13T08:44:01.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:01.955+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:01.957340+0000) 2019-10-13T08:44:02.787 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:44:02.787 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:02.791 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:44:02.791 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:04.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:04.311+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.316837+0000) 2019-10-13T08:44:04.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:04.311+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.318360+0000) 2019-10-13T08:44:04.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:04.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.318094+0000) 2019-10-13T08:44:04.319 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:04.311+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.318680+0000) 2019-10-13T08:44:04.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:04.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:04.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:04.311+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.319062+0000) 2019-10-13T08:44:04.321 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:04.315+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.319720+0000) 2019-10-13T08:44:04.322 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:04.315+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.320568+0000) 2019-10-13T08:44:04.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:04.323+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.328789+0000) 2019-10-13T08:44:04.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:04.983+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.985704+0000) 2019-10-13T08:44:06.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:06.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:06.207365+0000) 2019-10-13T08:44:08.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:08.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.474813+0000) 2019-10-13T08:44:09.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:09.315+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.322313+0000) 2019-10-13T08:44:09.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:09.315+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.323149+0000) 2019-10-13T08:44:09.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:09.319+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.323852+0000) 2019-10-13T08:44:09.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:09.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:09.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:09.319+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.325004+0000) 2019-10-13T08:44:09.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:09.319+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.325513+0000) 2019-10-13T08:44:09.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:09.319+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.325836+0000) 2019-10-13T08:44:09.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:09.319+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.327060+0000) 2019-10-13T08:44:09.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:09.327+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.330296+0000) 2019-10-13T08:44:09.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:09.459+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:09.460524+0000) 2019-10-13T08:44:10.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:10.831+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:10.835531+0000) 2019-10-13T08:44:11.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:11.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:11.717828+0000) 2019-10-13T08:44:11.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:11.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:11.763596+0000) 2019-10-13T08:44:11.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:11.955+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:11.957563+0000) 2019-10-13T08:44:14.328 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:14.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.327839+0000) 2019-10-13T08:44:14.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:14.327+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.328238+0000) 2019-10-13T08:44:14.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:14.323+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.328842+0000) 2019-10-13T08:44:14.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:14.327+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.328604+0000) 2019-10-13T08:44:14.330 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:14.327+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.329766+0000) 2019-10-13T08:44:14.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:14.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:14.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:14.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.331003+0000) 2019-10-13T08:44:14.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:14.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.331169+0000) 2019-10-13T08:44:14.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:14.331+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.336734+0000) 2019-10-13T08:44:14.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:14.983+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.985906+0000) 2019-10-13T08:44:16.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:16.207+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:16.207555+0000) 2019-10-13T08:44:18.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:18.471+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.475012+0000) 2019-10-13T08:44:19.334 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:19.331+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.333802+0000) 2019-10-13T08:44:19.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:19.331+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.334608+0000) 2019-10-13T08:44:19.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:19.331+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.334911+0000) 2019-10-13T08:44:19.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:19.335+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.335491+0000) 2019-10-13T08:44:19.336 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:19.335+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.335940+0000) 2019-10-13T08:44:19.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:19.335+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.336537+0000) 2019-10-13T08:44:19.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:19.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:19.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:19.335+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.337190+0000) 2019-10-13T08:44:19.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:19.339+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.343271+0000) 2019-10-13T08:44:19.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:19.459+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:19.460713+0000) 2019-10-13T08:44:20.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:20.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:20.842067+0000) 2019-10-13T08:44:21.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:21.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:21.718043+0000) 2019-10-13T08:44:21.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:21.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:21.763834+0000) 2019-10-13T08:44:21.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:21.955+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:21.957791+0000) 2019-10-13T08:44:24.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:24.338+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.340337+0000) 2019-10-13T08:44:24.343 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:24.338+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.340672+0000) 2019-10-13T08:44:24.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:24.338+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.341768+0000) 2019-10-13T08:44:24.343 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:24.338+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.342033+0000) 2019-10-13T08:44:24.345 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:24.339+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.342406+0000) 2019-10-13T08:44:24.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:24.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:24.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:24.338+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.343078+0000) 2019-10-13T08:44:24.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:24.342+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.343683+0000) 2019-10-13T08:44:24.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:24.343+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.347254+0000) 2019-10-13T08:44:24.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:24.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.986088+0000) 2019-10-13T08:44:26.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:26.206+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:26.207763+0000) 2019-10-13T08:44:28.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:28.470+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.475212+0000) 2019-10-13T08:44:29.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:29.342+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.346425+0000) 2019-10-13T08:44:29.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:29.342+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.346781+0000) 2019-10-13T08:44:29.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:29.342+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.347190+0000) 2019-10-13T08:44:29.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:29.346+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.347482+0000) 2019-10-13T08:44:29.349 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:29.343+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.348060+0000) 2019-10-13T08:44:29.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:29.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:29.351 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:29.346+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.349721+0000) 2019-10-13T08:44:29.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:29.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.351671+0000) 2019-10-13T08:44:29.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:29.355+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.358420+0000) 2019-10-13T08:44:29.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:29.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:29.460931+0000) 2019-10-13T08:44:30.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:30.851+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:30.853447+0000) 2019-10-13T08:44:31.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:31.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:31.718248+0000) 2019-10-13T08:44:31.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:31.762+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:31.764042+0000) 2019-10-13T08:44:31.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:31.954+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:31.957993+0000) 2019-10-13T08:44:32.877 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:44:32.877 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:32.880 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:44:32.880 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:34.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:34.346+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.351118+0000) 2019-10-13T08:44:34.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:34.350+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.352147+0000) 2019-10-13T08:44:34.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:34.350+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.352383+0000) 2019-10-13T08:44:34.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:34.350+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.352718+0000) 2019-10-13T08:44:34.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:34.350+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.353064+0000) 2019-10-13T08:44:34.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:34.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:34.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:34.347+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.353682+0000) 2019-10-13T08:44:34.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:34.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.354182+0000) 2019-10-13T08:44:34.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:34.355+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.361879+0000) 2019-10-13T08:44:34.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:34.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.986262+0000) 2019-10-13T08:44:36.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:36.206+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:36.207988+0000) 2019-10-13T08:44:38.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:38.474+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.475412+0000) 2019-10-13T08:44:39.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:39.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.356567+0000) 2019-10-13T08:44:39.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:39.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.357261+0000) 2019-10-13T08:44:39.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:39.354+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.358479+0000) 2019-10-13T08:44:39.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:39.354+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.358579+0000) 2019-10-13T08:44:39.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:39.354+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.359253+0000) 2019-10-13T08:44:39.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:39.358+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.359590+0000) 2019-10-13T08:44:39.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:39.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:39.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:39.355+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.360260+0000) 2019-10-13T08:44:39.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:39.359+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.365571+0000) 2019-10-13T08:44:39.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:39.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:39.461103+0000) 2019-10-13T08:44:40.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:40.859+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:40.860006+0000) 2019-10-13T08:44:41.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:41.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:41.718458+0000) 2019-10-13T08:44:41.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:41.762+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:41.764262+0000) 2019-10-13T08:44:41.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:41.954+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:41.958215+0000) 2019-10-13T08:44:44.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:44.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.362787+0000) 2019-10-13T08:44:44.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:44.358+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.362962+0000) 2019-10-13T08:44:44.364 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:44.362+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.363931+0000) 2019-10-13T08:44:44.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:44.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.364419+0000) 2019-10-13T08:44:44.365 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:44.362+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.365115+0000) 2019-10-13T08:44:44.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:44.359+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.366046+0000) 2019-10-13T08:44:44.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:44.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:44.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:44.362+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.366374+0000) 2019-10-13T08:44:44.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:44.367+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.371092+0000) 2019-10-13T08:44:44.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:44.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.986463+0000) 2019-10-13T08:44:46.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:46.206+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:46.208158+0000) 2019-10-13T08:44:48.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:48.474+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.475626+0000) 2019-10-13T08:44:49.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:49.366+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.369398+0000) 2019-10-13T08:44:49.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:49.366+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.370102+0000) 2019-10-13T08:44:49.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:49.366+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.370636+0000) 2019-10-13T08:44:49.372 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:49.366+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.370954+0000) 2019-10-13T08:44:49.373 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:49.370+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.371776+0000) 2019-10-13T08:44:49.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:49.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:49.373 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:49.370+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.372636+0000) 2019-10-13T08:44:49.374 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:49.367+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.372081+0000) 2019-10-13T08:44:49.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:49.371+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.377816+0000) 2019-10-13T08:44:49.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:49.455+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:49.461305+0000) 2019-10-13T08:44:50.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:50.863+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:50.866477+0000) 2019-10-13T08:44:51.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:51.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:51.718681+0000) 2019-10-13T08:44:51.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:51.762+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:51.764502+0000) 2019-10-13T08:44:51.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:51.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:51.958415+0000) 2019-10-13T08:44:54.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:54.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.375767+0000) 2019-10-13T08:44:54.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:54.374+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.376030+0000) 2019-10-13T08:44:54.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:54.374+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.376356+0000) 2019-10-13T08:44:54.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:54.374+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.377321+0000) 2019-10-13T08:44:54.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:54.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:54.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:54.374+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.377652+0000) 2019-10-13T08:44:54.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:54.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.378481+0000) 2019-10-13T08:44:54.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:54.375+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.378932+0000) 2019-10-13T08:44:54.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:54.383+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.386575+0000) 2019-10-13T08:44:54.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:54.986+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.986639+0000) 2019-10-13T08:44:56.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:56.206+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:56.208363+0000) 2019-10-13T08:44:58.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:58.474+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.475827+0000) 2019-10-13T08:44:59.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:44:59.378+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.380817+0000) 2019-10-13T08:44:59.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:59.375+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.381045+0000) 2019-10-13T08:44:59.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:44:59.378+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.382095+0000) 2019-10-13T08:44:59.383 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:44:59.378+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.382521+0000) 2019-10-13T08:44:59.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:44:59.378+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.382727+0000) 2019-10-13T08:44:59.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:44:59.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 58988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:44:59.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:44:59.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.383198+0000) 2019-10-13T08:44:59.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:44:59.382+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.383542+0000) 2019-10-13T08:44:59.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:44:59.383+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.388015+0000) 2019-10-13T08:44:59.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:44:59.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:59.461439+0000) 2019-10-13T08:45:00.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:00.874+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:00.878184+0000) 2019-10-13T08:45:01.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:01.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:01.718884+0000) 2019-10-13T08:45:01.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:01.762+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:01.764734+0000) 2019-10-13T08:45:01.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:01.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:01.958617+0000) 2019-10-13T08:45:03.017 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:45:03.017 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:03.020 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:45:03.021 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:04.387 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:04.382+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.386192+0000) 2019-10-13T08:45:04.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:04.382+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.385417+0000) 2019-10-13T08:45:04.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:04.382+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.386676+0000) 2019-10-13T08:45:04.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:04.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.387682+0000) 2019-10-13T08:45:04.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:04.386+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.388450+0000) 2019-10-13T08:45:04.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:04.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.388810+0000) 2019-10-13T08:45:04.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:04.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:04.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:04.386+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.389651+0000) 2019-10-13T08:45:04.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:04.390+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.394207+0000) 2019-10-13T08:45:04.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:04.982+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.986810+0000) 2019-10-13T08:45:06.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:06.206+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:06.208557+0000) 2019-10-13T08:45:08.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:08.473+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.476022+0000) 2019-10-13T08:45:09.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:09.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.393123+0000) 2019-10-13T08:45:09.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:09.389+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.393529+0000) 2019-10-13T08:45:09.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:09.390+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.394133+0000) 2019-10-13T08:45:09.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:09.389+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:09.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:09.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.394997+0000) 2019-10-13T08:45:09.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:09.393+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.395390+0000) 2019-10-13T08:45:09.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:09.393+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.395698+0000) 2019-10-13T08:45:09.396 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:09.393+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.396283+0000) 2019-10-13T08:45:09.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:09.394+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.400513+0000) 2019-10-13T08:45:09.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:09.458+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:09.461610+0000) 2019-10-13T08:45:10.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:10.878+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:10.884743+0000) 2019-10-13T08:45:11.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:11.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:11.719111+0000) 2019-10-13T08:45:11.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:11.761+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:11.764943+0000) 2019-10-13T08:45:11.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:11.953+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:11.958868+0000) 2019-10-13T08:45:14.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:14.393+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.397604+0000) 2019-10-13T08:45:14.399 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:14.394+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.398598+0000) 2019-10-13T08:45:14.399 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:14.393+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.398523+0000) 2019-10-13T08:45:14.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:14.397+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.399471+0000) 2019-10-13T08:45:14.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:14.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59090 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:14.401 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:14.397+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.399659+0000) 2019-10-13T08:45:14.401 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:14.397+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.400786+0000) 2019-10-13T08:45:14.402 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:14.397+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.401292+0000) 2019-10-13T08:45:14.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:14.398+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.404652+0000) 2019-10-13T08:45:14.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:14.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.987012+0000) 2019-10-13T08:45:16.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:16.205+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:16.208768+0000) 2019-10-13T08:45:18.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:18.473+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.476236+0000) 2019-10-13T08:45:19.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:19.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.403069+0000) 2019-10-13T08:45:19.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:19.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.403661+0000) 2019-10-13T08:45:19.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:19.401+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.404169+0000) 2019-10-13T08:45:19.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:19.401+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.404662+0000) 2019-10-13T08:45:19.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:19.398+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.405292+0000) 2019-10-13T08:45:19.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:19.401+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.405594+0000) 2019-10-13T08:45:19.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:19.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:19.407 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:19.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.406330+0000) 2019-10-13T08:45:19.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:19.406+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.411879+0000) 2019-10-13T08:45:19.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:19.454+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:19.461780+0000) 2019-10-13T08:45:20.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:20.890+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:20.891186+0000) 2019-10-13T08:45:21.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:21.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:21.719336+0000) 2019-10-13T08:45:21.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:21.761+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:21.765143+0000) 2019-10-13T08:45:21.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:21.953+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:21.959098+0000) 2019-10-13T08:45:24.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:24.405+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.408788+0000) 2019-10-13T08:45:24.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:24.405+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.409739+0000) 2019-10-13T08:45:24.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:24.405+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.409819+0000) 2019-10-13T08:45:24.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:24.402+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.410074+0000) 2019-10-13T08:45:24.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:24.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:24.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:24.409+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.411366+0000) 2019-10-13T08:45:24.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:24.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.411629+0000) 2019-10-13T08:45:24.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:24.409+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.411913+0000) 2019-10-13T08:45:24.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:24.414+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.419619+0000) 2019-10-13T08:45:24.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:24.981+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.987220+0000) 2019-10-13T08:45:26.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:26.205+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:26.208958+0000) 2019-10-13T08:45:28.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:28.473+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.476463+0000) 2019-10-13T08:45:29.414 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:29.409+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.413926+0000) 2019-10-13T08:45:29.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:29.409+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.414777+0000) 2019-10-13T08:45:29.415 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:29.409+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.414935+0000) 2019-10-13T08:45:29.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:29.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:29.416 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:29.413+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.415854+0000) 2019-10-13T08:45:29.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:29.413+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.417592+0000) 2019-10-13T08:45:29.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:29.414+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.418795+0000) 2019-10-13T08:45:29.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:29.413+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.416310+0000) 2019-10-13T08:45:29.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:29.422+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.428556+0000) 2019-10-13T08:45:29.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:29.458+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:29.461918+0000) 2019-10-13T08:45:30.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:30.898+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:30.902424+0000) 2019-10-13T08:45:31.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:31.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:31.719535+0000) 2019-10-13T08:45:31.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:31.761+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:31.765382+0000) 2019-10-13T08:45:31.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:31.957+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:31.959338+0000) 2019-10-13T08:45:33.151 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:45:33.151 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:33.155 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:45:33.155 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:34.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:34.413+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.418664+0000) 2019-10-13T08:45:34.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:34.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.420113+0000) 2019-10-13T08:45:34.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:34.417+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.420924+0000) 2019-10-13T08:45:34.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:34.417+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.421086+0000) 2019-10-13T08:45:34.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:34.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:34.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:34.417+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.421701+0000) 2019-10-13T08:45:34.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:34.417+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.423125+0000) 2019-10-13T08:45:34.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:34.414+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.420459+0000) 2019-10-13T08:45:34.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:34.426+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.430808+0000) 2019-10-13T08:45:34.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:34.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.987367+0000) 2019-10-13T08:45:36.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:36.205+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:36.209154+0000) 2019-10-13T08:45:38.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:38.473+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.476694+0000) 2019-10-13T08:45:39.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:39.421+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.424549+0000) 2019-10-13T08:45:39.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:39.421+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.425437+0000) 2019-10-13T08:45:39.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:39.421+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.426628+0000) 2019-10-13T08:45:39.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:39.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:39.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:39.422+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.426816+0000) 2019-10-13T08:45:39.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:39.425+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.427361+0000) 2019-10-13T08:45:39.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:39.425+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.430280+0000) 2019-10-13T08:45:39.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:39.425+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.428340+0000) 2019-10-13T08:45:39.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:39.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.437852+0000) 2019-10-13T08:45:39.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:39.458+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:39.462082+0000) 2019-10-13T08:45:40.911 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:40.906+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:40.908940+0000) 2019-10-13T08:45:41.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:41.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:41.719737+0000) 2019-10-13T08:45:41.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:41.761+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:41.765582+0000) 2019-10-13T08:45:41.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:41.957+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:41.959570+0000) 2019-10-13T08:45:44.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:44.425+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.429646+0000) 2019-10-13T08:45:44.430 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:44.425+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.430156+0000) 2019-10-13T08:45:44.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:44.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.431513+0000) 2019-10-13T08:45:44.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:44.426+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.432044+0000) 2019-10-13T08:45:44.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:44.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:44.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:44.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.432454+0000) 2019-10-13T08:45:44.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:44.429+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.432769+0000) 2019-10-13T08:45:44.434 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:44.429+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.433421+0000) 2019-10-13T08:45:44.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:44.430+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.436882+0000) 2019-10-13T08:45:44.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:44.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.987549+0000) 2019-10-13T08:45:46.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:46.205+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:46.209341+0000) 2019-10-13T08:45:48.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:48.473+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.476924+0000) 2019-10-13T08:45:49.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:49.434+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.434942+0000) 2019-10-13T08:45:49.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:49.433+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.435491+0000) 2019-10-13T08:45:49.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:49.433+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.436100+0000) 2019-10-13T08:45:49.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:49.433+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.437036+0000) 2019-10-13T08:45:49.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:49.433+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.437325+0000) 2019-10-13T08:45:49.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:49.433+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.437952+0000) 2019-10-13T08:45:49.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:49.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:49.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:49.433+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.438842+0000) 2019-10-13T08:45:49.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:49.442+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.444634+0000) 2019-10-13T08:45:49.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:49.462+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:49.462265+0000) 2019-10-13T08:45:50.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:50.918+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:50.919399+0000) 2019-10-13T08:45:51.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:51.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:51.719965+0000) 2019-10-13T08:45:51.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:51.760+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:51.765793+0000) 2019-10-13T08:45:51.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:51.956+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:51.959773+0000) 2019-10-13T08:45:54.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:54.436+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.441650+0000) 2019-10-13T08:45:54.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:54.442+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.442340+0000) 2019-10-13T08:45:54.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:54.436+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.442195+0000) 2019-10-13T08:45:54.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:54.436+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.442872+0000) 2019-10-13T08:45:54.445 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:54.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:54.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:54.440+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.444487+0000) 2019-10-13T08:45:54.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:54.440+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.445092+0000) 2019-10-13T08:45:54.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:54.440+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.445477+0000) 2019-10-13T08:45:54.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:54.450+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.452334+0000) 2019-10-13T08:45:54.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:54.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.987725+0000) 2019-10-13T08:45:56.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:56.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:56.209537+0000) 2019-10-13T08:45:58.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:58.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.477122+0000) 2019-10-13T08:45:59.448 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:45:59.444+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.447669+0000) 2019-10-13T08:45:59.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:45:59.444+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.448176+0000) 2019-10-13T08:45:59.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:59.445+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.449252+0000) 2019-10-13T08:45:59.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:45:59.444+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.449617+0000) 2019-10-13T08:45:59.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:45:59.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:45:59.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:45:59.444+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.450586+0000) 2019-10-13T08:45:59.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:45:59.448+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.451954+0000) 2019-10-13T08:45:59.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:45:59.448+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.453282+0000) 2019-10-13T08:45:59.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:45:59.453+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.454940+0000) 2019-10-13T08:45:59.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:45:59.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:59.462393+0000) 2019-10-13T08:46:00.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:00.921+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:00.925750+0000) 2019-10-13T08:46:01.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:01.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:01.720157+0000) 2019-10-13T08:46:01.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:01.760+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:01.766055+0000) 2019-10-13T08:46:01.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:01.956+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:01.959998+0000) 2019-10-13T08:46:03.232 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:46:03.233 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:03.236 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:46:03.237 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:04.454 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:04.448+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.453493+0000) 2019-10-13T08:46:04.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:04.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.453882+0000) 2019-10-13T08:46:04.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:04.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.454591+0000) 2019-10-13T08:46:04.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:04.448+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.455153+0000) 2019-10-13T08:46:04.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:04.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59424 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:04.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:04.452+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.456302+0000) 2019-10-13T08:46:04.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:04.452+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.455894+0000) 2019-10-13T08:46:04.458 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:04.452+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.458038+0000) 2019-10-13T08:46:04.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:04.457+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.461518+0000) 2019-10-13T08:46:04.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:04.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.987920+0000) 2019-10-13T08:46:06.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:06.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:06.209734+0000) 2019-10-13T08:46:08.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:08.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.477354+0000) 2019-10-13T08:46:09.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:09.452+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.459194+0000) 2019-10-13T08:46:09.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:09.456+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.459710+0000) 2019-10-13T08:46:09.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:09.456+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.460022+0000) 2019-10-13T08:46:09.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:09.457+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.460774+0000) 2019-10-13T08:46:09.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:09.456+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.460446+0000) 2019-10-13T08:46:09.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:09.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:09.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:09.456+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.461943+0000) 2019-10-13T08:46:09.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:09.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.462545+0000) 2019-10-13T08:46:09.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:09.456+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.461282+0000) 2019-10-13T08:46:09.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:09.465+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:09.468634+0000) 2019-10-13T08:46:10.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:10.929+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:10.932123+0000) 2019-10-13T08:46:11.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:11.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:11.720374+0000) 2019-10-13T08:46:11.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:11.760+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:11.766289+0000) 2019-10-13T08:46:11.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:11.956+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:11.960229+0000) 2019-10-13T08:46:14.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:14.460+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.464159+0000) 2019-10-13T08:46:14.465 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:14.460+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.464785+0000) 2019-10-13T08:46:14.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:14.460+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.465520+0000) 2019-10-13T08:46:14.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:14.460+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.465817+0000) 2019-10-13T08:46:14.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:14.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:14.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:14.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.466127+0000) 2019-10-13T08:46:14.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:14.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.466749+0000) 2019-10-13T08:46:14.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:14.460+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.466805+0000) 2019-10-13T08:46:14.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:14.473+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.474499+0000) 2019-10-13T08:46:14.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:14.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.988113+0000) 2019-10-13T08:46:16.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:16.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:16.209933+0000) 2019-10-13T08:46:18.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:18.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.477576+0000) 2019-10-13T08:46:19.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:19.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.462688+0000) 2019-10-13T08:46:19.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:19.464+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.469189+0000) 2019-10-13T08:46:19.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:19.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.469471+0000) 2019-10-13T08:46:19.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:19.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.469695+0000) 2019-10-13T08:46:19.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:19.464+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.470515+0000) 2019-10-13T08:46:19.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:19.468+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.471444+0000) 2019-10-13T08:46:19.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:19.468+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.471987+0000) 2019-10-13T08:46:19.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:19.468+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.472417+0000) 2019-10-13T08:46:19.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:19.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:19.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:19.473+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:19.476045+0000) 2019-10-13T08:46:20.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:20.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:20.943035+0000) 2019-10-13T08:46:21.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:21.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:21.720635+0000) 2019-10-13T08:46:21.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:21.760+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:21.766487+0000) 2019-10-13T08:46:21.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:21.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:21.960427+0000) 2019-10-13T08:46:24.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:24.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.476338+0000) 2019-10-13T08:46:24.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:24.473+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.476508+0000) 2019-10-13T08:46:24.478 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:24.472+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.477421+0000) 2019-10-13T08:46:24.478 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:24.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.477817+0000) 2019-10-13T08:46:24.479 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:24.472+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.478216+0000) 2019-10-13T08:46:24.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:24.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:24.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:24.472+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.478753+0000) 2019-10-13T08:46:24.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:24.476+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.480659+0000) 2019-10-13T08:46:24.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:24.481+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.483239+0000) 2019-10-13T08:46:24.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:24.988+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.988294+0000) 2019-10-13T08:46:26.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:26.204+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:26.210087+0000) 2019-10-13T08:46:28.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:28.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.477774+0000) 2019-10-13T08:46:29.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:29.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.462917+0000) 2019-10-13T08:46:29.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:29.476+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.482098+0000) 2019-10-13T08:46:29.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:29.481+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.482125+0000) 2019-10-13T08:46:29.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:29.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.482990+0000) 2019-10-13T08:46:29.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:29.480+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.483387+0000) 2019-10-13T08:46:29.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:29.480+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.484035+0000) 2019-10-13T08:46:29.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:29.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59591 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:29.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:29.480+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.484350+0000) 2019-10-13T08:46:29.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:29.480+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.485415+0000) 2019-10-13T08:46:29.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:29.489+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:29.490576+0000) 2019-10-13T08:46:30.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:30.949+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:30.949399+0000) 2019-10-13T08:46:31.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:31.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:31.720840+0000) 2019-10-13T08:46:31.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:31.760+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:31.766721+0000) 2019-10-13T08:46:31.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:31.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:31.960625+0000) 2019-10-13T08:46:33.304 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:46:33.305 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:33.308 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:46:33.309 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:34.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:34.484+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.487589+0000) 2019-10-13T08:46:34.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:34.485+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.488315+0000) 2019-10-13T08:46:34.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:34.484+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.487893+0000) 2019-10-13T08:46:34.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:34.484+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.488858+0000) 2019-10-13T08:46:34.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:34.484+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.489439+0000) 2019-10-13T08:46:34.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:34.484+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.489588+0000) 2019-10-13T08:46:34.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:34.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:34.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:34.484+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.491147+0000) 2019-10-13T08:46:34.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:34.493+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.496441+0000) 2019-10-13T08:46:34.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:34.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.988477+0000) 2019-10-13T08:46:36.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:36.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:36.210276+0000) 2019-10-13T08:46:38.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:38.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.477976+0000) 2019-10-13T08:46:39.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:39.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.463127+0000) 2019-10-13T08:46:39.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:39.487+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.493341+0000) 2019-10-13T08:46:39.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:39.487+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.493888+0000) 2019-10-13T08:46:39.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:39.487+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.494164+0000) 2019-10-13T08:46:39.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:39.493+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.494454+0000) 2019-10-13T08:46:39.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:39.487+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.494966+0000) 2019-10-13T08:46:39.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:39.487+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.495188+0000) 2019-10-13T08:46:39.497 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:39.491+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.495408+0000) 2019-10-13T08:46:39.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:39.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:39.503 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:39.501+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:39.502691+0000) 2019-10-13T08:46:40.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:40.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:40.955876+0000) 2019-10-13T08:46:41.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:41.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:41.721063+0000) 2019-10-13T08:46:41.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:41.759+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:41.766956+0000) 2019-10-13T08:46:41.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:41.959+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:41.960840+0000) 2019-10-13T08:46:44.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:44.495+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.498783+0000) 2019-10-13T08:46:44.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:44.495+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.499238+0000) 2019-10-13T08:46:44.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:44.495+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.499700+0000) 2019-10-13T08:46:44.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:44.495+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.500312+0000) 2019-10-13T08:46:44.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:44.497+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.501498+0000) 2019-10-13T08:46:44.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:44.495+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.501895+0000) 2019-10-13T08:46:44.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:44.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:44.503 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:44.495+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.502675+0000) 2019-10-13T08:46:44.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:44.509+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.510498+0000) 2019-10-13T08:46:44.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:44.983+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.988653+0000) 2019-10-13T08:46:46.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:46.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:46.210496+0000) 2019-10-13T08:46:48.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:48.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.478187+0000) 2019-10-13T08:46:49.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:49.461+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.463338+0000) 2019-10-13T08:46:49.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:49.499+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.505319+0000) 2019-10-13T08:46:49.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:49.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.505811+0000) 2019-10-13T08:46:49.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:49.499+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.506395+0000) 2019-10-13T08:46:49.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:49.505+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.506871+0000) 2019-10-13T08:46:49.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:49.499+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.507198+0000) 2019-10-13T08:46:49.508 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:49.503+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.507661+0000) 2019-10-13T08:46:49.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:49.503+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.508270+0000) 2019-10-13T08:46:49.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:49.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:49.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:49.513+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:49.515058+0000) 2019-10-13T08:46:50.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:50.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:50.962451+0000) 2019-10-13T08:46:51.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:51.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:51.721303+0000) 2019-10-13T08:46:51.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:51.759+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:51.767155+0000) 2019-10-13T08:46:51.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:51.959+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:51.961101+0000) 2019-10-13T08:46:54.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:54.507+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.512061+0000) 2019-10-13T08:46:54.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:54.507+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.513029+0000) 2019-10-13T08:46:54.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:54.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59759 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:54.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:54.507+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.514235+0000) 2019-10-13T08:46:54.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:54.512+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.514469+0000) 2019-10-13T08:46:54.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:54.507+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.515114+0000) 2019-10-13T08:46:54.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:54.511+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.515750+0000) 2019-10-13T08:46:54.518 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:54.511+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.517616+0000) 2019-10-13T08:46:54.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:54.516+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.520029+0000) 2019-10-13T08:46:54.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:54.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.988864+0000) 2019-10-13T08:46:56.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:56.203+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:56.210694+0000) 2019-10-13T08:46:58.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:58.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.478388+0000) 2019-10-13T08:46:59.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:59.460+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.463532+0000) 2019-10-13T08:46:59.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:46:59.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.517077+0000) 2019-10-13T08:46:59.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:46:59.512+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.517687+0000) 2019-10-13T08:46:59.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:46:59.511+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.518131+0000) 2019-10-13T08:46:59.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:46:59.511+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.518589+0000) 2019-10-13T08:46:59.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:46:59.515+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.519816+0000) 2019-10-13T08:46:59.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:46:59.515+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.519966+0000) 2019-10-13T08:46:59.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:46:59.515+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:46:59.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:46:59.515+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.520179+0000) 2019-10-13T08:46:59.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:46:59.524+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:59.526455+0000) 2019-10-13T08:47:00.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:00.968+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:00.969010+0000) 2019-10-13T08:47:01.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:01.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:01.721505+0000) 2019-10-13T08:47:01.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:01.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:01.767383+0000) 2019-10-13T08:47:01.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:01.959+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:01.961298+0000) 2019-10-13T08:47:03.445 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:47:03.445 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:03.449 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:47:03.449 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:04.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:04.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.523287+0000) 2019-10-13T08:47:04.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:04.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.523814+0000) 2019-10-13T08:47:04.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:04.519+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.523997+0000) 2019-10-13T08:47:04.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:04.519+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.524316+0000) 2019-10-13T08:47:04.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:04.519+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.525266+0000) 2019-10-13T08:47:04.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:04.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:04.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:04.520+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.525980+0000) 2019-10-13T08:47:04.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:04.519+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.527204+0000) 2019-10-13T08:47:04.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:04.532+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.535728+0000) 2019-10-13T08:47:04.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:04.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.989068+0000) 2019-10-13T08:47:06.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:06.207+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:06.210883+0000) 2019-10-13T08:47:08.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:08.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.478587+0000) 2019-10-13T08:47:09.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:09.460+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.463730+0000) 2019-10-13T08:47:09.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:09.527+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.528098+0000) 2019-10-13T08:47:09.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:09.527+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.528780+0000) 2019-10-13T08:47:09.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:09.527+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.529491+0000) 2019-10-13T08:47:09.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:09.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.530131+0000) 2019-10-13T08:47:09.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:09.527+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.530522+0000) 2019-10-13T08:47:09.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:09.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.530953+0000) 2019-10-13T08:47:09.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:09.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:09.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:09.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.531521+0000) 2019-10-13T08:47:09.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:09.532+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:09.537555+0000) 2019-10-13T08:47:10.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:10.972+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:10.975701+0000) 2019-10-13T08:47:11.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:11.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:11.721705+0000) 2019-10-13T08:47:11.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:11.767+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:11.767630+0000) 2019-10-13T08:47:11.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:11.959+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:11.961503+0000) 2019-10-13T08:47:14.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:14.531+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.534628+0000) 2019-10-13T08:47:14.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:14.535+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.535394+0000) 2019-10-13T08:47:14.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:14.535+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.535631+0000) 2019-10-13T08:47:14.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:14.535+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.536271+0000) 2019-10-13T08:47:14.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:14.535+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.537045+0000) 2019-10-13T08:47:14.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:14.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:14.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:14.535+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.537402+0000) 2019-10-13T08:47:14.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:14.532+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.537608+0000) 2019-10-13T08:47:14.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:14.536+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.540369+0000) 2019-10-13T08:47:14.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:14.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.989206+0000) 2019-10-13T08:47:16.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:16.211+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:16.211116+0000) 2019-10-13T08:47:18.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:18.474+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.478786+0000) 2019-10-13T08:47:19.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:19.460+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.463926+0000) 2019-10-13T08:47:19.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:19.538+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.540874+0000) 2019-10-13T08:47:19.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:19.538+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.541190+0000) 2019-10-13T08:47:19.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:19.536+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.541483+0000) 2019-10-13T08:47:19.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:19.538+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.541949+0000) 2019-10-13T08:47:19.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:19.538+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.542598+0000) 2019-10-13T08:47:19.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:19.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:19.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:19.542+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.543939+0000) 2019-10-13T08:47:19.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:19.538+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.543093+0000) 2019-10-13T08:47:19.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:19.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:19.550424+0000) 2019-10-13T08:47:20.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:20.984+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:20.986857+0000) 2019-10-13T08:47:21.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:21.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:21.721927+0000) 2019-10-13T08:47:21.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:21.766+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:21.767850+0000) 2019-10-13T08:47:21.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:21.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:21.961706+0000) 2019-10-13T08:47:24.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:24.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.546845+0000) 2019-10-13T08:47:24.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:24.546+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.547377+0000) 2019-10-13T08:47:24.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:24.546+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.547824+0000) 2019-10-13T08:47:24.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:24.544+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.548774+0000) 2019-10-13T08:47:24.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:24.546+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:24.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:24.546+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.549738+0000) 2019-10-13T08:47:24.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:24.546+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.550083+0000) 2019-10-13T08:47:24.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:24.546+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.550826+0000) 2019-10-13T08:47:24.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:24.552+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.555578+0000) 2019-10-13T08:47:24.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:24.986+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.989418+0000) 2019-10-13T08:47:26.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:26.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:26.211342+0000) 2019-10-13T08:47:28.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:28.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.478991+0000) 2019-10-13T08:47:29.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:29.460+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.464153+0000) 2019-10-13T08:47:29.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:29.550+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.552631+0000) 2019-10-13T08:47:29.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:29.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.553190+0000) 2019-10-13T08:47:29.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:29.550+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.553845+0000) 2019-10-13T08:47:29.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:29.550+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.554351+0000) 2019-10-13T08:47:29.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:29.550+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.554650+0000) 2019-10-13T08:47:29.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:29.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 59992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:29.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:29.554+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.555585+0000) 2019-10-13T08:47:29.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:29.554+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.556565+0000) 2019-10-13T08:47:29.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:29.560+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:29.562436+0000) 2019-10-13T08:47:30.994 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:30.988+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:30.993426+0000) 2019-10-13T08:47:31.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:31.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:31.722155+0000) 2019-10-13T08:47:31.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:31.766+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:31.768065+0000) 2019-10-13T08:47:31.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:31.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:31.961910+0000) 2019-10-13T08:47:33.526 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:47:33.527 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:33.530 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:47:33.530 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:34.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:34.554+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.558725+0000) 2019-10-13T08:47:34.560 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:34.554+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.559257+0000) 2019-10-13T08:47:34.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:34.558+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.559972+0000) 2019-10-13T08:47:34.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:34.558+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.560385+0000) 2019-10-13T08:47:34.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:34.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:34.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:34.556+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.561360+0000) 2019-10-13T08:47:34.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:34.558+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.562334+0000) 2019-10-13T08:47:34.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:34.558+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.562964+0000) 2019-10-13T08:47:34.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:34.564+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.566512+0000) 2019-10-13T08:47:34.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:34.986+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.989592+0000) 2019-10-13T08:47:36.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:36.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:36.211544+0000) 2019-10-13T08:47:38.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:38.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.479196+0000) 2019-10-13T08:47:39.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:39.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.464382+0000) 2019-10-13T08:47:39.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:39.562+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.564411+0000) 2019-10-13T08:47:39.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:39.562+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.565770+0000) 2019-10-13T08:47:39.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:39.562+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.566130+0000) 2019-10-13T08:47:39.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:39.562+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:39.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:39.562+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.566474+0000) 2019-10-13T08:47:39.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:39.562+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.567171+0000) 2019-10-13T08:47:39.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:39.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.564982+0000) 2019-10-13T08:47:39.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:39.566+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.567472+0000) 2019-10-13T08:47:39.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:39.572+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:39.575680+0000) 2019-10-13T08:47:41.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:40.996+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.000307+0000) 2019-10-13T08:47:41.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:41.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.722349+0000) 2019-10-13T08:47:41.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:41.766+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.768263+0000) 2019-10-13T08:47:41.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:41.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.962104+0000) 2019-10-13T08:47:44.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:44.566+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.569460+0000) 2019-10-13T08:47:44.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:44.566+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.569872+0000) 2019-10-13T08:47:44.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:44.566+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.570553+0000) 2019-10-13T08:47:44.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:44.568+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.570992+0000) 2019-10-13T08:47:44.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:44.570+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.571575+0000) 2019-10-13T08:47:44.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:44.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:44.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:44.570+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.573036+0000) 2019-10-13T08:47:44.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:44.570+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.573628+0000) 2019-10-13T08:47:44.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:44.572+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.577714+0000) 2019-10-13T08:47:44.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:44.986+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.989801+0000) 2019-10-13T08:47:46.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:46.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:46.211714+0000) 2019-10-13T08:47:48.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:48.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.479412+0000) 2019-10-13T08:47:49.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:49.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.464572+0000) 2019-10-13T08:47:49.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:49.574+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.575297+0000) 2019-10-13T08:47:49.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:49.574+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.575972+0000) 2019-10-13T08:47:49.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:49.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.576525+0000) 2019-10-13T08:47:49.577 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:49.574+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.577329+0000) 2019-10-13T08:47:49.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:49.574+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.577830+0000) 2019-10-13T08:47:49.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:49.575+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.578130+0000) 2019-10-13T08:47:49.579 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:49.574+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:49.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:49.574+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.579116+0000) 2019-10-13T08:47:49.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:49.583+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:49.587077+0000) 2019-10-13T08:47:51.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:51.003+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.006679+0000) 2019-10-13T08:47:51.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:51.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.722580+0000) 2019-10-13T08:47:51.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:51.766+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.768427+0000) 2019-10-13T08:47:51.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:51.958+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.962329+0000) 2019-10-13T08:47:54.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:54.578+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.581617+0000) 2019-10-13T08:47:54.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:54.578+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.581765+0000) 2019-10-13T08:47:54.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:54.578+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.582208+0000) 2019-10-13T08:47:54.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:54.579+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.582891+0000) 2019-10-13T08:47:54.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:54.578+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.583242+0000) 2019-10-13T08:47:54.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:54.582+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.583673+0000) 2019-10-13T08:47:54.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:54.582+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.584068+0000) 2019-10-13T08:47:54.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:54.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:54.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:54.591+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.594237+0000) 2019-10-13T08:47:54.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:54.986+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.989982+0000) 2019-10-13T08:47:56.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:56.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:56.211871+0000) 2019-10-13T08:47:58.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:58.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.479631+0000) 2019-10-13T08:47:59.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:59.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.464778+0000) 2019-10-13T08:47:59.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:47:59.583+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.587429+0000) 2019-10-13T08:47:59.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:47:59.586+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.587932+0000) 2019-10-13T08:47:59.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:47:59.586+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.589245+0000) 2019-10-13T08:47:59.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:47:59.586+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.589448+0000) 2019-10-13T08:47:59.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:47:59.586+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.589726+0000) 2019-10-13T08:47:59.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:47:59.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:47:59.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:47:59.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.590336+0000) 2019-10-13T08:47:59.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:47:59.586+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.590776+0000) 2019-10-13T08:47:59.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:47:59.595+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:59.599149+0000) 2019-10-13T08:48:01.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:01.007+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.013070+0000) 2019-10-13T08:48:01.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:01.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.722783+0000) 2019-10-13T08:48:01.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:01.765+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.768627+0000) 2019-10-13T08:48:01.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:01.961+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.962524+0000) 2019-10-13T08:48:03.649 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:48:03.649 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:03.653 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:48:03.653 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:04.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:04.587+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.592844+0000) 2019-10-13T08:48:04.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:04.589+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.593869+0000) 2019-10-13T08:48:04.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:04.589+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.594592+0000) 2019-10-13T08:48:04.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:04.589+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.594902+0000) 2019-10-13T08:48:04.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:04.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:04.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:04.593+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.596171+0000) 2019-10-13T08:48:04.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:04.593+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.596110+0000) 2019-10-13T08:48:04.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:04.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.596870+0000) 2019-10-13T08:48:04.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:04.595+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.601696+0000) 2019-10-13T08:48:04.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:04.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.990186+0000) 2019-10-13T08:48:06.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:06.209+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:06.212037+0000) 2019-10-13T08:48:08.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:08.477+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.479855+0000) 2019-10-13T08:48:09.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:09.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.464989+0000) 2019-10-13T08:48:09.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:09.593+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.598252+0000) 2019-10-13T08:48:09.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:09.597+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.599589+0000) 2019-10-13T08:48:09.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:09.595+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.599900+0000) 2019-10-13T08:48:09.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:09.597+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.600147+0000) 2019-10-13T08:48:09.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:09.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:09.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:09.597+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.600570+0000) 2019-10-13T08:48:09.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:09.597+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.601228+0000) 2019-10-13T08:48:09.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:09.597+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.601657+0000) 2019-10-13T08:48:09.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:09.599+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:09.606026+0000) 2019-10-13T08:48:11.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:11.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.019503+0000) 2019-10-13T08:48:11.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:11.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.722982+0000) 2019-10-13T08:48:11.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:11.765+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.768825+0000) 2019-10-13T08:48:11.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:11.957+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.962720+0000) 2019-10-13T08:48:14.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:14.601+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.603631+0000) 2019-10-13T08:48:14.605 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:14.601+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.604370+0000) 2019-10-13T08:48:14.605 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:14.599+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.604769+0000) 2019-10-13T08:48:14.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:14.601+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.605146+0000) 2019-10-13T08:48:14.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:14.601+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.605979+0000) 2019-10-13T08:48:14.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:14.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:14.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:14.601+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.606601+0000) 2019-10-13T08:48:14.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:14.601+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.607069+0000) 2019-10-13T08:48:14.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:14.607+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.613776+0000) 2019-10-13T08:48:14.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:14.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.990367+0000) 2019-10-13T08:48:16.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:16.209+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:16.212232+0000) 2019-10-13T08:48:18.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:18.477+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.480052+0000) 2019-10-13T08:48:19.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:19.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.465181+0000) 2019-10-13T08:48:19.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:19.605+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.609595+0000) 2019-10-13T08:48:19.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:19.605+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.610975+0000) 2019-10-13T08:48:19.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:19.605+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.611116+0000) 2019-10-13T08:48:19.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:19.609+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.611525+0000) 2019-10-13T08:48:19.613 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:19.607+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.612195+0000) 2019-10-13T08:48:19.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:19.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:19.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:19.609+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.612512+0000) 2019-10-13T08:48:19.613 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:19.609+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.613245+0000) 2019-10-13T08:48:19.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:19.611+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:19.616701+0000) 2019-10-13T08:48:21.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:21.023+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.025912+0000) 2019-10-13T08:48:21.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:21.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.723175+0000) 2019-10-13T08:48:21.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:21.765+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.769024+0000) 2019-10-13T08:48:21.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:21.961+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.962920+0000) 2019-10-13T08:48:24.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:24.613+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.615748+0000) 2019-10-13T08:48:24.617 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:24.611+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.616017+0000) 2019-10-13T08:48:24.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:24.613+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.616288+0000) 2019-10-13T08:48:24.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:24.613+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.616649+0000) 2019-10-13T08:48:24.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:24.613+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.617117+0000) 2019-10-13T08:48:24.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:24.613+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.617317+0000) 2019-10-13T08:48:24.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:24.613+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.618420+0000) 2019-10-13T08:48:24.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:24.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:24.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:24.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.625014+0000) 2019-10-13T08:48:24.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:24.989+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.990571+0000) 2019-10-13T08:48:26.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:26.209+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:26.212458+0000) 2019-10-13T08:48:28.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:28.477+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.480264+0000) 2019-10-13T08:48:29.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:29.459+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.465408+0000) 2019-10-13T08:48:29.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:29.617+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.622047+0000) 2019-10-13T08:48:29.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:29.617+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.622365+0000) 2019-10-13T08:48:29.623 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:29.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.622702+0000) 2019-10-13T08:48:29.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:29.621+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.623596+0000) 2019-10-13T08:48:29.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:29.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:29.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:29.621+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.623933+0000) 2019-10-13T08:48:29.626 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:29.621+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.624488+0000) 2019-10-13T08:48:29.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:29.621+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.625364+0000) 2019-10-13T08:48:29.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:29.623+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:29.629296+0000) 2019-10-13T08:48:31.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:31.031+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.032504+0000) 2019-10-13T08:48:31.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:31.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.723380+0000) 2019-10-13T08:48:31.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:31.765+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.769259+0000) 2019-10-13T08:48:31.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:31.957+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.963154+0000) 2019-10-13T08:48:33.747 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:48:33.747 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:33.751 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:48:33.751 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:34.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:34.625+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.627664+0000) 2019-10-13T08:48:34.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:34.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.627845+0000) 2019-10-13T08:48:34.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:34.623+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.628688+0000) 2019-10-13T08:48:34.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:34.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.628429+0000) 2019-10-13T08:48:34.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:34.625+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.629106+0000) 2019-10-13T08:48:34.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:34.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:34.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:34.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.629648+0000) 2019-10-13T08:48:34.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:34.625+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.629789+0000) 2019-10-13T08:48:34.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:34.631+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.638008+0000) 2019-10-13T08:48:34.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:34.985+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.990707+0000) 2019-10-13T08:48:36.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:36.209+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:36.212654+0000) 2019-10-13T08:48:38.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:38.477+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.480458+0000) 2019-10-13T08:48:39.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:39.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.465603+0000) 2019-10-13T08:48:39.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:39.627+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.633351+0000) 2019-10-13T08:48:39.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:39.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.633244+0000) 2019-10-13T08:48:39.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:39.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.634076+0000) 2019-10-13T08:48:39.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:39.629+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.634629+0000) 2019-10-13T08:48:39.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:39.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:39.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:39.629+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.634946+0000) 2019-10-13T08:48:39.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:39.633+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.635369+0000) 2019-10-13T08:48:39.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:39.633+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.636666+0000) 2019-10-13T08:48:39.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:39.639+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:39.644675+0000) 2019-10-13T08:48:41.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:41.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.044200+0000) 2019-10-13T08:48:41.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:41.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.723618+0000) 2019-10-13T08:48:41.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:41.765+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.769466+0000) 2019-10-13T08:48:41.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:41.961+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.963352+0000) 2019-10-13T08:48:44.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:44.630+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.637118+0000) 2019-10-13T08:48:44.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:44.633+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.637865+0000) 2019-10-13T08:48:44.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:44.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.638762+0000) 2019-10-13T08:48:44.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:44.633+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.639070+0000) 2019-10-13T08:48:44.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:44.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:44.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:44.637+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.639924+0000) 2019-10-13T08:48:44.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:44.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.641367+0000) 2019-10-13T08:48:44.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:44.637+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.641848+0000) 2019-10-13T08:48:44.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:44.642+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.647490+0000) 2019-10-13T08:48:44.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:44.988+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.990879+0000) 2019-10-13T08:48:46.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:46.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:46.212852+0000) 2019-10-13T08:48:48.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:48.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.480688+0000) 2019-10-13T08:48:49.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:49.462+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.465802+0000) 2019-10-13T08:48:49.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:49.636+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.642838+0000) 2019-10-13T08:48:49.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:49.640+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.644323+0000) 2019-10-13T08:48:49.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:49.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:49.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:49.640+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.644736+0000) 2019-10-13T08:48:49.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:49.638+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.645137+0000) 2019-10-13T08:48:49.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:49.640+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.646262+0000) 2019-10-13T08:48:49.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:49.640+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.646893+0000) 2019-10-13T08:48:49.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:49.644+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.648062+0000) 2019-10-13T08:48:49.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:49.646+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:49.651874+0000) 2019-10-13T08:48:51.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:51.046+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.050683+0000) 2019-10-13T08:48:51.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:51.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.723816+0000) 2019-10-13T08:48:51.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:51.764+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.769662+0000) 2019-10-13T08:48:51.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:51.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.963613+0000) 2019-10-13T08:48:54.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:54.642+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.648525+0000) 2019-10-13T08:48:54.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:54.644+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.649083+0000) 2019-10-13T08:48:54.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:54.644+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.649834+0000) 2019-10-13T08:48:54.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:54.644+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.650400+0000) 2019-10-13T08:48:54.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:54.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:54.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:54.644+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.650641+0000) 2019-10-13T08:48:54.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:54.644+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.651275+0000) 2019-10-13T08:48:54.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:54.648+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.651373+0000) 2019-10-13T08:48:54.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:54.650+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.655891+0000) 2019-10-13T08:48:54.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:54.984+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.991074+0000) 2019-10-13T08:48:56.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:56.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:56.213057+0000) 2019-10-13T08:48:58.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:58.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.480882+0000) 2019-10-13T08:48:59.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:59.462+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.465996+0000) 2019-10-13T08:48:59.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:48:59.648+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.653575+0000) 2019-10-13T08:48:59.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:48:59.648+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.653986+0000) 2019-10-13T08:48:59.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:48:59.648+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.654772+0000) 2019-10-13T08:48:59.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:48:59.652+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.655473+0000) 2019-10-13T08:48:59.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:48:59.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:48:59.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:48:59.650+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.656383+0000) 2019-10-13T08:48:59.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:48:59.652+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.657200+0000) 2019-10-13T08:48:59.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:48:59.652+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.657415+0000) 2019-10-13T08:48:59.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:48:59.662+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:59.667139+0000) 2019-10-13T08:49:01.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:01.054+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.057087+0000) 2019-10-13T08:49:01.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:01.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.724015+0000) 2019-10-13T08:49:01.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:01.764+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.769871+0000) 2019-10-13T08:49:01.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:01.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.963826+0000) 2019-10-13T08:49:03.896 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:49:03.896 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:03.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:49:03.900 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:04.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:04.656+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.659395+0000) 2019-10-13T08:49:04.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:04.656+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.660824+0000) 2019-10-13T08:49:04.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:04.656+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.661325+0000) 2019-10-13T08:49:04.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:04.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:04.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:04.656+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.661990+0000) 2019-10-13T08:49:04.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:04.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.662639+0000) 2019-10-13T08:49:04.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:04.658+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.663067+0000) 2019-10-13T08:49:04.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:04.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.664388+0000) 2019-10-13T08:49:04.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:04.662+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.669518+0000) 2019-10-13T08:49:04.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:04.988+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.991287+0000) 2019-10-13T08:49:06.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:06.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:06.213255+0000) 2019-10-13T08:49:08.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:08.476+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.481103+0000) 2019-10-13T08:49:09.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:09.462+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.466192+0000) 2019-10-13T08:49:09.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:09.662+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.665491+0000) 2019-10-13T08:49:09.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:09.660+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.666287+0000) 2019-10-13T08:49:09.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:09.660+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.666929+0000) 2019-10-13T08:49:09.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:09.664+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.667708+0000) 2019-10-13T08:49:09.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:09.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:09.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:09.664+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.668041+0000) 2019-10-13T08:49:09.669 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:09.664+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.668608+0000) 2019-10-13T08:49:09.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:09.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.668996+0000) 2019-10-13T08:49:09.674 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:09.670+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:09.674315+0000) 2019-10-13T08:49:11.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:11.062+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.063580+0000) 2019-10-13T08:49:11.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:11.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.724234+0000) 2019-10-13T08:49:11.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:11.764+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.770087+0000) 2019-10-13T08:49:11.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:11.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.964038+0000) 2019-10-13T08:49:14.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:14.664+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.670906+0000) 2019-10-13T08:49:14.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:14.666+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.671227+0000) 2019-10-13T08:49:14.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:14.668+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.672278+0000) 2019-10-13T08:49:14.674 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:14.668+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.672809+0000) 2019-10-13T08:49:14.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:14.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.673109+0000) 2019-10-13T08:49:14.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:14.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:14.675 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:14.668+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.675062+0000) 2019-10-13T08:49:14.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:14.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.676231+0000) 2019-10-13T08:49:14.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:14.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.682729+0000) 2019-10-13T08:49:14.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:14.988+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.991509+0000) 2019-10-13T08:49:16.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:16.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:16.213468+0000) 2019-10-13T08:49:18.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:18.480+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.481299+0000) 2019-10-13T08:49:19.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:19.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.466339+0000) 2019-10-13T08:49:19.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:19.670+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.676568+0000) 2019-10-13T08:49:19.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:19.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.677641+0000) 2019-10-13T08:49:19.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:19.672+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.678087+0000) 2019-10-13T08:49:19.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:19.672+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.678375+0000) 2019-10-13T08:49:19.680 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:19.676+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.679492+0000) 2019-10-13T08:49:19.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:19.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:19.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:19.676+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.680150+0000) 2019-10-13T08:49:19.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:19.676+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.680861+0000) 2019-10-13T08:49:19.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:19.678+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:19.683607+0000) 2019-10-13T08:49:21.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:21.066+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.070016+0000) 2019-10-13T08:49:21.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:21.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.724440+0000) 2019-10-13T08:49:21.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:21.764+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.770323+0000) 2019-10-13T08:49:21.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:21.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.964242+0000) 2019-10-13T08:49:24.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:24.680+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.683603+0000) 2019-10-13T08:49:24.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:24.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.684298+0000) 2019-10-13T08:49:24.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:24.680+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.684592+0000) 2019-10-13T08:49:24.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:24.680+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.685286+0000) 2019-10-13T08:49:24.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:24.680+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.685603+0000) 2019-10-13T08:49:24.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:24.678+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.686094+0000) 2019-10-13T08:49:24.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:24.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:24.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:24.680+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.686720+0000) 2019-10-13T08:49:24.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:24.686+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.692855+0000) 2019-10-13T08:49:24.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:24.988+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.991691+0000) 2019-10-13T08:49:26.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:26.208+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:26.213684+0000) 2019-10-13T08:49:28.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:28.479+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.481497+0000) 2019-10-13T08:49:29.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:29.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.466538+0000) 2019-10-13T08:49:29.689 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:29.683+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.688967+0000) 2019-10-13T08:49:29.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:29.686+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.689629+0000) 2019-10-13T08:49:29.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:29.683+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.690170+0000) 2019-10-13T08:49:29.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:29.683+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.690474+0000) 2019-10-13T08:49:29.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:29.683+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.691071+0000) 2019-10-13T08:49:29.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:29.687+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.691766+0000) 2019-10-13T08:49:29.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:29.687+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:29.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:29.687+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.692653+0000) 2019-10-13T08:49:29.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:29.698+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.699388+0000) 2019-10-13T08:49:31.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:31.074+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.076345+0000) 2019-10-13T08:49:31.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:31.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.724641+0000) 2019-10-13T08:49:31.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:31.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.770556+0000) 2019-10-13T08:49:31.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:31.963+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.964447+0000) 2019-10-13T08:49:34.010 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:49:34.010 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:34.014 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:49:34.014 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:34.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:34.694+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.694523+0000) 2019-10-13T08:49:34.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:34.691+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.695909+0000) 2019-10-13T08:49:34.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:34.691+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.696551+0000) 2019-10-13T08:49:34.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:34.691+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.696619+0000) 2019-10-13T08:49:34.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:34.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:34.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:34.691+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.697463+0000) 2019-10-13T08:49:34.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:34.691+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.697784+0000) 2019-10-13T08:49:34.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:34.691+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.697976+0000) 2019-10-13T08:49:34.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:34.698+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.701316+0000) 2019-10-13T08:49:34.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:34.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.991872+0000) 2019-10-13T08:49:36.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:36.211+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:36.213878+0000) 2019-10-13T08:49:38.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:38.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.481689+0000) 2019-10-13T08:49:39.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:39.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.466693+0000) 2019-10-13T08:49:39.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:39.695+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.700949+0000) 2019-10-13T08:49:39.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:39.695+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.701333+0000) 2019-10-13T08:49:39.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:39.695+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.702793+0000) 2019-10-13T08:49:39.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:39.701+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.702992+0000) 2019-10-13T08:49:39.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:39.699+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.703343+0000) 2019-10-13T08:49:39.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:39.699+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:39.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:39.699+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.703600+0000) 2019-10-13T08:49:39.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:39.699+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.704049+0000) 2019-10-13T08:49:39.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:39.709+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.710632+0000) 2019-10-13T08:49:41.083 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:41.081+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.082763+0000) 2019-10-13T08:49:41.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:41.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.724858+0000) 2019-10-13T08:49:41.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:41.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.770764+0000) 2019-10-13T08:49:41.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:41.963+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.964648+0000) 2019-10-13T08:49:44.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:44.699+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.706412+0000) 2019-10-13T08:49:44.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:44.699+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.706918+0000) 2019-10-13T08:49:44.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:44.703+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.707636+0000) 2019-10-13T08:49:44.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:44.703+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.708598+0000) 2019-10-13T08:49:44.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:44.703+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.708296+0000) 2019-10-13T08:49:44.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:44.705+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.709792+0000) 2019-10-13T08:49:44.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:44.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:44.711 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:44.703+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.710293+0000) 2019-10-13T08:49:44.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:44.713+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.714557+0000) 2019-10-13T08:49:44.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:44.987+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.992069+0000) 2019-10-13T08:49:46.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:46.207+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:46.214072+0000) 2019-10-13T08:49:48.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:48.475+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.481901+0000) 2019-10-13T08:49:49.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:49.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.466883+0000) 2019-10-13T08:49:49.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:49.713+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.714292+0000) 2019-10-13T08:49:49.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:49.707+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.713089+0000) 2019-10-13T08:49:49.721 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:49.707+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.713584+0000) 2019-10-13T08:49:49.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:49.707+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.714212+0000) 2019-10-13T08:49:49.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:49.707+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.714668+0000) 2019-10-13T08:49:49.722 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:49.707+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:49.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:49.707+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.715051+0000) 2019-10-13T08:49:49.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:49.711+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.715553+0000) 2019-10-13T08:49:49.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:49.717+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.721684+0000) 2019-10-13T08:49:51.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:51.085+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.089100+0000) 2019-10-13T08:49:51.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:51.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.725057+0000) 2019-10-13T08:49:51.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:51.763+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.770983+0000) 2019-10-13T08:49:51.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:51.963+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.964867+0000) 2019-10-13T08:49:54.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:54.711+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.718185+0000) 2019-10-13T08:49:54.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:54.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.718898+0000) 2019-10-13T08:49:54.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:54.711+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.718970+0000) 2019-10-13T08:49:54.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:54.715+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.719458+0000) 2019-10-13T08:49:54.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:54.717+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.720220+0000) 2019-10-13T08:49:54.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:54.715+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 60973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:54.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:54.715+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.721269+0000) 2019-10-13T08:49:54.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:54.715+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.721613+0000) 2019-10-13T08:49:54.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:54.725+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.726629+0000) 2019-10-13T08:49:54.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:54.991+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.992255+0000) 2019-10-13T08:49:56.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:56.207+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:56.214292+0000) 2019-10-13T08:49:58.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:58.479+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.482134+0000) 2019-10-13T08:49:59.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:59.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.467078+0000) 2019-10-13T08:49:59.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:49:59.719+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.723116+0000) 2019-10-13T08:49:59.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:49:59.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.723714+0000) 2019-10-13T08:49:59.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:49:59.723+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.724474+0000) 2019-10-13T08:49:59.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:49:59.723+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.724634+0000) 2019-10-13T08:49:59.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:49:59.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:49:59.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:49:59.721+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.725980+0000) 2019-10-13T08:49:59.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:49:59.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.726412+0000) 2019-10-13T08:49:59.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:49:59.723+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.726691+0000) 2019-10-13T08:49:59.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:49:59.733+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.736370+0000) 2019-10-13T08:50:01.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:01.093+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.095585+0000) 2019-10-13T08:50:01.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:01.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.725288+0000) 2019-10-13T08:50:01.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:01.767+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.771180+0000) 2019-10-13T08:50:01.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:01.963+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.965067+0000) 2019-10-13T08:50:04.080 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:50:04.080 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:04.084 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:50:04.084 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:04.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:04.727+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.729482+0000) 2019-10-13T08:50:04.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:04.727+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.729668+0000) 2019-10-13T08:50:04.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:04.727+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.730080+0000) 2019-10-13T08:50:04.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:04.727+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.730774+0000) 2019-10-13T08:50:04.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:04.727+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.731139+0000) 2019-10-13T08:50:04.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:04.731+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.731592+0000) 2019-10-13T08:50:04.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:04.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:04.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:04.729+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.732639+0000) 2019-10-13T08:50:04.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:04.737+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.739837+0000) 2019-10-13T08:50:04.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:04.991+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.992434+0000) 2019-10-13T08:50:06.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:06.211+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:06.214486+0000) 2019-10-13T08:50:08.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:08.479+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.482339+0000) 2019-10-13T08:50:09.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:09.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.467279+0000) 2019-10-13T08:50:09.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:09.735+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.736194+0000) 2019-10-13T08:50:09.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:09.735+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.737103+0000) 2019-10-13T08:50:09.738 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:09.735+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.737960+0000) 2019-10-13T08:50:09.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:09.735+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.738212+0000) 2019-10-13T08:50:09.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:09.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:09.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:09.735+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.738786+0000) 2019-10-13T08:50:09.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:09.739+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.739367+0000) 2019-10-13T08:50:09.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:09.737+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.740256+0000) 2019-10-13T08:50:09.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:09.741+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.743623+0000) 2019-10-13T08:50:11.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:11.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.101939+0000) 2019-10-13T08:50:11.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:11.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.725488+0000) 2019-10-13T08:50:11.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:11.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.771397+0000) 2019-10-13T08:50:11.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:11.962+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.965262+0000) 2019-10-13T08:50:14.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:14.738+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.742583+0000) 2019-10-13T08:50:14.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:14.738+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.742664+0000) 2019-10-13T08:50:14.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:14.738+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.743133+0000) 2019-10-13T08:50:14.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:14.742+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.743968+0000) 2019-10-13T08:50:14.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:14.742+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:14.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:14.741+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.744817+0000) 2019-10-13T08:50:14.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:14.742+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.745253+0000) 2019-10-13T08:50:14.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:14.742+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.745877+0000) 2019-10-13T08:50:14.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:14.749+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.751309+0000) 2019-10-13T08:50:14.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:14.990+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.992619+0000) 2019-10-13T08:50:16.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:16.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:16.214700+0000) 2019-10-13T08:50:18.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:18.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.482554+0000) 2019-10-13T08:50:19.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:19.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.467480+0000) 2019-10-13T08:50:19.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:19.746+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.747713+0000) 2019-10-13T08:50:19.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:19.745+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.748117+0000) 2019-10-13T08:50:19.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:19.746+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.748579+0000) 2019-10-13T08:50:19.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:19.746+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.748871+0000) 2019-10-13T08:50:19.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:19.746+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.749672+0000) 2019-10-13T08:50:19.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:19.746+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.750021+0000) 2019-10-13T08:50:19.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:19.746+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.750372+0000) 2019-10-13T08:50:19.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:19.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:19.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:19.757+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.758417+0000) 2019-10-13T08:50:21.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:21.105+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.108926+0000) 2019-10-13T08:50:21.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:21.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.725690+0000) 2019-10-13T08:50:21.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:21.770+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.771600+0000) 2019-10-13T08:50:21.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:21.962+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.965473+0000) 2019-10-13T08:50:24.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:24.749+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.753989+0000) 2019-10-13T08:50:24.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:24.750+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.754543+0000) 2019-10-13T08:50:24.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:24.750+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.754683+0000) 2019-10-13T08:50:24.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:24.754+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.755619+0000) 2019-10-13T08:50:24.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:24.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:24.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:24.754+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.756317+0000) 2019-10-13T08:50:24.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:24.754+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.757120+0000) 2019-10-13T08:50:24.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:24.754+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.759061+0000) 2019-10-13T08:50:24.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:24.765+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.767260+0000) 2019-10-13T08:50:24.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:24.990+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.992800+0000) 2019-10-13T08:50:26.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:26.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:26.214906+0000) 2019-10-13T08:50:28.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:28.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.482761+0000) 2019-10-13T08:50:29.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:29.465+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.467675+0000) 2019-10-13T08:50:29.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:29.758+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.759521+0000) 2019-10-13T08:50:29.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:29.758+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.760290+0000) 2019-10-13T08:50:29.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:29.758+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.760338+0000) 2019-10-13T08:50:29.763 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:29.757+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.760665+0000) 2019-10-13T08:50:29.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:29.758+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.761355+0000) 2019-10-13T08:50:29.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:29.758+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.761935+0000) 2019-10-13T08:50:29.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:29.758+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61211 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:29.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:29.758+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.762241+0000) 2019-10-13T08:50:29.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:29.765+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.769012+0000) 2019-10-13T08:50:31.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:31.113+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.115828+0000) 2019-10-13T08:50:31.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:31.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.725885+0000) 2019-10-13T08:50:31.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:31.770+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.771838+0000) 2019-10-13T08:50:31.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:31.962+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.965675+0000) 2019-10-13T08:50:34.183 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:50:34.183 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:34.187 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:50:34.187 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:34.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:34.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.765838+0000) 2019-10-13T08:50:34.767 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:34.762+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.766547+0000) 2019-10-13T08:50:34.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:34.762+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.767083+0000) 2019-10-13T08:50:34.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:34.766+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.767516+0000) 2019-10-13T08:50:34.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:34.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:34.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:34.766+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.768454+0000) 2019-10-13T08:50:34.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:34.766+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.768875+0000) 2019-10-13T08:50:34.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:34.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.770177+0000) 2019-10-13T08:50:34.775 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:34.772+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.774621+0000) 2019-10-13T08:50:34.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:34.990+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.992975+0000) 2019-10-13T08:50:36.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:36.214+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:36.215049+0000) 2019-10-13T08:50:38.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:38.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.482956+0000) 2019-10-13T08:50:39.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:39.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.467869+0000) 2019-10-13T08:50:39.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:39.770+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.771448+0000) 2019-10-13T08:50:39.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:39.770+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.772439+0000) 2019-10-13T08:50:39.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:39.770+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:39.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:39.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.773246+0000) 2019-10-13T08:50:39.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:39.770+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.773484+0000) 2019-10-13T08:50:39.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:39.770+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.773785+0000) 2019-10-13T08:50:39.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:39.770+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.774055+0000) 2019-10-13T08:50:39.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:39.770+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.775024+0000) 2019-10-13T08:50:39.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:39.776+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.778667+0000) 2019-10-13T08:50:41.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:41.120+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.122723+0000) 2019-10-13T08:50:41.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:41.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.726083+0000) 2019-10-13T08:50:41.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:41.770+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.772099+0000) 2019-10-13T08:50:41.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:41.962+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.965837+0000) 2019-10-13T08:50:44.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:44.774+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.775940+0000) 2019-10-13T08:50:44.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:44.774+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.776464+0000) 2019-10-13T08:50:44.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:44.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.777347+0000) 2019-10-13T08:50:44.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:44.774+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.777769+0000) 2019-10-13T08:50:44.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:44.774+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.778211+0000) 2019-10-13T08:50:44.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:44.774+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:44.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:44.776+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.779494+0000) 2019-10-13T08:50:44.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:44.778+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.780117+0000) 2019-10-13T08:50:44.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:44.784+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.786345+0000) 2019-10-13T08:50:44.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:44.990+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.993187+0000) 2019-10-13T08:50:46.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:46.210+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:46.215255+0000) 2019-10-13T08:50:48.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:48.478+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.483154+0000) 2019-10-13T08:50:49.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:49.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.468066+0000) 2019-10-13T08:50:49.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:49.778+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.782099+0000) 2019-10-13T08:50:49.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:49.780+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.782807+0000) 2019-10-13T08:50:49.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:49.782+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.783949+0000) 2019-10-13T08:50:49.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:49.782+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.784253+0000) 2019-10-13T08:50:49.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:49.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:49.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:49.782+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.784840+0000) 2019-10-13T08:50:49.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:49.782+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.784990+0000) 2019-10-13T08:50:49.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:49.782+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.785803+0000) 2019-10-13T08:50:49.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:49.784+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.789757+0000) 2019-10-13T08:50:51.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:51.124+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.129594+0000) 2019-10-13T08:50:51.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:51.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.726287+0000) 2019-10-13T08:50:51.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:51.770+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.772300+0000) 2019-10-13T08:50:51.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:51.962+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.966066+0000) 2019-10-13T08:50:54.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:54.784+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.788002+0000) 2019-10-13T08:50:54.789 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:54.786+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.788574+0000) 2019-10-13T08:50:54.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:54.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.789350+0000) 2019-10-13T08:50:54.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:54.786+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.789079+0000) 2019-10-13T08:50:54.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:54.786+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.789714+0000) 2019-10-13T08:50:54.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:54.786+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.790196+0000) 2019-10-13T08:50:54.791 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:54.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:54.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:54.786+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.791169+0000) 2019-10-13T08:50:54.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:54.796+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.798277+0000) 2019-10-13T08:50:54.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:54.990+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.993423+0000) 2019-10-13T08:50:56.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:56.213+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:56.215459+0000) 2019-10-13T08:50:58.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:58.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.483338+0000) 2019-10-13T08:50:59.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:59.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.468290+0000) 2019-10-13T08:50:59.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:50:59.789+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.793391+0000) 2019-10-13T08:50:59.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:50:59.789+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.794129+0000) 2019-10-13T08:50:59.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:50:59.789+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.793891+0000) 2019-10-13T08:50:59.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:50:59.789+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.794705+0000) 2019-10-13T08:50:59.797 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:50:59.792+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.795448+0000) 2019-10-13T08:50:59.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:50:59.789+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.795121+0000) 2019-10-13T08:50:59.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:50:59.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:50:59.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:50:59.793+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.796363+0000) 2019-10-13T08:50:59.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:50:59.796+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.801929+0000) 2019-10-13T08:51:01.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:01.136+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.136582+0000) 2019-10-13T08:51:01.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:01.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.726522+0000) 2019-10-13T08:51:01.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:01.769+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.772514+0000) 2019-10-13T08:51:01.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:01.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.966293+0000) 2019-10-13T08:51:04.255 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:51:04.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:04.259 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:51:04.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:04.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:04.793+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.799080+0000) 2019-10-13T08:51:04.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:04.797+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.799390+0000) 2019-10-13T08:51:04.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:04.797+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.800556+0000) 2019-10-13T08:51:04.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:04.796+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.800998+0000) 2019-10-13T08:51:04.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:04.797+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.801444+0000) 2019-10-13T08:51:04.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:04.797+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.801831+0000) 2019-10-13T08:51:04.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:04.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:04.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:04.797+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.803058+0000) 2019-10-13T08:51:04.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:04.804+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.809235+0000) 2019-10-13T08:51:04.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:04.989+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.993603+0000) 2019-10-13T08:51:06.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:06.213+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:06.215635+0000) 2019-10-13T08:51:08.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:08.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.483533+0000) 2019-10-13T08:51:09.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:09.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.468484+0000) 2019-10-13T08:51:09.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:09.801+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.805345+0000) 2019-10-13T08:51:09.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:09.801+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.805728+0000) 2019-10-13T08:51:09.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:09.801+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.806172+0000) 2019-10-13T08:51:09.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:09.801+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.806874+0000) 2019-10-13T08:51:09.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:09.804+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.807434+0000) 2019-10-13T08:51:09.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:09.805+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.807938+0000) 2019-10-13T08:51:09.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:09.805+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:09.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:09.805+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.808891+0000) 2019-10-13T08:51:09.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:09.812+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.814358+0000) 2019-10-13T08:51:11.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:11.144+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.145560+0000) 2019-10-13T08:51:11.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:11.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.726725+0000) 2019-10-13T08:51:11.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:11.769+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.772712+0000) 2019-10-13T08:51:11.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:11.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.966485+0000) 2019-10-13T08:51:14.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:14.809+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.811544+0000) 2019-10-13T08:51:14.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:14.809+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.811912+0000) 2019-10-13T08:51:14.813 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:14.809+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.812813+0000) 2019-10-13T08:51:14.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:14.809+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.813434+0000) 2019-10-13T08:51:14.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:14.809+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.814160+0000) 2019-10-13T08:51:14.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:14.809+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.813919+0000) 2019-10-13T08:51:14.815 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:14.812+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.814925+0000) 2019-10-13T08:51:14.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:14.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:14.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:14.816+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.818442+0000) 2019-10-13T08:51:14.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:14.989+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.993799+0000) 2019-10-13T08:51:16.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:16.213+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:16.215851+0000) 2019-10-13T08:51:18.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:18.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.483732+0000) 2019-10-13T08:51:19.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:19.464+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.468679+0000) 2019-10-13T08:51:19.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:19.813+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.818328+0000) 2019-10-13T08:51:19.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:19.813+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.818947+0000) 2019-10-13T08:51:19.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:19.817+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.819452+0000) 2019-10-13T08:51:19.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:19.817+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.820081+0000) 2019-10-13T08:51:19.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:19.817+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.820311+0000) 2019-10-13T08:51:19.821 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:19.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61549 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:19.821 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:19.816+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.820827+0000) 2019-10-13T08:51:19.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:19.817+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.821176+0000) 2019-10-13T08:51:19.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:19.824+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.829136+0000) 2019-10-13T08:51:21.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:21.152+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.157082+0000) 2019-10-13T08:51:21.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:21.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.726938+0000) 2019-10-13T08:51:21.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:21.769+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.772909+0000) 2019-10-13T08:51:21.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:21.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.966709+0000) 2019-10-13T08:51:24.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:24.821+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.824474+0000) 2019-10-13T08:51:24.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:24.821+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.824947+0000) 2019-10-13T08:51:24.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:24.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.825147+0000) 2019-10-13T08:51:24.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:24.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.825552+0000) 2019-10-13T08:51:24.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:24.821+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.826392+0000) 2019-10-13T08:51:24.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:24.824+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.826736+0000) 2019-10-13T08:51:24.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:24.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:24.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:24.825+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.827594+0000) 2019-10-13T08:51:24.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:24.828+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.832365+0000) 2019-10-13T08:51:24.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:24.989+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.993975+0000) 2019-10-13T08:51:26.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:26.213+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:26.216074+0000) 2019-10-13T08:51:28.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:28.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.483932+0000) 2019-10-13T08:51:29.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:29.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.468869+0000) 2019-10-13T08:51:29.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:29.825+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.830579+0000) 2019-10-13T08:51:29.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:29.825+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.831224+0000) 2019-10-13T08:51:29.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:29.829+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.832137+0000) 2019-10-13T08:51:29.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:29.827+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.832436+0000) 2019-10-13T08:51:29.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:29.829+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.832681+0000) 2019-10-13T08:51:29.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:29.829+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.833316+0000) 2019-10-13T08:51:29.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:29.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:29.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:29.829+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.833979+0000) 2019-10-13T08:51:29.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:29.835+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.839765+0000) 2019-10-13T08:51:31.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:31.159+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.163610+0000) 2019-10-13T08:51:31.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:31.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.727145+0000) 2019-10-13T08:51:31.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:31.769+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.773111+0000) 2019-10-13T08:51:31.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:31.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.966909+0000) 2019-10-13T08:51:34.387 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:51:34.388 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:34.391 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:51:34.391 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:34.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:34.833+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.837266+0000) 2019-10-13T08:51:34.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:34.833+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.838196+0000) 2019-10-13T08:51:34.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:34.833+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.838960+0000) 2019-10-13T08:51:34.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:34.837+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.839428+0000) 2019-10-13T08:51:34.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:34.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.839076+0000) 2019-10-13T08:51:34.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:34.835+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.839914+0000) 2019-10-13T08:51:34.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:34.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:34.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:34.837+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.840255+0000) 2019-10-13T08:51:34.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:34.839+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.844597+0000) 2019-10-13T08:51:34.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:34.993+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.994161+0000) 2019-10-13T08:51:36.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:36.213+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:36.216262+0000) 2019-10-13T08:51:38.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:38.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.484183+0000) 2019-10-13T08:51:39.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:39.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.469069+0000) 2019-10-13T08:51:39.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:39.840+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.843289+0000) 2019-10-13T08:51:39.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:39.840+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.844768+0000) 2019-10-13T08:51:39.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:39.840+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.845232+0000) 2019-10-13T08:51:39.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:39.839+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.845409+0000) 2019-10-13T08:51:39.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:39.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:39.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:39.840+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.846382+0000) 2019-10-13T08:51:39.848 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:39.840+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.846979+0000) 2019-10-13T08:51:39.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:39.844+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.847422+0000) 2019-10-13T08:51:39.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:39.847+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.852674+0000) 2019-10-13T08:51:41.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:41.167+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.170027+0000) 2019-10-13T08:51:41.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:41.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.727344+0000) 2019-10-13T08:51:41.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:41.768+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.773313+0000) 2019-10-13T08:51:41.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:41.960+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.967115+0000) 2019-10-13T08:51:44.849 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:44.844+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.849277+0000) 2019-10-13T08:51:44.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:44.844+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.849689+0000) 2019-10-13T08:51:44.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:44.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.850634+0000) 2019-10-13T08:51:44.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:44.844+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.850784+0000) 2019-10-13T08:51:44.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:44.844+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.851161+0000) 2019-10-13T08:51:44.852 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:44.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:44.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:44.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.851690+0000) 2019-10-13T08:51:44.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:44.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.852261+0000) 2019-10-13T08:51:44.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:44.851+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.856550+0000) 2019-10-13T08:51:44.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:44.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.994359+0000) 2019-10-13T08:51:46.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:46.212+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:46.216461+0000) 2019-10-13T08:51:48.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:48.480+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.484381+0000) 2019-10-13T08:51:49.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:49.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.469274+0000) 2019-10-13T08:51:49.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:49.848+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.854891+0000) 2019-10-13T08:51:49.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:49.848+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.855095+0000) 2019-10-13T08:51:49.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:49.852+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.855527+0000) 2019-10-13T08:51:49.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:49.852+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.856534+0000) 2019-10-13T08:51:49.857 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:49.852+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.856876+0000) 2019-10-13T08:51:49.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:49.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:49.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:49.852+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.858004+0000) 2019-10-13T08:51:49.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:49.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.858403+0000) 2019-10-13T08:51:49.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:49.859+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.862918+0000) 2019-10-13T08:51:51.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:51.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.176613+0000) 2019-10-13T08:51:51.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:51.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.727548+0000) 2019-10-13T08:51:51.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:51.768+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.773593+0000) 2019-10-13T08:51:51.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:51.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.967345+0000) 2019-10-13T08:51:54.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:54.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.860264+0000) 2019-10-13T08:51:54.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:54.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.860605+0000) 2019-10-13T08:51:54.862 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:54.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.861290+0000) 2019-10-13T08:51:54.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:54.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.861899+0000) 2019-10-13T08:51:54.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:54.856+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.862555+0000) 2019-10-13T08:51:54.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:54.856+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.862661+0000) 2019-10-13T08:51:54.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:54.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:54.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:54.860+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.864054+0000) 2019-10-13T08:51:54.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:54.863+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.869131+0000) 2019-10-13T08:51:54.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:54.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.994543+0000) 2019-10-13T08:51:56.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:56.216+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:56.216686+0000) 2019-10-13T08:51:58.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:58.480+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.484607+0000) 2019-10-13T08:51:59.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:59.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.469465+0000) 2019-10-13T08:51:59.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:51:59.860+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.866301+0000) 2019-10-13T08:51:59.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:51:59.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.867318+0000) 2019-10-13T08:51:59.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:51:59.864+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.867696+0000) 2019-10-13T08:51:59.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:51:59.864+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.868298+0000) 2019-10-13T08:51:59.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:51:59.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:51:59.869 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:51:59.864+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.868702+0000) 2019-10-13T08:51:59.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:51:59.864+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.868925+0000) 2019-10-13T08:51:59.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:51:59.864+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.869222+0000) 2019-10-13T08:51:59.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:51:59.871+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.874365+0000) 2019-10-13T08:52:01.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:01.187+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.188365+0000) 2019-10-13T08:52:01.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:01.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.727775+0000) 2019-10-13T08:52:01.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:01.768+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.773801+0000) 2019-10-13T08:52:01.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:01.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.967595+0000) 2019-10-13T08:52:04.501 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:52:04.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:04.505 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:52:04.505 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:04.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:04.867+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.871701+0000) 2019-10-13T08:52:04.873 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:04.868+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.872826+0000) 2019-10-13T08:52:04.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:04.868+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.873171+0000) 2019-10-13T08:52:04.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:04.868+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.873557+0000) 2019-10-13T08:52:04.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:04.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.873833+0000) 2019-10-13T08:52:04.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:04.868+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.874542+0000) 2019-10-13T08:52:04.875 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:04.868+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:04.876 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:04.872+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.875667+0000) 2019-10-13T08:52:04.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:04.879+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.882642+0000) 2019-10-13T08:52:04.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:04.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.994744+0000) 2019-10-13T08:52:06.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:06.212+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:06.216901+0000) 2019-10-13T08:52:08.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:08.480+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:08.484804+0000) 2019-10-13T08:52:09.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:09.463+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.469624+0000) 2019-10-13T08:52:09.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:09.871+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.876962+0000) 2019-10-13T08:52:09.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:09.872+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.878313+0000) 2019-10-13T08:52:09.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:09.876+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.879349+0000) 2019-10-13T08:52:09.880 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:09.876+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.879877+0000) 2019-10-13T08:52:09.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:09.872+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.878975+0000) 2019-10-13T08:52:09.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:09.876+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.880623+0000) 2019-10-13T08:52:09.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:09.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61890 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:09.882 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:09.876+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.881461+0000) 2019-10-13T08:52:09.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:09.883+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.888378+0000) 2019-10-13T08:52:11.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:11.191+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.194759+0000) 2019-10-13T08:52:11.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:11.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.727973+0000) 2019-10-13T08:52:11.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:11.768+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.774035+0000) 2019-10-13T08:52:11.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:11.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.967824+0000) 2019-10-13T08:52:14.885 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:14.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.884442+0000) 2019-10-13T08:52:14.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:14.880+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.885045+0000) 2019-10-13T08:52:14.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:14.880+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.885349+0000) 2019-10-13T08:52:14.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:14.880+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.885687+0000) 2019-10-13T08:52:14.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:14.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:14.887 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:14.880+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.886370+0000) 2019-10-13T08:52:14.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:14.883+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.886866+0000) 2019-10-13T08:52:14.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:14.884+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.887784+0000) 2019-10-13T08:52:14.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:14.887+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.893907+0000) 2019-10-13T08:52:14.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:14.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.994937+0000) 2019-10-13T08:52:16.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:16.212+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:16.217045+0000) 2019-10-13T08:52:18.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:18.484+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:18.485004+0000) 2019-10-13T08:52:19.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:19.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.469853+0000) 2019-10-13T08:52:19.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:19.884+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.889257+0000) 2019-10-13T08:52:19.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:19.884+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.890534+0000) 2019-10-13T08:52:19.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:19.884+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.890943+0000) 2019-10-13T08:52:19.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:19.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.891206+0000) 2019-10-13T08:52:19.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:19.888+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.891809+0000) 2019-10-13T08:52:19.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:19.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.891891+0000) 2019-10-13T08:52:19.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:19.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:19.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:19.888+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.893855+0000) 2019-10-13T08:52:19.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:19.895+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.900169+0000) 2019-10-13T08:52:21.202 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:21.199+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.201282+0000) 2019-10-13T08:52:21.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:21.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.728183+0000) 2019-10-13T08:52:21.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:21.768+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.774250+0000) 2019-10-13T08:52:21.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:21.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.968017+0000) 2019-10-13T08:52:24.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:24.890+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.895086+0000) 2019-10-13T08:52:24.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:24.891+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.896180+0000) 2019-10-13T08:52:24.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:24.891+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.896464+0000) 2019-10-13T08:52:24.897 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:24.891+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.896809+0000) 2019-10-13T08:52:24.897 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:24.891+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.897219+0000) 2019-10-13T08:52:24.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:24.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 61993 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:24.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:24.891+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.897955+0000) 2019-10-13T08:52:24.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:24.891+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.898002+0000) 2019-10-13T08:52:24.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:24.898+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.904735+0000) 2019-10-13T08:52:24.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:24.991+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.995118+0000) 2019-10-13T08:52:26.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:26.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:26.217240+0000) 2019-10-13T08:52:28.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:28.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:28.485203+0000) 2019-10-13T08:52:29.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:29.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.470051+0000) 2019-10-13T08:52:29.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:29.895+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.901736+0000) 2019-10-13T08:52:29.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:29.895+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.901937+0000) 2019-10-13T08:52:29.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:29.895+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.902725+0000) 2019-10-13T08:52:29.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:29.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:29.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:29.899+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.903532+0000) 2019-10-13T08:52:29.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:29.899+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.904136+0000) 2019-10-13T08:52:29.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:29.898+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.903000+0000) 2019-10-13T08:52:29.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:29.899+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.905278+0000) 2019-10-13T08:52:29.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:29.902+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.908591+0000) 2019-10-13T08:52:31.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:31.206+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.207758+0000) 2019-10-13T08:52:31.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:31.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.728382+0000) 2019-10-13T08:52:31.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:31.767+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.774480+0000) 2019-10-13T08:52:31.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:31.967+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.968257+0000) 2019-10-13T08:52:34.592 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:52:34.592 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:34.598 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:52:34.598 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:34.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:34.903+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.906558+0000) 2019-10-13T08:52:34.907 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:34.903+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.906958+0000) 2019-10-13T08:52:34.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:34.902+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.909022+0000) 2019-10-13T08:52:34.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:34.903+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.909378+0000) 2019-10-13T08:52:34.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:34.903+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.909653+0000) 2019-10-13T08:52:34.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:34.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:34.911 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:34.903+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.910037+0000) 2019-10-13T08:52:34.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:34.903+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.910880+0000) 2019-10-13T08:52:34.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:34.906+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.913228+0000) 2019-10-13T08:52:34.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:34.991+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.995351+0000) 2019-10-13T08:52:36.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:36.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:36.217451+0000) 2019-10-13T08:52:38.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:38.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:38.485403+0000) 2019-10-13T08:52:39.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:39.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.470273+0000) 2019-10-13T08:52:39.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:39.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.912706+0000) 2019-10-13T08:52:39.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:39.907+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.913241+0000) 2019-10-13T08:52:39.914 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:39.907+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.914268+0000) 2019-10-13T08:52:39.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:39.907+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.914802+0000) 2019-10-13T08:52:39.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:39.907+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.914673+0000) 2019-10-13T08:52:39.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:39.911+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.915802+0000) 2019-10-13T08:52:39.916 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:39.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:39.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:39.907+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.915168+0000) 2019-10-13T08:52:39.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:39.918+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.922733+0000) 2019-10-13T08:52:41.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:41.210+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.214245+0000) 2019-10-13T08:52:41.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:41.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.728585+0000) 2019-10-13T08:52:41.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:41.767+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.774688+0000) 2019-10-13T08:52:41.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:41.967+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.968460+0000) 2019-10-13T08:52:44.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:44.914+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.920019+0000) 2019-10-13T08:52:44.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:44.915+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.920264+0000) 2019-10-13T08:52:44.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:44.915+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.920812+0000) 2019-10-13T08:52:44.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:44.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:44.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:44.915+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.922024+0000) 2019-10-13T08:52:44.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:44.915+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.921733+0000) 2019-10-13T08:52:44.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:44.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.922363+0000) 2019-10-13T08:52:44.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:44.915+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.922488+0000) 2019-10-13T08:52:44.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:44.922+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.927238+0000) 2019-10-13T08:52:44.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:44.995+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.995558+0000) 2019-10-13T08:52:46.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:46.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:46.217680+0000) 2019-10-13T08:52:48.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:48.479+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:48.485588+0000) 2019-10-13T08:52:49.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:49.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.470467+0000) 2019-10-13T08:52:49.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:49.919+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.924424+0000) 2019-10-13T08:52:49.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:49.918+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.925290+0000) 2019-10-13T08:52:49.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:49.919+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.925905+0000) 2019-10-13T08:52:49.926 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:49.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.926376+0000) 2019-10-13T08:52:49.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:49.923+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.927407+0000) 2019-10-13T08:52:49.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:49.919+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.927118+0000) 2019-10-13T08:52:49.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:49.923+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.927653+0000) 2019-10-13T08:52:49.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:49.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:49.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:49.926+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.933379+0000) 2019-10-13T08:52:51.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:51.218+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.220436+0000) 2019-10-13T08:52:51.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:51.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.728850+0000) 2019-10-13T08:52:51.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:51.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.774889+0000) 2019-10-13T08:52:51.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:51.967+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.968673+0000) 2019-10-13T08:52:54.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:54.926+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.930711+0000) 2019-10-13T08:52:54.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:54.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.931225+0000) 2019-10-13T08:52:54.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:54.931+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.931942+0000) 2019-10-13T08:52:54.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:54.931+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.932460+0000) 2019-10-13T08:52:54.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:54.931+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.933169+0000) 2019-10-13T08:52:54.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:54.931+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.933263+0000) 2019-10-13T08:52:54.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:54.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:54.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:54.931+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.934177+0000) 2019-10-13T08:52:54.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:54.934+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.940425+0000) 2019-10-13T08:52:54.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:54.995+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.995733+0000) 2019-10-13T08:52:56.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:56.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:56.217876+0000) 2019-10-13T08:52:58.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:58.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:58.485791+0000) 2019-10-13T08:52:59.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:59.466+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.470681+0000) 2019-10-13T08:52:59.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:52:59.935+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.937409+0000) 2019-10-13T08:52:59.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:52:59.935+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.938219+0000) 2019-10-13T08:52:59.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:52:59.934+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.938672+0000) 2019-10-13T08:52:59.940 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:52:59.935+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.939029+0000) 2019-10-13T08:52:59.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:52:59.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.939690+0000) 2019-10-13T08:52:59.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:52:59.939+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.939700+0000) 2019-10-13T08:52:59.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:52:59.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:52:59.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:52:59.939+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.940148+0000) 2019-10-13T08:52:59.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:52:59.942+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.946579+0000) 2019-10-13T08:53:01.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:01.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.226790+0000) 2019-10-13T08:53:01.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:01.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.729078+0000) 2019-10-13T08:53:01.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:01.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.775088+0000) 2019-10-13T08:53:01.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:01.967+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.968885+0000) 2019-10-13T08:53:04.722 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:53:04.722 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:04.725 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:53:04.726 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:04.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:04.943+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.943678+0000) 2019-10-13T08:53:04.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:04.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.944627+0000) 2019-10-13T08:53:04.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:04.943+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.945112+0000) 2019-10-13T08:53:04.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:04.943+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.945353+0000) 2019-10-13T08:53:04.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:04.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.945808+0000) 2019-10-13T08:53:04.947 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:04.943+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.946162+0000) 2019-10-13T08:53:04.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:04.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62264 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:04.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:04.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.946410+0000) 2019-10-13T08:53:04.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:04.946+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.950295+0000) 2019-10-13T08:53:04.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:04.995+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.995903+0000) 2019-10-13T08:53:06.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:06.214+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:06.218066+0000) 2019-10-13T08:53:08.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:08.482+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:08.485990+0000) 2019-10-13T08:53:09.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:09.470+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.470887+0000) 2019-10-13T08:53:09.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:09.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.948873+0000) 2019-10-13T08:53:09.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:09.946+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.949659+0000) 2019-10-13T08:53:09.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:09.946+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.949961+0000) 2019-10-13T08:53:09.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:09.950+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.950668+0000) 2019-10-13T08:53:09.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:09.946+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.950977+0000) 2019-10-13T08:53:09.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:09.950+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.951439+0000) 2019-10-13T08:53:09.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:09.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:09.952 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:09.950+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.951788+0000) 2019-10-13T08:53:09.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:09.954+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.956632+0000) 2019-10-13T08:53:11.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:11.230+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.233199+0000) 2019-10-13T08:53:11.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:11.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.729300+0000) 2019-10-13T08:53:11.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:11.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.775291+0000) 2019-10-13T08:53:11.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:11.966+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.969080+0000) 2019-10-13T08:53:14.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:14.954+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.954400+0000) 2019-10-13T08:53:14.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:14.954+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.955392+0000) 2019-10-13T08:53:14.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:14.954+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.955888+0000) 2019-10-13T08:53:14.956 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:14.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.956245+0000) 2019-10-13T08:53:14.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:14.954+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.956743+0000) 2019-10-13T08:53:14.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:14.954+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.957541+0000) 2019-10-13T08:53:14.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:14.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:14.959 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:14.954+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.957795+0000) 2019-10-13T08:53:14.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:14.962+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.962827+0000) 2019-10-13T08:53:14.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:14.994+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.996107+0000) 2019-10-13T08:53:16.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:16.214+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:16.218261+0000) 2019-10-13T08:53:18.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:18.482+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:18.486194+0000) 2019-10-13T08:53:19.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:19.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.471120+0000) 2019-10-13T08:53:19.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:19.957+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.960132+0000) 2019-10-13T08:53:19.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:19.958+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.960771+0000) 2019-10-13T08:53:19.962 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:19.958+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.961653+0000) 2019-10-13T08:53:19.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:19.958+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.961944+0000) 2019-10-13T08:53:19.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:19.958+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.962383+0000) 2019-10-13T08:53:19.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:19.958+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.962615+0000) 2019-10-13T08:53:19.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:19.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:19.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:19.962+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.963997+0000) 2019-10-13T08:53:19.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:19.965+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.969020+0000) 2019-10-13T08:53:21.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:21.237+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.239538+0000) 2019-10-13T08:53:21.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:21.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.729524+0000) 2019-10-13T08:53:21.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:21.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.775506+0000) 2019-10-13T08:53:21.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:21.966+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.969303+0000) 2019-10-13T08:53:24.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:24.962+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.966080+0000) 2019-10-13T08:53:24.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:24.965+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.966516+0000) 2019-10-13T08:53:24.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:24.966+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.967360+0000) 2019-10-13T08:53:24.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:24.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.967079+0000) 2019-10-13T08:53:24.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:24.966+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.967925+0000) 2019-10-13T08:53:24.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:24.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:24.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:24.966+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.968850+0000) 2019-10-13T08:53:24.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:24.966+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.969569+0000) 2019-10-13T08:53:24.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:24.973+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.975261+0000) 2019-10-13T08:53:24.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:24.994+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.996310+0000) 2019-10-13T08:53:26.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:26.214+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:26.218452+0000) 2019-10-13T08:53:28.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:28.482+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:28.486391+0000) 2019-10-13T08:53:29.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:29.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.471354+0000) 2019-10-13T08:53:29.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:29.970+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.971992+0000) 2019-10-13T08:53:29.973 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:29.970+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.972696+0000) 2019-10-13T08:53:29.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:29.970+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.973366+0000) 2019-10-13T08:53:29.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:29.969+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.973585+0000) 2019-10-13T08:53:29.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:29.970+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.973076+0000) 2019-10-13T08:53:29.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:29.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:29.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:29.970+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.974996+0000) 2019-10-13T08:53:29.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:29.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.975426+0000) 2019-10-13T08:53:29.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:29.977+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.980814+0000) 2019-10-13T08:53:31.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:31.241+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.245898+0000) 2019-10-13T08:53:31.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:31.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.729737+0000) 2019-10-13T08:53:31.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:31.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.775665+0000) 2019-10-13T08:53:31.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:31.966+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.969520+0000) 2019-10-13T08:53:34.820 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:53:34.820 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:34.824 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:53:34.824 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:34.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:34.977+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.978371+0000) 2019-10-13T08:53:34.979 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:34.974+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.978795+0000) 2019-10-13T08:53:34.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:34.978+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.979288+0000) 2019-10-13T08:53:34.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:34.978+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.979949+0000) 2019-10-13T08:53:34.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:34.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:34.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:34.978+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.980309+0000) 2019-10-13T08:53:34.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:34.978+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.980862+0000) 2019-10-13T08:53:34.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:34.978+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.981708+0000) 2019-10-13T08:53:34.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:34.985+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.987469+0000) 2019-10-13T08:53:34.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:34.994+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.996535+0000) 2019-10-13T08:53:36.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:36.214+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:36.218650+0000) 2019-10-13T08:53:38.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:38.482+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:38.486615+0000) 2019-10-13T08:53:39.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:39.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.471551+0000) 2019-10-13T08:53:39.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:39.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.982756+0000) 2019-10-13T08:53:39.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:39.982+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.983925+0000) 2019-10-13T08:53:39.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:39.982+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.983555+0000) 2019-10-13T08:53:39.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:39.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:39.986 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:39.982+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.985217+0000) 2019-10-13T08:53:39.986 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:39.982+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.985744+0000) 2019-10-13T08:53:39.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:39.982+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.986251+0000) 2019-10-13T08:53:39.987 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:39.982+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.987117+0000) 2019-10-13T08:53:39.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:39.989+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.990526+0000) 2019-10-13T08:53:41.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:41.249+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.252373+0000) 2019-10-13T08:53:41.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:41.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.729943+0000) 2019-10-13T08:53:41.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:41.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.775889+0000) 2019-10-13T08:53:41.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:41.966+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.969671+0000) 2019-10-13T08:53:44.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:44.986+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.988699+0000) 2019-10-13T08:53:44.989 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:44.986+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.989225+0000) 2019-10-13T08:53:44.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:44.986+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.989985+0000) 2019-10-13T08:53:44.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:44.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:44.991 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:44.986+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.990932+0000) 2019-10-13T08:53:44.992 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:44.990+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.991627+0000) 2019-10-13T08:53:44.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:44.989+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.991921+0000) 2019-10-13T08:53:44.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:44.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.992402+0000) 2019-10-13T08:53:44.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:44.994+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.996685+0000) 2019-10-13T08:53:44.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:44.993+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.996830+0000) 2019-10-13T08:53:46.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:46.218+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:46.218844+0000) 2019-10-13T08:53:48.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:48.482+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:48.486810+0000) 2019-10-13T08:53:49.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:49.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.471746+0000) 2019-10-13T08:53:49.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:49.989+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.993725+0000) 2019-10-13T08:53:49.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:49.989+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.993977+0000) 2019-10-13T08:53:49.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:49.989+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.994281+0000) 2019-10-13T08:53:49.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:49.993+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.995802+0000) 2019-10-13T08:53:49.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:49.993+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.996057+0000) 2019-10-13T08:53:49.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:49.993+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.996561+0000) 2019-10-13T08:53:49.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:49.993+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.996887+0000) 2019-10-13T08:53:49.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:49.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:50.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:49.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:50.001591+0000) 2019-10-13T08:53:51.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:51.257+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.258740+0000) 2019-10-13T08:53:51.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:51.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.730213+0000) 2019-10-13T08:53:51.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:51.773+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.776099+0000) 2019-10-13T08:53:51.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:51.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.969899+0000) 2019-10-13T08:53:54.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:54.993+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.996853+0000) 2019-10-13T08:53:55.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:54.997+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.999774+0000) 2019-10-13T08:53:55.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:53:54.997+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.000410+0000) 2019-10-13T08:53:55.001 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:54.997+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.000662+0000) 2019-10-13T08:53:55.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:53:54.997+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.001558+0000) 2019-10-13T08:53:55.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:53:54.997+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.001372+0000) 2019-10-13T08:53:55.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:53:54.997+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.001927+0000) 2019-10-13T08:53:55.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:54.997+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.002220+0000) 2019-10-13T08:53:55.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:53:55.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:53:55.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:53:55.009+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:55.010507+0000) 2019-10-13T08:53:56.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:53:56.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:56.219043+0000) 2019-10-13T08:53:58.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:53:58.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:58.486997+0000) 2019-10-13T08:53:59.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:53:59.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.471966+0000) 2019-10-13T08:54:00.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:00.001+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.006310+0000) 2019-10-13T08:54:00.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:00.005+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.007147+0000) 2019-10-13T08:54:00.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:00.005+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.007591+0000) 2019-10-13T08:54:00.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:00.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:00.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:00.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.009772+0000) 2019-10-13T08:54:00.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:00.005+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.010093+0000) 2019-10-13T08:54:00.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:00.005+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.011007+0000) 2019-10-13T08:54:00.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:00.009+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.011629+0000) 2019-10-13T08:54:00.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:00.013+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:00.014242+0000) 2019-10-13T08:54:01.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:01.261+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.265165+0000) 2019-10-13T08:54:01.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:01.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.730479+0000) 2019-10-13T08:54:01.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:01.773+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.776340+0000) 2019-10-13T08:54:01.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:01.969+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.970103+0000) 2019-10-13T08:54:04.935 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:54:04.935 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:04.938 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:54:04.939 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:04.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:04.993+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.997070+0000) 2019-10-13T08:54:05.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:05.009+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.011333+0000) 2019-10-13T08:54:05.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:05.009+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.011435+0000) 2019-10-13T08:54:05.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:05.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.011787+0000) 2019-10-13T08:54:05.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:05.009+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.012932+0000) 2019-10-13T08:54:05.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:05.009+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.013601+0000) 2019-10-13T08:54:05.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:05.009+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.013794+0000) 2019-10-13T08:54:05.017 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:05.009+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.013988+0000) 2019-10-13T08:54:05.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:05.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:05.023 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:05.021+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:05.022786+0000) 2019-10-13T08:54:06.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:06.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:06.219241+0000) 2019-10-13T08:54:08.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:08.481+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:08.487211+0000) 2019-10-13T08:54:09.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:09.469+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.472168+0000) 2019-10-13T08:54:10.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:10.013+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.017019+0000) 2019-10-13T08:54:10.018 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:10.013+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.017259+0000) 2019-10-13T08:54:10.018 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:10.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.017665+0000) 2019-10-13T08:54:10.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:10.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.018129+0000) 2019-10-13T08:54:10.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:10.013+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.018995+0000) 2019-10-13T08:54:10.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:10.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:10.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:10.017+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.019685+0000) 2019-10-13T08:54:10.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:10.017+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.020557+0000) 2019-10-13T08:54:10.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:10.017+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:10.019453+0000) 2019-10-13T08:54:11.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:11.269+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.271467+0000) 2019-10-13T08:54:11.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:11.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.730678+0000) 2019-10-13T08:54:11.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:11.773+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.776559+0000) 2019-10-13T08:54:11.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:11.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.970258+0000) 2019-10-13T08:54:14.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:14.993+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.997282+0000) 2019-10-13T08:54:15.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:15.017+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.021615+0000) 2019-10-13T08:54:15.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:15.017+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.022342+0000) 2019-10-13T08:54:15.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:15.017+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.023133+0000) 2019-10-13T08:54:15.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:15.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.023756+0000) 2019-10-13T08:54:15.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:15.021+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.024073+0000) 2019-10-13T08:54:15.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:15.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:15.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:15.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.025034+0000) 2019-10-13T08:54:15.026 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:15.021+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.025599+0000) 2019-10-13T08:54:15.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:15.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:15.030748+0000) 2019-10-13T08:54:16.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:16.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:16.219474+0000) 2019-10-13T08:54:18.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:18.485+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:18.487418+0000) 2019-10-13T08:54:19.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:19.468+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.472413+0000) 2019-10-13T08:54:20.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:20.025+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.027568+0000) 2019-10-13T08:54:20.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:20.025+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.027891+0000) 2019-10-13T08:54:20.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:20.025+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.028631+0000) 2019-10-13T08:54:20.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:20.025+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.029186+0000) 2019-10-13T08:54:20.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:20.025+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.030184+0000) 2019-10-13T08:54:20.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:20.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:20.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:20.028+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.031013+0000) 2019-10-13T08:54:20.032 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:20.025+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.031021+0000) 2019-10-13T08:54:20.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:20.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:20.033776+0000) 2019-10-13T08:54:21.278 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:21.272+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.277847+0000) 2019-10-13T08:54:21.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:21.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.730938+0000) 2019-10-13T08:54:21.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:21.773+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.776793+0000) 2019-10-13T08:54:21.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:21.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.970464+0000) 2019-10-13T08:54:24.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:24.993+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.997507+0000) 2019-10-13T08:54:25.033 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:25.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.032717+0000) 2019-10-13T08:54:25.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:25.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.033400+0000) 2019-10-13T08:54:25.034 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:25.029+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.033390+0000) 2019-10-13T08:54:25.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:25.029+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.033206+0000) 2019-10-13T08:54:25.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:25.029+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.035149+0000) 2019-10-13T08:54:25.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:25.032+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.036139+0000) 2019-10-13T08:54:25.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:25.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:25.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:25.033+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.036546+0000) 2019-10-13T08:54:25.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:25.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:25.041201+0000) 2019-10-13T08:54:26.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:26.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:26.219700+0000) 2019-10-13T08:54:28.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:28.485+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:28.487575+0000) 2019-10-13T08:54:29.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:29.468+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.472567+0000) 2019-10-13T08:54:30.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:30.033+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.038753+0000) 2019-10-13T08:54:30.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:30.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.039410+0000) 2019-10-13T08:54:30.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:30.037+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.039739+0000) 2019-10-13T08:54:30.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:30.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.040714+0000) 2019-10-13T08:54:30.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:30.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.041171+0000) 2019-10-13T08:54:30.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:30.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.041510+0000) 2019-10-13T08:54:30.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:30.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:30.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:30.037+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.042485+0000) 2019-10-13T08:54:30.047 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:30.044+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:30.046973+0000) 2019-10-13T08:54:31.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:31.288+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.289267+0000) 2019-10-13T08:54:31.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:31.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.731178+0000) 2019-10-13T08:54:31.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:31.773+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.777005+0000) 2019-10-13T08:54:31.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:31.965+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.970668+0000) 2019-10-13T08:54:34.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:34.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.997745+0000) 2019-10-13T08:54:35.015 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:54:35.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:35.018 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:54:35.018 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:35.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:35.040+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.044374+0000) 2019-10-13T08:54:35.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:35.040+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.044648+0000) 2019-10-13T08:54:35.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:35.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.045057+0000) 2019-10-13T08:54:35.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:35.040+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.045440+0000) 2019-10-13T08:54:35.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:35.040+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.045431+0000) 2019-10-13T08:54:35.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:35.040+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.046617+0000) 2019-10-13T08:54:35.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:35.040+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.047205+0000) 2019-10-13T08:54:35.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:35.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:35.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:35.048+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:35.051317+0000) 2019-10-13T08:54:36.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:36.216+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:36.219911+0000) 2019-10-13T08:54:38.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:38.484+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:38.487811+0000) 2019-10-13T08:54:39.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:39.468+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.472777+0000) 2019-10-13T08:54:40.051 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:40.044+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.050618+0000) 2019-10-13T08:54:40.051 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:40.044+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.051119+0000) 2019-10-13T08:54:40.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:40.048+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.051569+0000) 2019-10-13T08:54:40.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:40.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.052269+0000) 2019-10-13T08:54:40.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:40.048+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.052889+0000) 2019-10-13T08:54:40.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:40.048+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.052867+0000) 2019-10-13T08:54:40.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:40.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:40.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:40.048+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.054292+0000) 2019-10-13T08:54:40.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:40.056+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:40.061423+0000) 2019-10-13T08:54:41.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:41.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.295695+0000) 2019-10-13T08:54:41.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:41.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.731412+0000) 2019-10-13T08:54:41.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:41.772+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.777235+0000) 2019-10-13T08:54:41.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:41.964+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.970821+0000) 2019-10-13T08:54:44.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:44.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.997954+0000) 2019-10-13T08:54:45.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:45.052+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.056685+0000) 2019-10-13T08:54:45.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:45.052+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.058286+0000) 2019-10-13T08:54:45.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:45.052+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.058664+0000) 2019-10-13T08:54:45.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:45.052+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.059245+0000) 2019-10-13T08:54:45.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:45.056+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.059602+0000) 2019-10-13T08:54:45.060 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:45.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:45.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:45.056+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.060040+0000) 2019-10-13T08:54:45.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:45.056+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.061044+0000) 2019-10-13T08:54:45.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:45.060+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:45.065130+0000) 2019-10-13T08:54:46.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:46.216+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:46.220110+0000) 2019-10-13T08:54:48.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:48.484+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:48.488071+0000) 2019-10-13T08:54:49.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:49.468+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.472973+0000) 2019-10-13T08:54:50.063 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:50.056+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.062272+0000) 2019-10-13T08:54:50.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:50.060+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.064083+0000) 2019-10-13T08:54:50.065 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:50.060+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.064411+0000) 2019-10-13T08:54:50.065 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:50.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 62976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:50.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:50.060+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.064707+0000) 2019-10-13T08:54:50.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:50.060+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.065061+0000) 2019-10-13T08:54:50.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:50.060+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.065438+0000) 2019-10-13T08:54:50.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:50.060+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.066770+0000) 2019-10-13T08:54:50.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:50.068+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:50.070724+0000) 2019-10-13T08:54:51.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:51.300+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.302163+0000) 2019-10-13T08:54:51.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:51.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.731637+0000) 2019-10-13T08:54:51.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:51.772+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.777486+0000) 2019-10-13T08:54:51.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:51.968+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.971032+0000) 2019-10-13T08:54:54.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:54.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.998167+0000) 2019-10-13T08:54:55.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:55.064+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.067720+0000) 2019-10-13T08:54:55.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:55.064+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.068123+0000) 2019-10-13T08:54:55.071 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:54:55.064+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.070704+0000) 2019-10-13T08:54:55.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:54:55.064+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.070756+0000) 2019-10-13T08:54:55.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:54:55.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:54:55.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:55.068+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.071589+0000) 2019-10-13T08:54:55.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:54:55.064+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.071164+0000) 2019-10-13T08:54:55.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:54:55.068+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.072104+0000) 2019-10-13T08:54:55.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:54:55.072+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:55.076498+0000) 2019-10-13T08:54:56.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:54:56.216+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:56.220299+0000) 2019-10-13T08:54:58.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:54:58.484+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:58.488276+0000) 2019-10-13T08:54:59.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:54:59.472+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.473194+0000) 2019-10-13T08:55:00.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:00.068+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.074465+0000) 2019-10-13T08:55:00.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:00.068+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.074942+0000) 2019-10-13T08:55:00.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:00.072+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.076022+0000) 2019-10-13T08:55:00.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:00.072+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.076011+0000) 2019-10-13T08:55:00.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:00.072+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.076226+0000) 2019-10-13T08:55:00.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:00.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:00.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:00.072+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.076883+0000) 2019-10-13T08:55:00.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:00.072+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.077791+0000) 2019-10-13T08:55:00.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:00.076+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:00.081811+0000) 2019-10-13T08:55:01.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:01.304+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.308619+0000) 2019-10-13T08:55:01.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:01.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.731858+0000) 2019-10-13T08:55:01.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:01.772+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.777709+0000) 2019-10-13T08:55:01.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:01.968+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.971231+0000) 2019-10-13T08:55:04.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:04.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.998365+0000) 2019-10-13T08:55:05.081 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:05.076+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.080595+0000) 2019-10-13T08:55:05.082 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:05.076+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.081816+0000) 2019-10-13T08:55:05.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:05.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.082101+0000) 2019-10-13T08:55:05.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:05.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:05.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:05.080+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.082321+0000) 2019-10-13T08:55:05.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:05.076+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.082740+0000) 2019-10-13T08:55:05.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:05.076+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.083173+0000) 2019-10-13T08:55:05.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:05.080+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.084048+0000) 2019-10-13T08:55:05.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:05.088+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:05.091257+0000) 2019-10-13T08:55:05.125 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:55:05.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:05.129 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:55:05.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:06.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:06.216+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:06.220468+0000) 2019-10-13T08:55:08.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:08.484+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:08.488472+0000) 2019-10-13T08:55:09.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:09.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.473389+0000) 2019-10-13T08:55:10.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:10.080+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.085823+0000) 2019-10-13T08:55:10.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:10.080+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.086167+0000) 2019-10-13T08:55:10.088 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:10.080+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.086904+0000) 2019-10-13T08:55:10.088 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:10.080+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.087195+0000) 2019-10-13T08:55:10.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:10.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63112 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:10.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:10.083+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.087900+0000) 2019-10-13T08:55:10.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:10.084+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.088236+0000) 2019-10-13T08:55:10.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:10.084+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.088644+0000) 2019-10-13T08:55:10.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:10.091+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:10.096182+0000) 2019-10-13T08:55:11.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:11.311+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.315075+0000) 2019-10-13T08:55:11.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:11.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.732088+0000) 2019-10-13T08:55:11.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:11.772+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.777923+0000) 2019-10-13T08:55:11.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:11.968+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.971468+0000) 2019-10-13T08:55:14.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:14.992+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.998577+0000) 2019-10-13T08:55:15.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:15.087+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.090669+0000) 2019-10-13T08:55:15.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:15.088+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.091343+0000) 2019-10-13T08:55:15.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:15.088+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.092623+0000) 2019-10-13T08:55:15.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:15.088+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:15.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:15.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.093335+0000) 2019-10-13T08:55:15.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:15.088+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.093842+0000) 2019-10-13T08:55:15.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:15.088+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.095044+0000) 2019-10-13T08:55:15.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:15.092+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.095526+0000) 2019-10-13T08:55:15.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:15.095+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:15.100579+0000) 2019-10-13T08:55:16.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:16.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:16.220692+0000) 2019-10-13T08:55:18.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:18.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:18.488700+0000) 2019-10-13T08:55:19.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:19.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.473614+0000) 2019-10-13T08:55:20.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:20.091+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.096415+0000) 2019-10-13T08:55:20.097 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:20.091+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.096875+0000) 2019-10-13T08:55:20.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:20.091+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.097532+0000) 2019-10-13T08:55:20.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:20.095+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.098549+0000) 2019-10-13T08:55:20.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:20.091+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:20.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:20.091+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.098932+0000) 2019-10-13T08:55:20.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:20.095+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.099626+0000) 2019-10-13T08:55:20.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:20.095+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.100731+0000) 2019-10-13T08:55:20.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:20.103+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:20.107038+0000) 2019-10-13T08:55:21.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:21.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.321469+0000) 2019-10-13T08:55:21.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:21.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.732290+0000) 2019-10-13T08:55:21.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:21.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.778136+0000) 2019-10-13T08:55:21.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:21.967+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.971687+0000) 2019-10-13T08:55:24.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:24.991+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.998770+0000) 2019-10-13T08:55:25.103 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:25.095+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.102660+0000) 2019-10-13T08:55:25.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:25.095+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.103040+0000) 2019-10-13T08:55:25.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:25.099+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.103912+0000) 2019-10-13T08:55:25.104 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:25.099+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.104306+0000) 2019-10-13T08:55:25.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:25.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:25.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:25.099+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.104871+0000) 2019-10-13T08:55:25.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:25.099+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.105251+0000) 2019-10-13T08:55:25.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:25.099+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.105734+0000) 2019-10-13T08:55:25.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:25.103+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:25.110063+0000) 2019-10-13T08:55:26.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:26.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:26.220892+0000) 2019-10-13T08:55:28.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:28.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:28.488929+0000) 2019-10-13T08:55:29.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:29.471+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.473829+0000) 2019-10-13T08:55:30.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:30.103+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.107710+0000) 2019-10-13T08:55:30.109 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:30.103+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.108165+0000) 2019-10-13T08:55:30.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:30.103+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.108762+0000) 2019-10-13T08:55:30.109 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:30.103+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.109174+0000) 2019-10-13T08:55:30.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:30.103+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.109515+0000) 2019-10-13T08:55:30.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:30.103+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.109805+0000) 2019-10-13T08:55:30.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:30.103+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.110108+0000) 2019-10-13T08:55:30.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:30.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:30.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:30.115+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:30.119975+0000) 2019-10-13T08:55:31.328 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:31.327+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.327914+0000) 2019-10-13T08:55:31.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:31.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.732494+0000) 2019-10-13T08:55:31.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:31.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.778351+0000) 2019-10-13T08:55:31.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:31.971+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.971843+0000) 2019-10-13T08:55:34.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:34.995+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.999004+0000) 2019-10-13T08:55:35.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:35.107+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.113582+0000) 2019-10-13T08:55:35.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:35.107+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.114020+0000) 2019-10-13T08:55:35.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:35.107+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.114703+0000) 2019-10-13T08:55:35.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:35.111+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.115491+0000) 2019-10-13T08:55:35.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:35.111+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.115668+0000) 2019-10-13T08:55:35.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:35.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:35.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:35.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.116490+0000) 2019-10-13T08:55:35.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:35.111+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.116743+0000) 2019-10-13T08:55:35.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:35.119+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:35.125288+0000) 2019-10-13T08:55:35.246 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:55:35.247 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:35.250 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:55:35.250 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:36.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:36.215+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:36.221114+0000) 2019-10-13T08:55:38.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:38.483+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:38.489142+0000) 2019-10-13T08:55:39.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:39.467+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.474019+0000) 2019-10-13T08:55:40.120 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:40.115+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.119438+0000) 2019-10-13T08:55:40.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:40.115+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.119726+0000) 2019-10-13T08:55:40.121 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:40.115+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.120319+0000) 2019-10-13T08:55:40.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:40.115+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.120547+0000) 2019-10-13T08:55:40.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:40.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.120771+0000) 2019-10-13T08:55:40.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:40.115+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.121232+0000) 2019-10-13T08:55:40.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:40.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:40.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:40.115+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.122256+0000) 2019-10-13T08:55:40.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:40.123+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:40.128055+0000) 2019-10-13T08:55:41.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:41.331+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.337478+0000) 2019-10-13T08:55:41.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:41.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.732699+0000) 2019-10-13T08:55:41.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:41.771+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.778571+0000) 2019-10-13T08:55:41.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:41.971+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.972037+0000) 2019-10-13T08:55:44.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:44.995+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.999209+0000) 2019-10-13T08:55:45.125 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:45.119+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.124483+0000) 2019-10-13T08:55:45.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:45.119+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.125021+0000) 2019-10-13T08:55:45.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:45.119+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.126157+0000) 2019-10-13T08:55:45.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:45.119+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.126229+0000) 2019-10-13T08:55:45.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:45.123+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.126491+0000) 2019-10-13T08:55:45.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:45.119+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.127132+0000) 2019-10-13T08:55:45.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:45.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:45.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:45.123+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.128212+0000) 2019-10-13T08:55:45.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:45.127+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:45.133887+0000) 2019-10-13T08:55:46.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:46.219+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:46.221350+0000) 2019-10-13T08:55:48.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:48.487+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:48.489373+0000) 2019-10-13T08:55:49.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:49.471+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.474244+0000) 2019-10-13T08:55:50.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:50.127+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.130111+0000) 2019-10-13T08:55:50.132 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:50.131+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.131381+0000) 2019-10-13T08:55:50.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:50.131+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.131842+0000) 2019-10-13T08:55:50.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:50.131+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.132466+0000) 2019-10-13T08:55:50.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:50.131+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.132959+0000) 2019-10-13T08:55:50.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:50.131+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:50.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:50.127+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.133631+0000) 2019-10-13T08:55:50.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:50.131+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.133889+0000) 2019-10-13T08:55:50.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:50.135+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:50.139121+0000) 2019-10-13T08:55:51.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:51.339+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.343727+0000) 2019-10-13T08:55:51.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:51.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.732919+0000) 2019-10-13T08:55:51.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:51.775+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.778777+0000) 2019-10-13T08:55:51.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:51.971+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.972251+0000) 2019-10-13T08:55:55.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:54.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.999406+0000) 2019-10-13T08:55:55.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:55.131+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.136607+0000) 2019-10-13T08:55:55.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:55:55.135+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.137056+0000) 2019-10-13T08:55:55.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:55:55.135+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.137619+0000) 2019-10-13T08:55:55.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:55:55.135+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.138232+0000) 2019-10-13T08:55:55.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:55.135+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.137999+0000) 2019-10-13T08:55:55.140 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:55:55.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:55:55.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:55:55.139+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.139335+0000) 2019-10-13T08:55:55.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:55.139+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.139858+0000) 2019-10-13T08:55:55.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:55:55.143+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:55.146956+0000) 2019-10-13T08:55:56.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:55:56.219+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:56.221542+0000) 2019-10-13T08:55:58.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:55:58.487+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:58.489620+0000) 2019-10-13T08:55:59.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:55:59.471+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.474457+0000) 2019-10-13T08:56:00.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:00.139+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.142623+0000) 2019-10-13T08:56:00.144 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:00.142+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.143707+0000) 2019-10-13T08:56:00.145 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:00.142+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.144091+0000) 2019-10-13T08:56:00.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:00.142+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.143429+0000) 2019-10-13T08:56:00.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:00.142+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.144927+0000) 2019-10-13T08:56:00.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:00.142+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.145145+0000) 2019-10-13T08:56:00.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:00.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:00.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:00.142+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.145857+0000) 2019-10-13T08:56:00.153 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:00.147+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:00.152862+0000) 2019-10-13T08:56:01.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:01.347+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.350172+0000) 2019-10-13T08:56:01.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:01.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.733151+0000) 2019-10-13T08:56:01.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:01.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.779002+0000) 2019-10-13T08:56:01.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:01.970+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.972450+0000) 2019-10-13T08:56:05.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:04.998+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.999604+0000) 2019-10-13T08:56:05.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:05.146+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.149447+0000) 2019-10-13T08:56:05.151 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:05.146+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.149775+0000) 2019-10-13T08:56:05.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:05.146+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.150406+0000) 2019-10-13T08:56:05.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:05.146+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.151125+0000) 2019-10-13T08:56:05.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:05.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:05.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:05.146+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.152189+0000) 2019-10-13T08:56:05.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:05.150+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.152471+0000) 2019-10-13T08:56:05.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:05.150+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.153125+0000) 2019-10-13T08:56:05.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:05.154+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:05.160147+0000) 2019-10-13T08:56:05.355 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:56:05.356 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:05.359 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:56:05.360 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:06.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:06.218+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:06.221738+0000) 2019-10-13T08:56:08.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:08.486+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:08.489821+0000) 2019-10-13T08:56:09.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:09.470+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.474682+0000) 2019-10-13T08:56:10.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:10.150+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.154070+0000) 2019-10-13T08:56:10.155 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:10.150+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.155193+0000) 2019-10-13T08:56:10.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:10.150+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.155512+0000) 2019-10-13T08:56:10.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:10.154+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.156187+0000) 2019-10-13T08:56:10.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:10.154+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.156417+0000) 2019-10-13T08:56:10.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:10.154+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.157067+0000) 2019-10-13T08:56:10.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:10.154+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:10.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:10.154+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.158058+0000) 2019-10-13T08:56:10.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:10.158+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:10.162851+0000) 2019-10-13T08:56:11.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:11.350+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.356644+0000) 2019-10-13T08:56:11.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:11.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.733394+0000) 2019-10-13T08:56:11.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:11.774+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.779225+0000) 2019-10-13T08:56:11.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:11.970+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.972670+0000) 2019-10-13T08:56:15.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:14.998+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.999832+0000) 2019-10-13T08:56:15.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:15.158+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.160747+0000) 2019-10-13T08:56:15.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:15.158+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.161084+0000) 2019-10-13T08:56:15.162 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:15.158+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.162390+0000) 2019-10-13T08:56:15.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:15.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:15.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:15.158+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.162701+0000) 2019-10-13T08:56:15.164 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:15.158+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.163730+0000) 2019-10-13T08:56:15.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:15.162+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.163539+0000) 2019-10-13T08:56:15.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:15.162+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.164230+0000) 2019-10-13T08:56:15.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:15.162+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:15.167973+0000) 2019-10-13T08:56:16.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:16.218+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:16.221947+0000) 2019-10-13T08:56:18.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:18.486+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:18.490050+0000) 2019-10-13T08:56:19.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:19.474+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.474874+0000) 2019-10-13T08:56:20.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:20.162+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.166405+0000) 2019-10-13T08:56:20.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:20.162+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.166870+0000) 2019-10-13T08:56:20.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:20.166+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.168486+0000) 2019-10-13T08:56:20.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:20.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.168591+0000) 2019-10-13T08:56:20.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:20.166+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.168948+0000) 2019-10-13T08:56:20.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:20.166+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.169080+0000) 2019-10-13T08:56:20.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:20.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:20.170 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:20.162+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.169315+0000) 2019-10-13T08:56:20.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:20.170+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:20.176163+0000) 2019-10-13T08:56:21.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:21.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.362409+0000) 2019-10-13T08:56:21.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:21.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.733601+0000) 2019-10-13T08:56:21.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:21.778+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.779422+0000) 2019-10-13T08:56:21.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:21.970+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.972871+0000) 2019-10-13T08:56:25.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:24.998+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.000052+0000) 2019-10-13T08:56:25.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:25.170+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.172247+0000) 2019-10-13T08:56:25.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:25.170+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.173603+0000) 2019-10-13T08:56:25.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:25.170+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.174060+0000) 2019-10-13T08:56:25.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:25.170+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.174233+0000) 2019-10-13T08:56:25.175 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:25.170+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.174930+0000) 2019-10-13T08:56:25.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:25.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:25.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:25.170+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.175020+0000) 2019-10-13T08:56:25.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:25.174+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.175908+0000) 2019-10-13T08:56:25.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:25.174+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:25.179867+0000) 2019-10-13T08:56:26.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:26.218+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:26.222146+0000) 2019-10-13T08:56:28.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:28.486+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:28.490245+0000) 2019-10-13T08:56:29.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:29.474+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.475076+0000) 2019-10-13T08:56:30.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:30.174+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.178116+0000) 2019-10-13T08:56:30.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:30.178+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.179336+0000) 2019-10-13T08:56:30.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:30.178+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.179539+0000) 2019-10-13T08:56:30.180 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:30.178+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.179835+0000) 2019-10-13T08:56:30.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:30.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:30.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:30.178+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.180634+0000) 2019-10-13T08:56:30.182 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:30.178+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.181009+0000) 2019-10-13T08:56:30.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:30.174+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.181449+0000) 2019-10-13T08:56:30.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:30.182+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:30.186496+0000) 2019-10-13T08:56:31.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:31.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.372156+0000) 2019-10-13T08:56:31.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:31.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.733808+0000) 2019-10-13T08:56:31.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:31.778+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.779624+0000) 2019-10-13T08:56:31.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:31.970+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.973071+0000) 2019-10-13T08:56:35.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:34.998+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.000242+0000) 2019-10-13T08:56:35.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:35.182+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.184242+0000) 2019-10-13T08:56:35.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:35.182+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.184704+0000) 2019-10-13T08:56:35.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:35.182+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.185420+0000) 2019-10-13T08:56:35.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:35.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.185768+0000) 2019-10-13T08:56:35.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:35.182+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.186269+0000) 2019-10-13T08:56:35.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:35.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:35.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:35.182+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.187007+0000) 2019-10-13T08:56:35.188 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:35.182+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.186711+0000) 2019-10-13T08:56:35.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:35.186+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:35.193791+0000) 2019-10-13T08:56:35.417 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:56:35.417 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:35.420 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:56:35.421 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:36.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:36.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:36.222343+0000) 2019-10-13T08:56:38.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:38.486+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:38.490441+0000) 2019-10-13T08:56:39.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:39.474+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.475306+0000) 2019-10-13T08:56:40.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:40.186+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.189794+0000) 2019-10-13T08:56:40.191 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:40.186+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.190270+0000) 2019-10-13T08:56:40.191 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:40.186+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.190809+0000) 2019-10-13T08:56:40.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:40.186+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.190862+0000) 2019-10-13T08:56:40.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:40.186+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.191250+0000) 2019-10-13T08:56:40.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:40.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:40.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:40.190+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.192209+0000) 2019-10-13T08:56:40.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:40.190+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.192582+0000) 2019-10-13T08:56:40.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:40.198+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:40.203466+0000) 2019-10-13T08:56:41.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:41.378+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.378499+0000) 2019-10-13T08:56:41.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:41.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.734009+0000) 2019-10-13T08:56:41.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:41.778+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.779825+0000) 2019-10-13T08:56:41.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:41.970+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.973277+0000) 2019-10-13T08:56:45.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:44.997+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.000420+0000) 2019-10-13T08:56:45.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:45.190+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.194587+0000) 2019-10-13T08:56:45.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:45.193+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.195656+0000) 2019-10-13T08:56:45.196 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:45.193+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.196064+0000) 2019-10-13T08:56:45.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:45.193+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.196787+0000) 2019-10-13T08:56:45.197 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:45.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:45.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:45.193+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.198044+0000) 2019-10-13T08:56:45.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:45.193+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.198501+0000) 2019-10-13T08:56:45.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:45.193+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.198838+0000) 2019-10-13T08:56:45.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:45.198+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:45.204932+0000) 2019-10-13T08:56:46.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:46.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:46.222563+0000) 2019-10-13T08:56:48.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:48.485+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:48.490642+0000) 2019-10-13T08:56:49.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:49.474+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.475547+0000) 2019-10-13T08:56:50.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:50.198+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.200075+0000) 2019-10-13T08:56:50.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:50.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.201097+0000) 2019-10-13T08:56:50.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:50.197+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.201539+0000) 2019-10-13T08:56:50.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:50.197+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.202660+0000) 2019-10-13T08:56:50.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:50.197+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.203009+0000) 2019-10-13T08:56:50.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:50.201+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.203350+0000) 2019-10-13T08:56:50.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:50.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63797 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:50.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:50.201+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.203720+0000) 2019-10-13T08:56:50.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:50.206+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:50.208863+0000) 2019-10-13T08:56:51.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:51.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.385144+0000) 2019-10-13T08:56:51.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:51.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.734237+0000) 2019-10-13T08:56:51.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:51.777+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.780045+0000) 2019-10-13T08:56:51.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:51.969+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.973471+0000) 2019-10-13T08:56:55.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:54.997+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.000611+0000) 2019-10-13T08:56:55.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:56:55.201+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.206563+0000) 2019-10-13T08:56:55.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:56:55.205+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.207609+0000) 2019-10-13T08:56:55.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:56:55.205+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.208283+0000) 2019-10-13T08:56:55.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:55.206+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.209051+0000) 2019-10-13T08:56:55.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:56:55.205+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.209508+0000) 2019-10-13T08:56:55.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:56:55.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:56:55.210 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:55.205+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.209815+0000) 2019-10-13T08:56:55.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:55.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.210589+0000) 2019-10-13T08:56:55.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:56:55.214+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:55.216188+0000) 2019-10-13T08:56:56.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:56:56.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:56.222754+0000) 2019-10-13T08:56:58.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:56:58.489+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:58.490843+0000) 2019-10-13T08:56:59.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:56:59.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.475697+0000) 2019-10-13T08:57:00.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:00.209+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.212736+0000) 2019-10-13T08:57:00.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:00.209+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.213717+0000) 2019-10-13T08:57:00.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:00.209+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.214365+0000) 2019-10-13T08:57:00.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:00.213+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.214789+0000) 2019-10-13T08:57:00.215 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:00.209+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.215112+0000) 2019-10-13T08:57:00.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:00.213+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.215740+0000) 2019-10-13T08:57:00.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:00.213+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.215620+0000) 2019-10-13T08:57:00.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:00.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:00.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:00.217+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:00.221225+0000) 2019-10-13T08:57:01.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:01.389+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.390858+0000) 2019-10-13T08:57:01.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:01.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.734434+0000) 2019-10-13T08:57:01.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:01.777+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.780285+0000) 2019-10-13T08:57:01.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:01.969+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.973671+0000) 2019-10-13T08:57:05.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:04.997+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.000808+0000) 2019-10-13T08:57:05.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:05.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.218846+0000) 2019-10-13T08:57:05.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:05.217+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.219715+0000) 2019-10-13T08:57:05.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:05.217+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.220428+0000) 2019-10-13T08:57:05.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:05.217+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.221010+0000) 2019-10-13T08:57:05.223 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:05.217+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.221772+0000) 2019-10-13T08:57:05.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:05.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:05.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:05.217+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.221609+0000) 2019-10-13T08:57:05.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:05.217+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.222669+0000) 2019-10-13T08:57:05.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:05.221+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:05.225708+0000) 2019-10-13T08:57:05.549 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:57:05.549 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:05.553 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:57:05.553 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:06.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:06.217+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:06.222992+0000) 2019-10-13T08:57:08.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:08.489+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:08.491039+0000) 2019-10-13T08:57:09.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:09.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.475914+0000) 2019-10-13T08:57:10.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:10.221+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.223507+0000) 2019-10-13T08:57:10.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:10.221+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.224032+0000) 2019-10-13T08:57:10.226 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:10.221+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.225200+0000) 2019-10-13T08:57:10.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:10.221+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.226047+0000) 2019-10-13T08:57:10.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:10.221+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.226423+0000) 2019-10-13T08:57:10.227 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:10.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.227356+0000) 2019-10-13T08:57:10.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:10.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:10.229 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:10.225+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.228133+0000) 2019-10-13T08:57:10.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:10.229+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:10.233087+0000) 2019-10-13T08:57:11.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:11.393+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.397199+0000) 2019-10-13T08:57:11.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:11.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.734669+0000) 2019-10-13T08:57:11.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:11.777+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.780488+0000) 2019-10-13T08:57:11.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:11.973+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.973878+0000) 2019-10-13T08:57:15.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:14.997+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.001035+0000) 2019-10-13T08:57:15.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:15.225+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.231030+0000) 2019-10-13T08:57:15.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:15.229+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.231612+0000) 2019-10-13T08:57:15.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:15.229+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.232481+0000) 2019-10-13T08:57:15.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:15.229+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.233029+0000) 2019-10-13T08:57:15.234 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:15.229+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.233845+0000) 2019-10-13T08:57:15.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:15.229+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.233538+0000) 2019-10-13T08:57:15.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:15.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 63969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:15.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:15.229+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.234474+0000) 2019-10-13T08:57:15.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:15.237+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:15.238147+0000) 2019-10-13T08:57:16.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:16.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:16.223228+0000) 2019-10-13T08:57:18.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:18.489+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:18.491255+0000) 2019-10-13T08:57:19.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:19.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.476104+0000) 2019-10-13T08:57:20.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:20.233+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.237308+0000) 2019-10-13T08:57:20.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:20.233+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.237634+0000) 2019-10-13T08:57:20.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:20.233+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.238065+0000) 2019-10-13T08:57:20.239 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:20.233+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.238456+0000) 2019-10-13T08:57:20.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:20.237+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.239752+0000) 2019-10-13T08:57:20.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:20.237+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.240081+0000) 2019-10-13T08:57:20.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:20.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:20.241 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:20.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.240739+0000) 2019-10-13T08:57:20.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:20.245+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:20.248575+0000) 2019-10-13T08:57:21.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:21.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.403608+0000) 2019-10-13T08:57:21.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:21.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.734869+0000) 2019-10-13T08:57:21.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:21.777+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.780688+0000) 2019-10-13T08:57:21.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:21.973+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.974103+0000) 2019-10-13T08:57:25.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:24.997+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.001265+0000) 2019-10-13T08:57:25.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:25.241+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.244159+0000) 2019-10-13T08:57:25.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:25.241+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.244443+0000) 2019-10-13T08:57:25.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:25.241+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.244709+0000) 2019-10-13T08:57:25.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:25.241+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.244965+0000) 2019-10-13T08:57:25.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:25.241+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.245678+0000) 2019-10-13T08:57:25.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:25.241+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.246069+0000) 2019-10-13T08:57:25.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:25.241+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:25.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:25.241+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.246449+0000) 2019-10-13T08:57:25.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:25.249+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:25.251802+0000) 2019-10-13T08:57:26.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:26.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:26.223407+0000) 2019-10-13T08:57:28.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:28.488+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:28.491488+0000) 2019-10-13T08:57:29.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:29.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.476300+0000) 2019-10-13T08:57:30.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:30.244+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.249440+0000) 2019-10-13T08:57:30.250 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:30.244+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.249860+0000) 2019-10-13T08:57:30.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:30.245+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.250101+0000) 2019-10-13T08:57:30.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:30.248+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.251512+0000) 2019-10-13T08:57:30.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:30.248+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.252276+0000) 2019-10-13T08:57:30.253 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:30.248+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.252460+0000) 2019-10-13T08:57:30.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:30.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:30.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:30.248+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.252890+0000) 2019-10-13T08:57:30.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:30.253+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:30.257971+0000) 2019-10-13T08:57:31.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:31.405+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.410031+0000) 2019-10-13T08:57:31.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:31.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.735065+0000) 2019-10-13T08:57:31.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:31.776+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.780898+0000) 2019-10-13T08:57:31.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:31.968+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.974328+0000) 2019-10-13T08:57:35.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:34.996+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.001475+0000) 2019-10-13T08:57:35.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:35.252+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.255885+0000) 2019-10-13T08:57:35.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:35.252+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.256721+0000) 2019-10-13T08:57:35.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:35.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.257379+0000) 2019-10-13T08:57:35.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:35.252+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.258037+0000) 2019-10-13T08:57:35.259 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:35.257+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.258233+0000) 2019-10-13T08:57:35.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:35.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:35.259 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:35.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.258893+0000) 2019-10-13T08:57:35.260 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:35.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.259229+0000) 2019-10-13T08:57:35.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:35.261+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:35.265193+0000) 2019-10-13T08:57:35.673 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:57:35.673 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:35.677 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:57:35.677 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:36.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:36.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:36.223582+0000) 2019-10-13T08:57:38.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:38.488+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:38.491687+0000) 2019-10-13T08:57:39.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:39.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.476497+0000) 2019-10-13T08:57:40.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:40.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.262366+0000) 2019-10-13T08:57:40.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:40.256+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.262485+0000) 2019-10-13T08:57:40.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:40.256+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.262932+0000) 2019-10-13T08:57:40.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:40.260+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.264139+0000) 2019-10-13T08:57:40.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:40.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:40.265 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:40.261+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.265027+0000) 2019-10-13T08:57:40.266 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:40.260+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.265701+0000) 2019-10-13T08:57:40.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:40.264+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.267675+0000) 2019-10-13T08:57:40.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:40.269+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:40.270956+0000) 2019-10-13T08:57:41.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:41.413+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.416426+0000) 2019-10-13T08:57:41.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:41.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.735268+0000) 2019-10-13T08:57:41.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:41.776+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.781094+0000) 2019-10-13T08:57:41.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:41.972+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.974529+0000) 2019-10-13T08:57:45.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:44.996+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.001668+0000) 2019-10-13T08:57:45.268 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:45.264+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.267595+0000) 2019-10-13T08:57:45.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:45.264+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.267866+0000) 2019-10-13T08:57:45.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:45.264+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.269134+0000) 2019-10-13T08:57:45.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:45.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.269241+0000) 2019-10-13T08:57:45.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:45.264+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.269781+0000) 2019-10-13T08:57:45.271 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:45.264+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.270101+0000) 2019-10-13T08:57:45.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:45.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:45.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:45.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.271937+0000) 2019-10-13T08:57:45.278 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:45.273+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:45.277525+0000) 2019-10-13T08:57:46.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:46.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:46.223713+0000) 2019-10-13T08:57:48.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:48.488+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:48.491925+0000) 2019-10-13T08:57:49.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:49.473+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.476651+0000) 2019-10-13T08:57:50.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:50.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.273559+0000) 2019-10-13T08:57:50.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:50.268+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.274672+0000) 2019-10-13T08:57:50.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:50.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.275084+0000) 2019-10-13T08:57:50.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:50.273+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.276489+0000) 2019-10-13T08:57:50.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:50.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:50.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:50.272+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.275630+0000) 2019-10-13T08:57:50.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:50.272+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.276128+0000) 2019-10-13T08:57:50.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:50.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.277165+0000) 2019-10-13T08:57:50.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:50.277+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:50.280471+0000) 2019-10-13T08:57:51.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:51.424+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.428202+0000) 2019-10-13T08:57:51.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:51.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.735530+0000) 2019-10-13T08:57:51.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:51.776+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.781327+0000) 2019-10-13T08:57:51.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:51.972+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.974725+0000) 2019-10-13T08:57:55.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:54.996+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.001862+0000) 2019-10-13T08:57:55.281 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:57:55.276+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.280129+0000) 2019-10-13T08:57:55.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:57:55.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.280924+0000) 2019-10-13T08:57:55.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:57:55.276+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.281108+0000) 2019-10-13T08:57:55.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:57:55.276+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.281504+0000) 2019-10-13T08:57:55.283 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:55.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.281822+0000) 2019-10-13T08:57:55.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:55.276+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.282330+0000) 2019-10-13T08:57:55.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:57:55.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:57:55.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:55.280+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.282956+0000) 2019-10-13T08:57:55.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:57:55.284+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:55.289935+0000) 2019-10-13T08:57:56.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:57:56.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:56.223902+0000) 2019-10-13T08:57:58.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:57:58.488+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:58.492128+0000) 2019-10-13T08:57:59.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:57:59.472+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.476875+0000) 2019-10-13T08:58:00.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:00.280+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.285921+0000) 2019-10-13T08:58:00.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:00.280+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.286558+0000) 2019-10-13T08:58:00.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:00.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.286950+0000) 2019-10-13T08:58:00.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:00.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:00.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:00.284+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.288055+0000) 2019-10-13T08:58:00.289 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:00.284+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.288400+0000) 2019-10-13T08:58:00.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:00.284+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.288734+0000) 2019-10-13T08:58:00.292 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:00.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.291393+0000) 2019-10-13T08:58:00.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:00.288+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:00.293870+0000) 2019-10-13T08:58:01.435 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:01.432+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.434803+0000) 2019-10-13T08:58:01.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:01.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.735732+0000) 2019-10-13T08:58:01.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:01.776+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.781554+0000) 2019-10-13T08:58:01.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:01.972+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.974926+0000) 2019-10-13T08:58:05.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:04.996+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.002075+0000) 2019-10-13T08:58:05.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:05.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.291404+0000) 2019-10-13T08:58:05.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:05.288+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.292217+0000) 2019-10-13T08:58:05.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:05.288+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.292720+0000) 2019-10-13T08:58:05.293 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:05.288+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.293235+0000) 2019-10-13T08:58:05.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:05.288+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.293878+0000) 2019-10-13T08:58:05.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:05.292+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.294405+0000) 2019-10-13T08:58:05.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:05.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:05.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:05.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.294991+0000) 2019-10-13T08:58:05.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:05.296+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:05.301719+0000) 2019-10-13T08:58:05.778 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:58:05.778 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:05.782 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:58:05.782 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:06.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:06.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:06.224079+0000) 2019-10-13T08:58:08.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:08.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:08.492329+0000) 2019-10-13T08:58:09.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:09.472+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.477067+0000) 2019-10-13T08:58:10.298 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:10.292+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.297684+0000) 2019-10-13T08:58:10.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:10.292+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.297980+0000) 2019-10-13T08:58:10.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:10.292+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.298623+0000) 2019-10-13T08:58:10.299 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:10.292+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.298880+0000) 2019-10-13T08:58:10.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:10.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.299345+0000) 2019-10-13T08:58:10.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:10.296+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.299847+0000) 2019-10-13T08:58:10.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:10.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:10.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:10.296+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.301366+0000) 2019-10-13T08:58:10.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:10.300+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:10.304431+0000) 2019-10-13T08:58:11.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:11.436+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.441367+0000) 2019-10-13T08:58:11.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:11.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.735964+0000) 2019-10-13T08:58:11.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:11.775+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.781751+0000) 2019-10-13T08:58:11.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:11.971+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.975122+0000) 2019-10-13T08:58:15.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:14.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.002304+0000) 2019-10-13T08:58:15.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:15.299+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.304630+0000) 2019-10-13T08:58:15.305 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:15.299+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.305122+0000) 2019-10-13T08:58:15.306 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:15.299+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.305456+0000) 2019-10-13T08:58:15.306 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:15.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.306029+0000) 2019-10-13T08:58:15.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:15.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64383 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:15.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:15.304+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.306734+0000) 2019-10-13T08:58:15.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:15.299+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.307165+0000) 2019-10-13T08:58:15.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:15.303+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.309192+0000) 2019-10-13T08:58:15.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:15.308+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:15.311483+0000) 2019-10-13T08:58:16.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:16.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:16.224263+0000) 2019-10-13T08:58:18.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:18.491+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:18.492525+0000) 2019-10-13T08:58:19.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:19.476+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.477259+0000) 2019-10-13T08:58:20.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:20.304+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.309525+0000) 2019-10-13T08:58:20.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:20.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.309823+0000) 2019-10-13T08:58:20.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:20.303+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.310173+0000) 2019-10-13T08:58:20.313 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:20.303+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.311219+0000) 2019-10-13T08:58:20.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:20.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.311643+0000) 2019-10-13T08:58:20.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:20.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.312338+0000) 2019-10-13T08:58:20.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:20.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:20.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:20.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.313183+0000) 2019-10-13T08:58:20.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:20.312+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:20.316868+0000) 2019-10-13T08:58:21.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:21.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.447810+0000) 2019-10-13T08:58:21.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:21.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.736161+0000) 2019-10-13T08:58:21.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:21.775+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.781957+0000) 2019-10-13T08:58:21.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:21.971+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.975354+0000) 2019-10-13T08:58:25.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:24.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.002513+0000) 2019-10-13T08:58:25.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:25.312+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.316473+0000) 2019-10-13T08:58:25.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:25.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.316680+0000) 2019-10-13T08:58:25.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:25.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.317114+0000) 2019-10-13T08:58:25.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:25.311+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.317567+0000) 2019-10-13T08:58:25.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:25.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.318038+0000) 2019-10-13T08:58:25.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:25.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.318798+0000) 2019-10-13T08:58:25.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:25.311+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.319131+0000) 2019-10-13T08:58:25.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:25.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:25.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:25.316+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:25.321939+0000) 2019-10-13T08:58:26.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:26.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:26.224472+0000) 2019-10-13T08:58:28.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:28.487+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:28.492776+0000) 2019-10-13T08:58:29.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:29.472+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.477455+0000) 2019-10-13T08:58:30.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:30.315+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.321151+0000) 2019-10-13T08:58:30.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:30.315+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.321939+0000) 2019-10-13T08:58:30.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:30.315+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.322977+0000) 2019-10-13T08:58:30.324 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:30.320+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.323358+0000) 2019-10-13T08:58:30.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:30.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:30.325 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:30.319+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.323752+0000) 2019-10-13T08:58:30.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:30.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.323987+0000) 2019-10-13T08:58:30.328 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:30.319+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.325017+0000) 2019-10-13T08:58:30.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:30.324+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:30.329422+0000) 2019-10-13T08:58:31.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:31.452+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.454196+0000) 2019-10-13T08:58:31.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:31.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.736370+0000) 2019-10-13T08:58:31.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:31.775+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.782244+0000) 2019-10-13T08:58:31.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:31.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.975555+0000) 2019-10-13T08:58:35.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:34.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.002723+0000) 2019-10-13T08:58:35.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:35.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.327047+0000) 2019-10-13T08:58:35.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:35.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.327395+0000) 2019-10-13T08:58:35.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:35.324+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.327991+0000) 2019-10-13T08:58:35.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:35.323+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.329160+0000) 2019-10-13T08:58:35.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:35.323+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.329390+0000) 2019-10-13T08:58:35.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:35.323+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.329829+0000) 2019-10-13T08:58:35.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:35.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:35.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:35.323+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.330413+0000) 2019-10-13T08:58:35.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:35.332+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:35.335237+0000) 2019-10-13T08:58:35.910 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:58:35.910 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:35.914 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:58:35.914 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:36.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:36.219+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:36.224683+0000) 2019-10-13T08:58:38.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:38.491+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:38.492933+0000) 2019-10-13T08:58:39.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:39.476+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.477643+0000) 2019-10-13T08:58:40.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:40.328+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.332875+0000) 2019-10-13T08:58:40.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:40.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.333604+0000) 2019-10-13T08:58:40.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:40.327+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.334155+0000) 2019-10-13T08:58:40.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:40.327+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.334416+0000) 2019-10-13T08:58:40.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:40.327+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.334705+0000) 2019-10-13T08:58:40.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:40.331+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.335674+0000) 2019-10-13T08:58:40.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:40.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:40.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:40.331+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.336151+0000) 2019-10-13T08:58:40.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:40.336+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:40.341843+0000) 2019-10-13T08:58:41.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:41.460+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.460521+0000) 2019-10-13T08:58:41.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:41.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.736580+0000) 2019-10-13T08:58:41.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:41.779+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.782493+0000) 2019-10-13T08:58:41.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:41.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.975789+0000) 2019-10-13T08:58:45.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:44.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.002922+0000) 2019-10-13T08:58:45.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:45.335+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.339115+0000) 2019-10-13T08:58:45.341 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:45.339+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.340410+0000) 2019-10-13T08:58:45.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:45.336+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.340649+0000) 2019-10-13T08:58:45.342 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:45.339+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.341146+0000) 2019-10-13T08:58:45.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:45.339+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.341928+0000) 2019-10-13T08:58:45.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:45.339+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.342011+0000) 2019-10-13T08:58:45.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:45.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64591 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:45.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:45.339+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.342407+0000) 2019-10-13T08:58:45.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:45.344+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:45.348222+0000) 2019-10-13T08:58:46.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:46.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:46.224860+0000) 2019-10-13T08:58:48.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:48.491+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:48.493127+0000) 2019-10-13T08:58:49.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:49.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.477831+0000) 2019-10-13T08:58:50.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:50.339+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.345404+0000) 2019-10-13T08:58:50.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:50.343+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.346209+0000) 2019-10-13T08:58:50.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:50.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.346872+0000) 2019-10-13T08:58:50.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:50.343+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.347244+0000) 2019-10-13T08:58:50.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:50.347+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.348058+0000) 2019-10-13T08:58:50.350 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:50.347+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.348404+0000) 2019-10-13T08:58:50.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:50.347+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:50.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:50.347+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.349056+0000) 2019-10-13T08:58:50.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:50.351+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:50.354506+0000) 2019-10-13T08:58:51.468 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:51.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.467103+0000) 2019-10-13T08:58:51.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:51.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.736812+0000) 2019-10-13T08:58:51.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:51.779+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.782741+0000) 2019-10-13T08:58:51.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:51.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.976045+0000) 2019-10-13T08:58:55.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:54.998+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.003153+0000) 2019-10-13T08:58:55.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:55.350+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.352435+0000) 2019-10-13T08:58:55.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:58:55.350+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.352314+0000) 2019-10-13T08:58:55.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:58:55.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.353928+0000) 2019-10-13T08:58:55.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:55.350+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.353602+0000) 2019-10-13T08:58:55.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:55.351+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.354223+0000) 2019-10-13T08:58:55.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:58:55.350+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.354530+0000) 2019-10-13T08:58:55.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:58:55.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:58:55.356 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:58:55.350+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.355134+0000) 2019-10-13T08:58:55.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:58:55.359+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:55.363158+0000) 2019-10-13T08:58:56.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:58:56.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:56.225034+0000) 2019-10-13T08:58:58.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:58:58.490+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:58.493336+0000) 2019-10-13T08:58:59.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:58:59.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.478021+0000) 2019-10-13T08:59:00.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:00.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.358305+0000) 2019-10-13T08:59:00.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:00.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.358688+0000) 2019-10-13T08:59:00.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:00.355+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.359118+0000) 2019-10-13T08:59:00.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:00.358+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.359799+0000) 2019-10-13T08:59:00.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:00.358+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.359858+0000) 2019-10-13T08:59:00.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:00.358+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.360173+0000) 2019-10-13T08:59:00.361 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:00.358+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.360290+0000) 2019-10-13T08:59:00.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:00.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:00.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:00.363+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:00.368017+0000) 2019-10-13T08:59:01.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:01.471+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.473589+0000) 2019-10-13T08:59:01.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:01.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.737064+0000) 2019-10-13T08:59:01.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:01.778+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.782974+0000) 2019-10-13T08:59:01.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:01.974+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.976256+0000) 2019-10-13T08:59:05.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:05.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.003386+0000) 2019-10-13T08:59:05.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:05.362+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.364436+0000) 2019-10-13T08:59:05.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:05.362+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.365168+0000) 2019-10-13T08:59:05.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:05.362+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.365737+0000) 2019-10-13T08:59:05.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:05.362+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.366240+0000) 2019-10-13T08:59:05.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:05.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:05.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:05.362+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.367172+0000) 2019-10-13T08:59:05.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:05.363+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.368564+0000) 2019-10-13T08:59:05.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:05.366+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.369250+0000) 2019-10-13T08:59:05.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:05.371+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:05.375011+0000) 2019-10-13T08:59:05.958 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:59:05.958 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:05.962 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:59:05.962 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:06.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:06.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:06.225230+0000) 2019-10-13T08:59:08.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:08.490+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:08.493567+0000) 2019-10-13T08:59:09.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:09.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.478213+0000) 2019-10-13T08:59:10.370 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:10.366+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.369682+0000) 2019-10-13T08:59:10.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:10.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.370542+0000) 2019-10-13T08:59:10.372 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:10.367+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.371635+0000) 2019-10-13T08:59:10.372 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:10.370+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.372194+0000) 2019-10-13T08:59:10.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:10.370+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.372739+0000) 2019-10-13T08:59:10.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:10.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:10.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:10.370+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.373331+0000) 2019-10-13T08:59:10.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:10.370+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.373044+0000) 2019-10-13T08:59:10.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:10.371+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:10.377660+0000) 2019-10-13T08:59:11.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:11.479+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.479803+0000) 2019-10-13T08:59:11.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:11.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.737269+0000) 2019-10-13T08:59:11.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:11.778+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.783188+0000) 2019-10-13T08:59:11.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:11.974+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.976489+0000) 2019-10-13T08:59:15.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:15.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.003631+0000) 2019-10-13T08:59:15.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:15.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.376275+0000) 2019-10-13T08:59:15.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:15.374+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.377217+0000) 2019-10-13T08:59:15.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:15.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.377699+0000) 2019-10-13T08:59:15.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:15.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.378298+0000) 2019-10-13T08:59:15.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:15.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.378719+0000) 2019-10-13T08:59:15.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:15.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64797 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:15.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:15.374+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.378994+0000) 2019-10-13T08:59:15.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:15.378+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.379721+0000) 2019-10-13T08:59:15.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:15.383+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:15.386668+0000) 2019-10-13T08:59:16.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:16.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:16.225407+0000) 2019-10-13T08:59:18.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:18.490+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:18.493786+0000) 2019-10-13T08:59:19.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:19.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.478415+0000) 2019-10-13T08:59:20.383 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:20.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.382707+0000) 2019-10-13T08:59:20.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:20.378+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.383142+0000) 2019-10-13T08:59:20.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:20.382+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.383476+0000) 2019-10-13T08:59:20.384 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:20.379+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.384195+0000) 2019-10-13T08:59:20.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:20.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.384432+0000) 2019-10-13T08:59:20.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:20.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:20.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:20.382+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.385200+0000) 2019-10-13T08:59:20.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:20.382+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.386848+0000) 2019-10-13T08:59:20.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:20.387+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:20.393157+0000) 2019-10-13T08:59:21.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:21.483+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.486246+0000) 2019-10-13T08:59:21.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:21.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.737471+0000) 2019-10-13T08:59:21.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:21.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.783358+0000) 2019-10-13T08:59:21.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:21.974+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.976733+0000) 2019-10-13T08:59:25.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:25.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.003836+0000) 2019-10-13T08:59:25.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:25.386+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.387432+0000) 2019-10-13T08:59:25.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:25.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.387863+0000) 2019-10-13T08:59:25.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:25.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.389444+0000) 2019-10-13T08:59:25.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:25.386+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.389504+0000) 2019-10-13T08:59:25.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:25.387+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.390315+0000) 2019-10-13T08:59:25.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:25.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.390984+0000) 2019-10-13T08:59:25.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:25.390+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.391536+0000) 2019-10-13T08:59:25.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:25.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:25.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:25.391+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:25.396807+0000) 2019-10-13T08:59:26.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:26.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:26.225590+0000) 2019-10-13T08:59:28.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:28.490+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:28.493948+0000) 2019-10-13T08:59:29.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:29.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.478640+0000) 2019-10-13T08:59:30.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:30.390+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.395217+0000) 2019-10-13T08:59:30.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:30.394+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.395523+0000) 2019-10-13T08:59:30.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:30.391+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.396318+0000) 2019-10-13T08:59:30.398 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:30.394+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.396827+0000) 2019-10-13T08:59:30.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:30.394+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.397129+0000) 2019-10-13T08:59:30.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:30.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:30.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:30.394+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.398150+0000) 2019-10-13T08:59:30.399 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:30.394+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.398665+0000) 2019-10-13T08:59:30.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:30.399+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:30.404016+0000) 2019-10-13T08:59:31.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:31.495+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.497910+0000) 2019-10-13T08:59:31.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:31.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.737705+0000) 2019-10-13T08:59:31.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:31.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.783590+0000) 2019-10-13T08:59:31.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:31.974+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.976924+0000) 2019-10-13T08:59:35.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:35.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.004054+0000) 2019-10-13T08:59:35.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:35.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.402316+0000) 2019-10-13T08:59:35.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:35.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.402407+0000) 2019-10-13T08:59:35.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:35.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.402828+0000) 2019-10-13T08:59:35.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:35.398+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.403035+0000) 2019-10-13T08:59:35.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:35.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64933 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:35.412 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:35.402+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.403981+0000) 2019-10-13T08:59:35.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:35.402+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.404385+0000) 2019-10-13T08:59:35.413 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:35.395+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.401396+0000) 2019-10-13T08:59:35.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:35.407+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:35.411013+0000) 2019-10-13T08:59:36.056 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T08:59:36.056 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:36.060 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T08:59:36.060 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:36.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:36.222+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:36.225775+0000) 2019-10-13T08:59:38.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:38.493+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:38.494146+0000) 2019-10-13T08:59:39.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:39.475+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.478837+0000) 2019-10-13T08:59:40.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:40.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.407145+0000) 2019-10-13T08:59:40.414 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:40.405+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.407853+0000) 2019-10-13T08:59:40.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:40.405+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.408234+0000) 2019-10-13T08:59:40.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:40.405+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.408977+0000) 2019-10-13T08:59:40.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:40.405+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.408920+0000) 2019-10-13T08:59:40.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:40.403+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.409829+0000) 2019-10-13T08:59:40.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:40.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 64965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:40.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:40.405+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.410407+0000) 2019-10-13T08:59:40.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:40.411+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:40.416993+0000) 2019-10-13T08:59:41.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:41.506+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.508604+0000) 2019-10-13T08:59:41.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:41.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.737907+0000) 2019-10-13T08:59:41.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:41.781+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.783799+0000) 2019-10-13T08:59:41.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:41.973+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.977117+0000) 2019-10-13T08:59:45.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:45.001+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.004259+0000) 2019-10-13T08:59:45.414 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:45.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.413181+0000) 2019-10-13T08:59:45.414 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:45.409+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.413316+0000) 2019-10-13T08:59:45.415 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:45.406+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.413600+0000) 2019-10-13T08:59:45.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:45.409+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.414187+0000) 2019-10-13T08:59:45.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:45.409+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.415180+0000) 2019-10-13T08:59:45.416 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:45.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.415522+0000) 2019-10-13T08:59:45.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:45.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:45.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:45.413+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.416244+0000) 2019-10-13T08:59:45.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:45.418+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:45.422419+0000) 2019-10-13T08:59:46.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:46.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:46.225959+0000) 2019-10-13T08:59:48.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:48.489+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:48.494409+0000) 2019-10-13T08:59:49.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:49.474+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.478993+0000) 2019-10-13T08:59:50.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:50.417+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.419398+0000) 2019-10-13T08:59:50.420 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:50.414+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.419639+0000) 2019-10-13T08:59:50.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:50.417+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.420688+0000) 2019-10-13T08:59:50.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:50.417+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.421245+0000) 2019-10-13T08:59:50.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:50.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.421555+0000) 2019-10-13T08:59:50.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:50.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:50.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:50.417+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.422723+0000) 2019-10-13T08:59:50.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:50.417+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.423208+0000) 2019-10-13T08:59:50.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:50.422+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.429388+0000) 2019-10-13T08:59:51.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:51.518+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.520334+0000) 2019-10-13T08:59:51.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:51.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.738106+0000) 2019-10-13T08:59:51.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:51.781+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.784016+0000) 2019-10-13T08:59:51.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:51.973+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.977345+0000) 2019-10-13T08:59:55.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:55.001+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.004459+0000) 2019-10-13T08:59:55.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:55.418+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.425912+0000) 2019-10-13T08:59:55.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T08:59:55.421+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.426273+0000) 2019-10-13T08:59:55.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T08:59:55.421+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.426452+0000) 2019-10-13T08:59:55.428 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T08:59:55.425+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.427597+0000) 2019-10-13T08:59:55.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T08:59:55.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T08:59:55.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:55.425+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.427966+0000) 2019-10-13T08:59:55.429 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T08:59:55.425+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.428386+0000) 2019-10-13T08:59:55.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:55.425+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.431125+0000) 2019-10-13T08:59:55.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T08:59:55.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.437867+0000) 2019-10-13T08:59:56.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T08:59:56.225+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:56.226138+0000) 2019-10-13T08:59:58.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T08:59:58.493+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:58.494634+0000) 2019-10-13T08:59:59.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T08:59:59.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.479220+0000) 2019-10-13T09:00:00.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:00.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.431510+0000) 2019-10-13T09:00:00.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:00.429+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.432104+0000) 2019-10-13T09:00:00.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:00.429+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.432488+0000) 2019-10-13T09:00:00.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:00.429+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.432863+0000) 2019-10-13T09:00:00.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:00.429+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.432249+0000) 2019-10-13T09:00:00.434 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:00.429+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.433330+0000) 2019-10-13T09:00:00.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:00.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:00.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:00.426+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.434118+0000) 2019-10-13T09:00:00.440 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:00.434+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.439627+0000) 2019-10-13T09:00:01.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:01.526+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.526700+0000) 2019-10-13T09:00:01.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:01.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.738325+0000) 2019-10-13T09:00:01.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:01.781+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.784221+0000) 2019-10-13T09:00:01.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:01.973+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.977492+0000) 2019-10-13T09:00:05.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:05.001+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.004655+0000) 2019-10-13T09:00:05.437 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:05.433+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.436772+0000) 2019-10-13T09:00:05.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:05.433+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.437532+0000) 2019-10-13T09:00:05.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:05.433+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.437713+0000) 2019-10-13T09:00:05.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:05.433+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.438163+0000) 2019-10-13T09:00:05.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:05.433+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.438532+0000) 2019-10-13T09:00:05.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:05.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:05.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:05.434+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.439379+0000) 2019-10-13T09:00:05.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:05.442+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.446691+0000) 2019-10-13T09:00:05.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:05.437+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.440353+0000) 2019-10-13T09:00:06.163 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:00:06.163 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:06.167 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:00:06.167 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:06.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:06.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:06.226322+0000) 2019-10-13T09:00:08.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:08.493+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:08.494834+0000) 2019-10-13T09:00:09.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:09.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.479419+0000) 2019-10-13T09:00:10.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:10.437+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.442499+0000) 2019-10-13T09:00:10.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:10.438+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.442793+0000) 2019-10-13T09:00:10.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:10.437+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.443266+0000) 2019-10-13T09:00:10.444 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:10.441+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.443707+0000) 2019-10-13T09:00:10.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:10.441+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.444436+0000) 2019-10-13T09:00:10.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:10.441+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.444913+0000) 2019-10-13T09:00:10.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:10.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:10.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:10.441+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.445486+0000) 2019-10-13T09:00:10.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:10.446+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.452515+0000) 2019-10-13T09:00:11.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:11.530+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.533183+0000) 2019-10-13T09:00:11.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:11.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.738584+0000) 2019-10-13T09:00:11.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:11.781+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.784429+0000) 2019-10-13T09:00:11.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:11.977+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.977703+0000) 2019-10-13T09:00:15.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:15.001+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.004851+0000) 2019-10-13T09:00:15.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:15.442+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.448431+0000) 2019-10-13T09:00:15.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:15.445+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.448883+0000) 2019-10-13T09:00:15.450 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:15.445+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.449192+0000) 2019-10-13T09:00:15.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:15.445+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.449969+0000) 2019-10-13T09:00:15.451 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:15.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.449526+0000) 2019-10-13T09:00:15.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:15.445+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.450978+0000) 2019-10-13T09:00:15.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:15.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:15.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:15.449+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.451576+0000) 2019-10-13T09:00:15.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:15.454+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.458725+0000) 2019-10-13T09:00:16.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:16.221+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:16.226506+0000) 2019-10-13T09:00:18.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:18.489+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:18.494994+0000) 2019-10-13T09:00:19.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:19.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.479624+0000) 2019-10-13T09:00:20.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:20.449+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.454409+0000) 2019-10-13T09:00:20.456 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:20.449+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.455025+0000) 2019-10-13T09:00:20.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:20.449+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.455229+0000) 2019-10-13T09:00:20.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:20.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.455598+0000) 2019-10-13T09:00:20.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:20.453+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.456166+0000) 2019-10-13T09:00:20.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:20.450+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.457493+0000) 2019-10-13T09:00:20.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:20.453+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.457774+0000) 2019-10-13T09:00:20.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:20.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:20.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:20.458+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.463632+0000) 2019-10-13T09:00:21.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:21.538+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.539664+0000) 2019-10-13T09:00:21.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:21.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.738788+0000) 2019-10-13T09:00:21.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:21.780+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.784633+0000) 2019-10-13T09:00:21.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:21.976+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.977903+0000) 2019-10-13T09:00:25.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:25.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.005079+0000) 2019-10-13T09:00:25.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:25.456+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.461239+0000) 2019-10-13T09:00:25.462 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:25.456+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.461640+0000) 2019-10-13T09:00:25.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:25.456+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.461515+0000) 2019-10-13T09:00:25.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:25.456+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.462504+0000) 2019-10-13T09:00:25.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:25.460+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.463639+0000) 2019-10-13T09:00:25.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:25.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:25.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:25.458+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.463954+0000) 2019-10-13T09:00:25.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:25.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.464672+0000) 2019-10-13T09:00:25.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:25.462+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.469574+0000) 2019-10-13T09:00:26.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:26.220+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:26.226682+0000) 2019-10-13T09:00:28.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:28.488+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:28.495209+0000) 2019-10-13T09:00:29.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:29.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.479821+0000) 2019-10-13T09:00:30.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:30.466+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.467489+0000) 2019-10-13T09:00:30.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:30.464+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.468187+0000) 2019-10-13T09:00:30.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:30.464+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.468483+0000) 2019-10-13T09:00:30.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:30.464+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.469082+0000) 2019-10-13T09:00:30.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:30.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:30.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:30.464+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.470032+0000) 2019-10-13T09:00:30.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:30.464+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.470267+0000) 2019-10-13T09:00:30.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:30.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.470525+0000) 2019-10-13T09:00:30.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:30.474+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.475472+0000) 2019-10-13T09:00:31.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:31.542+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.546119+0000) 2019-10-13T09:00:31.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:31.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.739016+0000) 2019-10-13T09:00:31.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:31.780+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.784876+0000) 2019-10-13T09:00:31.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:31.976+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.978129+0000) 2019-10-13T09:00:35.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:35.000+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.005277+0000) 2019-10-13T09:00:35.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:35.470+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.472241+0000) 2019-10-13T09:00:35.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:35.468+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.472915+0000) 2019-10-13T09:00:35.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:35.468+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.473143+0000) 2019-10-13T09:00:35.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:35.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.473672+0000) 2019-10-13T09:00:35.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:35.468+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.474627+0000) 2019-10-13T09:00:35.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:35.468+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.475112+0000) 2019-10-13T09:00:35.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:35.472+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.475605+0000) 2019-10-13T09:00:35.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:35.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:35.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:35.482+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.484756+0000) 2019-10-13T09:00:36.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:36.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:36.226855+0000) 2019-10-13T09:00:36.319 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:00:36.320 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:36.324 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:00:36.324 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:38.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:38.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:38.495410+0000) 2019-10-13T09:00:39.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:39.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.480014+0000) 2019-10-13T09:00:40.480 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:40.476+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.479487+0000) 2019-10-13T09:00:40.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:40.476+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.479776+0000) 2019-10-13T09:00:40.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:40.476+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.480186+0000) 2019-10-13T09:00:40.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:40.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.480563+0000) 2019-10-13T09:00:40.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:40.476+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.481537+0000) 2019-10-13T09:00:40.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:40.476+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.482186+0000) 2019-10-13T09:00:40.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:40.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:40.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:40.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.482733+0000) 2019-10-13T09:00:40.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:40.489+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.491387+0000) 2019-10-13T09:00:41.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:41.549+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.552868+0000) 2019-10-13T09:00:41.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:41.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.739222+0000) 2019-10-13T09:00:41.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:41.780+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.785092+0000) 2019-10-13T09:00:41.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:41.976+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.978308+0000) 2019-10-13T09:00:45.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:45.000+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.005474+0000) 2019-10-13T09:00:45.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:45.480+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.485218+0000) 2019-10-13T09:00:45.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:45.481+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.485781+0000) 2019-10-13T09:00:45.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:45.480+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.486956+0000) 2019-10-13T09:00:45.488 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:45.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.486592+0000) 2019-10-13T09:00:45.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:45.484+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.487926+0000) 2019-10-13T09:00:45.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:45.484+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.487648+0000) 2019-10-13T09:00:45.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:45.484+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.488041+0000) 2019-10-13T09:00:45.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:45.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:45.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:45.493+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.494524+0000) 2019-10-13T09:00:46.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:46.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:46.227035+0000) 2019-10-13T09:00:48.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:48.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:48.495650+0000) 2019-10-13T09:00:49.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:49.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.480233+0000) 2019-10-13T09:00:50.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:50.489+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.492450+0000) 2019-10-13T09:00:50.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:50.488+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.492787+0000) 2019-10-13T09:00:50.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:50.488+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.493253+0000) 2019-10-13T09:00:50.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:50.488+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.493540+0000) 2019-10-13T09:00:50.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:50.488+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.493832+0000) 2019-10-13T09:00:50.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:50.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:50.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:50.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.494929+0000) 2019-10-13T09:00:50.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:50.492+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.495757+0000) 2019-10-13T09:00:50.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:50.497+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.501577+0000) 2019-10-13T09:00:51.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:51.557+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.559278+0000) 2019-10-13T09:00:51.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:51.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.739398+0000) 2019-10-13T09:00:51.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:51.780+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.785294+0000) 2019-10-13T09:00:51.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:51.976+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.978502+0000) 2019-10-13T09:00:55.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:55.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.005674+0000) 2019-10-13T09:00:55.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:00:55.492+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.498312+0000) 2019-10-13T09:00:55.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:00:55.492+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.498956+0000) 2019-10-13T09:00:55.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:55.496+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.499611+0000) 2019-10-13T09:00:55.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:00:55.496+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.500061+0000) 2019-10-13T09:00:55.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:55.497+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.500680+0000) 2019-10-13T09:00:55.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:00:55.496+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.500584+0000) 2019-10-13T09:00:55.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:00:55.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:00:55.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:55.496+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.501089+0000) 2019-10-13T09:00:55.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:00:55.505+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.508077+0000) 2019-10-13T09:00:56.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:00:56.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:56.227236+0000) 2019-10-13T09:00:58.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:00:58.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:58.495878+0000) 2019-10-13T09:00:59.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:00:59.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.480431+0000) 2019-10-13T09:01:00.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:00.500+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.504098+0000) 2019-10-13T09:01:00.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:00.501+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.504389+0000) 2019-10-13T09:01:00.505 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:00.500+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.505011+0000) 2019-10-13T09:01:00.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:00.500+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.505250+0000) 2019-10-13T09:01:00.506 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:00.500+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.506186+0000) 2019-10-13T09:01:00.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:00.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:00.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:00.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.506891+0000) 2019-10-13T09:01:00.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:00.504+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.507485+0000) 2019-10-13T09:01:00.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:00.513+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.514682+0000) 2019-10-13T09:01:01.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:01.561+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.565879+0000) 2019-10-13T09:01:01.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:01.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.739561+0000) 2019-10-13T09:01:01.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:01.780+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.785514+0000) 2019-10-13T09:01:01.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:01.976+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.978700+0000) 2019-10-13T09:01:05.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:05.003+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.005879+0000) 2019-10-13T09:01:05.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:05.505+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.509920+0000) 2019-10-13T09:01:05.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:05.503+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.510596+0000) 2019-10-13T09:01:05.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:05.503+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.510710+0000) 2019-10-13T09:01:05.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:05.507+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.512056+0000) 2019-10-13T09:01:05.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:05.507+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.512446+0000) 2019-10-13T09:01:05.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:05.507+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.512771+0000) 2019-10-13T09:01:05.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:05.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65545 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:05.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:05.507+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.513393+0000) 2019-10-13T09:01:05.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:05.517+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.518227+0000) 2019-10-13T09:01:06.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:06.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:06.227424+0000) 2019-10-13T09:01:06.417 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:01:06.417 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:06.420 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:01:06.420 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:08.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:08.491+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:08.496095+0000) 2019-10-13T09:01:09.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:09.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.480644+0000) 2019-10-13T09:01:10.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:10.513+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.516030+0000) 2019-10-13T09:01:10.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:10.511+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.517069+0000) 2019-10-13T09:01:10.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:10.511+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.517583+0000) 2019-10-13T09:01:10.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:10.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:10.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:10.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.518772+0000) 2019-10-13T09:01:10.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:10.511+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.519112+0000) 2019-10-13T09:01:10.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:10.515+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.519350+0000) 2019-10-13T09:01:10.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:10.515+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.519658+0000) 2019-10-13T09:01:10.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:10.521+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.524279+0000) 2019-10-13T09:01:11.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:11.569+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.572279+0000) 2019-10-13T09:01:11.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:11.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.739745+0000) 2019-10-13T09:01:11.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:11.779+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.785734+0000) 2019-10-13T09:01:11.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:11.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.978927+0000) 2019-10-13T09:01:15.007 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:14.999+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.006077+0000) 2019-10-13T09:01:15.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:15.515+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.521800+0000) 2019-10-13T09:01:15.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:15.515+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.522785+0000) 2019-10-13T09:01:15.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:15.521+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.523754+0000) 2019-10-13T09:01:15.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:15.519+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.523439+0000) 2019-10-13T09:01:15.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:15.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.524323+0000) 2019-10-13T09:01:15.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:15.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:15.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:15.519+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.525156+0000) 2019-10-13T09:01:15.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:15.519+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.524849+0000) 2019-10-13T09:01:15.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:15.525+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.528867+0000) 2019-10-13T09:01:16.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:16.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:16.227583+0000) 2019-10-13T09:01:18.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:18.491+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:18.496254+0000) 2019-10-13T09:01:19.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:19.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.480827+0000) 2019-10-13T09:01:20.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:20.523+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.529273+0000) 2019-10-13T09:01:20.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:20.523+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.529748+0000) 2019-10-13T09:01:20.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:20.523+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.530050+0000) 2019-10-13T09:01:20.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:20.523+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.530534+0000) 2019-10-13T09:01:20.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:20.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:20.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:20.529+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.531965+0000) 2019-10-13T09:01:20.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:20.527+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.532050+0000) 2019-10-13T09:01:20.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:20.527+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.531947+0000) 2019-10-13T09:01:20.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:20.533+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.537827+0000) 2019-10-13T09:01:21.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:21.577+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.578744+0000) 2019-10-13T09:01:21.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:21.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.739969+0000) 2019-10-13T09:01:21.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:21.779+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.785938+0000) 2019-10-13T09:01:21.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:21.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.979170+0000) 2019-10-13T09:01:25.007 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:25.003+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.006289+0000) 2019-10-13T09:01:25.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:25.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.534124+0000) 2019-10-13T09:01:25.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:25.527+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.534641+0000) 2019-10-13T09:01:25.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:25.527+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.535173+0000) 2019-10-13T09:01:25.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:25.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.535734+0000) 2019-10-13T09:01:25.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:25.531+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.536741+0000) 2019-10-13T09:01:25.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:25.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:25.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:25.531+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.537577+0000) 2019-10-13T09:01:25.538 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:25.531+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.538049+0000) 2019-10-13T09:01:25.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:25.541+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.545306+0000) 2019-10-13T09:01:26.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:26.227+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:26.227763+0000) 2019-10-13T09:01:28.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:28.495+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:28.496475+0000) 2019-10-13T09:01:29.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:29.477+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.481018+0000) 2019-10-13T09:01:30.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:30.535+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.540344+0000) 2019-10-13T09:01:30.542 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:30.535+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.542221+0000) 2019-10-13T09:01:30.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:30.535+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.542591+0000) 2019-10-13T09:01:30.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:30.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.542937+0000) 2019-10-13T09:01:30.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:30.535+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.543233+0000) 2019-10-13T09:01:30.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:30.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:30.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:30.539+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.543874+0000) 2019-10-13T09:01:30.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:30.539+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.544462+0000) 2019-10-13T09:01:30.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:30.545+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.548774+0000) 2019-10-13T09:01:31.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:31.584+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.585198+0000) 2019-10-13T09:01:31.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:31.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.740199+0000) 2019-10-13T09:01:31.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:31.779+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.786134+0000) 2019-10-13T09:01:31.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:31.975+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.979380+0000) 2019-10-13T09:01:35.007 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:35.003+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.006522+0000) 2019-10-13T09:01:35.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:35.544+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.546903+0000) 2019-10-13T09:01:35.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:35.547+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.547421+0000) 2019-10-13T09:01:35.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:35.547+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.547850+0000) 2019-10-13T09:01:35.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:35.547+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.548159+0000) 2019-10-13T09:01:35.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:35.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.548830+0000) 2019-10-13T09:01:35.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:35.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.549284+0000) 2019-10-13T09:01:35.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:35.547+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:35.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:35.547+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.550474+0000) 2019-10-13T09:01:35.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:35.556+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.559829+0000) 2019-10-13T09:01:36.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:36.227+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:36.227937+0000) 2019-10-13T09:01:36.754 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:01:36.754 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:36.758 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:01:36.758 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:38.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:38.495+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:38.496680+0000) 2019-10-13T09:01:39.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:39.476+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.481225+0000) 2019-10-13T09:01:40.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:40.551+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.552900+0000) 2019-10-13T09:01:40.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:40.551+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.553422+0000) 2019-10-13T09:01:40.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:40.551+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.553923+0000) 2019-10-13T09:01:40.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:40.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.554780+0000) 2019-10-13T09:01:40.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:40.555+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.555358+0000) 2019-10-13T09:01:40.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:40.552+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.555610+0000) 2019-10-13T09:01:40.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:40.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65780 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:40.557 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:40.555+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.556154+0000) 2019-10-13T09:01:40.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:40.560+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.563925+0000) 2019-10-13T09:01:41.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:41.588+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.591732+0000) 2019-10-13T09:01:41.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:41.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.740434+0000) 2019-10-13T09:01:41.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:41.783+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.786331+0000) 2019-10-13T09:01:41.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:41.979+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.979585+0000) 2019-10-13T09:01:45.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:45.003+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.006721+0000) 2019-10-13T09:01:45.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:45.555+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.558923+0000) 2019-10-13T09:01:45.560 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:45.556+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.559610+0000) 2019-10-13T09:01:45.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:45.559+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.560068+0000) 2019-10-13T09:01:45.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:45.559+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.560572+0000) 2019-10-13T09:01:45.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:45.559+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.561023+0000) 2019-10-13T09:01:45.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:45.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:45.563 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:45.559+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.561757+0000) 2019-10-13T09:01:45.564 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:45.559+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.561855+0000) 2019-10-13T09:01:45.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:45.564+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.568693+0000) 2019-10-13T09:01:46.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:46.227+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:46.228107+0000) 2019-10-13T09:01:48.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:48.494+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:48.496909+0000) 2019-10-13T09:01:49.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:49.476+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.481451+0000) 2019-10-13T09:01:50.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:50.562+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.565842+0000) 2019-10-13T09:01:50.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:50.562+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.566333+0000) 2019-10-13T09:01:50.568 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:50.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.566784+0000) 2019-10-13T09:01:50.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:50.562+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.567074+0000) 2019-10-13T09:01:50.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:50.566+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.567342+0000) 2019-10-13T09:01:50.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:50.566+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.567707+0000) 2019-10-13T09:01:50.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:50.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65848 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:50.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:50.566+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.568227+0000) 2019-10-13T09:01:50.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:50.568+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.573592+0000) 2019-10-13T09:01:51.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:51.600+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.602845+0000) 2019-10-13T09:01:51.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:51.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.740642+0000) 2019-10-13T09:01:51.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:51.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.786499+0000) 2019-10-13T09:01:51.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:51.978+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.979740+0000) 2019-10-13T09:01:55.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:55.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.006946+0000) 2019-10-13T09:01:55.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:55.568+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.570618+0000) 2019-10-13T09:01:55.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:01:55.570+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.571812+0000) 2019-10-13T09:01:55.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:01:55.570+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.572184+0000) 2019-10-13T09:01:55.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:55.570+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.572610+0000) 2019-10-13T09:01:55.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:01:55.570+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.573020+0000) 2019-10-13T09:01:55.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:55.570+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.573364+0000) 2019-10-13T09:01:55.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:01:55.570+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.573657+0000) 2019-10-13T09:01:55.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:01:55.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:01:55.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:01:55.572+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.577546+0000) 2019-10-13T09:01:56.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:01:56.226+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:56.228284+0000) 2019-10-13T09:01:58.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:01:58.494+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:58.497138+0000) 2019-10-13T09:01:59.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:01:59.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.481647+0000) 2019-10-13T09:02:00.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:00.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.577400+0000) 2019-10-13T09:02:00.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:00.574+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.577868+0000) 2019-10-13T09:02:00.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:00.574+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.578362+0000) 2019-10-13T09:02:00.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:00.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.578951+0000) 2019-10-13T09:02:00.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:00.576+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.579368+0000) 2019-10-13T09:02:00.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:00.578+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.580046+0000) 2019-10-13T09:02:00.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:00.578+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.581276+0000) 2019-10-13T09:02:00.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:00.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:00.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:00.584+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.587178+0000) 2019-10-13T09:02:01.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:01.608+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.609252+0000) 2019-10-13T09:02:01.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:01.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.740830+0000) 2019-10-13T09:02:01.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:01.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.786711+0000) 2019-10-13T09:02:01.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:01.978+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.979968+0000) 2019-10-13T09:02:05.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:05.002+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.007178+0000) 2019-10-13T09:02:05.585 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:05.580+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.584259+0000) 2019-10-13T09:02:05.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:05.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.584940+0000) 2019-10-13T09:02:05.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:05.582+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.584776+0000) 2019-10-13T09:02:05.586 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:05.582+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.585658+0000) 2019-10-13T09:02:05.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:05.582+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.586179+0000) 2019-10-13T09:02:05.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:05.582+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.586626+0000) 2019-10-13T09:02:05.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:05.582+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.587162+0000) 2019-10-13T09:02:05.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:05.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65953 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:05.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:05.588+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.591954+0000) 2019-10-13T09:02:06.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:06.226+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:06.228464+0000) 2019-10-13T09:02:06.805 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:02:06.805 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:06.809 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:02:06.809 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:08.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:08.494+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:08.497332+0000) 2019-10-13T09:02:09.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:09.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.481835+0000) 2019-10-13T09:02:10.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:10.586+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.589815+0000) 2019-10-13T09:02:10.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:10.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.591109+0000) 2019-10-13T09:02:10.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:10.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.591393+0000) 2019-10-13T09:02:10.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:10.590+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.592237+0000) 2019-10-13T09:02:10.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:10.590+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.592958+0000) 2019-10-13T09:02:10.594 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:10.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.593006+0000) 2019-10-13T09:02:10.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:10.590+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.593474+0000) 2019-10-13T09:02:10.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:10.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 65984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:10.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:10.596+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.600719+0000) 2019-10-13T09:02:11.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:11.612+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.615475+0000) 2019-10-13T09:02:11.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:11.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.741031+0000) 2019-10-13T09:02:11.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:11.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.786958+0000) 2019-10-13T09:02:11.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:11.978+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.980169+0000) 2019-10-13T09:02:15.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:15.006+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.007401+0000) 2019-10-13T09:02:15.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:15.592+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.596363+0000) 2019-10-13T09:02:15.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:15.594+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.596733+0000) 2019-10-13T09:02:15.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:15.594+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.597703+0000) 2019-10-13T09:02:15.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:15.594+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.598864+0000) 2019-10-13T09:02:15.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:15.594+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.599135+0000) 2019-10-13T09:02:15.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:15.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:15.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:15.598+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.599526+0000) 2019-10-13T09:02:15.612 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:15.598+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.600115+0000) 2019-10-13T09:02:15.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:15.604+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.606420+0000) 2019-10-13T09:02:16.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:16.226+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:16.228662+0000) 2019-10-13T09:02:18.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:18.494+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:18.497553+0000) 2019-10-13T09:02:19.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:19.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.482032+0000) 2019-10-13T09:02:20.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:20.598+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.602526+0000) 2019-10-13T09:02:20.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:20.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.604198+0000) 2019-10-13T09:02:20.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:20.602+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.604636+0000) 2019-10-13T09:02:20.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:20.602+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.605251+0000) 2019-10-13T09:02:20.606 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:20.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:20.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:20.602+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.605009+0000) 2019-10-13T09:02:20.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:20.600+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.605587+0000) 2019-10-13T09:02:20.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:20.602+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.606344+0000) 2019-10-13T09:02:20.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:20.608+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.613261+0000) 2019-10-13T09:02:21.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:21.616+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.621852+0000) 2019-10-13T09:02:21.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:21.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.741242+0000) 2019-10-13T09:02:21.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:21.782+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.787147+0000) 2019-10-13T09:02:21.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:21.978+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.980381+0000) 2019-10-13T09:02:25.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:25.006+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.007602+0000) 2019-10-13T09:02:25.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:25.606+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.608849+0000) 2019-10-13T09:02:25.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:25.606+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.609206+0000) 2019-10-13T09:02:25.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:25.606+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.609629+0000) 2019-10-13T09:02:25.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:25.606+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.609910+0000) 2019-10-13T09:02:25.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:25.606+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.611265+0000) 2019-10-13T09:02:25.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:25.606+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.610950+0000) 2019-10-13T09:02:25.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:25.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66089 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:25.613 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:25.608+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.612520+0000) 2019-10-13T09:02:25.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:25.616+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.618146+0000) 2019-10-13T09:02:26.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:26.226+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:26.228875+0000) 2019-10-13T09:02:28.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:28.494+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:28.497751+0000) 2019-10-13T09:02:29.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:29.479+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.482220+0000) 2019-10-13T09:02:30.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:30.610+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.614716+0000) 2019-10-13T09:02:30.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:30.614+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.615668+0000) 2019-10-13T09:02:30.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:30.614+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.616234+0000) 2019-10-13T09:02:30.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:30.611+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.617273+0000) 2019-10-13T09:02:30.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:30.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.616652+0000) 2019-10-13T09:02:30.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:30.614+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.617383+0000) 2019-10-13T09:02:30.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:30.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.617684+0000) 2019-10-13T09:02:30.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:30.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:30.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:30.619+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.624932+0000) 2019-10-13T09:02:31.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:31.627+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.628134+0000) 2019-10-13T09:02:31.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:31.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.741431+0000) 2019-10-13T09:02:31.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:31.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.787365+0000) 2019-10-13T09:02:31.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:31.977+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.980592+0000) 2019-10-13T09:02:35.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:35.005+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.007833+0000) 2019-10-13T09:02:35.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:35.617+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.621418+0000) 2019-10-13T09:02:35.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:35.617+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.622079+0000) 2019-10-13T09:02:35.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:35.617+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.622479+0000) 2019-10-13T09:02:35.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:35.617+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.622755+0000) 2019-10-13T09:02:35.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:35.621+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.623978+0000) 2019-10-13T09:02:35.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:35.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.624349+0000) 2019-10-13T09:02:35.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:35.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:35.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:35.621+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.625082+0000) 2019-10-13T09:02:35.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:35.627+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.630657+0000) 2019-10-13T09:02:36.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:36.225+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:36.229083+0000) 2019-10-13T09:02:36.917 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:02:36.917 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:36.920 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:02:36.921 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:38.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:38.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:38.498015+0000) 2019-10-13T09:02:39.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:39.479+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.482375+0000) 2019-10-13T09:02:40.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:40.625+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.628085+0000) 2019-10-13T09:02:40.808 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:40.627+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.630466+0000) 2019-10-13T09:02:40.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:40.631+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.636453+0000) 2019-10-13T09:02:40.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:40.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.628519+0000) 2019-10-13T09:02:40.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:40.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.629361+0000) 2019-10-13T09:02:40.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:40.625+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.630197+0000) 2019-10-13T09:02:40.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:40.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:40.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:40.625+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.630923+0000) 2019-10-13T09:02:40.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:40.629+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.631356+0000) 2019-10-13T09:02:41.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:41.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.634558+0000) 2019-10-13T09:02:41.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:41.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.741608+0000) 2019-10-13T09:02:41.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:41.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.787588+0000) 2019-10-13T09:02:41.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:41.977+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.980794+0000) 2019-10-13T09:02:45.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:45.005+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.008000+0000) 2019-10-13T09:02:45.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:45.629+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.634250+0000) 2019-10-13T09:02:45.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:45.629+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.634453+0000) 2019-10-13T09:02:45.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:45.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.635197+0000) 2019-10-13T09:02:45.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:45.633+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.635662+0000) 2019-10-13T09:02:45.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:45.631+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.635996+0000) 2019-10-13T09:02:45.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:45.633+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.636531+0000) 2019-10-13T09:02:45.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:45.633+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.637127+0000) 2019-10-13T09:02:45.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:45.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:45.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:45.643+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.646790+0000) 2019-10-13T09:02:46.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:46.225+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:46.229286+0000) 2019-10-13T09:02:48.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:48.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:48.498280+0000) 2019-10-13T09:02:49.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:49.479+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.482597+0000) 2019-10-13T09:02:50.642 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:50.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.641808+0000) 2019-10-13T09:02:50.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:50.637+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.642106+0000) 2019-10-13T09:02:50.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:50.637+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.642633+0000) 2019-10-13T09:02:50.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:50.637+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.642871+0000) 2019-10-13T09:02:50.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:50.641+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.643504+0000) 2019-10-13T09:02:50.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:50.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:50.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:50.639+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.644889+0000) 2019-10-13T09:02:50.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:50.641+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.645606+0000) 2019-10-13T09:02:50.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:50.643+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.649566+0000) 2019-10-13T09:02:51.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:51.643+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.646172+0000) 2019-10-13T09:02:51.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:51.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.741821+0000) 2019-10-13T09:02:51.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:51.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.787829+0000) 2019-10-13T09:02:51.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:51.977+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.981024+0000) 2019-10-13T09:02:55.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:55.005+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.008223+0000) 2019-10-13T09:02:55.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:02:55.641+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.647167+0000) 2019-10-13T09:02:55.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:55.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.647842+0000) 2019-10-13T09:02:55.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:02:55.645+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.648414+0000) 2019-10-13T09:02:55.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:55.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.649200+0000) 2019-10-13T09:02:55.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:02:55.645+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.649026+0000) 2019-10-13T09:02:55.650 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:55.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.649729+0000) 2019-10-13T09:02:55.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:02:55.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.650005+0000) 2019-10-13T09:02:55.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:02:55.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:02:55.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:02:55.651+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.656511+0000) 2019-10-13T09:02:56.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:02:56.225+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:56.229475+0000) 2019-10-13T09:02:58.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:02:58.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:58.498482+0000) 2019-10-13T09:02:59.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:02:59.479+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.482823+0000) 2019-10-13T09:03:00.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:00.649+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.653434+0000) 2019-10-13T09:03:00.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:00.649+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.654483+0000) 2019-10-13T09:03:00.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:00.651+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.654972+0000) 2019-10-13T09:03:00.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:00.649+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.655187+0000) 2019-10-13T09:03:00.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:00.653+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.655647+0000) 2019-10-13T09:03:00.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:00.653+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.655995+0000) 2019-10-13T09:03:00.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:00.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:00.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:00.653+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.657277+0000) 2019-10-13T09:03:00.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:00.659+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.662721+0000) 2019-10-13T09:03:01.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:01.651+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.652372+0000) 2019-10-13T09:03:01.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:01.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.742031+0000) 2019-10-13T09:03:01.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:01.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.788062+0000) 2019-10-13T09:03:01.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:01.977+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.981198+0000) 2019-10-13T09:03:05.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:05.005+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.008424+0000) 2019-10-13T09:03:05.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:05.657+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.659656+0000) 2019-10-13T09:03:05.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:05.657+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.660051+0000) 2019-10-13T09:03:05.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:05.657+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.660940+0000) 2019-10-13T09:03:05.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:05.657+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.661251+0000) 2019-10-13T09:03:05.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:05.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:05.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:05.659+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.662461+0000) 2019-10-13T09:03:05.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:05.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.663058+0000) 2019-10-13T09:03:05.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:05.661+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.663486+0000) 2019-10-13T09:03:05.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:05.667+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.670819+0000) 2019-10-13T09:03:06.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:06.225+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:06.229654+0000) 2019-10-13T09:03:07.045 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:03:07.045 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:07.056 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:03:07.056 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:08.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:08.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:08.498694+0000) 2019-10-13T09:03:09.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:09.479+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.483018+0000) 2019-10-13T09:03:10.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:10.663+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.668923+0000) 2019-10-13T09:03:10.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:10.661+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.665714+0000) 2019-10-13T09:03:10.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:10.661+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.666643+0000) 2019-10-13T09:03:10.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:10.665+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.667385+0000) 2019-10-13T09:03:10.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:10.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:10.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:10.665+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.668433+0000) 2019-10-13T09:03:10.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:10.665+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.668247+0000) 2019-10-13T09:03:10.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:10.665+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.669581+0000) 2019-10-13T09:03:10.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:10.667+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.671894+0000) 2019-10-13T09:03:11.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:11.655+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.658636+0000) 2019-10-13T09:03:11.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:11.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.742230+0000) 2019-10-13T09:03:11.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:11.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.788249+0000) 2019-10-13T09:03:11.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:11.981+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.981397+0000) 2019-10-13T09:03:15.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:15.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.008611+0000) 2019-10-13T09:03:15.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:15.664+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.671170+0000) 2019-10-13T09:03:15.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:15.668+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.671605+0000) 2019-10-13T09:03:15.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:15.668+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.672318+0000) 2019-10-13T09:03:15.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:15.668+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.672453+0000) 2019-10-13T09:03:15.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:15.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.673487+0000) 2019-10-13T09:03:15.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:15.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:15.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:15.671+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.674378+0000) 2019-10-13T09:03:15.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:15.668+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.674683+0000) 2019-10-13T09:03:15.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:15.675+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.680552+0000) 2019-10-13T09:03:16.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:16.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:16.229832+0000) 2019-10-13T09:03:18.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:18.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:18.498894+0000) 2019-10-13T09:03:19.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:19.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.483213+0000) 2019-10-13T09:03:20.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:20.672+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.677442+0000) 2019-10-13T09:03:20.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:20.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.677996+0000) 2019-10-13T09:03:20.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:20.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.678246+0000) 2019-10-13T09:03:20.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:20.672+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.678958+0000) 2019-10-13T09:03:20.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:20.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.679273+0000) 2019-10-13T09:03:20.689 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:20.676+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.679876+0000) 2019-10-13T09:03:20.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:20.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:20.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:20.675+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.681216+0000) 2019-10-13T09:03:20.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:20.683+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.687171+0000) 2019-10-13T09:03:21.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:21.662+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.665186+0000) 2019-10-13T09:03:21.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:21.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.742467+0000) 2019-10-13T09:03:21.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:21.784+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.788462+0000) 2019-10-13T09:03:21.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:21.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.981595+0000) 2019-10-13T09:03:25.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:25.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.008837+0000) 2019-10-13T09:03:25.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:25.680+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.683549+0000) 2019-10-13T09:03:25.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:25.678+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.684077+0000) 2019-10-13T09:03:25.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:25.680+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.684870+0000) 2019-10-13T09:03:25.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:25.680+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.685044+0000) 2019-10-13T09:03:25.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:25.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.685666+0000) 2019-10-13T09:03:25.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:25.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.685885+0000) 2019-10-13T09:03:25.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:25.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:25.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:25.680+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.686607+0000) 2019-10-13T09:03:25.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:25.690+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.694155+0000) 2019-10-13T09:03:26.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:26.228+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:26.230009+0000) 2019-10-13T09:03:28.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:28.492+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:28.499109+0000) 2019-10-13T09:03:29.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:29.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.483403+0000) 2019-10-13T09:03:30.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:30.684+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.689852+0000) 2019-10-13T09:03:30.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:30.684+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.690358+0000) 2019-10-13T09:03:30.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:30.686+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.690675+0000) 2019-10-13T09:03:30.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:30.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.691729+0000) 2019-10-13T09:03:30.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:30.688+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.692115+0000) 2019-10-13T09:03:30.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:30.688+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.692560+0000) 2019-10-13T09:03:30.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:30.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:30.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:30.688+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.693315+0000) 2019-10-13T09:03:30.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:30.690+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.698119+0000) 2019-10-13T09:03:31.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:31.670+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.671718+0000) 2019-10-13T09:03:31.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:31.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.742671+0000) 2019-10-13T09:03:31.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:31.784+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.788643+0000) 2019-10-13T09:03:31.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:31.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.981792+0000) 2019-10-13T09:03:35.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:35.008+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.009062+0000) 2019-10-13T09:03:35.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:35.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.695721+0000) 2019-10-13T09:03:35.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:35.692+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.697159+0000) 2019-10-13T09:03:35.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:35.690+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.697872+0000) 2019-10-13T09:03:35.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:35.692+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.698289+0000) 2019-10-13T09:03:35.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:35.692+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.698711+0000) 2019-10-13T09:03:35.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:35.692+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.699153+0000) 2019-10-13T09:03:35.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:35.696+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.699604+0000) 2019-10-13T09:03:35.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:35.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:35.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:35.702+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.706415+0000) 2019-10-13T09:03:36.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:36.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:36.230190+0000) 2019-10-13T09:03:37.172 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:03:37.173 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:37.176 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:03:37.176 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:38.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:38.496+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:38.499347+0000) 2019-10-13T09:03:39.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:39.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:39.483600+0000) 2019-10-13T09:03:40.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:40.698+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.703439+0000) 2019-10-13T09:03:40.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:40.700+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.703629+0000) 2019-10-13T09:03:40.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:40.700+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.704592+0000) 2019-10-13T09:03:40.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:40.700+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.704290+0000) 2019-10-13T09:03:40.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:40.700+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.704831+0000) 2019-10-13T09:03:40.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:40.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:40.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:40.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.706541+0000) 2019-10-13T09:03:40.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:40.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.706756+0000) 2019-10-13T09:03:40.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:40.706+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.711088+0000) 2019-10-13T09:03:41.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:41.674+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.678014+0000) 2019-10-13T09:03:41.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:41.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.742869+0000) 2019-10-13T09:03:41.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:41.784+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.788829+0000) 2019-10-13T09:03:41.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:41.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.982017+0000) 2019-10-13T09:03:45.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:45.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.009259+0000) 2019-10-13T09:03:45.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:45.704+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.709424+0000) 2019-10-13T09:03:45.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:45.704+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.710185+0000) 2019-10-13T09:03:45.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:45.704+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.710592+0000) 2019-10-13T09:03:45.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:45.706+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.710814+0000) 2019-10-13T09:03:45.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:45.704+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.711061+0000) 2019-10-13T09:03:45.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:45.708+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.711427+0000) 2019-10-13T09:03:45.713 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:45.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:45.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:45.708+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.713050+0000) 2019-10-13T09:03:45.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:45.714+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.719720+0000) 2019-10-13T09:03:46.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:46.228+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:46.230377+0000) 2019-10-13T09:03:48.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:48.496+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:48.499547+0000) 2019-10-13T09:03:49.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:49.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:49.483799+0000) 2019-10-13T09:03:50.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:50.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.715472+0000) 2019-10-13T09:03:50.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:50.710+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.715965+0000) 2019-10-13T09:03:50.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:50.712+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.716668+0000) 2019-10-13T09:03:50.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:50.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.717554+0000) 2019-10-13T09:03:50.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:50.712+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.718483+0000) 2019-10-13T09:03:50.719 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:50.712+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.718771+0000) 2019-10-13T09:03:50.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:50.712+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.718413+0000) 2019-10-13T09:03:50.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:50.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:50.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:50.718+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.725873+0000) 2019-10-13T09:03:51.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:51.682+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.684465+0000) 2019-10-13T09:03:51.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:51.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.743072+0000) 2019-10-13T09:03:51.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:51.784+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.789051+0000) 2019-10-13T09:03:51.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:51.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.982213+0000) 2019-10-13T09:03:55.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:55.004+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.009453+0000) 2019-10-13T09:03:55.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:03:55.716+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.721787+0000) 2019-10-13T09:03:55.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:03:55.716+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.722487+0000) 2019-10-13T09:03:55.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:55.716+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.723126+0000) 2019-10-13T09:03:55.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:55.718+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.723554+0000) 2019-10-13T09:03:55.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:03:55.720+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.723976+0000) 2019-10-13T09:03:55.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:55.716+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.722722+0000) 2019-10-13T09:03:55.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:03:55.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:03:55.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:03:55.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.725214+0000) 2019-10-13T09:03:55.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:03:55.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.733067+0000) 2019-10-13T09:03:56.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:03:56.224+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:56.230509+0000) 2019-10-13T09:03:58.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:03:58.495+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:58.499749+0000) 2019-10-13T09:03:59.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:03:59.478+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:59.483996+0000) 2019-10-13T09:04:00.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:00.722+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.727774+0000) 2019-10-13T09:04:00.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:00.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.728552+0000) 2019-10-13T09:04:00.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:00.723+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.728822+0000) 2019-10-13T09:04:00.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:00.723+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.729114+0000) 2019-10-13T09:04:00.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:00.723+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.729418+0000) 2019-10-13T09:04:00.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:00.723+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.729877+0000) 2019-10-13T09:04:00.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:00.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:00.733 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:00.727+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.732180+0000) 2019-10-13T09:04:00.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:00.730+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.737031+0000) 2019-10-13T09:04:01.691 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:01.690+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.690941+0000) 2019-10-13T09:04:01.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:01.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.743266+0000) 2019-10-13T09:04:01.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:01.783+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.789253+0000) 2019-10-13T09:04:01.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:01.979+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.982412+0000) 2019-10-13T09:04:05.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:05.007+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.009678+0000) 2019-10-13T09:04:05.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:05.727+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.733380+0000) 2019-10-13T09:04:05.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:05.727+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.734158+0000) 2019-10-13T09:04:05.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:05.730+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.734891+0000) 2019-10-13T09:04:05.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:05.727+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.735203+0000) 2019-10-13T09:04:05.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:05.731+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.735741+0000) 2019-10-13T09:04:05.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:05.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:05.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:05.731+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.736658+0000) 2019-10-13T09:04:05.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:05.731+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.737127+0000) 2019-10-13T09:04:05.745 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:05.738+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.744974+0000) 2019-10-13T09:04:06.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:06.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:06.230681+0000) 2019-10-13T09:04:07.276 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:04:07.276 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:07.279 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:04:07.280 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:08.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:08.495+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:08.499922+0000) 2019-10-13T09:04:09.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:09.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:09.484186+0000) 2019-10-13T09:04:10.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:10.735+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.739475+0000) 2019-10-13T09:04:10.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:10.735+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.740045+0000) 2019-10-13T09:04:10.741 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:10.735+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.740484+0000) 2019-10-13T09:04:10.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:10.735+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.741761+0000) 2019-10-13T09:04:10.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:10.742+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.742258+0000) 2019-10-13T09:04:10.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:10.735+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.742604+0000) 2019-10-13T09:04:10.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:10.735+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.742952+0000) 2019-10-13T09:04:10.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:10.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:10.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:10.746+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.747176+0000) 2019-10-13T09:04:11.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:11.694+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.697493+0000) 2019-10-13T09:04:11.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:11.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.743506+0000) 2019-10-13T09:04:11.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:11.783+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.789490+0000) 2019-10-13T09:04:11.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:11.979+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.982615+0000) 2019-10-13T09:04:15.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:15.003+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.009892+0000) 2019-10-13T09:04:15.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:15.739+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.746641+0000) 2019-10-13T09:04:15.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:15.739+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.746816+0000) 2019-10-13T09:04:15.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:15.743+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.747952+0000) 2019-10-13T09:04:15.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:15.743+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.748481+0000) 2019-10-13T09:04:15.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:15.743+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.748948+0000) 2019-10-13T09:04:15.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:15.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:15.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:15.743+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.749732+0000) 2019-10-13T09:04:15.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:15.750+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.750118+0000) 2019-10-13T09:04:15.756 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:15.754+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.756067+0000) 2019-10-13T09:04:16.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:16.223+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:16.230888+0000) 2019-10-13T09:04:18.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:18.499+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:18.500131+0000) 2019-10-13T09:04:19.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:19.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:19.484385+0000) 2019-10-13T09:04:20.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:20.747+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.753371+0000) 2019-10-13T09:04:20.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:20.747+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.753991+0000) 2019-10-13T09:04:20.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:20.753+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.754528+0000) 2019-10-13T09:04:20.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:20.747+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.755146+0000) 2019-10-13T09:04:20.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:20.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:20.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:20.751+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.755840+0000) 2019-10-13T09:04:20.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:20.751+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.756286+0000) 2019-10-13T09:04:20.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:20.751+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.756686+0000) 2019-10-13T09:04:20.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:20.757+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.759372+0000) 2019-10-13T09:04:21.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:21.701+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.703268+0000) 2019-10-13T09:04:21.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:21.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.743726+0000) 2019-10-13T09:04:21.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:21.783+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.789691+0000) 2019-10-13T09:04:21.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:21.979+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.982820+0000) 2019-10-13T09:04:25.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:25.007+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.010112+0000) 2019-10-13T09:04:25.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:25.755+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.758609+0000) 2019-10-13T09:04:25.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:25.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.759168+0000) 2019-10-13T09:04:25.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:25.755+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.759390+0000) 2019-10-13T09:04:25.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:25.757+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.760633+0000) 2019-10-13T09:04:25.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:25.755+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.760943+0000) 2019-10-13T09:04:25.762 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:25.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:25.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:25.755+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.761113+0000) 2019-10-13T09:04:25.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:25.755+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.762302+0000) 2019-10-13T09:04:25.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:25.765+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.766569+0000) 2019-10-13T09:04:26.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:26.227+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:26.231063+0000) 2019-10-13T09:04:28.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:28.499+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:28.500334+0000) 2019-10-13T09:04:29.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:29.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:29.484534+0000) 2019-10-13T09:04:30.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:30.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.764630+0000) 2019-10-13T09:04:30.766 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:30.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.765532+0000) 2019-10-13T09:04:30.766 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:30.763+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.765922+0000) 2019-10-13T09:04:30.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:30.763+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.766336+0000) 2019-10-13T09:04:30.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:30.763+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.766736+0000) 2019-10-13T09:04:30.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:30.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:30.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:30.767+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.767388+0000) 2019-10-13T09:04:30.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:30.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.769233+0000) 2019-10-13T09:04:30.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:30.769+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.771571+0000) 2019-10-13T09:04:31.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:31.705+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.709583+0000) 2019-10-13T09:04:31.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:31.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.743906+0000) 2019-10-13T09:04:31.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:31.787+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.789907+0000) 2019-10-13T09:04:31.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:31.979+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.983053+0000) 2019-10-13T09:04:35.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:35.007+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.010303+0000) 2019-10-13T09:04:35.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:35.767+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.770375+0000) 2019-10-13T09:04:35.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:35.767+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.770752+0000) 2019-10-13T09:04:35.772 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:35.771+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.771433+0000) 2019-10-13T09:04:35.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:35.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.772834+0000) 2019-10-13T09:04:35.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:35.771+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.773184+0000) 2019-10-13T09:04:35.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:35.769+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.773563+0000) 2019-10-13T09:04:35.774 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:35.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 66970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:35.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:35.771+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.774648+0000) 2019-10-13T09:04:35.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:35.777+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.778244+0000) 2019-10-13T09:04:36.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:36.231+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:36.231242+0000) 2019-10-13T09:04:37.395 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:04:37.395 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:37.399 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:04:37.399 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:38.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:38.499+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:38.500552+0000) 2019-10-13T09:04:39.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:39.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:39.484756+0000) 2019-10-13T09:04:40.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:40.775+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.776867+0000) 2019-10-13T09:04:40.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:40.775+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.777408+0000) 2019-10-13T09:04:40.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:40.775+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.777601+0000) 2019-10-13T09:04:40.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:40.775+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.777971+0000) 2019-10-13T09:04:40.779 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:40.775+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.778340+0000) 2019-10-13T09:04:40.780 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:40.777+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.779258+0000) 2019-10-13T09:04:40.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:40.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:40.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:40.779+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.780345+0000) 2019-10-13T09:04:40.788 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:40.785+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.787450+0000) 2019-10-13T09:04:41.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:41.713+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.716134+0000) 2019-10-13T09:04:41.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:41.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.744106+0000) 2019-10-13T09:04:41.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:41.787+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.790106+0000) 2019-10-13T09:04:41.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:41.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.983266+0000) 2019-10-13T09:04:45.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:45.006+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.010545+0000) 2019-10-13T09:04:45.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:45.778+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.782754+0000) 2019-10-13T09:04:45.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:45.782+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.784140+0000) 2019-10-13T09:04:45.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:45.782+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.784922+0000) 2019-10-13T09:04:45.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:45.782+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.785570+0000) 2019-10-13T09:04:45.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:45.782+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.785692+0000) 2019-10-13T09:04:45.787 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:45.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:45.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:45.782+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.786598+0000) 2019-10-13T09:04:45.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:45.785+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.786819+0000) 2019-10-13T09:04:45.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:45.789+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.793025+0000) 2019-10-13T09:04:46.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:46.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:46.231422+0000) 2019-10-13T09:04:48.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:48.498+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:48.500768+0000) 2019-10-13T09:04:49.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:49.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:49.484958+0000) 2019-10-13T09:04:50.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:50.786+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.789987+0000) 2019-10-13T09:04:50.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:50.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.790372+0000) 2019-10-13T09:04:50.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:50.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.791239+0000) 2019-10-13T09:04:50.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:50.790+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.791361+0000) 2019-10-13T09:04:50.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:50.790+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.791756+0000) 2019-10-13T09:04:50.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:50.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:50.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:50.789+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.792231+0000) 2019-10-13T09:04:50.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:50.790+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.792999+0000) 2019-10-13T09:04:50.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:50.793+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.795874+0000) 2019-10-13T09:04:51.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:51.721+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.722647+0000) 2019-10-13T09:04:51.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:51.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.744284+0000) 2019-10-13T09:04:51.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:51.786+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.790305+0000) 2019-10-13T09:04:51.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:51.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.983520+0000) 2019-10-13T09:04:55.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:55.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.010775+0000) 2019-10-13T09:04:55.797 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:55.794+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.795932+0000) 2019-10-13T09:04:55.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:04:55.794+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.795559+0000) 2019-10-13T09:04:55.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:04:55.794+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.796550+0000) 2019-10-13T09:04:55.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:55.794+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.796913+0000) 2019-10-13T09:04:55.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:04:55.794+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.797493+0000) 2019-10-13T09:04:55.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:04:55.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:04:55.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:55.797+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.799291+0000) 2019-10-13T09:04:55.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:04:55.794+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.798804+0000) 2019-10-13T09:04:55.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:04:55.801+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.805471+0000) 2019-10-13T09:04:56.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:04:56.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:56.231589+0000) 2019-10-13T09:04:58.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:04:58.498+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:58.500980+0000) 2019-10-13T09:04:59.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:04:59.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:59.485181+0000) 2019-10-13T09:05:00.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:00.798+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.800941+0000) 2019-10-13T09:05:00.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:00.798+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.801573+0000) 2019-10-13T09:05:00.803 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:00.797+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.801957+0000) 2019-10-13T09:05:00.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:00.798+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.802255+0000) 2019-10-13T09:05:00.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:00.798+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.802690+0000) 2019-10-13T09:05:00.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:00.802+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.803955+0000) 2019-10-13T09:05:00.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:00.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67138 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:00.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:00.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.804311+0000) 2019-10-13T09:05:00.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:00.809+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.811983+0000) 2019-10-13T09:05:01.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:01.725+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.729176+0000) 2019-10-13T09:05:01.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:01.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.744457+0000) 2019-10-13T09:05:01.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:01.786+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.790485+0000) 2019-10-13T09:05:01.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:01.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.983723+0000) 2019-10-13T09:05:05.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:05.006+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.010974+0000) 2019-10-13T09:05:05.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:05.806+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.808063+0000) 2019-10-13T09:05:05.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:05.805+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.808386+0000) 2019-10-13T09:05:05.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:05.806+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.808763+0000) 2019-10-13T09:05:05.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:05.806+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.809087+0000) 2019-10-13T09:05:05.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:05.806+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.809376+0000) 2019-10-13T09:05:05.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:05.806+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.809736+0000) 2019-10-13T09:05:05.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:05.806+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.810223+0000) 2019-10-13T09:05:05.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:05.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:05.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:05.817+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.820001+0000) 2019-10-13T09:05:06.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:06.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:06.231765+0000) 2019-10-13T09:05:07.511 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:05:07.511 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:07.515 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:05:07.515 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:08.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:08.498+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:08.501212+0000) 2019-10-13T09:05:09.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:09.481+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:09.485378+0000) 2019-10-13T09:05:10.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:10.809+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.813170+0000) 2019-10-13T09:05:10.814 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:10.810+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.814050+0000) 2019-10-13T09:05:10.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:10.810+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.814985+0000) 2019-10-13T09:05:10.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:10.814+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.815588+0000) 2019-10-13T09:05:10.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:10.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:10.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:10.814+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.815840+0000) 2019-10-13T09:05:10.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:10.814+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.816651+0000) 2019-10-13T09:05:10.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:10.814+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.817067+0000) 2019-10-13T09:05:10.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:10.821+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.822379+0000) 2019-10-13T09:05:11.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:11.732+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.735678+0000) 2019-10-13T09:05:11.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:11.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.744631+0000) 2019-10-13T09:05:11.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:11.786+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.790661+0000) 2019-10-13T09:05:11.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:11.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.983923+0000) 2019-10-13T09:05:15.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:15.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.011210+0000) 2019-10-13T09:05:15.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:15.814+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.818924+0000) 2019-10-13T09:05:15.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:15.816+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.819351+0000) 2019-10-13T09:05:15.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:15.818+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.819786+0000) 2019-10-13T09:05:15.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:15.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.820289+0000) 2019-10-13T09:05:15.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:15.818+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.821442+0000) 2019-10-13T09:05:15.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:15.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:15.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:15.818+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.821939+0000) 2019-10-13T09:05:15.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:15.818+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.823154+0000) 2019-10-13T09:05:15.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:15.824+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.829713+0000) 2019-10-13T09:05:16.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:16.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:16.231976+0000) 2019-10-13T09:05:18.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:18.498+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:18.501418+0000) 2019-10-13T09:05:19.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:19.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:19.485571+0000) 2019-10-13T09:05:20.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:20.822+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.825071+0000) 2019-10-13T09:05:20.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:20.822+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.825577+0000) 2019-10-13T09:05:20.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:20.822+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.825997+0000) 2019-10-13T09:05:20.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:20.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.826552+0000) 2019-10-13T09:05:20.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:20.822+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.827020+0000) 2019-10-13T09:05:20.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:20.824+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.827611+0000) 2019-10-13T09:05:20.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:20.826+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.828078+0000) 2019-10-13T09:05:20.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:20.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:20.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:20.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.835597+0000) 2019-10-13T09:05:21.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:21.736+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.741859+0000) 2019-10-13T09:05:21.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:21.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.744804+0000) 2019-10-13T09:05:21.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:21.786+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.790850+0000) 2019-10-13T09:05:21.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:21.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.984127+0000) 2019-10-13T09:05:25.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:25.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.011437+0000) 2019-10-13T09:05:25.833 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:25.829+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.831901+0000) 2019-10-13T09:05:25.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:25.829+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.832458+0000) 2019-10-13T09:05:25.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:25.829+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.833649+0000) 2019-10-13T09:05:25.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:25.832+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.834244+0000) 2019-10-13T09:05:25.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:25.829+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.834489+0000) 2019-10-13T09:05:25.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:25.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67306 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:25.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:25.833+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.835283+0000) 2019-10-13T09:05:25.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:25.833+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.835856+0000) 2019-10-13T09:05:25.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:25.836+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.839888+0000) 2019-10-13T09:05:26.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:26.229+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:26.232151+0000) 2019-10-13T09:05:28.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:28.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:28.501646+0000) 2019-10-13T09:05:29.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:29.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:29.485759+0000) 2019-10-13T09:05:30.838 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:30.832+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.837966+0000) 2019-10-13T09:05:30.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:30.833+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.838709+0000) 2019-10-13T09:05:30.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:30.833+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.839268+0000) 2019-10-13T09:05:30.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:30.837+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.839804+0000) 2019-10-13T09:05:30.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:30.837+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.840302+0000) 2019-10-13T09:05:30.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:30.837+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.840884+0000) 2019-10-13T09:05:30.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:30.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:30.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:30.837+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.841738+0000) 2019-10-13T09:05:30.852 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:30.848+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.852149+0000) 2019-10-13T09:05:31.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:31.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.744982+0000) 2019-10-13T09:05:31.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:31.744+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.748232+0000) 2019-10-13T09:05:31.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:31.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.791031+0000) 2019-10-13T09:05:31.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:31.981+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.984369+0000) 2019-10-13T09:05:35.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:35.009+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.011638+0000) 2019-10-13T09:05:35.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:35.841+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.844400+0000) 2019-10-13T09:05:35.846 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:35.841+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.845063+0000) 2019-10-13T09:05:35.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:35.841+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.844951+0000) 2019-10-13T09:05:35.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:35.841+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.845539+0000) 2019-10-13T09:05:35.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:35.841+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.845876+0000) 2019-10-13T09:05:35.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:35.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:35.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:35.844+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.847469+0000) 2019-10-13T09:05:35.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:35.845+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.848410+0000) 2019-10-13T09:05:35.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:35.852+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.854580+0000) 2019-10-13T09:05:36.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:36.229+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:36.232343+0000) 2019-10-13T09:05:37.617 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:05:37.618 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:37.621 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:05:37.621 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:38.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:38.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:38.501843+0000) 2019-10-13T09:05:39.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:39.480+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:39.485971+0000) 2019-10-13T09:05:40.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:40.845+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.850421+0000) 2019-10-13T09:05:40.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:40.849+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.852065+0000) 2019-10-13T09:05:40.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:40.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.852831+0000) 2019-10-13T09:05:40.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:40.849+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.853219+0000) 2019-10-13T09:05:40.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:40.849+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:40.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:40.849+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.853947+0000) 2019-10-13T09:05:40.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:40.848+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.854099+0000) 2019-10-13T09:05:40.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:40.849+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.854705+0000) 2019-10-13T09:05:40.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:40.852+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.857823+0000) 2019-10-13T09:05:41.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:41.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.745161+0000) 2019-10-13T09:05:41.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:41.752+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.754569+0000) 2019-10-13T09:05:41.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:41.785+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.791174+0000) 2019-10-13T09:05:41.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:41.981+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.984602+0000) 2019-10-13T09:05:45.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:45.009+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.011846+0000) 2019-10-13T09:05:45.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:45.853+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.856745+0000) 2019-10-13T09:05:45.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:45.853+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.857232+0000) 2019-10-13T09:05:45.859 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:45.853+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.858523+0000) 2019-10-13T09:05:45.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:45.853+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.858955+0000) 2019-10-13T09:05:45.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:45.856+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.859419+0000) 2019-10-13T09:05:45.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:45.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67440 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:45.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:45.857+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.860079+0000) 2019-10-13T09:05:45.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:45.857+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.860612+0000) 2019-10-13T09:05:45.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:45.864+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.869469+0000) 2019-10-13T09:05:46.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:46.229+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:46.232530+0000) 2019-10-13T09:05:48.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:48.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:48.502069+0000) 2019-10-13T09:05:49.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:49.484+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:49.486193+0000) 2019-10-13T09:05:50.863 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:50.857+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.862711+0000) 2019-10-13T09:05:50.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:50.861+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.863754+0000) 2019-10-13T09:05:50.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:50.861+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.864278+0000) 2019-10-13T09:05:50.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:50.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.864841+0000) 2019-10-13T09:05:50.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:50.861+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.865356+0000) 2019-10-13T09:05:50.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:50.861+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:50.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:50.860+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.865923+0000) 2019-10-13T09:05:50.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:50.861+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.866319+0000) 2019-10-13T09:05:50.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:50.868+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.871400+0000) 2019-10-13T09:05:51.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:51.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.745374+0000) 2019-10-13T09:05:51.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:51.760+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.764338+0000) 2019-10-13T09:05:51.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:51.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.791354+0000) 2019-10-13T09:05:51.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:51.981+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.984829+0000) 2019-10-13T09:05:55.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:55.009+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.012080+0000) 2019-10-13T09:05:55.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:55.864+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.869385+0000) 2019-10-13T09:05:55.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:05:55.865+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.869702+0000) 2019-10-13T09:05:55.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:05:55.865+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.870406+0000) 2019-10-13T09:05:55.871 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:05:55.865+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.870523+0000) 2019-10-13T09:05:55.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:55.865+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.870790+0000) 2019-10-13T09:05:55.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:05:55.865+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:05:55.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:55.869+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.872063+0000) 2019-10-13T09:05:55.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:05:55.869+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.872624+0000) 2019-10-13T09:05:55.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:05:55.876+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.878564+0000) 2019-10-13T09:05:56.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:05:56.229+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:56.232726+0000) 2019-10-13T09:05:58.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:05:58.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:58.502273+0000) 2019-10-13T09:05:59.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:05:59.484+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:59.486394+0000) 2019-10-13T09:06:00.875 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:00.869+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.874312+0000) 2019-10-13T09:06:00.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:00.869+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.874657+0000) 2019-10-13T09:06:00.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:00.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.876073+0000) 2019-10-13T09:06:00.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:00.873+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.876395+0000) 2019-10-13T09:06:00.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:00.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:00.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:00.872+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.877095+0000) 2019-10-13T09:06:00.878 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:00.873+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.877571+0000) 2019-10-13T09:06:00.880 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:00.873+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.878030+0000) 2019-10-13T09:06:00.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:00.880+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.883914+0000) 2019-10-13T09:06:01.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:01.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.745553+0000) 2019-10-13T09:06:01.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:01.768+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.770531+0000) 2019-10-13T09:06:01.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:01.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.791576+0000) 2019-10-13T09:06:01.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:01.981+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.985028+0000) 2019-10-13T09:06:05.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:05.009+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.012317+0000) 2019-10-13T09:06:05.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:05.877+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.880906+0000) 2019-10-13T09:06:05.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:05.877+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.881338+0000) 2019-10-13T09:06:05.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:05.877+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.882604+0000) 2019-10-13T09:06:05.883 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:05.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.882619+0000) 2019-10-13T09:06:05.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:05.877+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.882934+0000) 2019-10-13T09:06:05.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:05.880+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.883435+0000) 2019-10-13T09:06:05.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:05.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:05.885 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:05.881+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.884024+0000) 2019-10-13T09:06:05.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:05.884+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.886453+0000) 2019-10-13T09:06:06.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:06.229+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:06.232856+0000) 2019-10-13T09:06:07.721 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:06:07.722 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:07.726 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:06:07.727 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:08.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:08.497+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:08.502515+0000) 2019-10-13T09:06:09.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:09.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:09.486606+0000) 2019-10-13T09:06:10.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:10.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.886187+0000) 2019-10-13T09:06:10.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:10.883+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.886893+0000) 2019-10-13T09:06:10.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:10.884+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.887735+0000) 2019-10-13T09:06:10.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:10.884+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.888510+0000) 2019-10-13T09:06:10.890 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:10.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:10.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:10.884+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.888921+0000) 2019-10-13T09:06:10.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:10.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.889277+0000) 2019-10-13T09:06:10.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:10.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.889714+0000) 2019-10-13T09:06:10.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:10.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.893871+0000) 2019-10-13T09:06:11.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:11.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.745762+0000) 2019-10-13T09:06:11.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:11.775+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.776831+0000) 2019-10-13T09:06:11.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:11.788+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.791787+0000) 2019-10-13T09:06:11.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:11.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.985227+0000) 2019-10-13T09:06:15.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:15.008+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.012514+0000) 2019-10-13T09:06:15.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:15.888+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.892587+0000) 2019-10-13T09:06:15.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:15.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.892929+0000) 2019-10-13T09:06:15.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:15.888+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.893574+0000) 2019-10-13T09:06:15.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:15.888+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.894087+0000) 2019-10-13T09:06:15.895 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:15.888+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.894262+0000) 2019-10-13T09:06:15.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:15.888+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.894638+0000) 2019-10-13T09:06:15.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:15.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67640 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:15.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:15.892+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.895862+0000) 2019-10-13T09:06:15.902 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:15.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.901232+0000) 2019-10-13T09:06:16.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:16.228+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:16.232998+0000) 2019-10-13T09:06:18.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:18.496+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:18.502717+0000) 2019-10-13T09:06:19.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:19.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:19.486795+0000) 2019-10-13T09:06:20.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:20.892+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.898146+0000) 2019-10-13T09:06:20.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:20.895+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.898865+0000) 2019-10-13T09:06:20.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:20.896+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.899472+0000) 2019-10-13T09:06:20.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:20.896+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.899971+0000) 2019-10-13T09:06:20.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:20.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.900568+0000) 2019-10-13T09:06:20.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:20.896+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.900894+0000) 2019-10-13T09:06:20.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:20.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:20.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:20.896+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.901190+0000) 2019-10-13T09:06:20.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:20.903+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.907109+0000) 2019-10-13T09:06:21.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:21.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.745943+0000) 2019-10-13T09:06:21.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:21.779+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.783419+0000) 2019-10-13T09:06:21.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:21.788+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.791972+0000) 2019-10-13T09:06:21.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:21.980+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.985463+0000) 2019-10-13T09:06:25.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:25.008+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.012716+0000) 2019-10-13T09:06:25.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:25.899+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.903977+0000) 2019-10-13T09:06:25.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:25.900+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.905209+0000) 2019-10-13T09:06:25.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:25.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.905750+0000) 2019-10-13T09:06:25.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:25.900+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.906189+0000) 2019-10-13T09:06:25.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:25.900+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.906485+0000) 2019-10-13T09:06:25.908 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:25.900+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.907107+0000) 2019-10-13T09:06:25.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:25.900+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.906683+0000) 2019-10-13T09:06:25.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:25.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:25.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:25.907+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.911538+0000) 2019-10-13T09:06:26.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:26.232+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:26.233176+0000) 2019-10-13T09:06:28.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:28.496+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:28.502943+0000) 2019-10-13T09:06:29.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:29.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:29.487015+0000) 2019-10-13T09:06:30.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:30.908+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.911409+0000) 2019-10-13T09:06:30.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:30.908+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.912197+0000) 2019-10-13T09:06:30.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:30.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.912347+0000) 2019-10-13T09:06:30.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:30.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.912731+0000) 2019-10-13T09:06:30.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:30.908+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.913013+0000) 2019-10-13T09:06:30.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:30.908+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.913549+0000) 2019-10-13T09:06:30.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:30.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:30.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:30.908+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.914543+0000) 2019-10-13T09:06:30.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:30.915+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.919592+0000) 2019-10-13T09:06:31.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:31.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.746141+0000) 2019-10-13T09:06:31.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:31.788+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.792174+0000) 2019-10-13T09:06:31.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:31.791+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.794893+0000) 2019-10-13T09:06:31.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:31.984+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.985637+0000) 2019-10-13T09:06:35.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:35.008+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.012929+0000) 2019-10-13T09:06:35.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:35.912+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.917525+0000) 2019-10-13T09:06:35.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:35.912+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.918630+0000) 2019-10-13T09:06:35.919 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:35.912+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.918819+0000) 2019-10-13T09:06:35.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:35.916+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.919387+0000) 2019-10-13T09:06:35.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:35.915+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.920218+0000) 2019-10-13T09:06:35.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:35.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:35.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:35.916+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.920916+0000) 2019-10-13T09:06:35.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:35.916+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.920567+0000) 2019-10-13T09:06:35.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:35.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.930684+0000) 2019-10-13T09:06:36.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:36.232+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:36.233410+0000) 2019-10-13T09:06:37.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:06:37.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:37.853 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:06:37.853 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:38.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:38.496+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:38.503167+0000) 2019-10-13T09:06:39.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:39.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:39.487210+0000) 2019-10-13T09:06:40.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:40.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.923948+0000) 2019-10-13T09:06:40.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:40.920+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.924380+0000) 2019-10-13T09:06:40.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:40.920+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.924773+0000) 2019-10-13T09:06:40.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:40.920+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.925864+0000) 2019-10-13T09:06:40.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:40.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:40.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:40.920+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.926925+0000) 2019-10-13T09:06:40.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:40.920+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.927112+0000) 2019-10-13T09:06:40.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:40.924+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.927647+0000) 2019-10-13T09:06:40.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:40.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.932575+0000) 2019-10-13T09:06:41.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:41.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.746343+0000) 2019-10-13T09:06:41.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:41.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.792395+0000) 2019-10-13T09:06:41.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:41.799+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.801281+0000) 2019-10-13T09:06:41.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:41.984+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.985841+0000) 2019-10-13T09:06:45.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:45.012+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.013147+0000) 2019-10-13T09:06:45.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:45.924+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.930299+0000) 2019-10-13T09:06:45.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:45.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.930473+0000) 2019-10-13T09:06:45.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:45.928+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.931493+0000) 2019-10-13T09:06:45.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:45.927+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.931833+0000) 2019-10-13T09:06:45.933 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:45.928+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.932114+0000) 2019-10-13T09:06:45.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:45.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:45.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:45.928+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.932831+0000) 2019-10-13T09:06:45.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:45.928+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.933316+0000) 2019-10-13T09:06:45.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:45.931+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.935952+0000) 2019-10-13T09:06:46.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:46.232+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:46.233592+0000) 2019-10-13T09:06:48.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:48.500+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:48.503404+0000) 2019-10-13T09:06:49.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:49.483+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:49.487427+0000) 2019-10-13T09:06:50.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:50.932+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.935611+0000) 2019-10-13T09:06:50.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:50.932+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.936323+0000) 2019-10-13T09:06:50.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:50.932+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.937141+0000) 2019-10-13T09:06:50.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:50.932+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.937536+0000) 2019-10-13T09:06:50.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:50.932+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.937755+0000) 2019-10-13T09:06:50.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:50.931+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.938067+0000) 2019-10-13T09:06:50.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:50.932+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.938954+0000) 2019-10-13T09:06:50.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:50.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:50.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:50.939+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.945734+0000) 2019-10-13T09:06:51.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:51.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.746518+0000) 2019-10-13T09:06:51.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:51.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.792611+0000) 2019-10-13T09:06:51.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:51.811+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.812926+0000) 2019-10-13T09:06:51.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:51.984+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.986060+0000) 2019-10-13T09:06:55.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:55.011+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.013345+0000) 2019-10-13T09:06:55.944 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:55.935+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.943086+0000) 2019-10-13T09:06:55.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:06:55.939+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.943590+0000) 2019-10-13T09:06:55.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:55.939+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.943939+0000) 2019-10-13T09:06:55.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:06:55.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.944229+0000) 2019-10-13T09:06:55.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:06:55.939+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.944532+0000) 2019-10-13T09:06:55.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:55.939+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.945158+0000) 2019-10-13T09:06:55.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:06:55.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67904 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:06:55.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:06:55.939+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.946490+0000) 2019-10-13T09:06:55.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:06:55.947+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.952652+0000) 2019-10-13T09:06:56.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:06:56.231+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:56.233784+0000) 2019-10-13T09:06:58.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:06:58.499+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:58.503602+0000) 2019-10-13T09:06:59.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:06:59.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:59.487619+0000) 2019-10-13T09:07:00.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:00.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.948709+0000) 2019-10-13T09:07:00.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:00.943+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.949410+0000) 2019-10-13T09:07:00.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:00.947+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.950438+0000) 2019-10-13T09:07:00.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:00.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.950778+0000) 2019-10-13T09:07:00.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:00.943+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.951060+0000) 2019-10-13T09:07:00.954 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:00.947+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.951554+0000) 2019-10-13T09:07:00.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:00.947+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.951824+0000) 2019-10-13T09:07:00.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:00.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:00.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:00.955+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.960647+0000) 2019-10-13T09:07:01.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:01.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.746726+0000) 2019-10-13T09:07:01.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:01.791+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.792802+0000) 2019-10-13T09:07:01.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:01.818+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.819367+0000) 2019-10-13T09:07:01.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:01.983+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.986207+0000) 2019-10-13T09:07:05.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:05.007+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.013558+0000) 2019-10-13T09:07:05.956 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:05.951+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.956017+0000) 2019-10-13T09:07:05.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:05.951+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.956418+0000) 2019-10-13T09:07:05.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:05.951+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.957125+0000) 2019-10-13T09:07:05.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:05.950+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.957839+0000) 2019-10-13T09:07:05.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:05.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:05.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:05.951+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.957576+0000) 2019-10-13T09:07:05.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:05.951+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.958293+0000) 2019-10-13T09:07:05.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:05.951+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.958629+0000) 2019-10-13T09:07:05.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:05.954+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.961385+0000) 2019-10-13T09:07:06.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:06.231+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:06.233997+0000) 2019-10-13T09:07:08.095 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:07:08.096 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:08.099 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:07:08.099 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:08.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:08.499+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:08.503823+0000) 2019-10-13T09:07:09.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:09.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:09.487833+0000) 2019-10-13T09:07:10.962 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:10.955+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.961247+0000) 2019-10-13T09:07:10.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:10.955+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.962360+0000) 2019-10-13T09:07:10.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:10.955+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.963193+0000) 2019-10-13T09:07:10.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:10.959+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.963699+0000) 2019-10-13T09:07:10.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:10.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 67999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:10.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:10.958+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.964533+0000) 2019-10-13T09:07:10.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:10.959+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.964164+0000) 2019-10-13T09:07:10.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:10.959+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.965243+0000) 2019-10-13T09:07:10.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:10.962+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.968428+0000) 2019-10-13T09:07:11.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:11.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.746940+0000) 2019-10-13T09:07:11.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:11.791+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.793012+0000) 2019-10-13T09:07:11.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:11.818+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.825593+0000) 2019-10-13T09:07:11.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:11.983+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.986413+0000) 2019-10-13T09:07:15.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:15.011+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.013759+0000) 2019-10-13T09:07:15.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:15.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.967185+0000) 2019-10-13T09:07:15.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:15.963+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.967725+0000) 2019-10-13T09:07:15.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:15.963+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.968644+0000) 2019-10-13T09:07:15.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:15.963+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.969050+0000) 2019-10-13T09:07:15.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:15.963+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.969967+0000) 2019-10-13T09:07:15.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:15.963+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.970103+0000) 2019-10-13T09:07:15.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:15.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68034 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:15.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:15.966+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.971139+0000) 2019-10-13T09:07:15.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:15.970+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.976172+0000) 2019-10-13T09:07:16.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:16.231+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:16.234207+0000) 2019-10-13T09:07:18.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:18.503+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:18.504023+0000) 2019-10-13T09:07:19.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:19.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:19.488029+0000) 2019-10-13T09:07:20.975 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:20.967+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.973849+0000) 2019-10-13T09:07:20.975 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:20.967+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.974232+0000) 2019-10-13T09:07:20.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:20.967+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.974695+0000) 2019-10-13T09:07:20.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:20.971+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.975532+0000) 2019-10-13T09:07:20.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:20.971+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.976087+0000) 2019-10-13T09:07:20.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:20.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68065 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:20.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:20.971+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.976688+0000) 2019-10-13T09:07:20.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:20.970+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.977405+0000) 2019-10-13T09:07:20.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:20.978+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.982544+0000) 2019-10-13T09:07:21.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:21.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.747146+0000) 2019-10-13T09:07:21.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:21.791+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.793147+0000) 2019-10-13T09:07:21.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:21.826+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.828328+0000) 2019-10-13T09:07:21.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:21.983+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.986646+0000) 2019-10-13T09:07:25.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:25.011+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.013976+0000) 2019-10-13T09:07:25.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:25.979+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.979485+0000) 2019-10-13T09:07:25.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:25.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.980250+0000) 2019-10-13T09:07:25.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:25.979+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.980202+0000) 2019-10-13T09:07:25.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:25.979+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.980495+0000) 2019-10-13T09:07:25.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:25.979+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.980814+0000) 2019-10-13T09:07:25.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:25.979+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.982604+0000) 2019-10-13T09:07:25.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:25.983+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.983421+0000) 2019-10-13T09:07:25.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:25.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:25.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:25.982+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.988874+0000) 2019-10-13T09:07:26.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:26.231+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:26.234413+0000) 2019-10-13T09:07:28.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:28.503+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:28.504235+0000) 2019-10-13T09:07:29.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:29.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:29.488220+0000) 2019-10-13T09:07:30.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:30.983+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.986645+0000) 2019-10-13T09:07:30.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:30.987+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.987904+0000) 2019-10-13T09:07:30.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:30.987+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.988509+0000) 2019-10-13T09:07:30.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:30.987+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.989018+0000) 2019-10-13T09:07:30.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:30.987+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:30.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:30.987+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.989813+0000) 2019-10-13T09:07:30.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:30.986+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.990379+0000) 2019-10-13T09:07:30.991 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:30.987+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.990830+0000) 2019-10-13T09:07:30.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:30.990+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.996577+0000) 2019-10-13T09:07:31.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:31.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.747347+0000) 2019-10-13T09:07:31.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:31.791+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.793321+0000) 2019-10-13T09:07:31.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:31.834+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.834819+0000) 2019-10-13T09:07:31.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:31.983+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.986821+0000) 2019-10-13T09:07:35.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:35.011+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.014178+0000) 2019-10-13T09:07:35.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:35.990+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.992957+0000) 2019-10-13T09:07:35.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:35.990+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.993301+0000) 2019-10-13T09:07:35.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:35.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.994309+0000) 2019-10-13T09:07:35.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:35.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.994596+0000) 2019-10-13T09:07:35.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:35.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.994921+0000) 2019-10-13T09:07:35.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:35.994+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.995842+0000) 2019-10-13T09:07:35.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:35.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:35.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:35.994+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.996296+0000) 2019-10-13T09:07:36.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:35.994+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:36.000707+0000) 2019-10-13T09:07:36.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:36.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:36.234586+0000) 2019-10-13T09:07:38.188 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:07:38.188 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:38.192 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:07:38.192 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:38.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:38.502+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:38.504437+0000) 2019-10-13T09:07:39.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:39.482+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:39.488446+0000) 2019-10-13T09:07:41.000 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:40.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.999653+0000) 2019-10-13T09:07:41.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:40.998+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.000335+0000) 2019-10-13T09:07:41.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:40.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.000966+0000) 2019-10-13T09:07:41.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:40.994+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.001282+0000) 2019-10-13T09:07:41.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:40.998+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.001678+0000) 2019-10-13T09:07:41.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:40.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:41.004 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:40.998+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.002239+0000) 2019-10-13T09:07:41.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:40.998+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.002841+0000) 2019-10-13T09:07:41.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:40.998+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.006075+0000) 2019-10-13T09:07:41.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:41.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.747526+0000) 2019-10-13T09:07:41.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:41.790+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.793533+0000) 2019-10-13T09:07:41.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:41.838+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.841165+0000) 2019-10-13T09:07:41.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:41.986+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.986982+0000) 2019-10-13T09:07:45.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:45.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.014392+0000) 2019-10-13T09:07:46.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:46.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.005211+0000) 2019-10-13T09:07:46.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:46.002+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.006505+0000) 2019-10-13T09:07:46.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:46.002+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.006792+0000) 2019-10-13T09:07:46.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:46.002+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.007183+0000) 2019-10-13T09:07:46.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:46.002+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.007890+0000) 2019-10-13T09:07:46.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:46.006+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.007605+0000) 2019-10-13T09:07:46.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:46.006+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.008224+0000) 2019-10-13T09:07:46.010 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:46.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:46.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:46.006+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.012743+0000) 2019-10-13T09:07:46.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:46.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.234764+0000) 2019-10-13T09:07:48.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:48.502+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:48.504599+0000) 2019-10-13T09:07:49.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:49.486+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:49.488678+0000) 2019-10-13T09:07:51.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:51.010+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.012564+0000) 2019-10-13T09:07:51.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:51.010+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.012902+0000) 2019-10-13T09:07:51.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:51.010+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.013467+0000) 2019-10-13T09:07:51.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:51.010+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.013895+0000) 2019-10-13T09:07:51.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:51.010+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.014976+0000) 2019-10-13T09:07:51.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:51.010+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.015242+0000) 2019-10-13T09:07:51.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:51.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:51.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:51.014+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.015656+0000) 2019-10-13T09:07:51.023 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:51.022+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.022414+0000) 2019-10-13T09:07:51.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:51.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.747750+0000) 2019-10-13T09:07:51.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:51.790+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.793746+0000) 2019-10-13T09:07:51.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:51.850+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.852950+0000) 2019-10-13T09:07:51.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:51.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.987139+0000) 2019-10-13T09:07:55.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:55.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.014592+0000) 2019-10-13T09:07:56.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:07:56.014+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.017873+0000) 2019-10-13T09:07:56.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:56.014+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.019023+0000) 2019-10-13T09:07:56.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:07:56.018+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.019485+0000) 2019-10-13T09:07:56.020 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:07:56.018+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.020226+0000) 2019-10-13T09:07:56.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:56.018+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.020351+0000) 2019-10-13T09:07:56.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:56.018+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.021424+0000) 2019-10-13T09:07:56.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:07:56.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:07:56.022 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:07:56.018+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.021923+0000) 2019-10-13T09:07:56.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:07:56.026+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.026627+0000) 2019-10-13T09:07:56.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:07:56.230+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.234942+0000) 2019-10-13T09:07:58.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:07:58.502+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:58.504750+0000) 2019-10-13T09:07:59.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:07:59.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:59.488883+0000) 2019-10-13T09:08:01.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:01.022+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.024895+0000) 2019-10-13T09:08:01.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:01.022+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.025446+0000) 2019-10-13T09:08:01.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:01.021+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.025774+0000) 2019-10-13T09:08:01.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:01.022+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.026356+0000) 2019-10-13T09:08:01.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:01.022+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.026748+0000) 2019-10-13T09:08:01.028 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:01.026+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.027376+0000) 2019-10-13T09:08:01.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:01.026+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.027809+0000) 2019-10-13T09:08:01.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:01.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:01.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:01.029+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.031407+0000) 2019-10-13T09:08:01.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:01.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.747934+0000) 2019-10-13T09:08:01.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:01.790+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.793921+0000) 2019-10-13T09:08:01.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:01.857+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.859394+0000) 2019-10-13T09:08:01.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:01.982+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.987278+0000) 2019-10-13T09:08:05.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:05.010+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.014794+0000) 2019-10-13T09:08:06.032 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:06.026+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.031161+0000) 2019-10-13T09:08:06.033 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:06.030+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.032324+0000) 2019-10-13T09:08:06.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:06.030+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.032421+0000) 2019-10-13T09:08:06.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:06.029+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.033040+0000) 2019-10-13T09:08:06.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:06.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:06.034 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:06.030+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.033995+0000) 2019-10-13T09:08:06.035 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:06.030+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.034529+0000) 2019-10-13T09:08:06.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:06.030+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.035028+0000) 2019-10-13T09:08:06.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:06.033+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.038016+0000) 2019-10-13T09:08:06.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:06.234+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.235119+0000) 2019-10-13T09:08:08.342 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:08:08.342 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:08.346 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:08:08.346 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:08.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:08.502+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:08.504967+0000) 2019-10-13T09:08:09.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:09.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:09.489077+0000) 2019-10-13T09:08:11.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:11.034+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.036943+0000) 2019-10-13T09:08:11.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:11.034+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.036734+0000) 2019-10-13T09:08:11.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:11.033+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.037397+0000) 2019-10-13T09:08:11.039 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:11.034+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.038288+0000) 2019-10-13T09:08:11.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:11.034+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.038416+0000) 2019-10-13T09:08:11.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:11.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.039738+0000) 2019-10-13T09:08:11.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:11.038+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.040491+0000) 2019-10-13T09:08:11.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:11.038+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:11.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:11.045+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.047694+0000) 2019-10-13T09:08:11.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:11.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.748116+0000) 2019-10-13T09:08:11.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:11.790+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.794101+0000) 2019-10-13T09:08:11.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:11.865+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.865827+0000) 2019-10-13T09:08:11.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:11.986+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.987505+0000) 2019-10-13T09:08:15.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:15.014+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.014991+0000) 2019-10-13T09:08:16.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:16.041+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.043934+0000) 2019-10-13T09:08:16.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:16.042+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.044641+0000) 2019-10-13T09:08:16.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:16.042+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.045893+0000) 2019-10-13T09:08:16.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:16.042+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.045944+0000) 2019-10-13T09:08:16.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:16.042+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.046372+0000) 2019-10-13T09:08:16.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:16.042+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.046923+0000) 2019-10-13T09:08:16.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:16.046+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.047397+0000) 2019-10-13T09:08:16.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:16.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:16.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:16.049+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.053901+0000) 2019-10-13T09:08:16.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:16.234+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.235285+0000) 2019-10-13T09:08:18.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:18.502+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:18.505186+0000) 2019-10-13T09:08:19.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:19.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:19.489260+0000) 2019-10-13T09:08:21.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:21.045+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.050849+0000) 2019-10-13T09:08:21.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:21.049+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.051511+0000) 2019-10-13T09:08:21.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:21.049+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.051982+0000) 2019-10-13T09:08:21.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:21.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.052237+0000) 2019-10-13T09:08:21.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:21.049+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.053125+0000) 2019-10-13T09:08:21.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:21.049+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.053401+0000) 2019-10-13T09:08:21.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:21.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:21.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:21.049+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.054650+0000) 2019-10-13T09:08:21.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:21.061+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.062712+0000) 2019-10-13T09:08:21.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:21.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.748302+0000) 2019-10-13T09:08:21.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:21.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.794315+0000) 2019-10-13T09:08:21.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:21.869+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.872128+0000) 2019-10-13T09:08:21.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:21.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.987717+0000) 2019-10-13T09:08:25.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:25.013+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.015202+0000) 2019-10-13T09:08:26.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:26.053+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.057396+0000) 2019-10-13T09:08:26.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:26.053+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.058079+0000) 2019-10-13T09:08:26.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:26.053+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.058178+0000) 2019-10-13T09:08:26.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:26.057+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.059094+0000) 2019-10-13T09:08:26.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:26.057+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.059442+0000) 2019-10-13T09:08:26.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:26.057+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.059952+0000) 2019-10-13T09:08:26.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:26.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:26.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:26.057+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.061039+0000) 2019-10-13T09:08:26.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:26.065+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.066975+0000) 2019-10-13T09:08:26.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:26.233+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.235500+0000) 2019-10-13T09:08:28.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:28.501+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:28.505391+0000) 2019-10-13T09:08:29.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:29.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:29.489442+0000) 2019-10-13T09:08:31.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:31.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.064515+0000) 2019-10-13T09:08:31.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:31.061+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.065141+0000) 2019-10-13T09:08:31.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:31.061+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.065694+0000) 2019-10-13T09:08:31.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:31.061+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.065898+0000) 2019-10-13T09:08:31.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:31.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.066255+0000) 2019-10-13T09:08:31.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:31.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:31.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:31.065+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.066960+0000) 2019-10-13T09:08:31.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:31.065+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.067810+0000) 2019-10-13T09:08:31.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:31.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.073924+0000) 2019-10-13T09:08:31.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:31.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.748527+0000) 2019-10-13T09:08:31.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:31.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.794496+0000) 2019-10-13T09:08:31.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:31.877+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.878527+0000) 2019-10-13T09:08:31.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:31.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.987895+0000) 2019-10-13T09:08:35.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:35.013+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.015402+0000) 2019-10-13T09:08:36.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:36.065+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.070557+0000) 2019-10-13T09:08:36.072 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:36.065+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.070618+0000) 2019-10-13T09:08:36.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:36.065+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.070921+0000) 2019-10-13T09:08:36.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:36.069+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.071238+0000) 2019-10-13T09:08:36.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:36.069+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.072173+0000) 2019-10-13T09:08:36.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:36.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:36.074 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:36.069+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.072843+0000) 2019-10-13T09:08:36.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:36.069+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.073477+0000) 2019-10-13T09:08:36.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:36.077+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.081914+0000) 2019-10-13T09:08:36.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:36.233+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.235677+0000) 2019-10-13T09:08:38.439 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:08:38.440 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:38.443 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:08:38.443 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:38.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:38.501+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:38.505588+0000) 2019-10-13T09:08:39.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:39.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:39.489629+0000) 2019-10-13T09:08:41.076 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:41.073+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.075730+0000) 2019-10-13T09:08:41.077 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:41.073+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.076643+0000) 2019-10-13T09:08:41.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:41.073+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.077070+0000) 2019-10-13T09:08:41.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:41.073+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.077582+0000) 2019-10-13T09:08:41.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:41.073+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.077923+0000) 2019-10-13T09:08:41.079 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:41.073+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.078232+0000) 2019-10-13T09:08:41.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:41.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:41.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:41.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.078599+0000) 2019-10-13T09:08:41.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:41.081+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.083618+0000) 2019-10-13T09:08:41.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:41.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.748767+0000) 2019-10-13T09:08:41.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:41.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.794719+0000) 2019-10-13T09:08:41.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:41.881+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.884767+0000) 2019-10-13T09:08:41.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:41.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.988089+0000) 2019-10-13T09:08:45.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:45.013+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.015636+0000) 2019-10-13T09:08:46.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:46.077+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.081365+0000) 2019-10-13T09:08:46.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:46.077+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.081748+0000) 2019-10-13T09:08:46.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:46.077+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.082089+0000) 2019-10-13T09:08:46.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:46.077+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.083152+0000) 2019-10-13T09:08:46.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:46.077+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.082815+0000) 2019-10-13T09:08:46.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:46.081+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.084348+0000) 2019-10-13T09:08:46.086 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:46.081+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.084963+0000) 2019-10-13T09:08:46.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:46.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:46.093 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:46.089+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.091833+0000) 2019-10-13T09:08:46.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:46.233+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.235851+0000) 2019-10-13T09:08:48.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:48.501+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:48.505795+0000) 2019-10-13T09:08:49.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:49.485+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:49.489823+0000) 2019-10-13T09:08:51.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:51.085+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.088623+0000) 2019-10-13T09:08:51.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:51.085+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.089069+0000) 2019-10-13T09:08:51.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:51.089+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.090401+0000) 2019-10-13T09:08:51.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:51.085+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.090670+0000) 2019-10-13T09:08:51.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:51.085+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.091031+0000) 2019-10-13T09:08:51.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:51.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:51.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:51.089+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.091603+0000) 2019-10-13T09:08:51.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:51.089+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.093192+0000) 2019-10-13T09:08:51.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:51.093+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.095801+0000) 2019-10-13T09:08:51.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:51.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.748943+0000) 2019-10-13T09:08:51.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:51.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.794897+0000) 2019-10-13T09:08:51.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:51.888+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.890930+0000) 2019-10-13T09:08:51.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:51.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.988289+0000) 2019-10-13T09:08:55.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:55.013+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.015834+0000) 2019-10-13T09:08:56.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:56.089+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.094197+0000) 2019-10-13T09:08:56.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:08:56.089+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.094733+0000) 2019-10-13T09:08:56.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:08:56.093+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.095605+0000) 2019-10-13T09:08:56.097 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:08:56.093+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.096896+0000) 2019-10-13T09:08:56.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:56.092+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.097168+0000) 2019-10-13T09:08:56.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:56.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.097575+0000) 2019-10-13T09:08:56.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:08:56.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68682 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:08:56.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:08:56.093+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.098032+0000) 2019-10-13T09:08:56.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:08:56.096+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.102116+0000) 2019-10-13T09:08:56.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:08:56.233+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.236027+0000) 2019-10-13T09:08:58.507 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:08:58.501+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:58.506001+0000) 2019-10-13T09:08:59.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:08:59.484+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:59.490042+0000) 2019-10-13T09:09:01.102 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:01.097+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.101227+0000) 2019-10-13T09:09:01.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:01.097+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.101921+0000) 2019-10-13T09:09:01.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:01.100+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.102244+0000) 2019-10-13T09:09:01.103 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:01.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.102932+0000) 2019-10-13T09:09:01.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:01.101+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.103579+0000) 2019-10-13T09:09:01.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:01.101+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.103807+0000) 2019-10-13T09:09:01.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:01.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:01.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:01.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.104318+0000) 2019-10-13T09:09:01.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:01.108+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.112226+0000) 2019-10-13T09:09:01.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:01.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.749126+0000) 2019-10-13T09:09:01.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:01.789+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.795046+0000) 2019-10-13T09:09:01.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:01.896+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.897470+0000) 2019-10-13T09:09:01.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:01.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.988461+0000) 2019-10-13T09:09:05.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:05.012+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.016058+0000) 2019-10-13T09:09:06.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:06.100+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.106660+0000) 2019-10-13T09:09:06.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:06.104+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.106987+0000) 2019-10-13T09:09:06.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:06.104+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.108518+0000) 2019-10-13T09:09:06.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:06.104+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.109142+0000) 2019-10-13T09:09:06.111 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:06.104+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.109416+0000) 2019-10-13T09:09:06.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:06.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68747 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:06.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:06.104+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.109929+0000) 2019-10-13T09:09:06.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:06.104+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.110527+0000) 2019-10-13T09:09:06.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:06.108+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.114123+0000) 2019-10-13T09:09:06.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:06.232+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.236199+0000) 2019-10-13T09:09:08.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:08.500+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:08.506211+0000) 2019-10-13T09:09:08.555 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:09:08.555 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:08.559 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:09:08.559 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:09.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:09.488+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:09.490251+0000) 2019-10-13T09:09:11.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:11.108+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.113362+0000) 2019-10-13T09:09:11.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:11.108+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.113829+0000) 2019-10-13T09:09:11.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:11.108+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.114433+0000) 2019-10-13T09:09:11.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:11.108+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.114530+0000) 2019-10-13T09:09:11.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:11.112+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.115494+0000) 2019-10-13T09:09:11.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:11.112+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.116020+0000) 2019-10-13T09:09:11.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:11.112+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:11.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:11.112+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.116530+0000) 2019-10-13T09:09:11.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:11.120+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.123886+0000) 2019-10-13T09:09:11.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:11.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.749336+0000) 2019-10-13T09:09:11.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:11.788+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.795229+0000) 2019-10-13T09:09:11.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:11.904+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.908392+0000) 2019-10-13T09:09:11.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:11.984+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.988651+0000) 2019-10-13T09:09:15.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:15.012+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.016258+0000) 2019-10-13T09:09:16.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:16.116+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.119474+0000) 2019-10-13T09:09:16.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:16.116+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.119828+0000) 2019-10-13T09:09:16.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:16.116+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.120550+0000) 2019-10-13T09:09:16.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:16.116+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.121151+0000) 2019-10-13T09:09:16.123 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:16.116+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.122071+0000) 2019-10-13T09:09:16.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:16.116+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.122470+0000) 2019-10-13T09:09:16.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:16.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:16.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:16.120+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.122963+0000) 2019-10-13T09:09:16.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:16.124+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.127731+0000) 2019-10-13T09:09:16.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:16.232+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.236373+0000) 2019-10-13T09:09:18.507 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:18.500+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:18.506455+0000) 2019-10-13T09:09:19.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:19.488+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:19.490445+0000) 2019-10-13T09:09:21.127 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:21.120+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.126092+0000) 2019-10-13T09:09:21.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:21.120+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.127043+0000) 2019-10-13T09:09:21.128 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:21.124+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.127512+0000) 2019-10-13T09:09:21.129 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:21.124+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.128142+0000) 2019-10-13T09:09:21.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:21.124+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.128366+0000) 2019-10-13T09:09:21.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:21.124+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.128866+0000) 2019-10-13T09:09:21.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:21.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:21.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:21.124+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.130127+0000) 2019-10-13T09:09:21.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:21.136+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.138179+0000) 2019-10-13T09:09:21.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:21.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.749550+0000) 2019-10-13T09:09:21.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:21.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.795450+0000) 2019-10-13T09:09:21.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:21.912+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.914783+0000) 2019-10-13T09:09:21.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:21.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.988823+0000) 2019-10-13T09:09:25.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:25.012+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.016465+0000) 2019-10-13T09:09:26.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:26.128+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.131966+0000) 2019-10-13T09:09:26.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:26.128+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.132786+0000) 2019-10-13T09:09:26.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:26.128+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.133719+0000) 2019-10-13T09:09:26.134 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:26.128+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.133988+0000) 2019-10-13T09:09:26.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:26.128+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.134369+0000) 2019-10-13T09:09:26.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:26.128+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.134929+0000) 2019-10-13T09:09:26.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:26.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:26.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:26.132+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.135714+0000) 2019-10-13T09:09:26.140 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:26.136+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.139279+0000) 2019-10-13T09:09:26.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:26.236+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.236611+0000) 2019-10-13T09:09:28.507 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:28.504+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:28.506684+0000) 2019-10-13T09:09:29.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:29.488+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:29.490596+0000) 2019-10-13T09:09:31.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:31.132+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.137898+0000) 2019-10-13T09:09:31.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:31.132+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.138466+0000) 2019-10-13T09:09:31.140 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:31.132+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.139040+0000) 2019-10-13T09:09:31.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:31.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.139454+0000) 2019-10-13T09:09:31.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:31.136+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.140797+0000) 2019-10-13T09:09:31.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:31.136+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.141035+0000) 2019-10-13T09:09:31.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:31.136+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.141591+0000) 2019-10-13T09:09:31.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:31.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:31.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:31.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.147559+0000) 2019-10-13T09:09:31.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:31.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.749799+0000) 2019-10-13T09:09:31.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:31.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.795660+0000) 2019-10-13T09:09:31.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:31.920+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.921017+0000) 2019-10-13T09:09:31.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:31.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.989025+0000) 2019-10-13T09:09:35.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:35.012+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.016682+0000) 2019-10-13T09:09:36.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:36.140+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.145455+0000) 2019-10-13T09:09:36.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:36.140+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.146280+0000) 2019-10-13T09:09:36.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:36.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.146561+0000) 2019-10-13T09:09:36.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:36.140+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.147273+0000) 2019-10-13T09:09:36.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:36.144+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.147673+0000) 2019-10-13T09:09:36.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:36.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:36.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:36.144+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.148301+0000) 2019-10-13T09:09:36.151 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:36.144+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.148732+0000) 2019-10-13T09:09:36.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:36.152+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.154910+0000) 2019-10-13T09:09:36.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:36.236+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.236850+0000) 2019-10-13T09:09:38.507 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:38.500+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:38.506894+0000) 2019-10-13T09:09:38.603 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:09:38.603 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:38.609 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:09:38.609 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:39.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:39.488+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:39.490790+0000) 2019-10-13T09:09:41.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:41.148+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.151387+0000) 2019-10-13T09:09:41.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:41.148+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.152184+0000) 2019-10-13T09:09:41.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:41.148+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.153364+0000) 2019-10-13T09:09:41.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:41.148+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.153895+0000) 2019-10-13T09:09:41.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:41.148+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.154309+0000) 2019-10-13T09:09:41.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:41.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 68973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:41.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:41.152+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.154782+0000) 2019-10-13T09:09:41.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:41.148+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.155181+0000) 2019-10-13T09:09:41.160 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:41.156+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.159355+0000) 2019-10-13T09:09:41.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:41.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.749929+0000) 2019-10-13T09:09:41.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:41.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.795852+0000) 2019-10-13T09:09:41.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:41.928+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.931864+0000) 2019-10-13T09:09:41.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:41.984+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.989228+0000) 2019-10-13T09:09:45.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:45.016+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.016885+0000) 2019-10-13T09:09:46.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:46.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.160550+0000) 2019-10-13T09:09:46.166 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:46.151+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.157041+0000) 2019-10-13T09:09:46.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:46.151+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.157742+0000) 2019-10-13T09:09:46.166 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:46.151+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.158159+0000) 2019-10-13T09:09:46.166 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:46.155+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.159667+0000) 2019-10-13T09:09:46.167 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:46.155+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.160225+0000) 2019-10-13T09:09:46.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:46.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:46.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:46.155+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.160854+0000) 2019-10-13T09:09:46.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:46.159+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.165158+0000) 2019-10-13T09:09:46.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:46.235+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.237022+0000) 2019-10-13T09:09:48.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:48.503+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:48.507118+0000) 2019-10-13T09:09:49.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:49.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:49.491002+0000) 2019-10-13T09:09:51.165 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:51.159+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.164568+0000) 2019-10-13T09:09:51.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:51.159+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.165421+0000) 2019-10-13T09:09:51.166 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:51.159+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.166043+0000) 2019-10-13T09:09:51.167 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:51.159+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.166235+0000) 2019-10-13T09:09:51.167 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:51.159+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.166610+0000) 2019-10-13T09:09:51.167 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:51.159+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.167028+0000) 2019-10-13T09:09:51.168 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:51.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.167671+0000) 2019-10-13T09:09:51.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:51.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:51.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:51.167+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.172624+0000) 2019-10-13T09:09:51.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:51.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.750097+0000) 2019-10-13T09:09:51.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:51.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.796040+0000) 2019-10-13T09:09:51.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:51.935+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.938601+0000) 2019-10-13T09:09:51.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:51.983+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.989401+0000) 2019-10-13T09:09:55.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:55.015+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.017125+0000) 2019-10-13T09:09:56.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:09:56.163+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.170884+0000) 2019-10-13T09:09:56.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:09:56.167+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.171658+0000) 2019-10-13T09:09:56.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:09:56.167+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.171763+0000) 2019-10-13T09:09:56.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:09:56.167+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.172175+0000) 2019-10-13T09:09:56.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:56.167+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.173239+0000) 2019-10-13T09:09:56.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:09:56.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:09:56.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:56.167+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.174010+0000) 2019-10-13T09:09:56.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:56.167+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.174577+0000) 2019-10-13T09:09:56.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:09:56.175+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.178840+0000) 2019-10-13T09:09:56.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:09:56.235+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.237199+0000) 2019-10-13T09:09:58.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:09:58.503+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:58.507348+0000) 2019-10-13T09:09:59.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:09:59.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:59.491196+0000) 2019-10-13T09:10:01.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:01.171+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.176482+0000) 2019-10-13T09:10:01.178 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:01.171+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.177486+0000) 2019-10-13T09:10:01.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:01.171+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.177447+0000) 2019-10-13T09:10:01.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:01.171+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.179059+0000) 2019-10-13T09:10:01.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:01.175+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.179462+0000) 2019-10-13T09:10:01.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:01.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:01.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:01.175+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.180391+0000) 2019-10-13T09:10:01.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:01.175+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.180988+0000) 2019-10-13T09:10:01.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:01.179+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.184618+0000) 2019-10-13T09:10:01.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:01.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.750325+0000) 2019-10-13T09:10:01.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:01.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.796227+0000) 2019-10-13T09:10:01.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:01.943+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.945147+0000) 2019-10-13T09:10:01.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:01.987+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.989600+0000) 2019-10-13T09:10:05.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:05.015+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.017356+0000) 2019-10-13T09:10:06.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:06.179+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.183203+0000) 2019-10-13T09:10:06.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:06.179+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.184274+0000) 2019-10-13T09:10:06.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:06.179+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.184548+0000) 2019-10-13T09:10:06.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:06.179+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.185013+0000) 2019-10-13T09:10:06.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:06.179+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.185516+0000) 2019-10-13T09:10:06.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:06.179+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.186095+0000) 2019-10-13T09:10:06.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:06.179+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.186367+0000) 2019-10-13T09:10:06.190 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:06.179+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:06.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:06.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.194235+0000) 2019-10-13T09:10:06.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:06.235+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.237393+0000) 2019-10-13T09:10:08.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:08.507+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:08.507567+0000) 2019-10-13T09:10:08.733 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:10:08.734 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:08.738 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:10:08.738 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:09.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:09.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:09.491404+0000) 2019-10-13T09:10:11.190 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:11.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.189439+0000) 2019-10-13T09:10:11.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:11.183+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.190198+0000) 2019-10-13T09:10:11.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:11.183+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.190566+0000) 2019-10-13T09:10:11.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:11.183+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.190937+0000) 2019-10-13T09:10:11.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:11.187+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.191458+0000) 2019-10-13T09:10:11.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:11.187+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:11.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:11.187+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.193219+0000) 2019-10-13T09:10:11.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:11.187+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.193635+0000) 2019-10-13T09:10:11.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:11.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.199334+0000) 2019-10-13T09:10:11.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:11.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.750505+0000) 2019-10-13T09:10:11.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:11.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.796411+0000) 2019-10-13T09:10:11.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:11.947+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.951597+0000) 2019-10-13T09:10:11.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:11.987+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.989824+0000) 2019-10-13T09:10:15.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:15.015+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.017588+0000) 2019-10-13T09:10:16.197 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:16.195+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.196278+0000) 2019-10-13T09:10:16.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:16.195+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.196246+0000) 2019-10-13T09:10:16.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:16.195+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.196968+0000) 2019-10-13T09:10:16.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:16.191+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.198008+0000) 2019-10-13T09:10:16.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:16.195+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.198318+0000) 2019-10-13T09:10:16.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:16.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69197 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:16.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:16.195+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.199263+0000) 2019-10-13T09:10:16.200 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:16.199+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.199781+0000) 2019-10-13T09:10:16.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:16.199+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.204520+0000) 2019-10-13T09:10:16.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:16.235+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.237573+0000) 2019-10-13T09:10:18.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:18.507+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:18.507786+0000) 2019-10-13T09:10:19.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:19.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:19.491606+0000) 2019-10-13T09:10:21.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:21.199+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.201890+0000) 2019-10-13T09:10:21.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:21.199+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.203033+0000) 2019-10-13T09:10:21.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:21.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.203819+0000) 2019-10-13T09:10:21.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:21.203+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.204166+0000) 2019-10-13T09:10:21.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:21.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:21.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:21.203+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.204418+0000) 2019-10-13T09:10:21.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:21.203+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.205243+0000) 2019-10-13T09:10:21.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:21.203+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.207059+0000) 2019-10-13T09:10:21.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:21.203+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.209917+0000) 2019-10-13T09:10:21.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:21.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.750685+0000) 2019-10-13T09:10:21.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:21.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.796592+0000) 2019-10-13T09:10:21.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:21.955+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.958086+0000) 2019-10-13T09:10:21.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:21.987+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.989998+0000) 2019-10-13T09:10:25.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:25.015+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.017845+0000) 2019-10-13T09:10:26.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:26.207+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.208555+0000) 2019-10-13T09:10:26.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:26.207+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.208894+0000) 2019-10-13T09:10:26.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:26.207+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.209506+0000) 2019-10-13T09:10:26.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:26.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.209829+0000) 2019-10-13T09:10:26.211 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:26.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.210226+0000) 2019-10-13T09:10:26.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:26.207+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.210648+0000) 2019-10-13T09:10:26.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:26.207+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69261 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:26.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:26.207+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.210945+0000) 2019-10-13T09:10:26.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:26.211+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.215657+0000) 2019-10-13T09:10:26.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:26.235+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.237749+0000) 2019-10-13T09:10:28.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:28.507+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:28.507997+0000) 2019-10-13T09:10:29.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:29.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:29.491832+0000) 2019-10-13T09:10:31.214 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:31.207+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.213605+0000) 2019-10-13T09:10:31.215 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:31.210+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.214467+0000) 2019-10-13T09:10:31.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:31.210+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.214997+0000) 2019-10-13T09:10:31.216 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:31.214+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.215628+0000) 2019-10-13T09:10:31.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:31.214+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.215913+0000) 2019-10-13T09:10:31.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:31.214+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.216552+0000) 2019-10-13T09:10:31.218 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:31.214+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.217028+0000) 2019-10-13T09:10:31.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:31.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:31.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:31.219+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.224858+0000) 2019-10-13T09:10:31.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:31.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.750895+0000) 2019-10-13T09:10:31.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:31.794+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.796806+0000) 2019-10-13T09:10:31.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:31.967+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.969845+0000) 2019-10-13T09:10:31.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:31.986+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.990186+0000) 2019-10-13T09:10:35.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:35.014+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.018062+0000) 2019-10-13T09:10:36.221 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:36.218+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.220699+0000) 2019-10-13T09:10:36.222 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:36.218+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.221606+0000) 2019-10-13T09:10:36.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:36.218+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.222334+0000) 2019-10-13T09:10:36.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:36.219+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.222654+0000) 2019-10-13T09:10:36.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:36.222+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.223339+0000) 2019-10-13T09:10:36.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:36.222+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.223612+0000) 2019-10-13T09:10:36.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:36.222+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:36.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:36.222+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.223994+0000) 2019-10-13T09:10:36.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:36.223+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.227604+0000) 2019-10-13T09:10:36.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:36.234+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.237926+0000) 2019-10-13T09:10:38.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:38.506+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:38.508223+0000) 2019-10-13T09:10:38.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:10:38.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:38.851 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:10:38.852 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:39.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:39.487+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:39.492028+0000) 2019-10-13T09:10:41.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:41.222+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.226574+0000) 2019-10-13T09:10:41.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:41.226+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.227823+0000) 2019-10-13T09:10:41.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:41.226+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.228262+0000) 2019-10-13T09:10:41.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:41.226+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.229209+0000) 2019-10-13T09:10:41.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:41.226+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.229459+0000) 2019-10-13T09:10:41.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:41.226+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.230050+0000) 2019-10-13T09:10:41.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:41.226+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:41.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:41.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.230414+0000) 2019-10-13T09:10:41.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:41.230+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.234772+0000) 2019-10-13T09:10:41.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:41.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.751044+0000) 2019-10-13T09:10:41.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:41.794+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.796994+0000) 2019-10-13T09:10:41.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:41.974+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.976002+0000) 2019-10-13T09:10:41.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:41.986+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.990361+0000) 2019-10-13T09:10:45.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:45.014+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.018267+0000) 2019-10-13T09:10:46.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:46.230+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.233439+0000) 2019-10-13T09:10:46.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:46.230+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.234127+0000) 2019-10-13T09:10:46.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:46.230+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.234354+0000) 2019-10-13T09:10:46.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:46.230+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.234794+0000) 2019-10-13T09:10:46.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:46.230+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.235237+0000) 2019-10-13T09:10:46.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:46.234+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:46.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:46.234+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.235646+0000) 2019-10-13T09:10:46.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:46.230+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.237200+0000) 2019-10-13T09:10:46.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:46.234+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.238061+0000) 2019-10-13T09:10:46.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:46.238+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.242906+0000) 2019-10-13T09:10:48.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:48.506+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:48.508435+0000) 2019-10-13T09:10:49.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:49.486+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:49.492222+0000) 2019-10-13T09:10:51.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:51.234+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.238996+0000) 2019-10-13T09:10:51.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:51.238+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.239584+0000) 2019-10-13T09:10:51.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:51.238+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.240123+0000) 2019-10-13T09:10:51.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:51.238+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.240528+0000) 2019-10-13T09:10:51.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:51.238+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.241534+0000) 2019-10-13T09:10:51.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:51.238+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.241528+0000) 2019-10-13T09:10:51.243 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:51.238+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.242475+0000) 2019-10-13T09:10:51.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:51.238+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:51.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:51.242+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.249424+0000) 2019-10-13T09:10:51.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:51.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.751243+0000) 2019-10-13T09:10:51.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:51.794+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.797204+0000) 2019-10-13T09:10:51.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:51.986+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.986880+0000) 2019-10-13T09:10:51.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:51.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.990538+0000) 2019-10-13T09:10:55.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:55.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.018492+0000) 2019-10-13T09:10:56.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:56.238+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.239714+0000) 2019-10-13T09:10:56.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:10:56.242+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.246375+0000) 2019-10-13T09:10:56.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:10:56.242+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.246630+0000) 2019-10-13T09:10:56.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:10:56.246+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.247361+0000) 2019-10-13T09:10:56.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:10:56.246+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.247897+0000) 2019-10-13T09:10:56.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:56.246+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.248294+0000) 2019-10-13T09:10:56.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:10:56.246+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.248761+0000) 2019-10-13T09:10:56.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:10:56.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:10:56.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:56.242+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.249664+0000) 2019-10-13T09:10:56.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:10:56.250+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.254961+0000) 2019-10-13T09:10:58.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:10:58.506+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:58.508631+0000) 2019-10-13T09:10:59.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:10:59.486+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:59.492429+0000) 2019-10-13T09:11:01.253 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:01.250+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.252628+0000) 2019-10-13T09:11:01.254 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:01.250+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.253155+0000) 2019-10-13T09:11:01.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:01.250+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.253625+0000) 2019-10-13T09:11:01.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:01.250+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.254091+0000) 2019-10-13T09:11:01.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:01.250+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.255700+0000) 2019-10-13T09:11:01.257 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:01.250+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.255084+0000) 2019-10-13T09:11:01.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:01.254+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.256149+0000) 2019-10-13T09:11:01.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:01.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:01.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:01.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.260657+0000) 2019-10-13T09:11:01.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:01.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.751419+0000) 2019-10-13T09:11:01.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:01.794+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.797387+0000) 2019-10-13T09:11:01.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:01.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.990720+0000) 2019-10-13T09:11:01.994 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:01.990+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.993069+0000) 2019-10-13T09:11:05.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:05.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.018687+0000) 2019-10-13T09:11:06.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:06.238+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.239908+0000) 2019-10-13T09:11:06.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:06.254+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.259233+0000) 2019-10-13T09:11:06.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:06.258+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.260084+0000) 2019-10-13T09:11:06.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:06.258+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.260682+0000) 2019-10-13T09:11:06.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:06.258+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.261934+0000) 2019-10-13T09:11:06.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:06.258+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.262097+0000) 2019-10-13T09:11:06.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:06.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.262457+0000) 2019-10-13T09:11:06.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:06.258+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.262699+0000) 2019-10-13T09:11:06.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:06.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:06.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:06.262+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.267779+0000) 2019-10-13T09:11:08.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:08.506+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:08.508854+0000) 2019-10-13T09:11:08.958 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:11:08.958 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:08.962 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:11:08.963 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:09.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:09.486+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:09.492624+0000) 2019-10-13T09:11:11.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:11.262+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.266106+0000) 2019-10-13T09:11:11.267 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:11.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.266917+0000) 2019-10-13T09:11:11.268 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:11.266+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.267831+0000) 2019-10-13T09:11:11.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:11.266+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.268201+0000) 2019-10-13T09:11:11.269 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:11.266+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.268645+0000) 2019-10-13T09:11:11.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:11.266+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.269501+0000) 2019-10-13T09:11:11.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:11.266+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.270164+0000) 2019-10-13T09:11:11.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:11.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:11.276 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:11.270+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.275374+0000) 2019-10-13T09:11:11.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:11.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.751598+0000) 2019-10-13T09:11:11.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:11.794+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.797599+0000) 2019-10-13T09:11:11.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:11.986+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.990894+0000) 2019-10-13T09:11:12.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:11.994+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.999689+0000) 2019-10-13T09:11:15.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:15.017+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.018888+0000) 2019-10-13T09:11:16.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:16.237+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.240099+0000) 2019-10-13T09:11:16.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:16.270+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.273402+0000) 2019-10-13T09:11:16.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:16.269+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.274070+0000) 2019-10-13T09:11:16.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:16.269+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.274455+0000) 2019-10-13T09:11:16.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:16.269+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.274848+0000) 2019-10-13T09:11:16.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:16.269+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.275163+0000) 2019-10-13T09:11:16.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:16.273+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.275938+0000) 2019-10-13T09:11:16.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:16.273+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.276808+0000) 2019-10-13T09:11:16.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:16.273+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:16.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:16.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.285534+0000) 2019-10-13T09:11:18.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:18.505+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:18.509099+0000) 2019-10-13T09:11:19.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:19.486+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:19.492851+0000) 2019-10-13T09:11:21.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:21.277+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.281312+0000) 2019-10-13T09:11:21.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:21.277+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.281755+0000) 2019-10-13T09:11:21.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:21.277+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.282732+0000) 2019-10-13T09:11:21.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:21.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.283076+0000) 2019-10-13T09:11:21.284 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:21.277+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.282508+0000) 2019-10-13T09:11:21.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:21.281+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.283375+0000) 2019-10-13T09:11:21.285 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:21.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:21.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:21.281+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.284161+0000) 2019-10-13T09:11:21.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:21.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.290964+0000) 2019-10-13T09:11:21.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:21.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.751813+0000) 2019-10-13T09:11:21.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:21.793+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.797776+0000) 2019-10-13T09:11:21.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:21.989+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.991073+0000) 2019-10-13T09:11:22.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:22.002+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:22.006105+0000) 2019-10-13T09:11:25.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:25.013+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.019101+0000) 2019-10-13T09:11:26.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:26.237+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.240297+0000) 2019-10-13T09:11:26.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:26.285+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.287381+0000) 2019-10-13T09:11:26.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:26.285+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.287696+0000) 2019-10-13T09:11:26.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:26.285+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.288189+0000) 2019-10-13T09:11:26.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:26.285+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.288889+0000) 2019-10-13T09:11:26.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:26.285+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.289459+0000) 2019-10-13T09:11:26.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:26.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:26.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:26.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.290553+0000) 2019-10-13T09:11:26.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:26.285+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.291142+0000) 2019-10-13T09:11:26.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:26.294+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.299576+0000) 2019-10-13T09:11:28.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:28.505+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:28.509295+0000) 2019-10-13T09:11:29.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:29.490+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:29.493049+0000) 2019-10-13T09:11:31.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:31.289+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.293252+0000) 2019-10-13T09:11:31.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:31.289+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.294239+0000) 2019-10-13T09:11:31.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:31.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.294813+0000) 2019-10-13T09:11:31.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:31.289+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.295156+0000) 2019-10-13T09:11:31.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:31.293+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.296441+0000) 2019-10-13T09:11:31.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:31.293+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.296905+0000) 2019-10-13T09:11:31.298 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:31.293+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.297155+0000) 2019-10-13T09:11:31.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:31.293+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:31.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:31.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.301354+0000) 2019-10-13T09:11:31.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:31.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.751989+0000) 2019-10-13T09:11:31.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:31.793+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.797945+0000) 2019-10-13T09:11:31.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:31.985+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.991265+0000) 2019-10-13T09:11:32.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:32.010+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:32.012436+0000) 2019-10-13T09:11:35.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:35.017+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.019333+0000) 2019-10-13T09:11:36.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:36.237+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.240543+0000) 2019-10-13T09:11:36.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:36.297+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.300147+0000) 2019-10-13T09:11:36.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:36.297+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.301261+0000) 2019-10-13T09:11:36.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:36.297+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.301649+0000) 2019-10-13T09:11:36.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:36.297+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.302209+0000) 2019-10-13T09:11:36.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:36.297+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.302821+0000) 2019-10-13T09:11:36.304 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:36.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:36.304 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:36.301+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.303551+0000) 2019-10-13T09:11:36.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:36.301+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.304253+0000) 2019-10-13T09:11:36.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:36.305+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.308004+0000) 2019-10-13T09:11:38.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:38.505+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:38.509534+0000) 2019-10-13T09:11:39.019 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:11:39.019 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:39.023 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:11:39.023 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:39.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:39.489+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:39.493268+0000) 2019-10-13T09:11:41.309 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:41.305+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.308008+0000) 2019-10-13T09:11:41.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:41.305+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.308422+0000) 2019-10-13T09:11:41.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:41.305+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.309033+0000) 2019-10-13T09:11:41.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:41.305+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.308794+0000) 2019-10-13T09:11:41.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:41.305+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.309419+0000) 2019-10-13T09:11:41.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:41.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:41.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:41.305+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.310087+0000) 2019-10-13T09:11:41.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:41.305+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.311065+0000) 2019-10-13T09:11:41.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:41.309+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.313634+0000) 2019-10-13T09:11:41.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:41.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.752183+0000) 2019-10-13T09:11:41.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:41.793+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.798140+0000) 2019-10-13T09:11:41.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:41.989+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.991466+0000) 2019-10-13T09:11:42.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:42.017+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:42.018735+0000) 2019-10-13T09:11:45.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:45.017+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.019528+0000) 2019-10-13T09:11:46.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:46.237+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.240735+0000) 2019-10-13T09:11:46.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:46.309+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.313429+0000) 2019-10-13T09:11:46.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:46.309+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.313840+0000) 2019-10-13T09:11:46.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:46.309+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.314866+0000) 2019-10-13T09:11:46.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:46.313+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.315431+0000) 2019-10-13T09:11:46.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:46.313+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.315744+0000) 2019-10-13T09:11:46.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:46.313+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.316331+0000) 2019-10-13T09:11:46.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:46.313+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.316428+0000) 2019-10-13T09:11:46.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:46.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:46.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:46.321+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.323812+0000) 2019-10-13T09:11:48.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:48.505+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:48.509734+0000) 2019-10-13T09:11:49.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:49.489+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:49.493438+0000) 2019-10-13T09:11:51.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:51.317+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.319895+0000) 2019-10-13T09:11:51.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:51.317+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.320748+0000) 2019-10-13T09:11:51.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:51.317+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.321097+0000) 2019-10-13T09:11:51.322 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:51.317+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.321789+0000) 2019-10-13T09:11:51.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:51.317+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.322205+0000) 2019-10-13T09:11:51.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:51.317+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:51.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:51.317+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.322808+0000) 2019-10-13T09:11:51.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:51.317+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.323165+0000) 2019-10-13T09:11:51.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:51.329+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.331033+0000) 2019-10-13T09:11:51.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:51.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.752382+0000) 2019-10-13T09:11:51.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:51.793+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.798326+0000) 2019-10-13T09:11:51.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:51.989+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.991642+0000) 2019-10-13T09:11:52.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:52.021+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:52.025005+0000) 2019-10-13T09:11:55.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:55.017+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.019755+0000) 2019-10-13T09:11:56.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:56.236+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.240930+0000) 2019-10-13T09:11:56.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:11:56.320+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.326048+0000) 2019-10-13T09:11:56.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:11:56.320+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.326709+0000) 2019-10-13T09:11:56.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:11:56.324+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.327288+0000) 2019-10-13T09:11:56.329 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:11:56.324+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.327689+0000) 2019-10-13T09:11:56.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:56.324+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.328287+0000) 2019-10-13T09:11:56.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:56.325+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.328672+0000) 2019-10-13T09:11:56.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:11:56.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:11:56.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:11:56.324+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.329453+0000) 2019-10-13T09:11:56.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:11:56.333+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.336317+0000) 2019-10-13T09:11:58.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:11:58.504+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:58.509940+0000) 2019-10-13T09:11:59.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:11:59.489+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:59.493595+0000) 2019-10-13T09:12:01.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:01.328+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.332356+0000) 2019-10-13T09:12:01.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:01.328+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.332978+0000) 2019-10-13T09:12:01.334 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:01.328+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.333326+0000) 2019-10-13T09:12:01.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:01.328+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.334219+0000) 2019-10-13T09:12:01.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:01.328+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.334559+0000) 2019-10-13T09:12:01.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:01.333+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.334943+0000) 2019-10-13T09:12:01.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:01.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:01.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:01.332+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.335703+0000) 2019-10-13T09:12:01.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:01.341+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.344396+0000) 2019-10-13T09:12:01.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:01.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.752567+0000) 2019-10-13T09:12:01.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:01.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.798523+0000) 2019-10-13T09:12:01.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:01.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.991860+0000) 2019-10-13T09:12:02.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:02.029+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:02.031299+0000) 2019-10-13T09:12:05.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:05.016+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.019952+0000) 2019-10-13T09:12:06.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:06.236+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.241158+0000) 2019-10-13T09:12:06.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:06.332+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.338874+0000) 2019-10-13T09:12:06.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:06.336+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.339561+0000) 2019-10-13T09:12:06.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:06.336+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.339906+0000) 2019-10-13T09:12:06.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:06.341+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.342452+0000) 2019-10-13T09:12:06.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:06.336+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.340644+0000) 2019-10-13T09:12:06.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:06.336+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.341016+0000) 2019-10-13T09:12:06.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:06.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:06.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:06.336+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.341570+0000) 2019-10-13T09:12:06.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:06.345+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.346762+0000) 2019-10-13T09:12:08.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:08.508+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:08.510134+0000) 2019-10-13T09:12:09.113 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:12:09.113 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:09.117 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:12:09.117 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:09.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:09.489+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:09.493789+0000) 2019-10-13T09:12:11.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:11.340+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.344919+0000) 2019-10-13T09:12:11.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:11.340+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.345399+0000) 2019-10-13T09:12:11.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:11.340+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.345732+0000) 2019-10-13T09:12:11.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:11.340+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.346907+0000) 2019-10-13T09:12:11.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:11.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69931 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:11.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:11.340+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.347185+0000) 2019-10-13T09:12:11.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:11.345+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.347744+0000) 2019-10-13T09:12:11.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:11.344+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.348205+0000) 2019-10-13T09:12:11.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:11.349+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.352824+0000) 2019-10-13T09:12:11.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:11.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.752769+0000) 2019-10-13T09:12:11.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:11.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.798756+0000) 2019-10-13T09:12:11.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:11.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.992030+0000) 2019-10-13T09:12:12.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:12.033+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:12.037642+0000) 2019-10-13T09:12:15.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:15.016+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.020148+0000) 2019-10-13T09:12:16.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:16.240+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.241303+0000) 2019-10-13T09:12:16.351 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:16.344+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.350455+0000) 2019-10-13T09:12:16.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:16.344+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.351201+0000) 2019-10-13T09:12:16.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:16.348+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.351656+0000) 2019-10-13T09:12:16.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:16.348+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.352384+0000) 2019-10-13T09:12:16.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:16.348+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.353104+0000) 2019-10-13T09:12:16.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:16.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69963 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:16.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:16.348+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.353709+0000) 2019-10-13T09:12:16.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:16.349+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.353899+0000) 2019-10-13T09:12:16.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:16.353+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.357189+0000) 2019-10-13T09:12:18.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:18.504+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:18.510376+0000) 2019-10-13T09:12:19.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:19.489+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:19.494004+0000) 2019-10-13T09:12:21.356 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:21.352+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.355626+0000) 2019-10-13T09:12:21.357 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:21.353+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.356943+0000) 2019-10-13T09:12:21.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:21.352+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.357944+0000) 2019-10-13T09:12:21.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:21.352+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.358100+0000) 2019-10-13T09:12:21.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:21.352+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.358739+0000) 2019-10-13T09:12:21.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:21.352+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.359022+0000) 2019-10-13T09:12:21.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:21.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 69996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:21.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:21.356+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.359718+0000) 2019-10-13T09:12:21.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:21.361+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.365732+0000) 2019-10-13T09:12:21.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:21.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.752978+0000) 2019-10-13T09:12:21.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:21.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.798944+0000) 2019-10-13T09:12:21.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:21.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.992240+0000) 2019-10-13T09:12:22.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:22.041+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:22.043523+0000) 2019-10-13T09:12:25.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:25.016+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.020350+0000) 2019-10-13T09:12:26.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:26.240+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.241509+0000) 2019-10-13T09:12:26.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:26.356+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.362087+0000) 2019-10-13T09:12:26.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:26.360+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.363364+0000) 2019-10-13T09:12:26.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:26.360+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.364297+0000) 2019-10-13T09:12:26.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:26.360+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.364660+0000) 2019-10-13T09:12:26.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:26.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:26.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:26.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.365274+0000) 2019-10-13T09:12:26.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:26.360+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.365903+0000) 2019-10-13T09:12:26.368 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:26.360+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.366428+0000) 2019-10-13T09:12:26.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:26.369+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.371073+0000) 2019-10-13T09:12:28.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:28.504+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:28.510603+0000) 2019-10-13T09:12:29.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:29.493+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:29.494201+0000) 2019-10-13T09:12:31.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:31.364+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.368667+0000) 2019-10-13T09:12:31.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:31.364+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.369206+0000) 2019-10-13T09:12:31.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:31.364+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.369550+0000) 2019-10-13T09:12:31.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:31.364+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.370012+0000) 2019-10-13T09:12:31.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:31.364+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.370778+0000) 2019-10-13T09:12:31.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:31.364+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.371211+0000) 2019-10-13T09:12:31.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:31.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:31.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:31.368+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.372163+0000) 2019-10-13T09:12:31.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:31.376+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.379389+0000) 2019-10-13T09:12:31.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:31.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.753178+0000) 2019-10-13T09:12:31.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:31.792+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.799151+0000) 2019-10-13T09:12:31.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:31.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.992443+0000) 2019-10-13T09:12:32.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:32.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:32.049810+0000) 2019-10-13T09:12:35.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:35.020+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.020577+0000) 2019-10-13T09:12:36.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:36.240+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.241710+0000) 2019-10-13T09:12:36.375 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:36.368+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.374333+0000) 2019-10-13T09:12:36.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:36.372+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.375757+0000) 2019-10-13T09:12:36.376 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:36.372+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.376075+0000) 2019-10-13T09:12:36.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:36.372+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.376447+0000) 2019-10-13T09:12:36.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:36.372+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.377373+0000) 2019-10-13T09:12:36.378 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:36.372+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.377727+0000) 2019-10-13T09:12:36.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:36.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:36.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:36.372+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.378372+0000) 2019-10-13T09:12:36.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:36.384+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.386968+0000) 2019-10-13T09:12:38.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:38.504+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:38.510839+0000) 2019-10-13T09:12:39.241 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:12:39.242 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:39.245 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:12:39.245 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:39.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:39.492+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:39.494392+0000) 2019-10-13T09:12:41.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:41.375+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.381277+0000) 2019-10-13T09:12:41.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:41.376+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.381883+0000) 2019-10-13T09:12:41.383 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:41.375+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.382322+0000) 2019-10-13T09:12:41.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:41.375+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.382692+0000) 2019-10-13T09:12:41.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:41.379+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.383606+0000) 2019-10-13T09:12:41.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:41.379+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.384127+0000) 2019-10-13T09:12:41.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:41.379+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.384529+0000) 2019-10-13T09:12:41.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:41.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:41.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:41.388+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.390222+0000) 2019-10-13T09:12:41.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:41.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.753360+0000) 2019-10-13T09:12:41.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:41.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.799290+0000) 2019-10-13T09:12:41.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:41.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.992653+0000) 2019-10-13T09:12:42.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:42.052+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:42.056127+0000) 2019-10-13T09:12:45.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:45.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.020811+0000) 2019-10-13T09:12:46.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:46.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.241901+0000) 2019-10-13T09:12:46.389 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:46.384+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.388324+0000) 2019-10-13T09:12:46.390 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:46.383+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.388636+0000) 2019-10-13T09:12:46.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:46.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.389200+0000) 2019-10-13T09:12:46.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:46.383+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.389625+0000) 2019-10-13T09:12:46.391 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:46.383+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.389902+0000) 2019-10-13T09:12:46.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:46.383+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.390794+0000) 2019-10-13T09:12:46.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:46.383+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.391082+0000) 2019-10-13T09:12:46.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:46.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:46.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:46.396+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.398609+0000) 2019-10-13T09:12:48.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:48.503+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:48.511061+0000) 2019-10-13T09:12:49.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:49.492+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:49.494603+0000) 2019-10-13T09:12:51.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:51.387+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.395015+0000) 2019-10-13T09:12:51.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:51.391+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.395609+0000) 2019-10-13T09:12:51.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:51.391+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.396273+0000) 2019-10-13T09:12:51.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:51.392+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.397763+0000) 2019-10-13T09:12:51.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:51.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:51.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:51.391+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.398146+0000) 2019-10-13T09:12:51.399 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:51.391+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.398408+0000) 2019-10-13T09:12:51.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:51.395+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.399700+0000) 2019-10-13T09:12:51.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:51.404+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.407313+0000) 2019-10-13T09:12:51.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:51.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.753546+0000) 2019-10-13T09:12:51.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:51.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.799515+0000) 2019-10-13T09:12:51.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:51.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.992893+0000) 2019-10-13T09:12:52.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:52.060+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:52.062551+0000) 2019-10-13T09:12:55.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:55.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.021039+0000) 2019-10-13T09:12:56.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:56.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.242095+0000) 2019-10-13T09:12:56.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:56.396+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.401201+0000) 2019-10-13T09:12:56.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:12:56.395+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.401735+0000) 2019-10-13T09:12:56.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:12:56.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.402183+0000) 2019-10-13T09:12:56.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:12:56.395+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.402586+0000) 2019-10-13T09:12:56.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:12:56.395+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.402896+0000) 2019-10-13T09:12:56.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:56.395+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.403225+0000) 2019-10-13T09:12:56.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:12:56.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70220 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:12:56.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:12:56.399+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.404454+0000) 2019-10-13T09:12:56.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:12:56.408+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.410315+0000) 2019-10-13T09:12:58.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:12:58.507+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:58.511344+0000) 2019-10-13T09:12:59.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:12:59.492+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:59.494793+0000) 2019-10-13T09:13:01.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:01.408+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.410173+0000) 2019-10-13T09:13:01.418 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:01.403+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.408128+0000) 2019-10-13T09:13:01.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:01.403+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.408648+0000) 2019-10-13T09:13:01.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:01.403+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.409165+0000) 2019-10-13T09:13:01.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:01.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.409474+0000) 2019-10-13T09:13:01.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:01.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.409734+0000) 2019-10-13T09:13:01.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:01.403+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.410477+0000) 2019-10-13T09:13:01.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:01.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:01.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:01.416+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.418501+0000) 2019-10-13T09:13:01.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:01.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.753749+0000) 2019-10-13T09:13:01.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:01.799+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.799707+0000) 2019-10-13T09:13:01.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:01.987+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.993075+0000) 2019-10-13T09:13:02.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:02.068+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:02.068791+0000) 2019-10-13T09:13:05.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:05.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.021258+0000) 2019-10-13T09:13:06.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:06.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.242290+0000) 2019-10-13T09:13:06.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:06.407+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.414212+0000) 2019-10-13T09:13:06.415 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:06.412+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.414479+0000) 2019-10-13T09:13:06.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:06.407+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.415140+0000) 2019-10-13T09:13:06.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:06.411+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.415569+0000) 2019-10-13T09:13:06.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:06.411+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.416412+0000) 2019-10-13T09:13:06.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:06.411+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.417173+0000) 2019-10-13T09:13:06.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:06.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:06.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:06.411+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.417595+0000) 2019-10-13T09:13:06.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:06.420+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.422500+0000) 2019-10-13T09:13:08.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:08.511+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:08.511540+0000) 2019-10-13T09:13:09.352 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:13:09.353 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:09.357 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:13:09.357 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:09.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:09.492+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:09.494966+0000) 2019-10-13T09:13:11.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:11.419+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.421077+0000) 2019-10-13T09:13:11.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:11.420+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.422344+0000) 2019-10-13T09:13:11.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:11.419+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.422783+0000) 2019-10-13T09:13:11.424 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:11.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:11.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:11.419+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.423058+0000) 2019-10-13T09:13:11.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:11.423+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.423709+0000) 2019-10-13T09:13:11.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:11.423+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.424530+0000) 2019-10-13T09:13:11.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:11.423+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.424421+0000) 2019-10-13T09:13:11.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:11.428+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.430570+0000) 2019-10-13T09:13:11.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:11.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.753924+0000) 2019-10-13T09:13:11.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:11.799+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.799898+0000) 2019-10-13T09:13:11.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:11.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.993251+0000) 2019-10-13T09:13:12.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:12.072+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:12.074869+0000) 2019-10-13T09:13:15.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:15.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.021457+0000) 2019-10-13T09:13:16.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:16.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.242500+0000) 2019-10-13T09:13:16.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:16.423+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.426650+0000) 2019-10-13T09:13:16.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:16.427+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.427757+0000) 2019-10-13T09:13:16.429 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:16.427+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.428513+0000) 2019-10-13T09:13:16.429 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:16.424+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.428667+0000) 2019-10-13T09:13:16.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:16.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:16.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:16.427+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.429824+0000) 2019-10-13T09:13:16.433 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:16.427+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.430299+0000) 2019-10-13T09:13:16.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:16.427+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.430387+0000) 2019-10-13T09:13:16.435 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:16.432+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.434713+0000) 2019-10-13T09:13:18.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:18.511+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:18.511746+0000) 2019-10-13T09:13:19.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:19.492+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:19.495183+0000) 2019-10-13T09:13:21.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:21.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.432519+0000) 2019-10-13T09:13:21.434 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:21.431+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.432813+0000) 2019-10-13T09:13:21.434 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:21.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.433227+0000) 2019-10-13T09:13:21.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:21.431+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.434707+0000) 2019-10-13T09:13:21.436 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:21.431+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.434970+0000) 2019-10-13T09:13:21.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:21.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:21.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:21.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.435808+0000) 2019-10-13T09:13:21.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:21.435+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.435637+0000) 2019-10-13T09:13:21.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:21.440+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.443057+0000) 2019-10-13T09:13:21.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:21.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.754104+0000) 2019-10-13T09:13:21.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:21.799+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.800079+0000) 2019-10-13T09:13:21.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:21.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.993431+0000) 2019-10-13T09:13:22.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:22.080+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:22.081230+0000) 2019-10-13T09:13:25.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:25.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.021678+0000) 2019-10-13T09:13:26.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:26.238+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.242697+0000) 2019-10-13T09:13:26.440 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:26.438+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.439653+0000) 2019-10-13T09:13:26.441 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:26.438+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.439646+0000) 2019-10-13T09:13:26.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:26.438+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.440026+0000) 2019-10-13T09:13:26.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:26.438+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.440324+0000) 2019-10-13T09:13:26.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:26.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.440753+0000) 2019-10-13T09:13:26.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:26.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70415 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:26.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:26.438+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.441577+0000) 2019-10-13T09:13:26.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:26.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.441975+0000) 2019-10-13T09:13:26.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:26.443+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.448662+0000) 2019-10-13T09:13:28.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:28.510+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:28.511968+0000) 2019-10-13T09:13:29.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:29.491+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:29.495411+0000) 2019-10-13T09:13:31.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:31.442+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.444658+0000) 2019-10-13T09:13:31.446 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:31.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.444882+0000) 2019-10-13T09:13:31.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:31.442+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.446105+0000) 2019-10-13T09:13:31.447 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:31.442+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.446499+0000) 2019-10-13T09:13:31.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:31.443+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.447019+0000) 2019-10-13T09:13:31.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:31.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:31.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:31.446+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.447489+0000) 2019-10-13T09:13:31.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:31.446+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.448158+0000) 2019-10-13T09:13:31.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:31.447+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.453637+0000) 2019-10-13T09:13:31.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:31.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.754296+0000) 2019-10-13T09:13:31.802 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:31.798+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.800291+0000) 2019-10-13T09:13:31.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:31.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.993608+0000) 2019-10-13T09:13:32.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:32.091+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:32.092801+0000) 2019-10-13T09:13:35.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:35.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.021901+0000) 2019-10-13T09:13:36.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:36.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.242920+0000) 2019-10-13T09:13:36.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:36.446+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.449818+0000) 2019-10-13T09:13:36.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:36.446+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.451183+0000) 2019-10-13T09:13:36.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:36.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.452510+0000) 2019-10-13T09:13:36.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:36.450+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.452190+0000) 2019-10-13T09:13:36.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:36.450+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.452798+0000) 2019-10-13T09:13:36.454 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:36.450+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.453173+0000) 2019-10-13T09:13:36.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:36.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.453809+0000) 2019-10-13T09:13:36.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:36.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:36.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:36.455+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.458381+0000) 2019-10-13T09:13:38.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:38.510+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:38.512189+0000) 2019-10-13T09:13:39.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:39.491+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:39.495606+0000) 2019-10-13T09:13:39.505 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:13:39.505 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:39.509 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:13:39.509 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:41.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:41.454+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.458320+0000) 2019-10-13T09:13:41.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:41.454+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.458650+0000) 2019-10-13T09:13:41.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:41.455+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.458913+0000) 2019-10-13T09:13:41.461 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:41.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:41.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:41.458+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.460122+0000) 2019-10-13T09:13:41.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:41.458+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.460611+0000) 2019-10-13T09:13:41.462 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:41.458+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.462001+0000) 2019-10-13T09:13:41.465 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:41.458+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.461109+0000) 2019-10-13T09:13:41.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:41.463+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.466537+0000) 2019-10-13T09:13:41.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:41.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.754484+0000) 2019-10-13T09:13:41.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:41.798+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.800478+0000) 2019-10-13T09:13:41.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:41.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.993805+0000) 2019-10-13T09:13:42.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:42.095+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:42.099425+0000) 2019-10-13T09:13:45.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:45.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.022103+0000) 2019-10-13T09:13:46.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:46.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.243112+0000) 2019-10-13T09:13:46.465 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:46.462+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.463731+0000) 2019-10-13T09:13:46.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:46.462+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.464128+0000) 2019-10-13T09:13:46.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:46.459+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.465375+0000) 2019-10-13T09:13:46.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:46.462+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.465994+0000) 2019-10-13T09:13:46.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:46.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70545 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:46.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:46.462+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.465999+0000) 2019-10-13T09:13:46.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:46.462+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.466338+0000) 2019-10-13T09:13:46.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:46.462+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.467008+0000) 2019-10-13T09:13:46.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:46.467+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.470914+0000) 2019-10-13T09:13:48.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:48.510+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:48.512412+0000) 2019-10-13T09:13:49.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:49.491+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:49.495822+0000) 2019-10-13T09:13:51.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:51.466+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.468699+0000) 2019-10-13T09:13:51.470 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:51.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.469752+0000) 2019-10-13T09:13:51.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:51.466+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.470381+0000) 2019-10-13T09:13:51.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:51.466+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.470764+0000) 2019-10-13T09:13:51.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:51.467+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.471068+0000) 2019-10-13T09:13:51.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:51.470+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.472063+0000) 2019-10-13T09:13:51.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:51.470+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.472714+0000) 2019-10-13T09:13:51.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:51.470+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70576 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:51.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:51.475+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.480721+0000) 2019-10-13T09:13:51.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:51.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.754659+0000) 2019-10-13T09:13:51.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:51.798+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.800663+0000) 2019-10-13T09:13:51.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:51.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.993973+0000) 2019-10-13T09:13:52.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:52.103+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:52.105943+0000) 2019-10-13T09:13:55.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:55.018+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.022308+0000) 2019-10-13T09:13:56.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:56.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.243331+0000) 2019-10-13T09:13:56.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:56.471+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.475757+0000) 2019-10-13T09:13:56.477 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:13:56.474+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.476558+0000) 2019-10-13T09:13:56.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:13:56.474+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.477306+0000) 2019-10-13T09:13:56.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:56.474+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.478216+0000) 2019-10-13T09:13:56.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:13:56.474+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.478598+0000) 2019-10-13T09:13:56.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:13:56.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:13:56.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:13:56.474+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.479187+0000) 2019-10-13T09:13:56.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:13:56.474+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.479184+0000) 2019-10-13T09:13:56.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:13:56.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.482505+0000) 2019-10-13T09:13:58.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:13:58.510+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:58.512591+0000) 2019-10-13T09:13:59.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:13:59.495+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:59.496013+0000) 2019-10-13T09:14:01.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:01.478+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.482686+0000) 2019-10-13T09:14:01.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:01.482+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.483356+0000) 2019-10-13T09:14:01.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:01.478+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.483074+0000) 2019-10-13T09:14:01.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:01.482+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.483883+0000) 2019-10-13T09:14:01.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:01.479+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.484525+0000) 2019-10-13T09:14:01.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:01.482+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:01.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:01.482+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.485564+0000) 2019-10-13T09:14:01.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:01.482+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.486091+0000) 2019-10-13T09:14:01.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:01.487+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.493949+0000) 2019-10-13T09:14:01.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:01.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.754840+0000) 2019-10-13T09:14:01.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:01.798+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.800883+0000) 2019-10-13T09:14:01.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:01.990+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.994175+0000) 2019-10-13T09:14:02.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:02.107+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:02.112216+0000) 2019-10-13T09:14:05.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:05.022+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.022542+0000) 2019-10-13T09:14:06.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:06.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.243530+0000) 2019-10-13T09:14:06.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:06.485+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.488477+0000) 2019-10-13T09:14:06.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:06.485+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.489449+0000) 2019-10-13T09:14:06.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:06.485+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.489660+0000) 2019-10-13T09:14:06.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:06.485+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.490072+0000) 2019-10-13T09:14:06.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:06.485+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.490474+0000) 2019-10-13T09:14:06.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:06.487+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.491056+0000) 2019-10-13T09:14:06.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:06.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:06.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:06.489+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.492283+0000) 2019-10-13T09:14:06.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:06.495+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.498713+0000) 2019-10-13T09:14:08.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:08.509+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:08.512785+0000) 2019-10-13T09:14:09.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:09.495+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:09.496213+0000) 2019-10-13T09:14:09.625 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:14:09.625 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:09.629 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:14:09.629 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:11.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:11.489+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.494698+0000) 2019-10-13T09:14:11.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:11.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.495470+0000) 2019-10-13T09:14:11.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:11.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.495863+0000) 2019-10-13T09:14:11.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:11.491+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.496395+0000) 2019-10-13T09:14:11.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:11.493+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.496819+0000) 2019-10-13T09:14:11.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:11.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.497352+0000) 2019-10-13T09:14:11.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:11.493+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:11.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:11.493+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.498171+0000) 2019-10-13T09:14:11.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:11.499+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.504015+0000) 2019-10-13T09:14:11.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:11.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.755018+0000) 2019-10-13T09:14:11.802 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:11.797+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.801109+0000) 2019-10-13T09:14:11.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:11.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.994355+0000) 2019-10-13T09:14:12.119 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:12.115+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:12.118501+0000) 2019-10-13T09:14:15.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:15.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.022759+0000) 2019-10-13T09:14:16.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:16.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.243766+0000) 2019-10-13T09:14:16.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:16.497+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.501231+0000) 2019-10-13T09:14:16.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:16.497+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.501749+0000) 2019-10-13T09:14:16.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:16.497+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.502358+0000) 2019-10-13T09:14:16.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:16.497+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.502731+0000) 2019-10-13T09:14:16.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:16.499+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.502985+0000) 2019-10-13T09:14:16.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:16.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:16.505 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:16.501+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.504086+0000) 2019-10-13T09:14:16.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:16.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.504897+0000) 2019-10-13T09:14:16.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:16.507+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.513596+0000) 2019-10-13T09:14:18.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:18.509+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:18.512985+0000) 2019-10-13T09:14:19.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:19.495+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:19.496437+0000) 2019-10-13T09:14:21.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:21.501+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.507237+0000) 2019-10-13T09:14:21.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:21.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.507607+0000) 2019-10-13T09:14:21.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:21.505+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.508969+0000) 2019-10-13T09:14:21.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:21.505+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.509037+0000) 2019-10-13T09:14:21.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:21.505+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.509338+0000) 2019-10-13T09:14:21.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:21.505+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.509566+0000) 2019-10-13T09:14:21.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:21.505+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.510198+0000) 2019-10-13T09:14:21.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:21.509+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70771 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:21.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:21.510+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.517629+0000) 2019-10-13T09:14:21.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:21.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.755198+0000) 2019-10-13T09:14:21.802 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:21.797+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.801302+0000) 2019-10-13T09:14:21.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:21.989+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.994536+0000) 2019-10-13T09:14:22.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:22.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:22.124796+0000) 2019-10-13T09:14:25.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:25.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.022988+0000) 2019-10-13T09:14:26.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:26.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.243994+0000) 2019-10-13T09:14:26.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:26.506+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.514025+0000) 2019-10-13T09:14:26.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:26.509+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.514981+0000) 2019-10-13T09:14:26.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:26.513+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.515430+0000) 2019-10-13T09:14:26.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:26.513+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.516098+0000) 2019-10-13T09:14:26.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:26.513+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.516695+0000) 2019-10-13T09:14:26.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:26.513+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.516812+0000) 2019-10-13T09:14:26.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:26.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.517301+0000) 2019-10-13T09:14:26.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:26.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:26.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:26.518+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.524316+0000) 2019-10-13T09:14:28.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:28.509+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:28.513188+0000) 2019-10-13T09:14:29.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:29.494+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:29.496660+0000) 2019-10-13T09:14:31.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:31.517+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.521589+0000) 2019-10-13T09:14:31.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:31.518+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.522536+0000) 2019-10-13T09:14:31.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:31.517+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.523036+0000) 2019-10-13T09:14:31.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:31.521+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.523781+0000) 2019-10-13T09:14:31.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:31.521+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.524025+0000) 2019-10-13T09:14:31.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:31.521+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.524274+0000) 2019-10-13T09:14:31.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:31.521+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.524419+0000) 2019-10-13T09:14:31.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:31.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:31.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:31.526+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.530840+0000) 2019-10-13T09:14:31.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:31.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.755402+0000) 2019-10-13T09:14:31.802 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:31.797+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.801551+0000) 2019-10-13T09:14:31.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:31.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.994717+0000) 2019-10-13T09:14:32.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:32.130+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:32.131254+0000) 2019-10-13T09:14:35.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:35.017+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.023151+0000) 2019-10-13T09:14:36.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:36.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.244186+0000) 2019-10-13T09:14:36.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:36.525+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.528727+0000) 2019-10-13T09:14:36.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:36.525+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.529157+0000) 2019-10-13T09:14:36.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:36.525+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.529884+0000) 2019-10-13T09:14:36.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:36.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.529714+0000) 2019-10-13T09:14:36.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:36.526+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.530227+0000) 2019-10-13T09:14:36.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:36.525+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.531229+0000) 2019-10-13T09:14:36.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:36.529+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.532210+0000) 2019-10-13T09:14:36.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:36.529+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:36.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:36.530+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.538029+0000) 2019-10-13T09:14:38.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:38.509+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:38.513413+0000) 2019-10-13T09:14:39.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:39.494+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:39.496854+0000) 2019-10-13T09:14:39.880 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:14:39.881 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:39.885 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:14:39.885 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:41.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:41.533+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.536542+0000) 2019-10-13T09:14:41.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:41.533+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.536826+0000) 2019-10-13T09:14:41.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:41.533+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.537440+0000) 2019-10-13T09:14:41.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:41.530+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.536350+0000) 2019-10-13T09:14:41.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:41.533+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.538113+0000) 2019-10-13T09:14:41.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:41.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:41.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:41.533+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.538488+0000) 2019-10-13T09:14:41.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:41.533+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.539174+0000) 2019-10-13T09:14:41.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:41.538+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.542750+0000) 2019-10-13T09:14:41.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:41.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.755612+0000) 2019-10-13T09:14:41.802 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:41.797+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.801738+0000) 2019-10-13T09:14:41.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:41.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.994897+0000) 2019-10-13T09:14:42.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:42.134+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:42.137541+0000) 2019-10-13T09:14:45.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:45.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.023348+0000) 2019-10-13T09:14:46.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:46.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.244398+0000) 2019-10-13T09:14:46.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:46.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.542004+0000) 2019-10-13T09:14:46.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:46.537+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.542519+0000) 2019-10-13T09:14:46.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:46.537+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.543118+0000) 2019-10-13T09:14:46.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:46.541+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.544416+0000) 2019-10-13T09:14:46.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:46.538+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.544661+0000) 2019-10-13T09:14:46.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:46.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:46.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:46.541+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.545544+0000) 2019-10-13T09:14:46.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:46.541+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.546172+0000) 2019-10-13T09:14:46.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:46.542+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.549878+0000) 2019-10-13T09:14:48.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:48.509+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:48.513589+0000) 2019-10-13T09:14:49.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:49.494+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:49.497046+0000) 2019-10-13T09:14:51.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:51.542+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.547694+0000) 2019-10-13T09:14:51.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:51.544+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.548582+0000) 2019-10-13T09:14:51.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:51.544+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.549058+0000) 2019-10-13T09:14:51.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:51.544+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.549652+0000) 2019-10-13T09:14:51.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:51.544+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.550150+0000) 2019-10-13T09:14:51.551 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:51.544+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.550558+0000) 2019-10-13T09:14:51.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:51.548+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.551652+0000) 2019-10-13T09:14:51.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:51.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 70965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:51.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:51.554+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.558752+0000) 2019-10-13T09:14:51.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:51.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.755790+0000) 2019-10-13T09:14:51.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:51.796+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.801921+0000) 2019-10-13T09:14:51.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:51.988+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.995026+0000) 2019-10-13T09:14:52.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:52.142+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:52.143836+0000) 2019-10-13T09:14:55.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:55.020+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.023522+0000) 2019-10-13T09:14:56.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:56.240+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.244592+0000) 2019-10-13T09:14:56.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:14:56.548+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.555126+0000) 2019-10-13T09:14:56.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:14:56.552+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.555971+0000) 2019-10-13T09:14:56.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:14:56.552+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.556234+0000) 2019-10-13T09:14:56.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:56.550+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.556759+0000) 2019-10-13T09:14:56.561 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:14:56.552+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.557492+0000) 2019-10-13T09:14:56.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:56.552+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.557736+0000) 2019-10-13T09:14:56.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:14:56.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.558223+0000) 2019-10-13T09:14:56.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:14:56.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:14:56.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:14:56.562+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.566646+0000) 2019-10-13T09:14:58.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:14:58.508+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:58.513800+0000) 2019-10-13T09:14:59.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:14:59.494+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:59.497240+0000) 2019-10-13T09:15:01.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:01.556+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.561062+0000) 2019-10-13T09:15:01.563 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:01.556+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.562458+0000) 2019-10-13T09:15:01.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:01.556+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.562625+0000) 2019-10-13T09:15:01.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:01.556+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.563182+0000) 2019-10-13T09:15:01.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:01.558+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.563503+0000) 2019-10-13T09:15:01.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:01.560+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.564203+0000) 2019-10-13T09:15:01.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:01.560+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.564582+0000) 2019-10-13T09:15:01.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:01.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:01.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:01.562+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.567833+0000) 2019-10-13T09:15:01.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:01.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.755981+0000) 2019-10-13T09:15:01.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:01.796+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.802103+0000) 2019-10-13T09:15:01.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:01.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.995237+0000) 2019-10-13T09:15:02.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:02.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:02.154919+0000) 2019-10-13T09:15:05.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:05.020+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:05.023680+0000) 2019-10-13T09:15:06.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:06.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.244793+0000) 2019-10-13T09:15:06.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:06.564+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.568098+0000) 2019-10-13T09:15:06.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:06.564+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.568231+0000) 2019-10-13T09:15:06.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:06.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.569058+0000) 2019-10-13T09:15:06.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:06.564+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.568841+0000) 2019-10-13T09:15:06.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:06.564+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.569729+0000) 2019-10-13T09:15:06.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:06.564+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.570157+0000) 2019-10-13T09:15:06.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:06.564+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.571015+0000) 2019-10-13T09:15:06.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:06.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:06.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:06.570+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.576771+0000) 2019-10-13T09:15:08.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:08.508+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:08.513952+0000) 2019-10-13T09:15:09.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:09.494+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:09.497429+0000) 2019-10-13T09:15:10.004 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:15:10.004 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:10.008 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:15:10.008 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:11.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:11.568+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.574170+0000) 2019-10-13T09:15:11.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:11.568+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.574095+0000) 2019-10-13T09:15:11.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:11.574+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.575193+0000) 2019-10-13T09:15:11.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:11.586+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.589115+0000) 2019-10-13T09:15:11.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:11.572+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.575602+0000) 2019-10-13T09:15:11.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:11.572+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.576603+0000) 2019-10-13T09:15:11.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:11.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.576716+0000) 2019-10-13T09:15:11.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:11.572+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.577145+0000) 2019-10-13T09:15:11.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:11.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:11.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:11.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.756189+0000) 2019-10-13T09:15:11.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:11.796+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.802320+0000) 2019-10-13T09:15:11.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:11.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.995402+0000) 2019-10-13T09:15:12.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:12.158+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:12.161442+0000) 2019-10-13T09:15:15.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:15.020+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:15.023881+0000) 2019-10-13T09:15:16.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:16.240+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.244990+0000) 2019-10-13T09:15:16.581 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:16.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.580430+0000) 2019-10-13T09:15:16.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:16.581+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.582628+0000) 2019-10-13T09:15:16.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:16.576+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.582030+0000) 2019-10-13T09:15:16.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:16.576+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.582390+0000) 2019-10-13T09:15:16.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:16.576+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.583174+0000) 2019-10-13T09:15:16.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:16.580+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.583431+0000) 2019-10-13T09:15:16.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:16.580+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.583884+0000) 2019-10-13T09:15:16.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:16.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:16.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:16.585+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.588549+0000) 2019-10-13T09:15:18.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:18.512+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:18.514119+0000) 2019-10-13T09:15:19.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:19.493+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:19.497656+0000) 2019-10-13T09:15:21.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:21.580+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.586826+0000) 2019-10-13T09:15:21.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:21.584+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.587879+0000) 2019-10-13T09:15:21.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:21.584+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.588810+0000) 2019-10-13T09:15:21.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:21.585+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.589486+0000) 2019-10-13T09:15:21.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:21.584+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.589664+0000) 2019-10-13T09:15:21.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:21.584+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.590038+0000) 2019-10-13T09:15:21.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:21.584+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.590443+0000) 2019-10-13T09:15:21.592 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:21.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:21.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:21.597+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.599192+0000) 2019-10-13T09:15:21.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:21.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.756373+0000) 2019-10-13T09:15:21.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:21.796+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.802505+0000) 2019-10-13T09:15:21.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:21.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.995597+0000) 2019-10-13T09:15:22.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:22.165+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:22.168451+0000) 2019-10-13T09:15:25.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:25.020+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:25.024101+0000) 2019-10-13T09:15:26.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:26.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.245183+0000) 2019-10-13T09:15:26.595 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:26.593+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.594391+0000) 2019-10-13T09:15:26.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:26.592+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.595424+0000) 2019-10-13T09:15:26.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:26.592+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.595816+0000) 2019-10-13T09:15:26.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:26.592+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.596281+0000) 2019-10-13T09:15:26.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:26.592+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.596990+0000) 2019-10-13T09:15:26.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:26.592+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.597711+0000) 2019-10-13T09:15:26.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:26.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:26.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:26.592+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.598848+0000) 2019-10-13T09:15:26.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:26.601+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.603547+0000) 2019-10-13T09:15:28.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:28.508+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:28.514327+0000) 2019-10-13T09:15:29.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:29.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:29.497849+0000) 2019-10-13T09:15:31.603 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:31.596+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.601775+0000) 2019-10-13T09:15:31.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:31.596+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.601969+0000) 2019-10-13T09:15:31.603 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:31.601+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.602808+0000) 2019-10-13T09:15:31.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:31.596+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.603272+0000) 2019-10-13T09:15:31.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:31.600+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.604440+0000) 2019-10-13T09:15:31.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:31.600+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:31.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:31.600+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.605079+0000) 2019-10-13T09:15:31.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:31.600+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.605664+0000) 2019-10-13T09:15:31.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:31.609+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.610476+0000) 2019-10-13T09:15:31.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:31.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.756576+0000) 2019-10-13T09:15:31.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:31.796+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.802692+0000) 2019-10-13T09:15:31.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:31.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.995766+0000) 2019-10-13T09:15:32.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:32.173+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:32.174739+0000) 2019-10-13T09:15:35.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:35.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:35.024291+0000) 2019-10-13T09:15:36.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:36.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.245385+0000) 2019-10-13T09:15:36.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:36.603+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.608907+0000) 2019-10-13T09:15:36.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:36.603+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.609841+0000) 2019-10-13T09:15:36.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:36.603+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.610182+0000) 2019-10-13T09:15:36.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:36.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:36.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:36.603+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.611022+0000) 2019-10-13T09:15:36.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:36.603+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.610702+0000) 2019-10-13T09:15:36.613 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:36.609+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.611440+0000) 2019-10-13T09:15:36.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:36.607+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.612092+0000) 2019-10-13T09:15:36.620 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:36.613+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.617570+0000) 2019-10-13T09:15:38.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:38.511+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:38.514528+0000) 2019-10-13T09:15:39.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:39.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:39.498049+0000) 2019-10-13T09:15:40.110 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:15:40.111 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:40.114 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:15:40.114 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:41.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:41.607+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.614184+0000) 2019-10-13T09:15:41.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:41.607+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.614834+0000) 2019-10-13T09:15:41.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:41.613+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.615129+0000) 2019-10-13T09:15:41.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:41.611+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.615438+0000) 2019-10-13T09:15:41.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:41.611+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.616106+0000) 2019-10-13T09:15:41.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:41.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.616913+0000) 2019-10-13T09:15:41.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:41.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:41.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:41.611+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.617954+0000) 2019-10-13T09:15:41.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:41.621+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.624800+0000) 2019-10-13T09:15:41.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:41.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.756759+0000) 2019-10-13T09:15:41.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:41.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.802905+0000) 2019-10-13T09:15:41.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:41.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.995900+0000) 2019-10-13T09:15:42.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:42.181+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:42.181567+0000) 2019-10-13T09:15:45.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:45.019+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:45.024490+0000) 2019-10-13T09:15:46.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:46.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.245582+0000) 2019-10-13T09:15:46.621 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:46.615+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.620599+0000) 2019-10-13T09:15:46.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:46.615+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.621200+0000) 2019-10-13T09:15:46.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:46.615+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.621495+0000) 2019-10-13T09:15:46.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:46.615+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.622757+0000) 2019-10-13T09:15:46.624 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:46.621+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.623025+0000) 2019-10-13T09:15:46.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:46.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.623661+0000) 2019-10-13T09:15:46.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:46.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:46.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:46.619+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.624307+0000) 2019-10-13T09:15:46.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:46.629+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.630909+0000) 2019-10-13T09:15:48.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:48.507+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:48.514752+0000) 2019-10-13T09:15:49.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:49.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:49.498245+0000) 2019-10-13T09:15:51.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:51.623+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.627289+0000) 2019-10-13T09:15:51.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:51.623+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.627481+0000) 2019-10-13T09:15:51.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:51.623+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.628885+0000) 2019-10-13T09:15:51.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:51.625+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.629088+0000) 2019-10-13T09:15:51.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:51.623+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.629364+0000) 2019-10-13T09:15:51.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:51.623+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.629896+0000) 2019-10-13T09:15:51.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:51.623+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.629944+0000) 2019-10-13T09:15:51.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:51.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:51.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:51.633+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.634933+0000) 2019-10-13T09:15:51.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:51.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.756942+0000) 2019-10-13T09:15:51.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:51.795+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.803140+0000) 2019-10-13T09:15:51.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:51.995+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.996096+0000) 2019-10-13T09:15:52.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:52.185+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:52.187674+0000) 2019-10-13T09:15:55.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:55.023+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:55.024695+0000) 2019-10-13T09:15:56.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:56.239+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.245806+0000) 2019-10-13T09:15:56.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:15:56.627+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.634127+0000) 2019-10-13T09:15:56.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:56.627+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.634798+0000) 2019-10-13T09:15:56.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:15:56.627+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.635223+0000) 2019-10-13T09:15:56.636 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:56.633+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.635538+0000) 2019-10-13T09:15:56.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:15:56.631+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.636220+0000) 2019-10-13T09:15:56.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:15:56.631+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.636842+0000) 2019-10-13T09:15:56.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:15:56.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:15:56.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:15:56.631+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.637435+0000) 2019-10-13T09:15:56.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:15:56.641+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.643373+0000) 2019-10-13T09:15:58.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:15:58.511+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:58.514945+0000) 2019-10-13T09:15:59.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:15:59.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:59.498457+0000) 2019-10-13T09:16:01.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:01.635+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.640235+0000) 2019-10-13T09:16:01.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:01.635+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.641275+0000) 2019-10-13T09:16:01.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:01.637+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.641765+0000) 2019-10-13T09:16:01.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:01.635+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.642147+0000) 2019-10-13T09:16:01.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:01.635+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.642398+0000) 2019-10-13T09:16:01.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:01.635+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.642736+0000) 2019-10-13T09:16:01.644 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:01.639+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.643506+0000) 2019-10-13T09:16:01.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:01.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:01.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:01.649+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.650556+0000) 2019-10-13T09:16:01.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:01.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.757120+0000) 2019-10-13T09:16:01.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:01.799+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.803349+0000) 2019-10-13T09:16:01.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:01.995+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.996286+0000) 2019-10-13T09:16:02.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:02.193+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:02.194195+0000) 2019-10-13T09:16:05.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:05.023+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:05.024896+0000) 2019-10-13T09:16:06.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:06.243+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.245951+0000) 2019-10-13T09:16:06.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:06.643+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.647071+0000) 2019-10-13T09:16:06.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:06.645+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.648200+0000) 2019-10-13T09:16:06.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:06.647+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.647645+0000) 2019-10-13T09:16:06.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:06.647+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.648709+0000) 2019-10-13T09:16:06.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:06.647+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.649017+0000) 2019-10-13T09:16:06.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:06.647+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.649317+0000) 2019-10-13T09:16:06.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:06.647+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.650073+0000) 2019-10-13T09:16:06.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:06.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:06.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:06.653+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.656067+0000) 2019-10-13T09:16:08.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:08.511+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:08.515151+0000) 2019-10-13T09:16:09.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:09.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:09.498654+0000) 2019-10-13T09:16:10.213 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:16:10.214 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:10.217 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:16:10.217 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:11.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:11.651+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.654135+0000) 2019-10-13T09:16:11.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:11.651+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.654125+0000) 2019-10-13T09:16:11.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:11.655+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.655734+0000) 2019-10-13T09:16:11.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:11.655+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.656092+0000) 2019-10-13T09:16:11.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:11.652+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.656370+0000) 2019-10-13T09:16:11.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:11.655+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.656778+0000) 2019-10-13T09:16:11.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:11.655+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.657020+0000) 2019-10-13T09:16:11.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:11.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:11.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:11.660+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.663483+0000) 2019-10-13T09:16:11.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:11.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.757353+0000) 2019-10-13T09:16:11.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:11.803+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.803553+0000) 2019-10-13T09:16:11.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:11.995+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.996483+0000) 2019-10-13T09:16:12.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:12.196+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:12.200670+0000) 2019-10-13T09:16:15.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:15.023+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:15.025100+0000) 2019-10-13T09:16:16.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:16.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.246153+0000) 2019-10-13T09:16:16.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:16.658+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.661137+0000) 2019-10-13T09:16:16.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:16.658+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.661746+0000) 2019-10-13T09:16:16.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:16.658+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.661854+0000) 2019-10-13T09:16:16.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:16.658+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.663205+0000) 2019-10-13T09:16:16.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:16.662+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.663595+0000) 2019-10-13T09:16:16.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:16.662+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:16.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:16.662+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.664282+0000) 2019-10-13T09:16:16.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:16.660+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.664591+0000) 2019-10-13T09:16:16.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:16.664+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.669360+0000) 2019-10-13T09:16:18.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:18.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:18.515395+0000) 2019-10-13T09:16:19.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:19.496+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:19.498870+0000) 2019-10-13T09:16:21.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:21.662+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.666999+0000) 2019-10-13T09:16:21.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:21.666+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.668082+0000) 2019-10-13T09:16:21.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:21.666+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.668260+0000) 2019-10-13T09:16:21.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:21.666+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.668848+0000) 2019-10-13T09:16:21.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:21.666+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.669397+0000) 2019-10-13T09:16:21.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:21.664+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.669907+0000) 2019-10-13T09:16:21.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:21.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:21.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:21.666+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.670933+0000) 2019-10-13T09:16:21.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:21.672+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.674308+0000) 2019-10-13T09:16:21.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:21.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.757537+0000) 2019-10-13T09:16:21.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:21.802+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.803733+0000) 2019-10-13T09:16:21.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:21.994+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.996652+0000) 2019-10-13T09:16:22.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:22.204+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:22.207106+0000) 2019-10-13T09:16:25.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:25.022+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:25.025299+0000) 2019-10-13T09:16:26.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:26.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.246389+0000) 2019-10-13T09:16:26.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:26.670+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.673625+0000) 2019-10-13T09:16:26.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:26.670+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.674156+0000) 2019-10-13T09:16:26.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:26.670+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.674872+0000) 2019-10-13T09:16:26.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:26.672+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.675547+0000) 2019-10-13T09:16:26.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:26.674+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.675740+0000) 2019-10-13T09:16:26.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:26.674+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.676004+0000) 2019-10-13T09:16:26.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:26.674+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.676619+0000) 2019-10-13T09:16:26.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:26.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:26.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:26.676+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.680951+0000) 2019-10-13T09:16:28.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:28.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:28.515590+0000) 2019-10-13T09:16:29.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:29.496+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:29.499064+0000) 2019-10-13T09:16:31.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:31.678+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.679773+0000) 2019-10-13T09:16:31.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:31.678+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.681309+0000) 2019-10-13T09:16:31.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:31.678+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.682225+0000) 2019-10-13T09:16:31.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:31.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.682577+0000) 2019-10-13T09:16:31.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:31.680+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.682836+0000) 2019-10-13T09:16:31.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:31.678+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.683277+0000) 2019-10-13T09:16:31.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:31.682+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.683697+0000) 2019-10-13T09:16:31.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:31.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:31.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:31.684+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.688176+0000) 2019-10-13T09:16:31.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:31.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.757719+0000) 2019-10-13T09:16:31.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:31.802+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.803921+0000) 2019-10-13T09:16:31.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:31.994+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.996829+0000) 2019-10-13T09:16:32.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:32.212+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:32.213961+0000) 2019-10-13T09:16:35.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:35.022+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:35.025504+0000) 2019-10-13T09:16:36.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:36.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.246610+0000) 2019-10-13T09:16:36.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:36.682+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.687146+0000) 2019-10-13T09:16:36.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:36.686+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.687640+0000) 2019-10-13T09:16:36.689 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:36.686+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.688507+0000) 2019-10-13T09:16:36.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:36.686+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.688525+0000) 2019-10-13T09:16:36.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:36.686+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.689734+0000) 2019-10-13T09:16:36.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:36.688+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.690433+0000) 2019-10-13T09:16:36.691 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:36.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:36.693 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:36.690+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.691827+0000) 2019-10-13T09:16:36.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:36.696+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.700175+0000) 2019-10-13T09:16:38.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:38.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:38.515803+0000) 2019-10-13T09:16:39.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:39.496+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:39.499260+0000) 2019-10-13T09:16:40.348 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:16:40.348 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:40.352 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:16:40.352 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:41.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:41.690+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.693803+0000) 2019-10-13T09:16:41.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:41.690+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.694736+0000) 2019-10-13T09:16:41.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:41.692+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.695760+0000) 2019-10-13T09:16:41.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:41.694+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.696357+0000) 2019-10-13T09:16:41.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:41.694+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.696836+0000) 2019-10-13T09:16:41.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:41.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:41.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:41.694+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.697343+0000) 2019-10-13T09:16:41.699 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:41.694+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.697912+0000) 2019-10-13T09:16:41.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:41.696+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.701803+0000) 2019-10-13T09:16:41.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:41.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.757899+0000) 2019-10-13T09:16:41.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:41.802+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.804116+0000) 2019-10-13T09:16:41.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:41.994+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.997052+0000) 2019-10-13T09:16:42.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:42.216+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:42.220220+0000) 2019-10-13T09:16:45.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:45.022+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:45.025706+0000) 2019-10-13T09:16:46.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:46.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.246833+0000) 2019-10-13T09:16:46.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:46.698+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.701067+0000) 2019-10-13T09:16:46.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:46.698+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.701547+0000) 2019-10-13T09:16:46.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:46.698+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.702258+0000) 2019-10-13T09:16:46.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:46.698+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.702945+0000) 2019-10-13T09:16:46.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:46.696+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.701888+0000) 2019-10-13T09:16:46.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:46.698+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.703262+0000) 2019-10-13T09:16:46.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:46.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:46.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:46.702+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.704201+0000) 2019-10-13T09:16:46.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:46.704+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.709806+0000) 2019-10-13T09:16:48.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:48.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:48.516014+0000) 2019-10-13T09:16:49.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:49.496+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:49.499465+0000) 2019-10-13T09:16:51.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:51.704+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.706271+0000) 2019-10-13T09:16:51.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:51.702+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.707171+0000) 2019-10-13T09:16:51.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:51.706+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.707716+0000) 2019-10-13T09:16:51.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:51.706+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.708065+0000) 2019-10-13T09:16:51.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:51.706+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.708588+0000) 2019-10-13T09:16:51.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:51.706+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.708789+0000) 2019-10-13T09:16:51.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:51.706+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.709228+0000) 2019-10-13T09:16:51.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:51.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:51.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:51.716+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.719782+0000) 2019-10-13T09:16:51.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:51.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.758061+0000) 2019-10-13T09:16:51.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:51.802+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.804291+0000) 2019-10-13T09:16:51.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:51.994+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.997242+0000) 2019-10-13T09:16:52.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:52.224+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:52.226419+0000) 2019-10-13T09:16:55.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:55.022+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:55.025901+0000) 2019-10-13T09:16:56.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:56.242+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.247029+0000) 2019-10-13T09:16:56.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:16:56.710+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.713133+0000) 2019-10-13T09:16:56.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:56.710+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.713645+0000) 2019-10-13T09:16:56.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:56.708+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.714077+0000) 2019-10-13T09:16:56.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:16:56.714+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.715344+0000) 2019-10-13T09:16:56.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:16:56.714+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.715881+0000) 2019-10-13T09:16:56.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:16:56.714+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.716462+0000) 2019-10-13T09:16:56.717 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:16:56.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:16:56.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:16:56.714+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.717165+0000) 2019-10-13T09:16:56.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:16:56.716+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.720758+0000) 2019-10-13T09:16:58.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:16:58.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:58.516217+0000) 2019-10-13T09:16:59.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:16:59.496+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:59.499689+0000) 2019-10-13T09:17:01.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:01.713+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.719073+0000) 2019-10-13T09:17:01.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:01.717+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.720667+0000) 2019-10-13T09:17:01.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:01.716+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.721275+0000) 2019-10-13T09:17:01.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:01.717+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.721509+0000) 2019-10-13T09:17:01.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:01.717+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.721973+0000) 2019-10-13T09:17:01.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:01.717+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.722384+0000) 2019-10-13T09:17:01.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:01.717+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.723225+0000) 2019-10-13T09:17:01.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:01.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71810 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:01.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:01.728+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.731186+0000) 2019-10-13T09:17:01.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:01.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.758251+0000) 2019-10-13T09:17:01.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:01.801+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.804474+0000) 2019-10-13T09:17:01.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:01.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.997416+0000) 2019-10-13T09:17:02.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:02.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:02.232857+0000) 2019-10-13T09:17:05.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:05.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:05.026099+0000) 2019-10-13T09:17:06.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:06.241+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.247241+0000) 2019-10-13T09:17:06.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:06.721+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.727111+0000) 2019-10-13T09:17:06.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:06.725+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.727459+0000) 2019-10-13T09:17:06.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:06.723+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.728233+0000) 2019-10-13T09:17:06.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:06.725+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.727919+0000) 2019-10-13T09:17:06.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:06.725+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.729133+0000) 2019-10-13T09:17:06.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:06.725+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.729579+0000) 2019-10-13T09:17:06.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:06.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:06.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:06.725+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.730080+0000) 2019-10-13T09:17:06.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:06.735+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.738652+0000) 2019-10-13T09:17:08.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:08.513+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:08.516417+0000) 2019-10-13T09:17:09.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:09.495+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:09.499878+0000) 2019-10-13T09:17:10.479 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:17:10.479 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:10.483 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:17:10.483 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:11.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:11.729+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.733041+0000) 2019-10-13T09:17:11.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:11.729+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.733416+0000) 2019-10-13T09:17:11.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:11.729+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.733742+0000) 2019-10-13T09:17:11.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:11.731+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.734470+0000) 2019-10-13T09:17:11.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:11.729+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.734877+0000) 2019-10-13T09:17:11.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:11.733+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.735557+0000) 2019-10-13T09:17:11.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:11.733+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.735339+0000) 2019-10-13T09:17:11.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:11.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:11.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:11.739+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.744008+0000) 2019-10-13T09:17:11.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:11.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.758489+0000) 2019-10-13T09:17:11.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:11.801+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.804659+0000) 2019-10-13T09:17:11.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:11.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.997609+0000) 2019-10-13T09:17:12.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:12.235+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:12.238662+0000) 2019-10-13T09:17:15.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:15.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:15.026305+0000) 2019-10-13T09:17:16.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:16.245+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.247424+0000) 2019-10-13T09:17:16.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:16.737+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.740450+0000) 2019-10-13T09:17:16.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:16.737+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.740895+0000) 2019-10-13T09:17:16.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:16.737+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.741414+0000) 2019-10-13T09:17:16.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:16.737+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.741994+0000) 2019-10-13T09:17:16.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:16.739+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.742595+0000) 2019-10-13T09:17:16.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:16.737+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.743123+0000) 2019-10-13T09:17:16.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:16.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:16.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:16.741+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.744177+0000) 2019-10-13T09:17:16.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:16.747+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.754096+0000) 2019-10-13T09:17:18.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:18.513+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:18.516571+0000) 2019-10-13T09:17:19.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:19.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:19.500080+0000) 2019-10-13T09:17:21.747 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:21.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.746578+0000) 2019-10-13T09:17:21.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:21.745+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.747840+0000) 2019-10-13T09:17:21.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:21.745+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.748282+0000) 2019-10-13T09:17:21.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:21.745+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.748661+0000) 2019-10-13T09:17:21.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:21.743+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.749142+0000) 2019-10-13T09:17:21.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:21.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:21.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:21.745+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.749731+0000) 2019-10-13T09:17:21.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:21.745+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.749903+0000) 2019-10-13T09:17:21.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:21.751+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.757855+0000) 2019-10-13T09:17:21.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:21.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.758664+0000) 2019-10-13T09:17:21.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:21.801+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.804865+0000) 2019-10-13T09:17:21.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:21.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.997829+0000) 2019-10-13T09:17:22.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:22.243+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:22.244917+0000) 2019-10-13T09:17:25.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:25.025+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:25.026507+0000) 2019-10-13T09:17:26.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:26.245+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.247623+0000) 2019-10-13T09:17:26.753 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:26.749+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.752093+0000) 2019-10-13T09:17:26.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:26.749+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.752990+0000) 2019-10-13T09:17:26.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:26.747+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.753396+0000) 2019-10-13T09:17:26.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:26.749+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.753936+0000) 2019-10-13T09:17:26.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:26.749+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.754682+0000) 2019-10-13T09:17:26.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:26.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.755293+0000) 2019-10-13T09:17:26.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:26.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 71973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:26.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:26.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.756215+0000) 2019-10-13T09:17:26.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:26.755+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.761951+0000) 2019-10-13T09:17:28.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:28.513+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:28.516763+0000) 2019-10-13T09:17:29.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:29.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:29.500269+0000) 2019-10-13T09:17:31.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:31.753+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.758563+0000) 2019-10-13T09:17:31.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:31.755+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.759938+0000) 2019-10-13T09:17:31.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:31.757+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.760791+0000) 2019-10-13T09:17:31.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:31.757+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.760515+0000) 2019-10-13T09:17:31.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:31.757+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.761150+0000) 2019-10-13T09:17:31.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:31.757+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.762463+0000) 2019-10-13T09:17:31.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:31.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:31.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:31.757+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.763103+0000) 2019-10-13T09:17:31.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:31.767+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.770202+0000) 2019-10-13T09:17:31.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:31.801+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.805038+0000) 2019-10-13T09:17:31.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:31.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.998006+0000) 2019-10-13T09:17:32.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:32.255+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:32.256190+0000) 2019-10-13T09:17:35.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:35.021+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:35.026709+0000) 2019-10-13T09:17:36.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:36.245+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.247824+0000) 2019-10-13T09:17:36.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:36.761+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.765359+0000) 2019-10-13T09:17:36.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:36.761+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766249+0000) 2019-10-13T09:17:36.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:36.761+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766595+0000) 2019-10-13T09:17:36.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:36.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.767401+0000) 2019-10-13T09:17:36.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:36.765+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.767778+0000) 2019-10-13T09:17:36.771 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:36.763+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.769178+0000) 2019-10-13T09:17:36.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:36.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:36.772 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:36.765+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.769588+0000) 2019-10-13T09:17:36.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:36.767+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.773421+0000) 2019-10-13T09:17:38.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:38.505+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.509347+0000) 2019-10-13T09:17:38.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:38.513+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.516979+0000) 2019-10-13T09:17:39.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:39.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:39.500458+0000) 2019-10-13T09:17:40.581 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:17:40.582 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:40.585 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:17:40.585 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:41.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:41.769+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.772841+0000) 2019-10-13T09:17:41.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:41.769+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.773512+0000) 2019-10-13T09:17:41.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:41.769+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.774606+0000) 2019-10-13T09:17:41.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:41.771+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775244+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:41.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:41.777 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:41.773+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775707+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:41.773+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775984+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:41.773+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.776643+0000) 2019-10-13T09:17:41.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:41.779+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.783833+0000) 2019-10-13T09:17:41.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:41.801+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.805230+0000) 2019-10-13T09:17:41.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:41.993+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.998188+0000) 2019-10-13T09:17:42.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:42.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:42.262526+0000) 2019-10-13T09:17:45.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:45.024+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:45.026907+0000) 2019-10-13T09:17:46.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:46.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.248023+0000) 2019-10-13T09:17:46.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:46.772+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.778003+0000) 2019-10-13T09:17:46.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:46.772+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.779247+0000) 2019-10-13T09:17:46.781 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:46.775+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.779710+0000) 2019-10-13T09:17:46.782 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:46.776+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.780780+0000) 2019-10-13T09:17:46.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:46.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.781034+0000) 2019-10-13T09:17:46.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:46.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:46.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:46.776+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.781862+0000) 2019-10-13T09:17:46.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:46.776+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.782357+0000) 2019-10-13T09:17:46.788 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:46.783+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.787017+0000) 2019-10-13T09:17:48.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:48.508+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.509567+0000) 2019-10-13T09:17:48.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:48.512+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.517173+0000) 2019-10-13T09:17:49.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:49.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:49.500674+0000) 2019-10-13T09:17:51.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:51.779+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.784716+0000) 2019-10-13T09:17:51.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:51.780+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.785933+0000) 2019-10-13T09:17:51.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:51.780+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.786163+0000) 2019-10-13T09:17:51.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:51.780+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.786526+0000) 2019-10-13T09:17:51.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:51.784+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.787549+0000) 2019-10-13T09:17:51.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:51.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:51.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:51.784+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.788526+0000) 2019-10-13T09:17:51.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:51.784+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.788982+0000) 2019-10-13T09:17:51.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:51.787+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.793464+0000) 2019-10-13T09:17:51.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:51.800+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.805410+0000) 2019-10-13T09:17:51.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:51.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.998365+0000) 2019-10-13T09:17:52.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:52.263+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:52.268901+0000) 2019-10-13T09:17:55.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:55.024+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:55.027110+0000) 2019-10-13T09:17:56.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:56.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.248217+0000) 2019-10-13T09:17:56.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:56.788+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.791555+0000) 2019-10-13T09:17:56.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:56.787+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.792389+0000) 2019-10-13T09:17:56.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:17:56.788+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.792679+0000) 2019-10-13T09:17:56.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:17:56.788+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.793550+0000) 2019-10-13T09:17:56.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:56.788+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.793998+0000) 2019-10-13T09:17:56.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:17:56.788+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.794506+0000) 2019-10-13T09:17:56.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:17:56.788+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.794805+0000) 2019-10-13T09:17:56.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:17:56.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:17:56.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:17:56.791+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.796433+0000) 2019-10-13T09:17:58.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:17:58.504+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.509765+0000) 2019-10-13T09:17:58.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:17:58.512+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.517358+0000) 2019-10-13T09:17:59.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:17:59.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:59.500887+0000) 2019-10-13T09:18:01.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:01.792+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.798638+0000) 2019-10-13T09:18:01.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:01.792+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.798993+0000) 2019-10-13T09:18:01.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:01.796+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.799567+0000) 2019-10-13T09:18:01.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:01.796+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.800054+0000) 2019-10-13T09:18:01.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:01.794+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.800878+0000) 2019-10-13T09:18:01.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:01.796+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.801125+0000) 2019-10-13T09:18:01.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:01.796+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.801696+0000) 2019-10-13T09:18:01.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:01.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:01.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:01.798+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.805074+0000) 2019-10-13T09:18:01.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:01.800+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.805546+0000) 2019-10-13T09:18:01.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:01.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.998580+0000) 2019-10-13T09:18:02.276 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:02.274+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:02.275388+0000) 2019-10-13T09:18:05.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:05.024+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:05.027339+0000) 2019-10-13T09:18:06.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:06.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.248410+0000) 2019-10-13T09:18:06.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:06.800+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.805600+0000) 2019-10-13T09:18:06.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:06.802+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.806429+0000) 2019-10-13T09:18:06.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:06.800+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.806852+0000) 2019-10-13T09:18:06.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:06.800+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.807180+0000) 2019-10-13T09:18:06.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:06.804+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.807552+0000) 2019-10-13T09:18:06.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:06.804+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.808172+0000) 2019-10-13T09:18:06.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:06.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:06.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:06.804+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.809053+0000) 2019-10-13T09:18:06.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:06.806+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.813216+0000) 2019-10-13T09:18:08.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:08.504+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.510003+0000) 2019-10-13T09:18:08.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:08.512+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.517556+0000) 2019-10-13T09:18:09.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:09.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:09.501090+0000) 2019-10-13T09:18:10.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:18:10.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:10.872 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:18:10.872 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:11.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:11.804+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.805747+0000) 2019-10-13T09:18:11.813 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:11.808+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.812807+0000) 2019-10-13T09:18:11.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:11.808+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.813807+0000) 2019-10-13T09:18:11.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:11.808+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.813871+0000) 2019-10-13T09:18:11.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:11.808+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.814218+0000) 2019-10-13T09:18:11.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:11.808+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.814635+0000) 2019-10-13T09:18:11.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:11.808+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.815216+0000) 2019-10-13T09:18:11.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:11.810+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.814978+0000) 2019-10-13T09:18:11.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:11.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:11.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:11.814+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.819570+0000) 2019-10-13T09:18:12.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:11.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.998758+0000) 2019-10-13T09:18:12.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:12.278+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:12.281736+0000) 2019-10-13T09:18:15.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:15.024+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:15.027538+0000) 2019-10-13T09:18:16.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:16.248+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.248603+0000) 2019-10-13T09:18:16.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:16.812+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.818789+0000) 2019-10-13T09:18:16.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:16.816+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.820389+0000) 2019-10-13T09:18:16.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:16.816+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.820759+0000) 2019-10-13T09:18:16.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:16.816+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.821193+0000) 2019-10-13T09:18:16.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:16.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:16.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:16.816+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.821682+0000) 2019-10-13T09:18:16.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:16.818+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.823195+0000) 2019-10-13T09:18:16.825 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:16.816+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.822277+0000) 2019-10-13T09:18:16.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:16.822+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.827971+0000) 2019-10-13T09:18:18.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:18.504+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.510222+0000) 2019-10-13T09:18:18.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:18.516+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.517791+0000) 2019-10-13T09:18:19.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:19.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:19.501284+0000) 2019-10-13T09:18:21.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:21.800+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.805946+0000) 2019-10-13T09:18:21.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:21.820+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.825361+0000) 2019-10-13T09:18:21.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:21.820+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.825727+0000) 2019-10-13T09:18:21.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:21.820+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.826047+0000) 2019-10-13T09:18:21.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:21.820+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.826732+0000) 2019-10-13T09:18:21.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:21.820+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.826918+0000) 2019-10-13T09:18:21.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:21.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72329 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:21.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:21.824+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.828002+0000) 2019-10-13T09:18:21.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:21.822+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.829412+0000) 2019-10-13T09:18:21.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:21.830+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.836184+0000) 2019-10-13T09:18:22.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:21.992+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.998936+0000) 2019-10-13T09:18:22.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:22.286+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:22.288221+0000) 2019-10-13T09:18:25.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:25.024+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:25.027756+0000) 2019-10-13T09:18:26.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:26.243+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.248818+0000) 2019-10-13T09:18:26.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:26.826+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.831655+0000) 2019-10-13T09:18:26.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:26.827+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.832520+0000) 2019-10-13T09:18:26.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:26.827+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.833061+0000) 2019-10-13T09:18:26.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:26.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.833673+0000) 2019-10-13T09:18:26.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:26.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:26.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:26.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.834162+0000) 2019-10-13T09:18:26.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:26.827+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.834754+0000) 2019-10-13T09:18:26.839 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:26.827+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.835135+0000) 2019-10-13T09:18:26.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:26.830+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.838002+0000) 2019-10-13T09:18:28.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:28.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.510422+0000) 2019-10-13T09:18:28.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:28.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.517985+0000) 2019-10-13T09:18:29.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:29.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:29.501477+0000) 2019-10-13T09:18:31.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:31.799+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.806141+0000) 2019-10-13T09:18:31.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:31.831+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.836609+0000) 2019-10-13T09:18:31.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:31.831+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.837409+0000) 2019-10-13T09:18:31.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:31.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.837813+0000) 2019-10-13T09:18:31.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:31.834+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.838478+0000) 2019-10-13T09:18:31.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:31.831+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.839081+0000) 2019-10-13T09:18:31.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:31.835+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:31.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:31.835+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.840016+0000) 2019-10-13T09:18:31.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:31.835+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.840509+0000) 2019-10-13T09:18:31.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:31.838+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.846112+0000) 2019-10-13T09:18:32.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:31.991+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.999123+0000) 2019-10-13T09:18:32.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:32.290+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:32.293979+0000) 2019-10-13T09:18:35.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:35.027+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:35.027955+0000) 2019-10-13T09:18:36.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:36.243+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.249016+0000) 2019-10-13T09:18:36.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:36.839+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.842585+0000) 2019-10-13T09:18:36.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:36.839+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.843494+0000) 2019-10-13T09:18:36.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:36.839+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.844344+0000) 2019-10-13T09:18:36.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:36.839+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.844625+0000) 2019-10-13T09:18:36.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:36.838+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.845006+0000) 2019-10-13T09:18:36.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:36.839+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.845545+0000) 2019-10-13T09:18:36.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:36.839+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.846350+0000) 2019-10-13T09:18:36.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:36.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:36.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:36.846+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.850309+0000) 2019-10-13T09:18:38.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:38.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.510635+0000) 2019-10-13T09:18:38.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:38.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.518183+0000) 2019-10-13T09:18:39.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:39.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:39.501666+0000) 2019-10-13T09:18:40.978 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:18:40.979 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:40.982 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:18:40.982 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:41.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:41.803+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.806349+0000) 2019-10-13T09:18:41.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:41.843+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.849876+0000) 2019-10-13T09:18:41.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:41.843+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.850357+0000) 2019-10-13T09:18:41.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:41.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.850787+0000) 2019-10-13T09:18:41.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:41.843+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.851167+0000) 2019-10-13T09:18:41.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:41.847+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.851488+0000) 2019-10-13T09:18:41.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:41.847+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.853112+0000) 2019-10-13T09:18:41.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:41.847+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.853380+0000) 2019-10-13T09:18:41.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:41.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:41.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:41.854+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.858548+0000) 2019-10-13T09:18:42.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:41.995+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.999352+0000) 2019-10-13T09:18:42.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:42.298+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:42.300337+0000) 2019-10-13T09:18:45.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:45.023+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:45.028122+0000) 2019-10-13T09:18:46.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:46.247+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.249235+0000) 2019-10-13T09:18:46.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:46.851+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.857867+0000) 2019-10-13T09:18:46.859 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:46.851+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.858337+0000) 2019-10-13T09:18:46.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:46.851+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.858798+0000) 2019-10-13T09:18:46.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:46.851+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.859234+0000) 2019-10-13T09:18:46.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:46.855+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.859522+0000) 2019-10-13T09:18:46.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:46.854+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.860003+0000) 2019-10-13T09:18:46.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:46.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:46.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:46.855+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.860566+0000) 2019-10-13T09:18:46.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:46.862+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.863367+0000) 2019-10-13T09:18:48.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:48.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.510854+0000) 2019-10-13T09:18:48.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:48.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.518381+0000) 2019-10-13T09:18:49.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:49.498+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:49.501853+0000) 2019-10-13T09:18:51.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:51.803+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.806563+0000) 2019-10-13T09:18:51.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:51.859+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.864307+0000) 2019-10-13T09:18:51.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:51.859+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.865438+0000) 2019-10-13T09:18:51.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:51.859+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.865841+0000) 2019-10-13T09:18:51.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:51.859+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.866171+0000) 2019-10-13T09:18:51.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:51.859+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.866944+0000) 2019-10-13T09:18:51.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:51.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:51.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:51.866+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.867527+0000) 2019-10-13T09:18:51.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:51.863+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.867587+0000) 2019-10-13T09:18:51.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:51.870+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.871607+0000) 2019-10-13T09:18:52.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:51.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.999576+0000) 2019-10-13T09:18:52.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:52.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:52.306686+0000) 2019-10-13T09:18:55.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:55.027+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:55.028350+0000) 2019-10-13T09:18:56.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:56.247+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.249422+0000) 2019-10-13T09:18:56.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:18:56.867+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.870569+0000) 2019-10-13T09:18:56.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:56.871+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.871335+0000) 2019-10-13T09:18:56.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:56.871+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.871811+0000) 2019-10-13T09:18:56.873 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:18:56.871+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.872746+0000) 2019-10-13T09:18:56.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:56.869+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.873864+0000) 2019-10-13T09:18:56.876 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:18:56.871+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.873443+0000) 2019-10-13T09:18:56.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:18:56.871+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.873723+0000) 2019-10-13T09:18:56.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:18:56.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:18:56.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:18:56.877+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.878826+0000) 2019-10-13T09:18:58.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:18:58.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.511072+0000) 2019-10-13T09:18:58.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:18:58.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.518606+0000) 2019-10-13T09:18:59.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:18:59.497+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:59.502050+0000) 2019-10-13T09:19:01.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:01.803+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.806794+0000) 2019-10-13T09:19:01.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:01.875+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.877382+0000) 2019-10-13T09:19:01.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:01.875+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.877874+0000) 2019-10-13T09:19:01.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:01.875+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.878218+0000) 2019-10-13T09:19:01.880 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:01.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.878695+0000) 2019-10-13T09:19:01.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:01.877+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.879328+0000) 2019-10-13T09:19:01.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:01.879+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.880035+0000) 2019-10-13T09:19:01.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:01.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:01.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:01.879+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.881094+0000) 2019-10-13T09:19:01.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:01.885+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.887610+0000) 2019-10-13T09:19:02.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:01.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.999753+0000) 2019-10-13T09:19:02.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:02.309+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:02.312413+0000) 2019-10-13T09:19:05.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:05.027+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:05.028548+0000) 2019-10-13T09:19:06.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:06.247+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.249620+0000) 2019-10-13T09:19:06.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:06.883+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.883404+0000) 2019-10-13T09:19:06.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:06.883+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.883965+0000) 2019-10-13T09:19:06.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:06.883+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.884181+0000) 2019-10-13T09:19:06.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:06.881+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.885232+0000) 2019-10-13T09:19:06.886 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:06.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.885703+0000) 2019-10-13T09:19:06.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:06.883+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.886350+0000) 2019-10-13T09:19:06.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:06.883+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.886812+0000) 2019-10-13T09:19:06.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:06.883+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:06.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:06.893+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.894751+0000) 2019-10-13T09:19:08.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:08.511+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.511271+0000) 2019-10-13T09:19:08.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:08.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.518797+0000) 2019-10-13T09:19:09.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:09.501+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:09.502250+0000) 2019-10-13T09:19:11.057 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:19:11.057 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:11.061 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:19:11.061 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:11.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:11.802+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.806990+0000) 2019-10-13T09:19:11.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:11.886+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.890560+0000) 2019-10-13T09:19:11.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:11.886+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.891141+0000) 2019-10-13T09:19:11.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:11.890+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.891718+0000) 2019-10-13T09:19:11.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:11.889+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.892575+0000) 2019-10-13T09:19:11.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:11.890+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.893040+0000) 2019-10-13T09:19:11.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:11.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:11.896 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:11.890+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.893938+0000) 2019-10-13T09:19:11.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:11.890+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.894413+0000) 2019-10-13T09:19:11.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:11.897+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.898320+0000) 2019-10-13T09:19:12.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:11.998+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.999956+0000) 2019-10-13T09:19:12.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:12.317+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:12.318286+0000) 2019-10-13T09:19:15.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:15.026+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:15.028767+0000) 2019-10-13T09:19:16.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:16.246+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.249839+0000) 2019-10-13T09:19:16.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:16.894+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.896579+0000) 2019-10-13T09:19:16.898 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:16.894+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.896974+0000) 2019-10-13T09:19:16.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:16.893+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.897389+0000) 2019-10-13T09:19:16.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:16.894+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.897856+0000) 2019-10-13T09:19:16.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:16.894+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.898148+0000) 2019-10-13T09:19:16.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:16.894+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.899202+0000) 2019-10-13T09:19:16.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:16.898+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.899797+0000) 2019-10-13T09:19:16.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:16.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:16.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:16.901+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.904557+0000) 2019-10-13T09:19:18.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:18.510+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.511499+0000) 2019-10-13T09:19:18.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:18.518+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.518993+0000) 2019-10-13T09:19:19.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:19.501+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:19.502461+0000) 2019-10-13T09:19:21.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:21.806+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.807199+0000) 2019-10-13T09:19:21.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:21.898+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.903084+0000) 2019-10-13T09:19:21.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:21.898+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.902833+0000) 2019-10-13T09:19:21.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:21.902+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.904007+0000) 2019-10-13T09:19:21.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:21.902+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.904218+0000) 2019-10-13T09:19:21.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:21.902+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.904731+0000) 2019-10-13T09:19:21.906 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:21.902+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.905643+0000) 2019-10-13T09:19:21.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:21.905+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.906295+0000) 2019-10-13T09:19:21.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:21.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:21.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:21.909+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.913575+0000) 2019-10-13T09:19:22.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:21.998+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:22.000131+0000) 2019-10-13T09:19:22.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:22.321+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:22.324704+0000) 2019-10-13T09:19:25.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:25.026+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:25.028971+0000) 2019-10-13T09:19:26.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:26.246+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.250037+0000) 2019-10-13T09:19:26.910 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:26.906+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.908891+0000) 2019-10-13T09:19:26.910 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:26.906+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.909878+0000) 2019-10-13T09:19:26.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:26.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.910286+0000) 2019-10-13T09:19:26.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:26.906+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.910593+0000) 2019-10-13T09:19:26.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:26.909+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.911820+0000) 2019-10-13T09:19:26.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:26.910+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.912138+0000) 2019-10-13T09:19:26.913 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:26.910+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.912388+0000) 2019-10-13T09:19:26.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:26.910+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:26.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:26.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.918167+0000) 2019-10-13T09:19:28.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:28.510+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.511696+0000) 2019-10-13T09:19:28.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:28.514+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.519194+0000) 2019-10-13T09:19:29.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:29.501+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:29.502668+0000) 2019-10-13T09:19:31.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:31.806+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.807403+0000) 2019-10-13T09:19:31.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:31.914+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.917019+0000) 2019-10-13T09:19:31.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:31.914+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.917312+0000) 2019-10-13T09:19:31.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:31.914+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.917714+0000) 2019-10-13T09:19:31.920 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:31.914+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.918129+0000) 2019-10-13T09:19:31.920 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:31.914+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.919090+0000) 2019-10-13T09:19:31.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:31.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:31.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:31.917+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.920239+0000) 2019-10-13T09:19:31.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:31.918+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.920722+0000) 2019-10-13T09:19:31.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:31.921+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.925355+0000) 2019-10-13T09:19:32.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:31.998+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:32.000343+0000) 2019-10-13T09:19:32.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:32.333+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:32.336441+0000) 2019-10-13T09:19:35.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:35.026+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:35.029165+0000) 2019-10-13T09:19:36.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:36.246+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.250140+0000) 2019-10-13T09:19:36.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:36.922+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.924065+0000) 2019-10-13T09:19:36.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:36.922+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.924975+0000) 2019-10-13T09:19:36.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:36.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.925207+0000) 2019-10-13T09:19:36.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:36.922+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.925559+0000) 2019-10-13T09:19:36.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:36.922+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.925865+0000) 2019-10-13T09:19:36.927 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:36.922+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.926099+0000) 2019-10-13T09:19:36.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:36.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:36.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:36.922+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.927077+0000) 2019-10-13T09:19:36.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:36.929+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.934124+0000) 2019-10-13T09:19:38.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:38.510+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.511935+0000) 2019-10-13T09:19:38.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:38.518+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.519393+0000) 2019-10-13T09:19:39.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:39.501+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:39.502894+0000) 2019-10-13T09:19:41.112 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:19:41.112 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:41.116 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:19:41.116 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:41.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:41.806+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.807643+0000) 2019-10-13T09:19:41.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:41.926+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.929892+0000) 2019-10-13T09:19:41.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:41.926+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.930837+0000) 2019-10-13T09:19:41.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:41.926+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.931139+0000) 2019-10-13T09:19:41.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:41.930+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.931491+0000) 2019-10-13T09:19:41.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:41.930+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.932385+0000) 2019-10-13T09:19:41.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:41.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:41.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:41.929+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.932807+0000) 2019-10-13T09:19:41.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:41.930+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.933651+0000) 2019-10-13T09:19:41.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:41.937+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.939824+0000) 2019-10-13T09:19:42.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:41.998+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:42.000531+0000) 2019-10-13T09:19:42.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:42.341+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:42.342682+0000) 2019-10-13T09:19:45.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:45.026+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:45.029360+0000) 2019-10-13T09:19:46.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:46.250+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.250338+0000) 2019-10-13T09:19:46.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:46.934+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.935851+0000) 2019-10-13T09:19:46.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:46.934+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.936416+0000) 2019-10-13T09:19:46.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:46.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.937600+0000) 2019-10-13T09:19:46.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:46.934+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.937886+0000) 2019-10-13T09:19:46.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:46.934+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.938318+0000) 2019-10-13T09:19:46.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:46.937+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.938751+0000) 2019-10-13T09:19:46.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:46.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72883 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:46.940 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:46.934+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.939234+0000) 2019-10-13T09:19:46.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:46.941+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.943465+0000) 2019-10-13T09:19:48.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:48.510+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.512210+0000) 2019-10-13T09:19:48.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:48.518+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.519607+0000) 2019-10-13T09:19:49.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:49.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:49.503128+0000) 2019-10-13T09:19:51.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:51.806+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.807885+0000) 2019-10-13T09:19:51.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:51.940+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.942174+0000) 2019-10-13T09:19:51.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:51.938+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.942525+0000) 2019-10-13T09:19:51.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:51.942+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.943471+0000) 2019-10-13T09:19:51.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:51.942+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.943843+0000) 2019-10-13T09:19:51.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:51.942+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.944610+0000) 2019-10-13T09:19:51.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:51.942+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.945055+0000) 2019-10-13T09:19:51.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:51.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:51.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:51.942+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.945361+0000) 2019-10-13T09:19:51.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:51.948+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.953231+0000) 2019-10-13T09:19:52.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:51.998+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:52.000714+0000) 2019-10-13T09:19:52.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:52.344+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:52.349121+0000) 2019-10-13T09:19:55.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:55.025+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:55.029574+0000) 2019-10-13T09:19:56.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:56.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.250557+0000) 2019-10-13T09:19:56.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:56.945+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.948349+0000) 2019-10-13T09:19:56.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:19:56.945+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.948822+0000) 2019-10-13T09:19:56.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:19:56.945+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.949788+0000) 2019-10-13T09:19:56.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:56.944+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.950071+0000) 2019-10-13T09:19:56.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:19:56.945+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.950658+0000) 2019-10-13T09:19:56.952 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:56.945+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.950941+0000) 2019-10-13T09:19:56.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:19:56.945+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:19:56.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:19:56.949+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.951569+0000) 2019-10-13T09:19:56.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:19:56.952+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.957867+0000) 2019-10-13T09:19:58.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:19:58.509+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.512411+0000) 2019-10-13T09:19:58.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:19:58.517+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.519832+0000) 2019-10-13T09:19:59.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:19:59.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:59.503322+0000) 2019-10-13T09:20:01.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:01.805+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.808086+0000) 2019-10-13T09:20:01.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:01.949+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.954139+0000) 2019-10-13T09:20:01.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:01.949+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.954529+0000) 2019-10-13T09:20:01.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:01.953+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.955467+0000) 2019-10-13T09:20:01.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:01.953+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.956279+0000) 2019-10-13T09:20:01.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:01.953+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.956600+0000) 2019-10-13T09:20:01.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:01.953+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.957095+0000) 2019-10-13T09:20:01.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:01.952+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.957543+0000) 2019-10-13T09:20:01.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:01.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 72982 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:01.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:01.956+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.961815+0000) 2019-10-13T09:20:02.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:01.997+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.000919+0000) 2019-10-13T09:20:02.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:02.352+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.355569+0000) 2019-10-13T09:20:05.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:05.025+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:05.029772+0000) 2019-10-13T09:20:06.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:06.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.250742+0000) 2019-10-13T09:20:06.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:06.957+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.961161+0000) 2019-10-13T09:20:06.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:06.957+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.961560+0000) 2019-10-13T09:20:06.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:06.957+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.962628+0000) 2019-10-13T09:20:06.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:06.957+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.963236+0000) 2019-10-13T09:20:06.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:06.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.963373+0000) 2019-10-13T09:20:06.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:06.960+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.963843+0000) 2019-10-13T09:20:06.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:06.961+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.963937+0000) 2019-10-13T09:20:06.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:06.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73014 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:06.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:06.964+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.967595+0000) 2019-10-13T09:20:08.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:08.509+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.512605+0000) 2019-10-13T09:20:08.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:08.517+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.520045+0000) 2019-10-13T09:20:09.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:09.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:09.503546+0000) 2019-10-13T09:20:11.221 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:20:11.221 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:11.225 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:20:11.225 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:11.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:11.805+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.808290+0000) 2019-10-13T09:20:11.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:11.965+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.968453+0000) 2019-10-13T09:20:11.970 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:11.965+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.968853+0000) 2019-10-13T09:20:11.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:11.965+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.969167+0000) 2019-10-13T09:20:11.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:11.965+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.969824+0000) 2019-10-13T09:20:11.971 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:11.965+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.969512+0000) 2019-10-13T09:20:11.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:11.964+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.970058+0000) 2019-10-13T09:20:11.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:11.965+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.970584+0000) 2019-10-13T09:20:11.973 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:11.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73049 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:11.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:11.976+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.978232+0000) 2019-10-13T09:20:12.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:11.997+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.001113+0000) 2019-10-13T09:20:12.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:12.360+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.362176+0000) 2019-10-13T09:20:15.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:15.025+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:15.030030+0000) 2019-10-13T09:20:16.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:16.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.250932+0000) 2019-10-13T09:20:16.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:16.973+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.975455+0000) 2019-10-13T09:20:16.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:16.973+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.976800+0000) 2019-10-13T09:20:16.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:16.973+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.977162+0000) 2019-10-13T09:20:16.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:16.973+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.977459+0000) 2019-10-13T09:20:16.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:16.973+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.978041+0000) 2019-10-13T09:20:16.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:16.973+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.978593+0000) 2019-10-13T09:20:16.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:16.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:16.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:16.976+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.979693+0000) 2019-10-13T09:20:16.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:16.980+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.984177+0000) 2019-10-13T09:20:18.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:18.509+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.512875+0000) 2019-10-13T09:20:18.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:18.517+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.520269+0000) 2019-10-13T09:20:19.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:19.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:19.503746+0000) 2019-10-13T09:20:21.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:21.805+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.808501+0000) 2019-10-13T09:20:21.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:21.977+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.982734+0000) 2019-10-13T09:20:21.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:21.977+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.983249+0000) 2019-10-13T09:20:21.985 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:21.981+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.983516+0000) 2019-10-13T09:20:21.985 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:21.981+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.984507+0000) 2019-10-13T09:20:21.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:21.981+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.984955+0000) 2019-10-13T09:20:21.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:21.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:21.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:21.980+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.985599+0000) 2019-10-13T09:20:21.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:21.981+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.986049+0000) 2019-10-13T09:20:21.992 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:21.988+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.991610+0000) 2019-10-13T09:20:22.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:21.997+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.001266+0000) 2019-10-13T09:20:22.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:22.364+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.368659+0000) 2019-10-13T09:20:25.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:25.029+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:25.030260+0000) 2019-10-13T09:20:26.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:26.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.251124+0000) 2019-10-13T09:20:26.989 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:26.985+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.988342+0000) 2019-10-13T09:20:26.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:26.985+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.989218+0000) 2019-10-13T09:20:26.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:26.985+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.989377+0000) 2019-10-13T09:20:26.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:26.985+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.990022+0000) 2019-10-13T09:20:26.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:26.985+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.991015+0000) 2019-10-13T09:20:26.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:26.988+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.991211+0000) 2019-10-13T09:20:26.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:26.989+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.991622+0000) 2019-10-13T09:20:26.993 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:26.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:26.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:26.996+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.998427+0000) 2019-10-13T09:20:28.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:28.509+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.513082+0000) 2019-10-13T09:20:28.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:28.517+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.520476+0000) 2019-10-13T09:20:29.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:29.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:29.503955+0000) 2019-10-13T09:20:31.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:31.805+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.808703+0000) 2019-10-13T09:20:31.996 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:31.989+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.995253+0000) 2019-10-13T09:20:31.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:31.993+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.996007+0000) 2019-10-13T09:20:31.997 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:31.993+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.996612+0000) 2019-10-13T09:20:31.998 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:31.993+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.996864+0000) 2019-10-13T09:20:31.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:31.993+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.997231+0000) 2019-10-13T09:20:31.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:31.993+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.997446+0000) 2019-10-13T09:20:31.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:31.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:32.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:31.997+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.001379+0000) 2019-10-13T09:20:32.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:31.996+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.999114+0000) 2019-10-13T09:20:32.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:32.000+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.004983+0000) 2019-10-13T09:20:32.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:32.372+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.374988+0000) 2019-10-13T09:20:35.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:35.025+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:35.030457+0000) 2019-10-13T09:20:36.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:36.244+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.251267+0000) 2019-10-13T09:20:37.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:37.000+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.004488+0000) 2019-10-13T09:20:37.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:36.996+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.001274+0000) 2019-10-13T09:20:37.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:36.996+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.001928+0000) 2019-10-13T09:20:37.008 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:36.996+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.002252+0000) 2019-10-13T09:20:37.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:36.996+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.002464+0000) 2019-10-13T09:20:37.008 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:37.000+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.003360+0000) 2019-10-13T09:20:37.009 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:37.000+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.003583+0000) 2019-10-13T09:20:37.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:37.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:37.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:37.008+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.011279+0000) 2019-10-13T09:20:38.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:38.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.513287+0000) 2019-10-13T09:20:38.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:38.516+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.520703+0000) 2019-10-13T09:20:39.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:39.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:39.504166+0000) 2019-10-13T09:20:41.324 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:20:41.324 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:41.327 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:20:41.328 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:41.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:41.804+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.808977+0000) 2019-10-13T09:20:42.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:41.996+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.001612+0000) 2019-10-13T09:20:42.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:42.004+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.009782+0000) 2019-10-13T09:20:42.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:42.004+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.010508+0000) 2019-10-13T09:20:42.012 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:42.004+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.011122+0000) 2019-10-13T09:20:42.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:42.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.011706+0000) 2019-10-13T09:20:42.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:42.008+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.012252+0000) 2019-10-13T09:20:42.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:42.008+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.012534+0000) 2019-10-13T09:20:42.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:42.008+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.012959+0000) 2019-10-13T09:20:42.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:42.008+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:42.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:42.016+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.018975+0000) 2019-10-13T09:20:42.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:42.380+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.381211+0000) 2019-10-13T09:20:45.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:45.028+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:45.030692+0000) 2019-10-13T09:20:46.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:46.248+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.251489+0000) 2019-10-13T09:20:47.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:47.012+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.016265+0000) 2019-10-13T09:20:47.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:47.011+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.017205+0000) 2019-10-13T09:20:47.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:47.012+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.018197+0000) 2019-10-13T09:20:47.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:47.012+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.018803+0000) 2019-10-13T09:20:47.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:47.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:47.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:47.012+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.019104+0000) 2019-10-13T09:20:47.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:47.016+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.019722+0000) 2019-10-13T09:20:47.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:47.016+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.020089+0000) 2019-10-13T09:20:47.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:47.019+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.023479+0000) 2019-10-13T09:20:48.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:48.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.513475+0000) 2019-10-13T09:20:48.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:48.520+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.520900+0000) 2019-10-13T09:20:49.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:49.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:49.504361+0000) 2019-10-13T09:20:51.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:51.808+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.809227+0000) 2019-10-13T09:20:52.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:51.996+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.001835+0000) 2019-10-13T09:20:52.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:52.016+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.021795+0000) 2019-10-13T09:20:52.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:52.016+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.022124+0000) 2019-10-13T09:20:52.023 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:52.016+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.022719+0000) 2019-10-13T09:20:52.024 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:52.020+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.023341+0000) 2019-10-13T09:20:52.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:52.019+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.023636+0000) 2019-10-13T09:20:52.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:52.020+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.024971+0000) 2019-10-13T09:20:52.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:52.020+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.025462+0000) 2019-10-13T09:20:52.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:52.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:52.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:52.027+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.030918+0000) 2019-10-13T09:20:52.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:52.383+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.387518+0000) 2019-10-13T09:20:55.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:55.028+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:55.030889+0000) 2019-10-13T09:20:56.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:56.248+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.251675+0000) 2019-10-13T09:20:57.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:57.024+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.028562+0000) 2019-10-13T09:20:57.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:57.024+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.028930+0000) 2019-10-13T09:20:57.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:20:57.024+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.030323+0000) 2019-10-13T09:20:57.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:20:57.024+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.030532+0000) 2019-10-13T09:20:57.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:20:57.024+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.031140+0000) 2019-10-13T09:20:57.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:57.027+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.031718+0000) 2019-10-13T09:20:57.033 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:20:57.028+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.032187+0000) 2019-10-13T09:20:57.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:20:57.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:20:57.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:20:57.031+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.036499+0000) 2019-10-13T09:20:58.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:20:58.508+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.513708+0000) 2019-10-13T09:20:58.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:20:58.516+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.521116+0000) 2019-10-13T09:20:59.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:20:59.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:59.504584+0000) 2019-10-13T09:21:01.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:01.808+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.809464+0000) 2019-10-13T09:21:02.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:01.996+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.002033+0000) 2019-10-13T09:21:02.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:02.032+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.035867+0000) 2019-10-13T09:21:02.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:02.032+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.036522+0000) 2019-10-13T09:21:02.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:02.032+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.036870+0000) 2019-10-13T09:21:02.038 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:02.032+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.037204+0000) 2019-10-13T09:21:02.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:02.032+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.037554+0000) 2019-10-13T09:21:02.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:02.032+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.038402+0000) 2019-10-13T09:21:02.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:02.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73383 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:02.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:02.035+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.039272+0000) 2019-10-13T09:21:02.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:02.039+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.044411+0000) 2019-10-13T09:21:02.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:02.387+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.393800+0000) 2019-10-13T09:21:05.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:05.028+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:05.031090+0000) 2019-10-13T09:21:06.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:06.248+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.251894+0000) 2019-10-13T09:21:07.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:07.036+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.041933+0000) 2019-10-13T09:21:07.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:07.036+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.042637+0000) 2019-10-13T09:21:07.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:07.040+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.043515+0000) 2019-10-13T09:21:07.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:07.040+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.043937+0000) 2019-10-13T09:21:07.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:07.040+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.044339+0000) 2019-10-13T09:21:07.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:07.039+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.045469+0000) 2019-10-13T09:21:07.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:07.040+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.045520+0000) 2019-10-13T09:21:07.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:07.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:07.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:07.047+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.052127+0000) 2019-10-13T09:21:08.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:08.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.513909+0000) 2019-10-13T09:21:08.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:08.520+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.521316+0000) 2019-10-13T09:21:09.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:09.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:09.504779+0000) 2019-10-13T09:21:11.418 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:21:11.418 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:11.422 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:21:11.422 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:11.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:11.804+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.809660+0000) 2019-10-13T09:21:12.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:11.996+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.002228+0000) 2019-10-13T09:21:12.050 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:12.044+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.048738+0000) 2019-10-13T09:21:12.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:12.044+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.049419+0000) 2019-10-13T09:21:12.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:12.044+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.049873+0000) 2019-10-13T09:21:12.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:12.047+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.050969+0000) 2019-10-13T09:21:12.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:12.044+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.051257+0000) 2019-10-13T09:21:12.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:12.048+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.051982+0000) 2019-10-13T09:21:12.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:12.048+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.052341+0000) 2019-10-13T09:21:12.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:12.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:12.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:12.059+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.063011+0000) 2019-10-13T09:21:12.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:12.399+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.400190+0000) 2019-10-13T09:21:15.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:15.028+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:15.031291+0000) 2019-10-13T09:21:16.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:16.248+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.252090+0000) 2019-10-13T09:21:17.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:17.051+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.057014+0000) 2019-10-13T09:21:17.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:17.052+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.058144+0000) 2019-10-13T09:21:17.060 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:17.052+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.058463+0000) 2019-10-13T09:21:17.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:17.052+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.058747+0000) 2019-10-13T09:21:17.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:17.052+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.059131+0000) 2019-10-13T09:21:17.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:17.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:17.061 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:17.056+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.059717+0000) 2019-10-13T09:21:17.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:17.056+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.061392+0000) 2019-10-13T09:21:17.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:17.063+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.066315+0000) 2019-10-13T09:21:18.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:18.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.514107+0000) 2019-10-13T09:21:18.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:18.516+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.521526+0000) 2019-10-13T09:21:19.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:19.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:19.504967+0000) 2019-10-13T09:21:21.811 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:21.807+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.809866+0000) 2019-10-13T09:21:22.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:21.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.002432+0000) 2019-10-13T09:21:22.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:22.055+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.062676+0000) 2019-10-13T09:21:22.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:22.055+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.063031+0000) 2019-10-13T09:21:22.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:22.059+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.063216+0000) 2019-10-13T09:21:22.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:22.059+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.064457+0000) 2019-10-13T09:21:22.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:22.059+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.065475+0000) 2019-10-13T09:21:22.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:22.059+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.065904+0000) 2019-10-13T09:21:22.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:22.059+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.066716+0000) 2019-10-13T09:21:22.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:22.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:22.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:22.067+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.070871+0000) 2019-10-13T09:21:22.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:22.403+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.406460+0000) 2019-10-13T09:21:25.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:25.027+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:25.031529+0000) 2019-10-13T09:21:26.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:26.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.252285+0000) 2019-10-13T09:21:27.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:27.063+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.070272+0000) 2019-10-13T09:21:27.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:27.063+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.070600+0000) 2019-10-13T09:21:27.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:27.067+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.071707+0000) 2019-10-13T09:21:27.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:27.067+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.072347+0000) 2019-10-13T09:21:27.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:27.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.072790+0000) 2019-10-13T09:21:27.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:27.067+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.073243+0000) 2019-10-13T09:21:27.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:27.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:27.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:27.067+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.073982+0000) 2019-10-13T09:21:27.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:27.075+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.080985+0000) 2019-10-13T09:21:28.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:28.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.514305+0000) 2019-10-13T09:21:28.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:28.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.521715+0000) 2019-10-13T09:21:29.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:29.503+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:29.505190+0000) 2019-10-13T09:21:31.811 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:31.807+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.810068+0000) 2019-10-13T09:21:32.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:31.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.002631+0000) 2019-10-13T09:21:32.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:32.071+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.076554+0000) 2019-10-13T09:21:32.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:32.071+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.076882+0000) 2019-10-13T09:21:32.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:32.071+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.077259+0000) 2019-10-13T09:21:32.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:32.071+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.078340+0000) 2019-10-13T09:21:32.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:32.071+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.078733+0000) 2019-10-13T09:21:32.081 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:32.075+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.079895+0000) 2019-10-13T09:21:32.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:32.075+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.080233+0000) 2019-10-13T09:21:32.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:32.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:32.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:32.083+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.086954+0000) 2019-10-13T09:21:32.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:32.407+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.412704+0000) 2019-10-13T09:21:35.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:35.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:35.031754+0000) 2019-10-13T09:21:36.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:36.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.252485+0000) 2019-10-13T09:21:37.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:37.079+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.083881+0000) 2019-10-13T09:21:37.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:37.079+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.085107+0000) 2019-10-13T09:21:37.087 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:37.079+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.085288+0000) 2019-10-13T09:21:37.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:37.079+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.085603+0000) 2019-10-13T09:21:37.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:37.079+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.085814+0000) 2019-10-13T09:21:37.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:37.083+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.086405+0000) 2019-10-13T09:21:37.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:37.079+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.087015+0000) 2019-10-13T09:21:37.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:37.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:37.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:37.095+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.098323+0000) 2019-10-13T09:21:38.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:38.507+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.514501+0000) 2019-10-13T09:21:38.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:38.515+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.521908+0000) 2019-10-13T09:21:39.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:39.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:39.505374+0000) 2019-10-13T09:21:41.471 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:21:41.471 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:41.475 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:21:41.475 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:41.811 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:41.803+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.810303+0000) 2019-10-13T09:21:42.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:41.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.002834+0000) 2019-10-13T09:21:42.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:42.087+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.091795+0000) 2019-10-13T09:21:42.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:42.087+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.092402+0000) 2019-10-13T09:21:42.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:42.087+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.092486+0000) 2019-10-13T09:21:42.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:42.086+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.092819+0000) 2019-10-13T09:21:42.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:42.087+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.093739+0000) 2019-10-13T09:21:42.096 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:42.087+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.094164+0000) 2019-10-13T09:21:42.096 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:42.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:42.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:42.087+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.094026+0000) 2019-10-13T09:21:42.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:42.094+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.099836+0000) 2019-10-13T09:21:42.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:42.418+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.419084+0000) 2019-10-13T09:21:45.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:45.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:45.031916+0000) 2019-10-13T09:21:46.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:46.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.252704+0000) 2019-10-13T09:21:47.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:47.091+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.098094+0000) 2019-10-13T09:21:47.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:47.091+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.098879+0000) 2019-10-13T09:21:47.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:47.091+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.098884+0000) 2019-10-13T09:21:47.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:47.095+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.099555+0000) 2019-10-13T09:21:47.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:47.095+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.099413+0000) 2019-10-13T09:21:47.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:47.095+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.100567+0000) 2019-10-13T09:21:47.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:47.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:47.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:47.094+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.101518+0000) 2019-10-13T09:21:47.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:47.102+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.107563+0000) 2019-10-13T09:21:48.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:48.511+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.514701+0000) 2019-10-13T09:21:48.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:48.519+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.522106+0000) 2019-10-13T09:21:49.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:49.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:49.505525+0000) 2019-10-13T09:21:51.811 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:51.807+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.810512+0000) 2019-10-13T09:21:52.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:51.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.003058+0000) 2019-10-13T09:21:52.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:52.103+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.105088+0000) 2019-10-13T09:21:52.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:52.103+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.105163+0000) 2019-10-13T09:21:52.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:52.103+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.105559+0000) 2019-10-13T09:21:52.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:52.103+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.106244+0000) 2019-10-13T09:21:52.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:52.107+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.107509+0000) 2019-10-13T09:21:52.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:52.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:52.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:52.102+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.108247+0000) 2019-10-13T09:21:52.109 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:52.107+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.108734+0000) 2019-10-13T09:21:52.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:52.110+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.116086+0000) 2019-10-13T09:21:52.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:52.418+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.425705+0000) 2019-10-13T09:21:55.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:55.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:55.032149+0000) 2019-10-13T09:21:56.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:56.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.252898+0000) 2019-10-13T09:21:57.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:21:57.107+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.110550+0000) 2019-10-13T09:21:57.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:21:57.111+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.111481+0000) 2019-10-13T09:21:57.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:57.111+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.112040+0000) 2019-10-13T09:21:57.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:21:57.111+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.112867+0000) 2019-10-13T09:21:57.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:21:57.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.112884+0000) 2019-10-13T09:21:57.114 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:57.111+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.113590+0000) 2019-10-13T09:21:57.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:21:57.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:21:57.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:57.110+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.114372+0000) 2019-10-13T09:21:57.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:21:57.114+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.120222+0000) 2019-10-13T09:21:58.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:21:58.511+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.514894+0000) 2019-10-13T09:21:58.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:21:58.519+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.522310+0000) 2019-10-13T09:21:59.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:21:59.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:59.505715+0000) 2019-10-13T09:22:01.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:01.807+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.810735+0000) 2019-10-13T09:22:02.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:02.003+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.003288+0000) 2019-10-13T09:22:02.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:02.115+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.116965+0000) 2019-10-13T09:22:02.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:02.115+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.117393+0000) 2019-10-13T09:22:02.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:02.115+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.118190+0000) 2019-10-13T09:22:02.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:02.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.118967+0000) 2019-10-13T09:22:02.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:02.114+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.119210+0000) 2019-10-13T09:22:02.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:02.119+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.119720+0000) 2019-10-13T09:22:02.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:02.119+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.120262+0000) 2019-10-13T09:22:02.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:02.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:02.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:02.122+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.129009+0000) 2019-10-13T09:22:02.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:02.434+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.435218+0000) 2019-10-13T09:22:05.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:05.030+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:05.032357+0000) 2019-10-13T09:22:06.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:06.250+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.253090+0000) 2019-10-13T09:22:07.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:07.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.124058+0000) 2019-10-13T09:22:07.125 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:07.122+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.124633+0000) 2019-10-13T09:22:07.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:07.122+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.124908+0000) 2019-10-13T09:22:07.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:07.122+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.125325+0000) 2019-10-13T09:22:07.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:07.122+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.126288+0000) 2019-10-13T09:22:07.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:07.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:07.128 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:07.122+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.126986+0000) 2019-10-13T09:22:07.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:07.126+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.127503+0000) 2019-10-13T09:22:07.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:07.126+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.133066+0000) 2019-10-13T09:22:08.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:08.510+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.515119+0000) 2019-10-13T09:22:08.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:08.518+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.522532+0000) 2019-10-13T09:22:09.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:09.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:09.505886+0000) 2019-10-13T09:22:11.604 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:22:11.604 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:11.608 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:22:11.608 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:11.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:11.806+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.810978+0000) 2019-10-13T09:22:12.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:12.002+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.003522+0000) 2019-10-13T09:22:12.131 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:12.126+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.130186+0000) 2019-10-13T09:22:12.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:12.126+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.130538+0000) 2019-10-13T09:22:12.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:12.126+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.130916+0000) 2019-10-13T09:22:12.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:12.130+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.131933+0000) 2019-10-13T09:22:12.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:12.130+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.132356+0000) 2019-10-13T09:22:12.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:12.126+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.132671+0000) 2019-10-13T09:22:12.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:12.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:12.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:12.130+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.133751+0000) 2019-10-13T09:22:12.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:12.134+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.138321+0000) 2019-10-13T09:22:12.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:12.438+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.441345+0000) 2019-10-13T09:22:15.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:15.030+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:15.032509+0000) 2019-10-13T09:22:16.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:16.250+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.253285+0000) 2019-10-13T09:22:17.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:17.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.137305+0000) 2019-10-13T09:22:17.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:17.134+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.137711+0000) 2019-10-13T09:22:17.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:17.134+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.138495+0000) 2019-10-13T09:22:17.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:17.134+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.139038+0000) 2019-10-13T09:22:17.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:17.138+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.140137+0000) 2019-10-13T09:22:17.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:17.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:17.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:17.138+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.140835+0000) 2019-10-13T09:22:17.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:17.134+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.141176+0000) 2019-10-13T09:22:17.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:17.142+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.146582+0000) 2019-10-13T09:22:18.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:18.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.515350+0000) 2019-10-13T09:22:18.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:18.518+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.522736+0000) 2019-10-13T09:22:19.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:19.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:19.506077+0000) 2019-10-13T09:22:21.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:21.810+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.811192+0000) 2019-10-13T09:22:22.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:22.002+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.003732+0000) 2019-10-13T09:22:22.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:22.142+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.144166+0000) 2019-10-13T09:22:22.146 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:22.142+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.144601+0000) 2019-10-13T09:22:22.146 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:22.138+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.145216+0000) 2019-10-13T09:22:22.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:22.142+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.145930+0000) 2019-10-13T09:22:22.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:22.142+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.145651+0000) 2019-10-13T09:22:22.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:22.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:22.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:22.142+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.147241+0000) 2019-10-13T09:22:22.149 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:22.146+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.148263+0000) 2019-10-13T09:22:22.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:22.146+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.153693+0000) 2019-10-13T09:22:22.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:22.446+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.447655+0000) 2019-10-13T09:22:25.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:25.030+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:25.032714+0000) 2019-10-13T09:22:26.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:26.250+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.253438+0000) 2019-10-13T09:22:27.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:27.146+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.149649+0000) 2019-10-13T09:22:27.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:27.146+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.150207+0000) 2019-10-13T09:22:27.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:27.146+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.150880+0000) 2019-10-13T09:22:27.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:27.150+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.151481+0000) 2019-10-13T09:22:27.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:27.150+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.151793+0000) 2019-10-13T09:22:27.153 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:27.150+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.152085+0000) 2019-10-13T09:22:27.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:27.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.152576+0000) 2019-10-13T09:22:27.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:27.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:27.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:27.154+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.156972+0000) 2019-10-13T09:22:28.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:28.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.515550+0000) 2019-10-13T09:22:28.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:28.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.522935+0000) 2019-10-13T09:22:29.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:29.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:29.506329+0000) 2019-10-13T09:22:31.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:31.810+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.811441+0000) 2019-10-13T09:22:32.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:32.002+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.003930+0000) 2019-10-13T09:22:32.161 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:32.154+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.157138+0000) 2019-10-13T09:22:32.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:32.154+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.156846+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:32.154+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.156534+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:32.154+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.157707+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:32.154+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.158111+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:32.154+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.158596+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:32.154+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.158896+0000) 2019-10-13T09:22:32.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:32.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 73985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:32.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:32.166+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.167538+0000) 2019-10-13T09:22:32.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:32.454+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.455053+0000) 2019-10-13T09:22:35.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:35.030+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:35.032950+0000) 2019-10-13T09:22:36.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:36.250+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.253633+0000) 2019-10-13T09:22:37.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:37.158+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.162723+0000) 2019-10-13T09:22:37.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:37.162+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.163354+0000) 2019-10-13T09:22:37.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:37.162+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.164611+0000) 2019-10-13T09:22:37.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:37.161+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.164896+0000) 2019-10-13T09:22:37.166 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:37.162+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.165592+0000) 2019-10-13T09:22:37.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:37.162+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.165830+0000) 2019-10-13T09:22:37.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:37.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:37.167 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:37.162+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.166562+0000) 2019-10-13T09:22:37.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:37.173+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.175897+0000) 2019-10-13T09:22:38.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:38.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.515761+0000) 2019-10-13T09:22:38.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:38.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.523131+0000) 2019-10-13T09:22:39.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:39.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:39.506521+0000) 2019-10-13T09:22:41.711 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:22:41.711 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:41.715 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:22:41.715 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:41.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:41.810+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.811653+0000) 2019-10-13T09:22:42.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:42.002+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.004134+0000) 2019-10-13T09:22:42.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:42.166+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.169292+0000) 2019-10-13T09:22:42.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:42.165+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.169844+0000) 2019-10-13T09:22:42.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:42.166+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.170187+0000) 2019-10-13T09:22:42.172 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:42.166+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.170911+0000) 2019-10-13T09:22:42.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:42.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.171865+0000) 2019-10-13T09:22:42.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:42.170+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.172058+0000) 2019-10-13T09:22:42.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:42.170+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.172971+0000) 2019-10-13T09:22:42.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:42.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:42.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:42.177+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.179495+0000) 2019-10-13T09:22:42.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:42.457+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.461489+0000) 2019-10-13T09:22:45.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:45.030+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:45.033123+0000) 2019-10-13T09:22:46.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:46.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.253864+0000) 2019-10-13T09:22:47.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:47.173+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.176581+0000) 2019-10-13T09:22:47.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:47.173+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.177739+0000) 2019-10-13T09:22:47.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:47.173+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.177938+0000) 2019-10-13T09:22:47.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:47.173+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.178389+0000) 2019-10-13T09:22:47.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:47.177+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.178836+0000) 2019-10-13T09:22:47.180 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:47.177+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.179471+0000) 2019-10-13T09:22:47.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:47.177+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.179715+0000) 2019-10-13T09:22:47.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:47.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:47.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:47.185+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.188553+0000) 2019-10-13T09:22:48.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:48.513+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.515963+0000) 2019-10-13T09:22:48.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:48.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.523356+0000) 2019-10-13T09:22:49.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:49.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:49.506726+0000) 2019-10-13T09:22:51.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:51.809+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.811883+0000) 2019-10-13T09:22:52.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:52.001+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.004360+0000) 2019-10-13T09:22:52.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:52.177+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.183168+0000) 2019-10-13T09:22:52.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:52.181+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.183677+0000) 2019-10-13T09:22:52.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:52.181+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.184417+0000) 2019-10-13T09:22:52.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:52.181+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.184670+0000) 2019-10-13T09:22:52.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:52.181+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.185473+0000) 2019-10-13T09:22:52.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:52.181+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.185128+0000) 2019-10-13T09:22:52.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:52.185+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.186277+0000) 2019-10-13T09:22:52.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:52.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:52.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:52.189+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.192566+0000) 2019-10-13T09:22:52.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:52.465+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.467892+0000) 2019-10-13T09:22:55.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:55.029+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:55.033325+0000) 2019-10-13T09:22:56.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:56.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.254056+0000) 2019-10-13T09:22:57.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:22:57.185+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.190916+0000) 2019-10-13T09:22:57.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:57.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.191407+0000) 2019-10-13T09:22:57.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:22:57.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.191846+0000) 2019-10-13T09:22:57.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:57.189+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.192335+0000) 2019-10-13T09:22:57.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:22:57.189+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.192947+0000) 2019-10-13T09:22:57.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:22:57.189+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.193213+0000) 2019-10-13T09:22:57.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:57.189+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.193363+0000) 2019-10-13T09:22:57.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:22:57.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:22:57.202 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:22:57.197+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.201484+0000) 2019-10-13T09:22:58.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:22:58.513+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.516194+0000) 2019-10-13T09:22:58.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:22:58.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.523584+0000) 2019-10-13T09:22:59.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:22:59.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:59.506915+0000) 2019-10-13T09:23:01.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:01.809+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.812083+0000) 2019-10-13T09:23:02.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:02.001+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.004562+0000) 2019-10-13T09:23:02.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:02.193+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.197696+0000) 2019-10-13T09:23:02.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:02.193+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.197986+0000) 2019-10-13T09:23:02.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:02.193+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.198608+0000) 2019-10-13T09:23:02.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:02.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.199758+0000) 2019-10-13T09:23:02.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:02.197+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.201233+0000) 2019-10-13T09:23:02.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:02.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:02.203 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:02.201+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.202490+0000) 2019-10-13T09:23:02.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:02.197+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.201888+0000) 2019-10-13T09:23:02.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:02.205+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.206662+0000) 2019-10-13T09:23:02.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:02.473+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.474191+0000) 2019-10-13T09:23:05.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:05.029+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:05.033545+0000) 2019-10-13T09:23:06.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:06.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.254248+0000) 2019-10-13T09:23:07.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:07.201+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.206035+0000) 2019-10-13T09:23:07.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:07.201+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.205462+0000) 2019-10-13T09:23:07.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:07.201+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.206309+0000) 2019-10-13T09:23:07.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:07.201+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.207014+0000) 2019-10-13T09:23:07.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:07.205+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.207291+0000) 2019-10-13T09:23:07.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:07.205+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.207554+0000) 2019-10-13T09:23:07.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:07.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:07.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:07.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.209064+0000) 2019-10-13T09:23:07.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:07.213+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.214514+0000) 2019-10-13T09:23:08.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:08.513+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.516422+0000) 2019-10-13T09:23:08.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:08.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.523811+0000) 2019-10-13T09:23:09.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:09.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:09.507110+0000) 2019-10-13T09:23:11.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:11.809+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.812299+0000) 2019-10-13T09:23:11.825 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:23:11.825 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:11.829 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:23:11.829 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:12.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:12.001+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.004784+0000) 2019-10-13T09:23:12.213 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:12.209+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.211579+0000) 2019-10-13T09:23:12.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:12.209+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.212233+0000) 2019-10-13T09:23:12.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:12.209+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.212816+0000) 2019-10-13T09:23:12.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:12.209+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.213198+0000) 2019-10-13T09:23:12.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:12.209+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.213858+0000) 2019-10-13T09:23:12.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:12.209+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.214060+0000) 2019-10-13T09:23:12.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:12.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:12.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:12.209+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.214600+0000) 2019-10-13T09:23:12.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:12.217+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.221787+0000) 2019-10-13T09:23:12.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:12.477+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.480356+0000) 2019-10-13T09:23:15.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:15.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:15.033749+0000) 2019-10-13T09:23:16.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:16.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.254503+0000) 2019-10-13T09:23:17.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:17.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.217696+0000) 2019-10-13T09:23:17.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:17.213+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.218589+0000) 2019-10-13T09:23:17.220 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:17.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.219170+0000) 2019-10-13T09:23:17.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:17.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.219585+0000) 2019-10-13T09:23:17.221 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:17.217+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.220537+0000) 2019-10-13T09:23:17.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:17.217+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.220917+0000) 2019-10-13T09:23:17.223 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:17.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:17.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:17.217+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.221434+0000) 2019-10-13T09:23:17.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:17.225+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.226755+0000) 2019-10-13T09:23:18.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:18.513+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.516640+0000) 2019-10-13T09:23:18.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:18.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.524015+0000) 2019-10-13T09:23:19.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:19.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:19.507295+0000) 2019-10-13T09:23:21.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:21.809+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.812505+0000) 2019-10-13T09:23:22.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:22.001+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.005003+0000) 2019-10-13T09:23:22.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:22.221+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.224247+0000) 2019-10-13T09:23:22.226 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:22.221+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.224482+0000) 2019-10-13T09:23:22.226 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:22.221+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.225697+0000) 2019-10-13T09:23:22.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:22.221+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.226686+0000) 2019-10-13T09:23:22.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:22.225+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.227343+0000) 2019-10-13T09:23:22.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:22.225+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.227422+0000) 2019-10-13T09:23:22.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:22.225+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.228163+0000) 2019-10-13T09:23:22.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:22.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:22.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:22.229+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.232054+0000) 2019-10-13T09:23:22.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:22.485+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.486705+0000) 2019-10-13T09:23:25.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:25.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:25.033945+0000) 2019-10-13T09:23:26.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:26.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.254700+0000) 2019-10-13T09:23:27.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:27.229+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.231709+0000) 2019-10-13T09:23:27.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:27.229+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.232902+0000) 2019-10-13T09:23:27.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:27.229+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.233398+0000) 2019-10-13T09:23:27.234 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:27.229+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.233657+0000) 2019-10-13T09:23:27.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:27.229+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.233955+0000) 2019-10-13T09:23:27.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:27.233+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.234351+0000) 2019-10-13T09:23:27.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:27.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:27.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:27.233+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.235417+0000) 2019-10-13T09:23:27.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:27.237+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.241505+0000) 2019-10-13T09:23:28.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:28.513+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.516801+0000) 2019-10-13T09:23:28.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:28.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.524230+0000) 2019-10-13T09:23:29.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:29.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:29.507526+0000) 2019-10-13T09:23:31.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:31.808+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.812736+0000) 2019-10-13T09:23:32.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:32.000+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.005201+0000) 2019-10-13T09:23:32.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:32.232+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.238513+0000) 2019-10-13T09:23:32.241 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:32.236+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.239507+0000) 2019-10-13T09:23:32.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:32.236+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.239769+0000) 2019-10-13T09:23:32.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:32.236+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.240831+0000) 2019-10-13T09:23:32.242 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:32.236+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.241179+0000) 2019-10-13T09:23:32.243 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:32.236+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.241467+0000) 2019-10-13T09:23:32.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:32.236+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.242207+0000) 2019-10-13T09:23:32.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:32.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:32.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:32.240+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.244465+0000) 2019-10-13T09:23:32.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:32.488+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.493137+0000) 2019-10-13T09:23:35.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:35.032+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:35.034153+0000) 2019-10-13T09:23:36.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:36.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.254892+0000) 2019-10-13T09:23:37.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:37.240+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.245865+0000) 2019-10-13T09:23:37.247 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:37.244+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.246529+0000) 2019-10-13T09:23:37.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:37.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.246855+0000) 2019-10-13T09:23:37.248 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:37.244+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.247967+0000) 2019-10-13T09:23:37.250 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:37.244+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.249015+0000) 2019-10-13T09:23:37.250 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:37.244+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.249391+0000) 2019-10-13T09:23:37.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:37.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:37.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:37.244+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.249831+0000) 2019-10-13T09:23:37.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:37.252+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.254264+0000) 2019-10-13T09:23:38.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:38.516+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.517039+0000) 2019-10-13T09:23:38.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:38.520+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.524458+0000) 2019-10-13T09:23:39.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:39.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:39.507677+0000) 2019-10-13T09:23:41.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:41.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.812952+0000) 2019-10-13T09:23:41.932 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:23:41.932 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:41.936 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:23:41.936 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:42.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:42.000+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.005401+0000) 2019-10-13T09:23:42.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:42.248+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.253062+0000) 2019-10-13T09:23:42.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:42.248+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.253878+0000) 2019-10-13T09:23:42.255 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:42.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.254265+0000) 2019-10-13T09:23:42.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:42.248+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.255136+0000) 2019-10-13T09:23:42.256 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:42.248+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.255148+0000) 2019-10-13T09:23:42.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:42.252+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.255457+0000) 2019-10-13T09:23:42.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:42.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:42.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:42.252+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.256854+0000) 2019-10-13T09:23:42.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:42.256+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.261765+0000) 2019-10-13T09:23:42.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:42.496+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.499374+0000) 2019-10-13T09:23:45.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:45.032+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:45.034359+0000) 2019-10-13T09:23:46.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:46.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.255119+0000) 2019-10-13T09:23:47.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:47.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.259139+0000) 2019-10-13T09:23:47.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:47.256+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.260513+0000) 2019-10-13T09:23:47.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:47.256+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.261003+0000) 2019-10-13T09:23:47.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:47.256+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.261337+0000) 2019-10-13T09:23:47.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:47.256+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.261600+0000) 2019-10-13T09:23:47.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:47.256+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.261969+0000) 2019-10-13T09:23:47.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:47.256+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:47.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:47.256+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.262317+0000) 2019-10-13T09:23:47.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:47.264+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.267620+0000) 2019-10-13T09:23:48.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:48.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.517250+0000) 2019-10-13T09:23:48.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:48.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.524680+0000) 2019-10-13T09:23:49.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:49.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:49.507856+0000) 2019-10-13T09:23:51.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:51.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.813151+0000) 2019-10-13T09:23:52.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:52.000+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.005610+0000) 2019-10-13T09:23:52.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:52.260+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.265994+0000) 2019-10-13T09:23:52.268 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:52.260+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.266665+0000) 2019-10-13T09:23:52.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:52.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.267567+0000) 2019-10-13T09:23:52.269 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:52.264+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.267974+0000) 2019-10-13T09:23:52.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:52.264+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.268521+0000) 2019-10-13T09:23:52.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:52.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:52.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:52.264+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.269019+0000) 2019-10-13T09:23:52.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:52.264+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.269512+0000) 2019-10-13T09:23:52.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:52.276+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.279990+0000) 2019-10-13T09:23:52.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:52.504+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.505713+0000) 2019-10-13T09:23:55.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:55.028+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:55.034556+0000) 2019-10-13T09:23:56.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:56.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.255342+0000) 2019-10-13T09:23:57.273 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:57.268+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.272018+0000) 2019-10-13T09:23:57.273 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:57.268+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.272436+0000) 2019-10-13T09:23:57.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:23:57.268+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.273373+0000) 2019-10-13T09:23:57.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:23:57.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.273712+0000) 2019-10-13T09:23:57.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:23:57.268+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.274809+0000) 2019-10-13T09:23:57.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:23:57.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:23:57.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:23:57.272+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.275374+0000) 2019-10-13T09:23:57.277 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:57.272+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.275672+0000) 2019-10-13T09:23:57.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:23:57.276+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.279149+0000) 2019-10-13T09:23:58.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:23:58.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.517469+0000) 2019-10-13T09:23:58.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:23:58.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.524879+0000) 2019-10-13T09:23:59.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:23:59.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:59.508006+0000) 2019-10-13T09:24:01.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:01.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.813356+0000) 2019-10-13T09:24:02.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:02.000+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.005806+0000) 2019-10-13T09:24:02.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:02.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.277900+0000) 2019-10-13T09:24:02.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:02.276+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.279929+0000) 2019-10-13T09:24:02.281 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:02.276+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.280385+0000) 2019-10-13T09:24:02.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:02.276+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.280683+0000) 2019-10-13T09:24:02.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:02.276+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.281446+0000) 2019-10-13T09:24:02.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:02.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:02.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:02.276+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.281763+0000) 2019-10-13T09:24:02.283 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:02.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.282339+0000) 2019-10-13T09:24:02.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:02.284+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.288707+0000) 2019-10-13T09:24:02.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:02.508+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.512093+0000) 2019-10-13T09:24:05.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:05.032+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:05.034745+0000) 2019-10-13T09:24:06.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:06.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.255531+0000) 2019-10-13T09:24:07.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:07.280+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.286421+0000) 2019-10-13T09:24:07.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:07.280+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.286695+0000) 2019-10-13T09:24:07.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:07.280+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.285671+0000) 2019-10-13T09:24:07.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:07.284+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.287595+0000) 2019-10-13T09:24:07.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:07.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74629 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:07.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:07.284+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.288599+0000) 2019-10-13T09:24:07.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:07.284+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.289206+0000) 2019-10-13T09:24:07.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:07.284+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.289628+0000) 2019-10-13T09:24:07.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:07.292+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.295177+0000) 2019-10-13T09:24:08.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:08.512+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.517685+0000) 2019-10-13T09:24:08.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:08.520+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.525042+0000) 2019-10-13T09:24:09.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:09.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:09.508198+0000) 2019-10-13T09:24:11.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:11.808+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.813562+0000) 2019-10-13T09:24:12.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:12.000+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.006001+0000) 2019-10-13T09:24:12.064 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:24:12.064 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:12.068 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:24:12.068 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:12.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:12.284+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.291088+0000) 2019-10-13T09:24:12.293 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:12.288+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.292168+0000) 2019-10-13T09:24:12.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:12.288+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.292708+0000) 2019-10-13T09:24:12.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:12.288+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.293084+0000) 2019-10-13T09:24:12.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:12.288+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.293816+0000) 2019-10-13T09:24:12.295 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:12.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.294055+0000) 2019-10-13T09:24:12.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:12.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.294409+0000) 2019-10-13T09:24:12.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:12.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:12.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:12.292+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.296960+0000) 2019-10-13T09:24:12.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:12.520+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.521247+0000) 2019-10-13T09:24:15.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:15.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:15.034943+0000) 2019-10-13T09:24:16.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:16.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.255740+0000) 2019-10-13T09:24:17.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:17.291+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.298675+0000) 2019-10-13T09:24:17.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:17.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.298910+0000) 2019-10-13T09:24:17.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:17.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.299466+0000) 2019-10-13T09:24:17.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:17.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.300055+0000) 2019-10-13T09:24:17.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:17.295+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.300686+0000) 2019-10-13T09:24:17.302 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:17.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:17.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:17.296+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.301624+0000) 2019-10-13T09:24:17.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:17.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.302305+0000) 2019-10-13T09:24:17.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:17.300+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.302638+0000) 2019-10-13T09:24:18.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:18.515+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.517898+0000) 2019-10-13T09:24:18.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:18.519+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.525272+0000) 2019-10-13T09:24:19.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:19.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:19.508385+0000) 2019-10-13T09:24:21.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:21.811+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.813769+0000) 2019-10-13T09:24:22.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:21.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.006196+0000) 2019-10-13T09:24:22.306 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:22.299+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.304817+0000) 2019-10-13T09:24:22.306 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:22.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.305338+0000) 2019-10-13T09:24:22.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:22.300+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.305796+0000) 2019-10-13T09:24:22.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:22.299+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.306040+0000) 2019-10-13T09:24:22.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:22.299+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.306676+0000) 2019-10-13T09:24:22.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:22.299+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.307182+0000) 2019-10-13T09:24:22.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:22.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:22.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:22.303+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.307926+0000) 2019-10-13T09:24:22.316 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:22.312+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.314678+0000) 2019-10-13T09:24:22.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:22.524+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.527697+0000) 2019-10-13T09:24:25.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:25.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:25.035148+0000) 2019-10-13T09:24:26.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:26.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.255957+0000) 2019-10-13T09:24:27.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:27.307+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.311418+0000) 2019-10-13T09:24:27.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:27.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.311894+0000) 2019-10-13T09:24:27.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:27.307+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.313070+0000) 2019-10-13T09:24:27.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:27.307+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.313501+0000) 2019-10-13T09:24:27.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:27.307+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.313822+0000) 2019-10-13T09:24:27.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:27.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:27.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:27.307+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.314156+0000) 2019-10-13T09:24:27.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:27.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.315518+0000) 2019-10-13T09:24:27.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:27.315+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.319666+0000) 2019-10-13T09:24:28.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:28.511+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.518112+0000) 2019-10-13T09:24:28.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:28.523+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.525468+0000) 2019-10-13T09:24:29.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:29.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:29.508575+0000) 2019-10-13T09:24:31.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:31.811+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.814001+0000) 2019-10-13T09:24:32.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:31.999+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.006351+0000) 2019-10-13T09:24:32.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:32.311+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.317246+0000) 2019-10-13T09:24:32.319 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:32.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.318185+0000) 2019-10-13T09:24:32.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:32.311+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.318546+0000) 2019-10-13T09:24:32.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:32.315+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.319952+0000) 2019-10-13T09:24:32.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:32.315+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.320230+0000) 2019-10-13T09:24:32.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:32.315+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.320713+0000) 2019-10-13T09:24:32.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:32.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:32.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:32.315+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.320734+0000) 2019-10-13T09:24:32.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:32.319+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.324560+0000) 2019-10-13T09:24:32.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:32.531+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.534747+0000) 2019-10-13T09:24:35.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:35.031+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:35.035346+0000) 2019-10-13T09:24:36.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:36.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.256156+0000) 2019-10-13T09:24:37.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:37.319+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.325373+0000) 2019-10-13T09:24:37.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:37.319+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.325720+0000) 2019-10-13T09:24:37.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:37.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.326260+0000) 2019-10-13T09:24:37.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:37.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.326973+0000) 2019-10-13T09:24:37.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:37.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.327675+0000) 2019-10-13T09:24:37.328 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:37.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:37.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:37.323+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.327811+0000) 2019-10-13T09:24:37.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:37.323+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.328258+0000) 2019-10-13T09:24:37.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:37.327+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.330324+0000) 2019-10-13T09:24:38.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:38.515+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.518316+0000) 2019-10-13T09:24:38.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:38.523+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.525662+0000) 2019-10-13T09:24:39.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:39.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:39.508747+0000) 2019-10-13T09:24:41.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:41.811+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.814231+0000) 2019-10-13T09:24:42.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:42.003+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.006580+0000) 2019-10-13T09:24:42.170 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:24:42.170 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:42.174 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:24:42.174 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:42.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:42.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.332222+0000) 2019-10-13T09:24:42.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:42.327+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.332298+0000) 2019-10-13T09:24:42.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:42.327+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.332565+0000) 2019-10-13T09:24:42.334 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:42.327+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.333101+0000) 2019-10-13T09:24:42.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:42.327+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.333392+0000) 2019-10-13T09:24:42.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:42.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:42.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:42.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.334338+0000) 2019-10-13T09:24:42.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:42.327+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.334976+0000) 2019-10-13T09:24:42.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:42.331+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.338000+0000) 2019-10-13T09:24:42.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:42.539+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.541623+0000) 2019-10-13T09:24:45.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:45.035+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:45.035583+0000) 2019-10-13T09:24:46.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:46.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.256353+0000) 2019-10-13T09:24:47.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:47.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.337361+0000) 2019-10-13T09:24:47.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:47.335+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.337941+0000) 2019-10-13T09:24:47.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:47.335+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.338963+0000) 2019-10-13T09:24:47.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:47.335+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.338983+0000) 2019-10-13T09:24:47.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:47.335+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.339279+0000) 2019-10-13T09:24:47.341 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:47.339+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.339777+0000) 2019-10-13T09:24:47.341 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:47.339+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.340505+0000) 2019-10-13T09:24:47.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:47.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:47.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:47.343+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.346442+0000) 2019-10-13T09:24:48.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:48.515+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.518517+0000) 2019-10-13T09:24:48.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:48.523+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.525839+0000) 2019-10-13T09:24:49.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:49.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:49.508920+0000) 2019-10-13T09:24:51.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:51.811+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.814426+0000) 2019-10-13T09:24:52.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:52.003+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.006801+0000) 2019-10-13T09:24:52.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:52.343+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.343486+0000) 2019-10-13T09:24:52.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:52.343+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.344620+0000) 2019-10-13T09:24:52.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:52.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.344991+0000) 2019-10-13T09:24:52.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:52.343+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.345577+0000) 2019-10-13T09:24:52.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:52.343+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.345653+0000) 2019-10-13T09:24:52.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:52.339+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.345914+0000) 2019-10-13T09:24:52.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:52.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:52.348 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:52.343+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.346928+0000) 2019-10-13T09:24:52.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:52.347+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.352768+0000) 2019-10-13T09:24:52.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:52.547+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.548596+0000) 2019-10-13T09:24:55.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:55.035+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:55.035792+0000) 2019-10-13T09:24:56.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:56.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.256585+0000) 2019-10-13T09:24:57.350 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:57.346+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.348992+0000) 2019-10-13T09:24:57.350 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:24:57.346+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.349563+0000) 2019-10-13T09:24:57.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:24:57.346+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.350220+0000) 2019-10-13T09:24:57.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:57.347+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.350408+0000) 2019-10-13T09:24:57.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:24:57.346+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.350665+0000) 2019-10-13T09:24:57.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:57.346+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.350973+0000) 2019-10-13T09:24:57.352 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:24:57.350+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.351482+0000) 2019-10-13T09:24:57.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:24:57.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 74970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:24:57.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:24:57.355+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.358705+0000) 2019-10-13T09:24:58.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:24:58.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.518709+0000) 2019-10-13T09:24:58.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:24:58.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.526035+0000) 2019-10-13T09:24:59.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:24:59.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:59.509114+0000) 2019-10-13T09:25:01.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:01.810+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.814646+0000) 2019-10-13T09:25:02.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:02.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.007018+0000) 2019-10-13T09:25:02.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:02.354+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.355780+0000) 2019-10-13T09:25:02.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:02.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.357488+0000) 2019-10-13T09:25:02.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:02.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.358268+0000) 2019-10-13T09:25:02.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:02.355+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.358586+0000) 2019-10-13T09:25:02.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:02.354+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.358813+0000) 2019-10-13T09:25:02.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:02.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:02.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:02.354+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.359079+0000) 2019-10-13T09:25:02.361 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:02.358+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.359525+0000) 2019-10-13T09:25:02.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:02.359+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.364328+0000) 2019-10-13T09:25:02.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:02.551+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.555620+0000) 2019-10-13T09:25:05.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:05.034+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:05.036039+0000) 2019-10-13T09:25:06.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:06.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.256774+0000) 2019-10-13T09:25:07.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:07.358+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.361660+0000) 2019-10-13T09:25:07.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:07.358+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.362323+0000) 2019-10-13T09:25:07.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:07.358+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.362359+0000) 2019-10-13T09:25:07.364 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:07.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.363271+0000) 2019-10-13T09:25:07.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:07.362+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.363916+0000) 2019-10-13T09:25:07.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:07.358+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.362944+0000) 2019-10-13T09:25:07.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:07.362+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.364347+0000) 2019-10-13T09:25:07.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:07.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:07.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:07.367+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.370151+0000) 2019-10-13T09:25:08.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:08.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.518937+0000) 2019-10-13T09:25:08.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:08.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.526228+0000) 2019-10-13T09:25:09.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:09.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:09.509286+0000) 2019-10-13T09:25:11.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:11.810+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.814852+0000) 2019-10-13T09:25:12.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:12.002+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.007218+0000) 2019-10-13T09:25:12.294 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:25:12.294 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:12.297 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:25:12.298 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:12.369 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:12.366+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.367927+0000) 2019-10-13T09:25:12.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:12.363+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.368324+0000) 2019-10-13T09:25:12.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:12.366+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.369239+0000) 2019-10-13T09:25:12.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:12.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.370055+0000) 2019-10-13T09:25:12.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:12.366+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.370564+0000) 2019-10-13T09:25:12.372 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:12.366+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.370494+0000) 2019-10-13T09:25:12.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:12.370+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.371397+0000) 2019-10-13T09:25:12.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:12.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:12.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:12.371+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.375894+0000) 2019-10-13T09:25:12.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:12.555+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.561827+0000) 2019-10-13T09:25:15.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:15.034+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:15.036263+0000) 2019-10-13T09:25:16.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:16.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.256927+0000) 2019-10-13T09:25:17.376 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:17.374+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.375459+0000) 2019-10-13T09:25:17.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:17.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.375813+0000) 2019-10-13T09:25:17.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:17.374+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.377242+0000) 2019-10-13T09:25:17.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:17.374+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.377683+0000) 2019-10-13T09:25:17.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:17.374+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.378066+0000) 2019-10-13T09:25:17.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:17.374+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.378466+0000) 2019-10-13T09:25:17.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:17.375+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.378950+0000) 2019-10-13T09:25:17.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:17.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:17.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:17.375+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.380003+0000) 2019-10-13T09:25:18.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:18.514+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.519163+0000) 2019-10-13T09:25:18.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:18.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.526439+0000) 2019-10-13T09:25:19.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:19.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:19.509467+0000) 2019-10-13T09:25:21.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:21.814+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.815052+0000) 2019-10-13T09:25:22.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:22.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.007489+0000) 2019-10-13T09:25:22.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:22.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.382330+0000) 2019-10-13T09:25:22.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:22.378+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.382711+0000) 2019-10-13T09:25:22.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:22.378+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.383001+0000) 2019-10-13T09:25:22.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:22.382+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.384154+0000) 2019-10-13T09:25:22.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:22.382+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.384770+0000) 2019-10-13T09:25:22.386 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:22.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.385225+0000) 2019-10-13T09:25:22.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:22.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:22.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:22.382+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.386446+0000) 2019-10-13T09:25:22.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:22.386+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.390969+0000) 2019-10-13T09:25:22.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:22.562+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.568241+0000) 2019-10-13T09:25:25.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:25.034+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:25.036465+0000) 2019-10-13T09:25:26.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:26.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.257146+0000) 2019-10-13T09:25:27.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:27.386+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.387381+0000) 2019-10-13T09:25:27.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:27.386+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.389034+0000) 2019-10-13T09:25:27.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:27.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.389190+0000) 2019-10-13T09:25:27.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:27.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.389742+0000) 2019-10-13T09:25:27.391 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:27.386+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.389638+0000) 2019-10-13T09:25:27.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:27.386+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.390561+0000) 2019-10-13T09:25:27.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:27.386+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.391319+0000) 2019-10-13T09:25:27.393 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:27.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:27.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:27.394+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.398318+0000) 2019-10-13T09:25:28.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:28.518+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.519429+0000) 2019-10-13T09:25:28.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:28.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.526675+0000) 2019-10-13T09:25:29.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:29.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:29.509635+0000) 2019-10-13T09:25:31.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:31.814+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.815262+0000) 2019-10-13T09:25:32.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:32.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.007687+0000) 2019-10-13T09:25:32.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:32.390+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.394775+0000) 2019-10-13T09:25:32.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:32.390+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.394979+0000) 2019-10-13T09:25:32.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:32.394+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.396264+0000) 2019-10-13T09:25:32.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:32.390+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.397552+0000) 2019-10-13T09:25:32.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:32.394+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.396749+0000) 2019-10-13T09:25:32.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:32.394+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.396994+0000) 2019-10-13T09:25:32.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:32.394+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.396590+0000) 2019-10-13T09:25:32.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:32.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:32.406 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:32.398+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.404572+0000) 2019-10-13T09:25:32.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:32.570+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.574629+0000) 2019-10-13T09:25:35.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:35.034+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:35.036670+0000) 2019-10-13T09:25:36.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:36.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.257377+0000) 2019-10-13T09:25:37.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:37.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.402124+0000) 2019-10-13T09:25:37.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:37.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.402403+0000) 2019-10-13T09:25:37.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:37.398+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.403570+0000) 2019-10-13T09:25:37.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:37.402+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.403786+0000) 2019-10-13T09:25:37.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:37.402+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.404059+0000) 2019-10-13T09:25:37.406 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:37.402+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.404713+0000) 2019-10-13T09:25:37.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:37.398+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.405142+0000) 2019-10-13T09:25:37.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:37.402+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.405087+0000) 2019-10-13T09:25:37.408 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:37.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:38.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:38.518+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.519619+0000) 2019-10-13T09:25:38.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:38.522+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.526902+0000) 2019-10-13T09:25:39.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:39.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:39.509807+0000) 2019-10-13T09:25:41.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:41.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.815497+0000) 2019-10-13T09:25:42.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:42.005+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.007892+0000) 2019-10-13T09:25:42.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:42.405+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.410526+0000) 2019-10-13T09:25:42.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:42.405+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.411087+0000) 2019-10-13T09:25:42.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:42.409+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.411440+0000) 2019-10-13T09:25:42.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:42.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.412360+0000) 2019-10-13T09:25:42.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:42.409+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.412777+0000) 2019-10-13T09:25:42.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:42.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:42.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:42.406+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.413337+0000) 2019-10-13T09:25:42.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:42.409+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.413955+0000) 2019-10-13T09:25:42.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:42.414+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.419405+0000) 2019-10-13T09:25:42.425 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:25:42.425 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:42.433 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:25:42.433 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:42.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:42.578+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.580917+0000) 2019-10-13T09:25:45.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:45.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:45.036876+0000) 2019-10-13T09:25:46.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:46.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.257569+0000) 2019-10-13T09:25:47.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:47.413+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.416639+0000) 2019-10-13T09:25:47.418 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:47.413+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.416823+0000) 2019-10-13T09:25:47.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:47.413+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.417167+0000) 2019-10-13T09:25:47.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:47.413+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.417532+0000) 2019-10-13T09:25:47.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:47.414+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.418195+0000) 2019-10-13T09:25:47.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:47.413+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.417948+0000) 2019-10-13T09:25:47.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:47.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:47.420 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:47.417+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.419412+0000) 2019-10-13T09:25:47.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:47.418+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.425167+0000) 2019-10-13T09:25:48.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:48.517+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.519824+0000) 2019-10-13T09:25:48.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:48.521+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.527088+0000) 2019-10-13T09:25:49.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:49.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:49.509993+0000) 2019-10-13T09:25:51.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:51.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:51.815703+0000) 2019-10-13T09:25:52.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:52.005+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.008096+0000) 2019-10-13T09:25:52.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:52.417+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.422883+0000) 2019-10-13T09:25:52.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:52.421+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.423352+0000) 2019-10-13T09:25:52.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:52.421+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.423773+0000) 2019-10-13T09:25:52.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:52.421+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.424724+0000) 2019-10-13T09:25:52.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:52.418+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.424893+0000) 2019-10-13T09:25:52.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:52.421+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.425267+0000) 2019-10-13T09:25:52.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:52.421+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.425580+0000) 2019-10-13T09:25:52.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:52.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:52.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:52.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.434313+0000) 2019-10-13T09:25:52.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:52.586+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.587462+0000) 2019-10-13T09:25:55.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:55.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:55.037080+0000) 2019-10-13T09:25:56.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:56.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:56.257766+0000) 2019-10-13T09:25:57.430 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:25:57.425+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.429288+0000) 2019-10-13T09:25:57.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:25:57.425+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.429937+0000) 2019-10-13T09:25:57.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:25:57.425+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.430382+0000) 2019-10-13T09:25:57.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:57.425+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.430764+0000) 2019-10-13T09:25:57.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:57.426+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.431518+0000) 2019-10-13T09:25:57.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:57.429+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.431890+0000) 2019-10-13T09:25:57.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:25:57.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.432316+0000) 2019-10-13T09:25:57.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:25:57.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:25:57.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:25:57.434+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.438867+0000) 2019-10-13T09:25:58.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:25:58.517+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.520041+0000) 2019-10-13T09:25:58.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:25:58.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.527325+0000) 2019-10-13T09:25:59.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:25:59.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:59.510194+0000) 2019-10-13T09:26:01.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:01.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:01.815910+0000) 2019-10-13T09:26:02.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:02.005+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.008293+0000) 2019-10-13T09:26:02.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:02.433+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.435802+0000) 2019-10-13T09:26:02.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:02.433+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.436908+0000) 2019-10-13T09:26:02.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:02.433+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.437763+0000) 2019-10-13T09:26:02.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:02.430+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.438030+0000) 2019-10-13T09:26:02.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:02.433+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.438331+0000) 2019-10-13T09:26:02.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:02.433+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.438682+0000) 2019-10-13T09:26:02.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:02.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:02.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:02.437+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.439636+0000) 2019-10-13T09:26:02.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:02.438+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.445216+0000) 2019-10-13T09:26:02.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:02.590+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.593957+0000) 2019-10-13T09:26:05.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:05.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:05.037250+0000) 2019-10-13T09:26:06.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:06.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:06.257963+0000) 2019-10-13T09:26:07.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:07.437+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.442848+0000) 2019-10-13T09:26:07.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:07.441+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.443666+0000) 2019-10-13T09:26:07.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:07.441+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.443630+0000) 2019-10-13T09:26:07.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:07.441+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.444610+0000) 2019-10-13T09:26:07.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:07.441+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.445537+0000) 2019-10-13T09:26:07.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:07.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:07.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:07.446+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.446630+0000) 2019-10-13T09:26:07.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:07.441+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.446897+0000) 2019-10-13T09:26:07.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:07.450+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.452528+0000) 2019-10-13T09:26:08.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:08.517+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.520252+0000) 2019-10-13T09:26:08.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:08.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.527536+0000) 2019-10-13T09:26:09.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:09.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:09.510379+0000) 2019-10-13T09:26:11.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:11.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:11.816109+0000) 2019-10-13T09:26:12.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:12.005+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.008494+0000) 2019-10-13T09:26:12.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:12.446+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.449830+0000) 2019-10-13T09:26:12.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:12.445+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.450190+0000) 2019-10-13T09:26:12.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:12.449+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.451423+0000) 2019-10-13T09:26:12.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:12.449+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.452236+0000) 2019-10-13T09:26:12.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:12.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:12.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:12.449+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.452614+0000) 2019-10-13T09:26:12.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:12.449+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.453085+0000) 2019-10-13T09:26:12.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:12.449+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.453596+0000) 2019-10-13T09:26:12.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:12.454+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.457750+0000) 2019-10-13T09:26:12.498 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:26:12.498 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:12.502 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:26:12.502 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:12.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:12.598+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.600368+0000) 2019-10-13T09:26:15.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:15.033+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:15.037469+0000) 2019-10-13T09:26:16.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:16.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:16.258157+0000) 2019-10-13T09:26:17.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:17.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.456554+0000) 2019-10-13T09:26:17.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:17.453+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.456947+0000) 2019-10-13T09:26:17.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:17.453+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.456695+0000) 2019-10-13T09:26:17.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:17.453+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.457452+0000) 2019-10-13T09:26:17.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:17.457+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.458199+0000) 2019-10-13T09:26:17.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:17.453+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.457778+0000) 2019-10-13T09:26:17.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:17.453+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.458975+0000) 2019-10-13T09:26:17.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:17.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:17.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:17.461+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.465469+0000) 2019-10-13T09:26:18.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:18.517+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.520447+0000) 2019-10-13T09:26:18.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:18.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.527745+0000) 2019-10-13T09:26:19.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:19.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:19.510607+0000) 2019-10-13T09:26:21.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:21.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:21.816263+0000) 2019-10-13T09:26:22.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:22.005+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.008701+0000) 2019-10-13T09:26:22.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:22.461+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.463066+0000) 2019-10-13T09:26:22.465 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:22.461+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.463394+0000) 2019-10-13T09:26:22.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:22.461+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.464140+0000) 2019-10-13T09:26:22.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:22.461+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.464741+0000) 2019-10-13T09:26:22.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:22.461+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.465563+0000) 2019-10-13T09:26:22.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:22.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:22.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:22.461+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.465938+0000) 2019-10-13T09:26:22.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:22.461+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.467013+0000) 2019-10-13T09:26:22.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:22.469+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.471716+0000) 2019-10-13T09:26:22.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:22.605+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.606705+0000) 2019-10-13T09:26:25.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:25.032+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:25.037678+0000) 2019-10-13T09:26:26.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:26.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:26.258376+0000) 2019-10-13T09:26:27.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:27.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.469471+0000) 2019-10-13T09:26:27.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:27.464+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.469920+0000) 2019-10-13T09:26:27.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:27.464+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.470550+0000) 2019-10-13T09:26:27.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:27.469+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.470914+0000) 2019-10-13T09:26:27.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:27.468+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.471837+0000) 2019-10-13T09:26:27.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:27.468+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.472281+0000) 2019-10-13T09:26:27.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:27.468+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.472340+0000) 2019-10-13T09:26:27.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:27.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:27.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:27.477+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.479305+0000) 2019-10-13T09:26:28.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:28.516+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.520648+0000) 2019-10-13T09:26:28.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:28.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.527965+0000) 2019-10-13T09:26:29.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:29.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:29.510798+0000) 2019-10-13T09:26:31.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:31.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:31.816493+0000) 2019-10-13T09:26:32.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:32.008+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.008936+0000) 2019-10-13T09:26:32.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:32.473+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.475738+0000) 2019-10-13T09:26:32.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:32.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.476705+0000) 2019-10-13T09:26:32.478 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:32.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.476948+0000) 2019-10-13T09:26:32.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:32.472+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.477297+0000) 2019-10-13T09:26:32.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:32.472+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.477694+0000) 2019-10-13T09:26:32.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:32.472+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.478462+0000) 2019-10-13T09:26:32.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:32.476+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.479559+0000) 2019-10-13T09:26:32.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:32.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:32.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:32.481+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.486083+0000) 2019-10-13T09:26:32.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:32.609+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.613086+0000) 2019-10-13T09:26:35.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:35.036+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:35.037880+0000) 2019-10-13T09:26:36.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:36.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:36.258570+0000) 2019-10-13T09:26:37.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:37.476+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.482730+0000) 2019-10-13T09:26:37.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:37.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.483184+0000) 2019-10-13T09:26:37.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:37.480+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.483831+0000) 2019-10-13T09:26:37.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:37.480+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.484366+0000) 2019-10-13T09:26:37.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:37.481+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.485164+0000) 2019-10-13T09:26:37.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:37.480+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.485521+0000) 2019-10-13T09:26:37.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:37.480+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.486340+0000) 2019-10-13T09:26:37.487 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:37.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:37.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:37.489+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.493272+0000) 2019-10-13T09:26:38.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:38.516+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.520871+0000) 2019-10-13T09:26:38.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:38.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.528193+0000) 2019-10-13T09:26:39.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:39.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:39.510994+0000) 2019-10-13T09:26:41.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:41.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:41.816698+0000) 2019-10-13T09:26:42.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:42.004+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.009196+0000) 2019-10-13T09:26:42.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:42.484+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.489884+0000) 2019-10-13T09:26:42.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:42.484+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.490600+0000) 2019-10-13T09:26:42.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:42.489+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.490950+0000) 2019-10-13T09:26:42.492 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:42.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.491361+0000) 2019-10-13T09:26:42.492 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:42.488+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.491734+0000) 2019-10-13T09:26:42.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:42.488+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.492367+0000) 2019-10-13T09:26:42.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:42.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:42.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:42.488+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.493334+0000) 2019-10-13T09:26:42.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:42.497+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.498487+0000) 2019-10-13T09:26:42.620 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:42.617+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.619452+0000) 2019-10-13T09:26:42.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:26:42.639 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:42.643 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:26:42.643 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:45.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:45.036+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:45.038076+0000) 2019-10-13T09:26:46.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:46.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:46.258771+0000) 2019-10-13T09:26:47.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:47.492+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.497390+0000) 2019-10-13T09:26:47.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:47.492+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.498568+0000) 2019-10-13T09:26:47.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:47.492+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.499175+0000) 2019-10-13T09:26:47.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:47.496+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.499689+0000) 2019-10-13T09:26:47.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:47.496+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.499551+0000) 2019-10-13T09:26:47.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:47.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:47.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:47.496+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.500037+0000) 2019-10-13T09:26:47.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:47.497+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.500431+0000) 2019-10-13T09:26:47.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:47.501+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.503879+0000) 2019-10-13T09:26:48.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:48.516+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.521094+0000) 2019-10-13T09:26:48.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:48.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.528427+0000) 2019-10-13T09:26:49.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:49.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:49.511220+0000) 2019-10-13T09:26:51.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:51.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:51.816879+0000) 2019-10-13T09:26:52.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:52.004+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.009392+0000) 2019-10-13T09:26:52.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:52.496+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.502966+0000) 2019-10-13T09:26:52.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:52.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.503568+0000) 2019-10-13T09:26:52.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:52.500+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.505006+0000) 2019-10-13T09:26:52.506 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:52.500+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.505303+0000) 2019-10-13T09:26:52.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:52.500+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.505122+0000) 2019-10-13T09:26:52.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:52.501+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.505822+0000) 2019-10-13T09:26:52.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:52.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:52.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:52.500+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.506371+0000) 2019-10-13T09:26:52.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:52.509+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.510666+0000) 2019-10-13T09:26:52.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:52.621+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.626034+0000) 2019-10-13T09:26:55.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:55.036+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:55.038274+0000) 2019-10-13T09:26:56.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:56.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:56.258996+0000) 2019-10-13T09:26:57.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:57.504+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.509243+0000) 2019-10-13T09:26:57.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:26:57.504+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.509878+0000) 2019-10-13T09:26:57.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:26:57.504+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.510851+0000) 2019-10-13T09:26:57.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:26:57.504+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.510867+0000) 2019-10-13T09:26:57.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:57.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.511674+0000) 2019-10-13T09:26:57.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:26:57.508+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.512457+0000) 2019-10-13T09:26:57.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:57.508+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.512655+0000) 2019-10-13T09:26:57.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:26:57.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:26:57.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:26:57.517+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.518480+0000) 2019-10-13T09:26:58.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:26:58.516+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.521319+0000) 2019-10-13T09:26:58.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:26:58.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.528619+0000) 2019-10-13T09:26:59.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:26:59.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:59.511440+0000) 2019-10-13T09:27:01.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:01.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:01.817079+0000) 2019-10-13T09:27:02.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:02.008+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.009595+0000) 2019-10-13T09:27:02.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:02.512+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.516092+0000) 2019-10-13T09:27:02.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:02.512+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.517160+0000) 2019-10-13T09:27:02.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:02.512+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.517435+0000) 2019-10-13T09:27:02.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:02.512+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.517783+0000) 2019-10-13T09:27:02.519 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:02.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.518482+0000) 2019-10-13T09:27:02.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:02.517+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.518819+0000) 2019-10-13T09:27:02.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:02.512+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.519026+0000) 2019-10-13T09:27:02.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:02.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:02.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:02.521+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.525964+0000) 2019-10-13T09:27:02.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:02.629+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.632328+0000) 2019-10-13T09:27:05.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:05.036+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:05.038479+0000) 2019-10-13T09:27:06.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:06.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:06.259187+0000) 2019-10-13T09:27:07.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:07.519+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.523433+0000) 2019-10-13T09:27:07.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:07.519+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.524180+0000) 2019-10-13T09:27:07.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:07.519+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.524592+0000) 2019-10-13T09:27:07.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:07.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.525147+0000) 2019-10-13T09:27:07.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:07.519+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.525260+0000) 2019-10-13T09:27:07.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:07.525+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.526280+0000) 2019-10-13T09:27:07.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:07.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:07.528 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:07.519+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.526595+0000) 2019-10-13T09:27:07.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:07.529+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.531724+0000) 2019-10-13T09:27:08.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:08.519+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.521546+0000) 2019-10-13T09:27:08.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:08.523+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.528815+0000) 2019-10-13T09:27:09.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:09.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:09.511640+0000) 2019-10-13T09:27:11.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:11.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:11.817285+0000) 2019-10-13T09:27:12.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:12.007+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.009832+0000) 2019-10-13T09:27:12.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:12.523+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.530165+0000) 2019-10-13T09:27:12.531 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:12.523+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.530460+0000) 2019-10-13T09:27:12.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:12.523+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.531209+0000) 2019-10-13T09:27:12.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:12.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.531743+0000) 2019-10-13T09:27:12.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:12.527+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.531978+0000) 2019-10-13T09:27:12.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:12.527+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.532395+0000) 2019-10-13T09:27:12.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:12.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:12.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:12.527+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.533511+0000) 2019-10-13T09:27:12.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:12.532+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.538037+0000) 2019-10-13T09:27:12.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:12.636+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.638923+0000) 2019-10-13T09:27:12.711 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:27:12.712 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:12.715 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:27:12.716 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:15.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:15.035+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:15.038705+0000) 2019-10-13T09:27:16.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:16.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:16.259405+0000) 2019-10-13T09:27:17.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:17.531+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.536236+0000) 2019-10-13T09:27:17.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:17.531+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.536559+0000) 2019-10-13T09:27:17.538 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:17.531+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.537132+0000) 2019-10-13T09:27:17.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:17.531+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.537639+0000) 2019-10-13T09:27:17.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:17.531+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.538828+0000) 2019-10-13T09:27:17.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:17.536+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.539183+0000) 2019-10-13T09:27:17.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:17.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:17.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:17.535+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.540018+0000) 2019-10-13T09:27:17.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:17.540+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.545239+0000) 2019-10-13T09:27:18.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:18.515+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.521719+0000) 2019-10-13T09:27:18.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:18.523+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.529014+0000) 2019-10-13T09:27:19.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:19.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:19.511835+0000) 2019-10-13T09:27:21.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:21.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:21.817515+0000) 2019-10-13T09:27:22.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:22.003+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.010041+0000) 2019-10-13T09:27:22.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:22.540+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.543445+0000) 2019-10-13T09:27:22.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:22.539+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.544090+0000) 2019-10-13T09:27:22.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:22.539+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.545024+0000) 2019-10-13T09:27:22.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:22.539+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.545139+0000) 2019-10-13T09:27:22.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:22.539+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.545763+0000) 2019-10-13T09:27:22.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:22.539+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.546122+0000) 2019-10-13T09:27:22.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:22.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:22.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:22.539+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.546826+0000) 2019-10-13T09:27:22.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:22.548+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.550599+0000) 2019-10-13T09:27:22.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:22.640+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.645253+0000) 2019-10-13T09:27:25.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:25.035+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:25.038929+0000) 2019-10-13T09:27:26.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:26.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:26.259614+0000) 2019-10-13T09:27:27.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:27.543+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.550109+0000) 2019-10-13T09:27:27.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:27.548+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.550828+0000) 2019-10-13T09:27:27.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:27.543+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.550502+0000) 2019-10-13T09:27:27.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:27.543+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.551182+0000) 2019-10-13T09:27:27.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:27.547+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.551847+0000) 2019-10-13T09:27:27.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:27.547+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 75988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:27.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:27.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.552722+0000) 2019-10-13T09:27:27.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:27.547+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.553438+0000) 2019-10-13T09:27:27.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:27.556+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.560402+0000) 2019-10-13T09:27:28.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:28.519+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.521906+0000) 2019-10-13T09:27:28.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:28.527+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.529205+0000) 2019-10-13T09:27:29.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:29.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:29.512030+0000) 2019-10-13T09:27:31.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:31.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:31.817712+0000) 2019-10-13T09:27:32.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:32.007+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.010241+0000) 2019-10-13T09:27:32.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:32.551+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.555165+0000) 2019-10-13T09:27:32.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:32.551+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.557268+0000) 2019-10-13T09:27:32.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:32.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.557854+0000) 2019-10-13T09:27:32.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:32.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.558200+0000) 2019-10-13T09:27:32.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:32.551+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.558614+0000) 2019-10-13T09:27:32.560 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:32.551+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.558832+0000) 2019-10-13T09:27:32.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:32.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:32.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:32.551+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.559113+0000) 2019-10-13T09:27:32.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:32.560+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.563446+0000) 2019-10-13T09:27:32.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:32.648+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.650995+0000) 2019-10-13T09:27:35.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:35.035+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:35.039148+0000) 2019-10-13T09:27:36.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:36.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:36.259847+0000) 2019-10-13T09:27:37.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:37.559+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.562149+0000) 2019-10-13T09:27:37.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:37.560+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.563330+0000) 2019-10-13T09:27:37.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:37.563+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.563860+0000) 2019-10-13T09:27:37.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:37.563+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.564661+0000) 2019-10-13T09:27:37.566 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:37.563+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.565272+0000) 2019-10-13T09:27:37.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:37.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:37.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:37.563+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.566216+0000) 2019-10-13T09:27:37.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:37.563+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.566686+0000) 2019-10-13T09:27:37.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:37.568+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.570517+0000) 2019-10-13T09:27:38.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:38.519+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.522126+0000) 2019-10-13T09:27:38.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:38.527+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.529409+0000) 2019-10-13T09:27:39.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:39.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:39.512228+0000) 2019-10-13T09:27:41.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:41.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:41.817914+0000) 2019-10-13T09:27:42.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:42.007+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.010438+0000) 2019-10-13T09:27:42.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:42.567+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.569439+0000) 2019-10-13T09:27:42.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:42.564+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.569581+0000) 2019-10-13T09:27:42.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:42.567+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.570041+0000) 2019-10-13T09:27:42.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:42.571+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.571390+0000) 2019-10-13T09:27:42.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:42.571+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.571812+0000) 2019-10-13T09:27:42.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:42.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.571931+0000) 2019-10-13T09:27:42.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:42.571+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.572240+0000) 2019-10-13T09:27:42.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:42.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:42.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:42.576+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.579822+0000) 2019-10-13T09:27:42.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:42.652+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.657138+0000) 2019-10-13T09:27:42.767 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:27:42.767 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:42.771 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:27:42.771 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:45.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:45.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:45.039383+0000) 2019-10-13T09:27:46.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:46.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:46.260068+0000) 2019-10-13T09:27:47.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:47.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.575634+0000) 2019-10-13T09:27:47.577 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:47.575+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.576242+0000) 2019-10-13T09:27:47.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:47.575+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.577047+0000) 2019-10-13T09:27:47.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:47.575+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.577415+0000) 2019-10-13T09:27:47.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:47.575+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.577907+0000) 2019-10-13T09:27:47.581 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:47.576+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.579012+0000) 2019-10-13T09:27:47.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:47.579+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.579494+0000) 2019-10-13T09:27:47.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:47.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:47.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:47.584+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.586836+0000) 2019-10-13T09:27:48.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:48.519+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.522328+0000) 2019-10-13T09:27:48.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:48.527+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.529606+0000) 2019-10-13T09:27:49.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:49.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:49.512443+0000) 2019-10-13T09:27:51.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:51.814+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:51.818150+0000) 2019-10-13T09:27:52.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:52.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.010636+0000) 2019-10-13T09:27:52.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:52.582+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.583646+0000) 2019-10-13T09:27:52.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:52.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.584014+0000) 2019-10-13T09:27:52.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:52.582+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.584651+0000) 2019-10-13T09:27:52.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:52.582+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.585200+0000) 2019-10-13T09:27:52.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:52.582+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.586069+0000) 2019-10-13T09:27:52.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:52.584+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.586345+0000) 2019-10-13T09:27:52.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:52.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:52.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:52.582+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.586696+0000) 2019-10-13T09:27:52.596 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:52.592+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.595421+0000) 2019-10-13T09:27:52.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:52.660+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.663476+0000) 2019-10-13T09:27:55.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:55.038+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:55.039622+0000) 2019-10-13T09:27:56.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:56.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:56.260266+0000) 2019-10-13T09:27:57.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:27:57.586+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.590102+0000) 2019-10-13T09:27:57.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:27:57.586+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.590962+0000) 2019-10-13T09:27:57.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:57.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.591497+0000) 2019-10-13T09:27:57.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:27:57.590+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.591693+0000) 2019-10-13T09:27:57.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:27:57.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.591993+0000) 2019-10-13T09:27:57.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:57.590+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.592660+0000) 2019-10-13T09:27:57.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:57.590+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.593229+0000) 2019-10-13T09:27:57.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:27:57.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:27:57.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:27:57.596+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.599047+0000) 2019-10-13T09:27:58.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:27:58.518+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.522538+0000) 2019-10-13T09:27:58.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:27:58.526+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.529794+0000) 2019-10-13T09:27:59.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:27:59.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:59.512661+0000) 2019-10-13T09:28:01.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:01.814+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:01.818354+0000) 2019-10-13T09:28:02.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:02.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.010861+0000) 2019-10-13T09:28:02.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:02.594+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.597097+0000) 2019-10-13T09:28:02.598 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:02.594+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.597644+0000) 2019-10-13T09:28:02.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:02.592+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.597715+0000) 2019-10-13T09:28:02.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:02.594+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.598015+0000) 2019-10-13T09:28:02.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:02.594+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.598786+0000) 2019-10-13T09:28:02.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:02.598+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.599563+0000) 2019-10-13T09:28:02.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:02.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:02.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:02.598+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.600367+0000) 2019-10-13T09:28:02.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:02.600+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.604948+0000) 2019-10-13T09:28:02.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:02.668+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.669767+0000) 2019-10-13T09:28:05.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:05.038+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:05.039855+0000) 2019-10-13T09:28:06.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:06.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:06.260498+0000) 2019-10-13T09:28:07.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:07.598+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.603004+0000) 2019-10-13T09:28:07.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:07.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.603364+0000) 2019-10-13T09:28:07.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:07.602+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.603948+0000) 2019-10-13T09:28:07.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:07.602+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.604297+0000) 2019-10-13T09:28:07.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:07.599+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.604964+0000) 2019-10-13T09:28:07.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:07.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:07.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:07.602+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.606310+0000) 2019-10-13T09:28:07.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:07.602+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.606063+0000) 2019-10-13T09:28:07.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:07.607+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.612225+0000) 2019-10-13T09:28:08.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:08.518+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.522737+0000) 2019-10-13T09:28:08.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:08.526+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.529997+0000) 2019-10-13T09:28:09.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:09.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:09.512869+0000) 2019-10-13T09:28:11.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:11.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:11.818588+0000) 2019-10-13T09:28:12.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:12.010+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.011059+0000) 2019-10-13T09:28:12.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:12.606+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.609358+0000) 2019-10-13T09:28:12.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:12.606+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.609946+0000) 2019-10-13T09:28:12.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:12.606+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.610384+0000) 2019-10-13T09:28:12.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:12.606+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.610752+0000) 2019-10-13T09:28:12.612 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:12.610+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.611467+0000) 2019-10-13T09:28:12.613 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:12.607+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.612085+0000) 2019-10-13T09:28:12.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:12.610+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.612696+0000) 2019-10-13T09:28:12.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:12.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:12.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:12.615+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.620428+0000) 2019-10-13T09:28:12.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:12.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.675997+0000) 2019-10-13T09:28:12.882 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:28:12.882 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:12.886 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:28:12.886 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:15.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:15.038+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:15.040057+0000) 2019-10-13T09:28:16.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:16.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:16.260715+0000) 2019-10-13T09:28:17.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:17.614+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.616822+0000) 2019-10-13T09:28:17.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:17.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.617587+0000) 2019-10-13T09:28:17.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:17.614+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.617908+0000) 2019-10-13T09:28:17.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:17.614+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.619072+0000) 2019-10-13T09:28:17.620 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:17.615+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.619397+0000) 2019-10-13T09:28:17.622 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:17.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:17.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:17.618+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.619741+0000) 2019-10-13T09:28:17.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:17.618+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.620149+0000) 2019-10-13T09:28:17.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:17.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.624885+0000) 2019-10-13T09:28:18.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:18.518+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.522929+0000) 2019-10-13T09:28:18.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:18.526+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.530171+0000) 2019-10-13T09:28:19.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:19.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:19.513079+0000) 2019-10-13T09:28:21.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:21.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:21.818787+0000) 2019-10-13T09:28:22.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:22.006+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.011277+0000) 2019-10-13T09:28:22.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:22.622+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.623455+0000) 2019-10-13T09:28:22.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:22.622+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.623534+0000) 2019-10-13T09:28:22.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:22.622+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.623884+0000) 2019-10-13T09:28:22.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:22.622+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.624237+0000) 2019-10-13T09:28:22.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:22.622+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.624584+0000) 2019-10-13T09:28:22.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:22.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.625898+0000) 2019-10-13T09:28:22.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:22.622+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.626327+0000) 2019-10-13T09:28:22.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:22.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:22.634 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:22.627+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.633449+0000) 2019-10-13T09:28:22.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:22.679+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.682401+0000) 2019-10-13T09:28:25.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:25.038+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:25.040259+0000) 2019-10-13T09:28:26.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:26.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:26.260919+0000) 2019-10-13T09:28:27.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:27.626+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.629926+0000) 2019-10-13T09:28:27.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:27.626+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.630863+0000) 2019-10-13T09:28:27.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:27.626+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.631267+0000) 2019-10-13T09:28:27.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:27.630+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.631649+0000) 2019-10-13T09:28:27.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:27.630+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.632511+0000) 2019-10-13T09:28:27.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:27.630+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.632624+0000) 2019-10-13T09:28:27.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:27.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:27.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:27.627+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.632280+0000) 2019-10-13T09:28:27.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:27.635+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.638750+0000) 2019-10-13T09:28:28.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:28.522+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.523073+0000) 2019-10-13T09:28:28.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:28.526+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.530302+0000) 2019-10-13T09:28:29.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:29.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:29.513281+0000) 2019-10-13T09:28:31.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:31.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:31.818985+0000) 2019-10-13T09:28:32.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:32.010+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.011508+0000) 2019-10-13T09:28:32.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:32.633+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.636836+0000) 2019-10-13T09:28:32.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:32.633+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.637510+0000) 2019-10-13T09:28:32.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:32.633+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.637975+0000) 2019-10-13T09:28:32.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:32.633+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.638312+0000) 2019-10-13T09:28:32.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:32.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.638653+0000) 2019-10-13T09:28:32.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:32.637+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.639433+0000) 2019-10-13T09:28:32.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:32.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76433 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:32.641 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:32.635+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.639953+0000) 2019-10-13T09:28:32.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:32.639+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.645631+0000) 2019-10-13T09:28:32.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:32.683+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.688344+0000) 2019-10-13T09:28:35.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:35.037+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:35.040474+0000) 2019-10-13T09:28:36.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:36.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:36.261139+0000) 2019-10-13T09:28:37.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:37.637+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.643048+0000) 2019-10-13T09:28:37.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:37.641+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.643390+0000) 2019-10-13T09:28:37.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:37.641+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.644227+0000) 2019-10-13T09:28:37.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:37.641+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.645323+0000) 2019-10-13T09:28:37.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:37.641+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.645597+0000) 2019-10-13T09:28:37.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:37.641+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.645859+0000) 2019-10-13T09:28:37.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:37.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.646187+0000) 2019-10-13T09:28:37.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:37.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:37.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:37.647+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.653322+0000) 2019-10-13T09:28:38.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:38.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.523291+0000) 2019-10-13T09:28:38.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:38.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.530500+0000) 2019-10-13T09:28:39.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:39.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:39.513536+0000) 2019-10-13T09:28:41.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:41.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:41.819185+0000) 2019-10-13T09:28:42.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:42.009+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.011712+0000) 2019-10-13T09:28:42.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:42.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.649766+0000) 2019-10-13T09:28:42.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:42.645+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.649546+0000) 2019-10-13T09:28:42.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:42.647+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.650481+0000) 2019-10-13T09:28:42.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:42.645+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.650758+0000) 2019-10-13T09:28:42.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:42.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.651581+0000) 2019-10-13T09:28:42.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:42.649+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.652087+0000) 2019-10-13T09:28:42.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:42.649+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.652249+0000) 2019-10-13T09:28:42.654 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:42.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:42.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:42.655+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.659250+0000) 2019-10-13T09:28:42.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:42.691+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.694528+0000) 2019-10-13T09:28:42.992 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:28:42.992 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:42.996 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:28:42.996 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:45.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:45.037+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:45.040678+0000) 2019-10-13T09:28:46.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:46.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:46.261367+0000) 2019-10-13T09:28:47.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:47.651+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.656368+0000) 2019-10-13T09:28:47.658 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:47.653+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.656875+0000) 2019-10-13T09:28:47.658 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:47.653+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.657555+0000) 2019-10-13T09:28:47.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:47.653+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.658171+0000) 2019-10-13T09:28:47.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:47.653+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.658834+0000) 2019-10-13T09:28:47.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:47.653+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.659011+0000) 2019-10-13T09:28:47.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:47.657+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.659896+0000) 2019-10-13T09:28:47.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:47.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:47.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:47.659+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.664622+0000) 2019-10-13T09:28:48.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:48.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.523565+0000) 2019-10-13T09:28:48.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:48.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.530635+0000) 2019-10-13T09:28:49.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:49.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:49.513735+0000) 2019-10-13T09:28:51.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:51.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:51.819391+0000) 2019-10-13T09:28:52.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:52.009+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.011915+0000) 2019-10-13T09:28:52.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:52.661+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.664174+0000) 2019-10-13T09:28:52.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:52.659+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.664594+0000) 2019-10-13T09:28:52.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:52.661+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.664946+0000) 2019-10-13T09:28:52.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:52.661+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.665771+0000) 2019-10-13T09:28:52.667 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:52.661+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.666721+0000) 2019-10-13T09:28:52.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:52.661+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.667062+0000) 2019-10-13T09:28:52.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:52.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:52.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:52.665+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.667786+0000) 2019-10-13T09:28:52.674 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:52.667+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.673772+0000) 2019-10-13T09:28:52.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:52.695+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.698745+0000) 2019-10-13T09:28:55.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:55.037+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:55.040914+0000) 2019-10-13T09:28:56.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:56.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:56.261556+0000) 2019-10-13T09:28:57.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:28:57.665+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.670533+0000) 2019-10-13T09:28:57.672 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:28:57.669+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.671414+0000) 2019-10-13T09:28:57.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:57.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.671860+0000) 2019-10-13T09:28:57.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:28:57.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.672437+0000) 2019-10-13T09:28:57.674 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:28:57.669+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.673334+0000) 2019-10-13T09:28:57.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:57.669+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.673973+0000) 2019-10-13T09:28:57.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:57.669+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.674471+0000) 2019-10-13T09:28:57.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:28:57.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:28:57.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:28:57.675+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.679207+0000) 2019-10-13T09:28:58.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:28:58.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.523734+0000) 2019-10-13T09:28:58.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:28:58.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.530830+0000) 2019-10-13T09:28:59.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:28:59.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:59.513964+0000) 2019-10-13T09:29:01.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:01.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:01.819596+0000) 2019-10-13T09:29:02.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:02.009+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.012115+0000) 2019-10-13T09:29:02.679 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:02.673+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.677993+0000) 2019-10-13T09:29:02.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:02.673+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.677738+0000) 2019-10-13T09:29:02.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:02.673+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.679026+0000) 2019-10-13T09:29:02.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:02.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.679412+0000) 2019-10-13T09:29:02.681 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:02.677+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.680166+0000) 2019-10-13T09:29:02.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:02.677+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.680456+0000) 2019-10-13T09:29:02.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:02.677+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.681318+0000) 2019-10-13T09:29:02.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:02.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76636 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:02.687 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:02.682+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.686251+0000) 2019-10-13T09:29:02.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:02.698+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.703191+0000) 2019-10-13T09:29:05.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:05.037+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:05.041118+0000) 2019-10-13T09:29:06.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:06.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:06.261774+0000) 2019-10-13T09:29:07.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:07.681+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.685180+0000) 2019-10-13T09:29:07.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:07.678+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.686021+0000) 2019-10-13T09:29:07.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:07.681+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.686471+0000) 2019-10-13T09:29:07.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:07.681+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.686830+0000) 2019-10-13T09:29:07.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:07.685+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.687339+0000) 2019-10-13T09:29:07.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:07.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:07.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:07.685+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.688057+0000) 2019-10-13T09:29:07.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:07.685+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.688408+0000) 2019-10-13T09:29:07.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:07.690+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.694436+0000) 2019-10-13T09:29:08.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:08.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.523909+0000) 2019-10-13T09:29:08.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:08.525+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.531005+0000) 2019-10-13T09:29:09.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:09.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:09.514186+0000) 2019-10-13T09:29:11.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:11.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:11.819806+0000) 2019-10-13T09:29:12.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:12.009+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.012331+0000) 2019-10-13T09:29:12.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:12.685+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.691213+0000) 2019-10-13T09:29:12.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:12.689+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.692536+0000) 2019-10-13T09:29:12.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:12.689+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.693263+0000) 2019-10-13T09:29:12.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:12.689+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.693388+0000) 2019-10-13T09:29:12.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:12.689+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.693695+0000) 2019-10-13T09:29:12.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:12.689+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.694093+0000) 2019-10-13T09:29:12.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:12.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:12.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:12.690+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.695149+0000) 2019-10-13T09:29:12.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:12.694+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.701791+0000) 2019-10-13T09:29:12.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:12.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.707696+0000) 2019-10-13T09:29:13.046 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:29:13.046 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:13.050 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:29:13.050 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:15.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:15.041+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:15.041340+0000) 2019-10-13T09:29:16.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:16.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:16.262002+0000) 2019-10-13T09:29:17.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:17.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.698242+0000) 2019-10-13T09:29:17.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:17.694+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.698554+0000) 2019-10-13T09:29:17.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:17.696+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.699734+0000) 2019-10-13T09:29:17.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:17.696+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.699778+0000) 2019-10-13T09:29:17.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:17.696+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.700066+0000) 2019-10-13T09:29:17.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:17.696+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.700343+0000) 2019-10-13T09:29:17.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:17.696+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.700899+0000) 2019-10-13T09:29:17.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:17.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:17.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:17.702+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.707443+0000) 2019-10-13T09:29:18.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:18.520+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.524119+0000) 2019-10-13T09:29:18.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:18.524+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.531179+0000) 2019-10-13T09:29:19.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:19.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:19.514413+0000) 2019-10-13T09:29:21.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:21.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:21.820007+0000) 2019-10-13T09:29:22.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:22.008+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.012536+0000) 2019-10-13T09:29:22.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:22.700+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.705550+0000) 2019-10-13T09:29:22.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:22.700+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.706414+0000) 2019-10-13T09:29:22.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:22.702+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.706577+0000) 2019-10-13T09:29:22.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:22.700+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.706954+0000) 2019-10-13T09:29:22.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:22.704+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.707740+0000) 2019-10-13T09:29:22.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:22.704+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.708465+0000) 2019-10-13T09:29:22.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:22.704+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.708900+0000) 2019-10-13T09:29:22.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:22.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:22.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:22.706+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.713742+0000) 2019-10-13T09:29:22.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:22.706+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.713841+0000) 2019-10-13T09:29:25.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:25.040+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:25.041537+0000) 2019-10-13T09:29:26.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:26.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:26.262149+0000) 2019-10-13T09:29:27.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:27.706+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.713500+0000) 2019-10-13T09:29:27.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:27.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.714113+0000) 2019-10-13T09:29:27.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:27.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.714410+0000) 2019-10-13T09:29:27.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:27.708+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.715268+0000) 2019-10-13T09:29:27.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:27.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.715528+0000) 2019-10-13T09:29:27.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:27.712+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.716028+0000) 2019-10-13T09:29:27.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:27.712+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.716932+0000) 2019-10-13T09:29:27.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:27.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:27.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:27.714+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.721242+0000) 2019-10-13T09:29:28.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:28.520+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.524324+0000) 2019-10-13T09:29:28.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:28.528+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.531363+0000) 2019-10-13T09:29:29.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:29.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:29.514604+0000) 2019-10-13T09:29:31.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:31.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:31.820260+0000) 2019-10-13T09:29:32.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:32.012+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.012767+0000) 2019-10-13T09:29:32.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:32.710+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.714033+0000) 2019-10-13T09:29:32.721 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:32.716+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.720541+0000) 2019-10-13T09:29:32.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:32.716+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.720172+0000) 2019-10-13T09:29:32.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:32.714+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.721109+0000) 2019-10-13T09:29:32.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:32.716+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.721621+0000) 2019-10-13T09:29:32.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:32.716+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.722281+0000) 2019-10-13T09:29:32.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:32.716+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.722709+0000) 2019-10-13T09:29:32.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:32.720+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.723839+0000) 2019-10-13T09:29:32.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:32.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:32.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:32.722+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.727317+0000) 2019-10-13T09:29:35.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:35.040+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:35.041743+0000) 2019-10-13T09:29:36.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:36.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:36.262339+0000) 2019-10-13T09:29:37.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:37.724+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.727479+0000) 2019-10-13T09:29:37.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:37.724+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.728909+0000) 2019-10-13T09:29:37.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:37.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.729148+0000) 2019-10-13T09:29:37.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:37.722+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.729619+0000) 2019-10-13T09:29:37.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:37.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.730359+0000) 2019-10-13T09:29:37.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:37.724+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.730878+0000) 2019-10-13T09:29:37.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:37.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76867 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:37.732 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:37.728+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.731525+0000) 2019-10-13T09:29:37.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:37.730+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.737981+0000) 2019-10-13T09:29:38.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:38.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.524530+0000) 2019-10-13T09:29:38.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:38.528+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.531533+0000) 2019-10-13T09:29:39.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:39.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:39.514816+0000) 2019-10-13T09:29:41.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:41.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:41.820466+0000) 2019-10-13T09:29:42.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:42.012+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.012969+0000) 2019-10-13T09:29:42.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:42.718+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.720789+0000) 2019-10-13T09:29:42.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:42.728+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.734348+0000) 2019-10-13T09:29:42.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:42.728+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.734944+0000) 2019-10-13T09:29:42.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:42.732+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.735681+0000) 2019-10-13T09:29:42.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:42.732+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.736396+0000) 2019-10-13T09:29:42.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:42.732+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.736341+0000) 2019-10-13T09:29:42.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:42.730+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.736673+0000) 2019-10-13T09:29:42.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:42.732+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.737250+0000) 2019-10-13T09:29:42.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:42.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76900 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:42.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:42.738+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.745142+0000) 2019-10-13T09:29:43.194 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:29:43.194 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:43.197 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:29:43.197 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:45.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:45.036+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:45.041938+0000) 2019-10-13T09:29:46.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:46.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:46.262530+0000) 2019-10-13T09:29:47.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:47.736+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.741026+0000) 2019-10-13T09:29:47.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:47.736+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.741922+0000) 2019-10-13T09:29:47.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:47.736+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.742369+0000) 2019-10-13T09:29:47.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:47.736+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.742846+0000) 2019-10-13T09:29:47.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:47.736+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.743188+0000) 2019-10-13T09:29:47.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:47.740+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.743657+0000) 2019-10-13T09:29:47.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:47.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:47.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:47.742+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.744355+0000) 2019-10-13T09:29:47.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:47.750+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.750601+0000) 2019-10-13T09:29:48.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:48.520+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.524701+0000) 2019-10-13T09:29:48.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:48.528+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.531674+0000) 2019-10-13T09:29:49.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:49.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:49.515008+0000) 2019-10-13T09:29:51.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:51.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:51.820705+0000) 2019-10-13T09:29:52.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:52.008+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.013199+0000) 2019-10-13T09:29:52.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:52.726+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.727253+0000) 2019-10-13T09:29:52.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:52.740+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.746546+0000) 2019-10-13T09:29:52.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:52.744+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.747728+0000) 2019-10-13T09:29:52.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:52.746+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.748042+0000) 2019-10-13T09:29:52.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:52.744+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.748405+0000) 2019-10-13T09:29:52.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:52.744+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.748916+0000) 2019-10-13T09:29:52.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:52.744+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.749149+0000) 2019-10-13T09:29:52.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:52.744+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.749934+0000) 2019-10-13T09:29:52.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:52.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:52.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:52.754+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.757091+0000) 2019-10-13T09:29:55.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:55.040+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:55.042137+0000) 2019-10-13T09:29:56.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:56.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:56.262724+0000) 2019-10-13T09:29:57.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:57.748+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.753717+0000) 2019-10-13T09:29:57.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:29:57.748+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.755258+0000) 2019-10-13T09:29:57.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:57.753+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.755887+0000) 2019-10-13T09:29:57.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:29:57.752+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.756588+0000) 2019-10-13T09:29:57.758 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:29:57.752+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.756860+0000) 2019-10-13T09:29:57.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:29:57.752+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.757197+0000) 2019-10-13T09:29:57.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:29:57.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 76999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:29:57.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:57.752+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.757593+0000) 2019-10-13T09:29:57.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:29:57.757+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.761616+0000) 2019-10-13T09:29:58.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:29:58.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.524876+0000) 2019-10-13T09:29:58.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:29:58.528+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.531851+0000) 2019-10-13T09:29:59.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:29:59.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:59.515212+0000) 2019-10-13T09:30:01.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:01.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:01.820908+0000) 2019-10-13T09:30:02.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:02.011+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.013432+0000) 2019-10-13T09:30:02.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:02.729+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.733613+0000) 2019-10-13T09:30:02.761 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:02.755+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.760253+0000) 2019-10-13T09:30:02.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:02.755+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.760815+0000) 2019-10-13T09:30:02.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:02.755+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.760849+0000) 2019-10-13T09:30:02.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:02.755+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.762473+0000) 2019-10-13T09:30:02.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:02.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.762578+0000) 2019-10-13T09:30:02.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:02.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.763510+0000) 2019-10-13T09:30:02.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:02.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:02.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:02.759+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.763902+0000) 2019-10-13T09:30:02.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:02.769+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.770315+0000) 2019-10-13T09:30:05.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:05.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:05.042337+0000) 2019-10-13T09:30:06.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:06.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:06.262868+0000) 2019-10-13T09:30:07.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:07.759+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.766641+0000) 2019-10-13T09:30:07.768 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:07.765+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.766934+0000) 2019-10-13T09:30:07.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:07.763+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.767788+0000) 2019-10-13T09:30:07.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:07.763+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.768344+0000) 2019-10-13T09:30:07.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:07.763+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.769339+0000) 2019-10-13T09:30:07.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:07.763+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.769576+0000) 2019-10-13T09:30:07.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:07.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:07.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:07.763+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.770349+0000) 2019-10-13T09:30:07.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:07.773+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.776241+0000) 2019-10-13T09:30:08.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:08.519+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.525047+0000) 2019-10-13T09:30:08.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:08.527+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.532017+0000) 2019-10-13T09:30:09.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:09.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:09.515406+0000) 2019-10-13T09:30:11.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:11.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:11.821158+0000) 2019-10-13T09:30:12.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:12.011+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.013653+0000) 2019-10-13T09:30:12.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:12.737+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.740076+0000) 2019-10-13T09:30:12.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:12.769+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.772346+0000) 2019-10-13T09:30:12.774 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:12.767+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.772972+0000) 2019-10-13T09:30:12.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:12.767+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.773597+0000) 2019-10-13T09:30:12.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:12.767+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.774148+0000) 2019-10-13T09:30:12.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:12.767+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.775187+0000) 2019-10-13T09:30:12.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:12.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.775908+0000) 2019-10-13T09:30:12.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:12.771+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.776048+0000) 2019-10-13T09:30:12.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:12.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:12.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:12.777+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.780608+0000) 2019-10-13T09:30:13.317 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:30:13.317 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:13.321 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:30:13.321 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:15.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:15.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:15.042537+0000) 2019-10-13T09:30:16.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:16.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:16.263061+0000) 2019-10-13T09:30:17.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:17.775+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.780930+0000) 2019-10-13T09:30:17.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:17.775+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.781258+0000) 2019-10-13T09:30:17.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:17.775+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.781878+0000) 2019-10-13T09:30:17.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:17.775+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.783127+0000) 2019-10-13T09:30:17.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:17.775+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.783084+0000) 2019-10-13T09:30:17.785 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:17.781+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.784157+0000) 2019-10-13T09:30:17.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:17.779+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.784474+0000) 2019-10-13T09:30:17.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:17.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:17.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:17.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.791118+0000) 2019-10-13T09:30:18.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:18.523+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.525273+0000) 2019-10-13T09:30:18.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:18.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.532207+0000) 2019-10-13T09:30:19.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:19.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:19.515608+0000) 2019-10-13T09:30:21.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:21.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:21.821415+0000) 2019-10-13T09:30:22.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:22.007+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.013892+0000) 2019-10-13T09:30:22.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:22.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.746452+0000) 2019-10-13T09:30:22.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:22.783+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.788163+0000) 2019-10-13T09:30:22.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:22.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.788444+0000) 2019-10-13T09:30:22.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:22.783+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.788816+0000) 2019-10-13T09:30:22.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:22.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.790354+0000) 2019-10-13T09:30:22.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:22.783+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.790704+0000) 2019-10-13T09:30:22.793 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:22.783+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.791163+0000) 2019-10-13T09:30:22.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:22.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:22.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:22.789+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.791849+0000) 2019-10-13T09:30:22.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:22.793+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.796970+0000) 2019-10-13T09:30:25.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:25.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:25.042743+0000) 2019-10-13T09:30:26.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:26.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:26.263260+0000) 2019-10-13T09:30:27.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:27.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.794569+0000) 2019-10-13T09:30:27.796 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:27.793+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.795856+0000) 2019-10-13T09:30:27.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:27.791+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.796101+0000) 2019-10-13T09:30:27.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:27.791+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.796535+0000) 2019-10-13T09:30:27.798 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:27.791+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.797319+0000) 2019-10-13T09:30:27.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:27.791+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.797635+0000) 2019-10-13T09:30:27.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:27.791+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.797829+0000) 2019-10-13T09:30:27.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:27.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:27.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:27.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.808541+0000) 2019-10-13T09:30:28.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:28.523+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.525478+0000) 2019-10-13T09:30:28.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:28.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.532382+0000) 2019-10-13T09:30:29.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:29.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:29.515806+0000) 2019-10-13T09:30:31.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:31.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:31.821644+0000) 2019-10-13T09:30:32.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:32.011+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.014089+0000) 2019-10-13T09:30:32.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:32.749+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.752873+0000) 2019-10-13T09:30:32.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:32.797+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.801024+0000) 2019-10-13T09:30:32.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:32.799+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.802155+0000) 2019-10-13T09:30:32.803 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:32.799+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.802604+0000) 2019-10-13T09:30:32.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:32.799+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.802930+0000) 2019-10-13T09:30:32.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:32.803+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.803338+0000) 2019-10-13T09:30:32.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:32.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:32.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:32.803+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.804711+0000) 2019-10-13T09:30:32.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:32.803+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.805432+0000) 2019-10-13T09:30:32.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:32.809+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.810272+0000) 2019-10-13T09:30:35.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:35.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:35.042977+0000) 2019-10-13T09:30:36.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:36.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:36.263528+0000) 2019-10-13T09:30:37.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:37.807+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.807962+0000) 2019-10-13T09:30:37.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:37.807+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.808664+0000) 2019-10-13T09:30:37.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:37.807+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.808965+0000) 2019-10-13T09:30:37.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:37.805+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.809395+0000) 2019-10-13T09:30:37.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:37.807+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.809871+0000) 2019-10-13T09:30:37.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:37.807+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.810042+0000) 2019-10-13T09:30:37.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:37.807+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.810768+0000) 2019-10-13T09:30:37.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:37.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:37.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:37.817+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.818331+0000) 2019-10-13T09:30:38.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:38.523+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.525686+0000) 2019-10-13T09:30:38.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:38.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.532559+0000) 2019-10-13T09:30:39.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:39.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:39.516010+0000) 2019-10-13T09:30:41.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:41.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:41.821842+0000) 2019-10-13T09:30:42.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:42.011+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.014312+0000) 2019-10-13T09:30:42.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:42.757+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.759236+0000) 2019-10-13T09:30:42.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:42.810+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.814596+0000) 2019-10-13T09:30:42.816 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:42.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.815270+0000) 2019-10-13T09:30:42.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:42.814+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.815835+0000) 2019-10-13T09:30:42.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:42.813+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.816383+0000) 2019-10-13T09:30:42.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:42.814+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.817000+0000) 2019-10-13T09:30:42.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:42.814+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.817256+0000) 2019-10-13T09:30:42.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:42.814+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.817966+0000) 2019-10-13T09:30:42.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:42.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:42.825 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:42.821+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.824612+0000) 2019-10-13T09:30:43.503 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:30:43.503 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:43.507 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:30:43.507 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:45.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:45.038+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:45.043219+0000) 2019-10-13T09:30:46.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:46.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:46.263752+0000) 2019-10-13T09:30:47.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:47.818+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.822029+0000) 2019-10-13T09:30:47.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:47.818+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.822260+0000) 2019-10-13T09:30:47.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:47.818+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.822632+0000) 2019-10-13T09:30:47.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:47.818+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.823152+0000) 2019-10-13T09:30:47.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:47.820+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.823692+0000) 2019-10-13T09:30:47.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:47.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.824575+0000) 2019-10-13T09:30:47.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:47.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:47.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:47.822+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.825370+0000) 2019-10-13T09:30:47.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:47.828+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.830933+0000) 2019-10-13T09:30:48.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:48.522+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.525936+0000) 2019-10-13T09:30:48.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:48.530+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.532731+0000) 2019-10-13T09:30:49.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:49.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:49.516211+0000) 2019-10-13T09:30:51.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:51.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:51.822058+0000) 2019-10-13T09:30:52.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:52.010+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.014516+0000) 2019-10-13T09:30:52.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:52.768+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.770777+0000) 2019-10-13T09:30:52.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:52.824+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.829746+0000) 2019-10-13T09:30:52.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:52.826+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.829666+0000) 2019-10-13T09:30:52.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:52.826+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.830217+0000) 2019-10-13T09:30:52.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:52.826+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.830430+0000) 2019-10-13T09:30:52.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:52.826+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.830775+0000) 2019-10-13T09:30:52.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:52.830+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.831347+0000) 2019-10-13T09:30:52.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:52.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:52.834 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:52.830+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.831817+0000) 2019-10-13T09:30:52.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:52.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.837690+0000) 2019-10-13T09:30:55.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:55.042+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:55.043442+0000) 2019-10-13T09:30:56.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:56.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:56.263942+0000) 2019-10-13T09:30:57.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:30:57.830+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.834878+0000) 2019-10-13T09:30:57.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:30:57.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.836368+0000) 2019-10-13T09:30:57.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:57.834+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.836971+0000) 2019-10-13T09:30:57.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:30:57.834+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.837273+0000) 2019-10-13T09:30:57.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:57.834+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.837611+0000) 2019-10-13T09:30:57.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:57.836+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.838122+0000) 2019-10-13T09:30:57.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:30:57.834+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:30:57.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:30:57.834+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.838438+0000) 2019-10-13T09:30:57.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:30:57.836+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.841601+0000) 2019-10-13T09:30:58.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:30:58.522+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.526110+0000) 2019-10-13T09:30:58.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:30:58.530+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.532906+0000) 2019-10-13T09:30:59.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:30:59.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:59.516409+0000) 2019-10-13T09:31:01.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:01.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:01.822254+0000) 2019-10-13T09:31:02.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:02.010+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.014727+0000) 2019-10-13T09:31:02.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:02.772+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.777171+0000) 2019-10-13T09:31:02.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:02.838+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.841903+0000) 2019-10-13T09:31:02.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:02.840+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.842785+0000) 2019-10-13T09:31:02.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:02.842+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.843594+0000) 2019-10-13T09:31:02.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:02.842+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.843733+0000) 2019-10-13T09:31:02.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:02.842+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.843980+0000) 2019-10-13T09:31:02.845 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:02.842+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.844361+0000) 2019-10-13T09:31:02.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:02.842+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.845237+0000) 2019-10-13T09:31:02.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:02.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:02.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:02.852+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.854761+0000) 2019-10-13T09:31:05.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:05.042+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:05.043671+0000) 2019-10-13T09:31:06.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:06.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:06.264129+0000) 2019-10-13T09:31:07.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:07.846+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.849378+0000) 2019-10-13T09:31:07.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:07.844+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.849710+0000) 2019-10-13T09:31:07.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:07.846+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.850130+0000) 2019-10-13T09:31:07.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:07.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.851084+0000) 2019-10-13T09:31:07.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:07.850+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.851293+0000) 2019-10-13T09:31:07.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:07.850+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.852019+0000) 2019-10-13T09:31:07.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:07.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.852466+0000) 2019-10-13T09:31:07.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:07.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:07.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:07.856+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.858381+0000) 2019-10-13T09:31:08.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:08.522+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.526286+0000) 2019-10-13T09:31:08.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:08.530+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.533084+0000) 2019-10-13T09:31:09.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:09.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:09.516604+0000) 2019-10-13T09:31:11.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:11.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:11.822453+0000) 2019-10-13T09:31:12.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:12.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.014975+0000) 2019-10-13T09:31:12.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:12.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.787949+0000) 2019-10-13T09:31:12.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:12.850+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.855017+0000) 2019-10-13T09:31:12.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:12.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.855589+0000) 2019-10-13T09:31:12.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:12.854+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.856601+0000) 2019-10-13T09:31:12.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:12.854+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.857777+0000) 2019-10-13T09:31:12.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:12.856+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.858466+0000) 2019-10-13T09:31:12.860 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:12.854+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.858830+0000) 2019-10-13T09:31:12.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:12.854+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.859122+0000) 2019-10-13T09:31:12.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:12.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:12.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:12.860+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.864955+0000) 2019-10-13T09:31:13.581 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:31:13.581 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:13.585 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:31:13.585 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:15.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:15.042+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:15.043873+0000) 2019-10-13T09:31:16.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:16.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:16.264323+0000) 2019-10-13T09:31:17.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:17.858+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.862834+0000) 2019-10-13T09:31:17.864 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:17.858+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.863262+0000) 2019-10-13T09:31:17.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:17.862+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.863616+0000) 2019-10-13T09:31:17.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:17.862+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.864181+0000) 2019-10-13T09:31:17.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:17.860+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.864914+0000) 2019-10-13T09:31:17.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:17.862+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.865469+0000) 2019-10-13T09:31:17.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:17.862+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.865827+0000) 2019-10-13T09:31:17.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:17.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:17.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:17.868+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.873726+0000) 2019-10-13T09:31:18.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:18.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.526458+0000) 2019-10-13T09:31:18.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:18.530+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.533254+0000) 2019-10-13T09:31:19.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:19.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:19.516818+0000) 2019-10-13T09:31:21.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:21.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:21.822670+0000) 2019-10-13T09:31:22.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:22.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.015193+0000) 2019-10-13T09:31:22.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:22.792+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.794317+0000) 2019-10-13T09:31:22.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:22.868+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.870853+0000) 2019-10-13T09:31:22.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:22.872+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.874853+0000) 2019-10-13T09:31:22.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:22.866+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.869353+0000) 2019-10-13T09:31:22.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:22.866+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.870286+0000) 2019-10-13T09:31:22.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:22.866+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.871056+0000) 2019-10-13T09:31:22.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:22.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.871891+0000) 2019-10-13T09:31:22.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:22.870+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.872250+0000) 2019-10-13T09:31:22.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:22.870+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.872930+0000) 2019-10-13T09:31:22.880 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:22.870+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:25.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:25.042+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:25.044072+0000) 2019-10-13T09:31:26.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:26.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:26.264514+0000) 2019-10-13T09:31:27.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:27.873+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.876804+0000) 2019-10-13T09:31:27.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:27.873+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.877987+0000) 2019-10-13T09:31:27.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:27.876+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.878360+0000) 2019-10-13T09:31:27.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:27.873+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.878870+0000) 2019-10-13T09:31:27.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:27.877+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.879503+0000) 2019-10-13T09:31:27.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:27.877+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.879786+0000) 2019-10-13T09:31:27.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:27.877+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.880256+0000) 2019-10-13T09:31:27.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:27.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:27.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:27.884+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.887082+0000) 2019-10-13T09:31:28.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:28.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.526654+0000) 2019-10-13T09:31:28.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:28.529+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.533433+0000) 2019-10-13T09:31:29.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:29.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:29.517056+0000) 2019-10-13T09:31:31.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:31.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:31.822887+0000) 2019-10-13T09:31:32.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:32.013+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.015394+0000) 2019-10-13T09:31:32.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:32.804+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.804971+0000) 2019-10-13T09:31:32.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:32.881+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.884288+0000) 2019-10-13T09:31:32.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:32.881+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.885054+0000) 2019-10-13T09:31:32.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:32.881+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.885872+0000) 2019-10-13T09:31:32.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:32.881+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.886721+0000) 2019-10-13T09:31:32.888 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:32.881+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.886888+0000) 2019-10-13T09:31:32.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:32.885+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:32.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:32.884+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.887710+0000) 2019-10-13T09:31:32.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:32.885+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.888312+0000) 2019-10-13T09:31:32.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:32.888+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.892084+0000) 2019-10-13T09:31:35.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:35.041+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:35.044270+0000) 2019-10-13T09:31:36.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:36.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:36.264704+0000) 2019-10-13T09:31:37.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:37.885+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.890777+0000) 2019-10-13T09:31:37.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:37.889+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.891435+0000) 2019-10-13T09:31:37.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:37.889+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.892709+0000) 2019-10-13T09:31:37.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:37.888+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.892937+0000) 2019-10-13T09:31:37.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:37.889+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.893302+0000) 2019-10-13T09:31:37.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:37.889+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.893602+0000) 2019-10-13T09:31:37.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:37.889+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.893965+0000) 2019-10-13T09:31:37.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:37.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:37.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:37.896+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.900482+0000) 2019-10-13T09:31:38.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:38.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.526831+0000) 2019-10-13T09:31:38.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:38.529+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.533605+0000) 2019-10-13T09:31:39.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:39.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:39.517247+0000) 2019-10-13T09:31:41.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:41.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:41.823087+0000) 2019-10-13T09:31:42.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:42.013+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.015599+0000) 2019-10-13T09:31:42.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:42.807+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.811243+0000) 2019-10-13T09:31:42.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:42.893+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.897372+0000) 2019-10-13T09:31:42.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:42.893+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.897768+0000) 2019-10-13T09:31:42.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:42.893+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.898255+0000) 2019-10-13T09:31:42.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:42.893+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.898598+0000) 2019-10-13T09:31:42.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:42.895+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.899301+0000) 2019-10-13T09:31:42.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:42.893+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.898980+0000) 2019-10-13T09:31:42.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:42.897+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.900592+0000) 2019-10-13T09:31:42.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:42.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:42.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:42.899+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.903072+0000) 2019-10-13T09:31:43.697 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:31:43.697 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:43.701 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:31:43.701 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:45.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:45.041+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:45.044491+0000) 2019-10-13T09:31:46.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:46.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:46.264917+0000) 2019-10-13T09:31:47.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:47.901+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.903984+0000) 2019-10-13T09:31:47.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:47.901+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.905243+0000) 2019-10-13T09:31:47.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:47.901+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.904953+0000) 2019-10-13T09:31:47.907 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:47.901+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.905884+0000) 2019-10-13T09:31:47.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:47.901+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.906490+0000) 2019-10-13T09:31:47.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:47.903+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.906832+0000) 2019-10-13T09:31:47.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:47.905+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.907355+0000) 2019-10-13T09:31:47.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:47.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:47.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:47.907+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.911217+0000) 2019-10-13T09:31:48.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:48.525+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.527011+0000) 2019-10-13T09:31:48.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:48.533+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.533801+0000) 2019-10-13T09:31:49.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:49.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:49.517448+0000) 2019-10-13T09:31:51.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:51.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:51.823352+0000) 2019-10-13T09:31:52.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:52.013+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.015832+0000) 2019-10-13T09:31:52.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:52.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.822694+0000) 2019-10-13T09:31:52.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:52.909+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.911289+0000) 2019-10-13T09:31:52.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:52.909+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.911764+0000) 2019-10-13T09:31:52.913 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:52.909+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.912127+0000) 2019-10-13T09:31:52.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:52.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.912940+0000) 2019-10-13T09:31:52.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:52.909+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.913274+0000) 2019-10-13T09:31:52.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:52.909+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.913735+0000) 2019-10-13T09:31:52.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:52.909+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.914438+0000) 2019-10-13T09:31:52.916 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:52.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:52.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:52.915+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.919264+0000) 2019-10-13T09:31:55.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:55.041+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:55.044691+0000) 2019-10-13T09:31:56.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:56.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:56.265110+0000) 2019-10-13T09:31:57.920 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:31:57.913+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.919135+0000) 2019-10-13T09:31:57.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:31:57.917+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.919634+0000) 2019-10-13T09:31:57.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:57.917+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.919986+0000) 2019-10-13T09:31:57.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:57.917+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.920406+0000) 2019-10-13T09:31:57.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:31:57.917+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.921383+0000) 2019-10-13T09:31:57.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:57.915+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.921611+0000) 2019-10-13T09:31:57.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:31:57.917+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.922151+0000) 2019-10-13T09:31:57.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:31:57.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:31:57.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:31:57.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.932241+0000) 2019-10-13T09:31:58.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:31:58.521+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.527192+0000) 2019-10-13T09:31:58.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:31:58.529+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.533998+0000) 2019-10-13T09:31:59.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:31:59.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:59.517670+0000) 2019-10-13T09:32:01.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:01.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:01.823589+0000) 2019-10-13T09:32:02.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:02.013+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.016066+0000) 2019-10-13T09:32:02.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:02.827+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.829354+0000) 2019-10-13T09:32:02.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:02.921+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.926563+0000) 2019-10-13T09:32:02.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:02.921+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.927104+0000) 2019-10-13T09:32:02.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:02.925+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.927822+0000) 2019-10-13T09:32:02.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:02.925+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.928586+0000) 2019-10-13T09:32:02.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:02.925+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.929827+0000) 2019-10-13T09:32:02.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:02.925+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.929940+0000) 2019-10-13T09:32:02.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:02.927+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.930343+0000) 2019-10-13T09:32:02.932 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:02.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:02.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:02.931+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.935613+0000) 2019-10-13T09:32:05.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:05.041+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:05.044954+0000) 2019-10-13T09:32:06.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:06.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:06.265303+0000) 2019-10-13T09:32:07.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:07.929+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.933539+0000) 2019-10-13T09:32:07.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:07.929+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.933805+0000) 2019-10-13T09:32:07.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:07.931+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.934995+0000) 2019-10-13T09:32:07.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:07.933+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.935621+0000) 2019-10-13T09:32:07.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:07.933+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.936424+0000) 2019-10-13T09:32:07.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:07.933+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.936519+0000) 2019-10-13T09:32:07.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:07.933+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.936799+0000) 2019-10-13T09:32:07.938 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:07.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:07.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:07.939+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.942373+0000) 2019-10-13T09:32:08.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:08.525+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.527376+0000) 2019-10-13T09:32:08.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:08.533+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.534179+0000) 2019-10-13T09:32:09.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:09.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:09.517894+0000) 2019-10-13T09:32:11.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:11.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:11.823828+0000) 2019-10-13T09:32:12.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:12.012+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.016300+0000) 2019-10-13T09:32:12.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:12.831+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.835831+0000) 2019-10-13T09:32:12.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:12.936+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.940341+0000) 2019-10-13T09:32:12.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:12.935+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.941083+0000) 2019-10-13T09:32:12.942 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:12.936+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.941659+0000) 2019-10-13T09:32:12.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:12.936+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.943110+0000) 2019-10-13T09:32:12.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:12.940+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.943430+0000) 2019-10-13T09:32:12.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:12.940+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.943950+0000) 2019-10-13T09:32:12.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:12.940+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.944395+0000) 2019-10-13T09:32:12.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:12.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:12.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:12.943+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.948477+0000) 2019-10-13T09:32:13.771 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:32:13.771 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:13.775 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:32:13.775 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:15.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:15.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:15.045178+0000) 2019-10-13T09:32:16.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:16.264+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:16.265501+0000) 2019-10-13T09:32:17.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:17.944+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.948053+0000) 2019-10-13T09:32:17.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:17.943+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.948609+0000) 2019-10-13T09:32:17.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:17.944+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.949257+0000) 2019-10-13T09:32:17.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:17.944+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.949570+0000) 2019-10-13T09:32:17.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:17.944+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.949978+0000) 2019-10-13T09:32:17.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:17.944+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.950327+0000) 2019-10-13T09:32:17.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:17.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:17.953 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:17.948+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.951651+0000) 2019-10-13T09:32:17.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:17.951+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.956921+0000) 2019-10-13T09:32:18.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:18.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.527518+0000) 2019-10-13T09:32:18.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:18.528+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.534353+0000) 2019-10-13T09:32:19.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:19.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:19.518088+0000) 2019-10-13T09:32:21.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:21.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:21.824028+0000) 2019-10-13T09:32:22.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:22.012+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.016503+0000) 2019-10-13T09:32:22.843 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:22.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.842212+0000) 2019-10-13T09:32:22.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:22.951+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.954941+0000) 2019-10-13T09:32:22.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:22.948+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.955191+0000) 2019-10-13T09:32:22.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:22.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.955896+0000) 2019-10-13T09:32:22.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:22.952+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.955920+0000) 2019-10-13T09:32:22.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:22.952+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.956781+0000) 2019-10-13T09:32:22.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:22.952+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.957248+0000) 2019-10-13T09:32:22.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:22.952+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.957897+0000) 2019-10-13T09:32:22.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:22.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:22.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:22.959+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.963784+0000) 2019-10-13T09:32:25.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:25.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:25.045438+0000) 2019-10-13T09:32:26.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:26.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:26.265713+0000) 2019-10-13T09:32:27.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:27.956+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.961839+0000) 2019-10-13T09:32:27.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:27.956+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.962896+0000) 2019-10-13T09:32:27.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:27.956+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.962500+0000) 2019-10-13T09:32:27.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:27.956+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.963238+0000) 2019-10-13T09:32:27.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:27.960+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.963609+0000) 2019-10-13T09:32:27.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:27.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.964094+0000) 2019-10-13T09:32:27.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:27.960+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.964442+0000) 2019-10-13T09:32:27.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:27.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 77988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:27.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:27.967+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.970857+0000) 2019-10-13T09:32:28.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:28.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.527720+0000) 2019-10-13T09:32:28.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:28.532+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.534535+0000) 2019-10-13T09:32:29.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:29.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:29.518315+0000) 2019-10-13T09:32:31.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:31.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:31.824262+0000) 2019-10-13T09:32:32.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:32.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.016703+0000) 2019-10-13T09:32:32.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:32.847+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.848663+0000) 2019-10-13T09:32:32.970 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:32.964+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.968822+0000) 2019-10-13T09:32:32.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:32.964+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.969847+0000) 2019-10-13T09:32:32.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:32.967+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.970253+0000) 2019-10-13T09:32:32.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:32.964+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.970843+0000) 2019-10-13T09:32:32.972 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:32.964+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.971159+0000) 2019-10-13T09:32:32.973 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:32.968+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.971812+0000) 2019-10-13T09:32:32.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:32.968+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.972177+0000) 2019-10-13T09:32:32.973 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:32.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:32.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:32.975+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.980528+0000) 2019-10-13T09:32:35.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:35.040+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:35.045639+0000) 2019-10-13T09:32:36.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:36.264+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:36.265954+0000) 2019-10-13T09:32:37.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:37.972+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.975747+0000) 2019-10-13T09:32:37.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:37.972+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.976402+0000) 2019-10-13T09:32:37.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:37.972+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.976756+0000) 2019-10-13T09:32:37.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:37.970+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.977362+0000) 2019-10-13T09:32:37.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:37.972+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.977616+0000) 2019-10-13T09:32:37.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:37.972+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.978081+0000) 2019-10-13T09:32:37.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:37.972+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.978587+0000) 2019-10-13T09:32:37.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:37.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:37.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:37.982+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.986173+0000) 2019-10-13T09:32:38.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:38.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.527922+0000) 2019-10-13T09:32:38.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:38.532+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.534740+0000) 2019-10-13T09:32:39.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:39.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:39.518489+0000) 2019-10-13T09:32:41.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:41.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:41.824471+0000) 2019-10-13T09:32:42.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:42.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.016930+0000) 2019-10-13T09:32:42.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:42.854+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.855094+0000) 2019-10-13T09:32:42.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:42.976+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.982917+0000) 2019-10-13T09:32:42.985 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:42.980+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.983542+0000) 2019-10-13T09:32:42.985 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:42.980+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.983916+0000) 2019-10-13T09:32:42.985 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:42.980+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.984208+0000) 2019-10-13T09:32:42.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:42.978+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.984573+0000) 2019-10-13T09:32:42.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:42.980+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.984743+0000) 2019-10-13T09:32:42.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:42.980+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.985631+0000) 2019-10-13T09:32:42.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:42.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:42.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:42.982+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.988801+0000) 2019-10-13T09:32:44.018 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:32:44.018 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:44.025 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:32:44.025 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:45.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:45.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:45.045898+0000) 2019-10-13T09:32:46.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:46.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:46.266137+0000) 2019-10-13T09:32:47.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:47.984+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.989065+0000) 2019-10-13T09:32:47.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:47.984+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.989432+0000) 2019-10-13T09:32:47.991 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:47.984+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.989794+0000) 2019-10-13T09:32:47.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:47.984+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.990331+0000) 2019-10-13T09:32:47.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:47.984+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.990753+0000) 2019-10-13T09:32:47.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:47.986+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.992081+0000) 2019-10-13T09:32:47.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:47.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:47.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:47.988+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.992765+0000) 2019-10-13T09:32:48.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:47.994+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.998976+0000) 2019-10-13T09:32:48.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:48.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.528118+0000) 2019-10-13T09:32:48.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:48.532+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.534909+0000) 2019-10-13T09:32:49.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:49.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:49.518679+0000) 2019-10-13T09:32:51.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:51.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:51.824673+0000) 2019-10-13T09:32:52.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:52.012+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.017133+0000) 2019-10-13T09:32:52.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:52.862+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.866556+0000) 2019-10-13T09:32:52.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:52.991+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.996360+0000) 2019-10-13T09:32:52.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:52.991+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.996721+0000) 2019-10-13T09:32:52.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:52.991+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.997297+0000) 2019-10-13T09:32:52.999 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:52.991+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.997780+0000) 2019-10-13T09:32:52.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:52.991+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.997968+0000) 2019-10-13T09:32:53.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:52.994+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.999205+0000) 2019-10-13T09:32:53.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:52.995+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:53.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:52.995+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.999720+0000) 2019-10-13T09:32:53.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:52.998+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.005446+0000) 2019-10-13T09:32:55.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:55.043+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:55.046095+0000) 2019-10-13T09:32:56.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:56.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:56.266352+0000) 2019-10-13T09:32:58.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:32:57.995+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.002941+0000) 2019-10-13T09:32:58.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:57.999+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.003399+0000) 2019-10-13T09:32:58.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:57.998+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.004463+0000) 2019-10-13T09:32:58.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:32:57.999+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.004045+0000) 2019-10-13T09:32:58.006 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:57.999+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.005172+0000) 2019-10-13T09:32:58.006 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:32:57.999+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.005687+0000) 2019-10-13T09:32:58.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:32:57.999+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.006026+0000) 2019-10-13T09:32:58.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:32:57.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:32:58.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:32:58.010+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.015208+0000) 2019-10-13T09:32:58.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:32:58.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.528328+0000) 2019-10-13T09:32:58.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:32:58.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.535113+0000) 2019-10-13T09:32:59.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:32:59.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:59.518884+0000) 2019-10-13T09:33:01.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:01.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:01.824894+0000) 2019-10-13T09:33:02.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:02.011+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.017332+0000) 2019-10-13T09:33:02.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:02.870+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.872903+0000) 2019-10-13T09:33:03.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:03.003+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.009609+0000) 2019-10-13T09:33:03.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:03.003+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.010588+0000) 2019-10-13T09:33:03.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:03.007+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.011341+0000) 2019-10-13T09:33:03.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:03.007+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.012212+0000) 2019-10-13T09:33:03.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:03.006+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.013329+0000) 2019-10-13T09:33:03.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:03.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.013488+0000) 2019-10-13T09:33:03.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:03.007+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.013919+0000) 2019-10-13T09:33:03.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:03.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:03.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:03.014+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.021070+0000) 2019-10-13T09:33:05.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:05.043+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:05.046297+0000) 2019-10-13T09:33:06.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:06.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:06.266585+0000) 2019-10-13T09:33:08.018 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:08.011+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.017056+0000) 2019-10-13T09:33:08.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:08.011+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.018288+0000) 2019-10-13T09:33:08.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:08.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.018490+0000) 2019-10-13T09:33:08.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:08.015+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.019376+0000) 2019-10-13T09:33:08.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:08.015+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.019666+0000) 2019-10-13T09:33:08.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:08.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:08.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:08.015+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.020947+0000) 2019-10-13T09:33:08.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:08.015+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.021380+0000) 2019-10-13T09:33:08.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:08.022+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.027103+0000) 2019-10-13T09:33:08.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:08.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.528508+0000) 2019-10-13T09:33:08.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:08.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.535348+0000) 2019-10-13T09:33:09.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:09.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:09.519111+0000) 2019-10-13T09:33:11.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:11.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:11.825124+0000) 2019-10-13T09:33:12.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:12.015+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.017538+0000) 2019-10-13T09:33:12.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:12.882+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.882405+0000) 2019-10-13T09:33:13.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:13.019+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.023668+0000) 2019-10-13T09:33:13.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:13.019+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.025335+0000) 2019-10-13T09:33:13.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:13.019+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.025646+0000) 2019-10-13T09:33:13.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:13.019+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.026429+0000) 2019-10-13T09:33:13.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:13.022+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.026785+0000) 2019-10-13T09:33:13.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:13.023+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.027503+0000) 2019-10-13T09:33:13.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:13.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:13.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:13.023+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.027948+0000) 2019-10-13T09:33:13.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:13.026+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.032567+0000) 2019-10-13T09:33:14.151 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:33:14.152 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:14.155 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:33:14.155 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:15.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:15.039+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:15.046490+0000) 2019-10-13T09:33:16.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:16.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:16.266840+0000) 2019-10-13T09:33:18.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:18.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.031342+0000) 2019-10-13T09:33:18.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:18.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.031801+0000) 2019-10-13T09:33:18.033 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:18.027+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.032047+0000) 2019-10-13T09:33:18.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:18.027+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.032994+0000) 2019-10-13T09:33:18.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:18.027+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.033538+0000) 2019-10-13T09:33:18.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:18.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:18.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:18.030+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.034441+0000) 2019-10-13T09:33:18.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:18.027+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.034753+0000) 2019-10-13T09:33:18.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:18.034+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.040162+0000) 2019-10-13T09:33:18.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:18.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.528712+0000) 2019-10-13T09:33:18.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:18.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.535551+0000) 2019-10-13T09:33:19.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:19.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:19.519304+0000) 2019-10-13T09:33:21.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:21.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:21.825326+0000) 2019-10-13T09:33:22.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:22.015+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.017705+0000) 2019-10-13T09:33:22.889 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:22.886+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.888326+0000) 2019-10-13T09:33:23.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:23.035+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.037563+0000) 2019-10-13T09:33:23.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:23.035+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.038811+0000) 2019-10-13T09:33:23.040 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:23.039+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.039506+0000) 2019-10-13T09:33:23.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:23.039+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.039710+0000) 2019-10-13T09:33:23.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:23.039+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.040238+0000) 2019-10-13T09:33:23.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:23.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:23.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:23.039+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.041097+0000) 2019-10-13T09:33:23.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:23.034+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.041887+0000) 2019-10-13T09:33:23.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:23.038+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.044448+0000) 2019-10-13T09:33:25.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:25.043+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:25.046690+0000) 2019-10-13T09:33:26.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:26.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:26.267030+0000) 2019-10-13T09:33:28.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:28.043+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.044339+0000) 2019-10-13T09:33:28.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:28.043+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.045645+0000) 2019-10-13T09:33:28.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:28.043+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.045957+0000) 2019-10-13T09:33:28.048 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:28.043+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.046536+0000) 2019-10-13T09:33:28.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:28.043+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.046571+0000) 2019-10-13T09:33:28.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:28.046+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.048193+0000) 2019-10-13T09:33:28.049 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:28.047+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.047610+0000) 2019-10-13T09:33:28.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:28.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:28.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:28.050+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.050691+0000) 2019-10-13T09:33:28.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:28.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.528914+0000) 2019-10-13T09:33:28.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:28.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.535725+0000) 2019-10-13T09:33:29.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:29.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:29.519516+0000) 2019-10-13T09:33:31.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:31.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:31.825543+0000) 2019-10-13T09:33:32.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:32.015+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.017927+0000) 2019-10-13T09:33:32.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:32.897+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.900060+0000) 2019-10-13T09:33:33.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:33.047+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.050679+0000) 2019-10-13T09:33:33.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:33.047+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.051068+0000) 2019-10-13T09:33:33.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:33.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.052756+0000) 2019-10-13T09:33:33.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:33.051+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.053619+0000) 2019-10-13T09:33:33.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:33.049+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.053713+0000) 2019-10-13T09:33:33.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:33.051+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.054104+0000) 2019-10-13T09:33:33.063 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:33.051+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.054574+0000) 2019-10-13T09:33:33.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:33.051+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:33.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:33.057+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.061200+0000) 2019-10-13T09:33:35.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:35.043+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:35.046923+0000) 2019-10-13T09:33:36.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:36.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:36.267263+0000) 2019-10-13T09:33:38.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:38.057+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.058778+0000) 2019-10-13T09:33:38.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:38.054+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.059255+0000) 2019-10-13T09:33:38.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:38.058+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.059570+0000) 2019-10-13T09:33:38.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:38.058+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.060062+0000) 2019-10-13T09:33:38.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:38.058+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.060294+0000) 2019-10-13T09:33:38.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:38.058+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.060494+0000) 2019-10-13T09:33:38.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:38.058+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.061049+0000) 2019-10-13T09:33:38.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:38.058+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:38.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:38.065+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.068306+0000) 2019-10-13T09:33:38.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:38.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.529121+0000) 2019-10-13T09:33:38.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:38.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.535898+0000) 2019-10-13T09:33:39.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:39.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:39.519727+0000) 2019-10-13T09:33:41.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:41.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:41.825743+0000) 2019-10-13T09:33:42.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:42.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.018127+0000) 2019-10-13T09:33:42.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:42.905+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.906597+0000) 2019-10-13T09:33:43.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:43.061+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.065152+0000) 2019-10-13T09:33:43.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:43.062+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.066087+0000) 2019-10-13T09:33:43.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:43.062+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.066858+0000) 2019-10-13T09:33:43.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:43.062+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.066851+0000) 2019-10-13T09:33:43.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:43.062+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.067249+0000) 2019-10-13T09:33:43.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:43.066+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.067765+0000) 2019-10-13T09:33:43.070 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:43.066+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.068203+0000) 2019-10-13T09:33:43.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:43.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78492 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:43.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:43.073+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.076361+0000) 2019-10-13T09:33:44.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:33:44.263 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:44.267 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:33:44.267 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:45.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:45.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:45.047151+0000) 2019-10-13T09:33:46.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:46.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:46.267526+0000) 2019-10-13T09:33:48.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:48.070+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.072285+0000) 2019-10-13T09:33:48.074 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:48.070+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.072524+0000) 2019-10-13T09:33:48.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:48.070+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.074122+0000) 2019-10-13T09:33:48.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:48.073+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.074840+0000) 2019-10-13T09:33:48.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:48.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.075084+0000) 2019-10-13T09:33:48.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:48.074+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.075420+0000) 2019-10-13T09:33:48.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:48.074+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.075731+0000) 2019-10-13T09:33:48.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:48.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:48.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:48.077+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.080739+0000) 2019-10-13T09:33:48.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:48.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.529316+0000) 2019-10-13T09:33:48.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:48.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.536075+0000) 2019-10-13T09:33:49.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:49.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:49.519927+0000) 2019-10-13T09:33:51.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:51.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:51.825946+0000) 2019-10-13T09:33:52.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:52.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.018358+0000) 2019-10-13T09:33:52.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:52.909+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.913014+0000) 2019-10-13T09:33:53.081 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:53.078+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.079405+0000) 2019-10-13T09:33:53.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:53.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.078813+0000) 2019-10-13T09:33:53.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:53.077+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.080356+0000) 2019-10-13T09:33:53.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:53.078+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.080604+0000) 2019-10-13T09:33:53.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:53.078+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.081116+0000) 2019-10-13T09:33:53.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:53.078+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.081465+0000) 2019-10-13T09:33:53.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:53.078+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.082297+0000) 2019-10-13T09:33:53.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:53.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:53.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:53.085+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.086998+0000) 2019-10-13T09:33:55.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:55.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:55.047364+0000) 2019-10-13T09:33:56.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:56.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:56.267780+0000) 2019-10-13T09:33:58.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:58.082+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.085959+0000) 2019-10-13T09:33:58.089 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:33:58.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.086757+0000) 2019-10-13T09:33:58.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:33:58.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.086952+0000) 2019-10-13T09:33:58.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:33:58.086+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.087551+0000) 2019-10-13T09:33:58.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:58.085+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.088551+0000) 2019-10-13T09:33:58.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:33:58.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:33:58.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:33:58.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.089271+0000) 2019-10-13T09:33:58.091 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:58.086+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.090211+0000) 2019-10-13T09:33:58.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:33:58.093+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.095353+0000) 2019-10-13T09:33:58.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:33:58.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.529489+0000) 2019-10-13T09:33:58.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:33:58.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.536269+0000) 2019-10-13T09:33:59.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:33:59.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:59.520122+0000) 2019-10-13T09:34:01.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:01.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:01.826175+0000) 2019-10-13T09:34:02.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:02.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.018555+0000) 2019-10-13T09:34:02.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:02.917+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.919529+0000) 2019-10-13T09:34:03.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:03.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.091911+0000) 2019-10-13T09:34:03.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:03.090+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.093647+0000) 2019-10-13T09:34:03.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:03.090+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.094030+0000) 2019-10-13T09:34:03.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:03.093+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.094737+0000) 2019-10-13T09:34:03.096 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:03.090+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.095148+0000) 2019-10-13T09:34:03.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:03.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:03.097 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:03.094+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.096321+0000) 2019-10-13T09:34:03.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:03.094+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.096748+0000) 2019-10-13T09:34:03.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:03.097+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.102079+0000) 2019-10-13T09:34:05.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:05.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:05.047560+0000) 2019-10-13T09:34:06.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:06.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:06.268000+0000) 2019-10-13T09:34:08.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:08.098+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.099532+0000) 2019-10-13T09:34:08.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:08.098+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.100201+0000) 2019-10-13T09:34:08.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:08.098+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.101454+0000) 2019-10-13T09:34:08.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:08.097+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.101966+0000) 2019-10-13T09:34:08.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:08.098+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.101669+0000) 2019-10-13T09:34:08.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:08.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:08.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:08.098+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.102978+0000) 2019-10-13T09:34:08.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:08.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.103607+0000) 2019-10-13T09:34:08.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:08.105+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.108004+0000) 2019-10-13T09:34:08.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:08.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.529665+0000) 2019-10-13T09:34:08.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:08.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.536409+0000) 2019-10-13T09:34:09.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:09.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:09.520323+0000) 2019-10-13T09:34:11.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:11.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:11.826372+0000) 2019-10-13T09:34:12.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:12.014+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.018771+0000) 2019-10-13T09:34:12.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:12.925+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.926031+0000) 2019-10-13T09:34:13.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:13.102+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.105501+0000) 2019-10-13T09:34:13.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:13.106+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.107602+0000) 2019-10-13T09:34:13.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:13.105+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.108040+0000) 2019-10-13T09:34:13.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:13.106+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.108300+0000) 2019-10-13T09:34:13.110 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:13.106+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.108920+0000) 2019-10-13T09:34:13.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:13.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:13.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:13.106+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.109166+0000) 2019-10-13T09:34:13.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:13.106+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.109893+0000) 2019-10-13T09:34:13.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:13.109+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.113602+0000) 2019-10-13T09:34:14.399 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:34:14.399 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:14.409 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:34:14.409 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:15.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:15.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:15.047759+0000) 2019-10-13T09:34:16.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:16.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:16.268181+0000) 2019-10-13T09:34:18.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:18.110+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.112909+0000) 2019-10-13T09:34:18.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:18.110+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.113424+0000) 2019-10-13T09:34:18.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:18.110+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.114023+0000) 2019-10-13T09:34:18.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:18.113+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.114735+0000) 2019-10-13T09:34:18.116 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:18.110+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.115008+0000) 2019-10-13T09:34:18.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:18.114+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.115289+0000) 2019-10-13T09:34:18.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:18.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.115695+0000) 2019-10-13T09:34:18.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:18.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78726 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:18.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:18.117+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.119259+0000) 2019-10-13T09:34:18.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:18.526+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.529839+0000) 2019-10-13T09:34:18.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:18.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.536644+0000) 2019-10-13T09:34:19.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:19.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:19.520532+0000) 2019-10-13T09:34:21.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:21.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:21.826596+0000) 2019-10-13T09:34:22.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:22.013+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.018992+0000) 2019-10-13T09:34:22.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:22.929+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.932232+0000) 2019-10-13T09:34:23.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:23.113+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.119287+0000) 2019-10-13T09:34:23.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:23.117+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.119981+0000) 2019-10-13T09:34:23.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:23.117+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.120169+0000) 2019-10-13T09:34:23.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:23.117+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.121047+0000) 2019-10-13T09:34:23.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:23.117+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.121539+0000) 2019-10-13T09:34:23.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:23.117+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.121353+0000) 2019-10-13T09:34:23.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:23.117+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.121851+0000) 2019-10-13T09:34:23.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:23.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:23.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:23.125+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.129100+0000) 2019-10-13T09:34:25.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:25.045+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:25.047962+0000) 2019-10-13T09:34:26.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:26.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:26.268380+0000) 2019-10-13T09:34:28.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:28.121+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.125413+0000) 2019-10-13T09:34:28.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:28.121+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.126180+0000) 2019-10-13T09:34:28.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:28.121+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.126870+0000) 2019-10-13T09:34:28.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:28.124+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.127715+0000) 2019-10-13T09:34:28.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:28.121+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.127284+0000) 2019-10-13T09:34:28.129 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:28.125+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.127984+0000) 2019-10-13T09:34:28.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:28.125+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.128663+0000) 2019-10-13T09:34:28.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:28.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78793 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:28.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:28.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.135461+0000) 2019-10-13T09:34:28.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:28.525+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.530013+0000) 2019-10-13T09:34:28.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:28.533+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.536848+0000) 2019-10-13T09:34:29.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:29.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:29.520749+0000) 2019-10-13T09:34:31.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:31.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:31.826810+0000) 2019-10-13T09:34:32.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:32.017+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.019195+0000) 2019-10-13T09:34:32.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:32.936+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.938699+0000) 2019-10-13T09:34:33.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:33.129+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.132892+0000) 2019-10-13T09:34:33.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:33.129+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.133365+0000) 2019-10-13T09:34:33.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:33.129+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.133933+0000) 2019-10-13T09:34:33.136 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:33.129+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.134273+0000) 2019-10-13T09:34:33.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:33.129+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.134768+0000) 2019-10-13T09:34:33.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:33.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.135215+0000) 2019-10-13T09:34:33.137 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:33.133+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.135717+0000) 2019-10-13T09:34:33.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:33.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78828 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:33.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:33.140+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.142805+0000) 2019-10-13T09:34:35.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:35.045+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:35.048167+0000) 2019-10-13T09:34:36.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:36.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:36.268582+0000) 2019-10-13T09:34:38.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:38.137+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.139960+0000) 2019-10-13T09:34:38.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:38.136+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.140728+0000) 2019-10-13T09:34:38.142 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:38.137+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.141564+0000) 2019-10-13T09:34:38.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:38.137+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.141886+0000) 2019-10-13T09:34:38.143 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:38.137+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.142589+0000) 2019-10-13T09:34:38.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:38.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.142297+0000) 2019-10-13T09:34:38.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:38.137+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.142895+0000) 2019-10-13T09:34:38.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:38.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:38.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:38.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.147164+0000) 2019-10-13T09:34:38.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:38.525+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.530207+0000) 2019-10-13T09:34:38.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:38.533+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.537055+0000) 2019-10-13T09:34:39.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:39.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:39.520943+0000) 2019-10-13T09:34:41.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:41.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:41.827021+0000) 2019-10-13T09:34:42.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:42.017+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.019430+0000) 2019-10-13T09:34:42.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:42.940+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.945098+0000) 2019-10-13T09:34:43.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:43.145+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.147420+0000) 2019-10-13T09:34:43.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:43.145+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.147968+0000) 2019-10-13T09:34:43.149 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:43.145+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.148396+0000) 2019-10-13T09:34:43.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:43.145+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.148653+0000) 2019-10-13T09:34:43.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:43.145+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.148971+0000) 2019-10-13T09:34:43.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:43.145+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.149431+0000) 2019-10-13T09:34:43.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:43.148+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.150212+0000) 2019-10-13T09:34:43.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:43.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:43.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:43.152+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.157327+0000) 2019-10-13T09:34:44.502 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:34:44.502 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:44.506 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:34:44.506 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:45.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:45.045+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:45.048379+0000) 2019-10-13T09:34:46.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:46.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:46.268774+0000) 2019-10-13T09:34:48.155 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:48.149+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.154513+0000) 2019-10-13T09:34:48.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:48.149+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.155207+0000) 2019-10-13T09:34:48.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:48.152+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.155471+0000) 2019-10-13T09:34:48.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:48.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.156434+0000) 2019-10-13T09:34:48.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:48.153+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.156833+0000) 2019-10-13T09:34:48.158 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:48.153+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.157205+0000) 2019-10-13T09:34:48.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:48.153+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.157520+0000) 2019-10-13T09:34:48.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:48.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:48.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:48.160+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.164971+0000) 2019-10-13T09:34:48.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:48.525+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.530385+0000) 2019-10-13T09:34:48.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:48.533+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.537234+0000) 2019-10-13T09:34:49.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:49.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:49.521138+0000) 2019-10-13T09:34:51.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:51.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:51.827219+0000) 2019-10-13T09:34:52.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:52.017+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.019661+0000) 2019-10-13T09:34:52.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:52.948+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.951511+0000) 2019-10-13T09:34:53.163 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:53.157+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.161554+0000) 2019-10-13T09:34:53.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:53.157+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.162213+0000) 2019-10-13T09:34:53.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:53.157+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.162493+0000) 2019-10-13T09:34:53.164 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:53.161+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.163463+0000) 2019-10-13T09:34:53.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:53.161+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.163947+0000) 2019-10-13T09:34:53.165 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:53.160+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.164108+0000) 2019-10-13T09:34:53.166 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:53.161+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.164435+0000) 2019-10-13T09:34:53.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:53.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:53.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:53.168+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.172097+0000) 2019-10-13T09:34:55.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:55.045+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:55.048577+0000) 2019-10-13T09:34:56.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:56.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:56.268969+0000) 2019-10-13T09:34:58.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:58.165+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.168322+0000) 2019-10-13T09:34:58.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:58.165+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.168843+0000) 2019-10-13T09:34:58.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:34:58.165+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.169037+0000) 2019-10-13T09:34:58.171 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:34:58.165+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.169841+0000) 2019-10-13T09:34:58.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:34:58.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.170403+0000) 2019-10-13T09:34:58.172 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:34:58.169+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.171344+0000) 2019-10-13T09:34:58.172 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:34:58.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 78993 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:34:58.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:58.168+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.171893+0000) 2019-10-13T09:34:58.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:34:58.172+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.177055+0000) 2019-10-13T09:34:58.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:34:58.529+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.530566+0000) 2019-10-13T09:34:58.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:34:58.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.537411+0000) 2019-10-13T09:34:59.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:34:59.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:59.521334+0000) 2019-10-13T09:35:01.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:01.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:01.827420+0000) 2019-10-13T09:35:02.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:02.017+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.019869+0000) 2019-10-13T09:35:02.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:02.956+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.957917+0000) 2019-10-13T09:35:03.176 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:03.168+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.174858+0000) 2019-10-13T09:35:03.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:03.172+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.175520+0000) 2019-10-13T09:35:03.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:03.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.177194+0000) 2019-10-13T09:35:03.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:03.172+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.177617+0000) 2019-10-13T09:35:03.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:03.172+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.177939+0000) 2019-10-13T09:35:03.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:03.172+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.178404+0000) 2019-10-13T09:35:03.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:03.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:03.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:03.176+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.178791+0000) 2019-10-13T09:35:03.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:03.180+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.183029+0000) 2019-10-13T09:35:05.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:05.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:05.048789+0000) 2019-10-13T09:35:06.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:06.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:06.269170+0000) 2019-10-13T09:35:08.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:08.176+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.182132+0000) 2019-10-13T09:35:08.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:08.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.182534+0000) 2019-10-13T09:35:08.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:08.180+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.183500+0000) 2019-10-13T09:35:08.185 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:08.180+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.184596+0000) 2019-10-13T09:35:08.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:08.180+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.184769+0000) 2019-10-13T09:35:08.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:08.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:08.186 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:08.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.185378+0000) 2019-10-13T09:35:08.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:08.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.185905+0000) 2019-10-13T09:35:08.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:08.188+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.193245+0000) 2019-10-13T09:35:08.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:08.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.530765+0000) 2019-10-13T09:35:08.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:08.532+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.537606+0000) 2019-10-13T09:35:09.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:09.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:09.521553+0000) 2019-10-13T09:35:11.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:11.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:11.827624+0000) 2019-10-13T09:35:12.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:12.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.020083+0000) 2019-10-13T09:35:12.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:12.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.964407+0000) 2019-10-13T09:35:13.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:13.184+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.188255+0000) 2019-10-13T09:35:13.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:13.184+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.189673+0000) 2019-10-13T09:35:13.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:13.184+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.189960+0000) 2019-10-13T09:35:13.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:13.184+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.190674+0000) 2019-10-13T09:35:13.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:13.188+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.191249+0000) 2019-10-13T09:35:13.192 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:13.188+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.191557+0000) 2019-10-13T09:35:13.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:13.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:13.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:13.188+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.192629+0000) 2019-10-13T09:35:13.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:13.196+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.198158+0000) 2019-10-13T09:35:14.830 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:35:14.830 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:14.834 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:35:14.834 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:15.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:15.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:15.048984+0000) 2019-10-13T09:35:16.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:16.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:16.269370+0000) 2019-10-13T09:35:18.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:18.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.195548+0000) 2019-10-13T09:35:18.197 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:18.192+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.195998+0000) 2019-10-13T09:35:18.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:18.192+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.196834+0000) 2019-10-13T09:35:18.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:18.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.197102+0000) 2019-10-13T09:35:18.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:18.192+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.197369+0000) 2019-10-13T09:35:18.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:18.192+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.197764+0000) 2019-10-13T09:35:18.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:18.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.198116+0000) 2019-10-13T09:35:18.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:18.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:18.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:18.200+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.202660+0000) 2019-10-13T09:35:18.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:18.528+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.530980+0000) 2019-10-13T09:35:18.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:18.536+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.537782+0000) 2019-10-13T09:35:19.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:19.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:19.521745+0000) 2019-10-13T09:35:21.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:21.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:21.827853+0000) 2019-10-13T09:35:22.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:22.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.020283+0000) 2019-10-13T09:35:22.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:22.971+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.975536+0000) 2019-10-13T09:35:23.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:23.196+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.202003+0000) 2019-10-13T09:35:23.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:23.196+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.202448+0000) 2019-10-13T09:35:23.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:23.199+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.203191+0000) 2019-10-13T09:35:23.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:23.200+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.203476+0000) 2019-10-13T09:35:23.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:23.200+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.204165+0000) 2019-10-13T09:35:23.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:23.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.204617+0000) 2019-10-13T09:35:23.207 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:23.200+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.205077+0000) 2019-10-13T09:35:23.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:23.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:23.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:23.207+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.210194+0000) 2019-10-13T09:35:25.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:25.048+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:25.049203+0000) 2019-10-13T09:35:26.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:26.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:26.269591+0000) 2019-10-13T09:35:28.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:28.204+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.208916+0000) 2019-10-13T09:35:28.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:28.204+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.209234+0000) 2019-10-13T09:35:28.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:28.204+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.209571+0000) 2019-10-13T09:35:28.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:28.204+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.209960+0000) 2019-10-13T09:35:28.212 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:28.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.210589+0000) 2019-10-13T09:35:28.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:28.207+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.210985+0000) 2019-10-13T09:35:28.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:28.208+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.211435+0000) 2019-10-13T09:35:28.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:28.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:28.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:28.215+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.218364+0000) 2019-10-13T09:35:28.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:28.524+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.531157+0000) 2019-10-13T09:35:28.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:28.536+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.537961+0000) 2019-10-13T09:35:29.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:29.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:29.521944+0000) 2019-10-13T09:35:31.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:31.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:31.828065+0000) 2019-10-13T09:35:32.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:32.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.020500+0000) 2019-10-13T09:35:32.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:32.979+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.982168+0000) 2019-10-13T09:35:33.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:33.212+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.216180+0000) 2019-10-13T09:35:33.218 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:33.212+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.216863+0000) 2019-10-13T09:35:33.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:33.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.218116+0000) 2019-10-13T09:35:33.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:33.212+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.218524+0000) 2019-10-13T09:35:33.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:33.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.219037+0000) 2019-10-13T09:35:33.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:33.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:33.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:33.216+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.219424+0000) 2019-10-13T09:35:33.221 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:33.216+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.220456+0000) 2019-10-13T09:35:33.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:33.219+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.222197+0000) 2019-10-13T09:35:35.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:35.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:35.049404+0000) 2019-10-13T09:35:36.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:36.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:36.269802+0000) 2019-10-13T09:35:38.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:38.216+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.222190+0000) 2019-10-13T09:35:38.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:38.216+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.222728+0000) 2019-10-13T09:35:38.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:38.219+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.224298+0000) 2019-10-13T09:35:38.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:38.220+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.224644+0000) 2019-10-13T09:35:38.226 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:38.220+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.224932+0000) 2019-10-13T09:35:38.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:38.220+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.225292+0000) 2019-10-13T09:35:38.227 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:38.220+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.226651+0000) 2019-10-13T09:35:38.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:38.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:38.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:38.223+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.229902+0000) 2019-10-13T09:35:38.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:38.528+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.531340+0000) 2019-10-13T09:35:38.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:38.536+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.538139+0000) 2019-10-13T09:35:39.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:39.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:39.522180+0000) 2019-10-13T09:35:41.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:41.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:41.828288+0000) 2019-10-13T09:35:42.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:42.016+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.020702+0000) 2019-10-13T09:35:42.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:42.983+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.988484+0000) 2019-10-13T09:35:43.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:43.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.230195+0000) 2019-10-13T09:35:43.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:43.224+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.230812+0000) 2019-10-13T09:35:43.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:43.228+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.231805+0000) 2019-10-13T09:35:43.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:43.228+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.232473+0000) 2019-10-13T09:35:43.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:43.227+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.232825+0000) 2019-10-13T09:35:43.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:43.228+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.233093+0000) 2019-10-13T09:35:43.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:43.228+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.233369+0000) 2019-10-13T09:35:43.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:43.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:43.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:43.239+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.242402+0000) 2019-10-13T09:35:44.891 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:35:44.891 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:44.895 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:35:44.895 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:45.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:45.044+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:45.049626+0000) 2019-10-13T09:35:46.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:46.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:46.269997+0000) 2019-10-13T09:35:48.238 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:48.231+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.237080+0000) 2019-10-13T09:35:48.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:48.231+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.237749+0000) 2019-10-13T09:35:48.239 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:48.231+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.238120+0000) 2019-10-13T09:35:48.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:48.231+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.239009+0000) 2019-10-13T09:35:48.240 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:48.235+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.239894+0000) 2019-10-13T09:35:48.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:48.235+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.240234+0000) 2019-10-13T09:35:48.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:48.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:48.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:48.235+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.240848+0000) 2019-10-13T09:35:48.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:48.239+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.245040+0000) 2019-10-13T09:35:48.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:48.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.531554+0000) 2019-10-13T09:35:48.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:48.531+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.538322+0000) 2019-10-13T09:35:49.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:49.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:49.522397+0000) 2019-10-13T09:35:51.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:51.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:51.828521+0000) 2019-10-13T09:35:52.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:52.015+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.020905+0000) 2019-10-13T09:35:52.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:52.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.994764+0000) 2019-10-13T09:35:53.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:53.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.243609+0000) 2019-10-13T09:35:53.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:53.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.244431+0000) 2019-10-13T09:35:53.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:53.239+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.245246+0000) 2019-10-13T09:35:53.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:53.239+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.245724+0000) 2019-10-13T09:35:53.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:53.239+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.246234+0000) 2019-10-13T09:35:53.248 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:53.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:53.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:53.243+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.247358+0000) 2019-10-13T09:35:53.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:53.243+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.247725+0000) 2019-10-13T09:35:53.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:53.247+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.250503+0000) 2019-10-13T09:35:55.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:55.047+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:55.049857+0000) 2019-10-13T09:35:56.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:56.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:56.270220+0000) 2019-10-13T09:35:58.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:35:58.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.250409+0000) 2019-10-13T09:35:58.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:35:58.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.251063+0000) 2019-10-13T09:35:58.253 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:58.247+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.252054+0000) 2019-10-13T09:35:58.253 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:58.247+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.252382+0000) 2019-10-13T09:35:58.254 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:35:58.247+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.252729+0000) 2019-10-13T09:35:58.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:35:58.247+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.253292+0000) 2019-10-13T09:35:58.254 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:58.247+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.253604+0000) 2019-10-13T09:35:58.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:35:58.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:35:58.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:35:58.251+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.257875+0000) 2019-10-13T09:35:58.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:35:58.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.531690+0000) 2019-10-13T09:35:58.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:35:58.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.538512+0000) 2019-10-13T09:35:59.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:35:59.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:59.522592+0000) 2019-10-13T09:36:01.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:01.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:01.828732+0000) 2019-10-13T09:36:02.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:02.015+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.021115+0000) 2019-10-13T09:36:03.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:02.999+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.001140+0000) 2019-10-13T09:36:03.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:03.251+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.258123+0000) 2019-10-13T09:36:03.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:03.251+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.258412+0000) 2019-10-13T09:36:03.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:03.251+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.259043+0000) 2019-10-13T09:36:03.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:03.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.259413+0000) 2019-10-13T09:36:03.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:03.255+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.259783+0000) 2019-10-13T09:36:03.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:03.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.260896+0000) 2019-10-13T09:36:03.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:03.255+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.261424+0000) 2019-10-13T09:36:03.263 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:03.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:03.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:03.263+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.268285+0000) 2019-10-13T09:36:05.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:05.043+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:05.050081+0000) 2019-10-13T09:36:06.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:06.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:06.270409+0000) 2019-10-13T09:36:08.266 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:08.259+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.265270+0000) 2019-10-13T09:36:08.267 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:08.259+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.266663+0000) 2019-10-13T09:36:08.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:08.259+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.267163+0000) 2019-10-13T09:36:08.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:08.263+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.267636+0000) 2019-10-13T09:36:08.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:08.263+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.268096+0000) 2019-10-13T09:36:08.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:08.263+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.268796+0000) 2019-10-13T09:36:08.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:08.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:08.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:08.263+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.269040+0000) 2019-10-13T09:36:08.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:08.267+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.272691+0000) 2019-10-13T09:36:08.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:08.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.531891+0000) 2019-10-13T09:36:08.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:08.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.538687+0000) 2019-10-13T09:36:09.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:09.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:09.522807+0000) 2019-10-13T09:36:11.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:11.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:11.829008+0000) 2019-10-13T09:36:12.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:12.019+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.021351+0000) 2019-10-13T09:36:13.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:13.007+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.007526+0000) 2019-10-13T09:36:13.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:13.267+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.273112+0000) 2019-10-13T09:36:13.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:13.267+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.273437+0000) 2019-10-13T09:36:13.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:13.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.273857+0000) 2019-10-13T09:36:13.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:13.267+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.274666+0000) 2019-10-13T09:36:13.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:13.267+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.275049+0000) 2019-10-13T09:36:13.276 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:13.271+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.275474+0000) 2019-10-13T09:36:13.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:13.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:13.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:13.271+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.276168+0000) 2019-10-13T09:36:13.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:13.275+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.280795+0000) 2019-10-13T09:36:14.986 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:36:14.986 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:14.990 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:36:14.990 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:15.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:15.047+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:15.050268+0000) 2019-10-13T09:36:16.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:16.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:16.270565+0000) 2019-10-13T09:36:18.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:18.275+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.278846+0000) 2019-10-13T09:36:18.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:18.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.279821+0000) 2019-10-13T09:36:18.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:18.279+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.280838+0000) 2019-10-13T09:36:18.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:18.279+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.280319+0000) 2019-10-13T09:36:18.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:18.279+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.281165+0000) 2019-10-13T09:36:18.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:18.274+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.281597+0000) 2019-10-13T09:36:18.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:18.279+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:18.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:18.279+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.282696+0000) 2019-10-13T09:36:18.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:18.282+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.289271+0000) 2019-10-13T09:36:18.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:18.531+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.532073+0000) 2019-10-13T09:36:18.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:18.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.538869+0000) 2019-10-13T09:36:19.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:19.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:19.523005+0000) 2019-10-13T09:36:21.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:21.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:21.829245+0000) 2019-10-13T09:36:22.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:22.019+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:22.021549+0000) 2019-10-13T09:36:23.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:23.010+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.013936+0000) 2019-10-13T09:36:23.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:23.283+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.286191+0000) 2019-10-13T09:36:23.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:23.283+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.286657+0000) 2019-10-13T09:36:23.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:23.283+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.286882+0000) 2019-10-13T09:36:23.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:23.283+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.287092+0000) 2019-10-13T09:36:23.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:23.287+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.287886+0000) 2019-10-13T09:36:23.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:23.287+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.288135+0000) 2019-10-13T09:36:23.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:23.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.288523+0000) 2019-10-13T09:36:23.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:23.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:23.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:23.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.293984+0000) 2019-10-13T09:36:25.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:25.047+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:25.050464+0000) 2019-10-13T09:36:26.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:26.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:26.270760+0000) 2019-10-13T09:36:28.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:28.291+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.293471+0000) 2019-10-13T09:36:28.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:28.291+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.294114+0000) 2019-10-13T09:36:28.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:28.291+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.294159+0000) 2019-10-13T09:36:28.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:28.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.294528+0000) 2019-10-13T09:36:28.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:28.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.294981+0000) 2019-10-13T09:36:28.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:28.290+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.296199+0000) 2019-10-13T09:36:28.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:28.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.296234+0000) 2019-10-13T09:36:28.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:28.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79595 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:28.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:28.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.302792+0000) 2019-10-13T09:36:28.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:28.531+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.532256+0000) 2019-10-13T09:36:28.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:28.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.539053+0000) 2019-10-13T09:36:29.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:29.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:29.523213+0000) 2019-10-13T09:36:31.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:31.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:31.829443+0000) 2019-10-13T09:36:32.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:32.018+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:32.021783+0000) 2019-10-13T09:36:33.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:33.014+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.020752+0000) 2019-10-13T09:36:33.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:33.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.300190+0000) 2019-10-13T09:36:33.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:33.298+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.301085+0000) 2019-10-13T09:36:33.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:33.298+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.302043+0000) 2019-10-13T09:36:33.304 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:33.298+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.302774+0000) 2019-10-13T09:36:33.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:33.302+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.303572+0000) 2019-10-13T09:36:33.305 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:33.302+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.303942+0000) 2019-10-13T09:36:33.305 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:33.302+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.304190+0000) 2019-10-13T09:36:33.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:33.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:33.311 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:33.306+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.310369+0000) 2019-10-13T09:36:35.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:35.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:35.050670+0000) 2019-10-13T09:36:36.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:36.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:36.270949+0000) 2019-10-13T09:36:38.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:38.302+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.307562+0000) 2019-10-13T09:36:38.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:38.306+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.308979+0000) 2019-10-13T09:36:38.310 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:38.306+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.309325+0000) 2019-10-13T09:36:38.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:38.306+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.308711+0000) 2019-10-13T09:36:38.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:38.306+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.309640+0000) 2019-10-13T09:36:38.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:38.306+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.310104+0000) 2019-10-13T09:36:38.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:38.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79662 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:38.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:38.306+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.310288+0000) 2019-10-13T09:36:38.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:38.310+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.317296+0000) 2019-10-13T09:36:38.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:38.530+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.532449+0000) 2019-10-13T09:36:38.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:38.534+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.539259+0000) 2019-10-13T09:36:39.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:39.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:39.523410+0000) 2019-10-13T09:36:41.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:41.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:41.829642+0000) 2019-10-13T09:36:42.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:42.018+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:42.021984+0000) 2019-10-13T09:36:43.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:43.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.026653+0000) 2019-10-13T09:36:43.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:43.310+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.313723+0000) 2019-10-13T09:36:43.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:43.310+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.314291+0000) 2019-10-13T09:36:43.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:43.310+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.314646+0000) 2019-10-13T09:36:43.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:43.314+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.315758+0000) 2019-10-13T09:36:43.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:43.314+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.316133+0000) 2019-10-13T09:36:43.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:43.314+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.316512+0000) 2019-10-13T09:36:43.318 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:43.314+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.317094+0000) 2019-10-13T09:36:43.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:43.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:43.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:43.318+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.324844+0000) 2019-10-13T09:36:45.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:45.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:45.050932+0000) 2019-10-13T09:36:45.173 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:36:45.173 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:45.177 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:36:45.177 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:46.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:46.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:46.271138+0000) 2019-10-13T09:36:48.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:48.318+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.322094+0000) 2019-10-13T09:36:48.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:48.318+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.322519+0000) 2019-10-13T09:36:48.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:48.318+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.322877+0000) 2019-10-13T09:36:48.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:48.318+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.323115+0000) 2019-10-13T09:36:48.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:48.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.323589+0000) 2019-10-13T09:36:48.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:48.318+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.324145+0000) 2019-10-13T09:36:48.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:48.322+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.324406+0000) 2019-10-13T09:36:48.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:48.322+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:48.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:48.322+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.327929+0000) 2019-10-13T09:36:48.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:48.530+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.532599+0000) 2019-10-13T09:36:48.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:48.538+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.539492+0000) 2019-10-13T09:36:49.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:49.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:49.523623+0000) 2019-10-13T09:36:51.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:51.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:51.829867+0000) 2019-10-13T09:36:52.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:52.018+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:52.022185+0000) 2019-10-13T09:36:53.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:53.034+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.038331+0000) 2019-10-13T09:36:53.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:53.326+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.328909+0000) 2019-10-13T09:36:53.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:53.326+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.329563+0000) 2019-10-13T09:36:53.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:53.326+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.329477+0000) 2019-10-13T09:36:53.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:53.326+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.330395+0000) 2019-10-13T09:36:53.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:53.326+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.331143+0000) 2019-10-13T09:36:53.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:53.326+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.331892+0000) 2019-10-13T09:36:53.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:53.330+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.332488+0000) 2019-10-13T09:36:53.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:53.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:53.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:53.330+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.334827+0000) 2019-10-13T09:36:55.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:55.046+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:55.051167+0000) 2019-10-13T09:36:56.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:56.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:56.271367+0000) 2019-10-13T09:36:58.337 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:58.334+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.335900+0000) 2019-10-13T09:36:58.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:36:58.334+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.336372+0000) 2019-10-13T09:36:58.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:36:58.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.337865+0000) 2019-10-13T09:36:58.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:36:58.334+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.338235+0000) 2019-10-13T09:36:58.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:58.334+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.338753+0000) 2019-10-13T09:36:58.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:58.334+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.339178+0000) 2019-10-13T09:36:58.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:36:58.338+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.339889+0000) 2019-10-13T09:36:58.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:36:58.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:36:58.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:36:58.338+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.345975+0000) 2019-10-13T09:36:58.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:36:58.530+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.532772+0000) 2019-10-13T09:36:58.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:36:58.538+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.539701+0000) 2019-10-13T09:36:59.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:36:59.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:59.523864+0000) 2019-10-13T09:37:01.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:01.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:01.830065+0000) 2019-10-13T09:37:02.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:02.018+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:02.022383+0000) 2019-10-13T09:37:03.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:03.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.044568+0000) 2019-10-13T09:37:03.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:03.342+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.343553+0000) 2019-10-13T09:37:03.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:03.342+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.343916+0000) 2019-10-13T09:37:03.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:03.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.344399+0000) 2019-10-13T09:37:03.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:03.342+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.344626+0000) 2019-10-13T09:37:03.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:03.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.344961+0000) 2019-10-13T09:37:03.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:03.342+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.345446+0000) 2019-10-13T09:37:03.348 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:03.342+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.345920+0000) 2019-10-13T09:37:03.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:03.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:03.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:03.346+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.352882+0000) 2019-10-13T09:37:05.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:05.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:05.051353+0000) 2019-10-13T09:37:06.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:06.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:06.271568+0000) 2019-10-13T09:37:08.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:08.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.351171+0000) 2019-10-13T09:37:08.353 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:08.350+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.351455+0000) 2019-10-13T09:37:08.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:08.350+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.351712+0000) 2019-10-13T09:37:08.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:08.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.352368+0000) 2019-10-13T09:37:08.355 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:08.350+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.352552+0000) 2019-10-13T09:37:08.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:08.354+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.354209+0000) 2019-10-13T09:37:08.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:08.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.354604+0000) 2019-10-13T09:37:08.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:08.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:08.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:08.358+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.359974+0000) 2019-10-13T09:37:08.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:08.530+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.532946+0000) 2019-10-13T09:37:08.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:08.538+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.539872+0000) 2019-10-13T09:37:09.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:09.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:09.524013+0000) 2019-10-13T09:37:11.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:11.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:11.830268+0000) 2019-10-13T09:37:12.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:12.018+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:12.022579+0000) 2019-10-13T09:37:13.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:13.049+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.050977+0000) 2019-10-13T09:37:13.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:13.353+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.358000+0000) 2019-10-13T09:37:13.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:13.353+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.358769+0000) 2019-10-13T09:37:13.361 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:13.357+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.359333+0000) 2019-10-13T09:37:13.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:13.357+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.360434+0000) 2019-10-13T09:37:13.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:13.357+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.360249+0000) 2019-10-13T09:37:13.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:13.357+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.360224+0000) 2019-10-13T09:37:13.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:13.357+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.361337+0000) 2019-10-13T09:37:13.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:13.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:13.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:13.365+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.366835+0000) 2019-10-13T09:37:15.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:15.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:15.051561+0000) 2019-10-13T09:37:15.435 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:37:15.435 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:15.439 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:37:15.439 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:16.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:16.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:16.271823+0000) 2019-10-13T09:37:18.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:18.361+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.364596+0000) 2019-10-13T09:37:18.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:18.361+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.364866+0000) 2019-10-13T09:37:18.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:18.361+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.365305+0000) 2019-10-13T09:37:18.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:18.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.365701+0000) 2019-10-13T09:37:18.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:18.361+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.366058+0000) 2019-10-13T09:37:18.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:18.361+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.367072+0000) 2019-10-13T09:37:18.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:18.365+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.368094+0000) 2019-10-13T09:37:18.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:18.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:18.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:18.369+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.372934+0000) 2019-10-13T09:37:18.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:18.529+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.533125+0000) 2019-10-13T09:37:18.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:18.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.540053+0000) 2019-10-13T09:37:19.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:19.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:19.524239+0000) 2019-10-13T09:37:21.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:21.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:21.830484+0000) 2019-10-13T09:37:22.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:22.017+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:22.022773+0000) 2019-10-13T09:37:23.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:23.053+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.057599+0000) 2019-10-13T09:37:23.374 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:23.369+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.373027+0000) 2019-10-13T09:37:23.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:23.369+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.373414+0000) 2019-10-13T09:37:23.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:23.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.373987+0000) 2019-10-13T09:37:23.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:23.369+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.373743+0000) 2019-10-13T09:37:23.376 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:23.369+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.374694+0000) 2019-10-13T09:37:23.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:23.369+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.375003+0000) 2019-10-13T09:37:23.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:23.373+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.376154+0000) 2019-10-13T09:37:23.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:23.373+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:23.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:23.377+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.381464+0000) 2019-10-13T09:37:25.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:25.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:25.051745+0000) 2019-10-13T09:37:26.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:26.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:26.272016+0000) 2019-10-13T09:37:28.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:28.377+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.380755+0000) 2019-10-13T09:37:28.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:28.377+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.381034+0000) 2019-10-13T09:37:28.383 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:28.377+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.381389+0000) 2019-10-13T09:37:28.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:28.381+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.382117+0000) 2019-10-13T09:37:28.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:28.377+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.382283+0000) 2019-10-13T09:37:28.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:28.377+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.382859+0000) 2019-10-13T09:37:28.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:28.377+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.382975+0000) 2019-10-13T09:37:28.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:28.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 79997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:28.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:28.385+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.389644+0000) 2019-10-13T09:37:28.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:28.529+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.533337+0000) 2019-10-13T09:37:28.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:28.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.540248+0000) 2019-10-13T09:37:29.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:29.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:29.524448+0000) 2019-10-13T09:37:31.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:31.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:31.830697+0000) 2019-10-13T09:37:32.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:32.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:32.022982+0000) 2019-10-13T09:37:33.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:33.061+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.064032+0000) 2019-10-13T09:37:33.389 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:33.385+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.387815+0000) 2019-10-13T09:37:33.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:33.385+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.388130+0000) 2019-10-13T09:37:33.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:33.385+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.388527+0000) 2019-10-13T09:37:33.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:33.385+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.389442+0000) 2019-10-13T09:37:33.391 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:33.385+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.389012+0000) 2019-10-13T09:37:33.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:33.385+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.389644+0000) 2019-10-13T09:37:33.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:33.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:33.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:33.385+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.390854+0000) 2019-10-13T09:37:33.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:33.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.398267+0000) 2019-10-13T09:37:35.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:35.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:35.051950+0000) 2019-10-13T09:37:36.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:36.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:36.272275+0000) 2019-10-13T09:37:38.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:38.389+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.394174+0000) 2019-10-13T09:37:38.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:38.389+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.394571+0000) 2019-10-13T09:37:38.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:38.389+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.395187+0000) 2019-10-13T09:37:38.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:38.393+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.395490+0000) 2019-10-13T09:37:38.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:38.393+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.396384+0000) 2019-10-13T09:37:38.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:38.393+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.396907+0000) 2019-10-13T09:37:38.398 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:38.393+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.397394+0000) 2019-10-13T09:37:38.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:38.393+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:38.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:38.401+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.404118+0000) 2019-10-13T09:37:38.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:38.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.533511+0000) 2019-10-13T09:37:38.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:38.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.540427+0000) 2019-10-13T09:37:39.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:39.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:39.524635+0000) 2019-10-13T09:37:41.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:41.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:41.830892+0000) 2019-10-13T09:37:42.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:42.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:42.023176+0000) 2019-10-13T09:37:43.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:43.069+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.070397+0000) 2019-10-13T09:37:43.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:43.397+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.400828+0000) 2019-10-13T09:37:43.402 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:43.397+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.401426+0000) 2019-10-13T09:37:43.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:43.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.401792+0000) 2019-10-13T09:37:43.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:43.397+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.402166+0000) 2019-10-13T09:37:43.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:43.401+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.403171+0000) 2019-10-13T09:37:43.404 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:43.397+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.402948+0000) 2019-10-13T09:37:43.405 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:43.401+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.404173+0000) 2019-10-13T09:37:43.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:43.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:43.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:43.405+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.409674+0000) 2019-10-13T09:37:45.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:45.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:45.052179+0000) 2019-10-13T09:37:45.492 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:37:45.492 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:45.495 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:37:45.496 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:46.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:46.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:46.272466+0000) 2019-10-13T09:37:48.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:48.405+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.407878+0000) 2019-10-13T09:37:48.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:48.405+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.408712+0000) 2019-10-13T09:37:48.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:48.405+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.408935+0000) 2019-10-13T09:37:48.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:48.405+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.409819+0000) 2019-10-13T09:37:48.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:48.405+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.410332+0000) 2019-10-13T09:37:48.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:48.405+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.410893+0000) 2019-10-13T09:37:48.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:48.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:48.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:48.409+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.411907+0000) 2019-10-13T09:37:48.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:48.413+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.416033+0000) 2019-10-13T09:37:48.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:48.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.533687+0000) 2019-10-13T09:37:48.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:48.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.540601+0000) 2019-10-13T09:37:49.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:49.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:49.524869+0000) 2019-10-13T09:37:51.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:51.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:51.831109+0000) 2019-10-13T09:37:52.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:52.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:52.023405+0000) 2019-10-13T09:37:53.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:53.073+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.076855+0000) 2019-10-13T09:37:53.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:53.413+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.415632+0000) 2019-10-13T09:37:53.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:53.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.416267+0000) 2019-10-13T09:37:53.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:53.413+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.416781+0000) 2019-10-13T09:37:53.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:53.413+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.416976+0000) 2019-10-13T09:37:53.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:53.413+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.417552+0000) 2019-10-13T09:37:53.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:53.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.417823+0000) 2019-10-13T09:37:53.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:53.413+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.418562+0000) 2019-10-13T09:37:53.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:53.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:53.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:53.421+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.422368+0000) 2019-10-13T09:37:55.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:55.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:55.052413+0000) 2019-10-13T09:37:56.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:56.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:56.272664+0000) 2019-10-13T09:37:58.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:37:58.416+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.422182+0000) 2019-10-13T09:37:58.424 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:37:58.416+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.423050+0000) 2019-10-13T09:37:58.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:58.421+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.423426+0000) 2019-10-13T09:37:58.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:37:58.420+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.423957+0000) 2019-10-13T09:37:58.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:58.420+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.424344+0000) 2019-10-13T09:37:58.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:58.420+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.425271+0000) 2019-10-13T09:37:58.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:37:58.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:37:58.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:37:58.420+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.425796+0000) 2019-10-13T09:37:58.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:37:58.425+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.429655+0000) 2019-10-13T09:37:58.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:37:58.528+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.533893+0000) 2019-10-13T09:37:58.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:37:58.536+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.540773+0000) 2019-10-13T09:37:59.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:37:59.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:59.525081+0000) 2019-10-13T09:38:01.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:01.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:01.831338+0000) 2019-10-13T09:38:02.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:02.020+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:02.023638+0000) 2019-10-13T09:38:03.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:03.081+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.083179+0000) 2019-10-13T09:38:03.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:03.424+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.428961+0000) 2019-10-13T09:38:03.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:03.424+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.429370+0000) 2019-10-13T09:38:03.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:03.424+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.429880+0000) 2019-10-13T09:38:03.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:03.424+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.430097+0000) 2019-10-13T09:38:03.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:03.424+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.431010+0000) 2019-10-13T09:38:03.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:03.429+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.431485+0000) 2019-10-13T09:38:03.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:03.428+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.432024+0000) 2019-10-13T09:38:03.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:03.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80232 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:03.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:03.437+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.442129+0000) 2019-10-13T09:38:05.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:05.048+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:05.052640+0000) 2019-10-13T09:38:06.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:06.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:06.272880+0000) 2019-10-13T09:38:08.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:08.432+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.435950+0000) 2019-10-13T09:38:08.438 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:08.432+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.436612+0000) 2019-10-13T09:38:08.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:08.432+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.437052+0000) 2019-10-13T09:38:08.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:08.432+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.437454+0000) 2019-10-13T09:38:08.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:08.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.437744+0000) 2019-10-13T09:38:08.439 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:08.432+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.438038+0000) 2019-10-13T09:38:08.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:08.432+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.438687+0000) 2019-10-13T09:38:08.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:08.436+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:08.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:08.444+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.446238+0000) 2019-10-13T09:38:08.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:08.532+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.534068+0000) 2019-10-13T09:38:08.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:08.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.540948+0000) 2019-10-13T09:38:09.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:09.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:09.525305+0000) 2019-10-13T09:38:11.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:11.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:11.831545+0000) 2019-10-13T09:38:12.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:12.020+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:12.023838+0000) 2019-10-13T09:38:13.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:13.084+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.089473+0000) 2019-10-13T09:38:13.444 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:13.440+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.443421+0000) 2019-10-13T09:38:13.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:13.440+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.443880+0000) 2019-10-13T09:38:13.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:13.440+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.444205+0000) 2019-10-13T09:38:13.446 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:13.440+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.444885+0000) 2019-10-13T09:38:13.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:13.440+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.445735+0000) 2019-10-13T09:38:13.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:13.440+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.446054+0000) 2019-10-13T09:38:13.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:13.440+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.446735+0000) 2019-10-13T09:38:13.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:13.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:13.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:13.464+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.468845+0000) 2019-10-13T09:38:15.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:15.048+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:15.052838+0000) 2019-10-13T09:38:15.648 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:38:15.648 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:15.652 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:38:15.652 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:16.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:16.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:16.273074+0000) 2019-10-13T09:38:18.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:18.444+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.450893+0000) 2019-10-13T09:38:18.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:18.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.451800+0000) 2019-10-13T09:38:18.453 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:18.448+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.452142+0000) 2019-10-13T09:38:18.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:18.448+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.452482+0000) 2019-10-13T09:38:18.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:18.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.452927+0000) 2019-10-13T09:38:18.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:18.448+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.453211+0000) 2019-10-13T09:38:18.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:18.448+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.453682+0000) 2019-10-13T09:38:18.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:18.448+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:18.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:18.452+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.457739+0000) 2019-10-13T09:38:18.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:18.532+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.534249+0000) 2019-10-13T09:38:18.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:18.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.541124+0000) 2019-10-13T09:38:19.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:19.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:19.525491+0000) 2019-10-13T09:38:21.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:21.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:21.831765+0000) 2019-10-13T09:38:22.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:22.020+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:22.024044+0000) 2019-10-13T09:38:23.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:23.092+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.096136+0000) 2019-10-13T09:38:23.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:23.452+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.457576+0000) 2019-10-13T09:38:23.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:23.452+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.458919+0000) 2019-10-13T09:38:23.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:23.456+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.459256+0000) 2019-10-13T09:38:23.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:23.456+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.459609+0000) 2019-10-13T09:38:23.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:23.456+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.459895+0000) 2019-10-13T09:38:23.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:23.456+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.460472+0000) 2019-10-13T09:38:23.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:23.456+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.461161+0000) 2019-10-13T09:38:23.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:23.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:23.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:23.464+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.468190+0000) 2019-10-13T09:38:25.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:25.052+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:25.053030+0000) 2019-10-13T09:38:26.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:26.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:26.273293+0000) 2019-10-13T09:38:28.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:28.460+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.465974+0000) 2019-10-13T09:38:28.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:28.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.466907+0000) 2019-10-13T09:38:28.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:28.460+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.467269+0000) 2019-10-13T09:38:28.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:28.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.468159+0000) 2019-10-13T09:38:28.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:28.464+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.468602+0000) 2019-10-13T09:38:28.470 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:28.464+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.469018+0000) 2019-10-13T09:38:28.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:28.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80398 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:28.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:28.464+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.469965+0000) 2019-10-13T09:38:28.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:28.472+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.477246+0000) 2019-10-13T09:38:28.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:28.528+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.534442+0000) 2019-10-13T09:38:28.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:28.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.541314+0000) 2019-10-13T09:38:29.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:29.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:29.525704+0000) 2019-10-13T09:38:31.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:31.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:31.831959+0000) 2019-10-13T09:38:32.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:32.024+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:32.024247+0000) 2019-10-13T09:38:33.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:33.100+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.102506+0000) 2019-10-13T09:38:33.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:33.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.473184+0000) 2019-10-13T09:38:33.475 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:33.468+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.474325+0000) 2019-10-13T09:38:33.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:33.468+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.474790+0000) 2019-10-13T09:38:33.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:33.472+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.475394+0000) 2019-10-13T09:38:33.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:33.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.476093+0000) 2019-10-13T09:38:33.477 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:33.472+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.476740+0000) 2019-10-13T09:38:33.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:33.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:33.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:33.472+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.477360+0000) 2019-10-13T09:38:33.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:33.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.480228+0000) 2019-10-13T09:38:35.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:35.048+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:35.053251+0000) 2019-10-13T09:38:36.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:36.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:36.273526+0000) 2019-10-13T09:38:38.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:38.476+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.480069+0000) 2019-10-13T09:38:38.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:38.476+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.480605+0000) 2019-10-13T09:38:38.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:38.476+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.481382+0000) 2019-10-13T09:38:38.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:38.476+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.481141+0000) 2019-10-13T09:38:38.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:38.476+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.482030+0000) 2019-10-13T09:38:38.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:38.476+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.482680+0000) 2019-10-13T09:38:38.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:38.476+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.483090+0000) 2019-10-13T09:38:38.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:38.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:38.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:38.484+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.489867+0000) 2019-10-13T09:38:38.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:38.528+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.534619+0000) 2019-10-13T09:38:38.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:38.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.541485+0000) 2019-10-13T09:38:39.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:39.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:39.525898+0000) 2019-10-13T09:38:41.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:41.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:41.832161+0000) 2019-10-13T09:38:42.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:42.023+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:42.024512+0000) 2019-10-13T09:38:43.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:43.108+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.108992+0000) 2019-10-13T09:38:43.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:43.483+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.487576+0000) 2019-10-13T09:38:43.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:43.483+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.488028+0000) 2019-10-13T09:38:43.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:43.483+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.488991+0000) 2019-10-13T09:38:43.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:43.483+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.489666+0000) 2019-10-13T09:38:43.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:43.483+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.489357+0000) 2019-10-13T09:38:43.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:43.484+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.490060+0000) 2019-10-13T09:38:43.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:43.483+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.490573+0000) 2019-10-13T09:38:43.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:43.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:43.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:43.492+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.497112+0000) 2019-10-13T09:38:45.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:45.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:45.053459+0000) 2019-10-13T09:38:45.750 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:38:45.751 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:45.754 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:38:45.754 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:46.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:46.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:46.273729+0000) 2019-10-13T09:38:48.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:48.487+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.494190+0000) 2019-10-13T09:38:48.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:48.487+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.494949+0000) 2019-10-13T09:38:48.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:48.487+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.495253+0000) 2019-10-13T09:38:48.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:48.492+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.495526+0000) 2019-10-13T09:38:48.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:48.491+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.497005+0000) 2019-10-13T09:38:48.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:48.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:48.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:48.491+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.497465+0000) 2019-10-13T09:38:48.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:48.491+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.498132+0000) 2019-10-13T09:38:48.503 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:48.500+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.502441+0000) 2019-10-13T09:38:48.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:48.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.534803+0000) 2019-10-13T09:38:48.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:48.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.541661+0000) 2019-10-13T09:38:49.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:49.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:49.526090+0000) 2019-10-13T09:38:51.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:51.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:51.832366+0000) 2019-10-13T09:38:52.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:52.023+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:52.024740+0000) 2019-10-13T09:38:53.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:53.112+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.115357+0000) 2019-10-13T09:38:53.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:53.495+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.500182+0000) 2019-10-13T09:38:53.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:53.495+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.500561+0000) 2019-10-13T09:38:53.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:53.495+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.501086+0000) 2019-10-13T09:38:53.503 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:53.495+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.501458+0000) 2019-10-13T09:38:53.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:53.495+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.502198+0000) 2019-10-13T09:38:53.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:53.500+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.502901+0000) 2019-10-13T09:38:53.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:53.495+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.503243+0000) 2019-10-13T09:38:53.505 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:53.499+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:53.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:53.504+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.509581+0000) 2019-10-13T09:38:55.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:55.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:55.053661+0000) 2019-10-13T09:38:56.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:56.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:56.273928+0000) 2019-10-13T09:38:58.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:58.503+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.507404+0000) 2019-10-13T09:38:58.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:38:58.503+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.508231+0000) 2019-10-13T09:38:58.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:58.503+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.508460+0000) 2019-10-13T09:38:58.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:58.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.509462+0000) 2019-10-13T09:38:58.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:38:58.503+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.509891+0000) 2019-10-13T09:38:58.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:38:58.503+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.510345+0000) 2019-10-13T09:38:58.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:38:58.503+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.511081+0000) 2019-10-13T09:38:58.513 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:38:58.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:38:58.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:38:58.512+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.516439+0000) 2019-10-13T09:38:58.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:38:58.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.534960+0000) 2019-10-13T09:38:58.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:38:58.535+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.541851+0000) 2019-10-13T09:38:59.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:38:59.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:59.526321+0000) 2019-10-13T09:39:01.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:01.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:01.832575+0000) 2019-10-13T09:39:02.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:02.019+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:02.024957+0000) 2019-10-13T09:39:03.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:03.115+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.121677+0000) 2019-10-13T09:39:03.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:03.511+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.515881+0000) 2019-10-13T09:39:03.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:03.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.516150+0000) 2019-10-13T09:39:03.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:03.511+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.516637+0000) 2019-10-13T09:39:03.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:03.511+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.516946+0000) 2019-10-13T09:39:03.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:03.511+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.518094+0000) 2019-10-13T09:39:03.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:03.511+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.518544+0000) 2019-10-13T09:39:03.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:03.511+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.519056+0000) 2019-10-13T09:39:03.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:03.515+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:03.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:03.519+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.525939+0000) 2019-10-13T09:39:05.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:05.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:05.053852+0000) 2019-10-13T09:39:06.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:06.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:06.274151+0000) 2019-10-13T09:39:08.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:08.519+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.523763+0000) 2019-10-13T09:39:08.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:08.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.524043+0000) 2019-10-13T09:39:08.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:08.519+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.525104+0000) 2019-10-13T09:39:08.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:08.519+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.525687+0000) 2019-10-13T09:39:08.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:08.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.526145+0000) 2019-10-13T09:39:08.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:08.519+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.526477+0000) 2019-10-13T09:39:08.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:08.523+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.527435+0000) 2019-10-13T09:39:08.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:08.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:08.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:08.527+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.532403+0000) 2019-10-13T09:39:08.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:08.527+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.535085+0000) 2019-10-13T09:39:08.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:08.539+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.542071+0000) 2019-10-13T09:39:09.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:09.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:09.526561+0000) 2019-10-13T09:39:11.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:11.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:11.832772+0000) 2019-10-13T09:39:12.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:12.023+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:12.025186+0000) 2019-10-13T09:39:13.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:13.123+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.127977+0000) 2019-10-13T09:39:13.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:13.527+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.530329+0000) 2019-10-13T09:39:13.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:13.531+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.531726+0000) 2019-10-13T09:39:13.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:13.527+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.532411+0000) 2019-10-13T09:39:13.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:13.531+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.532977+0000) 2019-10-13T09:39:13.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:13.531+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.533386+0000) 2019-10-13T09:39:13.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:13.531+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.533641+0000) 2019-10-13T09:39:13.535 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:13.531+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.533975+0000) 2019-10-13T09:39:13.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:13.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:13.541 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:13.535+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.540790+0000) 2019-10-13T09:39:15.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:15.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:15.054048+0000) 2019-10-13T09:39:15.844 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:39:15.844 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:15.848 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:39:15.848 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:16.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:16.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:16.274355+0000) 2019-10-13T09:39:18.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:18.531+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.535257+0000) 2019-10-13T09:39:18.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:18.531+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.538044+0000) 2019-10-13T09:39:18.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:18.535+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.538735+0000) 2019-10-13T09:39:18.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:18.539+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.539362+0000) 2019-10-13T09:39:18.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:18.539+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.539714+0000) 2019-10-13T09:39:18.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:18.539+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.540529+0000) 2019-10-13T09:39:18.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:18.539+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.541036+0000) 2019-10-13T09:39:18.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:18.539+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.541411+0000) 2019-10-13T09:39:18.543 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:18.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:18.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:18.539+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.542197+0000) 2019-10-13T09:39:18.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:18.547+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.550216+0000) 2019-10-13T09:39:19.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:19.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:19.526745+0000) 2019-10-13T09:39:21.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:21.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:21.832976+0000) 2019-10-13T09:39:22.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:22.023+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:22.025385+0000) 2019-10-13T09:39:23.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:23.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.134367+0000) 2019-10-13T09:39:23.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:23.542+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.545138+0000) 2019-10-13T09:39:23.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:23.542+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.545461+0000) 2019-10-13T09:39:23.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:23.539+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.546000+0000) 2019-10-13T09:39:23.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:23.542+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.546528+0000) 2019-10-13T09:39:23.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:23.542+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.546937+0000) 2019-10-13T09:39:23.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:23.546+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.547295+0000) 2019-10-13T09:39:23.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:23.546+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.548283+0000) 2019-10-13T09:39:23.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:23.546+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:23.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:23.551+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.554220+0000) 2019-10-13T09:39:25.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:25.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:25.054250+0000) 2019-10-13T09:39:26.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:26.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:26.274577+0000) 2019-10-13T09:39:28.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:28.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.535500+0000) 2019-10-13T09:39:28.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:28.538+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.542388+0000) 2019-10-13T09:39:28.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:28.550+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.552384+0000) 2019-10-13T09:39:28.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:28.550+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.552857+0000) 2019-10-13T09:39:28.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:28.547+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.553238+0000) 2019-10-13T09:39:28.555 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:28.550+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.553838+0000) 2019-10-13T09:39:28.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:28.550+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.554520+0000) 2019-10-13T09:39:28.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:28.550+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.554325+0000) 2019-10-13T09:39:28.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:28.550+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.554718+0000) 2019-10-13T09:39:28.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:28.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:28.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:28.555+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.562073+0000) 2019-10-13T09:39:29.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:29.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.526926+0000) 2019-10-13T09:39:31.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:31.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:31.833182+0000) 2019-10-13T09:39:32.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:32.022+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:32.025599+0000) 2019-10-13T09:39:33.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:33.139+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.140777+0000) 2019-10-13T09:39:33.561 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:33.558+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.559640+0000) 2019-10-13T09:39:33.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:33.558+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.559993+0000) 2019-10-13T09:39:33.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:33.558+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.560366+0000) 2019-10-13T09:39:33.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:33.558+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.560743+0000) 2019-10-13T09:39:33.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:33.558+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.561268+0000) 2019-10-13T09:39:33.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:33.558+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.561574+0000) 2019-10-13T09:39:33.563 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:33.555+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.561996+0000) 2019-10-13T09:39:33.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:33.563+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.569417+0000) 2019-10-13T09:39:33.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:33.562+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:35.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:35.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:35.054433+0000) 2019-10-13T09:39:36.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:36.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:36.274772+0000) 2019-10-13T09:39:38.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:38.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.535695+0000) 2019-10-13T09:39:38.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:38.538+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.542584+0000) 2019-10-13T09:39:38.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:38.562+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.566754+0000) 2019-10-13T09:39:38.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:38.562+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.567154+0000) 2019-10-13T09:39:38.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:38.566+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.568210+0000) 2019-10-13T09:39:38.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:38.566+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.568870+0000) 2019-10-13T09:39:38.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:38.566+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.569497+0000) 2019-10-13T09:39:38.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:38.566+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.569595+0000) 2019-10-13T09:39:38.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:38.563+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.569809+0000) 2019-10-13T09:39:38.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:38.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:38.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:38.571+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.574943+0000) 2019-10-13T09:39:39.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:39.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.527114+0000) 2019-10-13T09:39:41.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:41.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:41.833334+0000) 2019-10-13T09:39:42.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:42.022+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:42.025830+0000) 2019-10-13T09:39:43.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:43.143+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.147018+0000) 2019-10-13T09:39:43.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:43.570+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.573465+0000) 2019-10-13T09:39:43.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:43.570+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.573860+0000) 2019-10-13T09:39:43.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:43.570+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.574901+0000) 2019-10-13T09:39:43.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:43.574+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.575331+0000) 2019-10-13T09:39:43.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:43.574+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.575819+0000) 2019-10-13T09:39:43.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:43.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.576140+0000) 2019-10-13T09:39:43.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:43.571+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.577322+0000) 2019-10-13T09:39:43.579 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:43.574+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:43.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:43.579+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.582910+0000) 2019-10-13T09:39:45.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:45.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:45.054605+0000) 2019-10-13T09:39:45.992 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:39:45.992 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:45.996 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:39:45.996 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:46.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:46.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:46.275004+0000) 2019-10-13T09:39:48.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:48.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.535914+0000) 2019-10-13T09:39:48.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:48.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.542789+0000) 2019-10-13T09:39:48.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:48.578+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.580286+0000) 2019-10-13T09:39:48.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:48.578+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.580963+0000) 2019-10-13T09:39:48.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:48.578+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.581383+0000) 2019-10-13T09:39:48.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:48.578+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.581783+0000) 2019-10-13T09:39:48.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:48.578+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.582765+0000) 2019-10-13T09:39:48.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:48.579+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.583058+0000) 2019-10-13T09:39:48.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:48.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.583771+0000) 2019-10-13T09:39:48.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:48.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80933 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:48.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:48.583+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.588910+0000) 2019-10-13T09:39:49.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:49.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.527287+0000) 2019-10-13T09:39:51.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:51.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:51.833601+0000) 2019-10-13T09:39:52.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:52.022+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:52.026056+0000) 2019-10-13T09:39:53.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:53.147+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.153518+0000) 2019-10-13T09:39:53.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:53.586+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.587838+0000) 2019-10-13T09:39:53.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:53.586+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.588224+0000) 2019-10-13T09:39:53.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:53.586+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.588621+0000) 2019-10-13T09:39:53.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:53.583+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.589499+0000) 2019-10-13T09:39:53.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:53.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.589361+0000) 2019-10-13T09:39:53.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:53.586+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.590578+0000) 2019-10-13T09:39:53.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:53.590+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.591447+0000) 2019-10-13T09:39:53.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:53.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 80970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:53.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:53.591+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.597623+0000) 2019-10-13T09:39:55.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:55.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:55.054803+0000) 2019-10-13T09:39:56.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:56.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:56.275197+0000) 2019-10-13T09:39:58.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:58.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.536156+0000) 2019-10-13T09:39:58.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:58.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.542993+0000) 2019-10-13T09:39:58.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:39:58.594+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.595353+0000) 2019-10-13T09:39:58.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:39:58.594+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.595645+0000) 2019-10-13T09:39:58.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:39:58.594+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.596684+0000) 2019-10-13T09:39:58.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:39:58.594+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.597286+0000) 2019-10-13T09:39:58.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:58.590+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.598067+0000) 2019-10-13T09:39:58.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:39:58.594+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.597758+0000) 2019-10-13T09:39:58.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:39:58.594+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.598953+0000) 2019-10-13T09:39:58.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:39:58.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:39:58.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:39:58.598+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.603952+0000) 2019-10-13T09:39:59.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:39:59.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.527466+0000) 2019-10-13T09:40:01.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:01.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:01.833804+0000) 2019-10-13T09:40:02.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:02.022+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:02.026281+0000) 2019-10-13T09:40:03.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:03.158+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.159982+0000) 2019-10-13T09:40:03.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:03.598+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.602569+0000) 2019-10-13T09:40:03.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:03.598+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.602915+0000) 2019-10-13T09:40:03.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:03.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.604141+0000) 2019-10-13T09:40:03.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:03.602+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.604922+0000) 2019-10-13T09:40:03.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:03.598+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.605202+0000) 2019-10-13T09:40:03.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:03.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.606039+0000) 2019-10-13T09:40:03.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:03.602+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.606378+0000) 2019-10-13T09:40:03.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:03.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:03.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:03.606+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.610154+0000) 2019-10-13T09:40:05.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:05.050+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:05.055024+0000) 2019-10-13T09:40:06.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:06.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:06.275386+0000) 2019-10-13T09:40:08.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:08.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.536379+0000) 2019-10-13T09:40:08.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:08.537+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.543150+0000) 2019-10-13T09:40:08.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:08.605+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.610209+0000) 2019-10-13T09:40:08.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:08.605+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.610625+0000) 2019-10-13T09:40:08.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:08.606+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.611461+0000) 2019-10-13T09:40:08.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:08.609+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.612063+0000) 2019-10-13T09:40:08.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:08.609+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.612703+0000) 2019-10-13T09:40:08.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:08.609+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.613205+0000) 2019-10-13T09:40:08.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:08.609+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.613166+0000) 2019-10-13T09:40:08.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:08.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:08.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:08.614+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.618468+0000) 2019-10-13T09:40:09.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:09.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.527661+0000) 2019-10-13T09:40:11.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:11.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:11.834002+0000) 2019-10-13T09:40:12.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:12.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:12.026483+0000) 2019-10-13T09:40:13.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:13.162+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.166326+0000) 2019-10-13T09:40:13.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:13.614+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.618179+0000) 2019-10-13T09:40:13.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:13.613+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.619037+0000) 2019-10-13T09:40:13.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:13.617+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.619505+0000) 2019-10-13T09:40:13.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:13.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.619878+0000) 2019-10-13T09:40:13.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:13.617+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.620779+0000) 2019-10-13T09:40:13.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:13.617+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.621479+0000) 2019-10-13T09:40:13.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:13.617+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.621790+0000) 2019-10-13T09:40:13.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:13.617+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:13.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:13.622+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.627141+0000) 2019-10-13T09:40:15.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:15.049+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:15.055226+0000) 2019-10-13T09:40:16.073 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:40:16.073 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:16.077 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:40:16.077 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:16.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:16.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:16.275589+0000) 2019-10-13T09:40:18.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:18.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.536547+0000) 2019-10-13T09:40:18.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:18.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.543346+0000) 2019-10-13T09:40:18.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:18.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.625090+0000) 2019-10-13T09:40:18.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:18.621+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.625529+0000) 2019-10-13T09:40:18.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:18.621+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.625946+0000) 2019-10-13T09:40:18.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:18.621+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.626338+0000) 2019-10-13T09:40:18.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:18.625+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.627886+0000) 2019-10-13T09:40:18.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:18.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.628451+0000) 2019-10-13T09:40:18.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:18.625+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.628805+0000) 2019-10-13T09:40:18.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:18.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:18.634 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:18.626+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.633654+0000) 2019-10-13T09:40:19.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:19.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.527865+0000) 2019-10-13T09:40:21.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:21.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:21.834203+0000) 2019-10-13T09:40:22.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:22.025+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:22.026703+0000) 2019-10-13T09:40:23.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:23.170+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.172925+0000) 2019-10-13T09:40:23.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:23.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.633358+0000) 2019-10-13T09:40:23.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:23.629+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.633595+0000) 2019-10-13T09:40:23.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:23.626+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.633928+0000) 2019-10-13T09:40:23.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:23.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.634299+0000) 2019-10-13T09:40:23.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:23.629+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.634680+0000) 2019-10-13T09:40:23.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:23.629+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.635008+0000) 2019-10-13T09:40:23.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:23.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.635632+0000) 2019-10-13T09:40:23.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:23.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:23.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:23.638+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.642450+0000) 2019-10-13T09:40:25.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:25.053+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:25.055444+0000) 2019-10-13T09:40:26.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:26.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:26.275807+0000) 2019-10-13T09:40:28.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:28.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.536718+0000) 2019-10-13T09:40:28.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:28.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.543543+0000) 2019-10-13T09:40:28.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:28.633+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.639144+0000) 2019-10-13T09:40:28.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:28.637+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.639642+0000) 2019-10-13T09:40:28.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:28.634+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.640743+0000) 2019-10-13T09:40:28.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:28.637+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.640966+0000) 2019-10-13T09:40:28.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:28.637+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.641337+0000) 2019-10-13T09:40:28.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:28.637+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.641570+0000) 2019-10-13T09:40:28.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:28.637+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.642650+0000) 2019-10-13T09:40:28.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:28.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:28.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:28.642+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.648482+0000) 2019-10-13T09:40:29.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:29.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.528037+0000) 2019-10-13T09:40:31.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:31.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:31.834420+0000) 2019-10-13T09:40:32.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:32.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:32.026934+0000) 2019-10-13T09:40:33.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:33.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.179397+0000) 2019-10-13T09:40:33.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:33.641+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.645929+0000) 2019-10-13T09:40:33.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:33.641+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.646436+0000) 2019-10-13T09:40:33.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:33.645+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.647676+0000) 2019-10-13T09:40:33.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:33.642+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.647819+0000) 2019-10-13T09:40:33.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:33.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.648465+0000) 2019-10-13T09:40:33.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:33.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.648701+0000) 2019-10-13T09:40:33.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:33.645+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.649675+0000) 2019-10-13T09:40:33.651 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:33.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:33.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:33.650+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.655503+0000) 2019-10-13T09:40:35.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:35.053+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:35.055632+0000) 2019-10-13T09:40:36.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:36.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:36.276035+0000) 2019-10-13T09:40:38.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:38.533+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.536912+0000) 2019-10-13T09:40:38.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:38.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.543766+0000) 2019-10-13T09:40:38.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:38.649+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.653357+0000) 2019-10-13T09:40:38.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:38.649+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.653962+0000) 2019-10-13T09:40:38.655 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:38.649+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.654134+0000) 2019-10-13T09:40:38.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:38.649+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.654502+0000) 2019-10-13T09:40:38.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:38.650+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.655147+0000) 2019-10-13T09:40:38.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:38.653+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.655467+0000) 2019-10-13T09:40:38.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:38.653+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.656766+0000) 2019-10-13T09:40:38.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:38.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:38.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:38.658+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.666132+0000) 2019-10-13T09:40:39.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:39.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.528234+0000) 2019-10-13T09:40:41.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:41.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:41.834620+0000) 2019-10-13T09:40:42.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:42.021+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:42.027163+0000) 2019-10-13T09:40:43.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:43.182+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.185716+0000) 2019-10-13T09:40:43.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:43.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.660359+0000) 2019-10-13T09:40:43.662 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:43.657+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.661350+0000) 2019-10-13T09:40:43.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:43.657+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.661739+0000) 2019-10-13T09:40:43.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:43.658+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.662106+0000) 2019-10-13T09:40:43.664 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:43.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.662377+0000) 2019-10-13T09:40:43.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:43.657+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.662699+0000) 2019-10-13T09:40:43.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:43.661+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.663581+0000) 2019-10-13T09:40:43.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:43.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:43.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:43.666+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.670811+0000) 2019-10-13T09:40:45.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:45.053+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:45.055856+0000) 2019-10-13T09:40:46.179 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:40:46.179 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:46.183 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:40:46.183 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:46.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:46.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:46.276235+0000) 2019-10-13T09:40:48.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:48.537+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.537144+0000) 2019-10-13T09:40:48.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:48.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.543960+0000) 2019-10-13T09:40:48.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:48.665+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.667679+0000) 2019-10-13T09:40:48.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:48.665+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.668315+0000) 2019-10-13T09:40:48.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:48.665+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.668715+0000) 2019-10-13T09:40:48.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:48.665+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.669066+0000) 2019-10-13T09:40:48.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:48.665+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.669418+0000) 2019-10-13T09:40:48.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:48.665+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.669841+0000) 2019-10-13T09:40:48.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:48.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:48.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:48.670+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.670461+0000) 2019-10-13T09:40:48.678 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:48.674+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.677552+0000) 2019-10-13T09:40:49.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:49.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.528401+0000) 2019-10-13T09:40:51.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:51.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:51.834831+0000) 2019-10-13T09:40:52.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:52.024+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:52.027369+0000) 2019-10-13T09:40:53.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:53.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.191602+0000) 2019-10-13T09:40:53.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:53.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.674332+0000) 2019-10-13T09:40:53.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:53.672+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.675780+0000) 2019-10-13T09:40:53.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:53.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.676207+0000) 2019-10-13T09:40:53.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:53.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.676450+0000) 2019-10-13T09:40:53.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:53.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.676791+0000) 2019-10-13T09:40:53.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:53.673+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.677711+0000) 2019-10-13T09:40:53.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:53.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:53.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:53.672+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.678168+0000) 2019-10-13T09:40:53.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:53.681+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.685494+0000) 2019-10-13T09:40:55.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:55.052+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:55.056050+0000) 2019-10-13T09:40:56.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:56.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:56.276433+0000) 2019-10-13T09:40:58.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:58.536+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.537350+0000) 2019-10-13T09:40:58.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:58.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.544154+0000) 2019-10-13T09:40:58.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:40:58.676+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.681689+0000) 2019-10-13T09:40:58.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:40:58.676+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.682122+0000) 2019-10-13T09:40:58.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:40:58.676+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.682788+0000) 2019-10-13T09:40:58.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:58.681+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.682989+0000) 2019-10-13T09:40:58.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:40:58.680+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.683576+0000) 2019-10-13T09:40:58.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:40:58.680+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.683995+0000) 2019-10-13T09:40:58.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:40:58.680+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.685095+0000) 2019-10-13T09:40:58.686 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:40:58.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:40:58.691 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:40:58.689+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.690508+0000) 2019-10-13T09:40:59.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:40:59.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.528602+0000) 2019-10-13T09:41:01.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:01.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:01.835060+0000) 2019-10-13T09:41:02.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:02.024+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:02.027569+0000) 2019-10-13T09:41:03.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:03.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.198192+0000) 2019-10-13T09:41:03.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:03.684+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.688951+0000) 2019-10-13T09:41:03.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:03.684+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.690112+0000) 2019-10-13T09:41:03.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:03.684+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.690430+0000) 2019-10-13T09:41:03.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:03.684+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.690786+0000) 2019-10-13T09:41:03.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:03.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.691539+0000) 2019-10-13T09:41:03.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:03.689+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.691983+0000) 2019-10-13T09:41:03.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:03.688+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.692301+0000) 2019-10-13T09:41:03.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:03.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:03.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:03.693+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.695223+0000) 2019-10-13T09:41:05.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:05.052+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:05.056253+0000) 2019-10-13T09:41:06.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:06.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:06.276653+0000) 2019-10-13T09:41:08.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:08.532+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.537570+0000) 2019-10-13T09:41:08.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:08.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.544352+0000) 2019-10-13T09:41:08.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:08.692+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.695613+0000) 2019-10-13T09:41:08.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:08.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.696543+0000) 2019-10-13T09:41:08.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:08.692+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.696802+0000) 2019-10-13T09:41:08.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:08.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.697384+0000) 2019-10-13T09:41:08.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:08.693+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.698031+0000) 2019-10-13T09:41:08.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:08.692+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.699247+0000) 2019-10-13T09:41:08.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:08.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.699702+0000) 2019-10-13T09:41:08.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:08.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:08.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:08.701+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.704829+0000) 2019-10-13T09:41:09.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:09.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.528799+0000) 2019-10-13T09:41:11.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:11.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:11.835255+0000) 2019-10-13T09:41:12.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:12.024+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:12.027768+0000) 2019-10-13T09:41:13.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:13.201+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.204491+0000) 2019-10-13T09:41:13.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:13.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.703888+0000) 2019-10-13T09:41:13.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:13.700+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.704692+0000) 2019-10-13T09:41:13.706 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:13.700+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.705190+0000) 2019-10-13T09:41:13.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:13.700+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.705597+0000) 2019-10-13T09:41:13.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:13.701+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.706024+0000) 2019-10-13T09:41:13.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:13.700+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.706153+0000) 2019-10-13T09:41:13.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:13.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.706597+0000) 2019-10-13T09:41:13.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:13.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:13.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:13.709+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.710780+0000) 2019-10-13T09:41:15.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:15.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:15.056451+0000) 2019-10-13T09:41:16.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:16.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:16.276848+0000) 2019-10-13T09:41:16.446 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:41:16.446 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:16.449 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:41:16.450 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:18.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:18.532+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.537724+0000) 2019-10-13T09:41:18.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:18.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.544546+0000) 2019-10-13T09:41:18.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:18.704+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.710699+0000) 2019-10-13T09:41:18.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:18.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.711197+0000) 2019-10-13T09:41:18.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:18.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.711572+0000) 2019-10-13T09:41:18.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:18.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.712045+0000) 2019-10-13T09:41:18.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:18.709+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.713107+0000) 2019-10-13T09:41:18.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:18.708+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.713437+0000) 2019-10-13T09:41:18.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:18.708+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.714173+0000) 2019-10-13T09:41:18.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:18.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:18.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:18.721+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.722960+0000) 2019-10-13T09:41:19.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:19.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.529008+0000) 2019-10-13T09:41:21.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:21.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:21.835491+0000) 2019-10-13T09:41:22.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:22.024+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:22.027971+0000) 2019-10-13T09:41:23.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:23.209+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.211058+0000) 2019-10-13T09:41:23.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:23.712+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.718375+0000) 2019-10-13T09:41:23.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:23.712+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.718722+0000) 2019-10-13T09:41:23.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:23.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.719135+0000) 2019-10-13T09:41:23.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:23.717+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.720476+0000) 2019-10-13T09:41:23.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:23.716+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.720725+0000) 2019-10-13T09:41:23.722 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:23.716+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.721350+0000) 2019-10-13T09:41:23.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:23.716+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.721030+0000) 2019-10-13T09:41:23.723 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:23.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:23.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:23.725+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.728298+0000) 2019-10-13T09:41:25.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:25.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:25.056656+0000) 2019-10-13T09:41:26.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:26.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:26.277039+0000) 2019-10-13T09:41:28.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:28.536+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.537938+0000) 2019-10-13T09:41:28.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:28.540+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.544712+0000) 2019-10-13T09:41:28.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:28.721+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.725947+0000) 2019-10-13T09:41:28.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:28.720+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.725404+0000) 2019-10-13T09:41:28.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:28.720+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.725977+0000) 2019-10-13T09:41:28.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:28.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.727258+0000) 2019-10-13T09:41:28.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:28.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.728068+0000) 2019-10-13T09:41:28.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:28.724+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.728555+0000) 2019-10-13T09:41:28.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:28.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.728944+0000) 2019-10-13T09:41:28.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:28.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:28.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:28.733+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.735281+0000) 2019-10-13T09:41:29.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:29.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.529205+0000) 2019-10-13T09:41:31.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:31.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:31.835698+0000) 2019-10-13T09:41:32.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:32.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:32.028172+0000) 2019-10-13T09:41:33.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:33.213+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.217502+0000) 2019-10-13T09:41:33.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:33.727+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.732841+0000) 2019-10-13T09:41:33.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:33.727+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.733377+0000) 2019-10-13T09:41:33.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:33.727+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.733947+0000) 2019-10-13T09:41:33.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:33.727+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.734474+0000) 2019-10-13T09:41:33.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:33.733+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.735050+0000) 2019-10-13T09:41:33.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:33.727+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.735078+0000) 2019-10-13T09:41:33.738 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:33.731+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.735511+0000) 2019-10-13T09:41:33.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:33.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:33.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:33.737+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.741892+0000) 2019-10-13T09:41:35.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:35.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:35.056848+0000) 2019-10-13T09:41:36.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:36.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:36.277222+0000) 2019-10-13T09:41:38.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:38.535+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.538083+0000) 2019-10-13T09:41:38.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:38.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.544942+0000) 2019-10-13T09:41:38.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:38.735+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.739858+0000) 2019-10-13T09:41:38.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:38.735+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.740522+0000) 2019-10-13T09:41:38.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:38.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.741027+0000) 2019-10-13T09:41:38.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:38.735+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.741773+0000) 2019-10-13T09:41:38.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:38.735+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.743060+0000) 2019-10-13T09:41:38.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:38.739+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.743659+0000) 2019-10-13T09:41:38.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:38.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:38.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:38.741+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.744253+0000) 2019-10-13T09:41:38.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:38.749+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.751518+0000) 2019-10-13T09:41:39.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:39.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.529384+0000) 2019-10-13T09:41:41.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:41.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:41.835930+0000) 2019-10-13T09:41:42.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:42.027+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:42.028377+0000) 2019-10-13T09:41:43.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:43.221+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.223894+0000) 2019-10-13T09:41:43.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:43.743+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.747758+0000) 2019-10-13T09:41:43.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:43.745+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.748195+0000) 2019-10-13T09:41:43.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:43.743+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.748726+0000) 2019-10-13T09:41:43.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:43.743+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.749076+0000) 2019-10-13T09:41:43.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:43.743+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.749376+0000) 2019-10-13T09:41:43.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:43.743+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.750143+0000) 2019-10-13T09:41:43.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:43.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:43.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:43.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.751197+0000) 2019-10-13T09:41:43.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:43.753+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.756999+0000) 2019-10-13T09:41:45.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:45.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:45.057069+0000) 2019-10-13T09:41:46.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:46.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:46.277417+0000) 2019-10-13T09:41:46.602 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:41:46.602 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:46.606 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:41:46.606 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:48.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:48.535+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.538278+0000) 2019-10-13T09:41:48.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:48.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.545165+0000) 2019-10-13T09:41:48.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:48.747+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.754101+0000) 2019-10-13T09:41:48.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:48.752+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.754397+0000) 2019-10-13T09:41:48.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:48.747+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.755010+0000) 2019-10-13T09:41:48.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:48.751+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.755289+0000) 2019-10-13T09:41:48.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:48.751+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.755708+0000) 2019-10-13T09:41:48.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:48.751+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.756161+0000) 2019-10-13T09:41:48.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:48.751+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.756646+0000) 2019-10-13T09:41:48.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:48.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:48.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:48.756+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.761422+0000) 2019-10-13T09:41:49.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:49.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.529592+0000) 2019-10-13T09:41:51.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:51.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:51.836155+0000) 2019-10-13T09:41:52.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:52.027+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:52.028581+0000) 2019-10-13T09:41:53.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:53.228+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.230627+0000) 2019-10-13T09:41:53.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:53.755+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.760469+0000) 2019-10-13T09:41:53.762 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:53.756+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.760959+0000) 2019-10-13T09:41:53.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:53.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.761598+0000) 2019-10-13T09:41:53.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:53.755+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.762062+0000) 2019-10-13T09:41:53.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:53.755+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.762530+0000) 2019-10-13T09:41:53.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:53.759+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.763369+0000) 2019-10-13T09:41:53.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:53.759+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.763744+0000) 2019-10-13T09:41:53.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:53.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:53.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:53.768+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.770241+0000) 2019-10-13T09:41:55.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:55.051+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:55.057319+0000) 2019-10-13T09:41:56.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:56.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:56.277605+0000) 2019-10-13T09:41:58.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:58.535+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.538476+0000) 2019-10-13T09:41:58.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:58.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.545389+0000) 2019-10-13T09:41:58.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:41:58.763+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.767950+0000) 2019-10-13T09:41:58.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:41:58.763+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.768586+0000) 2019-10-13T09:41:58.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:58.764+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.769072+0000) 2019-10-13T09:41:58.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:41:58.763+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.769644+0000) 2019-10-13T09:41:58.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:41:58.763+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.770966+0000) 2019-10-13T09:41:58.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:41:58.763+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.771213+0000) 2019-10-13T09:41:58.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:41:58.767+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.771477+0000) 2019-10-13T09:41:58.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:41:58.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:41:58.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:41:58.772+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.776684+0000) 2019-10-13T09:41:59.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:41:59.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.529774+0000) 2019-10-13T09:42:01.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:01.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:01.836357+0000) 2019-10-13T09:42:02.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:02.027+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:02.028801+0000) 2019-10-13T09:42:03.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:03.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.237334+0000) 2019-10-13T09:42:03.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:03.772+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.775639+0000) 2019-10-13T09:42:03.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:03.775+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.776067+0000) 2019-10-13T09:42:03.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:03.775+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.776392+0000) 2019-10-13T09:42:03.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:03.775+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.776690+0000) 2019-10-13T09:42:03.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:03.775+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.777771+0000) 2019-10-13T09:42:03.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:03.775+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.778140+0000) 2019-10-13T09:42:03.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:03.775+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.778431+0000) 2019-10-13T09:42:03.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:03.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:03.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:03.780+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.786028+0000) 2019-10-13T09:42:05.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:05.055+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:05.057523+0000) 2019-10-13T09:42:06.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:06.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:06.277803+0000) 2019-10-13T09:42:08.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:08.535+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.538671+0000) 2019-10-13T09:42:08.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:08.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.545589+0000) 2019-10-13T09:42:08.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:08.779+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.782942+0000) 2019-10-13T09:42:08.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:08.783+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.783709+0000) 2019-10-13T09:42:08.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:08.783+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.784098+0000) 2019-10-13T09:42:08.787 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:08.780+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.785109+0000) 2019-10-13T09:42:08.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:08.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.785771+0000) 2019-10-13T09:42:08.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:08.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:08.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:08.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.786288+0000) 2019-10-13T09:42:08.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:08.783+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.786798+0000) 2019-10-13T09:42:08.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:08.788+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.790913+0000) 2019-10-13T09:42:09.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:09.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.529964+0000) 2019-10-13T09:42:11.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:11.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:11.836563+0000) 2019-10-13T09:42:12.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:12.027+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:12.029009+0000) 2019-10-13T09:42:13.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:13.240+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.243859+0000) 2019-10-13T09:42:13.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:13.787+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.789106+0000) 2019-10-13T09:42:13.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:13.784+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.789824+0000) 2019-10-13T09:42:13.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:13.787+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.790222+0000) 2019-10-13T09:42:13.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:13.787+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.790389+0000) 2019-10-13T09:42:13.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:13.787+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.790760+0000) 2019-10-13T09:42:13.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:13.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.791424+0000) 2019-10-13T09:42:13.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:13.791+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.792637+0000) 2019-10-13T09:42:13.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:13.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:13.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:13.796+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.798610+0000) 2019-10-13T09:42:15.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:15.055+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:15.057722+0000) 2019-10-13T09:42:16.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:16.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:16.277998+0000) 2019-10-13T09:42:16.713 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:42:16.713 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:16.717 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:42:16.717 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:18.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:18.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.538907+0000) 2019-10-13T09:42:18.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:18.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.545784+0000) 2019-10-13T09:42:18.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:18.794+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.797512+0000) 2019-10-13T09:42:18.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:18.794+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.797816+0000) 2019-10-13T09:42:18.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:18.794+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.797807+0000) 2019-10-13T09:42:18.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:18.796+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.798579+0000) 2019-10-13T09:42:18.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:18.798+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.799347+0000) 2019-10-13T09:42:18.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:18.798+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.799981+0000) 2019-10-13T09:42:18.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:18.798+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.800504+0000) 2019-10-13T09:42:18.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:18.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:18.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:18.800+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.805481+0000) 2019-10-13T09:42:19.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:19.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.530175+0000) 2019-10-13T09:42:21.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:21.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:21.836807+0000) 2019-10-13T09:42:22.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:22.026+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:22.029218+0000) 2019-10-13T09:42:23.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:23.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.250249+0000) 2019-10-13T09:42:23.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:23.802+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.803353+0000) 2019-10-13T09:42:23.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:23.802+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.804543+0000) 2019-10-13T09:42:23.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:23.800+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.804906+0000) 2019-10-13T09:42:23.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:23.802+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.805531+0000) 2019-10-13T09:42:23.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:23.802+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.805977+0000) 2019-10-13T09:42:23.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:23.802+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.806578+0000) 2019-10-13T09:42:23.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:23.806+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.807725+0000) 2019-10-13T09:42:23.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:23.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 81976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:23.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:23.808+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.813907+0000) 2019-10-13T09:42:25.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:25.054+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:25.057925+0000) 2019-10-13T09:42:26.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:26.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:26.278251+0000) 2019-10-13T09:42:28.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:28.534+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.539125+0000) 2019-10-13T09:42:28.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:28.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.545980+0000) 2019-10-13T09:42:28.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:28.810+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.811920+0000) 2019-10-13T09:42:28.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:28.810+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.812314+0000) 2019-10-13T09:42:28.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:28.808+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.813063+0000) 2019-10-13T09:42:28.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:28.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.813326+0000) 2019-10-13T09:42:28.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:28.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.814031+0000) 2019-10-13T09:42:28.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:28.810+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.814406+0000) 2019-10-13T09:42:28.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:28.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:28.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:28.814+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.815634+0000) 2019-10-13T09:42:28.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:28.816+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.820141+0000) 2019-10-13T09:42:29.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:29.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.530378+0000) 2019-10-13T09:42:31.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:31.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:31.837009+0000) 2019-10-13T09:42:32.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:32.026+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:32.029457+0000) 2019-10-13T09:42:33.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:33.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.256501+0000) 2019-10-13T09:42:33.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:33.814+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.818238+0000) 2019-10-13T09:42:33.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:33.814+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.818707+0000) 2019-10-13T09:42:33.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:33.818+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.819768+0000) 2019-10-13T09:42:33.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:33.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.820363+0000) 2019-10-13T09:42:33.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:33.818+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.820748+0000) 2019-10-13T09:42:33.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:33.816+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.821223+0000) 2019-10-13T09:42:33.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:33.818+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.821580+0000) 2019-10-13T09:42:33.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:33.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:33.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:33.820+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.825597+0000) 2019-10-13T09:42:35.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:35.054+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:35.058143+0000) 2019-10-13T09:42:36.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:36.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:36.278450+0000) 2019-10-13T09:42:38.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:38.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.539367+0000) 2019-10-13T09:42:38.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:38.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.546176+0000) 2019-10-13T09:42:38.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:38.822+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.825984+0000) 2019-10-13T09:42:38.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:38.822+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.826174+0000) 2019-10-13T09:42:38.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:38.824+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.827130+0000) 2019-10-13T09:42:38.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:38.822+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.826775+0000) 2019-10-13T09:42:38.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:38.826+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.827845+0000) 2019-10-13T09:42:38.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:38.826+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.828647+0000) 2019-10-13T09:42:38.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:38.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.829441+0000) 2019-10-13T09:42:38.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:38.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:38.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:38.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.834423+0000) 2019-10-13T09:42:39.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:39.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.530573+0000) 2019-10-13T09:42:41.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:41.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:41.837212+0000) 2019-10-13T09:42:42.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:42.026+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:42.029675+0000) 2019-10-13T09:42:43.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:43.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.262915+0000) 2019-10-13T09:42:43.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:43.830+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.833051+0000) 2019-10-13T09:42:43.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:43.830+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.833811+0000) 2019-10-13T09:42:43.835 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:43.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.834272+0000) 2019-10-13T09:42:43.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:43.830+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.834806+0000) 2019-10-13T09:42:43.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:43.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.835380+0000) 2019-10-13T09:42:43.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:43.834+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.835774+0000) 2019-10-13T09:42:43.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:43.834+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.836090+0000) 2019-10-13T09:42:43.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:43.834+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82112 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:43.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:43.835+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.840756+0000) 2019-10-13T09:42:45.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:45.054+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:45.058345+0000) 2019-10-13T09:42:46.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:46.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:46.278673+0000) 2019-10-13T09:42:46.895 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:42:46.896 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:46.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:42:46.899 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:48.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:48.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.539568+0000) 2019-10-13T09:42:48.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:48.546+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.546376+0000) 2019-10-13T09:42:48.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:48.835+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.840799+0000) 2019-10-13T09:42:48.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:48.838+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.841265+0000) 2019-10-13T09:42:48.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:48.838+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.842338+0000) 2019-10-13T09:42:48.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:48.838+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.842574+0000) 2019-10-13T09:42:48.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:48.842+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.843440+0000) 2019-10-13T09:42:48.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:48.838+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.843110+0000) 2019-10-13T09:42:48.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:48.842+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.843914+0000) 2019-10-13T09:42:48.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:48.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:48.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:48.843+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.848361+0000) 2019-10-13T09:42:49.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:49.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.530743+0000) 2019-10-13T09:42:51.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:51.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:51.837448+0000) 2019-10-13T09:42:52.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:52.026+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:52.029872+0000) 2019-10-13T09:42:53.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:53.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.269434+0000) 2019-10-13T09:42:53.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:53.846+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.847507+0000) 2019-10-13T09:42:53.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:53.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.848113+0000) 2019-10-13T09:42:53.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:53.843+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.848438+0000) 2019-10-13T09:42:53.850 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:53.846+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.849379+0000) 2019-10-13T09:42:53.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:53.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.849894+0000) 2019-10-13T09:42:53.852 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:53.846+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.850736+0000) 2019-10-13T09:42:53.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:53.846+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.849880+0000) 2019-10-13T09:42:53.852 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:53.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:53.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:53.851+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.855399+0000) 2019-10-13T09:42:55.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:55.054+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:55.058561+0000) 2019-10-13T09:42:56.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:56.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:56.278867+0000) 2019-10-13T09:42:58.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:58.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.539792+0000) 2019-10-13T09:42:58.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:58.542+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.546572+0000) 2019-10-13T09:42:58.856 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:42:58.850+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.854737+0000) 2019-10-13T09:42:58.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:58.851+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.855054+0000) 2019-10-13T09:42:58.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:42:58.854+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.855880+0000) 2019-10-13T09:42:58.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:42:58.854+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.856215+0000) 2019-10-13T09:42:58.858 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:42:58.854+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.856727+0000) 2019-10-13T09:42:58.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:42:58.854+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.857288+0000) 2019-10-13T09:42:58.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:42:58.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.858146+0000) 2019-10-13T09:42:58.859 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:42:58.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:42:58.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:42:58.863+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.866246+0000) 2019-10-13T09:42:59.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:42:59.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.530943+0000) 2019-10-13T09:43:01.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:01.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:01.837670+0000) 2019-10-13T09:43:02.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:02.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:02.030075+0000) 2019-10-13T09:43:03.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:03.275+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.275778+0000) 2019-10-13T09:43:03.864 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:03.857+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.862389+0000) 2019-10-13T09:43:03.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:03.857+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.862833+0000) 2019-10-13T09:43:03.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:03.861+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.863868+0000) 2019-10-13T09:43:03.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:03.859+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.864340+0000) 2019-10-13T09:43:03.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:03.861+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.864558+0000) 2019-10-13T09:43:03.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:03.861+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.865405+0000) 2019-10-13T09:43:03.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:03.861+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.866016+0000) 2019-10-13T09:43:03.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:03.861+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:03.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:03.863+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.869535+0000) 2019-10-13T09:43:05.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:05.053+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:05.058774+0000) 2019-10-13T09:43:06.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:06.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:06.279079+0000) 2019-10-13T09:43:08.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:08.537+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.539985+0000) 2019-10-13T09:43:08.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:08.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.546764+0000) 2019-10-13T09:43:08.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:08.865+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.870225+0000) 2019-10-13T09:43:08.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:08.865+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.870349+0000) 2019-10-13T09:43:08.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:08.869+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.871567+0000) 2019-10-13T09:43:08.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:08.867+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.872004+0000) 2019-10-13T09:43:08.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:08.869+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.873028+0000) 2019-10-13T09:43:08.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:08.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:08.875 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:08.869+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.873388+0000) 2019-10-13T09:43:08.875 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:08.869+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.873678+0000) 2019-10-13T09:43:08.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:08.871+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.877116+0000) 2019-10-13T09:43:09.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:09.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.531116+0000) 2019-10-13T09:43:11.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:11.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:11.837867+0000) 2019-10-13T09:43:12.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:12.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:12.030287+0000) 2019-10-13T09:43:13.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:13.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.282195+0000) 2019-10-13T09:43:13.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:13.871+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.876390+0000) 2019-10-13T09:43:13.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:13.873+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.876842+0000) 2019-10-13T09:43:13.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:13.873+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.877199+0000) 2019-10-13T09:43:13.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:13.873+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.877739+0000) 2019-10-13T09:43:13.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:13.873+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.878117+0000) 2019-10-13T09:43:13.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:13.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.879074+0000) 2019-10-13T09:43:13.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:13.877+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.879296+0000) 2019-10-13T09:43:13.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:13.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:13.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:13.883+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.886886+0000) 2019-10-13T09:43:15.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:15.053+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:15.059005+0000) 2019-10-13T09:43:16.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:16.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:16.279269+0000) 2019-10-13T09:43:17.023 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:43:17.023 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:17.027 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:43:17.027 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:18.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:18.537+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.540199+0000) 2019-10-13T09:43:18.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:18.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.546965+0000) 2019-10-13T09:43:18.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:18.881+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.883694+0000) 2019-10-13T09:43:18.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:18.881+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.884428+0000) 2019-10-13T09:43:18.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:18.879+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.885425+0000) 2019-10-13T09:43:18.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:18.881+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.885787+0000) 2019-10-13T09:43:18.887 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:18.881+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.886127+0000) 2019-10-13T09:43:18.888 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:18.881+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.886915+0000) 2019-10-13T09:43:18.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:18.885+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:18.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:18.885+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.887686+0000) 2019-10-13T09:43:18.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:18.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.893780+0000) 2019-10-13T09:43:19.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:19.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.531304+0000) 2019-10-13T09:43:21.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:21.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:21.838056+0000) 2019-10-13T09:43:22.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:22.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:22.030492+0000) 2019-10-13T09:43:23.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:23.287+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.288582+0000) 2019-10-13T09:43:23.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:23.885+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.891099+0000) 2019-10-13T09:43:23.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:23.889+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.891617+0000) 2019-10-13T09:43:23.893 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:23.889+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.891993+0000) 2019-10-13T09:43:23.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:23.889+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.892749+0000) 2019-10-13T09:43:23.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:23.889+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.893731+0000) 2019-10-13T09:43:23.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:23.891+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.894343+0000) 2019-10-13T09:43:23.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:23.889+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.894762+0000) 2019-10-13T09:43:23.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:23.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:23.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:23.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.901723+0000) 2019-10-13T09:43:25.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:25.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:25.059199+0000) 2019-10-13T09:43:26.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:26.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:26.279531+0000) 2019-10-13T09:43:28.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:28.537+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.540407+0000) 2019-10-13T09:43:28.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:28.541+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.547162+0000) 2019-10-13T09:43:28.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:28.893+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.898574+0000) 2019-10-13T09:43:28.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:28.897+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.899376+0000) 2019-10-13T09:43:28.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:28.895+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.900129+0000) 2019-10-13T09:43:28.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:28.897+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.899881+0000) 2019-10-13T09:43:28.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:28.897+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.901190+0000) 2019-10-13T09:43:28.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:28.897+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.901566+0000) 2019-10-13T09:43:28.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:28.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.902294+0000) 2019-10-13T09:43:28.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:28.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:28.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:28.903+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.906958+0000) 2019-10-13T09:43:29.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:29.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.531474+0000) 2019-10-13T09:43:31.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:31.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:31.838257+0000) 2019-10-13T09:43:32.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:32.025+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:32.030709+0000) 2019-10-13T09:43:33.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:33.291+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.295835+0000) 2019-10-13T09:43:33.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:33.901+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.905853+0000) 2019-10-13T09:43:33.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:33.901+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.906399+0000) 2019-10-13T09:43:33.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:33.903+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.907097+0000) 2019-10-13T09:43:33.909 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:33.905+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.907710+0000) 2019-10-13T09:43:33.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:33.905+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.908040+0000) 2019-10-13T09:43:33.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:33.905+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.908865+0000) 2019-10-13T09:43:33.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:33.905+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.908993+0000) 2019-10-13T09:43:33.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:33.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:33.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:33.911+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.915470+0000) 2019-10-13T09:43:35.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:35.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:35.059399+0000) 2019-10-13T09:43:36.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:36.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:36.279750+0000) 2019-10-13T09:43:38.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:38.537+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.540604+0000) 2019-10-13T09:43:38.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:38.545+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.547372+0000) 2019-10-13T09:43:38.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:38.909+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.913114+0000) 2019-10-13T09:43:38.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:38.909+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.913822+0000) 2019-10-13T09:43:38.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:38.909+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.914203+0000) 2019-10-13T09:43:38.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:38.910+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.914589+0000) 2019-10-13T09:43:38.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:38.909+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.915056+0000) 2019-10-13T09:43:38.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:38.913+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.915531+0000) 2019-10-13T09:43:38.917 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:38.913+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.915657+0000) 2019-10-13T09:43:38.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:38.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:38.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:38.918+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.924635+0000) 2019-10-13T09:43:39.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:39.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.531645+0000) 2019-10-13T09:43:41.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:41.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:41.838470+0000) 2019-10-13T09:43:42.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:42.025+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:42.030917+0000) 2019-10-13T09:43:43.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:43.302+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.303014+0000) 2019-10-13T09:43:43.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:43.916+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.920470+0000) 2019-10-13T09:43:43.922 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:43.916+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.920903+0000) 2019-10-13T09:43:43.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:43.916+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.921684+0000) 2019-10-13T09:43:43.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:43.916+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.922222+0000) 2019-10-13T09:43:43.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:43.916+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.922484+0000) 2019-10-13T09:43:43.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:43.916+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.923132+0000) 2019-10-13T09:43:43.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:43.918+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.923804+0000) 2019-10-13T09:43:43.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:43.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:43.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:43.926+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.931260+0000) 2019-10-13T09:43:45.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:45.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:45.059627+0000) 2019-10-13T09:43:46.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:46.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:46.279938+0000) 2019-10-13T09:43:47.162 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:43:47.162 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:47.166 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:43:47.166 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:48.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:48.536+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.540808+0000) 2019-10-13T09:43:48.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:48.544+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.547556+0000) 2019-10-13T09:43:48.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:48.924+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.927541+0000) 2019-10-13T09:43:48.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:48.924+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.928071+0000) 2019-10-13T09:43:48.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:48.924+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.928306+0000) 2019-10-13T09:43:48.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:48.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.929128+0000) 2019-10-13T09:43:48.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:48.924+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.928771+0000) 2019-10-13T09:43:48.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:48.924+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.929770+0000) 2019-10-13T09:43:48.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:48.926+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.930979+0000) 2019-10-13T09:43:48.932 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:48.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:48.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:48.934+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.939834+0000) 2019-10-13T09:43:49.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:49.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.531850+0000) 2019-10-13T09:43:51.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:51.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:51.838668+0000) 2019-10-13T09:43:52.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:52.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:52.031130+0000) 2019-10-13T09:43:53.311 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:53.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.310178+0000) 2019-10-13T09:43:53.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:53.928+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.934985+0000) 2019-10-13T09:43:53.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:53.932+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.935428+0000) 2019-10-13T09:43:53.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:53.932+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.936572+0000) 2019-10-13T09:43:53.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:53.932+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.936804+0000) 2019-10-13T09:43:53.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:53.932+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.937325+0000) 2019-10-13T09:43:53.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:53.930+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.937649+0000) 2019-10-13T09:43:53.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:53.932+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.938046+0000) 2019-10-13T09:43:53.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:53.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:53.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:53.934+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.940906+0000) 2019-10-13T09:43:55.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:55.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:55.059891+0000) 2019-10-13T09:43:56.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:56.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:56.280135+0000) 2019-10-13T09:43:58.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:58.536+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.541002+0000) 2019-10-13T09:43:58.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:58.544+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.547778+0000) 2019-10-13T09:43:58.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:58.938+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.942221+0000) 2019-10-13T09:43:58.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:43:58.936+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.942659+0000) 2019-10-13T09:43:58.944 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:43:58.940+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.943474+0000) 2019-10-13T09:43:58.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:43:58.940+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.943833+0000) 2019-10-13T09:43:58.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:43:58.940+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.944388+0000) 2019-10-13T09:43:58.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:43:58.940+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.944556+0000) 2019-10-13T09:43:58.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:43:58.940+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.944841+0000) 2019-10-13T09:43:58.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:43:58.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:43:58.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:43:58.950+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.955697+0000) 2019-10-13T09:43:59.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:43:59.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.532023+0000) 2019-10-13T09:44:01.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:01.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:01.838870+0000) 2019-10-13T09:44:02.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:02.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:02.031344+0000) 2019-10-13T09:44:03.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:03.310+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.317047+0000) 2019-10-13T09:44:03.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:03.944+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.950213+0000) 2019-10-13T09:44:03.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:03.944+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.951011+0000) 2019-10-13T09:44:03.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:03.946+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.951528+0000) 2019-10-13T09:44:03.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:03.948+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.952104+0000) 2019-10-13T09:44:03.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:03.948+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.952571+0000) 2019-10-13T09:44:03.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:03.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:03.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:03.948+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.953625+0000) 2019-10-13T09:44:03.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:03.948+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.954454+0000) 2019-10-13T09:44:03.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:03.954+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.959549+0000) 2019-10-13T09:44:05.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:05.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:05.060087+0000) 2019-10-13T09:44:06.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:06.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:06.280321+0000) 2019-10-13T09:44:08.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:08.536+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.541195+0000) 2019-10-13T09:44:08.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:08.544+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.547978+0000) 2019-10-13T09:44:08.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:08.952+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.956922+0000) 2019-10-13T09:44:08.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:08.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.957950+0000) 2019-10-13T09:44:08.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:08.952+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.958054+0000) 2019-10-13T09:44:08.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:08.952+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.959195+0000) 2019-10-13T09:44:08.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:08.954+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.959728+0000) 2019-10-13T09:44:08.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:08.956+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.960516+0000) 2019-10-13T09:44:08.962 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:08.956+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.960916+0000) 2019-10-13T09:44:08.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:08.956+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:08.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:08.958+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.963931+0000) 2019-10-13T09:44:09.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:09.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.532196+0000) 2019-10-13T09:44:11.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:11.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:11.839127+0000) 2019-10-13T09:44:12.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:12.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:12.031542+0000) 2019-10-13T09:44:13.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:13.322+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.324000+0000) 2019-10-13T09:44:13.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:13.960+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.965138+0000) 2019-10-13T09:44:13.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:13.960+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.965827+0000) 2019-10-13T09:44:13.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:13.960+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.965539+0000) 2019-10-13T09:44:13.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:13.960+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.966786+0000) 2019-10-13T09:44:13.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:13.964+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.967426+0000) 2019-10-13T09:44:13.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:13.962+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.967784+0000) 2019-10-13T09:44:13.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:13.964+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:13.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:13.964+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.968572+0000) 2019-10-13T09:44:13.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:13.970+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.974358+0000) 2019-10-13T09:44:15.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:15.056+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:15.060291+0000) 2019-10-13T09:44:16.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:16.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:16.280522+0000) 2019-10-13T09:44:17.221 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:44:17.221 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:17.224 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:44:17.225 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:18.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:18.540+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.541403+0000) 2019-10-13T09:44:18.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:18.544+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.548147+0000) 2019-10-13T09:44:18.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:18.964+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.971214+0000) 2019-10-13T09:44:18.973 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:18.968+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.971712+0000) 2019-10-13T09:44:18.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:18.968+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.972778+0000) 2019-10-13T09:44:18.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:18.968+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.972388+0000) 2019-10-13T09:44:18.975 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:18.968+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.973658+0000) 2019-10-13T09:44:18.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:18.968+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.974207+0000) 2019-10-13T09:44:18.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:18.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:18.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:18.970+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.975210+0000) 2019-10-13T09:44:18.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:18.974+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.980581+0000) 2019-10-13T09:44:19.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:19.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.532367+0000) 2019-10-13T09:44:21.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:21.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:21.839377+0000) 2019-10-13T09:44:22.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:22.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:22.031799+0000) 2019-10-13T09:44:23.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:23.330+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.330966+0000) 2019-10-13T09:44:23.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:23.972+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.977790+0000) 2019-10-13T09:44:23.979 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:23.972+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.978158+0000) 2019-10-13T09:44:23.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:23.972+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.978657+0000) 2019-10-13T09:44:23.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:23.978+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.979830+0000) 2019-10-13T09:44:23.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:23.976+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.980156+0000) 2019-10-13T09:44:23.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:23.976+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.980844+0000) 2019-10-13T09:44:23.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:23.976+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.981812+0000) 2019-10-13T09:44:23.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:23.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:23.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:23.986+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.987563+0000) 2019-10-13T09:44:25.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:25.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:25.060514+0000) 2019-10-13T09:44:26.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:26.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:26.280715+0000) 2019-10-13T09:44:28.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:28.535+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.541599+0000) 2019-10-13T09:44:28.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:28.547+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.548357+0000) 2019-10-13T09:44:29.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:28.979+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.985681+0000) 2019-10-13T09:44:29.017 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:28.979+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.986580+0000) 2019-10-13T09:44:29.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:28.979+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.986697+0000) 2019-10-13T09:44:29.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:28.983+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.987356+0000) 2019-10-13T09:44:29.017 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:28.983+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.987757+0000) 2019-10-13T09:44:29.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:28.982+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.985063+0000) 2019-10-13T09:44:29.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:28.994+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.994493+0000) 2019-10-13T09:44:29.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:28.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:29.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:28.983+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.988999+0000) 2019-10-13T09:44:29.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:29.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.532539+0000) 2019-10-13T09:44:31.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:31.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:31.839634+0000) 2019-10-13T09:44:32.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:32.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:32.031997+0000) 2019-10-13T09:44:33.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:33.333+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.337857+0000) 2019-10-13T09:44:33.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:33.987+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.992053+0000) 2019-10-13T09:44:33.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:33.989+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.992365+0000) 2019-10-13T09:44:33.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:33.987+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.993159+0000) 2019-10-13T09:44:33.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:33.987+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.994037+0000) 2019-10-13T09:44:33.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:33.987+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.994368+0000) 2019-10-13T09:44:33.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:33.987+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.994779+0000) 2019-10-13T09:44:33.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:33.991+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.995435+0000) 2019-10-13T09:44:33.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:33.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:34.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:33.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.999925+0000) 2019-10-13T09:44:35.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:35.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:35.060737+0000) 2019-10-13T09:44:36.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:36.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:36.280923+0000) 2019-10-13T09:44:38.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:38.539+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.541749+0000) 2019-10-13T09:44:38.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:38.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.548558+0000) 2019-10-13T09:44:39.000 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:38.995+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.999396+0000) 2019-10-13T09:44:39.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:38.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.999880+0000) 2019-10-13T09:44:39.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:38.995+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.000289+0000) 2019-10-13T09:44:39.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:38.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.001994+0000) 2019-10-13T09:44:39.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:38.995+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.002521+0000) 2019-10-13T09:44:39.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:38.995+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.002967+0000) 2019-10-13T09:44:39.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:38.999+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.003392+0000) 2019-10-13T09:44:39.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:38.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:39.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:39.005+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.007642+0000) 2019-10-13T09:44:39.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:39.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.532709+0000) 2019-10-13T09:44:41.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:41.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:41.839835+0000) 2019-10-13T09:44:42.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:42.027+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:42.032158+0000) 2019-10-13T09:44:43.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:43.341+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.344881+0000) 2019-10-13T09:44:44.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:44.005+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.006462+0000) 2019-10-13T09:44:44.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:44.003+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.007542+0000) 2019-10-13T09:44:44.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:44.003+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.008389+0000) 2019-10-13T09:44:44.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:44.003+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.009051+0000) 2019-10-13T09:44:44.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:44.003+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.009557+0000) 2019-10-13T09:44:44.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:44.003+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.009939+0000) 2019-10-13T09:44:44.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:44.003+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.010711+0000) 2019-10-13T09:44:44.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:44.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:44.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:44.009+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.013645+0000) 2019-10-13T09:44:45.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:45.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:45.060955+0000) 2019-10-13T09:44:46.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:46.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:46.281147+0000) 2019-10-13T09:44:47.354 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:44:47.354 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:47.357 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:44:47.357 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:48.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:48.539+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.541956+0000) 2019-10-13T09:44:48.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:48.543+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.548770+0000) 2019-10-13T09:44:49.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:49.007+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.013524+0000) 2019-10-13T09:44:49.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:49.007+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.014562+0000) 2019-10-13T09:44:49.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:49.011+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.015285+0000) 2019-10-13T09:44:49.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:49.013+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.016006+0000) 2019-10-13T09:44:49.017 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:49.011+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.016187+0000) 2019-10-13T09:44:49.018 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:49.011+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.016606+0000) 2019-10-13T09:44:49.018 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:49.011+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.016949+0000) 2019-10-13T09:44:49.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:49.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:49.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:49.017+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.020599+0000) 2019-10-13T09:44:49.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:49.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.532874+0000) 2019-10-13T09:44:51.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:51.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:51.840055+0000) 2019-10-13T09:44:52.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:52.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:52.032350+0000) 2019-10-13T09:44:53.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:53.349+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.351816+0000) 2019-10-13T09:44:54.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:54.015+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.020516+0000) 2019-10-13T09:44:54.023 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:54.015+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.021366+0000) 2019-10-13T09:44:54.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:54.015+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.021579+0000) 2019-10-13T09:44:54.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:54.015+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.022614+0000) 2019-10-13T09:44:54.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:54.015+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.022828+0000) 2019-10-13T09:44:54.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:54.021+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.024000+0000) 2019-10-13T09:44:54.026 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:54.019+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.023675+0000) 2019-10-13T09:44:54.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:54.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 82996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:54.030 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:54.025+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.028744+0000) 2019-10-13T09:44:55.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:55.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:55.061201+0000) 2019-10-13T09:44:56.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:56.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:56.281354+0000) 2019-10-13T09:44:58.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:58.539+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.542125+0000) 2019-10-13T09:44:58.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:58.547+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.548983+0000) 2019-10-13T09:44:59.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:44:59.027+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.027763+0000) 2019-10-13T09:44:59.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:44:59.027+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.028431+0000) 2019-10-13T09:44:59.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:44:59.027+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.028150+0000) 2019-10-13T09:44:59.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:44:59.027+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.029274+0000) 2019-10-13T09:44:59.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:59.025+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.029741+0000) 2019-10-13T09:44:59.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:44:59.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.030065+0000) 2019-10-13T09:44:59.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:44:59.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.030669+0000) 2019-10-13T09:44:59.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:44:59.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:44:59.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:44:59.033+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.035668+0000) 2019-10-13T09:44:59.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:44:59.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.533048+0000) 2019-10-13T09:45:01.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:01.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:01.840254+0000) 2019-10-13T09:45:02.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:02.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:02.032561+0000) 2019-10-13T09:45:03.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:03.357+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.358853+0000) 2019-10-13T09:45:04.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:04.031+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.034280+0000) 2019-10-13T09:45:04.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:04.031+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.034626+0000) 2019-10-13T09:45:04.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:04.035+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.035496+0000) 2019-10-13T09:45:04.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:04.033+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.036415+0000) 2019-10-13T09:45:04.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:04.035+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.036659+0000) 2019-10-13T09:45:04.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:04.035+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.037130+0000) 2019-10-13T09:45:04.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:04.035+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.037606+0000) 2019-10-13T09:45:04.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:04.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:04.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:04.041+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.045389+0000) 2019-10-13T09:45:05.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:05.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:05.061428+0000) 2019-10-13T09:45:06.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:06.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:06.281552+0000) 2019-10-13T09:45:08.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:08.539+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.542351+0000) 2019-10-13T09:45:08.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:08.547+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.549177+0000) 2019-10-13T09:45:09.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:09.041+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.043173+0000) 2019-10-13T09:45:09.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:09.043+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.043523+0000) 2019-10-13T09:45:09.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:09.043+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.044141+0000) 2019-10-13T09:45:09.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:09.043+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.044173+0000) 2019-10-13T09:45:09.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:09.043+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.046063+0000) 2019-10-13T09:45:09.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:09.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:09.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:09.043+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.046782+0000) 2019-10-13T09:45:09.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:09.043+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.047182+0000) 2019-10-13T09:45:09.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:09.049+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.051080+0000) 2019-10-13T09:45:09.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:09.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.533255+0000) 2019-10-13T09:45:11.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:11.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:11.840453+0000) 2019-10-13T09:45:12.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:12.030+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:12.032799+0000) 2019-10-13T09:45:13.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:13.361+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.365858+0000) 2019-10-13T09:45:14.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:14.046+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.049874+0000) 2019-10-13T09:45:14.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:14.046+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.050259+0000) 2019-10-13T09:45:14.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:14.046+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.050784+0000) 2019-10-13T09:45:14.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:14.049+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.051969+0000) 2019-10-13T09:45:14.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:14.050+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.051669+0000) 2019-10-13T09:45:14.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:14.050+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.052682+0000) 2019-10-13T09:45:14.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:14.050+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.053639+0000) 2019-10-13T09:45:14.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:14.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:14.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:14.057+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.058678+0000) 2019-10-13T09:45:15.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:15.058+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:15.061639+0000) 2019-10-13T09:45:16.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:16.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:16.281749+0000) 2019-10-13T09:45:17.426 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:45:17.426 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:17.430 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:45:17.430 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:18.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:18.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.542545+0000) 2019-10-13T09:45:18.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:18.546+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.549393+0000) 2019-10-13T09:45:19.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:19.053+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.057617+0000) 2019-10-13T09:45:19.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:19.054+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.057928+0000) 2019-10-13T09:45:19.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:19.054+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.058875+0000) 2019-10-13T09:45:19.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:19.058+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.059415+0000) 2019-10-13T09:45:19.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:19.058+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.059890+0000) 2019-10-13T09:45:19.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:19.058+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.060489+0000) 2019-10-13T09:45:19.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:19.058+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:19.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:19.058+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.061254+0000) 2019-10-13T09:45:19.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:19.061+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.065267+0000) 2019-10-13T09:45:19.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:19.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.533445+0000) 2019-10-13T09:45:21.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:21.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:21.840646+0000) 2019-10-13T09:45:22.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:22.030+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:22.033039+0000) 2019-10-13T09:45:23.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:23.369+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.372959+0000) 2019-10-13T09:45:24.065 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:24.062+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.063760+0000) 2019-10-13T09:45:24.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:24.062+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.064642+0000) 2019-10-13T09:45:24.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:24.062+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.065033+0000) 2019-10-13T09:45:24.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:24.062+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.065405+0000) 2019-10-13T09:45:24.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:24.062+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.065752+0000) 2019-10-13T09:45:24.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:24.065+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.066477+0000) 2019-10-13T09:45:24.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:24.062+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.066256+0000) 2019-10-13T09:45:24.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:24.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:24.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:24.073+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.074870+0000) 2019-10-13T09:45:25.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:25.058+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:25.061840+0000) 2019-10-13T09:45:26.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:26.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:26.281954+0000) 2019-10-13T09:45:28.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:28.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.542775+0000) 2019-10-13T09:45:28.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:28.546+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.549644+0000) 2019-10-13T09:45:29.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:29.066+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.071070+0000) 2019-10-13T09:45:29.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:29.070+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.071815+0000) 2019-10-13T09:45:29.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:29.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.072499+0000) 2019-10-13T09:45:29.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:29.070+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.072754+0000) 2019-10-13T09:45:29.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:29.070+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.073379+0000) 2019-10-13T09:45:29.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:29.068+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.073686+0000) 2019-10-13T09:45:29.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:29.070+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.074134+0000) 2019-10-13T09:45:29.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:29.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:29.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:29.076+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.080949+0000) 2019-10-13T09:45:29.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:29.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.533617+0000) 2019-10-13T09:45:31.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:31.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:31.840888+0000) 2019-10-13T09:45:32.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:32.030+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:32.033239+0000) 2019-10-13T09:45:33.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:33.376+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.380078+0000) 2019-10-13T09:45:34.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:34.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.079015+0000) 2019-10-13T09:45:34.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:34.078+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.079555+0000) 2019-10-13T09:45:34.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:34.078+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.079798+0000) 2019-10-13T09:45:34.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:34.078+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.080343+0000) 2019-10-13T09:45:34.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:34.078+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.080808+0000) 2019-10-13T09:45:34.082 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:34.078+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.081161+0000) 2019-10-13T09:45:34.085 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:34.078+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.081903+0000) 2019-10-13T09:45:34.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:34.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:34.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:34.084+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.087843+0000) 2019-10-13T09:45:35.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:35.058+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:35.062020+0000) 2019-10-13T09:45:36.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:36.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:36.282163+0000) 2019-10-13T09:45:38.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:38.542+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.542974+0000) 2019-10-13T09:45:38.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:38.546+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.549859+0000) 2019-10-13T09:45:39.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:39.082+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.085584+0000) 2019-10-13T09:45:39.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:39.082+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.086146+0000) 2019-10-13T09:45:39.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:39.082+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.086754+0000) 2019-10-13T09:45:39.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:39.082+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.087121+0000) 2019-10-13T09:45:39.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:39.084+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.087448+0000) 2019-10-13T09:45:39.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:39.086+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.088531+0000) 2019-10-13T09:45:39.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:39.086+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.088948+0000) 2019-10-13T09:45:39.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:39.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83301 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:39.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:39.092+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.095789+0000) 2019-10-13T09:45:39.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:39.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.533785+0000) 2019-10-13T09:45:41.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:41.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:41.841087+0000) 2019-10-13T09:45:42.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:42.030+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:42.033440+0000) 2019-10-13T09:45:43.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:43.384+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.387179+0000) 2019-10-13T09:45:44.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:44.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.093420+0000) 2019-10-13T09:45:44.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:44.090+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.093730+0000) 2019-10-13T09:45:44.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:44.092+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.094204+0000) 2019-10-13T09:45:44.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:44.090+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.094628+0000) 2019-10-13T09:45:44.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:44.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.094780+0000) 2019-10-13T09:45:44.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:44.090+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.095276+0000) 2019-10-13T09:45:44.097 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:44.094+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.096142+0000) 2019-10-13T09:45:44.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:44.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:44.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:44.100+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.102777+0000) 2019-10-13T09:45:45.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:45.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:45.062229+0000) 2019-10-13T09:45:46.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:46.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:46.282356+0000) 2019-10-13T09:45:47.541 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:45:47.541 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:47.545 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:45:47.545 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:48.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:48.538+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.543195+0000) 2019-10-13T09:45:48.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:48.546+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.550031+0000) 2019-10-13T09:45:49.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:49.098+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.099928+0000) 2019-10-13T09:45:49.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:49.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.100198+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:49.104+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.110123+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:49.098+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.101143+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:49.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.102118+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:49.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.102489+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:49.098+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.102944+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:49.102+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.103358+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:49.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:49.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:49.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.533951+0000) 2019-10-13T09:45:51.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:51.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:51.841292+0000) 2019-10-13T09:45:52.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:52.030+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:52.033636+0000) 2019-10-13T09:45:53.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:53.392+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.394358+0000) 2019-10-13T09:45:54.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:54.105+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.107290+0000) 2019-10-13T09:45:54.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:54.105+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.107809+0000) 2019-10-13T09:45:54.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:54.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.109777+0000) 2019-10-13T09:45:54.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:54.112+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.114627+0000) 2019-10-13T09:45:54.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:54.105+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.108921+0000) 2019-10-13T09:45:54.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:54.105+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.109291+0000) 2019-10-13T09:45:54.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:54.105+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.110343+0000) 2019-10-13T09:45:54.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:54.105+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.110775+0000) 2019-10-13T09:45:54.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:54.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83403 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:55.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:55.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:55.062410+0000) 2019-10-13T09:45:56.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:56.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:56.282550+0000) 2019-10-13T09:45:58.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:58.541+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.543414+0000) 2019-10-13T09:45:58.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:58.545+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.550217+0000) 2019-10-13T09:45:59.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:45:59.109+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.114390+0000) 2019-10-13T09:45:59.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:45:59.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.115472+0000) 2019-10-13T09:45:59.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:45:59.113+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.115760+0000) 2019-10-13T09:45:59.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:45:59.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.116090+0000) 2019-10-13T09:45:59.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:45:59.113+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.116676+0000) 2019-10-13T09:45:59.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:59.112+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.117450+0000) 2019-10-13T09:45:59.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:45:59.113+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.117050+0000) 2019-10-13T09:45:59.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:45:59.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:45:59.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:45:59.116+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.121530+0000) 2019-10-13T09:45:59.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:45:59.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.534131+0000) 2019-10-13T09:46:01.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:01.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:01.841456+0000) 2019-10-13T09:46:02.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:02.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:02.033848+0000) 2019-10-13T09:46:03.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:03.400+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.401341+0000) 2019-10-13T09:46:04.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:04.117+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.121309+0000) 2019-10-13T09:46:04.123 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:04.117+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.121646+0000) 2019-10-13T09:46:04.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:04.117+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.122538+0000) 2019-10-13T09:46:04.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:04.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.123221+0000) 2019-10-13T09:46:04.125 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:04.121+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.123849+0000) 2019-10-13T09:46:04.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:04.121+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.124175+0000) 2019-10-13T09:46:04.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:04.120+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.124622+0000) 2019-10-13T09:46:04.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:04.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:04.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:04.128+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.130770+0000) 2019-10-13T09:46:05.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:05.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:05.062615+0000) 2019-10-13T09:46:06.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:06.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:06.282742+0000) 2019-10-13T09:46:08.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:08.541+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.543576+0000) 2019-10-13T09:46:08.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:08.545+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.550418+0000) 2019-10-13T09:46:09.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:09.124+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.128733+0000) 2019-10-13T09:46:09.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:09.125+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.129076+0000) 2019-10-13T09:46:09.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:09.125+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.130251+0000) 2019-10-13T09:46:09.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:09.125+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.130349+0000) 2019-10-13T09:46:09.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:09.129+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.131515+0000) 2019-10-13T09:46:09.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:09.129+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.131794+0000) 2019-10-13T09:46:09.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:09.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:09.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:09.129+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.132293+0000) 2019-10-13T09:46:09.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:09.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.135833+0000) 2019-10-13T09:46:09.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:09.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.534377+0000) 2019-10-13T09:46:11.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:11.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:11.841660+0000) 2019-10-13T09:46:12.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:12.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:12.034053+0000) 2019-10-13T09:46:13.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:13.408+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.408335+0000) 2019-10-13T09:46:14.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:14.129+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.134962+0000) 2019-10-13T09:46:14.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:14.129+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.135256+0000) 2019-10-13T09:46:14.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:14.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.136869+0000) 2019-10-13T09:46:14.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:14.133+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.137373+0000) 2019-10-13T09:46:14.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:14.133+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.137586+0000) 2019-10-13T09:46:14.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:14.133+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.138141+0000) 2019-10-13T09:46:14.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:14.133+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.138703+0000) 2019-10-13T09:46:14.140 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:14.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:14.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:14.140+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.142735+0000) 2019-10-13T09:46:15.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:15.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:15.062794+0000) 2019-10-13T09:46:16.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:16.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:16.282956+0000) 2019-10-13T09:46:17.662 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:46:17.662 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:17.666 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:46:17.666 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:18.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:18.541+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.543795+0000) 2019-10-13T09:46:18.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:18.545+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.550616+0000) 2019-10-13T09:46:19.143 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:19.137+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.142053+0000) 2019-10-13T09:46:19.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:19.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.142382+0000) 2019-10-13T09:46:19.144 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:19.137+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.143036+0000) 2019-10-13T09:46:19.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:19.140+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.143772+0000) 2019-10-13T09:46:19.145 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:19.141+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.144091+0000) 2019-10-13T09:46:19.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:19.141+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.145028+0000) 2019-10-13T09:46:19.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:19.141+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.145693+0000) 2019-10-13T09:46:19.147 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:19.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:19.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:19.148+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.150465+0000) 2019-10-13T09:46:19.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:19.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.534549+0000) 2019-10-13T09:46:21.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:21.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:21.841876+0000) 2019-10-13T09:46:22.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:22.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:22.034280+0000) 2019-10-13T09:46:23.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:23.411+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.415586+0000) 2019-10-13T09:46:24.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:24.147+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.150339+0000) 2019-10-13T09:46:24.153 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:24.149+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.151641+0000) 2019-10-13T09:46:24.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:24.149+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.151892+0000) 2019-10-13T09:46:24.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:24.149+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.152337+0000) 2019-10-13T09:46:24.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:24.149+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.152707+0000) 2019-10-13T09:46:24.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:24.149+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.153061+0000) 2019-10-13T09:46:24.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:24.149+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.153320+0000) 2019-10-13T09:46:24.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:24.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:24.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:24.151+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.157589+0000) 2019-10-13T09:46:25.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:25.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:25.062973+0000) 2019-10-13T09:46:26.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:26.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:26.283145+0000) 2019-10-13T09:46:28.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:28.541+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.543966+0000) 2019-10-13T09:46:28.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:28.545+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.550843+0000) 2019-10-13T09:46:29.158 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:29.153+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.157434+0000) 2019-10-13T09:46:29.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:29.151+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.157869+0000) 2019-10-13T09:46:29.160 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:29.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.158273+0000) 2019-10-13T09:46:29.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:29.153+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.158739+0000) 2019-10-13T09:46:29.161 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:29.153+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.159288+0000) 2019-10-13T09:46:29.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:29.157+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.159732+0000) 2019-10-13T09:46:29.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:29.157+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.160630+0000) 2019-10-13T09:46:29.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:29.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:29.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:29.163+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.167735+0000) 2019-10-13T09:46:29.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:29.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.534718+0000) 2019-10-13T09:46:31.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:31.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:31.842095+0000) 2019-10-13T09:46:32.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:32.029+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:32.034497+0000) 2019-10-13T09:46:33.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:33.419+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.422642+0000) 2019-10-13T09:46:34.166 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:34.161+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.164421+0000) 2019-10-13T09:46:34.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:34.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.164960+0000) 2019-10-13T09:46:34.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:34.161+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.166354+0000) 2019-10-13T09:46:34.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:34.161+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.166725+0000) 2019-10-13T09:46:34.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:34.161+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.167037+0000) 2019-10-13T09:46:34.169 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:34.165+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.167794+0000) 2019-10-13T09:46:34.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:34.165+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.167899+0000) 2019-10-13T09:46:34.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:34.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:34.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:34.171+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.174835+0000) 2019-10-13T09:46:35.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:35.057+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:35.063146+0000) 2019-10-13T09:46:36.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:36.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:36.283346+0000) 2019-10-13T09:46:38.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:38.540+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.544176+0000) 2019-10-13T09:46:38.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:38.544+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.551038+0000) 2019-10-13T09:46:39.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:39.168+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.171435+0000) 2019-10-13T09:46:39.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:39.168+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.172750+0000) 2019-10-13T09:46:39.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:39.168+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.173121+0000) 2019-10-13T09:46:39.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:39.168+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.173398+0000) 2019-10-13T09:46:39.175 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:39.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.173862+0000) 2019-10-13T09:46:39.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:39.168+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.174694+0000) 2019-10-13T09:46:39.176 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:39.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.175557+0000) 2019-10-13T09:46:39.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:39.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:39.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:39.179+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.183521+0000) 2019-10-13T09:46:39.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:39.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.534893+0000) 2019-10-13T09:46:41.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:41.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:41.842315+0000) 2019-10-13T09:46:42.036 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:42.028+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:42.034691+0000) 2019-10-13T09:46:43.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:43.427+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:43.429854+0000) 2019-10-13T09:46:44.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:44.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.179970+0000) 2019-10-13T09:46:44.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:44.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.180836+0000) 2019-10-13T09:46:44.182 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:44.176+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.180711+0000) 2019-10-13T09:46:44.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:44.176+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.181723+0000) 2019-10-13T09:46:44.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:44.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.182452+0000) 2019-10-13T09:46:44.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:44.179+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.182565+0000) 2019-10-13T09:46:44.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:44.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.182953+0000) 2019-10-13T09:46:44.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:44.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:44.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:44.183+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.188226+0000) 2019-10-13T09:46:45.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:45.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:45.063348+0000) 2019-10-13T09:46:46.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:46.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:46.283541+0000) 2019-10-13T09:46:47.715 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:46:47.715 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:47.719 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:46:47.719 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:48.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:48.540+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.544389+0000) 2019-10-13T09:46:48.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:48.548+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.551253+0000) 2019-10-13T09:46:49.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:49.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.186691+0000) 2019-10-13T09:46:49.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:49.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.187112+0000) 2019-10-13T09:46:49.189 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:49.184+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.188073+0000) 2019-10-13T09:46:49.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:49.184+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.188369+0000) 2019-10-13T09:46:49.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:49.184+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.188905+0000) 2019-10-13T09:46:49.190 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:49.184+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.189160+0000) 2019-10-13T09:46:49.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:49.184+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.189435+0000) 2019-10-13T09:46:49.191 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:49.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:49.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:49.187+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.194046+0000) 2019-10-13T09:46:49.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:49.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.535066+0000) 2019-10-13T09:46:51.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:51.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:51.842540+0000) 2019-10-13T09:46:52.036 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:52.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:52.034895+0000) 2019-10-13T09:46:53.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:53.435+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:53.437045+0000) 2019-10-13T09:46:54.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:54.191+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.194457+0000) 2019-10-13T09:46:54.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:54.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.194840+0000) 2019-10-13T09:46:54.196 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:54.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.195417+0000) 2019-10-13T09:46:54.197 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:54.192+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.195851+0000) 2019-10-13T09:46:54.197 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:54.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.196174+0000) 2019-10-13T09:46:54.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:54.192+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.196887+0000) 2019-10-13T09:46:54.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:54.192+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.197738+0000) 2019-10-13T09:46:54.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:54.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:54.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:54.199+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.203365+0000) 2019-10-13T09:46:55.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:55.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:55.063526+0000) 2019-10-13T09:46:56.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:56.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:56.283766+0000) 2019-10-13T09:46:58.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:58.544+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.544606+0000) 2019-10-13T09:46:58.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:58.548+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.551478+0000) 2019-10-13T09:46:59.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:46:59.196+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.201597+0000) 2019-10-13T09:46:59.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:46:59.196+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.202774+0000) 2019-10-13T09:46:59.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:46:59.196+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.202514+0000) 2019-10-13T09:46:59.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:46:59.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.203355+0000) 2019-10-13T09:46:59.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:59.199+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.204350+0000) 2019-10-13T09:46:59.206 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:46:59.200+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.205062+0000) 2019-10-13T09:46:59.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:46:59.200+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.205435+0000) 2019-10-13T09:46:59.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:46:59.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:46:59.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:46:59.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.211213+0000) 2019-10-13T09:46:59.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:46:59.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.535275+0000) 2019-10-13T09:47:01.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:01.840+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:01.842782+0000) 2019-10-13T09:47:02.036 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:02.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:02.035128+0000) 2019-10-13T09:47:03.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:03.443+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:03.444231+0000) 2019-10-13T09:47:04.212 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:04.204+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.210941+0000) 2019-10-13T09:47:04.213 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:04.204+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.211041+0000) 2019-10-13T09:47:04.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:04.208+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.211443+0000) 2019-10-13T09:47:04.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:04.208+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.212192+0000) 2019-10-13T09:47:04.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:04.208+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.212551+0000) 2019-10-13T09:47:04.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:04.208+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.212744+0000) 2019-10-13T09:47:04.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:04.207+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.213295+0000) 2019-10-13T09:47:04.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:04.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:04.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:04.211+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.216755+0000) 2019-10-13T09:47:05.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:05.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:05.063707+0000) 2019-10-13T09:47:06.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:06.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:06.283963+0000) 2019-10-13T09:47:08.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:08.544+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.544817+0000) 2019-10-13T09:47:08.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:08.548+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.551678+0000) 2019-10-13T09:47:09.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:09.212+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.217081+0000) 2019-10-13T09:47:09.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:09.212+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.217272+0000) 2019-10-13T09:47:09.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:09.212+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.217720+0000) 2019-10-13T09:47:09.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:09.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.218302+0000) 2019-10-13T09:47:09.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:09.212+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.218782+0000) 2019-10-13T09:47:09.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:09.216+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.219782+0000) 2019-10-13T09:47:09.221 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:09.216+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.220194+0000) 2019-10-13T09:47:09.222 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:09.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:09.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:09.219+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.222857+0000) 2019-10-13T09:47:09.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:09.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.535448+0000) 2019-10-13T09:47:11.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:11.840+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:11.842985+0000) 2019-10-13T09:47:12.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:12.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:12.035356+0000) 2019-10-13T09:47:13.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:13.451+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:13.451215+0000) 2019-10-13T09:47:14.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:14.220+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.225064+0000) 2019-10-13T09:47:14.227 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:14.220+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.225554+0000) 2019-10-13T09:47:14.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:14.220+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.226040+0000) 2019-10-13T09:47:14.228 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:14.220+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.226617+0000) 2019-10-13T09:47:14.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:14.220+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.227091+0000) 2019-10-13T09:47:14.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:14.223+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.227222+0000) 2019-10-13T09:47:14.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:14.224+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.227762+0000) 2019-10-13T09:47:14.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:14.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:14.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:14.227+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.233670+0000) 2019-10-13T09:47:15.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:15.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:15.063899+0000) 2019-10-13T09:47:16.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:16.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:16.284157+0000) 2019-10-13T09:47:17.844 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:47:17.844 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:17.848 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:47:17.848 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:18.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:18.544+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.545013+0000) 2019-10-13T09:47:18.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:18.548+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.551892+0000) 2019-10-13T09:47:19.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:19.228+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.231439+0000) 2019-10-13T09:47:19.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:19.228+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.231802+0000) 2019-10-13T09:47:19.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:19.228+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.232114+0000) 2019-10-13T09:47:19.234 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:19.228+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.233365+0000) 2019-10-13T09:47:19.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:19.228+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.233717+0000) 2019-10-13T09:47:19.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:19.230+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.234507+0000) 2019-10-13T09:47:19.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:19.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 83979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:19.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:19.232+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.235474+0000) 2019-10-13T09:47:19.241 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:19.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.240595+0000) 2019-10-13T09:47:19.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:19.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.535621+0000) 2019-10-13T09:47:21.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:21.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:21.843188+0000) 2019-10-13T09:47:22.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:22.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:22.035553+0000) 2019-10-13T09:47:23.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:23.454+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:23.458494+0000) 2019-10-13T09:47:24.240 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:24.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.238627+0000) 2019-10-13T09:47:24.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:24.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.239193+0000) 2019-10-13T09:47:24.241 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:24.235+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.239979+0000) 2019-10-13T09:47:24.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:24.235+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.240866+0000) 2019-10-13T09:47:24.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:24.235+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.241188+0000) 2019-10-13T09:47:24.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:24.235+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.241576+0000) 2019-10-13T09:47:24.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:24.235+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.242468+0000) 2019-10-13T09:47:24.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:24.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:24.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:24.242+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.247018+0000) 2019-10-13T09:47:25.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:25.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:25.064091+0000) 2019-10-13T09:47:26.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:26.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:26.284388+0000) 2019-10-13T09:47:28.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:28.539+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.545237+0000) 2019-10-13T09:47:28.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:28.547+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.552105+0000) 2019-10-13T09:47:29.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:29.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.246385+0000) 2019-10-13T09:47:29.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:29.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.246623+0000) 2019-10-13T09:47:29.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:29.243+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.247660+0000) 2019-10-13T09:47:29.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:29.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84047 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:29.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:29.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.249076+0000) 2019-10-13T09:47:29.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:29.242+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.249438+0000) 2019-10-13T09:47:29.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:29.243+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.249772+0000) 2019-10-13T09:47:29.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:29.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.250196+0000) 2019-10-13T09:47:29.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:29.250+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.255745+0000) 2019-10-13T09:47:29.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:29.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.535787+0000) 2019-10-13T09:47:31.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:31.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:31.843401+0000) 2019-10-13T09:47:32.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:32.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:32.035759+0000) 2019-10-13T09:47:33.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:33.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:33.465459+0000) 2019-10-13T09:47:34.254 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:34.246+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.252901+0000) 2019-10-13T09:47:34.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:34.247+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.253224+0000) 2019-10-13T09:47:34.255 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:34.247+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.253902+0000) 2019-10-13T09:47:34.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:34.247+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.254277+0000) 2019-10-13T09:47:34.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:34.247+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.254932+0000) 2019-10-13T09:47:34.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:34.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.255791+0000) 2019-10-13T09:47:34.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:34.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:34.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:34.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.256540+0000) 2019-10-13T09:47:34.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:34.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.259707+0000) 2019-10-13T09:47:35.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:35.063+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:35.064224+0000) 2019-10-13T09:47:36.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:36.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:36.284582+0000) 2019-10-13T09:47:38.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:38.543+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.545453+0000) 2019-10-13T09:47:38.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:38.551+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.552297+0000) 2019-10-13T09:47:39.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:39.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.258538+0000) 2019-10-13T09:47:39.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:39.255+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.258844+0000) 2019-10-13T09:47:39.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:39.255+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.259838+0000) 2019-10-13T09:47:39.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:39.255+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.259935+0000) 2019-10-13T09:47:39.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:39.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.260692+0000) 2019-10-13T09:47:39.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:39.255+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.260874+0000) 2019-10-13T09:47:39.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:39.254+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.261866+0000) 2019-10-13T09:47:39.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:39.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:39.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:39.262+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.268538+0000) 2019-10-13T09:47:39.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:39.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.535969+0000) 2019-10-13T09:47:41.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:41.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:41.843651+0000) 2019-10-13T09:47:42.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:42.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:42.035977+0000) 2019-10-13T09:47:43.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:43.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:43.471567+0000) 2019-10-13T09:47:44.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:44.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.266431+0000) 2019-10-13T09:47:44.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:44.259+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.266887+0000) 2019-10-13T09:47:44.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:44.263+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.267592+0000) 2019-10-13T09:47:44.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:44.263+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.267896+0000) 2019-10-13T09:47:44.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:44.263+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.268511+0000) 2019-10-13T09:47:44.272 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:44.263+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.268893+0000) 2019-10-13T09:47:44.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:44.263+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.270154+0000) 2019-10-13T09:47:44.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:44.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:44.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:44.270+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.276568+0000) 2019-10-13T09:47:45.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:45.059+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:45.064405+0000) 2019-10-13T09:47:46.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:46.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:46.284803+0000) 2019-10-13T09:47:47.918 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:47:47.918 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:47.921 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:47:47.922 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:48.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:48.543+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.545652+0000) 2019-10-13T09:47:48.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:48.551+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.552457+0000) 2019-10-13T09:47:49.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:49.267+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.274003+0000) 2019-10-13T09:47:49.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:49.267+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.274937+0000) 2019-10-13T09:47:49.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:49.267+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.275265+0000) 2019-10-13T09:47:49.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:49.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.276487+0000) 2019-10-13T09:47:49.278 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:49.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.276699+0000) 2019-10-13T09:47:49.278 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:49.271+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.277171+0000) 2019-10-13T09:47:49.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:49.271+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.277699+0000) 2019-10-13T09:47:49.280 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:49.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:49.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:49.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.285652+0000) 2019-10-13T09:47:49.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:49.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.536160+0000) 2019-10-13T09:47:51.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:51.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:51.843855+0000) 2019-10-13T09:47:52.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:52.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:52.036181+0000) 2019-10-13T09:47:53.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:53.478+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:53.478570+0000) 2019-10-13T09:47:54.284 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:54.279+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.281939+0000) 2019-10-13T09:47:54.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:54.279+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.282371+0000) 2019-10-13T09:47:54.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:54.278+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.283098+0000) 2019-10-13T09:47:54.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:54.279+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.283248+0000) 2019-10-13T09:47:54.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:54.283+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.284280+0000) 2019-10-13T09:47:54.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:54.283+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.284623+0000) 2019-10-13T09:47:54.287 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:54.283+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.285349+0000) 2019-10-13T09:47:54.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:54.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:54.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:54.286+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.293292+0000) 2019-10-13T09:47:55.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:55.063+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:55.064600+0000) 2019-10-13T09:47:56.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:56.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:56.284997+0000) 2019-10-13T09:47:58.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:58.543+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.545856+0000) 2019-10-13T09:47:58.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:58.551+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.552676+0000) 2019-10-13T09:47:59.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:47:59.287+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.290168+0000) 2019-10-13T09:47:59.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:59.286+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.290542+0000) 2019-10-13T09:47:59.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:47:59.287+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.291213+0000) 2019-10-13T09:47:59.292 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:47:59.287+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.291257+0000) 2019-10-13T09:47:59.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:47:59.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.292307+0000) 2019-10-13T09:47:59.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:47:59.291+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.293209+0000) 2019-10-13T09:47:59.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:47:59.291+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.293781+0000) 2019-10-13T09:47:59.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:47:59.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:47:59.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:47:59.294+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.299443+0000) 2019-10-13T09:47:59.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:47:59.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.536366+0000) 2019-10-13T09:48:01.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:01.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:01.844058+0000) 2019-10-13T09:48:02.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:02.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:02.036417+0000) 2019-10-13T09:48:03.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:03.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:03.485579+0000) 2019-10-13T09:48:04.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:04.294+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.297313+0000) 2019-10-13T09:48:04.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:04.294+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.297958+0000) 2019-10-13T09:48:04.299 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:04.294+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.298570+0000) 2019-10-13T09:48:04.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:04.298+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.299186+0000) 2019-10-13T09:48:04.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:04.298+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.299527+0000) 2019-10-13T09:48:04.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:04.298+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.299783+0000) 2019-10-13T09:48:04.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:04.298+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.300866+0000) 2019-10-13T09:48:04.302 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:04.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:04.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:04.310+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.310787+0000) 2019-10-13T09:48:05.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:05.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:05.064776+0000) 2019-10-13T09:48:06.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:06.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:06.285215+0000) 2019-10-13T09:48:08.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:08.542+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.546096+0000) 2019-10-13T09:48:08.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:08.550+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.552915+0000) 2019-10-13T09:48:09.306 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:09.302+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.304769+0000) 2019-10-13T09:48:09.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:09.302+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.305446+0000) 2019-10-13T09:48:09.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:09.302+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.305664+0000) 2019-10-13T09:48:09.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:09.302+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.306595+0000) 2019-10-13T09:48:09.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:09.302+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.307024+0000) 2019-10-13T09:48:09.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:09.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.307741+0000) 2019-10-13T09:48:09.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:09.306+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.308026+0000) 2019-10-13T09:48:09.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:09.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:09.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:09.310+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.313736+0000) 2019-10-13T09:48:09.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:09.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.536542+0000) 2019-10-13T09:48:11.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:11.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:11.844255+0000) 2019-10-13T09:48:12.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:12.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:12.036629+0000) 2019-10-13T09:48:13.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:13.489+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:13.492446+0000) 2019-10-13T09:48:14.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:14.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.311804+0000) 2019-10-13T09:48:14.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:14.309+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.313388+0000) 2019-10-13T09:48:14.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:14.310+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.313912+0000) 2019-10-13T09:48:14.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:14.310+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.314430+0000) 2019-10-13T09:48:14.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:14.310+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.314807+0000) 2019-10-13T09:48:14.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:14.310+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.315245+0000) 2019-10-13T09:48:14.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:14.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:14.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:14.314+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.315691+0000) 2019-10-13T09:48:14.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:14.317+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.318502+0000) 2019-10-13T09:48:15.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:15.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:15.064953+0000) 2019-10-13T09:48:16.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:16.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:16.285418+0000) 2019-10-13T09:48:18.191 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:48:18.191 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:18.195 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:48:18.195 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:18.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:18.542+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.546296+0000) 2019-10-13T09:48:18.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:18.550+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.553131+0000) 2019-10-13T09:48:19.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:19.314+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.319249+0000) 2019-10-13T09:48:19.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:19.318+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.320037+0000) 2019-10-13T09:48:19.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:19.318+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.320127+0000) 2019-10-13T09:48:19.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:19.318+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.320587+0000) 2019-10-13T09:48:19.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:19.321+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.322347+0000) 2019-10-13T09:48:19.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:19.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:19.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:19.318+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.322898+0000) 2019-10-13T09:48:19.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:19.318+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.320755+0000) 2019-10-13T09:48:19.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:19.325+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.329642+0000) 2019-10-13T09:48:19.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:19.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.536719+0000) 2019-10-13T09:48:21.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:21.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:21.844452+0000) 2019-10-13T09:48:22.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:22.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:22.036863+0000) 2019-10-13T09:48:23.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:23.497+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:23.499313+0000) 2019-10-13T09:48:24.328 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:24.322+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.326274+0000) 2019-10-13T09:48:24.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:24.322+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.326700+0000) 2019-10-13T09:48:24.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:24.326+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.327949+0000) 2019-10-13T09:48:24.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:24.326+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.328622+0000) 2019-10-13T09:48:24.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:24.326+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.328675+0000) 2019-10-13T09:48:24.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:24.326+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.329029+0000) 2019-10-13T09:48:24.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:24.325+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.329868+0000) 2019-10-13T09:48:24.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:24.326+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:24.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:24.329+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.332526+0000) 2019-10-13T09:48:25.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:25.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:25.065163+0000) 2019-10-13T09:48:26.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:26.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:26.285648+0000) 2019-10-13T09:48:28.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:28.546+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.546538+0000) 2019-10-13T09:48:28.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:28.550+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.553326+0000) 2019-10-13T09:48:29.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:29.330+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.334113+0000) 2019-10-13T09:48:29.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:29.330+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.334541+0000) 2019-10-13T09:48:29.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:29.330+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.334922+0000) 2019-10-13T09:48:29.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:29.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.335715+0000) 2019-10-13T09:48:29.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:29.334+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.335521+0000) 2019-10-13T09:48:29.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:29.334+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.336697+0000) 2019-10-13T09:48:29.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:29.333+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.337208+0000) 2019-10-13T09:48:29.339 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:29.334+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:29.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:29.345+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.346670+0000) 2019-10-13T09:48:29.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:29.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.536910+0000) 2019-10-13T09:48:31.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:31.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:31.844650+0000) 2019-10-13T09:48:32.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:32.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:32.037073+0000) 2019-10-13T09:48:33.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:33.505+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:33.506417+0000) 2019-10-13T09:48:34.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:34.338+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.340660+0000) 2019-10-13T09:48:34.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:34.337+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.341213+0000) 2019-10-13T09:48:34.343 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:34.338+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.341518+0000) 2019-10-13T09:48:34.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:34.338+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.342097+0000) 2019-10-13T09:48:34.344 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:34.338+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.342638+0000) 2019-10-13T09:48:34.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:34.338+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.342906+0000) 2019-10-13T09:48:34.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:34.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.343823+0000) 2019-10-13T09:48:34.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:34.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:34.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:34.349+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.350383+0000) 2019-10-13T09:48:35.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:35.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:35.065342+0000) 2019-10-13T09:48:36.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:36.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:36.285846+0000) 2019-10-13T09:48:38.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:38.542+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.546733+0000) 2019-10-13T09:48:38.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:38.550+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.553521+0000) 2019-10-13T09:48:39.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:39.346+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.347846+0000) 2019-10-13T09:48:39.350 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:39.346+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.348482+0000) 2019-10-13T09:48:39.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:39.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.348829+0000) 2019-10-13T09:48:39.350 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:39.346+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.349453+0000) 2019-10-13T09:48:39.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:39.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.349710+0000) 2019-10-13T09:48:39.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:39.349+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.350447+0000) 2019-10-13T09:48:39.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:39.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:39.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:39.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.352085+0000) 2019-10-13T09:48:39.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:39.353+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.356319+0000) 2019-10-13T09:48:39.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:39.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.537085+0000) 2019-10-13T09:48:41.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:41.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:41.844846+0000) 2019-10-13T09:48:42.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:42.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:42.037282+0000) 2019-10-13T09:48:43.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:43.509+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:43.513343+0000) 2019-10-13T09:48:44.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:44.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.354972+0000) 2019-10-13T09:48:44.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:44.354+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.355550+0000) 2019-10-13T09:48:44.357 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:44.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.356642+0000) 2019-10-13T09:48:44.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:44.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.357251+0000) 2019-10-13T09:48:44.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:44.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.357508+0000) 2019-10-13T09:48:44.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:44.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.358131+0000) 2019-10-13T09:48:44.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:44.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:44.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:44.357+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.359154+0000) 2019-10-13T09:48:44.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:44.365+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.366870+0000) 2019-10-13T09:48:45.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:45.062+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:45.065516+0000) 2019-10-13T09:48:46.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:46.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:46.286073+0000) 2019-10-13T09:48:48.294 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:48:48.295 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:48.299 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:48:48.299 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:48.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:48.541+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.546931+0000) 2019-10-13T09:48:48.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:48.549+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.553749+0000) 2019-10-13T09:48:49.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:49.357+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.362402+0000) 2019-10-13T09:48:49.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:49.357+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.362820+0000) 2019-10-13T09:48:49.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:49.361+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.363712+0000) 2019-10-13T09:48:49.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:49.361+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.364216+0000) 2019-10-13T09:48:49.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:49.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.365398+0000) 2019-10-13T09:48:49.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:49.361+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.364876+0000) 2019-10-13T09:48:49.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:49.361+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.365649+0000) 2019-10-13T09:48:49.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:49.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:49.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:49.365+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.368793+0000) 2019-10-13T09:48:49.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:49.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.537253+0000) 2019-10-13T09:48:51.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:51.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:51.845067+0000) 2019-10-13T09:48:52.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:52.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:52.037520+0000) 2019-10-13T09:48:53.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:53.517+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:53.520386+0000) 2019-10-13T09:48:54.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:54.365+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.369814+0000) 2019-10-13T09:48:54.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:54.365+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.370288+0000) 2019-10-13T09:48:54.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:54.365+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.371248+0000) 2019-10-13T09:48:54.373 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:54.369+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.372000+0000) 2019-10-13T09:48:54.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:54.369+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.372504+0000) 2019-10-13T09:48:54.374 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:54.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.372779+0000) 2019-10-13T09:48:54.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:54.369+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.372863+0000) 2019-10-13T09:48:54.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:54.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:54.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:54.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.378064+0000) 2019-10-13T09:48:55.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:55.061+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:55.065693+0000) 2019-10-13T09:48:56.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:56.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:56.286301+0000) 2019-10-13T09:48:58.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:58.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.547167+0000) 2019-10-13T09:48:58.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:58.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.553944+0000) 2019-10-13T09:48:59.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:48:59.373+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.376414+0000) 2019-10-13T09:48:59.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:48:59.373+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.377462+0000) 2019-10-13T09:48:59.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:59.377+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.378577+0000) 2019-10-13T09:48:59.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:48:59.373+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.379235+0000) 2019-10-13T09:48:59.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:48:59.377+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.379779+0000) 2019-10-13T09:48:59.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:48:59.377+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.380177+0000) 2019-10-13T09:48:59.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:48:59.377+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.380518+0000) 2019-10-13T09:48:59.382 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:48:59.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:48:59.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:48:59.381+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.385565+0000) 2019-10-13T09:48:59.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:48:59.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.537460+0000) 2019-10-13T09:49:01.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:01.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:01.845340+0000) 2019-10-13T09:49:02.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:02.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:02.037738+0000) 2019-10-13T09:49:03.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:03.525+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:03.527878+0000) 2019-10-13T09:49:04.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:04.381+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.385209+0000) 2019-10-13T09:49:04.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:04.381+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.385653+0000) 2019-10-13T09:49:04.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:04.381+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.385837+0000) 2019-10-13T09:49:04.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:04.381+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.386370+0000) 2019-10-13T09:49:04.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:04.381+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.387212+0000) 2019-10-13T09:49:04.389 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:04.381+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.386842+0000) 2019-10-13T09:49:04.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:04.385+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.387566+0000) 2019-10-13T09:49:04.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:04.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:04.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:04.393+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.397030+0000) 2019-10-13T09:49:05.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:05.061+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.065917+0000) 2019-10-13T09:49:06.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:06.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:06.286509+0000) 2019-10-13T09:49:08.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:08.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.547369+0000) 2019-10-13T09:49:08.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:08.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.554144+0000) 2019-10-13T09:49:09.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:09.385+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.391198+0000) 2019-10-13T09:49:09.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:09.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.392326+0000) 2019-10-13T09:49:09.394 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:09.389+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.393169+0000) 2019-10-13T09:49:09.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:09.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.393436+0000) 2019-10-13T09:49:09.395 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:09.392+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.394437+0000) 2019-10-13T09:49:09.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:09.389+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.394842+0000) 2019-10-13T09:49:09.396 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:09.389+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.395119+0000) 2019-10-13T09:49:09.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:09.393+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:09.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:09.400+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.403709+0000) 2019-10-13T09:49:09.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:09.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.537588+0000) 2019-10-13T09:49:11.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:11.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:11.845583+0000) 2019-10-13T09:49:12.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:12.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:12.037943+0000) 2019-10-13T09:49:13.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:13.532+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:13.534799+0000) 2019-10-13T09:49:14.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:14.393+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.399102+0000) 2019-10-13T09:49:14.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:14.397+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.399497+0000) 2019-10-13T09:49:14.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:14.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.399962+0000) 2019-10-13T09:49:14.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:14.397+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.400983+0000) 2019-10-13T09:49:14.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:14.397+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.401895+0000) 2019-10-13T09:49:14.404 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:14.397+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.402407+0000) 2019-10-13T09:49:14.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:14.400+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.403348+0000) 2019-10-13T09:49:14.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:14.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:14.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:14.404+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.406898+0000) 2019-10-13T09:49:15.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:15.061+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.066112+0000) 2019-10-13T09:49:16.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:16.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:16.286704+0000) 2019-10-13T09:49:18.448 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:49:18.448 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:18.452 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:49:18.452 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:18.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:18.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.547566+0000) 2019-10-13T09:49:18.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:18.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.554358+0000) 2019-10-13T09:49:19.408 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:19.401+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.406514+0000) 2019-10-13T09:49:19.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:19.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.406961+0000) 2019-10-13T09:49:19.409 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:19.405+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.407601+0000) 2019-10-13T09:49:19.409 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:19.405+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.407785+0000) 2019-10-13T09:49:19.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:19.405+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.408921+0000) 2019-10-13T09:49:19.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:19.405+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.409994+0000) 2019-10-13T09:49:19.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:19.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:19.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:19.408+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.410843+0000) 2019-10-13T09:49:19.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:19.412+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.415198+0000) 2019-10-13T09:49:19.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:19.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.537763+0000) 2019-10-13T09:49:21.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:21.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:21.845741+0000) 2019-10-13T09:49:22.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:22.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:22.038171+0000) 2019-10-13T09:49:23.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:23.536+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:23.541785+0000) 2019-10-13T09:49:24.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:24.409+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.414185+0000) 2019-10-13T09:49:24.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:24.412+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.414623+0000) 2019-10-13T09:49:24.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:24.409+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.415009+0000) 2019-10-13T09:49:24.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:24.413+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.415996+0000) 2019-10-13T09:49:24.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:24.413+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.416288+0000) 2019-10-13T09:49:24.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:24.413+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.416825+0000) 2019-10-13T09:49:24.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:24.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.417917+0000) 2019-10-13T09:49:24.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:24.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:24.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:24.416+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.421714+0000) 2019-10-13T09:49:25.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:25.065+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.066314+0000) 2019-10-13T09:49:26.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:26.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:26.286917+0000) 2019-10-13T09:49:28.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:28.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.547814+0000) 2019-10-13T09:49:28.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:28.549+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.554561+0000) 2019-10-13T09:49:29.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:29.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.421739+0000) 2019-10-13T09:49:29.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:29.417+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.422164+0000) 2019-10-13T09:49:29.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:29.417+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.422797+0000) 2019-10-13T09:49:29.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:29.421+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.424489+0000) 2019-10-13T09:49:29.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:29.420+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.424902+0000) 2019-10-13T09:49:29.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:29.421+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.424076+0000) 2019-10-13T09:49:29.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:29.421+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.426099+0000) 2019-10-13T09:49:29.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:29.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:29.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:29.428+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.432519+0000) 2019-10-13T09:49:29.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:29.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.537914+0000) 2019-10-13T09:49:31.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:31.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:31.845975+0000) 2019-10-13T09:49:32.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:32.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:32.038375+0000) 2019-10-13T09:49:33.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:33.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:33.548886+0000) 2019-10-13T09:49:34.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:34.424+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.429415+0000) 2019-10-13T09:49:34.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:34.424+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.430338+0000) 2019-10-13T09:49:34.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:34.428+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.431655+0000) 2019-10-13T09:49:34.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:34.428+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.431962+0000) 2019-10-13T09:49:34.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:34.428+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.433875+0000) 2019-10-13T09:49:34.436 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:34.428+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.432765+0000) 2019-10-13T09:49:34.437 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:34.428+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.433257+0000) 2019-10-13T09:49:34.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:34.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:34.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:34.436+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.441288+0000) 2019-10-13T09:49:35.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:35.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.066490+0000) 2019-10-13T09:49:36.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:36.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:36.287115+0000) 2019-10-13T09:49:38.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:38.544+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.548016+0000) 2019-10-13T09:49:38.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:38.548+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.554762+0000) 2019-10-13T09:49:39.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:39.432+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.437780+0000) 2019-10-13T09:49:39.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:39.436+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.438567+0000) 2019-10-13T09:49:39.440 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:39.432+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.438331+0000) 2019-10-13T09:49:39.440 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:39.436+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.439704+0000) 2019-10-13T09:49:39.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:39.436+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.439930+0000) 2019-10-13T09:49:39.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:39.436+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.440881+0000) 2019-10-13T09:49:39.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:39.436+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:39.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:39.436+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.441557+0000) 2019-10-13T09:49:39.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:39.444+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.447956+0000) 2019-10-13T09:49:39.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:39.536+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.538098+0000) 2019-10-13T09:49:41.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:41.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:41.846182+0000) 2019-10-13T09:49:42.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:42.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:42.038614+0000) 2019-10-13T09:49:43.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:43.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:43.555178+0000) 2019-10-13T09:49:44.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:44.440+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.444099+0000) 2019-10-13T09:49:44.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:44.440+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.444960+0000) 2019-10-13T09:49:44.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:44.440+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.446513+0000) 2019-10-13T09:49:44.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:44.444+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.447052+0000) 2019-10-13T09:49:44.448 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:44.444+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.447482+0000) 2019-10-13T09:49:44.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:44.444+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.448091+0000) 2019-10-13T09:49:44.449 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:44.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:44.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:44.444+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.449077+0000) 2019-10-13T09:49:44.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:44.452+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.454977+0000) 2019-10-13T09:49:45.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:45.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.066694+0000) 2019-10-13T09:49:46.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:46.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:46.287345+0000) 2019-10-13T09:49:48.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:48.544+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.548223+0000) 2019-10-13T09:49:48.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:48.552+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.554957+0000) 2019-10-13T09:49:48.595 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:49:48.595 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:48.599 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:49:48.599 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:49.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:49.444+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.451181+0000) 2019-10-13T09:49:49.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:49.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.453612+0000) 2019-10-13T09:49:49.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:49.448+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.452384+0000) 2019-10-13T09:49:49.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:49.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.453182+0000) 2019-10-13T09:49:49.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:49.448+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.453892+0000) 2019-10-13T09:49:49.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:49.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.454304+0000) 2019-10-13T09:49:49.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:49.448+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.454689+0000) 2019-10-13T09:49:49.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:49.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 84985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:49.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:49.456+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.458600+0000) 2019-10-13T09:49:49.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:49.536+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.538324+0000) 2019-10-13T09:49:51.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:51.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:51.846381+0000) 2019-10-13T09:49:52.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:52.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:52.038815+0000) 2019-10-13T09:49:53.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:53.556+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:53.562116+0000) 2019-10-13T09:49:54.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:54.456+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.459558+0000) 2019-10-13T09:49:54.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:54.456+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.459891+0000) 2019-10-13T09:49:54.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:54.456+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.460751+0000) 2019-10-13T09:49:54.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:54.456+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.460155+0000) 2019-10-13T09:49:54.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:54.456+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.460384+0000) 2019-10-13T09:49:54.463 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:54.456+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.461185+0000) 2019-10-13T09:49:54.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:54.456+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.461865+0000) 2019-10-13T09:49:54.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:54.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:54.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:54.464+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.468238+0000) 2019-10-13T09:49:55.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:55.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.066875+0000) 2019-10-13T09:49:56.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:56.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:56.287562+0000) 2019-10-13T09:49:58.549 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:58.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.548430+0000) 2019-10-13T09:49:58.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:58.552+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.555181+0000) 2019-10-13T09:49:59.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:49:59.460+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.466815+0000) 2019-10-13T09:49:59.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:59.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.467113+0000) 2019-10-13T09:49:59.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:49:59.464+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.467655+0000) 2019-10-13T09:49:59.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:49:59.464+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.467703+0000) 2019-10-13T09:49:59.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:49:59.464+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.468074+0000) 2019-10-13T09:49:59.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:49:59.464+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.468848+0000) 2019-10-13T09:49:59.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:49:59.464+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.469268+0000) 2019-10-13T09:49:59.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:49:59.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:49:59.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:49:59.472+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.475522+0000) 2019-10-13T09:49:59.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:49:59.536+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.538458+0000) 2019-10-13T09:50:01.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:01.840+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:01.846554+0000) 2019-10-13T09:50:02.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:02.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:02.039019+0000) 2019-10-13T09:50:03.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:03.563+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:03.568972+0000) 2019-10-13T09:50:04.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:04.468+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.473086+0000) 2019-10-13T09:50:04.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:04.467+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.473354+0000) 2019-10-13T09:50:04.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:04.468+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.473877+0000) 2019-10-13T09:50:04.476 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:04.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.474600+0000) 2019-10-13T09:50:04.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:04.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.475911+0000) 2019-10-13T09:50:04.477 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:04.472+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.475916+0000) 2019-10-13T09:50:04.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:04.472+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.476959+0000) 2019-10-13T09:50:04.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:04.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85085 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:04.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:04.475+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.481848+0000) 2019-10-13T09:50:05.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:05.060+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.067076+0000) 2019-10-13T09:50:06.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:06.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:06.287792+0000) 2019-10-13T09:50:08.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:08.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.548631+0000) 2019-10-13T09:50:08.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:08.552+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.555415+0000) 2019-10-13T09:50:09.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:09.476+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.481287+0000) 2019-10-13T09:50:09.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:09.476+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.481524+0000) 2019-10-13T09:50:09.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:09.476+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.482553+0000) 2019-10-13T09:50:09.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:09.480+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.483336+0000) 2019-10-13T09:50:09.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:09.480+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.483586+0000) 2019-10-13T09:50:09.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:09.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.484035+0000) 2019-10-13T09:50:09.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:09.479+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.484494+0000) 2019-10-13T09:50:09.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:09.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:09.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:09.483+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.488670+0000) 2019-10-13T09:50:09.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:09.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.538621+0000) 2019-10-13T09:50:11.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:11.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:11.846761+0000) 2019-10-13T09:50:12.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:12.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:12.039223+0000) 2019-10-13T09:50:13.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:13.575+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:13.575967+0000) 2019-10-13T09:50:14.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:14.483+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.488911+0000) 2019-10-13T09:50:14.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:14.483+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.489288+0000) 2019-10-13T09:50:14.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:14.483+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.490035+0000) 2019-10-13T09:50:14.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:14.483+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.491018+0000) 2019-10-13T09:50:14.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:14.483+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.491194+0000) 2019-10-13T09:50:14.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:14.483+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.490870+0000) 2019-10-13T09:50:14.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:14.487+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.492056+0000) 2019-10-13T09:50:14.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:14.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:14.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:14.491+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.497044+0000) 2019-10-13T09:50:15.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:15.063+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.067283+0000) 2019-10-13T09:50:16.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:16.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:16.287991+0000) 2019-10-13T09:50:18.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:18.547+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.548826+0000) 2019-10-13T09:50:18.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:18.551+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.555644+0000) 2019-10-13T09:50:18.835 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:50:18.836 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:18.839 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:50:18.839 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:19.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:19.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.496843+0000) 2019-10-13T09:50:19.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:19.491+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.497263+0000) 2019-10-13T09:50:19.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:19.491+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.497557+0000) 2019-10-13T09:50:19.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:19.491+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.497892+0000) 2019-10-13T09:50:19.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:19.491+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.498676+0000) 2019-10-13T09:50:19.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:19.495+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.499534+0000) 2019-10-13T09:50:19.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:19.495+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.500044+0000) 2019-10-13T09:50:19.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:19.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:19.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:19.499+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.504932+0000) 2019-10-13T09:50:19.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:19.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.538799+0000) 2019-10-13T09:50:21.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:21.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:21.846955+0000) 2019-10-13T09:50:22.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:22.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:22.039425+0000) 2019-10-13T09:50:23.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:23.579+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:23.582998+0000) 2019-10-13T09:50:24.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:24.499+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.504518+0000) 2019-10-13T09:50:24.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:24.499+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.504878+0000) 2019-10-13T09:50:24.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:24.499+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.505193+0000) 2019-10-13T09:50:24.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:24.499+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.505565+0000) 2019-10-13T09:50:24.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:24.499+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.506440+0000) 2019-10-13T09:50:24.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:24.503+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.507506+0000) 2019-10-13T09:50:24.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:24.503+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.508095+0000) 2019-10-13T09:50:24.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:24.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:24.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:24.507+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.512134+0000) 2019-10-13T09:50:25.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:25.063+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.067513+0000) 2019-10-13T09:50:26.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:26.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:26.288213+0000) 2019-10-13T09:50:28.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:28.547+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.549023+0000) 2019-10-13T09:50:28.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:28.555+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.555874+0000) 2019-10-13T09:50:29.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:29.507+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.512502+0000) 2019-10-13T09:50:29.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:29.507+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.512805+0000) 2019-10-13T09:50:29.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:29.507+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.513141+0000) 2019-10-13T09:50:29.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:29.511+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.514477+0000) 2019-10-13T09:50:29.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:29.507+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.513708+0000) 2019-10-13T09:50:29.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:29.507+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.514086+0000) 2019-10-13T09:50:29.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:29.507+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.515102+0000) 2019-10-13T09:50:29.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:29.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:29.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:29.511+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.518023+0000) 2019-10-13T09:50:29.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:29.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.538958+0000) 2019-10-13T09:50:31.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:31.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:31.847126+0000) 2019-10-13T09:50:32.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:32.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:32.039625+0000) 2019-10-13T09:50:33.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:33.587+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:33.589970+0000) 2019-10-13T09:50:34.519 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:34.515+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.518355+0000) 2019-10-13T09:50:34.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:34.515+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.519593+0000) 2019-10-13T09:50:34.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:34.515+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.520088+0000) 2019-10-13T09:50:34.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:34.515+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.521152+0000) 2019-10-13T09:50:34.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:34.515+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.521253+0000) 2019-10-13T09:50:34.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:34.515+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.521590+0000) 2019-10-13T09:50:34.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:34.519+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.522652+0000) 2019-10-13T09:50:34.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:34.515+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:34.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:34.523+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.527007+0000) 2019-10-13T09:50:35.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:35.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.067693+0000) 2019-10-13T09:50:36.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:36.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:36.288405+0000) 2019-10-13T09:50:38.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:38.547+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.549235+0000) 2019-10-13T09:50:38.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:38.555+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.556070+0000) 2019-10-13T09:50:39.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:39.519+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.526056+0000) 2019-10-13T09:50:39.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:39.519+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.526995+0000) 2019-10-13T09:50:39.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:39.523+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.527771+0000) 2019-10-13T09:50:39.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:39.523+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.528766+0000) 2019-10-13T09:50:39.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:39.523+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.529026+0000) 2019-10-13T09:50:39.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:39.523+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.529141+0000) 2019-10-13T09:50:39.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:39.523+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.529961+0000) 2019-10-13T09:50:39.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:39.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:39.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:39.531+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.534532+0000) 2019-10-13T09:50:39.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:39.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.539094+0000) 2019-10-13T09:50:41.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:41.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:41.847351+0000) 2019-10-13T09:50:42.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:42.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:42.039826+0000) 2019-10-13T09:50:43.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:43.591+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:43.597053+0000) 2019-10-13T09:50:44.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:44.527+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.533760+0000) 2019-10-13T09:50:44.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:44.531+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.534204+0000) 2019-10-13T09:50:44.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:44.531+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.534837+0000) 2019-10-13T09:50:44.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:44.535+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.535415+0000) 2019-10-13T09:50:44.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:44.535+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.535646+0000) 2019-10-13T09:50:44.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:44.535+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.536016+0000) 2019-10-13T09:50:44.538 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:44.535+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.536898+0000) 2019-10-13T09:50:44.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:44.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:44.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:44.539+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.543596+0000) 2019-10-13T09:50:45.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:45.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.067878+0000) 2019-10-13T09:50:46.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:46.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:46.288625+0000) 2019-10-13T09:50:48.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:48.547+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.549431+0000) 2019-10-13T09:50:48.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:48.555+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.556267+0000) 2019-10-13T09:50:48.934 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:50:48.934 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:48.938 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:50:48.938 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:49.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:49.535+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.541693+0000) 2019-10-13T09:50:49.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:49.539+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.542056+0000) 2019-10-13T09:50:49.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:49.539+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.542310+0000) 2019-10-13T09:50:49.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:49.539+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.542953+0000) 2019-10-13T09:50:49.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:49.539+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.543204+0000) 2019-10-13T09:50:49.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:49.543+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.544165+0000) 2019-10-13T09:50:49.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:49.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:49.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:49.543+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.545561+0000) 2019-10-13T09:50:49.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:49.543+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.549985+0000) 2019-10-13T09:50:51.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:51.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:51.847571+0000) 2019-10-13T09:50:52.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:52.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:52.040046+0000) 2019-10-13T09:50:53.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:53.603+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:53.604008+0000) 2019-10-13T09:50:54.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:54.543+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.547258+0000) 2019-10-13T09:50:54.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:54.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.548921+0000) 2019-10-13T09:50:54.550 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:54.542+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.549200+0000) 2019-10-13T09:50:54.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:54.547+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.550267+0000) 2019-10-13T09:50:54.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:54.547+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.550666+0000) 2019-10-13T09:50:54.552 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:54.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.550883+0000) 2019-10-13T09:50:54.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:54.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.551381+0000) 2019-10-13T09:50:54.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:54.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85420 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:54.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:54.550+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.557237+0000) 2019-10-13T09:50:55.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:55.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.068045+0000) 2019-10-13T09:50:56.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:56.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:56.288854+0000) 2019-10-13T09:50:56.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:56.282+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:56.289753+0000) 2019-10-13T09:50:58.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:58.546+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.549629+0000) 2019-10-13T09:50:58.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:58.554+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.556468+0000) 2019-10-13T09:50:59.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:50:59.550+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.555266+0000) 2019-10-13T09:50:59.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:50:59.554+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.556036+0000) 2019-10-13T09:50:59.558 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:50:59.550+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.556690+0000) 2019-10-13T09:50:59.558 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:50:59.554+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.557153+0000) 2019-10-13T09:50:59.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:50:59.554+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.557293+0000) 2019-10-13T09:50:59.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:50:59.554+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.558232+0000) 2019-10-13T09:50:59.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:50:59.554+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.558827+0000) 2019-10-13T09:50:59.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:50:59.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:50:59.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:50:59.558+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.564712+0000) 2019-10-13T09:51:01.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:01.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:01.847741+0000) 2019-10-13T09:51:02.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:02.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:02.040245+0000) 2019-10-13T09:51:03.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:03.610+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:03.611015+0000) 2019-10-13T09:51:04.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:04.562+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.563697+0000) 2019-10-13T09:51:04.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:04.562+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.564246+0000) 2019-10-13T09:51:04.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:04.562+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.564828+0000) 2019-10-13T09:51:04.566 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:04.558+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.565052+0000) 2019-10-13T09:51:04.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:04.562+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.565656+0000) 2019-10-13T09:51:04.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:04.562+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.565767+0000) 2019-10-13T09:51:04.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:04.562+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.566443+0000) 2019-10-13T09:51:04.568 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:04.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:04.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:04.566+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.573295+0000) 2019-10-13T09:51:05.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:05.066+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.068204+0000) 2019-10-13T09:51:06.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:06.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:06.289046+0000) 2019-10-13T09:51:06.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:06.282+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:06.289945+0000) 2019-10-13T09:51:08.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:08.546+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.549854+0000) 2019-10-13T09:51:08.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:08.554+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.556708+0000) 2019-10-13T09:51:09.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:09.566+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.570391+0000) 2019-10-13T09:51:09.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:09.566+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.571049+0000) 2019-10-13T09:51:09.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:09.570+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.571367+0000) 2019-10-13T09:51:09.573 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:09.570+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.572060+0000) 2019-10-13T09:51:09.574 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:09.566+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.572675+0000) 2019-10-13T09:51:09.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:09.570+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.572875+0000) 2019-10-13T09:51:09.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:09.570+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.573211+0000) 2019-10-13T09:51:09.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:09.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:09.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:09.574+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.579034+0000) 2019-10-13T09:51:11.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:11.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:11.847979+0000) 2019-10-13T09:51:12.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:12.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:12.040446+0000) 2019-10-13T09:51:13.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:13.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:13.618089+0000) 2019-10-13T09:51:14.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:14.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.578089+0000) 2019-10-13T09:51:14.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:14.574+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.579046+0000) 2019-10-13T09:51:14.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:14.574+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.578367+0000) 2019-10-13T09:51:14.581 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:14.574+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.580554+0000) 2019-10-13T09:51:14.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:14.578+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.580907+0000) 2019-10-13T09:51:14.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:14.578+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.581299+0000) 2019-10-13T09:51:14.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:14.578+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.581893+0000) 2019-10-13T09:51:14.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:14.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:14.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:14.582+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.588117+0000) 2019-10-13T09:51:15.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:15.066+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.068397+0000) 2019-10-13T09:51:16.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:16.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:16.289239+0000) 2019-10-13T09:51:16.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:16.286+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:16.290169+0000) 2019-10-13T09:51:18.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:18.546+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.550067+0000) 2019-10-13T09:51:18.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:18.554+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.556931+0000) 2019-10-13T09:51:19.249 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:51:19.249 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:19.252 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:51:19.253 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:19.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:19.582+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.585324+0000) 2019-10-13T09:51:19.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:19.582+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.586033+0000) 2019-10-13T09:51:19.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:19.582+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.586679+0000) 2019-10-13T09:51:19.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:19.582+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.587524+0000) 2019-10-13T09:51:19.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:19.586+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.588152+0000) 2019-10-13T09:51:19.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:19.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.588665+0000) 2019-10-13T09:51:19.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:19.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:19.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:19.586+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.589511+0000) 2019-10-13T09:51:19.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:19.586+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.592095+0000) 2019-10-13T09:51:21.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:21.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:21.848193+0000) 2019-10-13T09:51:22.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:22.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:22.040659+0000) 2019-10-13T09:51:23.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:23.618+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:23.625326+0000) 2019-10-13T09:51:24.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:24.590+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.593776+0000) 2019-10-13T09:51:24.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:24.590+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.594058+0000) 2019-10-13T09:51:24.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:24.590+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.594380+0000) 2019-10-13T09:51:24.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:24.590+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.594828+0000) 2019-10-13T09:51:24.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:24.590+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.595201+0000) 2019-10-13T09:51:24.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:24.590+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.595604+0000) 2019-10-13T09:51:24.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:24.594+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.596237+0000) 2019-10-13T09:51:24.598 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:24.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:24.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:24.594+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.601159+0000) 2019-10-13T09:51:25.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:25.066+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.068531+0000) 2019-10-13T09:51:26.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:26.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:26.289450+0000) 2019-10-13T09:51:26.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:26.286+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:26.290369+0000) 2019-10-13T09:51:28.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:28.550+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.550333+0000) 2019-10-13T09:51:28.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:28.554+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.557157+0000) 2019-10-13T09:51:29.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:29.594+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.600071+0000) 2019-10-13T09:51:29.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:29.598+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.600873+0000) 2019-10-13T09:51:29.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:29.598+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.601127+0000) 2019-10-13T09:51:29.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:29.598+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.601372+0000) 2019-10-13T09:51:29.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:29.598+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.602477+0000) 2019-10-13T09:51:29.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:29.598+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.601978+0000) 2019-10-13T09:51:29.605 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:29.598+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.603105+0000) 2019-10-13T09:51:29.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:29.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:29.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:29.602+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.609297+0000) 2019-10-13T09:51:31.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:31.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:31.848356+0000) 2019-10-13T09:51:32.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:32.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:32.040873+0000) 2019-10-13T09:51:33.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:33.626+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:33.632339+0000) 2019-10-13T09:51:34.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:34.602+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.607154+0000) 2019-10-13T09:51:34.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:34.602+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.607780+0000) 2019-10-13T09:51:34.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:34.606+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.608307+0000) 2019-10-13T09:51:34.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:34.606+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.608680+0000) 2019-10-13T09:51:34.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:34.606+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.609044+0000) 2019-10-13T09:51:34.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:34.606+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.609619+0000) 2019-10-13T09:51:34.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:34.606+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.610062+0000) 2019-10-13T09:51:34.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:34.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:34.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:34.610+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.617172+0000) 2019-10-13T09:51:35.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:35.066+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.068715+0000) 2019-10-13T09:51:36.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:36.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:36.289644+0000) 2019-10-13T09:51:36.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:36.286+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:36.290590+0000) 2019-10-13T09:51:38.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:38.546+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.550529+0000) 2019-10-13T09:51:38.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:38.554+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.557353+0000) 2019-10-13T09:51:39.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:39.610+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.614432+0000) 2019-10-13T09:51:39.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:39.610+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.614945+0000) 2019-10-13T09:51:39.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:39.614+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.615625+0000) 2019-10-13T09:51:39.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:39.614+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.615958+0000) 2019-10-13T09:51:39.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:39.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.616391+0000) 2019-10-13T09:51:39.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:39.614+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.617251+0000) 2019-10-13T09:51:39.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:39.614+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.617668+0000) 2019-10-13T09:51:39.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:39.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:39.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:39.618+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.623696+0000) 2019-10-13T09:51:41.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:41.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:41.848561+0000) 2019-10-13T09:51:42.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:42.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:42.041068+0000) 2019-10-13T09:51:43.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:43.638+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:43.639475+0000) 2019-10-13T09:51:44.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:44.617+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.621824+0000) 2019-10-13T09:51:44.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:44.617+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.622429+0000) 2019-10-13T09:51:44.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:44.621+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.623507+0000) 2019-10-13T09:51:44.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:44.622+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.624148+0000) 2019-10-13T09:51:44.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:44.621+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.624603+0000) 2019-10-13T09:51:44.626 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:44.621+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.625030+0000) 2019-10-13T09:51:44.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:44.621+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.625253+0000) 2019-10-13T09:51:44.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:44.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:44.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:44.630+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.630525+0000) 2019-10-13T09:51:45.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:45.065+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.068896+0000) 2019-10-13T09:51:46.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:46.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:46.289842+0000) 2019-10-13T09:51:46.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:46.290+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:46.290826+0000) 2019-10-13T09:51:48.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:48.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.550724+0000) 2019-10-13T09:51:48.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:48.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.557553+0000) 2019-10-13T09:51:49.490 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:51:49.490 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:49.494 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:51:49.494 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:49.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:49.625+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.629754+0000) 2019-10-13T09:51:49.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:49.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.630346+0000) 2019-10-13T09:51:49.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:49.625+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.630756+0000) 2019-10-13T09:51:49.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:49.629+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.631669+0000) 2019-10-13T09:51:49.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:49.629+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.632403+0000) 2019-10-13T09:51:49.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:49.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.633313+0000) 2019-10-13T09:51:49.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:49.629+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.633655+0000) 2019-10-13T09:51:49.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:49.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:49.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:49.633+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.637351+0000) 2019-10-13T09:51:51.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:51.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:51.848780+0000) 2019-10-13T09:51:52.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:52.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:52.041269+0000) 2019-10-13T09:51:53.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:53.645+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:53.646407+0000) 2019-10-13T09:51:54.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:54.633+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.637754+0000) 2019-10-13T09:51:54.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:54.633+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.638655+0000) 2019-10-13T09:51:54.640 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:54.637+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.639135+0000) 2019-10-13T09:51:54.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:54.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.639668+0000) 2019-10-13T09:51:54.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:54.637+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.639665+0000) 2019-10-13T09:51:54.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:54.637+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.640014+0000) 2019-10-13T09:51:54.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:54.637+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.640563+0000) 2019-10-13T09:51:54.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:54.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:54.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:54.641+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.645428+0000) 2019-10-13T09:51:55.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:55.065+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.069103+0000) 2019-10-13T09:51:56.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:56.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:56.290040+0000) 2019-10-13T09:51:56.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:56.289+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:56.291039+0000) 2019-10-13T09:51:58.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:58.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.550958+0000) 2019-10-13T09:51:58.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:58.557+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.557751+0000) 2019-10-13T09:51:59.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:51:59.641+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.643679+0000) 2019-10-13T09:51:59.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:51:59.641+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.644400+0000) 2019-10-13T09:51:59.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:51:59.641+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.644744+0000) 2019-10-13T09:51:59.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:51:59.641+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.645182+0000) 2019-10-13T09:51:59.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:51:59.641+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.645215+0000) 2019-10-13T09:51:59.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:51:59.645+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.646354+0000) 2019-10-13T09:51:59.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:51:59.641+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.646303+0000) 2019-10-13T09:51:59.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:51:59.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:51:59.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:51:59.649+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.650298+0000) 2019-10-13T09:52:01.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:01.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:01.848972+0000) 2019-10-13T09:52:02.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:02.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:02.041467+0000) 2019-10-13T09:52:03.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:03.649+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:03.653443+0000) 2019-10-13T09:52:04.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:04.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.651752+0000) 2019-10-13T09:52:04.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:04.649+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.652159+0000) 2019-10-13T09:52:04.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:04.649+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.652830+0000) 2019-10-13T09:52:04.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:04.649+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.653061+0000) 2019-10-13T09:52:04.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:04.649+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.653366+0000) 2019-10-13T09:52:04.655 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:04.649+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.654347+0000) 2019-10-13T09:52:04.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:04.649+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.654627+0000) 2019-10-13T09:52:04.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:04.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:04.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:04.657+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.661672+0000) 2019-10-13T09:52:05.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:05.065+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.069276+0000) 2019-10-13T09:52:06.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:06.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:06.290261+0000) 2019-10-13T09:52:06.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:06.289+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:06.291230+0000) 2019-10-13T09:52:08.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:08.545+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.551150+0000) 2019-10-13T09:52:08.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:08.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.557956+0000) 2019-10-13T09:52:09.659 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:09.653+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.658197+0000) 2019-10-13T09:52:09.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:09.653+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.658701+0000) 2019-10-13T09:52:09.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:09.657+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.659063+0000) 2019-10-13T09:52:09.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:09.653+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.658647+0000) 2019-10-13T09:52:09.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:09.657+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.659552+0000) 2019-10-13T09:52:09.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:09.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.660610+0000) 2019-10-13T09:52:09.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:09.657+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.660930+0000) 2019-10-13T09:52:09.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:09.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:09.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:09.661+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.663687+0000) 2019-10-13T09:52:11.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:11.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:11.849202+0000) 2019-10-13T09:52:12.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:12.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:12.041636+0000) 2019-10-13T09:52:13.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:13.657+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:13.660459+0000) 2019-10-13T09:52:14.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:14.661+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.665415+0000) 2019-10-13T09:52:14.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:14.661+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.665886+0000) 2019-10-13T09:52:14.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:14.661+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.666680+0000) 2019-10-13T09:52:14.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:14.665+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.667572+0000) 2019-10-13T09:52:14.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:14.665+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.668024+0000) 2019-10-13T09:52:14.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:14.665+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.668963+0000) 2019-10-13T09:52:14.671 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:14.665+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.669629+0000) 2019-10-13T09:52:14.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:14.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:14.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:14.673+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.674273+0000) 2019-10-13T09:52:15.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:15.065+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.069455+0000) 2019-10-13T09:52:16.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:16.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:16.290455+0000) 2019-10-13T09:52:16.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:16.289+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:16.291429+0000) 2019-10-13T09:52:18.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:18.549+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.551345+0000) 2019-10-13T09:52:18.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:18.553+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.558150+0000) 2019-10-13T09:52:19.609 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:52:19.610 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:19.613 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:52:19.613 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:19.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:19.669+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.672982+0000) 2019-10-13T09:52:19.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:19.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.673556+0000) 2019-10-13T09:52:19.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:19.673+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.674457+0000) 2019-10-13T09:52:19.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:19.669+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.674326+0000) 2019-10-13T09:52:19.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:19.669+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.674719+0000) 2019-10-13T09:52:19.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:19.669+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.675054+0000) 2019-10-13T09:52:19.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:19.673+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.675741+0000) 2019-10-13T09:52:19.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:19.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 85988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:19.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:19.681+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.682452+0000) 2019-10-13T09:52:21.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:21.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:21.849432+0000) 2019-10-13T09:52:22.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:22.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:22.041867+0000) 2019-10-13T09:52:23.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:23.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:23.667478+0000) 2019-10-13T09:52:24.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:24.676+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.679991+0000) 2019-10-13T09:52:24.681 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:24.676+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.680618+0000) 2019-10-13T09:52:24.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:24.676+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.681157+0000) 2019-10-13T09:52:24.682 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:24.676+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.681569+0000) 2019-10-13T09:52:24.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:24.677+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.681927+0000) 2019-10-13T09:52:24.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:24.676+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.682682+0000) 2019-10-13T09:52:24.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:24.680+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.683364+0000) 2019-10-13T09:52:24.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:24.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:24.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:24.685+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.688808+0000) 2019-10-13T09:52:25.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:25.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.069585+0000) 2019-10-13T09:52:26.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:26.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:26.290668+0000) 2019-10-13T09:52:26.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:26.289+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:26.291620+0000) 2019-10-13T09:52:28.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:28.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.551545+0000) 2019-10-13T09:52:28.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:28.556+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.558347+0000) 2019-10-13T09:52:29.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:29.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.686866+0000) 2019-10-13T09:52:29.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:29.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.687690+0000) 2019-10-13T09:52:29.689 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:29.684+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.688443+0000) 2019-10-13T09:52:29.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:29.684+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.688521+0000) 2019-10-13T09:52:29.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:29.685+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.689378+0000) 2019-10-13T09:52:29.691 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:29.684+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.690063+0000) 2019-10-13T09:52:29.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:29.684+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.690715+0000) 2019-10-13T09:52:29.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:29.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:29.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:29.693+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.695600+0000) 2019-10-13T09:52:31.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:31.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:31.849642+0000) 2019-10-13T09:52:32.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:32.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:32.042042+0000) 2019-10-13T09:52:33.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:33.669+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:33.673886+0000) 2019-10-13T09:52:34.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:34.692+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.696235+0000) 2019-10-13T09:52:34.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:34.692+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.696733+0000) 2019-10-13T09:52:34.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:34.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.697525+0000) 2019-10-13T09:52:34.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:34.692+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.697227+0000) 2019-10-13T09:52:34.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:34.693+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.697970+0000) 2019-10-13T09:52:34.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:34.692+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.698387+0000) 2019-10-13T09:52:34.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:34.696+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.699409+0000) 2019-10-13T09:52:34.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:34.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86089 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:34.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:34.705+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.708455+0000) 2019-10-13T09:52:35.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:35.068+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.069820+0000) 2019-10-13T09:52:36.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:36.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:36.290878+0000) 2019-10-13T09:52:36.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:36.289+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:36.291807+0000) 2019-10-13T09:52:38.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:38.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.551746+0000) 2019-10-13T09:52:38.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:38.552+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.558536+0000) 2019-10-13T09:52:39.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:39.696+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.702062+0000) 2019-10-13T09:52:39.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:39.701+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.702392+0000) 2019-10-13T09:52:39.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:39.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.703195+0000) 2019-10-13T09:52:39.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:39.700+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.703696+0000) 2019-10-13T09:52:39.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:39.700+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.704512+0000) 2019-10-13T09:52:39.707 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:39.700+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.705207+0000) 2019-10-13T09:52:39.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:39.700+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.705631+0000) 2019-10-13T09:52:39.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:39.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:39.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:39.709+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.710895+0000) 2019-10-13T09:52:41.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:41.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:41.849847+0000) 2019-10-13T09:52:42.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:42.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:42.042250+0000) 2019-10-13T09:52:43.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:43.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:43.680365+0000) 2019-10-13T09:52:44.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:44.704+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.710738+0000) 2019-10-13T09:52:44.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:44.704+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.711015+0000) 2019-10-13T09:52:44.713 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:44.708+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.711337+0000) 2019-10-13T09:52:44.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:44.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.711797+0000) 2019-10-13T09:52:44.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:44.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.712133+0000) 2019-10-13T09:52:44.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:44.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.712322+0000) 2019-10-13T09:52:44.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:44.708+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.712863+0000) 2019-10-13T09:52:44.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:44.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:44.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:44.712+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.717206+0000) 2019-10-13T09:52:45.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:45.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.070017+0000) 2019-10-13T09:52:46.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:46.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:46.291070+0000) 2019-10-13T09:52:46.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:46.288+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:46.292005+0000) 2019-10-13T09:52:48.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:48.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.551902+0000) 2019-10-13T09:52:48.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:48.556+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.558775+0000) 2019-10-13T09:52:49.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:49.712+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.716843+0000) 2019-10-13T09:52:49.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:49.712+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.717163+0000) 2019-10-13T09:52:49.719 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:49.712+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.718111+0000) 2019-10-13T09:52:49.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:49.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.718109+0000) 2019-10-13T09:52:49.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:49.712+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.718286+0000) 2019-10-13T09:52:49.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:49.712+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.718522+0000) 2019-10-13T09:52:49.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:49.716+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.719325+0000) 2019-10-13T09:52:49.722 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:49.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:49.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:49.720+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.725458+0000) 2019-10-13T09:52:49.746 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:52:49.746 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:49.749 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:52:49.749 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:51.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:51.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:51.850049+0000) 2019-10-13T09:52:52.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:52.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:52.042448+0000) 2019-10-13T09:52:53.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:53.684+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:53.686937+0000) 2019-10-13T09:52:54.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:54.720+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.724958+0000) 2019-10-13T09:52:54.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:54.720+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.725814+0000) 2019-10-13T09:52:54.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:54.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.726316+0000) 2019-10-13T09:52:54.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:54.720+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.726549+0000) 2019-10-13T09:52:54.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:54.720+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.726852+0000) 2019-10-13T09:52:54.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:54.724+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.727629+0000) 2019-10-13T09:52:54.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:54.724+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.728251+0000) 2019-10-13T09:52:54.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:54.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:54.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:54.732+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.734328+0000) 2019-10-13T09:52:55.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:55.068+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.070214+0000) 2019-10-13T09:52:56.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:56.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:56.291299+0000) 2019-10-13T09:52:56.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:56.288+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:56.292185+0000) 2019-10-13T09:52:58.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:58.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.552129+0000) 2019-10-13T09:52:58.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:58.556+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.558966+0000) 2019-10-13T09:52:59.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:52:59.728+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.732526+0000) 2019-10-13T09:52:59.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:52:59.728+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.733020+0000) 2019-10-13T09:52:59.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:52:59.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.733838+0000) 2019-10-13T09:52:59.735 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:52:59.728+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.734079+0000) 2019-10-13T09:52:59.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:52:59.728+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.734458+0000) 2019-10-13T09:52:59.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:52:59.728+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.735026+0000) 2019-10-13T09:52:59.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:52:59.732+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.735712+0000) 2019-10-13T09:52:59.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:52:59.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:52:59.742 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:52:59.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.741570+0000) 2019-10-13T09:53:01.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:01.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:01.850262+0000) 2019-10-13T09:53:02.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:02.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:02.042655+0000) 2019-10-13T09:53:03.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:03.692+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:03.693258+0000) 2019-10-13T09:53:04.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:04.736+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.740160+0000) 2019-10-13T09:53:04.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:04.736+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.740614+0000) 2019-10-13T09:53:04.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:04.736+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.741249+0000) 2019-10-13T09:53:04.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:04.736+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.741988+0000) 2019-10-13T09:53:04.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:04.736+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.742876+0000) 2019-10-13T09:53:04.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:04.736+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.742968+0000) 2019-10-13T09:53:04.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:04.740+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.743976+0000) 2019-10-13T09:53:04.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:04.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:04.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:04.748+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.752561+0000) 2019-10-13T09:53:05.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:05.064+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.070389+0000) 2019-10-13T09:53:06.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:06.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:06.291522+0000) 2019-10-13T09:53:06.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:06.288+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:06.292372+0000) 2019-10-13T09:53:08.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:08.551+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.552335+0000) 2019-10-13T09:53:08.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:08.555+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.559166+0000) 2019-10-13T09:53:09.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:09.743+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.747435+0000) 2019-10-13T09:53:09.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:09.743+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.748633+0000) 2019-10-13T09:53:09.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:09.743+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.748839+0000) 2019-10-13T09:53:09.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:09.744+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.749397+0000) 2019-10-13T09:53:09.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:09.743+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.750482+0000) 2019-10-13T09:53:09.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:09.743+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.751245+0000) 2019-10-13T09:53:09.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:09.747+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.751538+0000) 2019-10-13T09:53:09.753 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:09.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:09.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:09.756+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.758995+0000) 2019-10-13T09:53:11.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:11.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:11.850465+0000) 2019-10-13T09:53:12.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:12.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:12.042853+0000) 2019-10-13T09:53:13.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:13.696+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:13.699767+0000) 2019-10-13T09:53:14.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:14.747+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.755094+0000) 2019-10-13T09:53:14.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:14.751+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.755780+0000) 2019-10-13T09:53:14.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:14.751+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.756179+0000) 2019-10-13T09:53:14.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:14.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.757295+0000) 2019-10-13T09:53:14.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:14.751+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.757650+0000) 2019-10-13T09:53:14.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:14.752+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.758103+0000) 2019-10-13T09:53:14.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:14.751+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.759015+0000) 2019-10-13T09:53:14.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:14.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:14.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:14.760+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.762745+0000) 2019-10-13T09:53:15.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:15.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.070571+0000) 2019-10-13T09:53:16.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:16.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:16.291715+0000) 2019-10-13T09:53:16.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:16.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:16.292554+0000) 2019-10-13T09:53:18.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:18.551+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.552544+0000) 2019-10-13T09:53:18.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:18.555+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.559394+0000) 2019-10-13T09:53:19.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:19.755+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.762800+0000) 2019-10-13T09:53:19.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:19.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.763232+0000) 2019-10-13T09:53:19.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:19.759+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.763699+0000) 2019-10-13T09:53:19.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:19.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.764457+0000) 2019-10-13T09:53:19.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:19.759+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.764633+0000) 2019-10-13T09:53:19.766 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:19.760+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.765256+0000) 2019-10-13T09:53:19.767 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:19.759+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.766317+0000) 2019-10-13T09:53:19.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:19.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:19.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:19.768+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.771758+0000) 2019-10-13T09:53:19.851 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:53:19.851 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:19.855 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:53:19.855 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:21.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:21.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:21.850694+0000) 2019-10-13T09:53:22.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:22.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:22.043054+0000) 2019-10-13T09:53:23.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:23.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:23.706105+0000) 2019-10-13T09:53:24.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:24.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.771547+0000) 2019-10-13T09:53:24.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:24.767+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.771545+0000) 2019-10-13T09:53:24.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:24.767+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.771871+0000) 2019-10-13T09:53:24.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:24.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.772860+0000) 2019-10-13T09:53:24.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:24.767+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.772523+0000) 2019-10-13T09:53:24.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:24.767+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.773144+0000) 2019-10-13T09:53:24.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:24.767+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.773487+0000) 2019-10-13T09:53:24.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:24.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:24.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:24.776+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.780401+0000) 2019-10-13T09:53:25.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:25.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.070755+0000) 2019-10-13T09:53:26.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:26.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:26.291930+0000) 2019-10-13T09:53:26.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:26.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:26.292745+0000) 2019-10-13T09:53:28.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:28.551+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.552738+0000) 2019-10-13T09:53:28.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:28.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.559613+0000) 2019-10-13T09:53:29.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:29.771+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.778071+0000) 2019-10-13T09:53:29.780 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:29.776+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.778593+0000) 2019-10-13T09:53:29.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:29.771+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.779192+0000) 2019-10-13T09:53:29.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:29.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.778884+0000) 2019-10-13T09:53:29.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:29.775+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.779665+0000) 2019-10-13T09:53:29.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:29.775+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.780069+0000) 2019-10-13T09:53:29.782 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:29.775+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.780495+0000) 2019-10-13T09:53:29.782 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:29.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:29.788 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:29.784+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.787201+0000) 2019-10-13T09:53:31.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:31.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:31.850865+0000) 2019-10-13T09:53:32.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:32.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:32.043274+0000) 2019-10-13T09:53:33.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:33.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:33.712716+0000) 2019-10-13T09:53:34.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:34.779+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.784406+0000) 2019-10-13T09:53:34.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:34.780+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.785010+0000) 2019-10-13T09:53:34.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:34.779+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.785926+0000) 2019-10-13T09:53:34.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:34.779+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.786221+0000) 2019-10-13T09:53:34.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:34.779+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.786604+0000) 2019-10-13T09:53:34.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:34.783+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.787538+0000) 2019-10-13T09:53:34.789 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:34.783+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.788261+0000) 2019-10-13T09:53:34.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:34.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:34.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:34.788+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.792880+0000) 2019-10-13T09:53:35.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:35.063+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.070938+0000) 2019-10-13T09:53:36.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:36.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:36.292127+0000) 2019-10-13T09:53:36.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:36.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:36.292940+0000) 2019-10-13T09:53:38.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:38.551+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.552935+0000) 2019-10-13T09:53:38.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:38.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.559860+0000) 2019-10-13T09:53:39.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:39.791+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.792570+0000) 2019-10-13T09:53:39.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:39.787+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.792973+0000) 2019-10-13T09:53:39.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:39.791+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.793426+0000) 2019-10-13T09:53:39.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:39.791+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.793634+0000) 2019-10-13T09:53:39.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:39.791+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.794663+0000) 2019-10-13T09:53:39.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:39.791+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.794817+0000) 2019-10-13T09:53:39.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:39.795+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.796081+0000) 2019-10-13T09:53:39.797 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:39.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:39.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:39.799+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.802492+0000) 2019-10-13T09:53:41.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:41.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:41.851069+0000) 2019-10-13T09:53:42.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:42.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:42.043520+0000) 2019-10-13T09:53:43.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:43.715+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:43.719140+0000) 2019-10-13T09:53:44.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:44.799+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.800053+0000) 2019-10-13T09:53:44.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:44.799+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.800404+0000) 2019-10-13T09:53:44.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:44.799+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.800892+0000) 2019-10-13T09:53:44.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:44.799+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.801233+0000) 2019-10-13T09:53:44.803 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:44.795+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.801624+0000) 2019-10-13T09:53:44.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:44.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.802527+0000) 2019-10-13T09:53:44.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:44.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.802612+0000) 2019-10-13T09:53:44.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:44.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:44.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:44.803+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.810068+0000) 2019-10-13T09:53:45.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:45.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.071120+0000) 2019-10-13T09:53:46.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:46.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:46.292320+0000) 2019-10-13T09:53:46.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:46.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:46.293133+0000) 2019-10-13T09:53:48.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:48.551+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.553152+0000) 2019-10-13T09:53:48.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:48.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.560062+0000) 2019-10-13T09:53:49.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:49.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.806422+0000) 2019-10-13T09:53:49.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:49.806+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.807757+0000) 2019-10-13T09:53:49.809 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:49.803+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.808004+0000) 2019-10-13T09:53:49.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:49.806+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.809113+0000) 2019-10-13T09:53:49.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:49.806+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.808975+0000) 2019-10-13T09:53:49.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:49.806+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.809520+0000) 2019-10-13T09:53:49.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:49.806+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.810309+0000) 2019-10-13T09:53:49.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:49.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:49.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:49.811+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.814787+0000) 2019-10-13T09:53:49.966 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:53:49.966 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:49.970 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:53:49.970 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:51.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:51.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:51.851264+0000) 2019-10-13T09:53:52.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:52.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:52.043728+0000) 2019-10-13T09:53:53.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:53.719+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:53.725462+0000) 2019-10-13T09:53:54.815 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:54.807+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.813589+0000) 2019-10-13T09:53:54.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:54.810+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.814789+0000) 2019-10-13T09:53:54.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:54.810+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.815072+0000) 2019-10-13T09:53:54.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:54.814+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.816034+0000) 2019-10-13T09:53:54.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:54.814+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.815613+0000) 2019-10-13T09:53:54.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:54.814+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.816553+0000) 2019-10-13T09:53:54.818 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:54.814+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.816962+0000) 2019-10-13T09:53:54.818 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:54.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:54.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:54.815+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.820053+0000) 2019-10-13T09:53:55.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:55.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.071297+0000) 2019-10-13T09:53:56.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:56.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:56.292521+0000) 2019-10-13T09:53:56.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:56.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:56.293331+0000) 2019-10-13T09:53:58.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:58.550+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.553346+0000) 2019-10-13T09:53:58.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:58.558+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.560261+0000) 2019-10-13T09:53:59.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:53:59.818+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.820309+0000) 2019-10-13T09:53:59.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:53:59.815+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.821129+0000) 2019-10-13T09:53:59.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:53:59.818+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.821373+0000) 2019-10-13T09:53:59.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:53:59.818+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.822197+0000) 2019-10-13T09:53:59.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:53:59.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.822531+0000) 2019-10-13T09:53:59.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:53:59.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.823609+0000) 2019-10-13T09:53:59.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:53:59.822+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.823916+0000) 2019-10-13T09:53:59.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:53:59.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:53:59.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:53:59.823+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.828332+0000) 2019-10-13T09:54:01.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:01.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:01.851497+0000) 2019-10-13T09:54:02.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:02.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:02.043953+0000) 2019-10-13T09:54:03.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:03.727+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:03.731565+0000) 2019-10-13T09:54:04.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:04.826+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.828314+0000) 2019-10-13T09:54:04.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:04.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.829077+0000) 2019-10-13T09:54:04.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:04.826+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.829623+0000) 2019-10-13T09:54:04.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:04.826+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.829949+0000) 2019-10-13T09:54:04.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:04.827+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.830453+0000) 2019-10-13T09:54:04.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:04.826+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.830798+0000) 2019-10-13T09:54:04.832 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:04.830+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.831660+0000) 2019-10-13T09:54:04.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:04.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:04.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:04.831+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.836874+0000) 2019-10-13T09:54:05.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:05.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.071519+0000) 2019-10-13T09:54:06.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:06.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:06.292747+0000) 2019-10-13T09:54:06.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:06.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:06.293530+0000) 2019-10-13T09:54:08.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:08.550+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.553543+0000) 2019-10-13T09:54:08.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:08.558+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.560471+0000) 2019-10-13T09:54:09.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:09.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.835514+0000) 2019-10-13T09:54:09.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:09.834+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.836075+0000) 2019-10-13T09:54:09.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:09.834+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.836433+0000) 2019-10-13T09:54:09.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:09.834+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.837473+0000) 2019-10-13T09:54:09.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:09.834+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.838129+0000) 2019-10-13T09:54:09.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:09.834+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.839164+0000) 2019-10-13T09:54:09.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:09.838+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:09.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:09.835+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.839040+0000) 2019-10-13T09:54:09.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:09.839+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.844349+0000) 2019-10-13T09:54:11.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:11.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:11.851686+0000) 2019-10-13T09:54:12.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:12.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:12.044174+0000) 2019-10-13T09:54:13.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:13.735+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:13.738023+0000) 2019-10-13T09:54:14.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:14.842+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.843500+0000) 2019-10-13T09:54:14.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:14.842+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.843679+0000) 2019-10-13T09:54:14.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:14.839+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.844421+0000) 2019-10-13T09:54:14.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:14.842+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.844142+0000) 2019-10-13T09:54:14.846 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:14.842+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.844813+0000) 2019-10-13T09:54:14.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:14.842+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.845434+0000) 2019-10-13T09:54:14.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:14.842+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.846386+0000) 2019-10-13T09:54:14.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:14.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:14.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:14.847+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.852821+0000) 2019-10-13T09:54:15.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:15.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.071725+0000) 2019-10-13T09:54:16.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:16.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:16.292942+0000) 2019-10-13T09:54:16.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:16.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:16.293745+0000) 2019-10-13T09:54:18.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:18.550+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.553744+0000) 2019-10-13T09:54:18.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:18.558+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.560662+0000) 2019-10-13T09:54:19.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:19.846+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.850124+0000) 2019-10-13T09:54:19.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:19.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.850536+0000) 2019-10-13T09:54:19.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:19.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.851476+0000) 2019-10-13T09:54:19.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:19.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.851717+0000) 2019-10-13T09:54:19.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:19.850+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.852220+0000) 2019-10-13T09:54:19.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:19.850+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.853017+0000) 2019-10-13T09:54:19.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:19.850+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.853850+0000) 2019-10-13T09:54:19.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:19.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:19.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:19.855+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.860981+0000) 2019-10-13T09:54:20.097 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:54:20.097 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:20.101 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:54:20.101 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:21.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:21.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:21.851888+0000) 2019-10-13T09:54:22.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:22.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:22.044383+0000) 2019-10-13T09:54:23.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:23.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:23.744496+0000) 2019-10-13T09:54:24.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:24.854+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.858197+0000) 2019-10-13T09:54:24.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:24.854+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.859277+0000) 2019-10-13T09:54:24.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:24.858+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.859544+0000) 2019-10-13T09:54:24.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:24.858+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.859940+0000) 2019-10-13T09:54:24.862 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:24.858+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.861045+0000) 2019-10-13T09:54:24.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:24.855+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.861264+0000) 2019-10-13T09:54:24.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:24.858+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.861790+0000) 2019-10-13T09:54:24.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:24.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:24.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:24.863+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.868971+0000) 2019-10-13T09:54:25.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:25.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.071887+0000) 2019-10-13T09:54:26.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:26.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:26.293139+0000) 2019-10-13T09:54:26.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:26.287+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:26.293958+0000) 2019-10-13T09:54:28.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:28.550+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.553940+0000) 2019-10-13T09:54:28.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:28.558+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.560814+0000) 2019-10-13T09:54:29.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:29.862+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.865680+0000) 2019-10-13T09:54:29.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:29.862+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.866477+0000) 2019-10-13T09:54:29.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:29.862+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.866873+0000) 2019-10-13T09:54:29.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:29.866+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.867994+0000) 2019-10-13T09:54:29.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:29.866+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.868310+0000) 2019-10-13T09:54:29.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:29.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.868655+0000) 2019-10-13T09:54:29.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:29.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.869306+0000) 2019-10-13T09:54:29.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:29.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:29.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:29.867+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.873485+0000) 2019-10-13T09:54:31.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:31.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:31.852092+0000) 2019-10-13T09:54:32.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:32.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:32.044581+0000) 2019-10-13T09:54:33.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:33.747+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:33.750878+0000) 2019-10-13T09:54:34.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:34.869+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.872571+0000) 2019-10-13T09:54:34.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:34.869+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.873146+0000) 2019-10-13T09:54:34.875 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:34.869+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.874287+0000) 2019-10-13T09:54:34.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:34.869+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.875208+0000) 2019-10-13T09:54:34.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:34.873+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.875726+0000) 2019-10-13T09:54:34.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:34.870+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.876222+0000) 2019-10-13T09:54:34.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:34.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:34.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:34.873+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.876643+0000) 2019-10-13T09:54:34.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:34.878+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.883634+0000) 2019-10-13T09:54:35.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:35.069+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.072088+0000) 2019-10-13T09:54:36.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:36.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:36.293333+0000) 2019-10-13T09:54:36.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:36.290+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:36.294169+0000) 2019-10-13T09:54:38.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:38.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.554171+0000) 2019-10-13T09:54:38.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:38.557+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.561012+0000) 2019-10-13T09:54:39.882 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:39.877+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.881173+0000) 2019-10-13T09:54:39.883 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:39.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.881396+0000) 2019-10-13T09:54:39.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:39.874+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.881655+0000) 2019-10-13T09:54:39.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:39.877+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.881945+0000) 2019-10-13T09:54:39.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:39.877+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.882269+0000) 2019-10-13T09:54:39.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:39.877+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.883247+0000) 2019-10-13T09:54:39.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:39.881+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.883999+0000) 2019-10-13T09:54:39.885 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:39.881+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:39.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:39.882+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.886589+0000) 2019-10-13T09:54:41.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:41.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:41.852321+0000) 2019-10-13T09:54:42.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:42.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:42.044787+0000) 2019-10-13T09:54:43.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:43.754+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:43.757213+0000) 2019-10-13T09:54:44.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:44.885+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.888295+0000) 2019-10-13T09:54:44.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:44.885+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.888606+0000) 2019-10-13T09:54:44.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:44.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.888898+0000) 2019-10-13T09:54:44.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:44.885+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.889281+0000) 2019-10-13T09:54:44.891 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:44.885+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.889710+0000) 2019-10-13T09:54:44.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:44.885+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.890282+0000) 2019-10-13T09:54:44.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:44.885+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.890877+0000) 2019-10-13T09:54:44.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:44.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 86970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:44.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:44.894+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.899812+0000) 2019-10-13T09:54:45.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:45.069+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.072273+0000) 2019-10-13T09:54:46.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:46.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:46.293524+0000) 2019-10-13T09:54:46.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:46.290+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:46.294350+0000) 2019-10-13T09:54:48.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:48.549+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.554370+0000) 2019-10-13T09:54:48.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:48.557+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.561199+0000) 2019-10-13T09:54:49.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:49.893+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.895927+0000) 2019-10-13T09:54:49.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:49.893+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.896270+0000) 2019-10-13T09:54:49.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:49.890+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.896642+0000) 2019-10-13T09:54:49.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:49.893+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.897581+0000) 2019-10-13T09:54:49.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:49.893+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.897978+0000) 2019-10-13T09:54:49.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:49.893+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.898455+0000) 2019-10-13T09:54:49.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:49.893+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.898717+0000) 2019-10-13T09:54:49.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:49.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:49.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:49.898+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.903735+0000) 2019-10-13T09:54:50.216 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:54:50.216 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:50.220 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:54:50.220 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:51.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:51.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:51.852533+0000) 2019-10-13T09:54:52.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:52.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:52.044987+0000) 2019-10-13T09:54:53.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:53.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:53.766539+0000) 2019-10-13T09:54:54.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:54.898+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.902064+0000) 2019-10-13T09:54:54.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:54.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.902797+0000) 2019-10-13T09:54:54.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:54.901+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.903589+0000) 2019-10-13T09:54:54.905 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:54.901+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.903890+0000) 2019-10-13T09:54:54.906 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:54.901+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.905118+0000) 2019-10-13T09:54:54.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:54.901+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.905509+0000) 2019-10-13T09:54:54.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:54.901+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.906610+0000) 2019-10-13T09:54:54.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:54.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:54.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:54.906+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.910779+0000) 2019-10-13T09:54:55.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:55.069+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.072469+0000) 2019-10-13T09:54:56.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:56.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:56.293756+0000) 2019-10-13T09:54:56.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:56.290+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:56.294576+0000) 2019-10-13T09:54:58.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:58.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.554567+0000) 2019-10-13T09:54:58.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:58.557+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.561392+0000) 2019-10-13T09:54:59.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:54:59.905+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.909905+0000) 2019-10-13T09:54:59.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:54:59.905+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.911119+0000) 2019-10-13T09:54:59.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:54:59.909+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.911531+0000) 2019-10-13T09:54:59.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:54:59.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.911777+0000) 2019-10-13T09:54:59.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:54:59.909+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.912014+0000) 2019-10-13T09:54:59.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:54:59.909+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.912388+0000) 2019-10-13T09:54:59.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:54:59.909+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.913031+0000) 2019-10-13T09:54:59.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:54:59.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:54:59.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:54:59.914+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.919260+0000) 2019-10-13T09:55:01.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:01.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:01.852772+0000) 2019-10-13T09:55:02.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:02.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:02.045193+0000) 2019-10-13T09:55:03.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:03.766+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:03.772915+0000) 2019-10-13T09:55:04.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:04.914+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.918653+0000) 2019-10-13T09:55:04.920 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:04.913+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.918009+0000) 2019-10-13T09:55:04.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:04.913+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.919169+0000) 2019-10-13T09:55:04.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:04.917+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.919694+0000) 2019-10-13T09:55:04.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:04.917+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.920123+0000) 2019-10-13T09:55:04.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:04.917+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.920580+0000) 2019-10-13T09:55:04.922 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:04.917+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.921018+0000) 2019-10-13T09:55:04.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:04.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:04.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:04.918+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.924810+0000) 2019-10-13T09:55:05.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:05.069+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.072604+0000) 2019-10-13T09:55:06.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:06.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:06.293953+0000) 2019-10-13T09:55:06.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:06.294+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:06.294769+0000) 2019-10-13T09:55:08.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:08.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.554764+0000) 2019-10-13T09:55:08.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:08.561+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.561591+0000) 2019-10-13T09:55:09.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:09.918+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.925511+0000) 2019-10-13T09:55:09.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:09.921+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.925687+0000) 2019-10-13T09:55:09.927 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:09.921+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.926352+0000) 2019-10-13T09:55:09.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:09.921+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.926638+0000) 2019-10-13T09:55:09.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:09.921+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.926950+0000) 2019-10-13T09:55:09.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:09.925+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.927814+0000) 2019-10-13T09:55:09.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:09.925+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.928406+0000) 2019-10-13T09:55:09.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:09.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:09.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:09.926+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.931472+0000) 2019-10-13T09:55:11.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:11.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:11.852997+0000) 2019-10-13T09:55:12.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:12.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:12.045395+0000) 2019-10-13T09:55:13.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:13.778+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:13.779304+0000) 2019-10-13T09:55:14.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:14.929+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.931750+0000) 2019-10-13T09:55:14.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:14.929+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.933168+0000) 2019-10-13T09:55:14.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:14.929+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.933423+0000) 2019-10-13T09:55:14.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:14.929+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.933870+0000) 2019-10-13T09:55:14.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:14.930+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.934888+0000) 2019-10-13T09:55:14.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:14.929+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.934640+0000) 2019-10-13T09:55:14.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:14.929+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.935022+0000) 2019-10-13T09:55:14.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:14.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:14.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:14.934+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.942070+0000) 2019-10-13T09:55:15.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:15.069+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.072788+0000) 2019-10-13T09:55:16.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:16.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:16.294179+0000) 2019-10-13T09:55:16.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:16.290+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:16.294962+0000) 2019-10-13T09:55:18.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:18.548+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.554997+0000) 2019-10-13T09:55:18.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:18.560+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.561791+0000) 2019-10-13T09:55:19.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:19.932+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.939255+0000) 2019-10-13T09:55:19.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:19.934+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.940506+0000) 2019-10-13T09:55:19.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:19.936+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.940987+0000) 2019-10-13T09:55:19.942 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:19.936+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.941143+0000) 2019-10-13T09:55:19.943 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:19.936+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.941490+0000) 2019-10-13T09:55:19.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:19.936+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.942001+0000) 2019-10-13T09:55:19.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:19.936+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.942233+0000) 2019-10-13T09:55:19.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:19.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:19.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:19.946+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.952203+0000) 2019-10-13T09:55:20.319 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:55:20.319 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:20.323 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:55:20.323 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:21.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:21.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:21.853199+0000) 2019-10-13T09:55:22.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:22.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:22.045603+0000) 2019-10-13T09:55:23.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:23.782+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:23.785737+0000) 2019-10-13T09:55:24.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:24.940+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.946358+0000) 2019-10-13T09:55:24.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:24.944+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.948249+0000) 2019-10-13T09:55:24.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:24.944+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.948810+0000) 2019-10-13T09:55:24.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:24.944+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.949256+0000) 2019-10-13T09:55:24.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:24.946+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.949869+0000) 2019-10-13T09:55:24.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:24.944+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.950288+0000) 2019-10-13T09:55:24.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:24.944+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.950834+0000) 2019-10-13T09:55:24.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:24.944+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:24.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:24.954+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.956089+0000) 2019-10-13T09:55:25.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:25.068+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.072967+0000) 2019-10-13T09:55:26.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:26.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:26.294374+0000) 2019-10-13T09:55:26.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:26.294+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:26.295146+0000) 2019-10-13T09:55:28.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:28.552+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.555202+0000) 2019-10-13T09:55:28.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:28.560+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.561984+0000) 2019-10-13T09:55:29.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:29.949+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.953863+0000) 2019-10-13T09:55:29.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:29.948+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.954810+0000) 2019-10-13T09:55:29.956 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:29.952+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.955791+0000) 2019-10-13T09:55:29.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:29.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.956576+0000) 2019-10-13T09:55:29.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:29.952+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.957452+0000) 2019-10-13T09:55:29.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:29.952+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.957855+0000) 2019-10-13T09:55:29.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:29.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:29.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:29.952+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.958807+0000) 2019-10-13T09:55:29.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:29.957+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.961370+0000) 2019-10-13T09:55:31.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:31.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:31.853401+0000) 2019-10-13T09:55:32.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:32.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:32.045836+0000) 2019-10-13T09:55:33.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:33.789+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:33.792142+0000) 2019-10-13T09:55:34.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:34.956+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.962438+0000) 2019-10-13T09:55:34.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:34.961+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.962979+0000) 2019-10-13T09:55:34.965 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:34.960+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.964049+0000) 2019-10-13T09:55:34.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:34.960+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.964445+0000) 2019-10-13T09:55:34.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:34.960+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.964789+0000) 2019-10-13T09:55:34.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:34.960+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.965132+0000) 2019-10-13T09:55:34.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:34.960+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.965662+0000) 2019-10-13T09:55:34.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:34.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:34.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:34.969+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.971574+0000) 2019-10-13T09:55:35.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:35.068+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.073148+0000) 2019-10-13T09:55:36.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:36.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:36.294589+0000) 2019-10-13T09:55:36.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:36.293+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:36.295355+0000) 2019-10-13T09:55:38.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:38.552+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.555430+0000) 2019-10-13T09:55:38.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:38.556+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.562179+0000) 2019-10-13T09:55:39.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:39.964+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.969203+0000) 2019-10-13T09:55:39.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:39.964+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.970179+0000) 2019-10-13T09:55:39.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:39.964+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.970736+0000) 2019-10-13T09:55:39.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:39.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.971345+0000) 2019-10-13T09:55:39.973 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:39.968+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.971680+0000) 2019-10-13T09:55:39.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:39.968+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.971989+0000) 2019-10-13T09:55:39.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:39.968+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.973161+0000) 2019-10-13T09:55:39.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:39.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:39.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:39.973+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.977260+0000) 2019-10-13T09:55:41.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:41.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:41.853604+0000) 2019-10-13T09:55:42.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:42.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:42.046037+0000) 2019-10-13T09:55:43.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:43.797+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:43.798893+0000) 2019-10-13T09:55:44.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:44.972+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.976171+0000) 2019-10-13T09:55:44.978 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:44.972+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.977071+0000) 2019-10-13T09:55:44.979 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:44.972+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.977911+0000) 2019-10-13T09:55:44.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:44.972+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.978210+0000) 2019-10-13T09:55:44.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:44.972+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.978564+0000) 2019-10-13T09:55:44.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:44.977+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.979149+0000) 2019-10-13T09:55:44.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:44.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:44.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:44.976+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.979898+0000) 2019-10-13T09:55:44.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:44.985+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.986845+0000) 2019-10-13T09:55:45.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:45.072+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.073328+0000) 2019-10-13T09:55:46.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:46.293+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:46.295575+0000) 2019-10-13T09:55:46.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:46.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:46.294804+0000) 2019-10-13T09:55:48.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:48.552+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.555628+0000) 2019-10-13T09:55:48.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:48.556+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.562369+0000) 2019-10-13T09:55:49.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:49.980+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.983349+0000) 2019-10-13T09:55:49.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:49.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.983952+0000) 2019-10-13T09:55:49.986 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:49.980+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.984384+0000) 2019-10-13T09:55:49.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:49.980+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.984753+0000) 2019-10-13T09:55:49.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:49.980+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.985886+0000) 2019-10-13T09:55:49.987 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:49.980+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.986650+0000) 2019-10-13T09:55:49.988 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:49.980+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.986993+0000) 2019-10-13T09:55:49.988 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:49.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:49.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:49.989+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.992026+0000) 2019-10-13T09:55:50.448 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:55:50.448 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:50.452 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:55:50.452 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:51.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:51.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:51.853817+0000) 2019-10-13T09:55:52.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:52.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:52.046251+0000) 2019-10-13T09:55:53.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:53.801+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:53.805388+0000) 2019-10-13T09:55:54.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:54.984+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.990134+0000) 2019-10-13T09:55:54.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:54.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.991160+0000) 2019-10-13T09:55:54.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:54.988+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.991743+0000) 2019-10-13T09:55:54.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:54.989+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.992646+0000) 2019-10-13T09:55:54.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:54.988+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.992797+0000) 2019-10-13T09:55:54.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:54.988+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.993357+0000) 2019-10-13T09:55:54.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:54.988+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.993675+0000) 2019-10-13T09:55:54.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:54.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:55:55.004 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:55:54.997+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.000883+0000) 2019-10-13T09:55:55.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:55.072+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.073503+0000) 2019-10-13T09:55:56.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:56.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:56.294990+0000) 2019-10-13T09:55:56.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:56.293+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:56.295768+0000) 2019-10-13T09:55:58.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:58.552+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.555845+0000) 2019-10-13T09:55:58.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:58.560+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.562595+0000) 2019-10-13T09:55:59.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:55:59.991+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.997328+0000) 2019-10-13T09:55:59.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:55:59.991+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.997792+0000) 2019-10-13T09:55:59.999 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:55:59.991+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.998136+0000) 2019-10-13T09:56:00.000 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:55:59.991+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.999107+0000) 2019-10-13T09:56:00.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:55:59.997+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.999231+0000) 2019-10-13T09:56:00.001 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:55:59.995+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.999585+0000) 2019-10-13T09:56:00.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:55:59.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.999817+0000) 2019-10-13T09:56:00.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:55:59.995+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:00.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:00.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.006425+0000) 2019-10-13T09:56:01.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:01.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:01.854054+0000) 2019-10-13T09:56:02.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:02.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:02.046497+0000) 2019-10-13T09:56:03.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:03.809+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:03.811803+0000) 2019-10-13T09:56:05.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:04.999+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.004213+0000) 2019-10-13T09:56:05.006 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:04.999+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.004970+0000) 2019-10-13T09:56:05.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:05.001+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.005737+0000) 2019-10-13T09:56:05.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:04.999+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.005455+0000) 2019-10-13T09:56:05.008 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:04.999+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.006516+0000) 2019-10-13T09:56:05.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:04.999+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.006082+0000) 2019-10-13T09:56:05.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:05.003+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.007382+0000) 2019-10-13T09:56:05.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:05.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:05.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:05.009+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.012881+0000) 2019-10-13T09:56:05.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:05.071+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.073708+0000) 2019-10-13T09:56:06.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:06.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:06.295144+0000) 2019-10-13T09:56:06.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:06.293+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:06.295961+0000) 2019-10-13T09:56:08.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:08.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.556043+0000) 2019-10-13T09:56:08.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:08.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.562799+0000) 2019-10-13T09:56:10.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:10.007+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.011572+0000) 2019-10-13T09:56:10.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:10.007+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.012424+0000) 2019-10-13T09:56:10.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:10.007+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.012538+0000) 2019-10-13T09:56:10.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:10.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.012945+0000) 2019-10-13T09:56:10.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:10.007+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.013145+0000) 2019-10-13T09:56:10.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:10.007+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.013437+0000) 2019-10-13T09:56:10.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:10.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.014075+0000) 2019-10-13T09:56:10.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:10.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:10.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:10.017+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.020880+0000) 2019-10-13T09:56:11.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:11.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:11.854273+0000) 2019-10-13T09:56:12.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:12.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:12.046697+0000) 2019-10-13T09:56:13.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:13.817+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:13.818245+0000) 2019-10-13T09:56:15.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:15.015+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.019652+0000) 2019-10-13T09:56:15.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:15.017+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.020447+0000) 2019-10-13T09:56:15.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:15.015+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.021022+0000) 2019-10-13T09:56:15.022 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:15.015+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.021421+0000) 2019-10-13T09:56:15.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:15.015+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.022064+0000) 2019-10-13T09:56:15.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:15.015+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.022349+0000) 2019-10-13T09:56:15.024 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:15.015+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.023084+0000) 2019-10-13T09:56:15.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:15.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:15.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:15.025+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.027868+0000) 2019-10-13T09:56:15.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:15.067+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.073919+0000) 2019-10-13T09:56:16.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:16.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:16.295337+0000) 2019-10-13T09:56:16.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:16.293+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:16.296188+0000) 2019-10-13T09:56:18.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:18.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.556249+0000) 2019-10-13T09:56:18.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:18.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.562997+0000) 2019-10-13T09:56:20.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:20.023+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.027473+0000) 2019-10-13T09:56:20.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:20.023+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.027702+0000) 2019-10-13T09:56:20.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:20.023+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.028204+0000) 2019-10-13T09:56:20.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:20.025+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.028893+0000) 2019-10-13T09:56:20.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:20.023+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.029118+0000) 2019-10-13T09:56:20.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:20.023+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.029451+0000) 2019-10-13T09:56:20.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:20.023+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.029919+0000) 2019-10-13T09:56:20.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:20.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:20.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:20.033+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.034675+0000) 2019-10-13T09:56:20.592 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:56:20.592 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:20.596 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:56:20.596 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:21.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:21.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:21.854476+0000) 2019-10-13T09:56:22.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:22.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:22.046904+0000) 2019-10-13T09:56:23.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:23.829+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:23.830107+0000) 2019-10-13T09:56:25.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:25.027+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.034341+0000) 2019-10-13T09:56:25.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:25.027+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.034919+0000) 2019-10-13T09:56:25.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:25.027+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.035264+0000) 2019-10-13T09:56:25.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:25.032+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.035711+0000) 2019-10-13T09:56:25.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:25.031+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.036570+0000) 2019-10-13T09:56:25.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:25.031+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.036667+0000) 2019-10-13T09:56:25.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:25.031+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.037590+0000) 2019-10-13T09:56:25.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:25.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:25.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:25.040+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.043037+0000) 2019-10-13T09:56:25.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:25.071+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.074090+0000) 2019-10-13T09:56:26.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:26.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:26.295533+0000) 2019-10-13T09:56:26.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:26.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:26.296379+0000) 2019-10-13T09:56:28.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:28.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.556480+0000) 2019-10-13T09:56:28.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:28.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.563190+0000) 2019-10-13T09:56:30.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:30.039+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.042361+0000) 2019-10-13T09:56:30.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:30.039+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.043002+0000) 2019-10-13T09:56:30.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:30.039+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.044455+0000) 2019-10-13T09:56:30.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:30.039+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.044463+0000) 2019-10-13T09:56:30.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:30.040+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.045409+0000) 2019-10-13T09:56:30.047 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:30.039+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.046159+0000) 2019-10-13T09:56:30.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:30.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:30.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:30.039+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.046704+0000) 2019-10-13T09:56:30.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:30.052+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.054166+0000) 2019-10-13T09:56:31.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:31.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:31.854742+0000) 2019-10-13T09:56:32.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:32.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:32.047101+0000) 2019-10-13T09:56:33.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:33.832+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:33.836543+0000) 2019-10-13T09:56:35.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:35.047+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.050303+0000) 2019-10-13T09:56:35.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:35.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.050755+0000) 2019-10-13T09:56:35.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:35.047+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.051222+0000) 2019-10-13T09:56:35.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:35.051+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.051609+0000) 2019-10-13T09:56:35.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:35.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.051949+0000) 2019-10-13T09:56:35.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:35.051+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.052194+0000) 2019-10-13T09:56:35.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:35.051+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.053549+0000) 2019-10-13T09:56:35.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:35.051+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:35.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:35.056+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.060252+0000) 2019-10-13T09:56:35.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:35.071+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.074281+0000) 2019-10-13T09:56:36.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:36.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:36.295727+0000) 2019-10-13T09:56:36.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:36.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:36.296585+0000) 2019-10-13T09:56:38.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:38.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.556708+0000) 2019-10-13T09:56:38.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:38.563+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.563382+0000) 2019-10-13T09:56:40.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:40.055+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.057843+0000) 2019-10-13T09:56:40.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:40.055+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.058607+0000) 2019-10-13T09:56:40.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:40.055+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.058975+0000) 2019-10-13T09:56:40.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:40.059+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.059640+0000) 2019-10-13T09:56:40.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:40.056+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.060335+0000) 2019-10-13T09:56:40.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:40.059+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.060530+0000) 2019-10-13T09:56:40.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:40.059+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.061082+0000) 2019-10-13T09:56:40.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:40.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:40.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:40.064+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.069781+0000) 2019-10-13T09:56:41.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:41.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:41.854977+0000) 2019-10-13T09:56:42.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:42.047+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:42.047331+0000) 2019-10-13T09:56:43.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:43.840+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:43.843161+0000) 2019-10-13T09:56:45.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:45.062+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.064766+0000) 2019-10-13T09:56:45.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:45.062+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.065239+0000) 2019-10-13T09:56:45.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:45.062+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.065765+0000) 2019-10-13T09:56:45.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:45.064+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.066395+0000) 2019-10-13T09:56:45.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:45.062+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.066639+0000) 2019-10-13T09:56:45.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:45.066+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.067473+0000) 2019-10-13T09:56:45.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:45.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:45.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:45.066+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.068794+0000) 2019-10-13T09:56:45.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:45.068+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.072735+0000) 2019-10-13T09:56:45.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:45.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.074415+0000) 2019-10-13T09:56:46.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:46.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:46.295933+0000) 2019-10-13T09:56:46.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:46.296+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:46.296776+0000) 2019-10-13T09:56:48.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:48.554+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.556934+0000) 2019-10-13T09:56:48.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:48.562+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.563538+0000) 2019-10-13T09:56:50.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:50.070+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.072033+0000) 2019-10-13T09:56:50.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:50.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.072349+0000) 2019-10-13T09:56:50.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:50.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.072634+0000) 2019-10-13T09:56:50.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:50.070+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.073175+0000) 2019-10-13T09:56:50.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:50.070+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.074116+0000) 2019-10-13T09:56:50.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:50.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:50.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:50.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.074702+0000) 2019-10-13T09:56:50.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:50.072+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.075305+0000) 2019-10-13T09:56:50.083 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:50.080+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.082510+0000) 2019-10-13T09:56:50.812 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:56:50.812 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:50.815 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:56:50.816 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:51.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:51.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:51.855175+0000) 2019-10-13T09:56:52.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:52.046+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:52.047546+0000) 2019-10-13T09:56:53.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:53.848+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:53.849482+0000) 2019-10-13T09:56:55.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:55.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.074541+0000) 2019-10-13T09:56:55.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:56:55.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.077372+0000) 2019-10-13T09:56:55.079 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:55.074+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.077958+0000) 2019-10-13T09:56:55.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:56:55.074+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.078430+0000) 2019-10-13T09:56:55.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:56:55.074+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.078741+0000) 2019-10-13T09:56:55.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:55.074+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.079171+0000) 2019-10-13T09:56:55.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:55.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.079499+0000) 2019-10-13T09:56:55.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:55.078+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.080629+0000) 2019-10-13T09:56:55.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:56:55.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:56:55.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:56:55.084+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.086735+0000) 2019-10-13T09:56:56.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:56:56.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:56.296147+0000) 2019-10-13T09:56:56.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:56:56.296+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:56.296968+0000) 2019-10-13T09:56:58.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:56:58.554+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.557156+0000) 2019-10-13T09:56:58.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:56:58.562+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.563743+0000) 2019-10-13T09:57:00.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:00.082+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.084886+0000) 2019-10-13T09:57:00.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:00.080+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.086031+0000) 2019-10-13T09:57:00.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:00.082+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.086501+0000) 2019-10-13T09:57:00.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:00.082+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.086825+0000) 2019-10-13T09:57:00.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:00.086+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.087726+0000) 2019-10-13T09:57:00.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:00.086+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.088088+0000) 2019-10-13T09:57:00.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:00.086+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.088776+0000) 2019-10-13T09:57:00.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:00.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:00.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:00.088+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.094127+0000) 2019-10-13T09:57:01.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:01.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:01.855376+0000) 2019-10-13T09:57:02.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:02.046+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:02.047751+0000) 2019-10-13T09:57:03.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:03.860+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:03.861139+0000) 2019-10-13T09:57:05.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:05.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.074748+0000) 2019-10-13T09:57:05.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:05.088+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.092205+0000) 2019-10-13T09:57:05.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:05.090+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.092640+0000) 2019-10-13T09:57:05.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:05.090+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.093481+0000) 2019-10-13T09:57:05.095 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:05.090+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.093806+0000) 2019-10-13T09:57:05.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:05.090+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.094556+0000) 2019-10-13T09:57:05.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:05.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.095022+0000) 2019-10-13T09:57:05.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:05.094+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.095637+0000) 2019-10-13T09:57:05.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:05.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87920 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:05.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:05.100+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.102997+0000) 2019-10-13T09:57:06.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:06.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:06.296375+0000) 2019-10-13T09:57:06.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:06.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:06.297161+0000) 2019-10-13T09:57:08.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:08.554+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.557357+0000) 2019-10-13T09:57:08.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:08.562+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.563944+0000) 2019-10-13T09:57:10.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:10.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.100322+0000) 2019-10-13T09:57:10.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:10.098+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.100749+0000) 2019-10-13T09:57:10.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:10.098+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.101175+0000) 2019-10-13T09:57:10.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:10.098+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.101432+0000) 2019-10-13T09:57:10.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:10.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.102566+0000) 2019-10-13T09:57:10.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:10.100+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.103176+0000) 2019-10-13T09:57:10.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:10.102+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.103400+0000) 2019-10-13T09:57:10.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:10.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:10.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:10.104+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.109363+0000) 2019-10-13T09:57:11.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:11.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:11.855590+0000) 2019-10-13T09:57:12.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:12.046+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:12.047959+0000) 2019-10-13T09:57:13.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:13.864+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:13.867572+0000) 2019-10-13T09:57:15.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:15.074+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.074949+0000) 2019-10-13T09:57:15.110 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:15.106+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.108445+0000) 2019-10-13T09:57:15.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:15.106+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.108826+0000) 2019-10-13T09:57:15.111 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:15.106+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.109114+0000) 2019-10-13T09:57:15.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:15.106+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.109515+0000) 2019-10-13T09:57:15.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:15.104+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.109858+0000) 2019-10-13T09:57:15.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:15.106+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.110656+0000) 2019-10-13T09:57:15.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:15.106+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.110161+0000) 2019-10-13T09:57:15.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:15.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 87987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:15.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:15.112+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.117228+0000) 2019-10-13T09:57:16.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:16.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:16.296566+0000) 2019-10-13T09:57:16.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:16.292+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:16.297354+0000) 2019-10-13T09:57:18.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:18.554+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.557559+0000) 2019-10-13T09:57:18.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:18.562+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.564134+0000) 2019-10-13T09:57:20.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:20.114+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.116130+0000) 2019-10-13T09:57:20.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:20.114+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.116410+0000) 2019-10-13T09:57:20.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:20.111+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.117403+0000) 2019-10-13T09:57:20.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:20.114+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.116999+0000) 2019-10-13T09:57:20.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:20.114+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.117666+0000) 2019-10-13T09:57:20.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:20.114+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.118029+0000) 2019-10-13T09:57:20.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:20.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.118321+0000) 2019-10-13T09:57:20.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:20.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:20.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:20.119+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.124662+0000) 2019-10-13T09:57:20.869 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:57:20.870 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:20.873 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:57:20.873 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:21.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:21.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:21.855814+0000) 2019-10-13T09:57:22.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:22.046+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:22.048160+0000) 2019-10-13T09:57:23.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:23.871+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:23.874076+0000) 2019-10-13T09:57:25.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:25.070+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.075168+0000) 2019-10-13T09:57:25.124 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:25.118+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.123026+0000) 2019-10-13T09:57:25.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:25.122+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.123789+0000) 2019-10-13T09:57:25.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:25.122+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.124794+0000) 2019-10-13T09:57:25.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:25.122+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.124632+0000) 2019-10-13T09:57:25.126 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:25.119+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.125625+0000) 2019-10-13T09:57:25.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:25.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.125989+0000) 2019-10-13T09:57:25.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:25.122+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.126537+0000) 2019-10-13T09:57:25.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:25.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:25.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:25.123+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.129923+0000) 2019-10-13T09:57:26.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:26.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:26.296760+0000) 2019-10-13T09:57:26.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:26.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:26.297561+0000) 2019-10-13T09:57:28.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:28.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.557790+0000) 2019-10-13T09:57:28.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:28.561+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.564361+0000) 2019-10-13T09:57:30.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:30.125+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.129844+0000) 2019-10-13T09:57:30.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:30.125+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.130137+0000) 2019-10-13T09:57:30.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:30.125+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.130699+0000) 2019-10-13T09:57:30.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:30.129+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.131632+0000) 2019-10-13T09:57:30.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:30.129+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.132289+0000) 2019-10-13T09:57:30.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:30.127+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.133177+0000) 2019-10-13T09:57:30.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:30.129+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.133694+0000) 2019-10-13T09:57:30.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:30.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:30.140 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:30.135+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.139099+0000) 2019-10-13T09:57:31.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:31.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:31.856048+0000) 2019-10-13T09:57:32.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:32.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:32.048368+0000) 2019-10-13T09:57:33.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:33.879+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:33.880666+0000) 2019-10-13T09:57:35.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:35.073+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.075391+0000) 2019-10-13T09:57:35.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:35.133+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.137302+0000) 2019-10-13T09:57:35.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:35.133+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.137870+0000) 2019-10-13T09:57:35.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:35.133+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.138246+0000) 2019-10-13T09:57:35.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:35.133+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.138905+0000) 2019-10-13T09:57:35.140 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:35.137+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.139291+0000) 2019-10-13T09:57:35.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:35.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.141385+0000) 2019-10-13T09:57:35.143 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:35.137+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.141697+0000) 2019-10-13T09:57:35.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:35.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:35.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:35.143+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.146874+0000) 2019-10-13T09:57:36.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:36.295+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:36.297750+0000) 2019-10-13T09:57:36.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:36.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:36.296982+0000) 2019-10-13T09:57:38.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:38.557+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.558013+0000) 2019-10-13T09:57:38.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:38.561+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.564592+0000) 2019-10-13T09:57:40.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:40.141+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.146056+0000) 2019-10-13T09:57:40.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:40.145+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.147757+0000) 2019-10-13T09:57:40.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:40.143+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.148142+0000) 2019-10-13T09:57:40.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:40.145+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.148524+0000) 2019-10-13T09:57:40.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:40.145+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.148947+0000) 2019-10-13T09:57:40.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:40.145+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.149831+0000) 2019-10-13T09:57:40.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:40.145+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.150022+0000) 2019-10-13T09:57:40.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:40.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:40.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:40.147+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.153848+0000) 2019-10-13T09:57:41.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:41.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:41.856284+0000) 2019-10-13T09:57:42.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:42.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:42.048573+0000) 2019-10-13T09:57:43.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:43.883+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:43.887235+0000) 2019-10-13T09:57:45.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:45.073+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.075595+0000) 2019-10-13T09:57:45.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:45.149+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.154178+0000) 2019-10-13T09:57:45.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:45.149+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.154753+0000) 2019-10-13T09:57:45.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:45.149+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.155144+0000) 2019-10-13T09:57:45.157 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:45.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.156297+0000) 2019-10-13T09:57:45.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:45.151+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.156324+0000) 2019-10-13T09:57:45.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:45.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.156754+0000) 2019-10-13T09:57:45.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:45.153+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.158027+0000) 2019-10-13T09:57:45.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:45.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:45.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:45.163+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.166365+0000) 2019-10-13T09:57:46.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:46.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:46.297179+0000) 2019-10-13T09:57:46.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:46.291+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:46.297964+0000) 2019-10-13T09:57:48.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:48.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.558217+0000) 2019-10-13T09:57:48.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:48.561+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.564784+0000) 2019-10-13T09:57:50.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:50.157+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.161549+0000) 2019-10-13T09:57:50.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:50.159+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.162233+0000) 2019-10-13T09:57:50.163 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:50.157+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.162480+0000) 2019-10-13T09:57:50.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:50.161+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.163702+0000) 2019-10-13T09:57:50.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:50.161+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.164049+0000) 2019-10-13T09:57:50.166 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:50.161+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.163666+0000) 2019-10-13T09:57:50.166 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:50.161+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.164516+0000) 2019-10-13T09:57:50.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:50.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:50.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:50.163+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.170003+0000) 2019-10-13T09:57:50.975 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:57:50.976 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:50.979 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:57:50.980 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:51.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:51.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:51.856506+0000) 2019-10-13T09:57:52.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:52.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:52.048805+0000) 2019-10-13T09:57:53.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:53.887+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:53.893733+0000) 2019-10-13T09:57:55.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:55.073+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.075815+0000) 2019-10-13T09:57:55.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:57:55.165+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.169646+0000) 2019-10-13T09:57:55.171 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:55.165+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.170047+0000) 2019-10-13T09:57:55.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:57:55.165+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.170586+0000) 2019-10-13T09:57:55.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:55.165+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.170707+0000) 2019-10-13T09:57:55.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:55.167+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.171123+0000) 2019-10-13T09:57:55.173 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:55.169+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.171382+0000) 2019-10-13T09:57:55.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:57:55.169+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.172798+0000) 2019-10-13T09:57:55.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:57:55.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:57:55.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:57:55.175+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.180499+0000) 2019-10-13T09:57:56.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:57:56.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:56.297376+0000) 2019-10-13T09:57:56.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:57:56.295+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:56.298187+0000) 2019-10-13T09:57:58.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:57:58.553+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.558413+0000) 2019-10-13T09:57:58.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:57:58.561+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.565016+0000) 2019-10-13T09:58:00.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:00.173+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.176722+0000) 2019-10-13T09:58:00.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:00.173+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.177090+0000) 2019-10-13T09:58:00.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:00.173+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.177833+0000) 2019-10-13T09:58:00.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:00.173+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.178419+0000) 2019-10-13T09:58:00.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:00.177+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.179512+0000) 2019-10-13T09:58:00.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:00.177+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.180209+0000) 2019-10-13T09:58:00.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:00.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.180673+0000) 2019-10-13T09:58:00.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:00.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:00.186 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:00.179+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.185321+0000) 2019-10-13T09:58:01.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:01.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:01.856727+0000) 2019-10-13T09:58:02.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:02.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:02.049009+0000) 2019-10-13T09:58:03.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:03.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:03.900087+0000) 2019-10-13T09:58:05.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:05.073+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.076043+0000) 2019-10-13T09:58:05.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:05.181+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.184374+0000) 2019-10-13T09:58:05.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:05.181+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.184933+0000) 2019-10-13T09:58:05.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:05.181+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.185190+0000) 2019-10-13T09:58:05.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:05.181+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.185429+0000) 2019-10-13T09:58:05.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:05.181+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.186282+0000) 2019-10-13T09:58:05.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:05.181+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.186734+0000) 2019-10-13T09:58:05.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:05.183+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.187012+0000) 2019-10-13T09:58:05.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:05.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:05.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:05.191+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.194366+0000) 2019-10-13T09:58:06.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:06.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:06.297568+0000) 2019-10-13T09:58:06.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:06.295+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:06.298384+0000) 2019-10-13T09:58:08.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:08.557+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.558647+0000) 2019-10-13T09:58:08.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:08.565+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.565212+0000) 2019-10-13T09:58:10.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:10.184+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.190898+0000) 2019-10-13T09:58:10.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:10.188+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.191507+0000) 2019-10-13T09:58:10.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:10.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.192227+0000) 2019-10-13T09:58:10.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:10.188+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.192939+0000) 2019-10-13T09:58:10.194 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:10.188+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.193661+0000) 2019-10-13T09:58:10.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:10.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.194465+0000) 2019-10-13T09:58:10.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:10.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.194296+0000) 2019-10-13T09:58:10.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:10.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:10.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:10.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.201693+0000) 2019-10-13T09:58:11.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:11.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:11.856938+0000) 2019-10-13T09:58:12.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:12.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:12.049214+0000) 2019-10-13T09:58:13.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:13.903+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:13.906589+0000) 2019-10-13T09:58:15.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:15.072+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.076251+0000) 2019-10-13T09:58:15.200 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:15.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.198676+0000) 2019-10-13T09:58:15.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:15.194+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.201361+0000) 2019-10-13T09:58:15.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:15.196+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.199712+0000) 2019-10-13T09:58:15.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:15.196+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.200326+0000) 2019-10-13T09:58:15.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:15.196+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.200568+0000) 2019-10-13T09:58:15.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:15.196+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.200957+0000) 2019-10-13T09:58:15.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:15.196+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.202207+0000) 2019-10-13T09:58:15.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:15.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:15.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:15.202+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.209318+0000) 2019-10-13T09:58:16.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:16.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:16.297769+0000) 2019-10-13T09:58:16.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:16.294+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:16.298592+0000) 2019-10-13T09:58:18.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:18.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.558851+0000) 2019-10-13T09:58:18.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:18.564+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.565406+0000) 2019-10-13T09:58:20.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:20.200+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.206620+0000) 2019-10-13T09:58:20.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:20.200+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.206053+0000) 2019-10-13T09:58:20.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:20.200+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.207069+0000) 2019-10-13T09:58:20.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:20.204+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.207575+0000) 2019-10-13T09:58:20.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:20.202+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.208100+0000) 2019-10-13T09:58:20.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:20.204+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.208359+0000) 2019-10-13T09:58:20.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:20.204+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.209043+0000) 2019-10-13T09:58:20.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:20.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:20.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:20.210+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.216965+0000) 2019-10-13T09:58:21.099 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:58:21.099 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:21.103 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:58:21.104 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:21.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:21.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:21.857135+0000) 2019-10-13T09:58:22.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:22.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:22.049454+0000) 2019-10-13T09:58:23.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:23.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:23.913089+0000) 2019-10-13T09:58:25.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:25.072+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.076482+0000) 2019-10-13T09:58:25.215 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:25.208+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.213969+0000) 2019-10-13T09:58:25.216 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:25.208+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.214541+0000) 2019-10-13T09:58:25.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:25.212+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.215337+0000) 2019-10-13T09:58:25.217 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:25.212+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.215725+0000) 2019-10-13T09:58:25.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:25.210+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.216061+0000) 2019-10-13T09:58:25.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:25.212+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.216419+0000) 2019-10-13T09:58:25.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:25.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.217725+0000) 2019-10-13T09:58:25.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:25.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:25.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:25.218+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.225530+0000) 2019-10-13T09:58:26.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:26.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.297960+0000) 2019-10-13T09:58:26.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:26.294+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.298783+0000) 2019-10-13T09:58:28.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:28.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.559062+0000) 2019-10-13T09:58:28.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:28.560+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.565609+0000) 2019-10-13T09:58:30.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:30.216+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.221110+0000) 2019-10-13T09:58:30.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:30.214+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.221430+0000) 2019-10-13T09:58:30.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:30.216+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.222984+0000) 2019-10-13T09:58:30.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:30.216+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.223141+0000) 2019-10-13T09:58:30.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:30.220+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.223799+0000) 2019-10-13T09:58:30.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:30.220+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.224072+0000) 2019-10-13T09:58:30.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:30.220+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.224417+0000) 2019-10-13T09:58:30.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:30.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:30.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:30.226+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.231677+0000) 2019-10-13T09:58:31.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:31.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.857341+0000) 2019-10-13T09:58:32.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:32.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:32.049699+0000) 2019-10-13T09:58:33.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:33.918+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:33.919774+0000) 2019-10-13T09:58:35.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:35.076+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.076717+0000) 2019-10-13T09:58:35.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:35.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.228793+0000) 2019-10-13T09:58:35.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:35.224+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.229174+0000) 2019-10-13T09:58:35.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:35.224+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.229517+0000) 2019-10-13T09:58:35.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:35.224+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.229926+0000) 2019-10-13T09:58:35.232 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:35.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.230454+0000) 2019-10-13T09:58:35.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:35.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.230820+0000) 2019-10-13T09:58:35.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:35.228+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.231808+0000) 2019-10-13T09:58:35.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:35.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:35.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:35.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.238164+0000) 2019-10-13T09:58:36.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:36.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.298158+0000) 2019-10-13T09:58:36.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:36.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.298981+0000) 2019-10-13T09:58:38.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:38.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.559264+0000) 2019-10-13T09:58:38.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:38.560+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.565789+0000) 2019-10-13T09:58:40.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:40.232+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.235264+0000) 2019-10-13T09:58:40.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:40.232+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.236041+0000) 2019-10-13T09:58:40.237 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:40.232+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.236458+0000) 2019-10-13T09:58:40.238 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:40.232+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.236985+0000) 2019-10-13T09:58:40.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:40.230+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.237405+0000) 2019-10-13T09:58:40.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:40.232+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.237891+0000) 2019-10-13T09:58:40.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:40.232+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.238351+0000) 2019-10-13T09:58:40.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:40.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:40.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:40.238+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.246122+0000) 2019-10-13T09:58:41.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:41.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.857546+0000) 2019-10-13T09:58:42.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:42.048+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:42.049924+0000) 2019-10-13T09:58:43.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:43.922+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:43.926362+0000) 2019-10-13T09:58:45.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:45.072+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.076948+0000) 2019-10-13T09:58:45.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:45.240+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.243804+0000) 2019-10-13T09:58:45.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:45.238+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.243884+0000) 2019-10-13T09:58:45.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:45.240+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.244197+0000) 2019-10-13T09:58:45.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:45.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.245229+0000) 2019-10-13T09:58:45.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:45.240+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.245600+0000) 2019-10-13T09:58:45.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:45.240+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.246045+0000) 2019-10-13T09:58:45.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:45.240+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.246728+0000) 2019-10-13T09:58:45.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:45.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:45.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:45.250+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.255380+0000) 2019-10-13T09:58:46.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:46.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.298366+0000) 2019-10-13T09:58:46.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:46.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.299193+0000) 2019-10-13T09:58:48.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:48.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.559509+0000) 2019-10-13T09:58:48.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:48.564+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.565986+0000) 2019-10-13T09:58:50.253 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:50.248+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.252183+0000) 2019-10-13T09:58:50.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:50.248+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.252771+0000) 2019-10-13T09:58:50.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:50.248+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.253719+0000) 2019-10-13T09:58:50.256 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:50.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.254613+0000) 2019-10-13T09:58:50.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:50.248+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.255205+0000) 2019-10-13T09:58:50.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:50.252+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.255602+0000) 2019-10-13T09:58:50.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:50.252+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.255942+0000) 2019-10-13T09:58:50.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:50.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:50.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:50.258+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.265985+0000) 2019-10-13T09:58:51.189 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:58:51.189 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:51.193 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:58:51.193 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:51.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:51.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.857747+0000) 2019-10-13T09:58:52.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:52.048+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:52.050136+0000) 2019-10-13T09:58:53.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:53.930+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:53.932833+0000) 2019-10-13T09:58:55.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:55.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.077167+0000) 2019-10-13T09:58:55.261 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:58:55.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.259659+0000) 2019-10-13T09:58:55.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:58:55.255+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.260183+0000) 2019-10-13T09:58:55.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:58:55.255+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.260537+0000) 2019-10-13T09:58:55.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:55.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.261268+0000) 2019-10-13T09:58:55.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:55.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.261797+0000) 2019-10-13T09:58:55.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:55.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.262491+0000) 2019-10-13T09:58:55.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:55.259+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.263565+0000) 2019-10-13T09:58:55.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:58:55.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:58:55.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:58:55.266+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.271365+0000) 2019-10-13T09:58:56.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:58:56.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.298578+0000) 2019-10-13T09:58:56.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:58:56.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.299389+0000) 2019-10-13T09:58:58.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:58:58.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.559708+0000) 2019-10-13T09:58:58.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:58:58.563+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.566219+0000) 2019-10-13T09:59:00.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:00.263+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.268298+0000) 2019-10-13T09:59:00.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:00.263+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.268601+0000) 2019-10-13T09:59:00.270 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:00.263+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.269759+0000) 2019-10-13T09:59:00.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:00.263+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.270082+0000) 2019-10-13T09:59:00.271 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:00.263+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.270635+0000) 2019-10-13T09:59:00.272 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:00.263+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.270960+0000) 2019-10-13T09:59:00.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:00.266+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.271424+0000) 2019-10-13T09:59:00.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:00.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:00.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:00.270+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.274209+0000) 2019-10-13T09:59:01.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:01.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.857942+0000) 2019-10-13T09:59:02.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:02.047+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:02.050384+0000) 2019-10-13T09:59:03.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:03.938+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:03.939239+0000) 2019-10-13T09:59:05.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:05.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.077402+0000) 2019-10-13T09:59:05.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:05.271+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.275454+0000) 2019-10-13T09:59:05.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:05.271+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.275726+0000) 2019-10-13T09:59:05.278 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:05.274+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.276108+0000) 2019-10-13T09:59:05.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:05.271+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.276688+0000) 2019-10-13T09:59:05.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:05.271+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.277970+0000) 2019-10-13T09:59:05.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:05.271+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.279069+0000) 2019-10-13T09:59:05.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:05.275+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.279731+0000) 2019-10-13T09:59:05.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:05.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:05.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:05.282+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.284454+0000) 2019-10-13T09:59:06.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:06.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.298804+0000) 2019-10-13T09:59:06.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:06.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.299608+0000) 2019-10-13T09:59:08.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:08.555+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.559924+0000) 2019-10-13T09:59:08.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:08.559+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.566437+0000) 2019-10-13T09:59:10.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:10.279+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.283051+0000) 2019-10-13T09:59:10.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:10.279+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.283599+0000) 2019-10-13T09:59:10.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:10.279+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.284550+0000) 2019-10-13T09:59:10.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:10.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.285083+0000) 2019-10-13T09:59:10.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:10.279+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.285580+0000) 2019-10-13T09:59:10.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:10.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.285814+0000) 2019-10-13T09:59:10.287 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:10.279+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.286404+0000) 2019-10-13T09:59:10.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:10.279+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:10.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:10.294+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.295262+0000) 2019-10-13T09:59:11.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:11.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.858171+0000) 2019-10-13T09:59:12.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:12.047+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:12.050602+0000) 2019-10-13T09:59:13.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:13.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:13.945652+0000) 2019-10-13T09:59:15.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:15.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.077603+0000) 2019-10-13T09:59:15.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:15.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.290358+0000) 2019-10-13T09:59:15.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:15.283+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.290890+0000) 2019-10-13T09:59:15.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:15.287+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.291782+0000) 2019-10-13T09:59:15.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:15.287+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.292260+0000) 2019-10-13T09:59:15.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:15.287+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.293255+0000) 2019-10-13T09:59:15.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:15.289+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.293547+0000) 2019-10-13T09:59:15.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:15.287+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.293832+0000) 2019-10-13T09:59:15.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:15.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:15.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:15.297+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.298972+0000) 2019-10-13T09:59:16.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:16.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.298999+0000) 2019-10-13T09:59:16.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:16.297+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.299798+0000) 2019-10-13T09:59:18.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:18.559+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.560125+0000) 2019-10-13T09:59:18.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:18.563+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.566583+0000) 2019-10-13T09:59:20.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:20.291+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.297844+0000) 2019-10-13T09:59:20.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:20.297+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.298299+0000) 2019-10-13T09:59:20.300 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:20.291+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.298577+0000) 2019-10-13T09:59:20.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:20.291+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.299040+0000) 2019-10-13T09:59:20.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:20.295+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.299808+0000) 2019-10-13T09:59:20.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:20.295+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.300139+0000) 2019-10-13T09:59:20.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:20.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.301041+0000) 2019-10-13T09:59:20.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:20.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:20.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:20.305+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.308469+0000) 2019-10-13T09:59:21.312 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:59:21.313 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:21.316 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:59:21.316 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:21.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:21.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.858391+0000) 2019-10-13T09:59:22.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:22.047+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:22.050838+0000) 2019-10-13T09:59:23.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:23.949+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:23.952099+0000) 2019-10-13T09:59:25.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:25.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.077807+0000) 2019-10-13T09:59:25.306 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:25.303+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.305029+0000) 2019-10-13T09:59:25.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:25.303+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.306385+0000) 2019-10-13T09:59:25.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:25.303+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.306926+0000) 2019-10-13T09:59:25.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:25.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.307933+0000) 2019-10-13T09:59:25.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:25.305+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.307488+0000) 2019-10-13T09:59:25.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:25.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.308593+0000) 2019-10-13T09:59:25.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:25.307+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.309061+0000) 2019-10-13T09:59:25.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:25.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:25.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:25.309+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.313291+0000) 2019-10-13T09:59:26.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:26.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.299202+0000) 2019-10-13T09:59:26.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:26.297+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.299991+0000) 2019-10-13T09:59:28.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:28.559+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.560321+0000) 2019-10-13T09:59:28.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:28.563+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.566778+0000) 2019-10-13T09:59:30.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:30.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.312642+0000) 2019-10-13T09:59:30.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:30.311+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.313175+0000) 2019-10-13T09:59:30.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:30.311+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.313390+0000) 2019-10-13T09:59:30.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:30.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.313972+0000) 2019-10-13T09:59:30.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:30.311+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.314143+0000) 2019-10-13T09:59:30.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:30.313+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.314990+0000) 2019-10-13T09:59:30.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:30.315+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.315709+0000) 2019-10-13T09:59:30.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:30.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:30.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:30.317+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.321698+0000) 2019-10-13T09:59:31.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:31.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.858638+0000) 2019-10-13T09:59:32.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:32.047+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:32.051043+0000) 2019-10-13T09:59:33.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:33.957+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:33.958738+0000) 2019-10-13T09:59:35.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:35.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.078037+0000) 2019-10-13T09:59:35.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:35.319+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.320268+0000) 2019-10-13T09:59:35.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:35.317+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.321972+0000) 2019-10-13T09:59:35.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:35.319+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.322224+0000) 2019-10-13T09:59:35.324 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:35.319+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.322525+0000) 2019-10-13T09:59:35.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:35.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.323205+0000) 2019-10-13T09:59:35.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:35.323+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.323595+0000) 2019-10-13T09:59:35.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:35.323+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.324140+0000) 2019-10-13T09:59:35.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:35.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:35.328 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:35.325+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.327756+0000) 2019-10-13T09:59:36.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:36.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.299377+0000) 2019-10-13T09:59:36.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:36.297+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.300163+0000) 2019-10-13T09:59:38.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:38.558+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.560519+0000) 2019-10-13T09:59:38.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:38.562+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.566994+0000) 2019-10-13T09:59:40.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:40.326+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.328951+0000) 2019-10-13T09:59:40.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:40.326+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.329198+0000) 2019-10-13T09:59:40.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:40.326+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.329898+0000) 2019-10-13T09:59:40.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:40.326+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.330294+0000) 2019-10-13T09:59:40.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:40.326+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.330980+0000) 2019-10-13T09:59:40.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:40.330+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.331594+0000) 2019-10-13T09:59:40.334 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:40.329+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.332034+0000) 2019-10-13T09:59:40.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:40.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 88977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:40.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:40.333+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.337622+0000) 2019-10-13T09:59:41.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:41.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.858880+0000) 2019-10-13T09:59:42.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:42.050+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:42.051243+0000) 2019-10-13T09:59:43.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:43.961+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:43.965109+0000) 2019-10-13T09:59:45.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:45.074+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.078270+0000) 2019-10-13T09:59:45.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:45.330+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.335149+0000) 2019-10-13T09:59:45.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:45.334+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.336237+0000) 2019-10-13T09:59:45.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:45.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.336709+0000) 2019-10-13T09:59:45.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:45.333+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.337116+0000) 2019-10-13T09:59:45.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:45.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.337404+0000) 2019-10-13T09:59:45.339 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:45.334+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.337741+0000) 2019-10-13T09:59:45.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:45.334+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.338597+0000) 2019-10-13T09:59:45.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:45.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:45.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:45.345+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.346153+0000) 2019-10-13T09:59:46.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:46.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.299558+0000) 2019-10-13T09:59:46.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:46.297+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.300332+0000) 2019-10-13T09:59:48.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:48.558+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.560713+0000) 2019-10-13T09:59:48.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:48.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.567194+0000) 2019-10-13T09:59:50.343 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:50.338+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.342188+0000) 2019-10-13T09:59:50.344 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:50.338+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.343232+0000) 2019-10-13T09:59:50.344 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:50.341+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.343785+0000) 2019-10-13T09:59:50.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:50.342+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.343973+0000) 2019-10-13T09:59:50.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:50.342+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.344887+0000) 2019-10-13T09:59:50.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:50.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.345210+0000) 2019-10-13T09:59:50.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:50.342+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.345599+0000) 2019-10-13T09:59:50.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:50.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:50.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:50.349+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.352519+0000) 2019-10-13T09:59:51.430 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T09:59:51.430 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:51.434 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T09:59:51.435 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:51.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:51.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.859098+0000) 2019-10-13T09:59:52.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:52.050+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:52.051507+0000) 2019-10-13T09:59:53.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:53.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:53.971751+0000) 2019-10-13T09:59:55.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:55.074+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.078469+0000) 2019-10-13T09:59:55.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T09:59:55.346+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.350276+0000) 2019-10-13T09:59:55.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:55.349+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.350635+0000) 2019-10-13T09:59:55.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:55.350+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.351538+0000) 2019-10-13T09:59:55.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T09:59:55.350+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.352219+0000) 2019-10-13T09:59:55.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T09:59:55.350+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.352835+0000) 2019-10-13T09:59:55.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:55.350+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.353903+0000) 2019-10-13T09:59:55.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:55.350+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.354286+0000) 2019-10-13T09:59:55.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T09:59:55.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T09:59:55.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T09:59:55.357+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.359774+0000) 2019-10-13T09:59:56.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T09:59:56.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.299726+0000) 2019-10-13T09:59:56.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T09:59:56.297+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.300511+0000) 2019-10-13T09:59:58.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T09:59:58.558+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.560963+0000) 2019-10-13T09:59:58.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T09:59:58.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.567409+0000) 2019-10-13T10:00:00.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:00.357+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.358282+0000) 2019-10-13T10:00:00.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:00.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.359129+0000) 2019-10-13T10:00:00.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:00.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.359946+0000) 2019-10-13T10:00:00.362 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:00.358+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.360107+0000) 2019-10-13T10:00:00.362 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:00.358+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.360415+0000) 2019-10-13T10:00:00.363 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:00.358+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.360986+0000) 2019-10-13T10:00:00.363 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:00.358+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.361635+0000) 2019-10-13T10:00:00.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:00.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:00.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:00.365+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.368618+0000) 2019-10-13T10:00:01.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:01.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.859343+0000) 2019-10-13T10:00:02.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:02.050+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:02.051718+0000) 2019-10-13T10:00:03.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:03.977+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:03.978348+0000) 2019-10-13T10:00:05.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:05.074+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.078692+0000) 2019-10-13T10:00:05.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:05.362+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.364555+0000) 2019-10-13T10:00:05.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:05.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.365220+0000) 2019-10-13T10:00:05.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:05.362+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.365987+0000) 2019-10-13T10:00:05.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:05.362+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.366843+0000) 2019-10-13T10:00:05.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:05.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.367914+0000) 2019-10-13T10:00:05.369 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:05.366+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.368314+0000) 2019-10-13T10:00:05.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:05.366+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.369074+0000) 2019-10-13T10:00:05.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:05.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:05.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:05.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.374385+0000) 2019-10-13T10:00:06.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:06.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.299904+0000) 2019-10-13T10:00:06.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:06.296+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.300715+0000) 2019-10-13T10:00:08.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:08.558+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.561160+0000) 2019-10-13T10:00:08.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:08.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.567601+0000) 2019-10-13T10:00:10.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:10.370+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.372968+0000) 2019-10-13T10:00:10.375 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:10.370+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.373803+0000) 2019-10-13T10:00:10.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:10.374+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.375433+0000) 2019-10-13T10:00:10.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:10.372+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.375837+0000) 2019-10-13T10:00:10.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:10.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.376680+0000) 2019-10-13T10:00:10.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:10.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:10.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:10.374+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.377594+0000) 2019-10-13T10:00:10.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:10.374+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.378008+0000) 2019-10-13T10:00:10.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:10.376+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.379668+0000) 2019-10-13T10:00:11.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:11.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.859547+0000) 2019-10-13T10:00:12.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:12.050+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:12.051925+0000) 2019-10-13T10:00:13.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:13.980+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:13.984821+0000) 2019-10-13T10:00:15.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:15.078+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.078954+0000) 2019-10-13T10:00:15.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:15.378+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.381672+0000) 2019-10-13T10:00:15.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:15.378+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.382128+0000) 2019-10-13T10:00:15.384 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:15.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.382308+0000) 2019-10-13T10:00:15.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:15.378+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.382626+0000) 2019-10-13T10:00:15.384 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:15.378+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.383265+0000) 2019-10-13T10:00:15.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:15.380+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.383696+0000) 2019-10-13T10:00:15.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:15.382+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.384633+0000) 2019-10-13T10:00:15.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:15.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:15.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:15.388+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.390871+0000) 2019-10-13T10:00:16.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:16.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.300081+0000) 2019-10-13T10:00:16.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:16.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.300922+0000) 2019-10-13T10:00:18.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:18.558+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.561369+0000) 2019-10-13T10:00:18.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:18.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.567800+0000) 2019-10-13T10:00:20.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:20.386+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.388224+0000) 2019-10-13T10:00:20.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:20.386+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.388688+0000) 2019-10-13T10:00:20.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:20.386+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.389638+0000) 2019-10-13T10:00:20.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:20.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.390815+0000) 2019-10-13T10:00:20.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:20.388+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.391271+0000) 2019-10-13T10:00:20.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:20.390+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.392106+0000) 2019-10-13T10:00:20.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:20.390+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.392302+0000) 2019-10-13T10:00:20.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:20.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:20.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:20.400+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.402190+0000) 2019-10-13T10:00:21.489 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:00:21.489 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:21.493 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:00:21.493 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:21.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:21.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.859751+0000) 2019-10-13T10:00:22.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:22.049+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:22.052146+0000) 2019-10-13T10:00:23.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:23.988+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:23.991443+0000) 2019-10-13T10:00:25.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:25.077+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.079215+0000) 2019-10-13T10:00:25.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:25.393+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.396578+0000) 2019-10-13T10:00:25.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:25.393+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.397305+0000) 2019-10-13T10:00:25.398 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:25.393+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.397707+0000) 2019-10-13T10:00:25.399 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:25.393+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.398131+0000) 2019-10-13T10:00:25.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:25.393+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.398521+0000) 2019-10-13T10:00:25.400 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:25.396+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.398856+0000) 2019-10-13T10:00:25.400 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:25.397+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.399769+0000) 2019-10-13T10:00:25.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:25.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:25.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:25.404+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.406336+0000) 2019-10-13T10:00:26.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:26.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.300278+0000) 2019-10-13T10:00:26.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:26.296+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.301096+0000) 2019-10-13T10:00:28.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:28.557+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.561574+0000) 2019-10-13T10:00:28.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:28.565+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.568036+0000) 2019-10-13T10:00:30.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:30.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.403647+0000) 2019-10-13T10:00:30.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:30.401+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.404464+0000) 2019-10-13T10:00:30.406 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:30.400+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.405184+0000) 2019-10-13T10:00:30.406 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:30.401+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.405487+0000) 2019-10-13T10:00:30.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:30.401+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.405737+0000) 2019-10-13T10:00:30.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:30.401+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.406368+0000) 2019-10-13T10:00:30.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:30.401+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.407133+0000) 2019-10-13T10:00:30.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:30.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:30.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:30.408+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.413741+0000) 2019-10-13T10:00:31.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:31.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.859963+0000) 2019-10-13T10:00:32.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:32.049+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:32.052346+0000) 2019-10-13T10:00:33.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:33.992+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:33.997590+0000) 2019-10-13T10:00:35.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:35.077+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.079492+0000) 2019-10-13T10:00:35.412 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:35.405+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.411114+0000) 2019-10-13T10:00:35.413 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:35.409+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.411428+0000) 2019-10-13T10:00:35.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:35.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.412348+0000) 2019-10-13T10:00:35.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:35.408+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.413099+0000) 2019-10-13T10:00:35.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:35.409+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.413420+0000) 2019-10-13T10:00:35.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:35.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.413814+0000) 2019-10-13T10:00:35.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:35.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:35.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:35.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.414887+0000) 2019-10-13T10:00:35.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:35.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.418070+0000) 2019-10-13T10:00:36.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:36.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.300486+0000) 2019-10-13T10:00:36.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:36.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.301262+0000) 2019-10-13T10:00:38.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:38.557+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.561771+0000) 2019-10-13T10:00:38.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:38.565+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.568283+0000) 2019-10-13T10:00:40.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:40.413+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.417782+0000) 2019-10-13T10:00:40.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:40.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.418402+0000) 2019-10-13T10:00:40.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:40.417+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.419542+0000) 2019-10-13T10:00:40.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:40.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.420167+0000) 2019-10-13T10:00:40.422 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:40.417+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.420491+0000) 2019-10-13T10:00:40.422 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:40.417+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.420767+0000) 2019-10-13T10:00:40.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:40.416+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.421251+0000) 2019-10-13T10:00:40.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:40.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:40.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:40.420+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.425046+0000) 2019-10-13T10:00:41.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:41.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.860168+0000) 2019-10-13T10:00:42.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:42.049+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:42.052549+0000) 2019-10-13T10:00:44.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:44.000+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.004092+0000) 2019-10-13T10:00:45.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:45.077+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.079658+0000) 2019-10-13T10:00:45.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:45.421+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.425316+0000) 2019-10-13T10:00:45.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:45.421+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.425939+0000) 2019-10-13T10:00:45.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:45.421+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.426638+0000) 2019-10-13T10:00:45.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:45.421+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.426972+0000) 2019-10-13T10:00:45.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:45.424+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.427657+0000) 2019-10-13T10:00:45.429 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:45.425+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.427990+0000) 2019-10-13T10:00:45.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:45.425+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.428576+0000) 2019-10-13T10:00:45.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:45.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:45.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:45.432+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.434790+0000) 2019-10-13T10:00:46.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:46.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.300684+0000) 2019-10-13T10:00:46.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:46.296+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.301440+0000) 2019-10-13T10:00:48.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:48.561+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.561971+0000) 2019-10-13T10:00:48.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:48.565+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.568510+0000) 2019-10-13T10:00:50.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:50.429+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.432775+0000) 2019-10-13T10:00:50.434 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:50.429+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.432313+0000) 2019-10-13T10:00:50.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:50.429+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.434237+0000) 2019-10-13T10:00:50.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:50.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.434637+0000) 2019-10-13T10:00:50.436 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:50.429+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.434844+0000) 2019-10-13T10:00:50.436 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:50.433+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.435341+0000) 2019-10-13T10:00:50.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:50.433+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.435413+0000) 2019-10-13T10:00:50.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:50.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:50.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:50.436+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.440286+0000) 2019-10-13T10:00:51.619 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:00:51.620 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:51.623 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:00:51.624 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:51.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:51.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.860402+0000) 2019-10-13T10:00:52.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:52.049+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:52.052754+0000) 2019-10-13T10:00:54.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:54.008+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.010642+0000) 2019-10-13T10:00:55.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:55.077+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.079882+0000) 2019-10-13T10:00:55.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:00:55.437+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.440225+0000) 2019-10-13T10:00:55.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:55.437+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.440487+0000) 2019-10-13T10:00:55.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:55.437+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.441107+0000) 2019-10-13T10:00:55.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:55.436+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.441872+0000) 2019-10-13T10:00:55.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:55.437+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.442235+0000) 2019-10-13T10:00:55.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:00:55.437+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.442488+0000) 2019-10-13T10:00:55.444 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:00:55.437+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.442990+0000) 2019-10-13T10:00:55.445 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:00:55.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:00:55.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:00:55.444+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.447881+0000) 2019-10-13T10:00:56.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:00:56.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.300895+0000) 2019-10-13T10:00:56.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:00:56.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.301641+0000) 2019-10-13T10:00:58.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:00:58.561+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.562172+0000) 2019-10-13T10:00:58.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:00:58.565+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.568706+0000) 2019-10-13T10:01:00.449 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:00.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.447763+0000) 2019-10-13T10:01:00.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:00.445+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.448340+0000) 2019-10-13T10:01:00.450 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:00.445+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.448956+0000) 2019-10-13T10:01:00.450 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:00.445+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.449003+0000) 2019-10-13T10:01:00.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:00.445+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.449275+0000) 2019-10-13T10:01:00.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:00.448+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.450280+0000) 2019-10-13T10:01:00.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:00.445+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.450510+0000) 2019-10-13T10:01:00.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:00.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:00.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:00.452+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.457055+0000) 2019-10-13T10:01:01.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:01.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.860611+0000) 2019-10-13T10:01:02.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:02.049+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:02.052994+0000) 2019-10-13T10:01:04.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:04.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.017140+0000) 2019-10-13T10:01:05.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:05.076+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.080113+0000) 2019-10-13T10:01:05.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:05.452+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.455814+0000) 2019-10-13T10:01:05.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:05.452+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.456022+0000) 2019-10-13T10:01:05.458 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:05.452+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.456703+0000) 2019-10-13T10:01:05.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:05.452+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.457056+0000) 2019-10-13T10:01:05.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:05.452+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.457923+0000) 2019-10-13T10:01:05.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:05.452+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.458247+0000) 2019-10-13T10:01:05.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:05.455+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.458808+0000) 2019-10-13T10:01:05.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:05.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:05.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:05.459+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.463910+0000) 2019-10-13T10:01:06.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:06.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.301137+0000) 2019-10-13T10:01:06.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:06.299+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.301873+0000) 2019-10-13T10:01:08.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:08.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.562390+0000) 2019-10-13T10:01:08.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:08.564+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.568901+0000) 2019-10-13T10:01:10.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:10.456+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.462606+0000) 2019-10-13T10:01:10.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:10.460+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.463516+0000) 2019-10-13T10:01:10.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:10.459+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.463866+0000) 2019-10-13T10:01:10.465 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:10.460+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.464113+0000) 2019-10-13T10:01:10.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:10.460+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.464505+0000) 2019-10-13T10:01:10.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:10.460+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.464719+0000) 2019-10-13T10:01:10.466 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:10.460+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.465171+0000) 2019-10-13T10:01:10.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:10.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:10.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:10.467+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.472062+0000) 2019-10-13T10:01:11.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:11.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.860815+0000) 2019-10-13T10:01:12.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:12.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:12.053198+0000) 2019-10-13T10:01:14.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:14.019+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.023738+0000) 2019-10-13T10:01:15.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:15.076+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.080311+0000) 2019-10-13T10:01:15.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:15.464+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.469478+0000) 2019-10-13T10:01:15.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:15.467+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.470420+0000) 2019-10-13T10:01:15.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:15.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.470162+0000) 2019-10-13T10:01:15.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:15.464+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.471094+0000) 2019-10-13T10:01:15.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:15.464+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.471138+0000) 2019-10-13T10:01:15.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:15.468+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.471810+0000) 2019-10-13T10:01:15.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:15.468+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.472667+0000) 2019-10-13T10:01:15.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:15.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:15.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:15.475+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.478412+0000) 2019-10-13T10:01:16.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:16.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.301311+0000) 2019-10-13T10:01:16.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:16.299+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.302054+0000) 2019-10-13T10:01:18.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:18.560+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.562626+0000) 2019-10-13T10:01:18.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:18.564+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.569049+0000) 2019-10-13T10:01:20.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:20.471+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.476957+0000) 2019-10-13T10:01:20.478 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:20.472+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.477250+0000) 2019-10-13T10:01:20.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:20.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.477851+0000) 2019-10-13T10:01:20.480 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:20.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.477892+0000) 2019-10-13T10:01:20.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:20.472+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.478295+0000) 2019-10-13T10:01:20.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:20.472+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.479276+0000) 2019-10-13T10:01:20.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:20.476+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.480048+0000) 2019-10-13T10:01:20.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:20.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:20.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:20.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.485227+0000) 2019-10-13T10:01:21.731 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:01:21.731 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:21.735 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:01:21.735 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:21.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:21.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.861019+0000) 2019-10-13T10:01:22.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:22.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:22.053405+0000) 2019-10-13T10:01:24.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:24.027+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.030228+0000) 2019-10-13T10:01:25.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:25.076+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.080554+0000) 2019-10-13T10:01:25.486 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:25.480+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.484819+0000) 2019-10-13T10:01:25.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:25.480+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.485936+0000) 2019-10-13T10:01:25.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:25.483+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.486349+0000) 2019-10-13T10:01:25.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:25.480+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.486933+0000) 2019-10-13T10:01:25.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:25.480+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.487165+0000) 2019-10-13T10:01:25.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:25.484+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.487548+0000) 2019-10-13T10:01:25.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:25.484+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.487959+0000) 2019-10-13T10:01:25.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:25.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:25.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:25.487+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.491763+0000) 2019-10-13T10:01:26.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:26.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.301522+0000) 2019-10-13T10:01:26.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:26.299+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.302268+0000) 2019-10-13T10:01:28.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:28.560+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.562887+0000) 2019-10-13T10:01:28.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:28.568+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.569243+0000) 2019-10-13T10:01:30.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:30.488+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.492004+0000) 2019-10-13T10:01:30.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:30.488+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.492872+0000) 2019-10-13T10:01:30.495 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:30.488+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.493444+0000) 2019-10-13T10:01:30.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:30.488+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.493746+0000) 2019-10-13T10:01:30.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:30.488+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.493893+0000) 2019-10-13T10:01:30.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:30.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.494499+0000) 2019-10-13T10:01:30.496 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:30.488+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.495234+0000) 2019-10-13T10:01:30.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:30.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:30.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:30.495+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.501443+0000) 2019-10-13T10:01:31.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:31.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.861236+0000) 2019-10-13T10:01:32.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:32.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:32.053637+0000) 2019-10-13T10:01:34.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:34.035+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.036623+0000) 2019-10-13T10:01:35.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:35.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.080766+0000) 2019-10-13T10:01:35.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:35.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.499260+0000) 2019-10-13T10:01:35.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:35.495+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.500312+0000) 2019-10-13T10:01:35.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:35.496+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.500521+0000) 2019-10-13T10:01:35.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:35.496+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.500773+0000) 2019-10-13T10:01:35.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:35.496+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.501071+0000) 2019-10-13T10:01:35.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:35.496+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.501498+0000) 2019-10-13T10:01:35.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:35.496+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.502439+0000) 2019-10-13T10:01:35.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:35.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:35.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:35.503+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.507536+0000) 2019-10-13T10:01:36.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:36.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.301721+0000) 2019-10-13T10:01:36.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:36.299+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.302446+0000) 2019-10-13T10:01:38.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:38.556+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.563079+0000) 2019-10-13T10:01:38.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:38.568+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.569438+0000) 2019-10-13T10:01:40.508 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:40.500+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.506893+0000) 2019-10-13T10:01:40.508 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:40.500+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.507109+0000) 2019-10-13T10:01:40.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:40.503+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.507601+0000) 2019-10-13T10:01:40.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:40.504+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.508653+0000) 2019-10-13T10:01:40.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:40.504+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.508707+0000) 2019-10-13T10:01:40.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:40.504+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.509505+0000) 2019-10-13T10:01:40.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:40.504+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.510222+0000) 2019-10-13T10:01:40.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:40.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:40.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:40.511+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.515003+0000) 2019-10-13T10:01:41.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:41.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.861408+0000) 2019-10-13T10:01:42.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:42.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:42.053835+0000) 2019-10-13T10:01:44.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:44.039+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.043104+0000) 2019-10-13T10:01:45.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:45.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.080976+0000) 2019-10-13T10:01:45.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:45.507+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.513335+0000) 2019-10-13T10:01:45.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:45.508+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.513987+0000) 2019-10-13T10:01:45.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:45.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.514664+0000) 2019-10-13T10:01:45.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:45.508+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.515045+0000) 2019-10-13T10:01:45.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:45.512+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.515471+0000) 2019-10-13T10:01:45.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:45.512+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.516554+0000) 2019-10-13T10:01:45.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:45.512+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.516938+0000) 2019-10-13T10:01:45.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:45.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:45.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:45.519+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.523296+0000) 2019-10-13T10:01:46.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:46.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.301898+0000) 2019-10-13T10:01:46.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:46.299+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.302595+0000) 2019-10-13T10:01:48.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:48.559+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.563268+0000) 2019-10-13T10:01:48.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:48.567+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.569667+0000) 2019-10-13T10:01:50.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:50.515+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.521200+0000) 2019-10-13T10:01:50.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:50.515+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.521352+0000) 2019-10-13T10:01:50.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:50.515+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.522091+0000) 2019-10-13T10:01:50.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:50.515+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.522276+0000) 2019-10-13T10:01:50.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:50.515+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.523106+0000) 2019-10-13T10:01:50.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:50.519+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.523634+0000) 2019-10-13T10:01:50.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:50.519+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.524541+0000) 2019-10-13T10:01:50.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:50.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:50.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:50.527+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.530618+0000) 2019-10-13T10:01:51.824 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:01:51.824 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:51.828 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:01:51.828 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:51.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:51.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.861600+0000) 2019-10-13T10:01:52.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:52.051+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:52.054075+0000) 2019-10-13T10:01:54.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:54.047+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.049595+0000) 2019-10-13T10:01:55.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:55.075+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.081132+0000) 2019-10-13T10:01:55.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:01:55.523+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.528937+0000) 2019-10-13T10:01:55.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:55.523+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.529407+0000) 2019-10-13T10:01:55.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:55.523+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.529766+0000) 2019-10-13T10:01:55.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:01:55.523+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.530527+0000) 2019-10-13T10:01:55.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:55.523+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.531234+0000) 2019-10-13T10:01:55.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:55.527+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.531529+0000) 2019-10-13T10:01:55.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:01:55.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.531872+0000) 2019-10-13T10:01:55.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:01:55.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:01:55.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:01:55.535+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.541202+0000) 2019-10-13T10:01:56.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:01:56.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.302099+0000) 2019-10-13T10:01:56.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:01:56.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.302742+0000) 2019-10-13T10:01:58.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:01:58.559+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.563504+0000) 2019-10-13T10:01:58.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:01:58.563+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.569860+0000) 2019-10-13T10:02:00.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:00.531+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.536160+0000) 2019-10-13T10:02:00.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:00.531+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.537259+0000) 2019-10-13T10:02:00.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:00.530+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.538011+0000) 2019-10-13T10:02:00.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:00.531+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.538363+0000) 2019-10-13T10:02:00.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:00.531+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.538870+0000) 2019-10-13T10:02:00.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:00.535+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.539564+0000) 2019-10-13T10:02:00.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:00.535+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.539951+0000) 2019-10-13T10:02:00.542 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:00.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:00.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:00.538+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.543830+0000) 2019-10-13T10:02:01.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:01.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.861813+0000) 2019-10-13T10:02:02.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:02.051+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:02.054339+0000) 2019-10-13T10:02:04.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:04.054+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.056151+0000) 2019-10-13T10:02:05.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:05.079+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.081275+0000) 2019-10-13T10:02:05.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:05.539+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.544074+0000) 2019-10-13T10:02:05.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:05.538+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.544890+0000) 2019-10-13T10:02:05.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:05.539+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.544794+0000) 2019-10-13T10:02:05.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:05.539+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.546291+0000) 2019-10-13T10:02:05.548 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:05.539+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.546577+0000) 2019-10-13T10:02:05.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:05.539+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.547050+0000) 2019-10-13T10:02:05.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:05.543+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.547559+0000) 2019-10-13T10:02:05.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:05.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 89969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:05.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:05.546+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.551914+0000) 2019-10-13T10:02:06.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:06.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.302291+0000) 2019-10-13T10:02:06.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:06.302+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.302920+0000) 2019-10-13T10:02:08.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:08.563+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.563662+0000) 2019-10-13T10:02:08.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:08.567+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.570054+0000) 2019-10-13T10:02:10.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:10.547+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.551361+0000) 2019-10-13T10:02:10.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:10.547+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.553124+0000) 2019-10-13T10:02:10.555 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:10.547+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.553762+0000) 2019-10-13T10:02:10.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:10.547+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.553914+0000) 2019-10-13T10:02:10.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:10.550+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.554505+0000) 2019-10-13T10:02:10.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:10.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.554940+0000) 2019-10-13T10:02:10.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:10.547+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.555061+0000) 2019-10-13T10:02:10.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:10.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:10.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:10.554+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.561914+0000) 2019-10-13T10:02:11.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:11.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.862026+0000) 2019-10-13T10:02:12.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:12.051+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:12.054599+0000) 2019-10-13T10:02:14.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:14.058+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.062672+0000) 2019-10-13T10:02:15.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:15.079+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.081481+0000) 2019-10-13T10:02:15.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:15.555+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.560121+0000) 2019-10-13T10:02:15.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:15.555+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.559783+0000) 2019-10-13T10:02:15.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:15.554+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.561212+0000) 2019-10-13T10:02:15.562 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:15.555+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.560796+0000) 2019-10-13T10:02:15.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:15.555+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.561468+0000) 2019-10-13T10:02:15.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:15.555+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.561804+0000) 2019-10-13T10:02:15.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:15.559+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.563298+0000) 2019-10-13T10:02:15.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:15.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:15.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:15.562+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.569658+0000) 2019-10-13T10:02:16.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:16.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.302466+0000) 2019-10-13T10:02:16.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:16.298+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.303096+0000) 2019-10-13T10:02:18.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:18.563+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.563864+0000) 2019-10-13T10:02:18.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:18.567+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.570248+0000) 2019-10-13T10:02:20.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:20.563+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.567013+0000) 2019-10-13T10:02:20.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:20.567+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.567610+0000) 2019-10-13T10:02:20.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:20.567+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.567894+0000) 2019-10-13T10:02:20.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:20.567+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.568396+0000) 2019-10-13T10:02:20.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:20.562+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.568688+0000) 2019-10-13T10:02:20.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:20.567+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.568945+0000) 2019-10-13T10:02:20.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:20.567+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.570152+0000) 2019-10-13T10:02:20.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:20.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:20.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:20.570+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.575645+0000) 2019-10-13T10:02:21.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:21.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.862248+0000) 2019-10-13T10:02:21.940 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:02:21.940 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:21.943 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:02:21.944 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:22.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:22.051+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:22.054805+0000) 2019-10-13T10:02:24.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:24.066+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.069173+0000) 2019-10-13T10:02:25.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:25.079+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.081701+0000) 2019-10-13T10:02:25.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:25.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.574482+0000) 2019-10-13T10:02:25.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:25.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.575665+0000) 2019-10-13T10:02:25.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:25.575+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.575991+0000) 2019-10-13T10:02:25.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:25.575+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.576667+0000) 2019-10-13T10:02:25.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:25.575+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.577166+0000) 2019-10-13T10:02:25.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:25.575+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.578368+0000) 2019-10-13T10:02:25.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:25.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:25.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:25.575+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.579190+0000) 2019-10-13T10:02:25.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:25.578+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.583972+0000) 2019-10-13T10:02:26.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:26.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.302636+0000) 2019-10-13T10:02:26.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:26.302+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.303242+0000) 2019-10-13T10:02:28.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:28.563+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.564041+0000) 2019-10-13T10:02:28.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:28.567+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.570400+0000) 2019-10-13T10:02:30.584 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:30.579+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.582520+0000) 2019-10-13T10:02:30.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:30.583+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.583482+0000) 2019-10-13T10:02:30.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:30.583+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.583809+0000) 2019-10-13T10:02:30.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:30.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.582961+0000) 2019-10-13T10:02:30.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:30.583+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.584152+0000) 2019-10-13T10:02:30.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:30.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.584866+0000) 2019-10-13T10:02:30.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:30.583+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.585191+0000) 2019-10-13T10:02:30.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:30.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:30.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:30.586+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.593833+0000) 2019-10-13T10:02:31.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:31.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.862506+0000) 2019-10-13T10:02:32.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:32.050+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:32.055020+0000) 2019-10-13T10:02:34.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:34.074+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.075795+0000) 2019-10-13T10:02:35.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:35.078+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.081905+0000) 2019-10-13T10:02:35.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:35.586+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.589476+0000) 2019-10-13T10:02:35.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:35.582+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.589837+0000) 2019-10-13T10:02:35.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:35.586+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.590264+0000) 2019-10-13T10:02:35.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:35.586+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.590779+0000) 2019-10-13T10:02:35.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:35.590+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.591768+0000) 2019-10-13T10:02:35.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:35.590+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.591860+0000) 2019-10-13T10:02:35.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:35.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.592464+0000) 2019-10-13T10:02:35.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:35.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:35.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:35.590+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.597180+0000) 2019-10-13T10:02:36.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:36.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.302846+0000) 2019-10-13T10:02:36.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:36.302+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.303391+0000) 2019-10-13T10:02:38.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:38.562+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.564206+0000) 2019-10-13T10:02:38.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:38.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.570632+0000) 2019-10-13T10:02:40.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:40.594+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.596301+0000) 2019-10-13T10:02:40.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:40.594+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.596665+0000) 2019-10-13T10:02:40.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:40.590+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.597972+0000) 2019-10-13T10:02:40.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:40.594+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.598793+0000) 2019-10-13T10:02:40.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:40.594+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.599154+0000) 2019-10-13T10:02:40.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:40.598+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.599499+0000) 2019-10-13T10:02:40.601 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:40.598+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.600102+0000) 2019-10-13T10:02:40.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:40.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:40.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:40.598+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.604112+0000) 2019-10-13T10:02:41.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:41.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.862738+0000) 2019-10-13T10:02:42.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:42.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:42.055257+0000) 2019-10-13T10:02:44.083 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:44.082+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.082323+0000) 2019-10-13T10:02:45.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:45.078+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.082099+0000) 2019-10-13T10:02:45.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:45.602+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.603701+0000) 2019-10-13T10:02:45.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:45.602+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.604334+0000) 2019-10-13T10:02:45.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:45.602+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.605196+0000) 2019-10-13T10:02:45.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:45.602+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.605611+0000) 2019-10-13T10:02:45.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:45.606+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.606358+0000) 2019-10-13T10:02:45.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:45.602+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.606886+0000) 2019-10-13T10:02:45.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:45.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.607128+0000) 2019-10-13T10:02:45.609 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:45.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:45.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:45.610+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.612797+0000) 2019-10-13T10:02:46.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:46.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.303020+0000) 2019-10-13T10:02:46.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:46.302+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.303557+0000) 2019-10-13T10:02:48.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:48.562+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.564452+0000) 2019-10-13T10:02:48.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:48.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.570842+0000) 2019-10-13T10:02:50.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:50.610+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.611400+0000) 2019-10-13T10:02:50.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:50.610+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.612302+0000) 2019-10-13T10:02:50.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:50.610+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.613079+0000) 2019-10-13T10:02:50.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:50.610+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.612779+0000) 2019-10-13T10:02:50.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:50.610+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.613545+0000) 2019-10-13T10:02:50.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:50.610+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.614094+0000) 2019-10-13T10:02:50.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:50.610+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.615222+0000) 2019-10-13T10:02:50.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:50.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:50.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:50.622+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.624689+0000) 2019-10-13T10:02:51.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:51.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.862958+0000) 2019-10-13T10:02:52.055 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:02:52.055 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:52.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:52.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:52.055531+0000) 2019-10-13T10:02:52.059 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:02:52.059 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:54.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:54.085+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.088775+0000) 2019-10-13T10:02:55.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:55.078+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.082294+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:55.618+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.620640+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:02:55.618+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.620902+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:55.618+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.621323+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:55.618+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.622414+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:02:55.618+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.622960+0000) 2019-10-13T10:02:55.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:02:55.622+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.623377+0000) 2019-10-13T10:02:55.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:02:55.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:02:55.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:55.617+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.620288+0000) 2019-10-13T10:02:55.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:02:55.625+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.628990+0000) 2019-10-13T10:02:56.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:02:56.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.303220+0000) 2019-10-13T10:02:56.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:02:56.301+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.303731+0000) 2019-10-13T10:02:58.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:02:58.562+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.564652+0000) 2019-10-13T10:02:58.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:02:58.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.571042+0000) 2019-10-13T10:03:00.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:00.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.628101+0000) 2019-10-13T10:03:00.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:00.626+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.628350+0000) 2019-10-13T10:03:00.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:00.626+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.628686+0000) 2019-10-13T10:03:00.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:00.625+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.629307+0000) 2019-10-13T10:03:00.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:00.626+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.630958+0000) 2019-10-13T10:03:00.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:00.626+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.631276+0000) 2019-10-13T10:03:00.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:00.630+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.632431+0000) 2019-10-13T10:03:00.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:00.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:00.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:00.633+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.636338+0000) 2019-10-13T10:03:01.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:01.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.863164+0000) 2019-10-13T10:03:02.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:02.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:02.055737+0000) 2019-10-13T10:03:04.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:04.093+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.095260+0000) 2019-10-13T10:03:05.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:05.082+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.082496+0000) 2019-10-13T10:03:05.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:05.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.636792+0000) 2019-10-13T10:03:05.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:05.634+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.637059+0000) 2019-10-13T10:03:05.639 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:05.633+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.637898+0000) 2019-10-13T10:03:05.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:05.634+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.637432+0000) 2019-10-13T10:03:05.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:05.634+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.638249+0000) 2019-10-13T10:03:05.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:05.634+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.639253+0000) 2019-10-13T10:03:05.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:05.638+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.640371+0000) 2019-10-13T10:03:05.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:05.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:05.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:05.641+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.646098+0000) 2019-10-13T10:03:06.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:06.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.303392+0000) 2019-10-13T10:03:06.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:06.301+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.303902+0000) 2019-10-13T10:03:08.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:08.562+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.564867+0000) 2019-10-13T10:03:08.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:08.566+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.571245+0000) 2019-10-13T10:03:10.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:10.642+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.644088+0000) 2019-10-13T10:03:10.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:10.642+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.644749+0000) 2019-10-13T10:03:10.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:10.642+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.645302+0000) 2019-10-13T10:03:10.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:10.641+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.645562+0000) 2019-10-13T10:03:10.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:10.642+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.645830+0000) 2019-10-13T10:03:10.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:10.642+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.646299+0000) 2019-10-13T10:03:10.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:10.642+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.646646+0000) 2019-10-13T10:03:10.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:10.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:10.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:10.649+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.653480+0000) 2019-10-13T10:03:11.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:11.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.863413+0000) 2019-10-13T10:03:12.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:12.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:12.055938+0000) 2019-10-13T10:03:14.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:14.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.101840+0000) 2019-10-13T10:03:15.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:15.078+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.082697+0000) 2019-10-13T10:03:15.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:15.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.651122+0000) 2019-10-13T10:03:15.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:15.649+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.651419+0000) 2019-10-13T10:03:15.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:15.649+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.652381+0000) 2019-10-13T10:03:15.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:15.649+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.653241+0000) 2019-10-13T10:03:15.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:15.649+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.653766+0000) 2019-10-13T10:03:15.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:15.649+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.654292+0000) 2019-10-13T10:03:15.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:15.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.654696+0000) 2019-10-13T10:03:15.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:15.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:15.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:15.657+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.658555+0000) 2019-10-13T10:03:16.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:16.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.303567+0000) 2019-10-13T10:03:16.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:16.301+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.304073+0000) 2019-10-13T10:03:18.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:18.561+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.565095+0000) 2019-10-13T10:03:18.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:18.569+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.571500+0000) 2019-10-13T10:03:20.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:20.653+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.658297+0000) 2019-10-13T10:03:20.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:20.653+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.658921+0000) 2019-10-13T10:03:20.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:20.657+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.660180+0000) 2019-10-13T10:03:20.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:20.657+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.660135+0000) 2019-10-13T10:03:20.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:20.657+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.660615+0000) 2019-10-13T10:03:20.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:20.657+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.661532+0000) 2019-10-13T10:03:20.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:20.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.661729+0000) 2019-10-13T10:03:20.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:20.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:20.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:20.665+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.669115+0000) 2019-10-13T10:03:21.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:21.861+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.863659+0000) 2019-10-13T10:03:22.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:22.053+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:22.056139+0000) 2019-10-13T10:03:22.185 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:03:22.185 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:22.189 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:03:22.189 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:24.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:24.105+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.108427+0000) 2019-10-13T10:03:25.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:25.081+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.082903+0000) 2019-10-13T10:03:25.668 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:25.665+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.667443+0000) 2019-10-13T10:03:25.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:25.665+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.668316+0000) 2019-10-13T10:03:25.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:25.665+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.668728+0000) 2019-10-13T10:03:25.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:25.665+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.669520+0000) 2019-10-13T10:03:25.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:25.665+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.669962+0000) 2019-10-13T10:03:25.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:25.669+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.670422+0000) 2019-10-13T10:03:25.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:25.665+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.670824+0000) 2019-10-13T10:03:25.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:25.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:25.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:25.673+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.675518+0000) 2019-10-13T10:03:26.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:26.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.303740+0000) 2019-10-13T10:03:26.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:26.301+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.304254+0000) 2019-10-13T10:03:28.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:28.561+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:28.565318+0000) 2019-10-13T10:03:28.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:28.569+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:28.571727+0000) 2019-10-13T10:03:30.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:30.669+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.674780+0000) 2019-10-13T10:03:30.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:30.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.675127+0000) 2019-10-13T10:03:30.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:30.673+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.675668+0000) 2019-10-13T10:03:30.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:30.673+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.676259+0000) 2019-10-13T10:03:30.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:30.673+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.676422+0000) 2019-10-13T10:03:30.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:30.673+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.677391+0000) 2019-10-13T10:03:30.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:30.673+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.677851+0000) 2019-10-13T10:03:30.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:30.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:30.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:30.681+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.685075+0000) 2019-10-13T10:03:31.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:31.861+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.863876+0000) 2019-10-13T10:03:32.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:32.053+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:32.056340+0000) 2019-10-13T10:03:34.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:34.113+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.114838+0000) 2019-10-13T10:03:35.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:35.077+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.083106+0000) 2019-10-13T10:03:35.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:35.677+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.682207+0000) 2019-10-13T10:03:35.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:35.681+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.683868+0000) 2019-10-13T10:03:35.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:35.681+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.684491+0000) 2019-10-13T10:03:35.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:35.681+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.684774+0000) 2019-10-13T10:03:35.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:35.681+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.685639+0000) 2019-10-13T10:03:35.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:35.681+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.686161+0000) 2019-10-13T10:03:35.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:35.685+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.686917+0000) 2019-10-13T10:03:35.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:35.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:35.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:35.689+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.690983+0000) 2019-10-13T10:03:36.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:36.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.303870+0000) 2019-10-13T10:03:36.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:36.301+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.304424+0000) 2019-10-13T10:03:38.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:38.565+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:38.565550+0000) 2019-10-13T10:03:38.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:38.569+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:38.571889+0000) 2019-10-13T10:03:40.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:40.685+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.689909+0000) 2019-10-13T10:03:40.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:40.685+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.691255+0000) 2019-10-13T10:03:40.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:40.689+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.691513+0000) 2019-10-13T10:03:40.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:40.689+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.692757+0000) 2019-10-13T10:03:40.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:40.689+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.692273+0000) 2019-10-13T10:03:40.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:40.689+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.693051+0000) 2019-10-13T10:03:40.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:40.689+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.693828+0000) 2019-10-13T10:03:40.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:40.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:40.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:40.697+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.700970+0000) 2019-10-13T10:03:41.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:41.861+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.864085+0000) 2019-10-13T10:03:42.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:42.053+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:42.056548+0000) 2019-10-13T10:03:44.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:44.117+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.121370+0000) 2019-10-13T10:03:45.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:45.081+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.083334+0000) 2019-10-13T10:03:45.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:45.693+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.699225+0000) 2019-10-13T10:03:45.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:45.697+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.699498+0000) 2019-10-13T10:03:45.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:45.697+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.699868+0000) 2019-10-13T10:03:45.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:45.697+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.700563+0000) 2019-10-13T10:03:45.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:45.697+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.700920+0000) 2019-10-13T10:03:45.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:45.697+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.701521+0000) 2019-10-13T10:03:45.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:45.697+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.701899+0000) 2019-10-13T10:03:45.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:45.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:45.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:45.701+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.705246+0000) 2019-10-13T10:03:46.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:46.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.304042+0000) 2019-10-13T10:03:46.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:46.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.304614+0000) 2019-10-13T10:03:48.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:48.565+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:48.565779+0000) 2019-10-13T10:03:48.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:48.569+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:48.572065+0000) 2019-10-13T10:03:50.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:50.700+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.705760+0000) 2019-10-13T10:03:50.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:50.701+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.706692+0000) 2019-10-13T10:03:50.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:50.701+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.707189+0000) 2019-10-13T10:03:50.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:50.705+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.707620+0000) 2019-10-13T10:03:50.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:50.705+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.708154+0000) 2019-10-13T10:03:50.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:50.705+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.708601+0000) 2019-10-13T10:03:50.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:50.705+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.708946+0000) 2019-10-13T10:03:50.710 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:50.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:50.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:50.708+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.713518+0000) 2019-10-13T10:03:51.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:51.861+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.864306+0000) 2019-10-13T10:03:52.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:52.053+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:52.056747+0000) 2019-10-13T10:03:52.295 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:03:52.295 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:52.298 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:03:52.299 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:54.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:54.124+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.127894+0000) 2019-10-13T10:03:55.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:55.081+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.083528+0000) 2019-10-13T10:03:55.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:55.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.714034+0000) 2019-10-13T10:03:55.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:03:55.709+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.714488+0000) 2019-10-13T10:03:55.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:55.709+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.714563+0000) 2019-10-13T10:03:55.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:03:55.709+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.714830+0000) 2019-10-13T10:03:55.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:03:55.709+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.715166+0000) 2019-10-13T10:03:55.717 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:55.713+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.715669+0000) 2019-10-13T10:03:55.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:55.713+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.716011+0000) 2019-10-13T10:03:55.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:03:55.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:03:55.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:03:55.716+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.719121+0000) 2019-10-13T10:03:56.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:03:56.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.304219+0000) 2019-10-13T10:03:56.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:03:56.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.304788+0000) 2019-10-13T10:03:58.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:03:58.561+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:58.565974+0000) 2019-10-13T10:03:58.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:03:58.569+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:58.572287+0000) 2019-10-13T10:04:00.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:00.716+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.720002+0000) 2019-10-13T10:04:00.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:00.716+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.720630+0000) 2019-10-13T10:04:00.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:00.716+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.721019+0000) 2019-10-13T10:04:00.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:00.716+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.722540+0000) 2019-10-13T10:04:00.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:00.716+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.722997+0000) 2019-10-13T10:04:00.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:00.716+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.723191+0000) 2019-10-13T10:04:00.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:00.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.723823+0000) 2019-10-13T10:04:00.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:00.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:00.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:00.724+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.728472+0000) 2019-10-13T10:04:01.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:01.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.864509+0000) 2019-10-13T10:04:02.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:02.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:02.056948+0000) 2019-10-13T10:04:04.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:04.132+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.134411+0000) 2019-10-13T10:04:05.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:05.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.083744+0000) 2019-10-13T10:04:05.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:05.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.729133+0000) 2019-10-13T10:04:05.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:05.724+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.729345+0000) 2019-10-13T10:04:05.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:05.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.729598+0000) 2019-10-13T10:04:05.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:05.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.729894+0000) 2019-10-13T10:04:05.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:05.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.730457+0000) 2019-10-13T10:04:05.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:05.724+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.730771+0000) 2019-10-13T10:04:05.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:05.724+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.731007+0000) 2019-10-13T10:04:05.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:05.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:05.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:05.732+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.734710+0000) 2019-10-13T10:04:06.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:06.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.304392+0000) 2019-10-13T10:04:06.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:06.300+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.304981+0000) 2019-10-13T10:04:08.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:08.560+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:08.566174+0000) 2019-10-13T10:04:08.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:08.568+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:08.572480+0000) 2019-10-13T10:04:10.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:10.732+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.735760+0000) 2019-10-13T10:04:10.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:10.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.736116+0000) 2019-10-13T10:04:10.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:10.732+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.737014+0000) 2019-10-13T10:04:10.739 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:10.732+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.737510+0000) 2019-10-13T10:04:10.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:10.732+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.737889+0000) 2019-10-13T10:04:10.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:10.732+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.738074+0000) 2019-10-13T10:04:10.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:10.732+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.738580+0000) 2019-10-13T10:04:10.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:10.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:10.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:10.740+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.745525+0000) 2019-10-13T10:04:11.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:11.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.864725+0000) 2019-10-13T10:04:12.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:12.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:12.057150+0000) 2019-10-13T10:04:14.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:14.140+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.141420+0000) 2019-10-13T10:04:15.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:15.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.083943+0000) 2019-10-13T10:04:15.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:15.736+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.743008+0000) 2019-10-13T10:04:15.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:15.740+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.743645+0000) 2019-10-13T10:04:15.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:15.740+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.744152+0000) 2019-10-13T10:04:15.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:15.740+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.744331+0000) 2019-10-13T10:04:15.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:15.740+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.744678+0000) 2019-10-13T10:04:15.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:15.740+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.745578+0000) 2019-10-13T10:04:15.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:15.740+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.746058+0000) 2019-10-13T10:04:15.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:15.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:15.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:15.748+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.751389+0000) 2019-10-13T10:04:16.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:16.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.304571+0000) 2019-10-13T10:04:16.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:16.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.305186+0000) 2019-10-13T10:04:18.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:18.560+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:18.566379+0000) 2019-10-13T10:04:18.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:18.568+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:18.572701+0000) 2019-10-13T10:04:20.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:20.744+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.749814+0000) 2019-10-13T10:04:20.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:20.748+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.751638+0000) 2019-10-13T10:04:20.753 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:20.748+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.752067+0000) 2019-10-13T10:04:20.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:20.748+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.752276+0000) 2019-10-13T10:04:20.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:20.748+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.753200+0000) 2019-10-13T10:04:20.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:20.748+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.754071+0000) 2019-10-13T10:04:20.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:20.748+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:20.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:20.752+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.754682+0000) 2019-10-13T10:04:20.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:20.752+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.757612+0000) 2019-10-13T10:04:21.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:21.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.864939+0000) 2019-10-13T10:04:22.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:22.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:22.057349+0000) 2019-10-13T10:04:22.383 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:04:22.383 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:22.387 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:04:22.387 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:24.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:24.148+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.148478+0000) 2019-10-13T10:04:25.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:25.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.084165+0000) 2019-10-13T10:04:25.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:25.752+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.758377+0000) 2019-10-13T10:04:25.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:25.752+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.758467+0000) 2019-10-13T10:04:25.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:25.756+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.759669+0000) 2019-10-13T10:04:25.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:25.756+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.760172+0000) 2019-10-13T10:04:25.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:25.756+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.760490+0000) 2019-10-13T10:04:25.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:25.756+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.760952+0000) 2019-10-13T10:04:25.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:25.756+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.761300+0000) 2019-10-13T10:04:25.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:25.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:25.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:25.764+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.766361+0000) 2019-10-13T10:04:26.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:26.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.305375+0000) 2019-10-13T10:04:26.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:26.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.304780+0000) 2019-10-13T10:04:28.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:28.564+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:28.566581+0000) 2019-10-13T10:04:28.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:28.572+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:28.572896+0000) 2019-10-13T10:04:30.767 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:30.760+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.765783+0000) 2019-10-13T10:04:30.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:30.760+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.767027+0000) 2019-10-13T10:04:30.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:30.760+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.766656+0000) 2019-10-13T10:04:30.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:30.764+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.767840+0000) 2019-10-13T10:04:30.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:30.764+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.768567+0000) 2019-10-13T10:04:30.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:30.764+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.768748+0000) 2019-10-13T10:04:30.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:30.764+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.769057+0000) 2019-10-13T10:04:30.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:30.764+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:30.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:30.772+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.775497+0000) 2019-10-13T10:04:31.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:31.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.865174+0000) 2019-10-13T10:04:32.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:32.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:32.057554+0000) 2019-10-13T10:04:34.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:34.152+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.155588+0000) 2019-10-13T10:04:35.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:35.084+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.084394+0000) 2019-10-13T10:04:35.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:35.768+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.774102+0000) 2019-10-13T10:04:35.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:35.768+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.774447+0000) 2019-10-13T10:04:35.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:35.768+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.774831+0000) 2019-10-13T10:04:35.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:35.768+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.775169+0000) 2019-10-13T10:04:35.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:35.772+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.776608+0000) 2019-10-13T10:04:35.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:35.772+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.776837+0000) 2019-10-13T10:04:35.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:35.772+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.777732+0000) 2019-10-13T10:04:35.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:35.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 90972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:35.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:35.776+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.781775+0000) 2019-10-13T10:04:36.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:36.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.305007+0000) 2019-10-13T10:04:36.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:36.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.305582+0000) 2019-10-13T10:04:38.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:38.564+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:38.566784+0000) 2019-10-13T10:04:38.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:38.572+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:38.573103+0000) 2019-10-13T10:04:40.783 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:40.776+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.781798+0000) 2019-10-13T10:04:40.784 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:40.780+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.782261+0000) 2019-10-13T10:04:40.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:40.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.782686+0000) 2019-10-13T10:04:40.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:40.780+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.783679+0000) 2019-10-13T10:04:40.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:40.780+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.784701+0000) 2019-10-13T10:04:40.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:40.780+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.785152+0000) 2019-10-13T10:04:40.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:40.780+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.785447+0000) 2019-10-13T10:04:40.787 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:40.780+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:40.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:40.788+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.791915+0000) 2019-10-13T10:04:41.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:41.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.865445+0000) 2019-10-13T10:04:42.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:42.052+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:42.057749+0000) 2019-10-13T10:04:44.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:44.159+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.162653+0000) 2019-10-13T10:04:45.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:45.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.084629+0000) 2019-10-13T10:04:45.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:45.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.789311+0000) 2019-10-13T10:04:45.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:45.783+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.789759+0000) 2019-10-13T10:04:45.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:45.787+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.790337+0000) 2019-10-13T10:04:45.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:45.783+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.790526+0000) 2019-10-13T10:04:45.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:45.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.791028+0000) 2019-10-13T10:04:45.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:45.787+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.791824+0000) 2019-10-13T10:04:45.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:45.787+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.792422+0000) 2019-10-13T10:04:45.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:45.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:45.797 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:45.791+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.796577+0000) 2019-10-13T10:04:46.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:46.303+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.305196+0000) 2019-10-13T10:04:46.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:46.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.305712+0000) 2019-10-13T10:04:48.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:48.563+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:48.566987+0000) 2019-10-13T10:04:48.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:48.571+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:48.573313+0000) 2019-10-13T10:04:50.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:50.791+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.796602+0000) 2019-10-13T10:04:50.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:50.791+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.797262+0000) 2019-10-13T10:04:50.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:50.791+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.798195+0000) 2019-10-13T10:04:50.799 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:50.795+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.798668+0000) 2019-10-13T10:04:50.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:50.795+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.799295+0000) 2019-10-13T10:04:50.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:50.795+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.799848+0000) 2019-10-13T10:04:50.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:50.795+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.800208+0000) 2019-10-13T10:04:50.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:50.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:50.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:50.803+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.808480+0000) 2019-10-13T10:04:51.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:51.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.865651+0000) 2019-10-13T10:04:52.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:52.051+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:52.057952+0000) 2019-10-13T10:04:52.493 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:04:52.494 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:52.498 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:04:52.498 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:54.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:54.167+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.169819+0000) 2019-10-13T10:04:55.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:55.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.084817+0000) 2019-10-13T10:04:55.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:04:55.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.804132+0000) 2019-10-13T10:04:55.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:55.799+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.804479+0000) 2019-10-13T10:04:55.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:04:55.799+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.804958+0000) 2019-10-13T10:04:55.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:55.799+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.806138+0000) 2019-10-13T10:04:55.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:55.799+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.806653+0000) 2019-10-13T10:04:55.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:55.799+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.807015+0000) 2019-10-13T10:04:55.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:04:55.803+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.807373+0000) 2019-10-13T10:04:55.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:04:55.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:04:55.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:04:55.811+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.815523+0000) 2019-10-13T10:04:56.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:04:56.303+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.305348+0000) 2019-10-13T10:04:56.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:04:56.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.305890+0000) 2019-10-13T10:04:58.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:04:58.559+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:58.567186+0000) 2019-10-13T10:04:58.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:04:58.571+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:58.573538+0000) 2019-10-13T10:05:00.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:00.807+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.811925+0000) 2019-10-13T10:05:00.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:00.807+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.812853+0000) 2019-10-13T10:05:00.814 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:00.807+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.813242+0000) 2019-10-13T10:05:00.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:00.807+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.813720+0000) 2019-10-13T10:05:00.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:00.807+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.814186+0000) 2019-10-13T10:05:00.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:00.807+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.814693+0000) 2019-10-13T10:05:00.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:00.811+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.815403+0000) 2019-10-13T10:05:00.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:00.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:00.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:00.819+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.822420+0000) 2019-10-13T10:05:01.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:01.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.865856+0000) 2019-10-13T10:05:02.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:02.055+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:02.058158+0000) 2019-10-13T10:05:04.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:04.175+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.176899+0000) 2019-10-13T10:05:05.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:05.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.085025+0000) 2019-10-13T10:05:05.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:05.815+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.819880+0000) 2019-10-13T10:05:05.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:05.815+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.820266+0000) 2019-10-13T10:05:05.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:05.815+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.821538+0000) 2019-10-13T10:05:05.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:05.815+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.821057+0000) 2019-10-13T10:05:05.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:05.815+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.822100+0000) 2019-10-13T10:05:05.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:05.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.822683+0000) 2019-10-13T10:05:05.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:05.815+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.823189+0000) 2019-10-13T10:05:05.825 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:05.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:05.833 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:05.827+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.832191+0000) 2019-10-13T10:05:06.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:06.303+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.305499+0000) 2019-10-13T10:05:06.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:06.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.306061+0000) 2019-10-13T10:05:08.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:08.563+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:08.567411+0000) 2019-10-13T10:05:08.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:08.571+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:08.573733+0000) 2019-10-13T10:05:10.830 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:10.823+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.828436+0000) 2019-10-13T10:05:10.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:10.823+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.828715+0000) 2019-10-13T10:05:10.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:10.823+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.829559+0000) 2019-10-13T10:05:10.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:10.823+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.829839+0000) 2019-10-13T10:05:10.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:10.823+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.830086+0000) 2019-10-13T10:05:10.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:10.823+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.830560+0000) 2019-10-13T10:05:10.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:10.827+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.831407+0000) 2019-10-13T10:05:10.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:10.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:10.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:10.831+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.834641+0000) 2019-10-13T10:05:11.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:11.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.866077+0000) 2019-10-13T10:05:12.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:12.055+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:12.058392+0000) 2019-10-13T10:05:14.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:14.179+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.184061+0000) 2019-10-13T10:05:15.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:15.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.085223+0000) 2019-10-13T10:05:15.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:15.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.834788+0000) 2019-10-13T10:05:15.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:15.835+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.835430+0000) 2019-10-13T10:05:15.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:15.835+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.836432+0000) 2019-10-13T10:05:15.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:15.835+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.837040+0000) 2019-10-13T10:05:15.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:15.835+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.837656+0000) 2019-10-13T10:05:15.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:15.835+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.837787+0000) 2019-10-13T10:05:15.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:15.835+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.838835+0000) 2019-10-13T10:05:15.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:15.835+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:15.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:15.835+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.841517+0000) 2019-10-13T10:05:16.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:16.303+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.305648+0000) 2019-10-13T10:05:16.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:16.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.306250+0000) 2019-10-13T10:05:18.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:18.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:18.567621+0000) 2019-10-13T10:05:18.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:18.571+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:18.573935+0000) 2019-10-13T10:05:20.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:20.839+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.842464+0000) 2019-10-13T10:05:20.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:20.839+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.843090+0000) 2019-10-13T10:05:20.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:20.843+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.843523+0000) 2019-10-13T10:05:20.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:20.839+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.845019+0000) 2019-10-13T10:05:20.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:20.843+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.845212+0000) 2019-10-13T10:05:20.847 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:20.843+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.845568+0000) 2019-10-13T10:05:20.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:20.843+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.846006+0000) 2019-10-13T10:05:20.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:20.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:20.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:20.847+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.850537+0000) 2019-10-13T10:05:21.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:21.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.866331+0000) 2019-10-13T10:05:22.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:22.055+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:22.058596+0000) 2019-10-13T10:05:22.613 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:05:22.613 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:22.617 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:05:22.617 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:24.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:24.187+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.191159+0000) 2019-10-13T10:05:25.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:25.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.085416+0000) 2019-10-13T10:05:25.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:25.847+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.850540+0000) 2019-10-13T10:05:25.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:25.851+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.851612+0000) 2019-10-13T10:05:25.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:25.851+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.851961+0000) 2019-10-13T10:05:25.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:25.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.852550+0000) 2019-10-13T10:05:25.854 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:25.851+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.852301+0000) 2019-10-13T10:05:25.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:25.851+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.852825+0000) 2019-10-13T10:05:25.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:25.851+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.853933+0000) 2019-10-13T10:05:25.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:25.851+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:25.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:25.851+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.857519+0000) 2019-10-13T10:05:26.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:26.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.305846+0000) 2019-10-13T10:05:26.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:26.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.306439+0000) 2019-10-13T10:05:28.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:28.566+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:28.567863+0000) 2019-10-13T10:05:28.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:28.570+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:28.574155+0000) 2019-10-13T10:05:30.859 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:30.854+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.857854+0000) 2019-10-13T10:05:30.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:30.854+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.858579+0000) 2019-10-13T10:05:30.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:30.858+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.859553+0000) 2019-10-13T10:05:30.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:30.855+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.860132+0000) 2019-10-13T10:05:30.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:30.858+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.860498+0000) 2019-10-13T10:05:30.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:30.858+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.860934+0000) 2019-10-13T10:05:30.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:30.858+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.860981+0000) 2019-10-13T10:05:30.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:30.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:30.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:30.859+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.865437+0000) 2019-10-13T10:05:31.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:31.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.866535+0000) 2019-10-13T10:05:32.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:32.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:32.058797+0000) 2019-10-13T10:05:34.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:34.195+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.198379+0000) 2019-10-13T10:05:35.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:35.082+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.085617+0000) 2019-10-13T10:05:35.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:35.862+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.866759+0000) 2019-10-13T10:05:35.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:35.862+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.867177+0000) 2019-10-13T10:05:35.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:35.866+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.867641+0000) 2019-10-13T10:05:35.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:35.866+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.868340+0000) 2019-10-13T10:05:35.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:35.863+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.868517+0000) 2019-10-13T10:05:35.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:35.866+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.868814+0000) 2019-10-13T10:05:35.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:35.866+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.869636+0000) 2019-10-13T10:05:35.882 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:35.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:35.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:35.867+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.872799+0000) 2019-10-13T10:05:36.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:36.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.305997+0000) 2019-10-13T10:05:36.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:36.303+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.306610+0000) 2019-10-13T10:05:38.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:38.566+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:38.568055+0000) 2019-10-13T10:05:38.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:38.570+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:38.574299+0000) 2019-10-13T10:05:40.874 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:40.870+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.873280+0000) 2019-10-13T10:05:40.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:40.870+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.874277+0000) 2019-10-13T10:05:40.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:40.870+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.874664+0000) 2019-10-13T10:05:40.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:40.874+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.875448+0000) 2019-10-13T10:05:40.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:40.874+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.876020+0000) 2019-10-13T10:05:40.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:40.874+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.876280+0000) 2019-10-13T10:05:40.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:40.874+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.876642+0000) 2019-10-13T10:05:40.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:40.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:40.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:40.874+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.881390+0000) 2019-10-13T10:05:41.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:41.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.866731+0000) 2019-10-13T10:05:42.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:42.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:42.059033+0000) 2019-10-13T10:05:44.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:44.202+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.209808+0000) 2019-10-13T10:05:45.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:45.082+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.085759+0000) 2019-10-13T10:05:45.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:45.878+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.881303+0000) 2019-10-13T10:05:45.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:45.878+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.883627+0000) 2019-10-13T10:05:45.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:45.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.882022+0000) 2019-10-13T10:05:45.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:45.878+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.882189+0000) 2019-10-13T10:05:45.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:45.878+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.882740+0000) 2019-10-13T10:05:45.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:45.882+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.884605+0000) 2019-10-13T10:05:45.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:45.882+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.885150+0000) 2019-10-13T10:05:45.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:45.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:45.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:45.882+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.889714+0000) 2019-10-13T10:05:46.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:46.302+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.306785+0000) 2019-10-13T10:05:46.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:46.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.306180+0000) 2019-10-13T10:05:48.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:48.566+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:48.568263+0000) 2019-10-13T10:05:48.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:48.570+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:48.574489+0000) 2019-10-13T10:05:50.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:50.886+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.888961+0000) 2019-10-13T10:05:50.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:50.886+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.889608+0000) 2019-10-13T10:05:50.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:50.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.890037+0000) 2019-10-13T10:05:50.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:50.886+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.890263+0000) 2019-10-13T10:05:50.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:50.886+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.891121+0000) 2019-10-13T10:05:50.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:50.890+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.891924+0000) 2019-10-13T10:05:50.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:50.890+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.892549+0000) 2019-10-13T10:05:50.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:50.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:50.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:50.890+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.896891+0000) 2019-10-13T10:05:51.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:51.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.866917+0000) 2019-10-13T10:05:52.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:52.054+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:52.059244+0000) 2019-10-13T10:05:52.907 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:05:52.907 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:52.911 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:05:52.911 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:54.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:54.214+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.216490+0000) 2019-10-13T10:05:55.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:55.082+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.085949+0000) 2019-10-13T10:05:55.897 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:55.894+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.895888+0000) 2019-10-13T10:05:55.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:05:55.894+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.896764+0000) 2019-10-13T10:05:55.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:55.890+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.897123+0000) 2019-10-13T10:05:55.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:55.894+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.897815+0000) 2019-10-13T10:05:55.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:05:55.894+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.898268+0000) 2019-10-13T10:05:55.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:05:55.894+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.898770+0000) 2019-10-13T10:05:55.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:55.898+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.899567+0000) 2019-10-13T10:05:55.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:05:55.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:05:55.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:05:55.902+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.906472+0000) 2019-10-13T10:05:56.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:05:56.306+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.306360+0000) 2019-10-13T10:05:56.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:05:56.306+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.306958+0000) 2019-10-13T10:05:58.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:05:58.566+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:58.568466+0000) 2019-10-13T10:05:58.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:05:58.570+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:58.574713+0000) 2019-10-13T10:06:00.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:00.902+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.903841+0000) 2019-10-13T10:06:00.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:00.902+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.904546+0000) 2019-10-13T10:06:00.906 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:00.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.904561+0000) 2019-10-13T10:06:00.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:00.902+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.905292+0000) 2019-10-13T10:06:00.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:00.902+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.905873+0000) 2019-10-13T10:06:00.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:00.902+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.906888+0000) 2019-10-13T10:06:00.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:00.906+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.907487+0000) 2019-10-13T10:06:00.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:00.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:00.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:00.906+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.911397+0000) 2019-10-13T10:06:01.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:01.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.867104+0000) 2019-10-13T10:06:02.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:02.058+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:02.059487+0000) 2019-10-13T10:06:04.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:04.222+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.222970+0000) 2019-10-13T10:06:05.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:05.086+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.086184+0000) 2019-10-13T10:06:05.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:05.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.911743+0000) 2019-10-13T10:06:05.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:05.910+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.912204+0000) 2019-10-13T10:06:05.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:05.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.912607+0000) 2019-10-13T10:06:05.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:05.910+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.913284+0000) 2019-10-13T10:06:05.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:05.910+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.913633+0000) 2019-10-13T10:06:05.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:05.910+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.914438+0000) 2019-10-13T10:06:05.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:05.910+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.915253+0000) 2019-10-13T10:06:05.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:05.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:05.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:05.914+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.920302+0000) 2019-10-13T10:06:06.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:06.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.306481+0000) 2019-10-13T10:06:06.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:06.306+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.307082+0000) 2019-10-13T10:06:08.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:08.566+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:08.568668+0000) 2019-10-13T10:06:08.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:08.570+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:08.574942+0000) 2019-10-13T10:06:10.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:10.917+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.919293+0000) 2019-10-13T10:06:10.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:10.917+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.919560+0000) 2019-10-13T10:06:10.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:10.917+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.920001+0000) 2019-10-13T10:06:10.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:10.917+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.921238+0000) 2019-10-13T10:06:10.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:10.918+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.922244+0000) 2019-10-13T10:06:10.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:10.917+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.921727+0000) 2019-10-13T10:06:10.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:10.917+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.923017+0000) 2019-10-13T10:06:10.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:10.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:10.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:10.922+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.928874+0000) 2019-10-13T10:06:11.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:11.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.867286+0000) 2019-10-13T10:06:12.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:12.057+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:12.059690+0000) 2019-10-13T10:06:14.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:14.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.229449+0000) 2019-10-13T10:06:15.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:15.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.086388+0000) 2019-10-13T10:06:15.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:15.922+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.928343+0000) 2019-10-13T10:06:15.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:15.925+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.928723+0000) 2019-10-13T10:06:15.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:15.925+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.929281+0000) 2019-10-13T10:06:15.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:15.925+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.929976+0000) 2019-10-13T10:06:15.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:15.925+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.930919+0000) 2019-10-13T10:06:15.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:15.925+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.931145+0000) 2019-10-13T10:06:15.933 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:15.929+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.931478+0000) 2019-10-13T10:06:15.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:15.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:15.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:15.930+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.936958+0000) 2019-10-13T10:06:16.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:16.305+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.306651+0000) 2019-10-13T10:06:16.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:16.306+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.307308+0000) 2019-10-13T10:06:18.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:18.565+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:18.568883+0000) 2019-10-13T10:06:18.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:18.573+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:18.575178+0000) 2019-10-13T10:06:20.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:20.930+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.935904+0000) 2019-10-13T10:06:20.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:20.933+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.935815+0000) 2019-10-13T10:06:20.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:20.933+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.936263+0000) 2019-10-13T10:06:20.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:20.933+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.936834+0000) 2019-10-13T10:06:20.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:20.933+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.937202+0000) 2019-10-13T10:06:20.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:20.933+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.938078+0000) 2019-10-13T10:06:20.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:20.933+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.938522+0000) 2019-10-13T10:06:20.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:20.937+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:20.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:20.938+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.945006+0000) 2019-10-13T10:06:21.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:21.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.867513+0000) 2019-10-13T10:06:22.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:22.057+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:22.059903+0000) 2019-10-13T10:06:22.989 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:06:22.990 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:22.994 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:06:22.994 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:24.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:24.234+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.235809+0000) 2019-10-13T10:06:25.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:25.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.086609+0000) 2019-10-13T10:06:25.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:25.937+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.942462+0000) 2019-10-13T10:06:25.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:25.937+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.943255+0000) 2019-10-13T10:06:25.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:25.941+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.943689+0000) 2019-10-13T10:06:25.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:25.942+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.944103+0000) 2019-10-13T10:06:25.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:25.941+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.944442+0000) 2019-10-13T10:06:25.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:25.941+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.945077+0000) 2019-10-13T10:06:25.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:25.941+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.945144+0000) 2019-10-13T10:06:25.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:25.941+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:25.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:25.946+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.949322+0000) 2019-10-13T10:06:26.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:26.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.306821+0000) 2019-10-13T10:06:26.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:26.306+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.307434+0000) 2019-10-13T10:06:28.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:28.565+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:28.569123+0000) 2019-10-13T10:06:28.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:28.573+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:28.575392+0000) 2019-10-13T10:06:30.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:30.950+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.950982+0000) 2019-10-13T10:06:30.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:30.945+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.950182+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:30.949+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.951392+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:30.949+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.951765+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:30.949+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.952154+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:30.949+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.952313+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:30.949+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.952668+0000) 2019-10-13T10:06:30.957 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:30.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:30.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:30.958+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.958980+0000) 2019-10-13T10:06:31.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:31.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.867707+0000) 2019-10-13T10:06:32.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:32.057+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:32.060125+0000) 2019-10-13T10:06:34.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:34.241+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.242193+0000) 2019-10-13T10:06:35.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:35.081+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.086801+0000) 2019-10-13T10:06:35.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:35.953+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.957578+0000) 2019-10-13T10:06:35.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:35.953+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.958221+0000) 2019-10-13T10:06:35.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:35.953+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.958545+0000) 2019-10-13T10:06:35.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:35.953+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.959120+0000) 2019-10-13T10:06:35.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:35.957+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.959877+0000) 2019-10-13T10:06:35.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:35.957+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.961362+0000) 2019-10-13T10:06:35.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:35.957+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.961021+0000) 2019-10-13T10:06:35.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:35.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:35.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:35.965+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.966546+0000) 2019-10-13T10:06:36.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:36.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.307636+0000) 2019-10-13T10:06:36.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:36.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.307035+0000) 2019-10-13T10:06:38.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:38.565+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:38.569320+0000) 2019-10-13T10:06:38.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:38.573+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:38.575580+0000) 2019-10-13T10:06:40.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:40.961+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.965138+0000) 2019-10-13T10:06:40.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:40.961+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.966026+0000) 2019-10-13T10:06:40.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:40.961+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.966039+0000) 2019-10-13T10:06:40.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:40.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.966437+0000) 2019-10-13T10:06:40.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:40.961+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.967093+0000) 2019-10-13T10:06:40.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:40.965+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.967681+0000) 2019-10-13T10:06:40.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:40.965+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.967443+0000) 2019-10-13T10:06:40.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:40.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91809 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:40.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:40.973+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:40.974487+0000) 2019-10-13T10:06:41.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:41.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.867930+0000) 2019-10-13T10:06:42.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:42.057+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:42.060359+0000) 2019-10-13T10:06:44.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:44.245+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.248776+0000) 2019-10-13T10:06:45.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:45.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.086999+0000) 2019-10-13T10:06:45.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:45.969+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.972594+0000) 2019-10-13T10:06:45.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:45.969+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.974111+0000) 2019-10-13T10:06:45.975 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:45.969+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.974197+0000) 2019-10-13T10:06:45.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:45.969+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.974926+0000) 2019-10-13T10:06:45.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:45.973+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.975426+0000) 2019-10-13T10:06:45.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:45.973+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.975637+0000) 2019-10-13T10:06:45.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:45.973+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.975912+0000) 2019-10-13T10:06:45.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:45.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:45.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:45.977+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.981732+0000) 2019-10-13T10:06:46.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:46.301+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.307209+0000) 2019-10-13T10:06:46.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:46.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.307767+0000) 2019-10-13T10:06:48.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:48.569+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:48.569519+0000) 2019-10-13T10:06:48.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:48.573+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:48.575761+0000) 2019-10-13T10:06:50.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:50.977+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.980029+0000) 2019-10-13T10:06:50.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:50.977+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.980838+0000) 2019-10-13T10:06:50.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:50.977+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.980968+0000) 2019-10-13T10:06:50.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:50.977+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.981805+0000) 2019-10-13T10:06:50.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:50.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.982338+0000) 2019-10-13T10:06:50.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:50.977+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.982615+0000) 2019-10-13T10:06:50.984 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:50.977+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.983168+0000) 2019-10-13T10:06:50.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:50.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:50.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:50.989+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:50.990458+0000) 2019-10-13T10:06:51.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:51.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.868095+0000) 2019-10-13T10:06:52.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:52.057+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:52.060556+0000) 2019-10-13T10:06:53.073 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:06:53.073 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:53.077 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:06:53.077 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:54.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:54.253+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.255364+0000) 2019-10-13T10:06:55.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:55.080+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.087226+0000) 2019-10-13T10:06:55.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:06:55.984+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.987702+0000) 2019-10-13T10:06:55.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:06:55.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.987727+0000) 2019-10-13T10:06:55.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:55.985+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.988772+0000) 2019-10-13T10:06:55.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:55.984+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.988165+0000) 2019-10-13T10:06:55.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:55.984+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.989443+0000) 2019-10-13T10:06:55.991 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:55.984+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.989923+0000) 2019-10-13T10:06:55.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:06:55.984+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.990264+0000) 2019-10-13T10:06:55.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:06:55.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:06:55.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:06:55.993+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.997683+0000) 2019-10-13T10:06:56.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:06:56.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.307388+0000) 2019-10-13T10:06:56.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:06:56.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.307934+0000) 2019-10-13T10:06:58.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:06:58.564+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:58.569713+0000) 2019-10-13T10:06:58.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:06:58.572+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:58.575980+0000) 2019-10-13T10:07:00.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:00.992+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.995488+0000) 2019-10-13T10:07:00.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:00.993+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.996057+0000) 2019-10-13T10:07:00.998 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:00.992+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.996483+0000) 2019-10-13T10:07:00.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:00.992+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.996806+0000) 2019-10-13T10:07:00.998 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:00.992+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.997553+0000) 2019-10-13T10:07:00.999 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:00.992+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.997681+0000) 2019-10-13T10:07:01.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:00.992+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:00.998965+0000) 2019-10-13T10:07:01.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:00.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:01.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:00.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.001617+0000) 2019-10-13T10:07:01.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:01.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.868278+0000) 2019-10-13T10:07:02.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:02.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:02.060754+0000) 2019-10-13T10:07:04.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:04.257+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.261890+0000) 2019-10-13T10:07:05.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:05.084+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:05.087427+0000) 2019-10-13T10:07:06.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:05.996+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.002552+0000) 2019-10-13T10:07:06.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:05.996+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.003122+0000) 2019-10-13T10:07:06.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:06.001+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.003784+0000) 2019-10-13T10:07:06.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:06.000+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.004400+0000) 2019-10-13T10:07:06.006 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:06.000+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.004707+0000) 2019-10-13T10:07:06.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:06.000+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.004979+0000) 2019-10-13T10:07:06.007 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:06.000+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.006256+0000) 2019-10-13T10:07:06.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:06.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 91977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:06.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:06.009+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.012467+0000) 2019-10-13T10:07:06.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:06.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.307569+0000) 2019-10-13T10:07:06.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:06.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.308105+0000) 2019-10-13T10:07:08.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:08.568+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:08.569913+0000) 2019-10-13T10:07:08.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:08.572+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:08.576174+0000) 2019-10-13T10:07:11.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:11.004+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.009651+0000) 2019-10-13T10:07:11.012 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:11.004+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.010945+0000) 2019-10-13T10:07:11.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:11.004+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.011053+0000) 2019-10-13T10:07:11.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:11.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.011475+0000) 2019-10-13T10:07:11.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:11.008+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.012067+0000) 2019-10-13T10:07:11.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:11.008+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.012833+0000) 2019-10-13T10:07:11.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:11.008+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.013183+0000) 2019-10-13T10:07:11.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:11.008+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:11.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:11.017+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.019218+0000) 2019-10-13T10:07:11.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:11.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.868463+0000) 2019-10-13T10:07:12.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:12.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:12.060956+0000) 2019-10-13T10:07:14.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:14.265+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.267665+0000) 2019-10-13T10:07:15.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:15.084+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:15.087655+0000) 2019-10-13T10:07:16.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:16.012+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.018113+0000) 2019-10-13T10:07:16.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:16.017+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.018655+0000) 2019-10-13T10:07:16.020 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:16.012+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.019219+0000) 2019-10-13T10:07:16.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:16.016+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.019406+0000) 2019-10-13T10:07:16.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:16.016+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.019790+0000) 2019-10-13T10:07:16.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:16.016+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.020452+0000) 2019-10-13T10:07:16.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:16.016+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.021001+0000) 2019-10-13T10:07:16.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:16.016+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:16.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:16.025+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.027447+0000) 2019-10-13T10:07:16.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:16.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.307740+0000) 2019-10-13T10:07:16.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:16.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.308296+0000) 2019-10-13T10:07:18.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:18.564+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:18.570071+0000) 2019-10-13T10:07:18.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:18.572+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:18.576368+0000) 2019-10-13T10:07:21.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:21.020+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.026388+0000) 2019-10-13T10:07:21.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:21.020+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.026650+0000) 2019-10-13T10:07:21.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:21.024+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.027393+0000) 2019-10-13T10:07:21.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:21.024+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.028123+0000) 2019-10-13T10:07:21.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:21.025+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.028897+0000) 2019-10-13T10:07:21.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:21.024+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.029062+0000) 2019-10-13T10:07:21.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:21.024+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.029357+0000) 2019-10-13T10:07:21.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:21.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:21.036 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:21.033+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.034952+0000) 2019-10-13T10:07:21.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:21.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.868686+0000) 2019-10-13T10:07:22.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:22.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:22.061180+0000) 2019-10-13T10:07:23.200 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:07:23.201 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:23.204 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:07:23.204 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:24.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:24.273+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.274125+0000) 2019-10-13T10:07:25.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:25.084+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:25.087885+0000) 2019-10-13T10:07:26.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:26.029+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.033893+0000) 2019-10-13T10:07:26.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:26.028+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.034366+0000) 2019-10-13T10:07:26.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:26.028+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.035025+0000) 2019-10-13T10:07:26.036 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:26.032+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.035455+0000) 2019-10-13T10:07:26.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:26.032+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.036152+0000) 2019-10-13T10:07:26.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:26.032+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.036747+0000) 2019-10-13T10:07:26.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:26.032+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.037019+0000) 2019-10-13T10:07:26.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:26.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:26.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:26.041+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.042431+0000) 2019-10-13T10:07:26.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:26.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.307922+0000) 2019-10-13T10:07:26.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:26.305+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.308487+0000) 2019-10-13T10:07:28.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:28.568+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:28.570268+0000) 2019-10-13T10:07:28.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:28.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:28.576588+0000) 2019-10-13T10:07:31.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:31.036+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.041276+0000) 2019-10-13T10:07:31.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:31.036+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.041958+0000) 2019-10-13T10:07:31.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:31.040+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.042626+0000) 2019-10-13T10:07:31.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:31.036+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.042873+0000) 2019-10-13T10:07:31.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:31.040+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.043430+0000) 2019-10-13T10:07:31.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:31.040+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.043604+0000) 2019-10-13T10:07:31.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:31.040+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.044011+0000) 2019-10-13T10:07:31.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:31.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:31.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:31.048+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.050672+0000) 2019-10-13T10:07:31.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:31.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.868874+0000) 2019-10-13T10:07:32.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:32.056+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:32.061378+0000) 2019-10-13T10:07:34.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:34.276+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.280644+0000) 2019-10-13T10:07:35.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:35.088+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:35.088086+0000) 2019-10-13T10:07:36.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:36.044+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.049594+0000) 2019-10-13T10:07:36.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:36.044+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.050165+0000) 2019-10-13T10:07:36.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:36.044+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.050732+0000) 2019-10-13T10:07:36.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:36.044+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.051144+0000) 2019-10-13T10:07:36.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:36.048+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.051644+0000) 2019-10-13T10:07:36.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:36.048+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.051923+0000) 2019-10-13T10:07:36.054 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:36.048+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.052253+0000) 2019-10-13T10:07:36.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:36.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:36.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:36.052+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.056505+0000) 2019-10-13T10:07:36.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:36.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.308090+0000) 2019-10-13T10:07:36.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:36.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.308653+0000) 2019-10-13T10:07:38.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:38.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:38.570502+0000) 2019-10-13T10:07:38.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:38.575+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:38.576790+0000) 2019-10-13T10:07:41.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:41.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.057388+0000) 2019-10-13T10:07:41.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:41.051+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.057105+0000) 2019-10-13T10:07:41.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:41.051+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.058382+0000) 2019-10-13T10:07:41.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:41.056+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.058887+0000) 2019-10-13T10:07:41.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:41.051+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.058957+0000) 2019-10-13T10:07:41.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:41.055+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.059413+0000) 2019-10-13T10:07:41.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:41.055+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.059888+0000) 2019-10-13T10:07:41.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:41.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:41.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:41.064+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.066500+0000) 2019-10-13T10:07:41.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:41.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.869095+0000) 2019-10-13T10:07:42.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:42.059+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.061540+0000) 2019-10-13T10:07:44.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:44.284+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.286640+0000) 2019-10-13T10:07:45.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:45.083+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:45.088286+0000) 2019-10-13T10:07:46.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:46.059+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.064829+0000) 2019-10-13T10:07:46.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:46.059+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.065732+0000) 2019-10-13T10:07:46.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:46.059+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.066329+0000) 2019-10-13T10:07:46.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:46.059+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.066553+0000) 2019-10-13T10:07:46.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:46.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.068120+0000) 2019-10-13T10:07:46.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:46.063+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.068592+0000) 2019-10-13T10:07:46.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:46.063+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.068804+0000) 2019-10-13T10:07:46.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:46.063+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:46.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:46.068+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.074120+0000) 2019-10-13T10:07:46.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:46.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.308299+0000) 2019-10-13T10:07:46.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:46.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.308824+0000) 2019-10-13T10:07:48.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:48.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:48.570719+0000) 2019-10-13T10:07:48.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:48.571+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:48.576985+0000) 2019-10-13T10:07:51.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:51.067+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.073142+0000) 2019-10-13T10:07:51.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:51.067+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.073777+0000) 2019-10-13T10:07:51.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:51.067+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.074459+0000) 2019-10-13T10:07:51.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:51.071+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.075358+0000) 2019-10-13T10:07:51.077 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:51.071+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.075676+0000) 2019-10-13T10:07:51.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:51.071+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.076165+0000) 2019-10-13T10:07:51.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:51.072+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.077024+0000) 2019-10-13T10:07:51.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:51.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:51.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:51.076+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.081093+0000) 2019-10-13T10:07:51.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:51.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.869277+0000) 2019-10-13T10:07:52.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:52.059+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.061756+0000) 2019-10-13T10:07:53.303 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:07:53.303 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:53.307 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:07:53.307 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:54.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:54.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.293242+0000) 2019-10-13T10:07:55.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:55.087+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:55.088518+0000) 2019-10-13T10:07:56.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:07:56.075+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.081837+0000) 2019-10-13T10:07:56.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:07:56.075+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.082030+0000) 2019-10-13T10:07:56.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:07:56.075+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.082315+0000) 2019-10-13T10:07:56.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:56.075+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.082740+0000) 2019-10-13T10:07:56.085 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:56.079+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.084085+0000) 2019-10-13T10:07:56.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:56.079+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.084967+0000) 2019-10-13T10:07:56.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:56.080+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.085592+0000) 2019-10-13T10:07:56.087 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:07:56.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:07:56.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:07:56.092+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.094875+0000) 2019-10-13T10:07:56.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:07:56.303+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.308479+0000) 2019-10-13T10:07:56.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:07:56.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.308997+0000) 2019-10-13T10:07:58.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:07:58.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:58.570948+0000) 2019-10-13T10:07:58.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:07:58.575+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:58.577183+0000) 2019-10-13T10:08:01.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:01.083+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.090081+0000) 2019-10-13T10:08:01.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:01.083+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.090484+0000) 2019-10-13T10:08:01.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:01.087+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.091562+0000) 2019-10-13T10:08:01.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:01.087+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.091298+0000) 2019-10-13T10:08:01.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:01.087+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.092227+0000) 2019-10-13T10:08:01.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:01.087+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.092243+0000) 2019-10-13T10:08:01.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:01.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.093044+0000) 2019-10-13T10:08:01.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:01.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:01.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:01.100+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.102187+0000) 2019-10-13T10:08:01.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:01.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.869471+0000) 2019-10-13T10:08:02.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:02.059+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.061935+0000) 2019-10-13T10:08:04.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:04.296+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.299681+0000) 2019-10-13T10:08:05.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:05.087+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:05.088722+0000) 2019-10-13T10:08:06.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:06.091+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.097250+0000) 2019-10-13T10:08:06.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:06.091+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.097706+0000) 2019-10-13T10:08:06.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:06.091+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.098257+0000) 2019-10-13T10:08:06.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:06.091+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.098307+0000) 2019-10-13T10:08:06.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:06.091+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.099004+0000) 2019-10-13T10:08:06.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:06.095+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.100651+0000) 2019-10-13T10:08:06.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:06.096+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.101263+0000) 2019-10-13T10:08:06.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:06.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:06.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:06.100+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.106052+0000) 2019-10-13T10:08:06.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:06.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.308664+0000) 2019-10-13T10:08:06.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:06.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.309170+0000) 2019-10-13T10:08:08.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:08.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:08.571147+0000) 2019-10-13T10:08:08.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:08.575+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:08.577376+0000) 2019-10-13T10:08:11.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:11.103+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.105531+0000) 2019-10-13T10:08:11.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:11.103+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.106229+0000) 2019-10-13T10:08:11.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:11.103+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.106230+0000) 2019-10-13T10:08:11.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:11.103+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.106951+0000) 2019-10-13T10:08:11.109 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:11.107+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.107955+0000) 2019-10-13T10:08:11.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:11.107+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.108451+0000) 2019-10-13T10:08:11.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:11.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.108905+0000) 2019-10-13T10:08:11.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:11.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:11.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:11.108+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.112925+0000) 2019-10-13T10:08:11.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:11.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.869638+0000) 2019-10-13T10:08:12.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:12.059+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.062134+0000) 2019-10-13T10:08:14.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:14.308+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.310800+0000) 2019-10-13T10:08:15.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:15.087+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:15.088930+0000) 2019-10-13T10:08:16.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:16.111+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.113072+0000) 2019-10-13T10:08:16.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:16.111+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.113400+0000) 2019-10-13T10:08:16.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:16.111+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.114345+0000) 2019-10-13T10:08:16.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:16.111+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.114365+0000) 2019-10-13T10:08:16.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:16.111+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.114857+0000) 2019-10-13T10:08:16.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:16.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.115155+0000) 2019-10-13T10:08:16.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:16.112+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.116491+0000) 2019-10-13T10:08:16.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:16.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:16.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:16.120+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.122815+0000) 2019-10-13T10:08:16.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:16.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.308836+0000) 2019-10-13T10:08:16.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:16.304+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.309343+0000) 2019-10-13T10:08:18.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:18.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:18.571352+0000) 2019-10-13T10:08:18.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:18.575+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:18.577572+0000) 2019-10-13T10:08:21.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:21.118+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.120938+0000) 2019-10-13T10:08:21.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:21.118+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.121865+0000) 2019-10-13T10:08:21.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:21.120+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.122251+0000) 2019-10-13T10:08:21.124 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:21.118+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.122445+0000) 2019-10-13T10:08:21.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:21.118+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.122957+0000) 2019-10-13T10:08:21.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:21.122+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.123841+0000) 2019-10-13T10:08:21.126 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:21.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.124097+0000) 2019-10-13T10:08:21.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:21.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:21.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:21.128+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.130628+0000) 2019-10-13T10:08:21.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:21.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.869828+0000) 2019-10-13T10:08:22.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:22.058+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.062332+0000) 2019-10-13T10:08:23.406 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:08:23.407 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:23.410 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:08:23.410 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:24.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:24.315+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.317356+0000) 2019-10-13T10:08:25.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:25.086+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:25.089129+0000) 2019-10-13T10:08:26.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:26.126+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.128727+0000) 2019-10-13T10:08:26.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:26.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.129249+0000) 2019-10-13T10:08:26.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:26.126+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.130272+0000) 2019-10-13T10:08:26.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:26.126+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.130651+0000) 2019-10-13T10:08:26.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:26.130+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.131603+0000) 2019-10-13T10:08:26.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:26.130+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.131739+0000) 2019-10-13T10:08:26.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:26.130+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.132076+0000) 2019-10-13T10:08:26.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:26.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:26.140 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:26.135+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.139040+0000) 2019-10-13T10:08:26.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:26.306+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.309026+0000) 2019-10-13T10:08:26.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:26.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.309512+0000) 2019-10-13T10:08:28.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:28.570+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:28.571611+0000) 2019-10-13T10:08:28.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:28.574+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:28.577786+0000) 2019-10-13T10:08:31.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:31.131+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.136369+0000) 2019-10-13T10:08:31.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:31.134+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.137311+0000) 2019-10-13T10:08:31.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:31.134+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.137589+0000) 2019-10-13T10:08:31.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:31.134+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.138401+0000) 2019-10-13T10:08:31.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:31.134+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.138815+0000) 2019-10-13T10:08:31.140 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:31.134+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.139070+0000) 2019-10-13T10:08:31.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:31.138+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.139706+0000) 2019-10-13T10:08:31.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:31.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:31.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:31.139+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.144419+0000) 2019-10-13T10:08:31.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:31.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.870029+0000) 2019-10-13T10:08:32.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:32.058+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.062515+0000) 2019-10-13T10:08:34.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:34.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.323776+0000) 2019-10-13T10:08:35.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:35.086+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:35.089333+0000) 2019-10-13T10:08:36.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:36.142+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.145185+0000) 2019-10-13T10:08:36.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:36.142+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.145871+0000) 2019-10-13T10:08:36.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:36.142+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.146124+0000) 2019-10-13T10:08:36.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:36.142+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.146529+0000) 2019-10-13T10:08:36.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:36.146+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.147293+0000) 2019-10-13T10:08:36.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:36.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.147746+0000) 2019-10-13T10:08:36.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:36.146+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.147984+0000) 2019-10-13T10:08:36.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:36.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:36.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:36.151+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.154758+0000) 2019-10-13T10:08:36.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:36.306+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.309183+0000) 2019-10-13T10:08:36.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:36.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.309716+0000) 2019-10-13T10:08:38.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:38.570+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:38.571902+0000) 2019-10-13T10:08:38.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:38.574+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:38.577982+0000) 2019-10-13T10:08:41.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:41.150+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.152423+0000) 2019-10-13T10:08:41.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:41.150+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.152562+0000) 2019-10-13T10:08:41.154 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:41.150+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.152993+0000) 2019-10-13T10:08:41.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:41.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.153508+0000) 2019-10-13T10:08:41.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:41.150+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.153924+0000) 2019-10-13T10:08:41.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:41.154+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.155634+0000) 2019-10-13T10:08:41.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:41.151+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.155982+0000) 2019-10-13T10:08:41.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:41.154+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92606 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:41.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:41.155+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.161229+0000) 2019-10-13T10:08:41.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:41.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.870244+0000) 2019-10-13T10:08:42.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:42.058+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.062724+0000) 2019-10-13T10:08:44.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:44.327+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.330383+0000) 2019-10-13T10:08:45.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:45.086+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:45.089562+0000) 2019-10-13T10:08:46.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:46.158+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.159625+0000) 2019-10-13T10:08:46.161 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:46.158+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.160486+0000) 2019-10-13T10:08:46.162 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:46.158+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.160866+0000) 2019-10-13T10:08:46.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:46.158+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.161376+0000) 2019-10-13T10:08:46.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:46.158+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.161937+0000) 2019-10-13T10:08:46.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:46.158+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.162159+0000) 2019-10-13T10:08:46.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:46.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:46.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:46.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.163471+0000) 2019-10-13T10:08:46.172 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:46.163+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.168725+0000) 2019-10-13T10:08:46.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:46.306+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.309380+0000) 2019-10-13T10:08:46.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:46.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.309896+0000) 2019-10-13T10:08:48.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:48.570+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:48.572107+0000) 2019-10-13T10:08:48.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:48.574+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:48.578211+0000) 2019-10-13T10:08:51.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:51.166+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.167797+0000) 2019-10-13T10:08:51.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:51.166+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.168467+0000) 2019-10-13T10:08:51.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:51.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.169003+0000) 2019-10-13T10:08:51.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:51.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.169515+0000) 2019-10-13T10:08:51.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:51.166+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.169990+0000) 2019-10-13T10:08:51.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:51.166+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.170666+0000) 2019-10-13T10:08:51.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:51.170+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.171332+0000) 2019-10-13T10:08:51.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:51.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:51.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:51.171+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.176719+0000) 2019-10-13T10:08:51.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:51.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.870423+0000) 2019-10-13T10:08:52.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:52.058+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.062902+0000) 2019-10-13T10:08:53.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:08:53.535 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:53.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:08:53.539 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:54.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:54.335+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.336773+0000) 2019-10-13T10:08:55.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:55.086+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:55.089761+0000) 2019-10-13T10:08:56.177 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:56.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.175613+0000) 2019-10-13T10:08:56.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:08:56.174+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.176086+0000) 2019-10-13T10:08:56.178 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:56.171+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.177394+0000) 2019-10-13T10:08:56.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:08:56.174+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.177610+0000) 2019-10-13T10:08:56.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:56.174+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.178031+0000) 2019-10-13T10:08:56.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:56.174+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.178366+0000) 2019-10-13T10:08:56.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:08:56.174+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.178798+0000) 2019-10-13T10:08:56.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:08:56.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:08:56.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:08:56.179+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.183140+0000) 2019-10-13T10:08:56.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:08:56.306+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.309574+0000) 2019-10-13T10:08:56.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:08:56.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.310068+0000) 2019-10-13T10:08:58.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:08:58.570+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:58.572355+0000) 2019-10-13T10:08:58.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:08:58.574+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:58.578429+0000) 2019-10-13T10:09:01.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:01.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.182759+0000) 2019-10-13T10:09:01.184 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:01.182+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.183348+0000) 2019-10-13T10:09:01.185 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:01.182+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.183983+0000) 2019-10-13T10:09:01.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:01.182+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.184456+0000) 2019-10-13T10:09:01.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:01.182+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.185364+0000) 2019-10-13T10:09:01.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:01.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.185901+0000) 2019-10-13T10:09:01.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:01.182+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.186139+0000) 2019-10-13T10:09:01.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:01.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:01.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:01.187+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.191818+0000) 2019-10-13T10:09:01.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:01.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.870632+0000) 2019-10-13T10:09:02.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:02.062+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.063082+0000) 2019-10-13T10:09:04.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:04.343+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.343296+0000) 2019-10-13T10:09:05.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:05.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:05.089961+0000) 2019-10-13T10:09:06.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:06.185+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.190006+0000) 2019-10-13T10:09:06.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:06.189+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.191292+0000) 2019-10-13T10:09:06.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:06.185+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.191126+0000) 2019-10-13T10:09:06.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:06.189+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.192109+0000) 2019-10-13T10:09:06.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:06.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.192958+0000) 2019-10-13T10:09:06.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:06.189+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.193869+0000) 2019-10-13T10:09:06.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:06.189+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.194274+0000) 2019-10-13T10:09:06.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:06.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:06.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:06.195+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.199558+0000) 2019-10-13T10:09:06.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:06.305+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.309753+0000) 2019-10-13T10:09:06.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:06.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.310280+0000) 2019-10-13T10:09:08.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:08.569+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:08.572563+0000) 2019-10-13T10:09:08.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:08.573+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:08.578629+0000) 2019-10-13T10:09:11.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:11.193+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.197603+0000) 2019-10-13T10:09:11.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:11.193+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.198213+0000) 2019-10-13T10:09:11.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:11.195+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.198593+0000) 2019-10-13T10:09:11.200 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:11.193+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.198937+0000) 2019-10-13T10:09:11.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:11.197+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.199957+0000) 2019-10-13T10:09:11.202 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:11.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.201293+0000) 2019-10-13T10:09:11.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:11.197+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.201480+0000) 2019-10-13T10:09:11.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:11.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:11.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:11.203+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.208415+0000) 2019-10-13T10:09:11.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:11.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.870819+0000) 2019-10-13T10:09:12.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:12.061+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.063278+0000) 2019-10-13T10:09:14.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:14.347+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.349793+0000) 2019-10-13T10:09:15.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:15.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:15.090159+0000) 2019-10-13T10:09:16.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:16.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.206356+0000) 2019-10-13T10:09:16.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:16.201+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.206678+0000) 2019-10-13T10:09:16.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:16.205+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.207529+0000) 2019-10-13T10:09:16.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:16.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.208159+0000) 2019-10-13T10:09:16.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:16.205+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.208277+0000) 2019-10-13T10:09:16.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:16.205+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.208632+0000) 2019-10-13T10:09:16.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:16.205+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.209072+0000) 2019-10-13T10:09:16.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:16.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:16.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:16.207+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.212159+0000) 2019-10-13T10:09:16.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:16.307+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.310397+0000) 2019-10-13T10:09:16.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:16.305+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.309885+0000) 2019-10-13T10:09:18.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:18.569+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:18.572777+0000) 2019-10-13T10:09:18.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:18.577+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:18.578859+0000) 2019-10-13T10:09:21.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:21.206+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.213774+0000) 2019-10-13T10:09:21.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:21.209+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.214175+0000) 2019-10-13T10:09:21.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:21.209+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.214606+0000) 2019-10-13T10:09:21.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:21.209+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.214890+0000) 2019-10-13T10:09:21.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:21.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.215527+0000) 2019-10-13T10:09:21.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:21.213+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.215920+0000) 2019-10-13T10:09:21.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:21.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.217449+0000) 2019-10-13T10:09:21.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:21.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:21.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:21.214+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.219047+0000) 2019-10-13T10:09:21.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:21.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.871002+0000) 2019-10-13T10:09:22.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:22.061+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.063486+0000) 2019-10-13T10:09:23.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:09:23.638 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:23.643 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:09:23.643 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:24.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:24.354+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.356202+0000) 2019-10-13T10:09:25.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:25.089+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:25.090394+0000) 2019-10-13T10:09:26.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:26.217+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.221152+0000) 2019-10-13T10:09:26.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:26.217+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.221475+0000) 2019-10-13T10:09:26.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:26.218+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.222254+0000) 2019-10-13T10:09:26.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:26.217+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.222781+0000) 2019-10-13T10:09:26.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:26.217+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.222890+0000) 2019-10-13T10:09:26.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:26.221+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.223746+0000) 2019-10-13T10:09:26.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:26.221+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.224258+0000) 2019-10-13T10:09:26.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:26.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:26.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:26.222+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.229932+0000) 2019-10-13T10:09:26.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:26.305+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.310084+0000) 2019-10-13T10:09:26.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:26.306+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.310589+0000) 2019-10-13T10:09:28.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:28.569+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:28.572981+0000) 2019-10-13T10:09:28.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:28.577+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:28.579053+0000) 2019-10-13T10:09:31.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:31.222+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.228362+0000) 2019-10-13T10:09:31.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:31.225+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.229199+0000) 2019-10-13T10:09:31.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:31.225+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.229688+0000) 2019-10-13T10:09:31.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:31.225+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.230371+0000) 2019-10-13T10:09:31.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:31.225+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.230551+0000) 2019-10-13T10:09:31.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:31.225+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.231243+0000) 2019-10-13T10:09:31.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:31.229+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.231678+0000) 2019-10-13T10:09:31.234 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:31.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:31.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:31.230+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.236452+0000) 2019-10-13T10:09:31.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:31.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.871219+0000) 2019-10-13T10:09:32.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:32.061+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.063700+0000) 2019-10-13T10:09:34.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:34.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.362600+0000) 2019-10-13T10:09:35.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:35.089+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:35.090593+0000) 2019-10-13T10:09:36.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:36.233+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.236444+0000) 2019-10-13T10:09:36.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:36.233+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.237161+0000) 2019-10-13T10:09:36.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:36.233+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.237488+0000) 2019-10-13T10:09:36.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:36.233+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.238153+0000) 2019-10-13T10:09:36.240 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:36.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.238499+0000) 2019-10-13T10:09:36.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:36.237+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.239442+0000) 2019-10-13T10:09:36.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:36.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.240355+0000) 2019-10-13T10:09:36.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:36.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 92968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:36.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:36.238+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.244406+0000) 2019-10-13T10:09:36.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:36.309+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.310301+0000) 2019-10-13T10:09:36.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:36.310+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.310759+0000) 2019-10-13T10:09:38.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:38.569+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:38.573178+0000) 2019-10-13T10:09:38.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:38.577+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:38.579283+0000) 2019-10-13T10:09:41.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:41.241+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.244236+0000) 2019-10-13T10:09:41.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:41.241+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.245046+0000) 2019-10-13T10:09:41.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:41.241+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.245146+0000) 2019-10-13T10:09:41.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:41.241+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.245883+0000) 2019-10-13T10:09:41.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:41.241+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.246410+0000) 2019-10-13T10:09:41.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:41.242+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.247572+0000) 2019-10-13T10:09:41.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:41.245+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.247880+0000) 2019-10-13T10:09:41.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:41.245+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:41.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:41.250+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.254338+0000) 2019-10-13T10:09:41.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:41.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.871407+0000) 2019-10-13T10:09:42.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:42.061+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.063905+0000) 2019-10-13T10:09:44.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:44.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.369170+0000) 2019-10-13T10:09:45.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:45.085+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:45.090788+0000) 2019-10-13T10:09:46.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:46.249+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.252163+0000) 2019-10-13T10:09:46.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:46.249+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.253073+0000) 2019-10-13T10:09:46.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:46.249+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.253429+0000) 2019-10-13T10:09:46.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:46.249+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.254131+0000) 2019-10-13T10:09:46.255 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:46.249+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.254352+0000) 2019-10-13T10:09:46.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:46.249+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.254582+0000) 2019-10-13T10:09:46.256 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:46.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.255549+0000) 2019-10-13T10:09:46.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:46.253+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93034 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:46.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:46.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.262440+0000) 2019-10-13T10:09:46.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:46.305+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.310502+0000) 2019-10-13T10:09:46.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:46.310+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.310929+0000) 2019-10-13T10:09:48.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:48.572+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:48.573379+0000) 2019-10-13T10:09:48.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:48.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:48.579501+0000) 2019-10-13T10:09:51.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:51.256+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.260722+0000) 2019-10-13T10:09:51.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:51.256+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.261119+0000) 2019-10-13T10:09:51.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:51.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.261703+0000) 2019-10-13T10:09:51.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:51.258+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.262629+0000) 2019-10-13T10:09:51.264 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:51.256+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.262098+0000) 2019-10-13T10:09:51.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:51.256+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.263236+0000) 2019-10-13T10:09:51.265 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:51.260+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.263964+0000) 2019-10-13T10:09:51.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:51.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:51.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:51.262+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.266580+0000) 2019-10-13T10:09:51.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:51.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.871600+0000) 2019-10-13T10:09:52.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:52.060+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.064082+0000) 2019-10-13T10:09:53.753 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:09:53.753 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:53.757 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:09:53.757 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:54.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:54.374+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.375729+0000) 2019-10-13T10:09:55.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:55.088+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:55.091001+0000) 2019-10-13T10:09:56.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:09:56.264+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.268059+0000) 2019-10-13T10:09:56.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:56.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.268342+0000) 2019-10-13T10:09:56.273 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:56.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.268825+0000) 2019-10-13T10:09:56.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:56.264+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.269166+0000) 2019-10-13T10:09:56.274 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:09:56.264+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.269616+0000) 2019-10-13T10:09:56.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:56.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.270247+0000) 2019-10-13T10:09:56.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:09:56.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.271258+0000) 2019-10-13T10:09:56.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:09:56.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:09:56.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:09:56.274+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.278636+0000) 2019-10-13T10:09:56.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:09:56.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.310685+0000) 2019-10-13T10:09:56.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:09:56.310+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.311106+0000) 2019-10-13T10:09:58.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:09:58.568+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:58.573581+0000) 2019-10-13T10:09:58.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:09:58.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:58.579653+0000) 2019-10-13T10:10:01.277 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:01.272+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.276024+0000) 2019-10-13T10:10:01.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:01.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.276989+0000) 2019-10-13T10:10:01.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:01.272+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.276608+0000) 2019-10-13T10:10:01.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:01.272+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.277999+0000) 2019-10-13T10:10:01.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:01.272+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.278691+0000) 2019-10-13T10:10:01.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:01.274+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.279214+0000) 2019-10-13T10:10:01.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:01.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.279675+0000) 2019-10-13T10:10:01.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:01.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:01.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:01.282+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.286779+0000) 2019-10-13T10:10:01.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:01.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.871796+0000) 2019-10-13T10:10:02.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:02.060+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.064257+0000) 2019-10-13T10:10:04.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:04.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.382351+0000) 2019-10-13T10:10:05.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:05.084+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:05.091233+0000) 2019-10-13T10:10:06.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:06.280+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.283412+0000) 2019-10-13T10:10:06.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:06.280+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.283687+0000) 2019-10-13T10:10:06.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:06.280+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.285572+0000) 2019-10-13T10:10:06.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:06.280+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.285965+0000) 2019-10-13T10:10:06.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:06.280+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.286309+0000) 2019-10-13T10:10:06.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:06.280+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.286680+0000) 2019-10-13T10:10:06.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:06.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.284792+0000) 2019-10-13T10:10:06.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:06.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:06.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:06.290+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.291155+0000) 2019-10-13T10:10:06.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:06.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.310857+0000) 2019-10-13T10:10:06.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:06.310+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.311267+0000) 2019-10-13T10:10:08.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:08.572+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:08.573776+0000) 2019-10-13T10:10:08.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:08.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:08.579858+0000) 2019-10-13T10:10:11.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:11.290+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.290278+0000) 2019-10-13T10:10:11.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:11.288+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.291645+0000) 2019-10-13T10:10:11.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:11.288+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.292024+0000) 2019-10-13T10:10:11.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:11.288+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.292385+0000) 2019-10-13T10:10:11.294 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:11.288+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.292652+0000) 2019-10-13T10:10:11.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:11.288+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.293344+0000) 2019-10-13T10:10:11.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:11.288+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.293830+0000) 2019-10-13T10:10:11.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:11.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:11.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:11.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.301418+0000) 2019-10-13T10:10:11.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:11.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.872008+0000) 2019-10-13T10:10:12.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:12.060+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.064445+0000) 2019-10-13T10:10:14.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:14.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.388847+0000) 2019-10-13T10:10:15.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:15.088+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:15.091518+0000) 2019-10-13T10:10:16.309 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:16.292+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.297889+0000) 2019-10-13T10:10:16.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:16.297+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.299198+0000) 2019-10-13T10:10:16.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:16.296+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.299426+0000) 2019-10-13T10:10:16.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:16.296+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.299804+0000) 2019-10-13T10:10:16.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:16.296+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.300041+0000) 2019-10-13T10:10:16.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:16.296+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.300490+0000) 2019-10-13T10:10:16.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:16.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.300880+0000) 2019-10-13T10:10:16.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:16.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:16.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:16.305+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.307036+0000) 2019-10-13T10:10:16.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:16.304+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.310994+0000) 2019-10-13T10:10:16.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:16.309+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.311393+0000) 2019-10-13T10:10:18.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:18.572+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:18.574002+0000) 2019-10-13T10:10:18.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:18.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:18.580054+0000) 2019-10-13T10:10:21.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:21.300+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.306032+0000) 2019-10-13T10:10:21.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:21.300+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.306667+0000) 2019-10-13T10:10:21.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:21.300+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.307019+0000) 2019-10-13T10:10:21.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:21.304+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.307499+0000) 2019-10-13T10:10:21.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:21.305+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.308054+0000) 2019-10-13T10:10:21.310 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:21.304+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.308466+0000) 2019-10-13T10:10:21.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:21.304+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.308696+0000) 2019-10-13T10:10:21.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:21.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:21.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:21.313+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.314672+0000) 2019-10-13T10:10:21.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:21.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.872224+0000) 2019-10-13T10:10:22.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:22.060+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.064620+0000) 2019-10-13T10:10:23.876 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:10:23.877 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:23.880 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:10:23.881 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:24.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:24.393+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.395648+0000) 2019-10-13T10:10:25.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:25.088+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:25.091743+0000) 2019-10-13T10:10:26.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:26.308+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.311120+0000) 2019-10-13T10:10:26.314 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:26.308+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.313297+0000) 2019-10-13T10:10:26.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:26.308+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.313737+0000) 2019-10-13T10:10:26.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:26.308+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.314737+0000) 2019-10-13T10:10:26.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:26.312+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.315855+0000) 2019-10-13T10:10:26.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:26.312+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.316482+0000) 2019-10-13T10:10:26.318 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:26.312+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.316527+0000) 2019-10-13T10:10:26.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:26.313+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.317304+0000) 2019-10-13T10:10:26.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:26.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:26.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:26.321+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.323078+0000) 2019-10-13T10:10:28.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:28.568+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:28.574233+0000) 2019-10-13T10:10:28.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:28.580+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:28.580247+0000) 2019-10-13T10:10:31.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:31.315+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.321484+0000) 2019-10-13T10:10:31.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:31.315+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.321842+0000) 2019-10-13T10:10:31.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:31.315+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.323120+0000) 2019-10-13T10:10:31.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:31.321+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.323904+0000) 2019-10-13T10:10:31.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:31.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.323508+0000) 2019-10-13T10:10:31.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:31.319+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.324518+0000) 2019-10-13T10:10:31.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:31.319+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.325701+0000) 2019-10-13T10:10:31.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:31.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:31.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:31.329+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.331608+0000) 2019-10-13T10:10:31.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:31.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.872370+0000) 2019-10-13T10:10:32.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:32.059+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.064797+0000) 2019-10-13T10:10:33.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:33.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:33.062305+0000) 2019-10-13T10:10:34.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:34.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.402193+0000) 2019-10-13T10:10:35.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:35.087+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:35.091944+0000) 2019-10-13T10:10:36.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:36.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.311341+0000) 2019-10-13T10:10:36.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:36.323+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.329406+0000) 2019-10-13T10:10:36.331 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:36.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.330056+0000) 2019-10-13T10:10:36.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:36.329+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.330569+0000) 2019-10-13T10:10:36.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:36.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.331239+0000) 2019-10-13T10:10:36.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:36.327+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.331502+0000) 2019-10-13T10:10:36.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:36.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.331722+0000) 2019-10-13T10:10:36.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:36.327+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.332088+0000) 2019-10-13T10:10:36.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:36.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:36.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:36.337+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.339204+0000) 2019-10-13T10:10:38.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:38.567+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:38.574452+0000) 2019-10-13T10:10:38.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:38.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:38.580451+0000) 2019-10-13T10:10:41.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:41.331+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.337510+0000) 2019-10-13T10:10:41.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:41.331+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.338545+0000) 2019-10-13T10:10:41.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:41.331+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.338937+0000) 2019-10-13T10:10:41.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:41.331+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.339271+0000) 2019-10-13T10:10:41.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:41.335+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.339993+0000) 2019-10-13T10:10:41.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:41.335+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.340992+0000) 2019-10-13T10:10:41.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:41.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.341552+0000) 2019-10-13T10:10:41.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:41.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:41.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:41.345+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.347124+0000) 2019-10-13T10:10:41.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:41.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.872559+0000) 2019-10-13T10:10:42.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:42.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.064985+0000) 2019-10-13T10:10:43.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:43.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:43.062515+0000) 2019-10-13T10:10:44.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:44.405+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.408864+0000) 2019-10-13T10:10:45.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:45.087+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:45.092158+0000) 2019-10-13T10:10:46.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:46.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.311493+0000) 2019-10-13T10:10:46.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:46.339+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.345775+0000) 2019-10-13T10:10:46.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:46.339+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.346271+0000) 2019-10-13T10:10:46.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:46.339+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.346866+0000) 2019-10-13T10:10:46.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:46.343+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.347543+0000) 2019-10-13T10:10:46.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:46.339+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.347214+0000) 2019-10-13T10:10:46.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:46.343+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.348543+0000) 2019-10-13T10:10:46.350 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:46.345+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.349131+0000) 2019-10-13T10:10:46.351 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:46.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:46.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:46.357+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.358241+0000) 2019-10-13T10:10:48.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:48.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:48.574666+0000) 2019-10-13T10:10:48.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:48.575+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:48.580654+0000) 2019-10-13T10:10:51.355 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:51.347+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.354221+0000) 2019-10-13T10:10:51.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:51.347+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.353986+0000) 2019-10-13T10:10:51.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:51.347+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.355137+0000) 2019-10-13T10:10:51.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:51.347+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.355239+0000) 2019-10-13T10:10:51.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:51.351+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.355939+0000) 2019-10-13T10:10:51.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:51.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.355648+0000) 2019-10-13T10:10:51.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:51.351+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.356908+0000) 2019-10-13T10:10:51.359 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:51.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:51.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:51.357+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.359144+0000) 2019-10-13T10:10:51.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:51.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.872773+0000) 2019-10-13T10:10:52.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:52.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.065161+0000) 2019-10-13T10:10:53.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:53.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:53.062731+0000) 2019-10-13T10:10:54.024 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:10:54.024 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:54.032 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:10:54.032 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:54.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:54.413+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.415142+0000) 2019-10-13T10:10:55.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:55.091+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:55.092361+0000) 2019-10-13T10:10:56.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:56.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.311685+0000) 2019-10-13T10:10:56.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:10:56.357+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.361317+0000) 2019-10-13T10:10:56.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:10:56.355+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.361952+0000) 2019-10-13T10:10:56.364 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:10:56.355+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.362527+0000) 2019-10-13T10:10:56.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:56.359+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.363519+0000) 2019-10-13T10:10:56.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:56.359+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.363652+0000) 2019-10-13T10:10:56.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:10:56.359+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.364152+0000) 2019-10-13T10:10:56.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:10:56.359+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.364486+0000) 2019-10-13T10:10:56.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:10:56.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:10:56.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:10:56.365+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.368778+0000) 2019-10-13T10:10:58.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:10:58.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:58.574869+0000) 2019-10-13T10:10:58.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:10:58.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:58.580879+0000) 2019-10-13T10:11:01.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:01.363+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.368765+0000) 2019-10-13T10:11:01.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:01.363+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.369617+0000) 2019-10-13T10:11:01.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:01.365+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.370086+0000) 2019-10-13T10:11:01.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:01.363+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.370317+0000) 2019-10-13T10:11:01.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:01.363+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.370672+0000) 2019-10-13T10:11:01.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:01.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.371216+0000) 2019-10-13T10:11:01.373 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:01.367+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.372409+0000) 2019-10-13T10:11:01.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:01.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:01.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:01.377+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.378437+0000) 2019-10-13T10:11:01.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:01.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.872962+0000) 2019-10-13T10:11:02.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:02.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.065352+0000) 2019-10-13T10:11:03.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:03.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:03.062959+0000) 2019-10-13T10:11:04.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:04.417+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.422042+0000) 2019-10-13T10:11:05.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:05.091+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:05.092605+0000) 2019-10-13T10:11:06.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:06.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.311878+0000) 2019-10-13T10:11:06.378 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:06.375+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.376782+0000) 2019-10-13T10:11:06.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:06.375+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.377548+0000) 2019-10-13T10:11:06.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:06.373+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.377897+0000) 2019-10-13T10:11:06.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:06.375+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.378298+0000) 2019-10-13T10:11:06.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:06.379+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.379507+0000) 2019-10-13T10:11:06.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:06.379+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.379983+0000) 2019-10-13T10:11:06.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:06.379+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.380646+0000) 2019-10-13T10:11:06.382 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:06.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:06.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:06.385+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.386602+0000) 2019-10-13T10:11:08.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:08.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:08.575071+0000) 2019-10-13T10:11:08.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:08.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:08.581073+0000) 2019-10-13T10:11:11.386 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:11.383+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.384614+0000) 2019-10-13T10:11:11.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:11.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.385642+0000) 2019-10-13T10:11:11.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:11.384+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.386210+0000) 2019-10-13T10:11:11.388 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:11.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.386264+0000) 2019-10-13T10:11:11.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:11.383+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.387229+0000) 2019-10-13T10:11:11.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:11.387+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.387599+0000) 2019-10-13T10:11:11.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:11.387+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.387921+0000) 2019-10-13T10:11:11.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:11.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:11.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:11.388+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.392374+0000) 2019-10-13T10:11:11.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:11.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.873149+0000) 2019-10-13T10:11:12.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:12.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.065547+0000) 2019-10-13T10:11:13.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:13.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:13.063168+0000) 2019-10-13T10:11:14.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:14.424+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.428556+0000) 2019-10-13T10:11:15.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:15.090+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:15.092803+0000) 2019-10-13T10:11:16.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:16.310+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.312068+0000) 2019-10-13T10:11:16.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:16.388+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.393058+0000) 2019-10-13T10:11:16.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:16.390+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.393677+0000) 2019-10-13T10:11:16.397 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:16.390+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.393764+0000) 2019-10-13T10:11:16.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:16.390+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.394376+0000) 2019-10-13T10:11:16.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:16.390+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.394619+0000) 2019-10-13T10:11:16.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:16.390+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.395190+0000) 2019-10-13T10:11:16.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:16.394+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.396016+0000) 2019-10-13T10:11:16.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:16.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:16.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:16.400+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.402230+0000) 2019-10-13T10:11:18.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:18.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:18.575267+0000) 2019-10-13T10:11:18.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:18.578+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:18.581222+0000) 2019-10-13T10:11:21.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:21.398+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.400146+0000) 2019-10-13T10:11:21.403 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:21.396+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.401321+0000) 2019-10-13T10:11:21.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:21.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.401118+0000) 2019-10-13T10:11:21.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:21.398+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.401584+0000) 2019-10-13T10:11:21.405 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:21.398+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.402729+0000) 2019-10-13T10:11:21.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:21.402+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.403468+0000) 2019-10-13T10:11:21.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:21.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.403721+0000) 2019-10-13T10:11:21.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:21.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:21.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:21.408+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.410785+0000) 2019-10-13T10:11:21.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:21.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.873354+0000) 2019-10-13T10:11:22.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:22.062+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.065728+0000) 2019-10-13T10:11:23.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:23.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:23.063357+0000) 2019-10-13T10:11:24.125 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:11:24.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:24.129 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:11:24.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:24.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:24.432+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.434940+0000) 2019-10-13T10:11:25.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:25.090+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:25.092979+0000) 2019-10-13T10:11:26.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:26.310+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.312262+0000) 2019-10-13T10:11:26.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:26.406+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.408229+0000) 2019-10-13T10:11:26.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:26.406+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.408842+0000) 2019-10-13T10:11:26.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:26.406+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.409412+0000) 2019-10-13T10:11:26.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:26.406+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.409947+0000) 2019-10-13T10:11:26.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:26.406+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.410325+0000) 2019-10-13T10:11:26.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:26.408+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.410799+0000) 2019-10-13T10:11:26.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:26.406+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.411046+0000) 2019-10-13T10:11:26.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:26.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:26.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:26.412+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.415589+0000) 2019-10-13T10:11:28.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:28.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:28.575490+0000) 2019-10-13T10:11:28.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:28.578+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:28.581440+0000) 2019-10-13T10:11:31.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:31.414+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.415746+0000) 2019-10-13T10:11:31.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:31.414+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.416107+0000) 2019-10-13T10:11:31.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:31.414+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.416683+0000) 2019-10-13T10:11:31.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:31.414+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.417301+0000) 2019-10-13T10:11:31.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:31.414+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.417818+0000) 2019-10-13T10:11:31.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:31.416+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.418462+0000) 2019-10-13T10:11:31.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:31.414+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.419149+0000) 2019-10-13T10:11:31.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:31.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:31.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:31.420+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.425345+0000) 2019-10-13T10:11:31.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:31.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.873544+0000) 2019-10-13T10:11:32.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:32.062+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.065926+0000) 2019-10-13T10:11:33.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:33.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:33.063585+0000) 2019-10-13T10:11:34.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:34.436+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.441268+0000) 2019-10-13T10:11:35.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:35.090+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:35.093181+0000) 2019-10-13T10:11:36.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:36.310+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.312460+0000) 2019-10-13T10:11:36.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:36.422+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.424313+0000) 2019-10-13T10:11:36.426 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:36.422+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.424650+0000) 2019-10-13T10:11:36.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:36.422+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.425031+0000) 2019-10-13T10:11:36.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:36.422+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.425303+0000) 2019-10-13T10:11:36.427 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:36.420+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.426006+0000) 2019-10-13T10:11:36.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:36.422+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.426591+0000) 2019-10-13T10:11:36.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:36.422+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.426946+0000) 2019-10-13T10:11:36.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:36.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:36.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:36.432+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.434851+0000) 2019-10-13T10:11:38.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:38.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:38.575644+0000) 2019-10-13T10:11:38.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:38.578+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:38.581635+0000) 2019-10-13T10:11:41.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:41.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.430973+0000) 2019-10-13T10:11:41.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:41.428+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.431966+0000) 2019-10-13T10:11:41.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:41.430+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.431496+0000) 2019-10-13T10:11:41.434 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:41.430+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.432249+0000) 2019-10-13T10:11:41.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:41.430+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.433518+0000) 2019-10-13T10:11:41.435 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:41.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.434194+0000) 2019-10-13T10:11:41.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:41.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:41.436 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:41.430+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.435154+0000) 2019-10-13T10:11:41.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:41.436+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.439879+0000) 2019-10-13T10:11:41.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:41.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.873777+0000) 2019-10-13T10:11:42.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:42.062+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.066141+0000) 2019-10-13T10:11:43.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:43.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:43.063781+0000) 2019-10-13T10:11:44.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:44.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.447800+0000) 2019-10-13T10:11:45.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:45.090+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:45.093389+0000) 2019-10-13T10:11:46.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:46.310+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.312683+0000) 2019-10-13T10:11:46.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:46.434+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.437974+0000) 2019-10-13T10:11:46.440 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:46.434+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.438894+0000) 2019-10-13T10:11:46.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:46.434+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.439245+0000) 2019-10-13T10:11:46.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:46.438+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.439617+0000) 2019-10-13T10:11:46.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:46.436+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.441090+0000) 2019-10-13T10:11:46.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:46.438+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.440876+0000) 2019-10-13T10:11:46.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:46.438+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.441631+0000) 2019-10-13T10:11:46.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:46.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93815 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:46.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:46.444+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.447072+0000) 2019-10-13T10:11:48.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:48.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:48.575840+0000) 2019-10-13T10:11:48.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:48.578+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:48.581838+0000) 2019-10-13T10:11:51.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:51.440+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.445762+0000) 2019-10-13T10:11:51.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:51.442+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.446481+0000) 2019-10-13T10:11:51.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:51.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.446804+0000) 2019-10-13T10:11:51.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:51.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.447139+0000) 2019-10-13T10:11:51.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:51.446+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.448543+0000) 2019-10-13T10:11:51.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:51.446+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.448742+0000) 2019-10-13T10:11:51.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:51.446+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.449164+0000) 2019-10-13T10:11:51.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:51.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:51.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:51.452+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.457260+0000) 2019-10-13T10:11:51.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:51.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.873987+0000) 2019-10-13T10:11:52.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:52.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.066324+0000) 2019-10-13T10:11:53.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:53.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:53.063974+0000) 2019-10-13T10:11:54.254 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:11:54.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:54.258 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:11:54.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:54.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:54.452+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.454573+0000) 2019-10-13T10:11:55.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:55.090+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:55.093604+0000) 2019-10-13T10:11:56.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:56.310+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.312874+0000) 2019-10-13T10:11:56.455 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:56.450+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.453866+0000) 2019-10-13T10:11:56.456 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:11:56.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.454666+0000) 2019-10-13T10:11:56.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:11:56.452+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.456678+0000) 2019-10-13T10:11:56.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:11:56.454+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.455417+0000) 2019-10-13T10:11:56.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:11:56.454+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.456241+0000) 2019-10-13T10:11:56.458 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:11:56.454+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.456942+0000) 2019-10-13T10:11:56.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:56.454+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.457411+0000) 2019-10-13T10:11:56.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:11:56.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:11:56.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:11:56.460+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.464199+0000) 2019-10-13T10:11:58.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:11:58.573+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:58.575995+0000) 2019-10-13T10:11:58.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:11:58.581+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:58.582037+0000) 2019-10-13T10:12:01.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:01.457+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.463017+0000) 2019-10-13T10:12:01.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:01.461+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.463361+0000) 2019-10-13T10:12:01.465 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:01.461+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.463703+0000) 2019-10-13T10:12:01.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:01.461+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.464594+0000) 2019-10-13T10:12:01.470 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:01.461+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.466689+0000) 2019-10-13T10:12:01.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:01.461+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.467030+0000) 2019-10-13T10:12:01.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:01.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:01.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:01.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.466357+0000) 2019-10-13T10:12:01.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:01.468+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.471264+0000) 2019-10-13T10:12:01.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:01.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.874172+0000) 2019-10-13T10:12:02.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:02.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.066507+0000) 2019-10-13T10:12:03.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:03.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:03.064165+0000) 2019-10-13T10:12:04.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:04.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.465788+0000) 2019-10-13T10:12:05.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:05.089+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:05.093799+0000) 2019-10-13T10:12:06.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:06.309+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.313086+0000) 2019-10-13T10:12:06.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:06.465+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.470826+0000) 2019-10-13T10:12:06.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:06.469+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.472070+0000) 2019-10-13T10:12:06.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:06.469+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.473067+0000) 2019-10-13T10:12:06.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:06.469+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.473346+0000) 2019-10-13T10:12:06.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:06.469+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.473717+0000) 2019-10-13T10:12:06.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:06.469+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.474140+0000) 2019-10-13T10:12:06.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:06.471+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.474723+0000) 2019-10-13T10:12:06.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:06.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:06.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:06.475+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.480960+0000) 2019-10-13T10:12:08.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:08.573+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:08.576197+0000) 2019-10-13T10:12:08.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:08.577+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:08.582229+0000) 2019-10-13T10:12:11.480 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:11.473+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.478489+0000) 2019-10-13T10:12:11.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:11.473+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.479146+0000) 2019-10-13T10:12:11.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:11.477+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.479569+0000) 2019-10-13T10:12:11.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:11.477+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.480494+0000) 2019-10-13T10:12:11.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:11.477+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.480653+0000) 2019-10-13T10:12:11.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:11.477+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.481051+0000) 2019-10-13T10:12:11.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:11.475+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.481726+0000) 2019-10-13T10:12:11.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:11.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 93975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:11.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:11.483+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.488181+0000) 2019-10-13T10:12:11.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:11.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.874349+0000) 2019-10-13T10:12:12.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:12.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.066694+0000) 2019-10-13T10:12:13.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:13.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:13.064372+0000) 2019-10-13T10:12:14.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:14.467+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.472232+0000) 2019-10-13T10:12:15.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:15.093+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:15.094000+0000) 2019-10-13T10:12:16.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:16.309+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.313297+0000) 2019-10-13T10:12:16.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:16.481+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.485934+0000) 2019-10-13T10:12:16.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:16.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.486898+0000) 2019-10-13T10:12:16.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:16.485+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.487478+0000) 2019-10-13T10:12:16.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:16.485+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.487906+0000) 2019-10-13T10:12:16.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:16.483+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.488609+0000) 2019-10-13T10:12:16.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:16.485+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.489174+0000) 2019-10-13T10:12:16.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:16.485+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.490326+0000) 2019-10-13T10:12:16.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:16.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:16.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:16.491+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.495863+0000) 2019-10-13T10:12:18.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:18.573+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:18.576392+0000) 2019-10-13T10:12:18.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:18.581+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:18.582425+0000) 2019-10-13T10:12:21.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:21.489+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.493405+0000) 2019-10-13T10:12:21.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:21.489+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.494142+0000) 2019-10-13T10:12:21.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:21.489+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.494756+0000) 2019-10-13T10:12:21.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:21.489+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.495275+0000) 2019-10-13T10:12:21.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:21.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.495930+0000) 2019-10-13T10:12:21.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:21.493+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.496475+0000) 2019-10-13T10:12:21.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:21.493+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.496910+0000) 2019-10-13T10:12:21.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:21.493+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:21.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:21.499+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.503704+0000) 2019-10-13T10:12:21.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:21.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.874505+0000) 2019-10-13T10:12:22.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:22.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.066881+0000) 2019-10-13T10:12:23.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:23.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:23.064577+0000) 2019-10-13T10:12:24.378 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:12:24.378 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:24.382 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:12:24.382 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:24.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:24.475+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.478662+0000) 2019-10-13T10:12:25.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:25.093+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:25.094200+0000) 2019-10-13T10:12:26.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:26.309+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.313488+0000) 2019-10-13T10:12:26.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:26.497+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.501526+0000) 2019-10-13T10:12:26.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:26.497+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.502770+0000) 2019-10-13T10:12:26.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:26.497+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.503221+0000) 2019-10-13T10:12:26.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:26.501+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.503738+0000) 2019-10-13T10:12:26.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:26.499+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.504719+0000) 2019-10-13T10:12:26.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:26.501+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.505053+0000) 2019-10-13T10:12:26.508 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:26.501+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.504001+0000) 2019-10-13T10:12:26.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:26.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:26.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:26.507+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.510971+0000) 2019-10-13T10:12:28.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:28.573+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:28.576637+0000) 2019-10-13T10:12:28.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:28.577+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:28.582627+0000) 2019-10-13T10:12:31.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:31.507+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.510290+0000) 2019-10-13T10:12:31.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:31.505+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.510890+0000) 2019-10-13T10:12:31.512 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:31.505+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.511282+0000) 2019-10-13T10:12:31.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:31.509+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.511788+0000) 2019-10-13T10:12:31.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:31.509+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.512412+0000) 2019-10-13T10:12:31.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:31.509+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.512960+0000) 2019-10-13T10:12:31.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:31.509+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.513224+0000) 2019-10-13T10:12:31.515 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:31.509+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:31.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:31.511+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.517599+0000) 2019-10-13T10:12:31.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:31.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.874690+0000) 2019-10-13T10:12:32.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:32.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.067064+0000) 2019-10-13T10:12:33.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:33.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:33.064773+0000) 2019-10-13T10:12:34.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:34.483+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.485151+0000) 2019-10-13T10:12:35.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:35.089+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:35.094407+0000) 2019-10-13T10:12:36.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:36.309+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.313678+0000) 2019-10-13T10:12:36.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:36.513+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.517757+0000) 2019-10-13T10:12:36.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:36.515+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.518257+0000) 2019-10-13T10:12:36.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:36.513+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.518523+0000) 2019-10-13T10:12:36.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:36.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.519234+0000) 2019-10-13T10:12:36.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:36.517+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.520163+0000) 2019-10-13T10:12:36.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:36.517+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.520285+0000) 2019-10-13T10:12:36.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:36.517+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.520916+0000) 2019-10-13T10:12:36.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:36.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:36.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:36.523+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.527813+0000) 2019-10-13T10:12:38.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:38.573+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:38.576839+0000) 2019-10-13T10:12:38.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:38.581+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:38.582838+0000) 2019-10-13T10:12:41.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:41.519+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.524935+0000) 2019-10-13T10:12:41.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:41.521+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.525686+0000) 2019-10-13T10:12:41.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:41.521+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.526379+0000) 2019-10-13T10:12:41.528 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:41.521+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.527065+0000) 2019-10-13T10:12:41.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:41.525+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.527545+0000) 2019-10-13T10:12:41.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:41.525+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.528182+0000) 2019-10-13T10:12:41.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:41.525+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.528970+0000) 2019-10-13T10:12:41.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:41.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:41.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:41.531+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.534729+0000) 2019-10-13T10:12:41.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:41.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.874876+0000) 2019-10-13T10:12:42.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:42.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.067250+0000) 2019-10-13T10:12:43.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:43.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:43.064984+0000) 2019-10-13T10:12:44.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:44.487+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.491630+0000) 2019-10-13T10:12:45.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:45.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:45.094641+0000) 2019-10-13T10:12:46.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:46.312+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.313907+0000) 2019-10-13T10:12:46.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:46.528+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.533350+0000) 2019-10-13T10:12:46.541 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:46.528+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.533732+0000) 2019-10-13T10:12:46.541 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:46.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.534534+0000) 2019-10-13T10:12:46.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:46.528+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.534664+0000) 2019-10-13T10:12:46.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:46.531+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.535736+0000) 2019-10-13T10:12:46.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:46.528+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.535013+0000) 2019-10-13T10:12:46.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:46.532+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.536618+0000) 2019-10-13T10:12:46.545 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:46.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:46.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:46.539+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.542501+0000) 2019-10-13T10:12:48.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:48.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:48.577044+0000) 2019-10-13T10:12:48.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:48.576+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:48.583030+0000) 2019-10-13T10:12:51.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:51.536+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.540963+0000) 2019-10-13T10:12:51.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:51.536+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.541603+0000) 2019-10-13T10:12:51.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:51.536+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.541608+0000) 2019-10-13T10:12:51.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:51.536+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.541861+0000) 2019-10-13T10:12:51.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:51.536+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.542177+0000) 2019-10-13T10:12:51.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:51.540+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.544160+0000) 2019-10-13T10:12:51.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:51.539+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.544936+0000) 2019-10-13T10:12:51.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:51.540+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:51.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:51.547+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.551167+0000) 2019-10-13T10:12:51.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:51.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.875071+0000) 2019-10-13T10:12:52.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:52.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.067489+0000) 2019-10-13T10:12:53.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:53.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:53.065176+0000) 2019-10-13T10:12:54.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:54.499+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.503042+0000) 2019-10-13T10:12:54.548 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:12:54.548 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:54.551 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:12:54.552 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:55.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:55.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:55.094868+0000) 2019-10-13T10:12:56.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:56.308+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.314100+0000) 2019-10-13T10:12:56.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:56.544+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.549388+0000) 2019-10-13T10:12:56.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:56.544+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.549995+0000) 2019-10-13T10:12:56.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:12:56.544+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.550479+0000) 2019-10-13T10:12:56.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:12:56.547+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.550866+0000) 2019-10-13T10:12:56.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:12:56.548+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.552135+0000) 2019-10-13T10:12:56.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:12:56.548+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.552497+0000) 2019-10-13T10:12:56.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:12:56.548+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.552875+0000) 2019-10-13T10:12:56.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:12:56.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:12:56.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:12:56.555+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.560564+0000) 2019-10-13T10:12:58.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:12:58.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:58.577281+0000) 2019-10-13T10:12:58.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:12:58.580+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:58.583238+0000) 2019-10-13T10:13:01.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:01.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.557458+0000) 2019-10-13T10:13:01.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:01.552+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.557788+0000) 2019-10-13T10:13:01.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:01.552+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.558422+0000) 2019-10-13T10:13:01.560 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:01.554+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.559555+0000) 2019-10-13T10:13:01.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:01.556+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.560134+0000) 2019-10-13T10:13:01.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:01.556+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.560320+0000) 2019-10-13T10:13:01.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:01.556+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.560782+0000) 2019-10-13T10:13:01.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:01.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:01.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:01.558+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.564660+0000) 2019-10-13T10:13:01.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:01.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.875278+0000) 2019-10-13T10:13:02.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:02.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.067688+0000) 2019-10-13T10:13:03.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:03.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:03.065364+0000) 2019-10-13T10:13:04.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:04.506+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.508798+0000) 2019-10-13T10:13:05.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:05.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:05.095055+0000) 2019-10-13T10:13:06.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:06.308+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.314296+0000) 2019-10-13T10:13:06.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:06.560+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.566201+0000) 2019-10-13T10:13:06.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:06.560+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.567007+0000) 2019-10-13T10:13:06.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:06.560+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.566981+0000) 2019-10-13T10:13:06.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:06.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.567477+0000) 2019-10-13T10:13:06.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:06.564+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.568188+0000) 2019-10-13T10:13:06.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:06.564+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.568582+0000) 2019-10-13T10:13:06.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:06.564+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.569398+0000) 2019-10-13T10:13:06.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:06.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:06.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:06.566+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.571775+0000) 2019-10-13T10:13:08.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:08.572+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:08.577490+0000) 2019-10-13T10:13:08.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:08.580+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:08.583457+0000) 2019-10-13T10:13:11.575 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:11.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.573734+0000) 2019-10-13T10:13:11.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:11.568+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.574530+0000) 2019-10-13T10:13:11.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:11.568+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.575223+0000) 2019-10-13T10:13:11.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:11.572+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.575603+0000) 2019-10-13T10:13:11.577 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:11.572+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.576081+0000) 2019-10-13T10:13:11.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:11.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.576274+0000) 2019-10-13T10:13:11.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:11.572+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.577065+0000) 2019-10-13T10:13:11.579 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:11.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:11.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:11.578+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.582629+0000) 2019-10-13T10:13:11.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:11.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.875522+0000) 2019-10-13T10:13:12.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:12.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.067880+0000) 2019-10-13T10:13:13.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:13.058+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:13.065579+0000) 2019-10-13T10:13:14.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:14.514+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.515130+0000) 2019-10-13T10:13:15.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:15.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:15.095247+0000) 2019-10-13T10:13:16.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:16.312+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.314518+0000) 2019-10-13T10:13:16.582 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:16.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.581158+0000) 2019-10-13T10:13:16.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:16.576+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.581961+0000) 2019-10-13T10:13:16.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:16.576+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.582658+0000) 2019-10-13T10:13:16.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:16.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.583161+0000) 2019-10-13T10:13:16.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:16.580+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.583381+0000) 2019-10-13T10:13:16.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:16.580+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.583909+0000) 2019-10-13T10:13:16.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:16.580+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.584467+0000) 2019-10-13T10:13:16.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:16.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:16.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:16.586+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.593367+0000) 2019-10-13T10:13:18.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:18.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:18.577749+0000) 2019-10-13T10:13:18.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:18.580+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:18.583681+0000) 2019-10-13T10:13:21.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:21.584+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.590528+0000) 2019-10-13T10:13:21.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:21.586+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.590855+0000) 2019-10-13T10:13:21.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:21.584+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.591259+0000) 2019-10-13T10:13:21.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:21.588+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.591574+0000) 2019-10-13T10:13:21.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:21.588+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.591824+0000) 2019-10-13T10:13:21.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:21.588+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.592374+0000) 2019-10-13T10:13:21.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:21.588+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.593132+0000) 2019-10-13T10:13:21.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:21.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:21.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:21.590+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.597545+0000) 2019-10-13T10:13:21.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:21.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.875711+0000) 2019-10-13T10:13:22.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:22.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.068057+0000) 2019-10-13T10:13:23.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:23.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:23.065772+0000) 2019-10-13T10:13:24.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:24.518+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.521508+0000) 2019-10-13T10:13:24.653 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:13:24.653 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:24.657 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:13:24.657 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:25.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:25.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:25.095508+0000) 2019-10-13T10:13:26.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:26.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.314712+0000) 2019-10-13T10:13:26.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:26.591+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.597434+0000) 2019-10-13T10:13:26.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:26.591+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.597967+0000) 2019-10-13T10:13:26.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:26.591+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.598498+0000) 2019-10-13T10:13:26.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:26.594+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.598780+0000) 2019-10-13T10:13:26.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:26.595+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.600583+0000) 2019-10-13T10:13:26.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:26.595+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.601512+0000) 2019-10-13T10:13:26.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:26.595+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.601224+0000) 2019-10-13T10:13:26.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:26.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:26.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:26.598+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.605788+0000) 2019-10-13T10:13:28.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:28.575+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:28.577950+0000) 2019-10-13T10:13:28.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:28.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:28.583912+0000) 2019-10-13T10:13:31.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:31.599+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.605862+0000) 2019-10-13T10:13:31.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:31.599+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.606400+0000) 2019-10-13T10:13:31.608 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:31.602+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.606921+0000) 2019-10-13T10:13:31.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:31.599+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.607125+0000) 2019-10-13T10:13:31.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:31.603+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.607888+0000) 2019-10-13T10:13:31.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:31.603+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.608283+0000) 2019-10-13T10:13:31.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:31.603+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.609105+0000) 2019-10-13T10:13:31.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:31.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:31.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:31.606+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.611734+0000) 2019-10-13T10:13:31.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:31.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.875915+0000) 2019-10-13T10:13:32.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:32.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.068249+0000) 2019-10-13T10:13:33.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:33.058+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:33.065981+0000) 2019-10-13T10:13:34.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:34.526+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.528265+0000) 2019-10-13T10:13:35.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:35.091+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:35.095718+0000) 2019-10-13T10:13:36.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:36.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.314906+0000) 2019-10-13T10:13:36.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:36.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.613150+0000) 2019-10-13T10:13:36.615 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:36.606+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.613688+0000) 2019-10-13T10:13:36.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:36.607+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.614647+0000) 2019-10-13T10:13:36.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:36.607+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.614983+0000) 2019-10-13T10:13:36.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:36.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.615283+0000) 2019-10-13T10:13:36.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:36.611+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.615593+0000) 2019-10-13T10:13:36.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:36.611+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.615843+0000) 2019-10-13T10:13:36.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:36.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:36.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:36.614+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.620483+0000) 2019-10-13T10:13:38.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:38.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:38.578150+0000) 2019-10-13T10:13:38.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:38.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:38.584127+0000) 2019-10-13T10:13:41.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:41.615+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.620589+0000) 2019-10-13T10:13:41.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:41.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.621901+0000) 2019-10-13T10:13:41.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:41.615+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.622284+0000) 2019-10-13T10:13:41.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:41.615+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.622895+0000) 2019-10-13T10:13:41.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:41.619+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.623565+0000) 2019-10-13T10:13:41.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:41.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.623833+0000) 2019-10-13T10:13:41.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:41.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.624797+0000) 2019-10-13T10:13:41.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:41.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:41.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:41.626+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.631478+0000) 2019-10-13T10:13:41.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:41.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.876091+0000) 2019-10-13T10:13:42.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:42.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.068424+0000) 2019-10-13T10:13:43.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:43.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:43.066190+0000) 2019-10-13T10:13:44.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:44.534+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.537213+0000) 2019-10-13T10:13:45.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:45.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:45.095892+0000) 2019-10-13T10:13:46.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:46.307+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.315108+0000) 2019-10-13T10:13:46.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:46.623+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.629274+0000) 2019-10-13T10:13:46.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:46.623+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.629971+0000) 2019-10-13T10:13:46.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:46.623+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.630494+0000) 2019-10-13T10:13:46.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:46.623+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.631056+0000) 2019-10-13T10:13:46.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:46.627+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.631526+0000) 2019-10-13T10:13:46.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:46.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.631913+0000) 2019-10-13T10:13:46.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:46.627+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.632544+0000) 2019-10-13T10:13:46.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:46.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:46.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:46.638+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.639727+0000) 2019-10-13T10:13:48.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:48.571+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:48.578358+0000) 2019-10-13T10:13:48.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:48.579+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:48.584336+0000) 2019-10-13T10:13:51.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:51.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.635714+0000) 2019-10-13T10:13:51.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:51.631+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.636363+0000) 2019-10-13T10:13:51.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:51.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.637836+0000) 2019-10-13T10:13:51.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:51.631+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.638341+0000) 2019-10-13T10:13:51.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:51.631+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.638809+0000) 2019-10-13T10:13:51.640 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:51.638+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.639253+0000) 2019-10-13T10:13:51.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:51.635+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.639777+0000) 2019-10-13T10:13:51.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:51.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:51.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:51.646+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.646287+0000) 2019-10-13T10:13:51.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:51.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.876293+0000) 2019-10-13T10:13:52.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:52.063+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.068606+0000) 2019-10-13T10:13:53.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:53.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:53.066387+0000) 2019-10-13T10:13:54.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:54.541+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.543700+0000) 2019-10-13T10:13:54.773 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:13:54.773 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:54.777 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:13:54.777 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:55.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:55.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:55.096114+0000) 2019-10-13T10:13:56.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:56.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.315278+0000) 2019-10-13T10:13:56.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:13:56.643+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.644435+0000) 2019-10-13T10:13:56.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:13:56.643+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.645490+0000) 2019-10-13T10:13:56.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:56.643+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.645768+0000) 2019-10-13T10:13:56.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:13:56.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.646139+0000) 2019-10-13T10:13:56.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:13:56.645+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.646867+0000) 2019-10-13T10:13:56.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:13:56.647+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.647297+0000) 2019-10-13T10:13:56.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:56.647+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.647638+0000) 2019-10-13T10:13:56.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:13:56.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94676 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:13:56.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:13:56.649+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.653275+0000) 2019-10-13T10:13:58.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:13:58.575+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:58.578553+0000) 2019-10-13T10:13:58.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:13:58.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:58.584527+0000) 2019-10-13T10:14:01.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:01.651+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.652383+0000) 2019-10-13T10:14:01.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:01.651+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.653184+0000) 2019-10-13T10:14:01.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:01.651+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.653859+0000) 2019-10-13T10:14:01.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:01.651+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.654426+0000) 2019-10-13T10:14:01.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:01.653+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.655039+0000) 2019-10-13T10:14:01.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:01.655+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.655395+0000) 2019-10-13T10:14:01.657 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:01.655+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.655710+0000) 2019-10-13T10:14:01.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:01.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:01.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:01.657+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.661890+0000) 2019-10-13T10:14:01.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:01.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.876510+0000) 2019-10-13T10:14:02.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:02.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.068791+0000) 2019-10-13T10:14:03.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:03.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:03.066600+0000) 2019-10-13T10:14:04.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:04.545+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.549995+0000) 2019-10-13T10:14:05.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:05.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:05.096331+0000) 2019-10-13T10:14:06.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:06.315+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.315459+0000) 2019-10-13T10:14:06.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:06.659+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.660588+0000) 2019-10-13T10:14:06.663 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:06.659+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.661659+0000) 2019-10-13T10:14:06.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:06.659+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.661879+0000) 2019-10-13T10:14:06.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:06.659+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.662461+0000) 2019-10-13T10:14:06.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:06.663+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.663396+0000) 2019-10-13T10:14:06.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:06.663+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.664627+0000) 2019-10-13T10:14:06.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:06.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:06.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:06.661+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.664401+0000) 2019-10-13T10:14:06.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:06.665+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.667587+0000) 2019-10-13T10:14:08.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:08.575+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:08.578752+0000) 2019-10-13T10:14:08.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:08.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:08.584720+0000) 2019-10-13T10:14:11.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:11.666+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.668169+0000) 2019-10-13T10:14:11.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:11.666+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.668521+0000) 2019-10-13T10:14:11.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:11.666+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.669437+0000) 2019-10-13T10:14:11.671 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:11.666+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.669548+0000) 2019-10-13T10:14:11.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:11.665+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.669913+0000) 2019-10-13T10:14:11.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:11.666+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.670366+0000) 2019-10-13T10:14:11.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:11.666+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.671091+0000) 2019-10-13T10:14:11.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:11.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:11.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:11.673+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.675106+0000) 2019-10-13T10:14:11.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:11.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.876721+0000) 2019-10-13T10:14:12.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:12.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.068971+0000) 2019-10-13T10:14:13.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:13.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:13.066790+0000) 2019-10-13T10:14:14.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:14.553+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.556478+0000) 2019-10-13T10:14:15.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:15.094+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:15.096527+0000) 2019-10-13T10:14:16.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:16.314+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.315653+0000) 2019-10-13T10:14:16.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:16.674+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.675951+0000) 2019-10-13T10:14:16.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:16.674+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.677489+0000) 2019-10-13T10:14:16.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:16.674+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.677756+0000) 2019-10-13T10:14:16.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:16.674+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.678794+0000) 2019-10-13T10:14:16.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:16.677+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.679013+0000) 2019-10-13T10:14:16.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:16.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.679360+0000) 2019-10-13T10:14:16.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:16.678+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.680593+0000) 2019-10-13T10:14:16.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:16.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94810 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:16.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:16.681+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.683323+0000) 2019-10-13T10:14:18.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:18.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:18.578943+0000) 2019-10-13T10:14:18.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:18.582+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:18.584914+0000) 2019-10-13T10:14:21.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:21.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.685140+0000) 2019-10-13T10:14:21.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:21.682+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.685148+0000) 2019-10-13T10:14:21.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:21.681+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.685896+0000) 2019-10-13T10:14:21.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:21.682+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.686232+0000) 2019-10-13T10:14:21.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:21.682+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.686694+0000) 2019-10-13T10:14:21.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:21.686+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.687553+0000) 2019-10-13T10:14:21.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:21.686+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.688104+0000) 2019-10-13T10:14:21.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:21.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:21.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:21.689+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.692560+0000) 2019-10-13T10:14:21.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:21.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.876936+0000) 2019-10-13T10:14:22.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:22.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.069153+0000) 2019-10-13T10:14:23.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:23.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:23.066997+0000) 2019-10-13T10:14:24.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:24.561+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.563085+0000) 2019-10-13T10:14:24.895 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:14:24.895 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:24.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:14:24.899 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:25.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:25.094+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:25.096703+0000) 2019-10-13T10:14:26.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:26.314+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.315847+0000) 2019-10-13T10:14:26.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:26.690+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.691962+0000) 2019-10-13T10:14:26.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:26.689+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.692561+0000) 2019-10-13T10:14:26.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:26.690+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.692841+0000) 2019-10-13T10:14:26.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:26.690+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.694323+0000) 2019-10-13T10:14:26.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:26.690+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.694718+0000) 2019-10-13T10:14:26.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:26.690+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.695107+0000) 2019-10-13T10:14:26.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:26.694+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.695597+0000) 2019-10-13T10:14:26.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:26.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:26.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:26.701+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.703464+0000) 2019-10-13T10:14:28.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:28.574+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:28.579139+0000) 2019-10-13T10:14:28.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:28.582+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:28.585140+0000) 2019-10-13T10:14:31.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:31.698+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.700098+0000) 2019-10-13T10:14:31.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:31.697+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.701275+0000) 2019-10-13T10:14:31.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:31.698+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.701506+0000) 2019-10-13T10:14:31.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:31.698+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.701887+0000) 2019-10-13T10:14:31.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:31.698+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.702715+0000) 2019-10-13T10:14:31.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:31.702+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.703524+0000) 2019-10-13T10:14:31.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:31.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.704089+0000) 2019-10-13T10:14:31.705 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:31.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:31.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:31.709+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.713222+0000) 2019-10-13T10:14:31.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:31.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.877166+0000) 2019-10-13T10:14:32.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:32.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.069334+0000) 2019-10-13T10:14:33.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:33.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:33.067205+0000) 2019-10-13T10:14:34.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:34.565+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.569521+0000) 2019-10-13T10:14:35.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:35.094+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:35.096909+0000) 2019-10-13T10:14:36.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:36.314+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.316084+0000) 2019-10-13T10:14:36.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:36.706+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.708211+0000) 2019-10-13T10:14:36.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:36.705+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.709728+0000) 2019-10-13T10:14:36.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:36.706+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.710034+0000) 2019-10-13T10:14:36.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:36.706+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.710878+0000) 2019-10-13T10:14:36.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:36.710+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.711355+0000) 2019-10-13T10:14:36.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:36.710+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.711812+0000) 2019-10-13T10:14:36.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:36.710+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.712131+0000) 2019-10-13T10:14:36.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:36.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:36.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:36.717+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.718657+0000) 2019-10-13T10:14:38.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:38.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:38.579343+0000) 2019-10-13T10:14:38.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:38.582+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:38.585339+0000) 2019-10-13T10:14:41.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:41.714+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.716270+0000) 2019-10-13T10:14:41.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:41.714+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.715844+0000) 2019-10-13T10:14:41.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:41.714+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.717263+0000) 2019-10-13T10:14:41.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:41.714+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.717608+0000) 2019-10-13T10:14:41.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:41.717+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.718382+0000) 2019-10-13T10:14:41.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:41.714+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.718673+0000) 2019-10-13T10:14:41.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:41.718+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.719478+0000) 2019-10-13T10:14:41.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:41.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 94976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:41.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:41.721+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.725614+0000) 2019-10-13T10:14:41.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:41.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.877382+0000) 2019-10-13T10:14:42.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:42.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.069521+0000) 2019-10-13T10:14:43.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:43.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:43.067402+0000) 2019-10-13T10:14:44.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:44.573+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.576127+0000) 2019-10-13T10:14:45.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:45.094+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:45.097070+0000) 2019-10-13T10:14:46.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:46.314+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.316275+0000) 2019-10-13T10:14:46.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:46.722+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.723855+0000) 2019-10-13T10:14:46.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:46.722+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.724428+0000) 2019-10-13T10:14:46.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:46.722+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.724878+0000) 2019-10-13T10:14:46.726 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:46.722+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.725311+0000) 2019-10-13T10:14:46.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:46.722+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.726013+0000) 2019-10-13T10:14:46.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:46.725+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.726927+0000) 2019-10-13T10:14:46.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:46.722+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.725663+0000) 2019-10-13T10:14:46.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:46.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:46.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:46.729+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.733249+0000) 2019-10-13T10:14:48.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:48.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:48.579499+0000) 2019-10-13T10:14:48.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:48.582+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:48.585540+0000) 2019-10-13T10:14:51.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:51.730+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.731992+0000) 2019-10-13T10:14:51.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:51.730+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.732894+0000) 2019-10-13T10:14:51.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:51.730+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.732519+0000) 2019-10-13T10:14:51.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:51.730+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.733618+0000) 2019-10-13T10:14:51.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:51.730+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.734532+0000) 2019-10-13T10:14:51.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:51.730+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.734791+0000) 2019-10-13T10:14:51.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:51.732+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.735720+0000) 2019-10-13T10:14:51.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:51.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:51.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:51.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.741509+0000) 2019-10-13T10:14:51.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:51.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.877587+0000) 2019-10-13T10:14:52.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:52.066+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.069738+0000) 2019-10-13T10:14:53.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:53.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:53.067598+0000) 2019-10-13T10:14:54.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:54.584+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.587125+0000) 2019-10-13T10:14:54.969 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:14:54.969 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:54.973 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:14:54.973 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:55.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:55.093+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:55.097289+0000) 2019-10-13T10:14:56.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:56.313+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.316496+0000) 2019-10-13T10:14:56.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:14:56.737+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.739838+0000) 2019-10-13T10:14:56.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:14:56.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.739927+0000) 2019-10-13T10:14:56.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:14:56.737+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.740540+0000) 2019-10-13T10:14:56.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:56.737+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.740869+0000) 2019-10-13T10:14:56.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:14:56.740+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.742192+0000) 2019-10-13T10:14:56.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:14:56.737+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.742787+0000) 2019-10-13T10:14:56.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:56.737+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.743077+0000) 2019-10-13T10:14:56.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:14:56.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:14:56.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:14:56.744+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.748838+0000) 2019-10-13T10:14:58.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:14:58.577+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:58.579697+0000) 2019-10-13T10:14:58.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:14:58.581+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:58.585735+0000) 2019-10-13T10:15:01.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:01.744+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.747994+0000) 2019-10-13T10:15:01.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:01.745+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.748276+0000) 2019-10-13T10:15:01.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:01.745+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.748583+0000) 2019-10-13T10:15:01.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:01.745+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.749363+0000) 2019-10-13T10:15:01.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:01.745+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.749805+0000) 2019-10-13T10:15:01.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:01.745+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.750582+0000) 2019-10-13T10:15:01.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:01.745+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.750963+0000) 2019-10-13T10:15:01.754 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:01.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:01.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:01.756+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.758229+0000) 2019-10-13T10:15:01.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:01.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.877773+0000) 2019-10-13T10:15:02.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:02.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.069911+0000) 2019-10-13T10:15:03.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:03.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:03.067791+0000) 2019-10-13T10:15:04.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:04.596+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.598649+0000) 2019-10-13T10:15:05.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:05.093+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:05.097562+0000) 2019-10-13T10:15:06.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:06.313+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.316692+0000) 2019-10-13T10:15:06.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:06.753+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.756374+0000) 2019-10-13T10:15:06.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:06.752+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.756935+0000) 2019-10-13T10:15:06.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:06.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.757166+0000) 2019-10-13T10:15:06.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:06.753+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.757601+0000) 2019-10-13T10:15:06.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:06.753+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.758100+0000) 2019-10-13T10:15:06.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:06.753+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.758560+0000) 2019-10-13T10:15:06.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:06.753+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.758858+0000) 2019-10-13T10:15:06.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:06.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:06.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:06.764+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.768873+0000) 2019-10-13T10:15:08.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:08.577+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:08.579896+0000) 2019-10-13T10:15:08.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:08.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:08.585931+0000) 2019-10-13T10:15:11.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:11.760+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.763743+0000) 2019-10-13T10:15:11.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:11.761+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.764303+0000) 2019-10-13T10:15:11.766 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:11.761+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.765037+0000) 2019-10-13T10:15:11.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:11.761+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.765334+0000) 2019-10-13T10:15:11.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:11.761+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.765695+0000) 2019-10-13T10:15:11.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:11.761+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.766200+0000) 2019-10-13T10:15:11.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:11.761+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.766434+0000) 2019-10-13T10:15:11.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:11.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:11.775 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:11.772+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.774422+0000) 2019-10-13T10:15:11.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:11.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.877956+0000) 2019-10-13T10:15:12.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:12.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.070041+0000) 2019-10-13T10:15:13.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:13.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:13.067982+0000) 2019-10-13T10:15:14.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:14.608+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.610338+0000) 2019-10-13T10:15:15.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:15.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:15.097796+0000) 2019-10-13T10:15:16.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:16.313+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.316911+0000) 2019-10-13T10:15:16.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:16.769+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.771589+0000) 2019-10-13T10:15:16.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:16.769+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.771959+0000) 2019-10-13T10:15:16.774 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:16.769+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.772776+0000) 2019-10-13T10:15:16.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:16.769+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.773168+0000) 2019-10-13T10:15:16.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:16.769+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.774227+0000) 2019-10-13T10:15:16.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:16.772+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.774895+0000) 2019-10-13T10:15:16.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:16.773+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.775389+0000) 2019-10-13T10:15:16.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:16.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:16.782 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:16.776+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.781419+0000) 2019-10-13T10:15:18.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:18.577+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:18.580094+0000) 2019-10-13T10:15:18.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:18.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:18.586131+0000) 2019-10-13T10:15:21.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:21.777+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.779622+0000) 2019-10-13T10:15:21.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:21.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.780280+0000) 2019-10-13T10:15:21.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:21.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.780774+0000) 2019-10-13T10:15:21.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:21.777+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.781071+0000) 2019-10-13T10:15:21.783 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:21.777+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.781548+0000) 2019-10-13T10:15:21.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:21.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.782159+0000) 2019-10-13T10:15:21.784 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:21.780+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.783051+0000) 2019-10-13T10:15:21.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:21.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:21.788 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:21.784+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.786860+0000) 2019-10-13T10:15:21.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:21.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.878142+0000) 2019-10-13T10:15:22.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:22.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.070217+0000) 2019-10-13T10:15:23.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:23.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:23.068173+0000) 2019-10-13T10:15:24.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:24.616+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.616686+0000) 2019-10-13T10:15:25.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:25.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:25.098042+0000) 2019-10-13T10:15:25.149 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:15:25.149 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:25.153 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:15:25.153 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:26.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:26.313+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.317132+0000) 2019-10-13T10:15:26.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:26.785+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.788045+0000) 2019-10-13T10:15:26.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:26.785+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.788479+0000) 2019-10-13T10:15:26.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:26.785+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.789374+0000) 2019-10-13T10:15:26.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:26.785+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.790081+0000) 2019-10-13T10:15:26.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:26.785+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.790448+0000) 2019-10-13T10:15:26.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:26.788+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.791408+0000) 2019-10-13T10:15:26.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:26.789+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.791860+0000) 2019-10-13T10:15:26.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:26.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:26.797 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:26.792+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.796191+0000) 2019-10-13T10:15:28.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:28.577+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:28.580274+0000) 2019-10-13T10:15:28.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:28.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:28.586329+0000) 2019-10-13T10:15:31.798 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:31.793+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.796833+0000) 2019-10-13T10:15:31.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:31.793+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.797220+0000) 2019-10-13T10:15:31.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:31.793+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.798385+0000) 2019-10-13T10:15:31.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:31.796+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.798489+0000) 2019-10-13T10:15:31.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:31.793+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.798759+0000) 2019-10-13T10:15:31.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:31.793+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.799115+0000) 2019-10-13T10:15:31.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:31.797+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.799509+0000) 2019-10-13T10:15:31.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:31.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:31.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:31.804+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.806688+0000) 2019-10-13T10:15:31.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:31.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.878329+0000) 2019-10-13T10:15:32.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:32.065+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.070410+0000) 2019-10-13T10:15:33.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:33.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:33.068368+0000) 2019-10-13T10:15:34.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:34.624+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.625765+0000) 2019-10-13T10:15:35.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:35.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:35.098244+0000) 2019-10-13T10:15:36.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:36.317+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.317327+0000) 2019-10-13T10:15:36.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:36.800+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.804385+0000) 2019-10-13T10:15:36.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:36.800+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.805439+0000) 2019-10-13T10:15:36.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:36.800+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.806205+0000) 2019-10-13T10:15:36.808 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:36.804+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.806662+0000) 2019-10-13T10:15:36.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:36.800+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.806864+0000) 2019-10-13T10:15:36.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:36.800+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.807083+0000) 2019-10-13T10:15:36.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:36.804+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.807528+0000) 2019-10-13T10:15:36.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:36.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:36.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:36.808+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.811729+0000) 2019-10-13T10:15:38.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:38.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:38.580477+0000) 2019-10-13T10:15:38.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:38.580+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:38.586556+0000) 2019-10-13T10:15:41.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:41.808+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.811733+0000) 2019-10-13T10:15:41.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:41.808+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.813303+0000) 2019-10-13T10:15:41.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:41.808+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.812898+0000) 2019-10-13T10:15:41.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:41.808+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.813779+0000) 2019-10-13T10:15:41.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:41.808+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.814288+0000) 2019-10-13T10:15:41.816 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:41.808+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.814719+0000) 2019-10-13T10:15:41.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:41.812+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.815092+0000) 2019-10-13T10:15:41.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:41.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:41.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:41.820+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.822606+0000) 2019-10-13T10:15:41.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:41.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.878545+0000) 2019-10-13T10:15:42.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:42.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.070588+0000) 2019-10-13T10:15:43.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:43.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:43.068556+0000) 2019-10-13T10:15:44.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:44.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.637401+0000) 2019-10-13T10:15:45.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:45.096+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:45.098480+0000) 2019-10-13T10:15:46.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:46.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.317519+0000) 2019-10-13T10:15:46.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:46.816+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.819373+0000) 2019-10-13T10:15:46.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:46.816+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.820120+0000) 2019-10-13T10:15:46.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:46.816+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.820932+0000) 2019-10-13T10:15:46.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:46.815+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.821438+0000) 2019-10-13T10:15:46.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:46.816+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.821681+0000) 2019-10-13T10:15:46.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:46.816+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.821942+0000) 2019-10-13T10:15:46.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:46.816+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.822266+0000) 2019-10-13T10:15:46.825 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:46.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:46.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:46.827+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.830688+0000) 2019-10-13T10:15:48.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:48.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:48.580706+0000) 2019-10-13T10:15:48.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:48.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:48.586749+0000) 2019-10-13T10:15:51.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:51.824+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.827925+0000) 2019-10-13T10:15:51.830 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:51.824+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.828477+0000) 2019-10-13T10:15:51.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:51.824+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.828612+0000) 2019-10-13T10:15:51.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:51.824+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.829298+0000) 2019-10-13T10:15:51.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:51.824+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.829526+0000) 2019-10-13T10:15:51.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:51.824+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.829979+0000) 2019-10-13T10:15:51.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:51.827+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.830579+0000) 2019-10-13T10:15:51.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:51.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:51.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:51.831+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.837368+0000) 2019-10-13T10:15:51.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:51.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.878731+0000) 2019-10-13T10:15:52.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:52.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.070756+0000) 2019-10-13T10:15:53.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:53.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:53.068780+0000) 2019-10-13T10:15:54.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:54.639+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.643851+0000) 2019-10-13T10:15:55.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:55.092+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:55.098674+0000) 2019-10-13T10:15:55.198 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:15:55.198 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:55.202 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:15:55.202 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:56.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:56.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.317712+0000) 2019-10-13T10:15:56.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:15:56.832+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.835942+0000) 2019-10-13T10:15:56.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:56.832+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.836300+0000) 2019-10-13T10:15:56.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:15:56.832+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.837182+0000) 2019-10-13T10:15:56.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:15:56.832+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.837690+0000) 2019-10-13T10:15:56.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:15:56.832+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.838289+0000) 2019-10-13T10:15:56.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:15:56.835+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.838921+0000) 2019-10-13T10:15:56.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:56.832+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.839047+0000) 2019-10-13T10:15:56.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:15:56.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95481 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:15:56.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:15:56.843+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.847354+0000) 2019-10-13T10:15:58.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:15:58.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:58.580860+0000) 2019-10-13T10:15:58.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:15:58.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:58.586948+0000) 2019-10-13T10:16:01.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:01.836+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.842544+0000) 2019-10-13T10:16:01.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:01.836+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.843222+0000) 2019-10-13T10:16:01.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:01.840+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.844110+0000) 2019-10-13T10:16:01.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:01.839+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.844465+0000) 2019-10-13T10:16:01.846 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:01.840+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.845271+0000) 2019-10-13T10:16:01.847 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:01.840+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.845928+0000) 2019-10-13T10:16:01.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:01.840+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.846088+0000) 2019-10-13T10:16:01.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:01.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:01.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:01.847+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.851815+0000) 2019-10-13T10:16:01.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:01.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.878922+0000) 2019-10-13T10:16:02.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:02.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.070933+0000) 2019-10-13T10:16:03.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:03.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:03.068995+0000) 2019-10-13T10:16:04.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:04.651+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.654796+0000) 2019-10-13T10:16:05.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:05.096+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:05.098903+0000) 2019-10-13T10:16:06.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:06.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.317908+0000) 2019-10-13T10:16:06.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:06.844+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.850641+0000) 2019-10-13T10:16:06.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:06.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.851363+0000) 2019-10-13T10:16:06.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:06.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.851823+0000) 2019-10-13T10:16:06.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:06.848+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.852163+0000) 2019-10-13T10:16:06.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:06.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.852980+0000) 2019-10-13T10:16:06.854 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:06.848+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.852556+0000) 2019-10-13T10:16:06.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:06.848+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.853850+0000) 2019-10-13T10:16:06.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:06.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:06.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:06.859+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.862949+0000) 2019-10-13T10:16:08.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:08.576+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:08.581087+0000) 2019-10-13T10:16:08.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:08.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:08.587149+0000) 2019-10-13T10:16:11.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:11.852+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.858374+0000) 2019-10-13T10:16:11.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:11.852+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.858738+0000) 2019-10-13T10:16:11.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:11.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.859610+0000) 2019-10-13T10:16:11.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:11.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.859956+0000) 2019-10-13T10:16:11.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:11.856+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.860426+0000) 2019-10-13T10:16:11.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:11.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.860922+0000) 2019-10-13T10:16:11.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:11.856+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.861240+0000) 2019-10-13T10:16:11.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:11.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:11.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:11.863+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.868051+0000) 2019-10-13T10:16:11.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:11.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.879133+0000) 2019-10-13T10:16:12.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:12.064+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.071115+0000) 2019-10-13T10:16:13.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:13.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:13.069182+0000) 2019-10-13T10:16:14.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:14.659+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.661173+0000) 2019-10-13T10:16:15.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:15.096+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:15.099104+0000) 2019-10-13T10:16:16.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:16.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.318107+0000) 2019-10-13T10:16:16.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:16.860+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.866570+0000) 2019-10-13T10:16:16.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:16.864+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.867690+0000) 2019-10-13T10:16:16.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:16.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.867840+0000) 2019-10-13T10:16:16.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:16.864+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.868086+0000) 2019-10-13T10:16:16.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:16.864+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.868665+0000) 2019-10-13T10:16:16.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:16.864+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.868985+0000) 2019-10-13T10:16:16.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:16.864+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.869369+0000) 2019-10-13T10:16:16.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:16.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:16.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:16.871+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:16.876912+0000) 2019-10-13T10:16:18.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:18.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:18.581305+0000) 2019-10-13T10:16:18.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:18.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:18.587344+0000) 2019-10-13T10:16:21.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:21.871+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.874929+0000) 2019-10-13T10:16:21.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:21.871+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.875543+0000) 2019-10-13T10:16:21.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:21.871+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.875807+0000) 2019-10-13T10:16:21.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:21.871+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.876073+0000) 2019-10-13T10:16:21.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:21.871+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.876781+0000) 2019-10-13T10:16:21.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:21.871+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.876986+0000) 2019-10-13T10:16:21.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:21.871+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.877979+0000) 2019-10-13T10:16:21.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:21.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:21.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:21.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.879258+0000) 2019-10-13T10:16:21.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:21.875+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.881563+0000) 2019-10-13T10:16:22.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:22.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.071344+0000) 2019-10-13T10:16:23.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:23.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:23.069378+0000) 2019-10-13T10:16:24.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:24.663+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.667568+0000) 2019-10-13T10:16:25.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:25.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:25.099350+0000) 2019-10-13T10:16:25.312 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:16:25.313 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:25.317 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:16:25.317 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:26.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:26.315+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.318301+0000) 2019-10-13T10:16:26.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:26.875+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.882012+0000) 2019-10-13T10:16:26.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:26.879+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.882820+0000) 2019-10-13T10:16:26.884 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:26.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.883085+0000) 2019-10-13T10:16:26.885 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:26.879+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.883434+0000) 2019-10-13T10:16:26.885 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:26.879+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.884216+0000) 2019-10-13T10:16:26.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:26.879+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.884792+0000) 2019-10-13T10:16:26.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:26.879+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.885404+0000) 2019-10-13T10:16:26.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:26.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:26.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:26.887+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:26.891168+0000) 2019-10-13T10:16:28.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:28.575+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:28.581511+0000) 2019-10-13T10:16:28.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:28.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:28.587536+0000) 2019-10-13T10:16:31.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:31.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.879462+0000) 2019-10-13T10:16:31.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:31.883+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.889894+0000) 2019-10-13T10:16:31.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:31.883+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.890128+0000) 2019-10-13T10:16:31.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:31.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.890584+0000) 2019-10-13T10:16:31.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:31.883+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.891026+0000) 2019-10-13T10:16:31.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:31.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.891548+0000) 2019-10-13T10:16:31.895 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:31.887+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.892287+0000) 2019-10-13T10:16:31.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:31.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.892690+0000) 2019-10-13T10:16:31.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:31.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:31.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:31.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.898654+0000) 2019-10-13T10:16:32.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:32.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.071491+0000) 2019-10-13T10:16:33.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:33.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.069574+0000) 2019-10-13T10:16:34.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:34.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.674105+0000) 2019-10-13T10:16:35.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:35.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:35.099594+0000) 2019-10-13T10:16:36.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:36.315+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.318500+0000) 2019-10-13T10:16:36.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:36.891+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.896753+0000) 2019-10-13T10:16:36.898 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:36.891+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.897235+0000) 2019-10-13T10:16:36.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:36.891+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.898077+0000) 2019-10-13T10:16:36.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:36.891+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.898570+0000) 2019-10-13T10:16:36.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:36.891+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.898309+0000) 2019-10-13T10:16:36.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:36.895+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.899554+0000) 2019-10-13T10:16:36.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:36.895+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.899969+0000) 2019-10-13T10:16:36.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:36.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:36.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:36.903+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:36.906897+0000) 2019-10-13T10:16:38.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:38.579+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.581743+0000) 2019-10-13T10:16:38.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:38.587+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.587736+0000) 2019-10-13T10:16:41.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:41.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.879694+0000) 2019-10-13T10:16:41.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:41.899+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.905278+0000) 2019-10-13T10:16:41.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:41.899+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.905402+0000) 2019-10-13T10:16:41.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:41.899+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.905857+0000) 2019-10-13T10:16:41.907 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:41.899+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.906170+0000) 2019-10-13T10:16:41.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:41.902+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.907005+0000) 2019-10-13T10:16:41.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:41.899+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.907252+0000) 2019-10-13T10:16:41.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:41.903+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.908645+0000) 2019-10-13T10:16:41.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:41.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:41.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:41.906+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.913614+0000) 2019-10-13T10:16:42.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:42.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.071630+0000) 2019-10-13T10:16:43.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:43.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.069798+0000) 2019-10-13T10:16:44.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:44.678+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.679932+0000) 2019-10-13T10:16:45.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:45.099+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:45.099800+0000) 2019-10-13T10:16:46.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:46.311+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.318693+0000) 2019-10-13T10:16:46.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:46.907+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.912414+0000) 2019-10-13T10:16:46.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:46.907+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.913546+0000) 2019-10-13T10:16:46.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:46.907+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.913656+0000) 2019-10-13T10:16:46.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:46.910+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.915190+0000) 2019-10-13T10:16:46.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:46.911+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.915542+0000) 2019-10-13T10:16:46.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:46.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.916107+0000) 2019-10-13T10:16:46.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:46.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:46.918 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:46.911+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.916795+0000) 2019-10-13T10:16:46.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:46.914+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:46.920892+0000) 2019-10-13T10:16:48.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:48.579+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.581939+0000) 2019-10-13T10:16:48.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:48.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.587929+0000) 2019-10-13T10:16:51.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:51.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.879895+0000) 2019-10-13T10:16:51.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:51.914+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.920605+0000) 2019-10-13T10:16:51.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:51.919+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.920953+0000) 2019-10-13T10:16:51.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:51.919+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.921414+0000) 2019-10-13T10:16:51.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:51.919+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.921630+0000) 2019-10-13T10:16:51.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:51.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.921937+0000) 2019-10-13T10:16:51.924 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:51.919+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.922586+0000) 2019-10-13T10:16:51.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:51.923+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.923489+0000) 2019-10-13T10:16:51.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:51.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:51.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:51.922+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.927162+0000) 2019-10-13T10:16:52.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:52.071+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.071846+0000) 2019-10-13T10:16:53.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:53.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.069992+0000) 2019-10-13T10:16:54.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:54.686+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.690880+0000) 2019-10-13T10:16:55.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:55.099+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:55.100007+0000) 2019-10-13T10:16:55.378 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:16:55.378 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:55.382 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:16:55.382 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:56.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:56.315+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.318890+0000) 2019-10-13T10:16:56.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:56.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.927834+0000) 2019-10-13T10:16:56.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:16:56.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.928574+0000) 2019-10-13T10:16:56.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:16:56.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.928975+0000) 2019-10-13T10:16:56.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:56.927+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.929269+0000) 2019-10-13T10:16:56.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:16:56.927+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.930334+0000) 2019-10-13T10:16:56.932 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:16:56.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.930667+0000) 2019-10-13T10:16:56.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:16:56.927+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.931127+0000) 2019-10-13T10:16:56.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:16:56.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95881 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:16:56.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:16:56.930+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:56.936312+0000) 2019-10-13T10:16:58.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:16:58.579+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.582133+0000) 2019-10-13T10:16:58.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:16:58.587+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.588119+0000) 2019-10-13T10:17:01.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:01.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.880115+0000) 2019-10-13T10:17:01.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:01.935+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.935732+0000) 2019-10-13T10:17:01.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:01.935+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.936565+0000) 2019-10-13T10:17:01.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:01.930+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.937355+0000) 2019-10-13T10:17:01.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:01.935+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.937679+0000) 2019-10-13T10:17:01.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:01.935+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.938234+0000) 2019-10-13T10:17:01.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:01.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.938383+0000) 2019-10-13T10:17:01.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:01.935+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.938554+0000) 2019-10-13T10:17:01.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:01.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95913 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:01.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:01.938+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.946060+0000) 2019-10-13T10:17:02.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:02.071+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.072041+0000) 2019-10-13T10:17:03.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:03.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.070182+0000) 2019-10-13T10:17:04.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:04.690+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.697375+0000) 2019-10-13T10:17:05.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:05.098+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:05.100206+0000) 2019-10-13T10:17:06.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:06.314+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.319109+0000) 2019-10-13T10:17:06.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:06.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.944176+0000) 2019-10-13T10:17:06.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:06.942+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.944911+0000) 2019-10-13T10:17:06.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:06.942+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.945381+0000) 2019-10-13T10:17:06.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:06.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.945773+0000) 2019-10-13T10:17:06.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:06.942+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.946562+0000) 2019-10-13T10:17:06.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:06.942+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.946951+0000) 2019-10-13T10:17:06.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:06.946+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.947699+0000) 2019-10-13T10:17:06.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:06.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:06.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:06.950+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:06.954529+0000) 2019-10-13T10:17:08.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:08.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.582361+0000) 2019-10-13T10:17:08.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:08.586+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.588349+0000) 2019-10-13T10:17:11.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:11.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.880348+0000) 2019-10-13T10:17:11.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:11.950+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.952355+0000) 2019-10-13T10:17:11.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:11.950+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.954450+0000) 2019-10-13T10:17:11.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:11.950+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.953580+0000) 2019-10-13T10:17:11.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:11.950+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.953698+0000) 2019-10-13T10:17:11.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:11.950+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.954716+0000) 2019-10-13T10:17:11.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:11.950+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.955150+0000) 2019-10-13T10:17:11.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:11.954+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.955723+0000) 2019-10-13T10:17:11.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:11.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 95979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:11.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:11.958+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.964529+0000) 2019-10-13T10:17:12.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:12.070+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.072237+0000) 2019-10-13T10:17:13.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:13.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.070376+0000) 2019-10-13T10:17:14.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:14.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.703714+0000) 2019-10-13T10:17:15.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:15.098+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:15.100439+0000) 2019-10-13T10:17:16.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:16.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.319349+0000) 2019-10-13T10:17:16.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:16.958+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.960572+0000) 2019-10-13T10:17:16.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:16.958+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.960893+0000) 2019-10-13T10:17:16.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:16.958+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.961765+0000) 2019-10-13T10:17:16.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:16.954+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.961836+0000) 2019-10-13T10:17:16.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:16.958+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.962668+0000) 2019-10-13T10:17:16.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:16.958+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.963023+0000) 2019-10-13T10:17:16.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:16.962+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.964152+0000) 2019-10-13T10:17:16.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:16.962+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:16.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:16.962+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:16.969911+0000) 2019-10-13T10:17:18.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:18.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.582579+0000) 2019-10-13T10:17:18.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:18.586+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.588545+0000) 2019-10-13T10:17:21.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:21.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.880546+0000) 2019-10-13T10:17:21.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:21.962+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.969458+0000) 2019-10-13T10:17:21.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:21.966+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.968045+0000) 2019-10-13T10:17:21.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:21.966+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.968691+0000) 2019-10-13T10:17:21.973 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:21.966+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.969966+0000) 2019-10-13T10:17:21.973 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:21.966+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.970539+0000) 2019-10-13T10:17:21.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:21.966+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.971169+0000) 2019-10-13T10:17:21.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:21.970+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.972191+0000) 2019-10-13T10:17:21.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:21.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96047 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:21.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:21.974+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.979790+0000) 2019-10-13T10:17:22.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:22.070+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.072458+0000) 2019-10-13T10:17:23.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:23.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.070574+0000) 2019-10-13T10:17:24.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:24.706+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.710207+0000) 2019-10-13T10:17:25.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:25.098+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:25.100677+0000) 2019-10-13T10:17:25.517 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:17:25.517 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:25.520 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:17:25.520 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:26.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:26.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.319585+0000) 2019-10-13T10:17:26.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:26.974+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.977024+0000) 2019-10-13T10:17:26.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:26.974+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.977736+0000) 2019-10-13T10:17:26.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:26.974+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.978327+0000) 2019-10-13T10:17:26.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:26.974+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.978718+0000) 2019-10-13T10:17:26.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:26.978+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.979111+0000) 2019-10-13T10:17:26.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:26.978+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.980144+0000) 2019-10-13T10:17:26.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:26.978+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.980284+0000) 2019-10-13T10:17:26.982 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:26.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:26.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:26.982+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:26.984032+0000) 2019-10-13T10:17:28.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:28.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.582773+0000) 2019-10-13T10:17:28.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:28.586+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.588738+0000) 2019-10-13T10:17:31.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:31.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.880726+0000) 2019-10-13T10:17:31.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:31.982+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.984094+0000) 2019-10-13T10:17:31.986 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:31.982+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.984835+0000) 2019-10-13T10:17:31.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:31.982+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.985250+0000) 2019-10-13T10:17:31.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:31.982+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.986141+0000) 2019-10-13T10:17:31.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:31.986+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.986700+0000) 2019-10-13T10:17:31.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:31.982+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.986985+0000) 2019-10-13T10:17:31.988 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:31.986+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.987339+0000) 2019-10-13T10:17:31.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:31.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:31.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:31.990+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.993673+0000) 2019-10-13T10:17:32.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:32.070+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.072595+0000) 2019-10-13T10:17:33.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:33.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.070783+0000) 2019-10-13T10:17:34.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:34.717+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.722028+0000) 2019-10-13T10:17:35.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:35.098+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:35.100875+0000) 2019-10-13T10:17:36.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:36.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.319771+0000) 2019-10-13T10:17:36.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:36.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.991485+0000) 2019-10-13T10:17:36.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:36.989+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.991828+0000) 2019-10-13T10:17:36.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:36.990+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.992311+0000) 2019-10-13T10:17:36.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:36.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.992837+0000) 2019-10-13T10:17:36.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:36.990+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.993258+0000) 2019-10-13T10:17:36.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:36.990+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.993743+0000) 2019-10-13T10:17:36.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:36.990+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.995253+0000) 2019-10-13T10:17:36.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:36.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:37.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:36.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:36.999234+0000) 2019-10-13T10:17:38.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:38.578+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.583007+0000) 2019-10-13T10:17:38.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:38.586+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.588928+0000) 2019-10-13T10:17:41.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:41.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:41.880897+0000) 2019-10-13T10:17:42.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:41.994+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:41.999175+0000) 2019-10-13T10:17:42.001 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:41.998+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:41.999589+0000) 2019-10-13T10:17:42.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:41.998+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.001195+0000) 2019-10-13T10:17:42.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:41.998+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.001598+0000) 2019-10-13T10:17:42.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:41.998+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.001713+0000) 2019-10-13T10:17:42.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:42.001+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.002178+0000) 2019-10-13T10:17:42.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:41.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.002552+0000) 2019-10-13T10:17:42.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:42.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:42.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:42.009+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.010568+0000) 2019-10-13T10:17:42.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:42.070+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.072785+0000) 2019-10-13T10:17:43.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:43.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.070994+0000) 2019-10-13T10:17:44.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:44.729+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.733363+0000) 2019-10-13T10:17:45.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:45.098+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:45.101110+0000) 2019-10-13T10:17:46.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:46.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:46.319978+0000) 2019-10-13T10:17:47.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:47.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.007209+0000) 2019-10-13T10:17:47.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:47.006+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.007776+0000) 2019-10-13T10:17:47.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:47.006+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.008617+0000) 2019-10-13T10:17:47.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:47.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.008934+0000) 2019-10-13T10:17:47.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:47.006+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.009245+0000) 2019-10-13T10:17:47.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:47.006+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.009691+0000) 2019-10-13T10:17:47.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:47.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:47.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:47.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.010601+0000) 2019-10-13T10:17:47.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:47.013+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.017435+0000) 2019-10-13T10:17:48.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:48.577+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.583243+0000) 2019-10-13T10:17:48.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:48.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.589126+0000) 2019-10-13T10:17:51.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:51.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:51.881101+0000) 2019-10-13T10:17:52.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:52.009+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.015090+0000) 2019-10-13T10:17:52.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:52.013+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.015554+0000) 2019-10-13T10:17:52.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:52.013+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.016124+0000) 2019-10-13T10:17:52.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:52.013+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.016821+0000) 2019-10-13T10:17:52.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:52.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.017213+0000) 2019-10-13T10:17:52.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:52.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.018000+0000) 2019-10-13T10:17:52.019 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:52.013+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.018322+0000) 2019-10-13T10:17:52.021 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:52.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:52.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:52.021+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.023155+0000) 2019-10-13T10:17:52.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:52.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.073009+0000) 2019-10-13T10:17:53.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:53.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.071160+0000) 2019-10-13T10:17:54.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:54.737+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.739300+0000) 2019-10-13T10:17:55.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:55.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:55.101325+0000) 2019-10-13T10:17:55.647 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:17:55.647 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:55.651 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:17:55.651 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:56.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:56.317+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:56.320170+0000) 2019-10-13T10:17:57.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:17:57.017+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.023176+0000) 2019-10-13T10:17:57.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:17:57.021+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.023533+0000) 2019-10-13T10:17:57.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:57.021+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.024136+0000) 2019-10-13T10:17:57.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:17:57.021+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.024469+0000) 2019-10-13T10:17:57.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:17:57.021+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.025441+0000) 2019-10-13T10:17:57.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:17:57.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.024787+0000) 2019-10-13T10:17:57.028 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:57.021+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.026392+0000) 2019-10-13T10:17:57.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:17:57.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:17:57.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:17:57.029+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.032264+0000) 2019-10-13T10:17:58.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:17:58.581+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.583511+0000) 2019-10-13T10:17:58.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:17:58.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.589344+0000) 2019-10-13T10:18:01.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:01.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:01.881299+0000) 2019-10-13T10:18:02.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:02.025+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.031056+0000) 2019-10-13T10:18:02.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:02.029+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.031801+0000) 2019-10-13T10:18:02.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:02.029+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.031982+0000) 2019-10-13T10:18:02.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:02.029+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.032562+0000) 2019-10-13T10:18:02.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:02.029+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.034015+0000) 2019-10-13T10:18:02.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:02.029+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.034962+0000) 2019-10-13T10:18:02.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:02.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.033867+0000) 2019-10-13T10:18:02.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:02.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:02.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:02.037+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.039154+0000) 2019-10-13T10:18:02.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:02.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.073179+0000) 2019-10-13T10:18:03.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:03.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.071327+0000) 2019-10-13T10:18:04.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:04.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.745736+0000) 2019-10-13T10:18:05.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:05.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:05.101524+0000) 2019-10-13T10:18:06.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:06.317+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:06.320361+0000) 2019-10-13T10:18:07.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:07.033+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.039091+0000) 2019-10-13T10:18:07.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:07.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.040150+0000) 2019-10-13T10:18:07.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:07.037+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.040559+0000) 2019-10-13T10:18:07.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:07.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.040963+0000) 2019-10-13T10:18:07.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:07.037+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.041147+0000) 2019-10-13T10:18:07.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:07.037+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.041875+0000) 2019-10-13T10:18:07.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:07.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.041461+0000) 2019-10-13T10:18:07.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:07.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:07.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:07.049+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.052066+0000) 2019-10-13T10:18:08.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:08.581+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.583710+0000) 2019-10-13T10:18:08.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:08.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.589535+0000) 2019-10-13T10:18:11.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:11.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:11.881513+0000) 2019-10-13T10:18:12.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:12.045+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.047556+0000) 2019-10-13T10:18:12.049 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:12.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.048003+0000) 2019-10-13T10:18:12.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:12.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.048063+0000) 2019-10-13T10:18:12.050 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:12.045+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.048991+0000) 2019-10-13T10:18:12.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:12.045+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.049436+0000) 2019-10-13T10:18:12.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:12.045+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.049974+0000) 2019-10-13T10:18:12.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:12.045+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.050475+0000) 2019-10-13T10:18:12.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:12.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:12.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:12.053+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.055086+0000) 2019-10-13T10:18:12.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:12.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.073387+0000) 2019-10-13T10:18:13.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:13.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.071533+0000) 2019-10-13T10:18:14.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:14.749+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.752119+0000) 2019-10-13T10:18:15.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:15.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:15.101740+0000) 2019-10-13T10:18:16.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:16.317+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:16.320556+0000) 2019-10-13T10:18:17.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:17.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.054600+0000) 2019-10-13T10:18:17.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:17.053+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.055777+0000) 2019-10-13T10:18:17.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:17.053+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.056381+0000) 2019-10-13T10:18:17.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:17.053+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.056518+0000) 2019-10-13T10:18:17.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:17.053+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.056960+0000) 2019-10-13T10:18:17.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:17.053+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.057051+0000) 2019-10-13T10:18:17.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:17.053+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.057481+0000) 2019-10-13T10:18:17.060 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:17.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:17.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:17.065+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.066879+0000) 2019-10-13T10:18:18.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:18.581+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.583932+0000) 2019-10-13T10:18:18.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:18.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.589729+0000) 2019-10-13T10:18:21.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:21.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:21.881759+0000) 2019-10-13T10:18:22.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:22.057+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.062276+0000) 2019-10-13T10:18:22.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:22.057+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.063041+0000) 2019-10-13T10:18:22.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:22.061+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.063497+0000) 2019-10-13T10:18:22.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:22.061+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.063855+0000) 2019-10-13T10:18:22.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:22.061+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.064522+0000) 2019-10-13T10:18:22.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:22.061+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.064756+0000) 2019-10-13T10:18:22.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:22.061+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.065679+0000) 2019-10-13T10:18:22.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:22.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:22.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:22.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.070679+0000) 2019-10-13T10:18:22.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:22.069+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.073513+0000) 2019-10-13T10:18:23.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:23.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.071731+0000) 2019-10-13T10:18:24.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:24.757+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.758473+0000) 2019-10-13T10:18:25.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:25.097+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:25.101996+0000) 2019-10-13T10:18:25.759 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:18:25.759 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:25.764 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:18:25.764 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:26.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:26.317+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:26.320775+0000) 2019-10-13T10:18:27.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:27.069+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.070415+0000) 2019-10-13T10:18:27.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:27.065+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.070698+0000) 2019-10-13T10:18:27.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:27.069+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.071608+0000) 2019-10-13T10:18:27.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:27.069+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.071597+0000) 2019-10-13T10:18:27.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:27.069+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.072119+0000) 2019-10-13T10:18:27.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:27.069+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.072213+0000) 2019-10-13T10:18:27.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:27.069+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.074199+0000) 2019-10-13T10:18:27.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:27.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:27.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:27.077+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.081676+0000) 2019-10-13T10:18:28.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:28.581+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.584127+0000) 2019-10-13T10:18:28.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:28.585+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.589925+0000) 2019-10-13T10:18:31.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:31.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:31.882025+0000) 2019-10-13T10:18:32.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:32.068+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.073681+0000) 2019-10-13T10:18:32.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:32.072+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.078969+0000) 2019-10-13T10:18:32.080 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:32.072+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.079058+0000) 2019-10-13T10:18:32.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:32.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.079583+0000) 2019-10-13T10:18:32.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:32.076+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.079854+0000) 2019-10-13T10:18:32.082 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:32.076+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.081204+0000) 2019-10-13T10:18:32.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:32.076+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.081672+0000) 2019-10-13T10:18:32.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:32.076+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.082118+0000) 2019-10-13T10:18:32.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:32.080+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:32.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:32.084+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.089983+0000) 2019-10-13T10:18:33.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:33.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.071977+0000) 2019-10-13T10:18:34.766 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:34.760+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.764905+0000) 2019-10-13T10:18:35.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:35.096+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:35.102218+0000) 2019-10-13T10:18:36.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:36.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:36.320987+0000) 2019-10-13T10:18:37.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:37.080+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.086576+0000) 2019-10-13T10:18:37.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:37.084+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.087461+0000) 2019-10-13T10:18:37.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:37.084+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.087788+0000) 2019-10-13T10:18:37.090 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:37.084+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.088689+0000) 2019-10-13T10:18:37.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:37.084+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.088228+0000) 2019-10-13T10:18:37.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:37.084+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.090002+0000) 2019-10-13T10:18:37.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:37.084+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.090594+0000) 2019-10-13T10:18:37.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:37.088+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:37.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:37.092+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.096402+0000) 2019-10-13T10:18:38.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:38.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.584326+0000) 2019-10-13T10:18:38.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:38.588+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.590154+0000) 2019-10-13T10:18:41.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:41.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:41.882237+0000) 2019-10-13T10:18:42.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:42.068+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.073866+0000) 2019-10-13T10:18:42.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:42.092+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.095777+0000) 2019-10-13T10:18:42.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:42.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.096299+0000) 2019-10-13T10:18:42.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:42.092+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.097200+0000) 2019-10-13T10:18:42.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:42.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.097616+0000) 2019-10-13T10:18:42.101 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:42.092+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.098165+0000) 2019-10-13T10:18:42.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:42.092+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.098449+0000) 2019-10-13T10:18:42.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:42.096+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.098948+0000) 2019-10-13T10:18:42.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:42.096+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:42.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:42.100+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.102945+0000) 2019-10-13T10:18:43.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:43.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.072184+0000) 2019-10-13T10:18:44.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:44.768+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.771744+0000) 2019-10-13T10:18:45.103 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:45.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:45.102385+0000) 2019-10-13T10:18:46.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:46.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:46.321172+0000) 2019-10-13T10:18:47.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:47.096+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.102421+0000) 2019-10-13T10:18:47.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:47.096+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.103108+0000) 2019-10-13T10:18:47.105 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:47.100+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.103498+0000) 2019-10-13T10:18:47.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:47.100+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.105017+0000) 2019-10-13T10:18:47.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:47.100+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.105525+0000) 2019-10-13T10:18:47.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:47.100+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.106005+0000) 2019-10-13T10:18:47.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:47.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.106543+0000) 2019-10-13T10:18:47.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:47.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:47.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:47.108+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.112225+0000) 2019-10-13T10:18:48.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:48.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.584553+0000) 2019-10-13T10:18:48.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:48.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.590344+0000) 2019-10-13T10:18:51.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:51.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:51.882434+0000) 2019-10-13T10:18:52.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:52.072+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.074100+0000) 2019-10-13T10:18:52.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:52.108+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.111546+0000) 2019-10-13T10:18:52.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:52.108+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.111936+0000) 2019-10-13T10:18:52.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:52.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.112740+0000) 2019-10-13T10:18:52.114 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:52.108+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.113088+0000) 2019-10-13T10:18:52.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:52.108+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.113447+0000) 2019-10-13T10:18:52.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:52.108+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.113870+0000) 2019-10-13T10:18:52.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:52.112+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.115296+0000) 2019-10-13T10:18:52.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:52.112+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:52.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:52.116+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.120957+0000) 2019-10-13T10:18:53.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:53.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.072355+0000) 2019-10-13T10:18:54.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:54.780+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.782915+0000) 2019-10-13T10:18:55.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:55.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:55.102593+0000) 2019-10-13T10:18:55.870 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:18:55.871 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:55.874 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:18:55.874 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:56.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:56.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:56.321370+0000) 2019-10-13T10:18:57.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:57.116+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.120113+0000) 2019-10-13T10:18:57.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:18:57.116+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.120725+0000) 2019-10-13T10:18:57.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:18:57.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.121071+0000) 2019-10-13T10:18:57.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:18:57.116+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.121561+0000) 2019-10-13T10:18:57.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:57.116+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.122038+0000) 2019-10-13T10:18:57.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:18:57.116+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.122413+0000) 2019-10-13T10:18:57.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:18:57.120+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.123482+0000) 2019-10-13T10:18:57.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:18:57.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:18:57.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:18:57.124+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.129001+0000) 2019-10-13T10:18:58.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:18:58.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.584749+0000) 2019-10-13T10:18:58.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:18:58.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.590535+0000) 2019-10-13T10:19:01.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:01.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:01.882672+0000) 2019-10-13T10:19:02.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:02.068+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.074326+0000) 2019-10-13T10:19:02.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:02.120+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.126947+0000) 2019-10-13T10:19:02.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:02.124+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.127451+0000) 2019-10-13T10:19:02.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:02.124+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.128458+0000) 2019-10-13T10:19:02.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:02.124+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.128822+0000) 2019-10-13T10:19:02.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:02.124+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.129838+0000) 2019-10-13T10:19:02.141 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:02.124+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.130160+0000) 2019-10-13T10:19:02.142 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:02.124+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.130534+0000) 2019-10-13T10:19:02.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:02.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:02.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:02.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.135452+0000) 2019-10-13T10:19:03.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:03.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.072530+0000) 2019-10-13T10:19:04.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:04.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.789499+0000) 2019-10-13T10:19:05.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:05.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:05.102799+0000) 2019-10-13T10:19:06.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:06.316+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:06.321560+0000) 2019-10-13T10:19:07.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:07.128+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.134518+0000) 2019-10-13T10:19:07.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:07.128+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.135197+0000) 2019-10-13T10:19:07.136 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:07.132+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.135588+0000) 2019-10-13T10:19:07.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:07.132+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.136045+0000) 2019-10-13T10:19:07.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:07.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.136728+0000) 2019-10-13T10:19:07.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:07.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.137149+0000) 2019-10-13T10:19:07.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:07.132+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.137767+0000) 2019-10-13T10:19:07.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:07.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:07.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:07.144+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.147092+0000) 2019-10-13T10:19:08.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:08.580+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.584952+0000) 2019-10-13T10:19:08.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:08.584+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.590727+0000) 2019-10-13T10:19:11.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:11.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:11.882870+0000) 2019-10-13T10:19:12.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:12.072+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.074528+0000) 2019-10-13T10:19:12.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:12.140+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.143219+0000) 2019-10-13T10:19:12.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:12.140+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.143759+0000) 2019-10-13T10:19:12.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:12.140+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.144232+0000) 2019-10-13T10:19:12.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:12.140+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.144882+0000) 2019-10-13T10:19:12.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:12.140+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.145718+0000) 2019-10-13T10:19:12.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:12.140+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.146010+0000) 2019-10-13T10:19:12.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:12.140+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.146493+0000) 2019-10-13T10:19:12.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:12.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:12.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:12.148+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.153388+0000) 2019-10-13T10:19:13.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:13.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.072734+0000) 2019-10-13T10:19:14.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:14.792+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.795247+0000) 2019-10-13T10:19:15.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:15.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:15.102997+0000) 2019-10-13T10:19:16.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:16.319+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:16.321760+0000) 2019-10-13T10:19:17.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:17.147+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.151481+0000) 2019-10-13T10:19:17.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:17.148+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.152202+0000) 2019-10-13T10:19:17.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:17.147+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.152835+0000) 2019-10-13T10:19:17.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:17.147+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.153341+0000) 2019-10-13T10:19:17.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:17.147+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.153702+0000) 2019-10-13T10:19:17.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:17.147+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.154129+0000) 2019-10-13T10:19:17.156 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:17.147+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.155187+0000) 2019-10-13T10:19:17.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:17.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:17.160 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:17.156+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.159188+0000) 2019-10-13T10:19:18.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:18.579+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.585167+0000) 2019-10-13T10:19:18.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:18.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.590922+0000) 2019-10-13T10:19:21.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:21.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:21.883075+0000) 2019-10-13T10:19:22.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:22.067+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.074756+0000) 2019-10-13T10:19:22.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:22.156+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.159291+0000) 2019-10-13T10:19:22.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:22.155+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.159710+0000) 2019-10-13T10:19:22.161 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:22.155+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.160542+0000) 2019-10-13T10:19:22.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:22.155+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.161075+0000) 2019-10-13T10:19:22.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:22.155+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.161630+0000) 2019-10-13T10:19:22.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:22.155+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.162493+0000) 2019-10-13T10:19:22.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:22.155+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.162881+0000) 2019-10-13T10:19:22.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:22.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:22.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:22.164+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.167070+0000) 2019-10-13T10:19:23.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:23.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.072913+0000) 2019-10-13T10:19:24.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:24.799+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.801824+0000) 2019-10-13T10:19:25.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:25.095+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:25.103198+0000) 2019-10-13T10:19:25.999 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:19:25.999 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:26.003 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:19:26.003 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:26.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:26.315+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:26.321992+0000) 2019-10-13T10:19:27.169 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:27.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.167955+0000) 2019-10-13T10:19:27.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:27.163+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.168258+0000) 2019-10-13T10:19:27.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:27.163+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.168677+0000) 2019-10-13T10:19:27.170 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:27.163+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.168950+0000) 2019-10-13T10:19:27.171 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:27.163+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.170095+0000) 2019-10-13T10:19:27.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:27.163+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.170761+0000) 2019-10-13T10:19:27.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:27.167+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.171505+0000) 2019-10-13T10:19:27.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:27.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:27.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:27.175+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.178603+0000) 2019-10-13T10:19:28.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:28.583+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.585406+0000) 2019-10-13T10:19:28.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:28.583+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.591117+0000) 2019-10-13T10:19:31.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:31.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:31.883276+0000) 2019-10-13T10:19:32.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:32.071+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.074957+0000) 2019-10-13T10:19:32.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:32.171+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.175862+0000) 2019-10-13T10:19:32.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:32.171+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.176290+0000) 2019-10-13T10:19:32.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:32.171+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.176951+0000) 2019-10-13T10:19:32.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:32.171+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.177627+0000) 2019-10-13T10:19:32.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:32.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.178203+0000) 2019-10-13T10:19:32.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:32.171+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.179158+0000) 2019-10-13T10:19:32.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:32.175+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.179605+0000) 2019-10-13T10:19:32.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:32.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:32.186 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:32.179+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.185161+0000) 2019-10-13T10:19:33.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:33.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.073087+0000) 2019-10-13T10:19:34.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:34.807+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.808267+0000) 2019-10-13T10:19:35.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:35.099+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:35.103422+0000) 2019-10-13T10:19:36.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:36.319+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:36.322191+0000) 2019-10-13T10:19:37.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:37.179+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.182882+0000) 2019-10-13T10:19:37.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:37.179+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.183392+0000) 2019-10-13T10:19:37.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:37.179+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.184109+0000) 2019-10-13T10:19:37.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:37.179+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.185004+0000) 2019-10-13T10:19:37.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:37.179+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.185602+0000) 2019-10-13T10:19:37.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:37.179+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.186184+0000) 2019-10-13T10:19:37.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:37.183+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.186782+0000) 2019-10-13T10:19:37.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:37.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96960 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:37.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:37.187+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.191661+0000) 2019-10-13T10:19:38.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:38.583+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.585630+0000) 2019-10-13T10:19:38.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:38.587+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.591331+0000) 2019-10-13T10:19:41.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:41.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:41.883564+0000) 2019-10-13T10:19:42.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:42.071+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.075185+0000) 2019-10-13T10:19:42.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:42.187+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.193224+0000) 2019-10-13T10:19:42.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:42.187+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.193553+0000) 2019-10-13T10:19:42.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:42.187+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.193793+0000) 2019-10-13T10:19:42.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:42.187+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.194069+0000) 2019-10-13T10:19:42.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:42.187+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.194417+0000) 2019-10-13T10:19:42.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:42.191+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.195257+0000) 2019-10-13T10:19:42.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:42.187+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.194737+0000) 2019-10-13T10:19:42.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:42.191+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 96991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:42.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:42.195+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.198822+0000) 2019-10-13T10:19:43.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:43.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.073292+0000) 2019-10-13T10:19:44.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:44.811+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.814639+0000) 2019-10-13T10:19:45.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:45.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:45.103631+0000) 2019-10-13T10:19:46.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:46.319+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:46.322416+0000) 2019-10-13T10:19:47.202 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:47.199+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.200434+0000) 2019-10-13T10:19:47.203 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:47.199+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.201036+0000) 2019-10-13T10:19:47.203 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:47.195+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.201943+0000) 2019-10-13T10:19:47.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:47.199+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.202520+0000) 2019-10-13T10:19:47.204 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:47.199+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.203041+0000) 2019-10-13T10:19:47.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:47.199+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.203274+0000) 2019-10-13T10:19:47.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:47.203+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.203772+0000) 2019-10-13T10:19:47.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:47.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:47.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:47.203+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.207087+0000) 2019-10-13T10:19:48.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:48.583+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.585832+0000) 2019-10-13T10:19:48.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:48.591+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.591492+0000) 2019-10-13T10:19:51.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:51.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:51.883764+0000) 2019-10-13T10:19:52.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:52.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.075379+0000) 2019-10-13T10:19:52.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:52.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.208247+0000) 2019-10-13T10:19:52.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:52.207+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.209338+0000) 2019-10-13T10:19:52.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:52.207+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.209820+0000) 2019-10-13T10:19:52.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:52.207+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.210235+0000) 2019-10-13T10:19:52.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:52.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.210769+0000) 2019-10-13T10:19:52.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:52.207+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.211207+0000) 2019-10-13T10:19:52.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:52.207+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.211829+0000) 2019-10-13T10:19:52.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:52.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:52.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:52.211+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.217564+0000) 2019-10-13T10:19:53.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:53.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.073463+0000) 2019-10-13T10:19:54.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:54.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.820456+0000) 2019-10-13T10:19:55.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:55.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:55.103853+0000) 2019-10-13T10:19:56.101 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:19:56.101 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:56.105 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:19:56.105 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:56.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:56.319+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:56.322607+0000) 2019-10-13T10:19:57.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:19:57.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.216841+0000) 2019-10-13T10:19:57.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:57.215+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.217191+0000) 2019-10-13T10:19:57.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:19:57.215+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.217476+0000) 2019-10-13T10:19:57.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:19:57.215+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.218283+0000) 2019-10-13T10:19:57.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:19:57.219+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.219438+0000) 2019-10-13T10:19:57.221 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:19:57.219+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.219810+0000) 2019-10-13T10:19:57.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:57.219+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.220989+0000) 2019-10-13T10:19:57.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:19:57.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:19:57.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:19:57.223+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.227336+0000) 2019-10-13T10:19:58.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:19:58.582+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.586029+0000) 2019-10-13T10:19:58.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:19:58.590+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.591635+0000) 2019-10-13T10:20:01.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:01.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:01.883971+0000) 2019-10-13T10:20:02.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:02.074+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.075587+0000) 2019-10-13T10:20:02.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:02.219+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.224362+0000) 2019-10-13T10:20:02.226 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:02.222+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.224692+0000) 2019-10-13T10:20:02.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:02.222+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.225184+0000) 2019-10-13T10:20:02.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:02.222+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.226356+0000) 2019-10-13T10:20:02.228 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:02.222+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.227037+0000) 2019-10-13T10:20:02.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:02.226+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.228046+0000) 2019-10-13T10:20:02.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:02.226+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.228490+0000) 2019-10-13T10:20:02.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:02.226+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:02.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:02.227+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.231884+0000) 2019-10-13T10:20:03.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:03.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.073641+0000) 2019-10-13T10:20:04.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:04.823+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.826823+0000) 2019-10-13T10:20:05.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:05.102+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:05.104014+0000) 2019-10-13T10:20:06.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:06.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:06.322811+0000) 2019-10-13T10:20:07.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:07.230+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.232016+0000) 2019-10-13T10:20:07.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:07.230+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.232481+0000) 2019-10-13T10:20:07.234 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:07.230+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.233092+0000) 2019-10-13T10:20:07.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:07.230+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.233405+0000) 2019-10-13T10:20:07.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:07.227+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.234112+0000) 2019-10-13T10:20:07.236 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:07.230+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.234597+0000) 2019-10-13T10:20:07.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:07.234+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.235554+0000) 2019-10-13T10:20:07.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:07.234+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:07.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:07.235+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.239140+0000) 2019-10-13T10:20:08.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:08.582+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.586237+0000) 2019-10-13T10:20:08.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:08.590+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.591853+0000) 2019-10-13T10:20:11.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:11.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:11.884174+0000) 2019-10-13T10:20:12.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:12.074+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.075820+0000) 2019-10-13T10:20:12.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:12.238+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.240125+0000) 2019-10-13T10:20:12.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:12.238+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.241094+0000) 2019-10-13T10:20:12.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:12.235+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.241696+0000) 2019-10-13T10:20:12.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:12.238+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.241941+0000) 2019-10-13T10:20:12.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:12.238+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.242218+0000) 2019-10-13T10:20:12.244 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:12.238+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.242626+0000) 2019-10-13T10:20:12.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:12.242+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.243422+0000) 2019-10-13T10:20:12.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:12.242+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:12.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:12.243+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.249481+0000) 2019-10-13T10:20:13.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:13.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.073813+0000) 2019-10-13T10:20:14.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:14.827+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.833162+0000) 2019-10-13T10:20:15.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:15.102+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:15.104228+0000) 2019-10-13T10:20:16.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:16.318+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:16.323007+0000) 2019-10-13T10:20:17.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:17.246+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.247825+0000) 2019-10-13T10:20:17.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:17.246+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.248401+0000) 2019-10-13T10:20:17.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:17.246+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.249077+0000) 2019-10-13T10:20:17.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:17.246+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.249668+0000) 2019-10-13T10:20:17.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:17.247+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.250284+0000) 2019-10-13T10:20:17.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:17.246+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.250749+0000) 2019-10-13T10:20:17.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:17.246+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.251031+0000) 2019-10-13T10:20:17.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:17.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:17.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:17.251+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.256836+0000) 2019-10-13T10:20:18.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:18.582+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.586445+0000) 2019-10-13T10:20:18.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:18.590+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.592061+0000) 2019-10-13T10:20:21.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:21.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:21.884323+0000) 2019-10-13T10:20:22.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:22.074+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.076010+0000) 2019-10-13T10:20:22.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:22.254+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.255985+0000) 2019-10-13T10:20:22.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:22.254+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.256575+0000) 2019-10-13T10:20:22.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:22.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.256990+0000) 2019-10-13T10:20:22.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:22.254+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.257340+0000) 2019-10-13T10:20:22.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:22.254+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.258049+0000) 2019-10-13T10:20:22.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:22.258+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.259471+0000) 2019-10-13T10:20:22.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:22.258+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.259597+0000) 2019-10-13T10:20:22.263 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:22.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:22.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:22.262+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.266423+0000) 2019-10-13T10:20:23.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:23.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.073983+0000) 2019-10-13T10:20:24.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:24.842+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.843257+0000) 2019-10-13T10:20:25.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:25.102+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:25.104428+0000) 2019-10-13T10:20:26.215 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:20:26.215 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:26.220 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:20:26.220 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:26.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:26.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:26.323240+0000) 2019-10-13T10:20:27.266 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:27.262+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.265184+0000) 2019-10-13T10:20:27.267 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:27.262+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.265595+0000) 2019-10-13T10:20:27.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:27.262+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.266214+0000) 2019-10-13T10:20:27.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:27.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.266460+0000) 2019-10-13T10:20:27.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:27.266+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.267280+0000) 2019-10-13T10:20:27.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:27.266+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.267846+0000) 2019-10-13T10:20:27.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:27.266+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.268421+0000) 2019-10-13T10:20:27.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:27.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:27.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:27.266+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.274050+0000) 2019-10-13T10:20:28.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:28.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.586643+0000) 2019-10-13T10:20:28.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:28.590+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.592272+0000) 2019-10-13T10:20:31.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:31.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:31.884517+0000) 2019-10-13T10:20:32.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:32.074+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.076216+0000) 2019-10-13T10:20:32.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:32.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.273192+0000) 2019-10-13T10:20:32.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:32.270+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.273411+0000) 2019-10-13T10:20:32.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:32.266+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.273678+0000) 2019-10-13T10:20:32.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:32.270+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.274026+0000) 2019-10-13T10:20:32.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:32.270+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.274504+0000) 2019-10-13T10:20:32.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:32.274+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.276293+0000) 2019-10-13T10:20:32.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:32.274+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.276762+0000) 2019-10-13T10:20:32.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:32.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:32.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:32.278+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.282272+0000) 2019-10-13T10:20:33.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:33.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.074181+0000) 2019-10-13T10:20:34.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:34.846+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.849615+0000) 2019-10-13T10:20:35.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:35.102+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:35.104633+0000) 2019-10-13T10:20:36.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:36.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:36.323468+0000) 2019-10-13T10:20:37.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:37.278+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.280741+0000) 2019-10-13T10:20:37.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:37.278+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.281551+0000) 2019-10-13T10:20:37.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:37.278+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.281873+0000) 2019-10-13T10:20:37.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:37.278+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.282416+0000) 2019-10-13T10:20:37.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:37.278+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.282567+0000) 2019-10-13T10:20:37.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:37.278+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.282938+0000) 2019-10-13T10:20:37.285 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:37.282+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.284218+0000) 2019-10-13T10:20:37.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:37.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97367 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:37.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:37.286+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.291741+0000) 2019-10-13T10:20:38.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:38.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.586843+0000) 2019-10-13T10:20:38.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:38.590+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.592484+0000) 2019-10-13T10:20:41.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:41.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:41.884751+0000) 2019-10-13T10:20:42.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:42.073+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.076414+0000) 2019-10-13T10:20:42.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:42.285+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.289395+0000) 2019-10-13T10:20:42.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:42.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.289957+0000) 2019-10-13T10:20:42.291 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:42.285+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.290282+0000) 2019-10-13T10:20:42.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:42.285+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.291018+0000) 2019-10-13T10:20:42.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:42.289+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.291861+0000) 2019-10-13T10:20:42.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:42.289+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.292123+0000) 2019-10-13T10:20:42.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:42.289+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.292852+0000) 2019-10-13T10:20:42.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:42.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:42.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:42.294+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.299663+0000) 2019-10-13T10:20:43.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:43.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.074363+0000) 2019-10-13T10:20:44.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:44.854+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.856016+0000) 2019-10-13T10:20:45.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:45.101+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:45.104832+0000) 2019-10-13T10:20:46.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:46.321+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:46.323686+0000) 2019-10-13T10:20:47.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:47.293+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.297241+0000) 2019-10-13T10:20:47.299 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:47.293+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.297654+0000) 2019-10-13T10:20:47.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:47.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.298675+0000) 2019-10-13T10:20:47.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:47.293+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.298963+0000) 2019-10-13T10:20:47.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:47.297+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.299297+0000) 2019-10-13T10:20:47.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:47.297+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.299685+0000) 2019-10-13T10:20:47.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:47.297+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.300476+0000) 2019-10-13T10:20:47.302 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:47.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:47.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:47.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.305397+0000) 2019-10-13T10:20:48.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:48.585+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.587042+0000) 2019-10-13T10:20:48.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:48.589+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.592682+0000) 2019-10-13T10:20:51.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:51.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:51.884943+0000) 2019-10-13T10:20:52.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:52.073+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.076616+0000) 2019-10-13T10:20:52.305 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:52.301+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.304224+0000) 2019-10-13T10:20:52.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:52.298+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.304660+0000) 2019-10-13T10:20:52.306 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:52.301+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.305088+0000) 2019-10-13T10:20:52.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:52.301+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.305483+0000) 2019-10-13T10:20:52.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:52.301+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.306331+0000) 2019-10-13T10:20:52.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:52.301+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.306863+0000) 2019-10-13T10:20:52.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:52.305+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.307824+0000) 2019-10-13T10:20:52.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:52.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97468 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:52.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:52.310+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.314294+0000) 2019-10-13T10:20:53.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:53.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.074555+0000) 2019-10-13T10:20:54.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:54.866+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.866901+0000) 2019-10-13T10:20:55.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:55.101+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:55.105025+0000) 2019-10-13T10:20:56.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:56.321+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:56.323887+0000) 2019-10-13T10:20:56.347 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:20:56.347 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:56.350 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:20:56.350 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:57.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:20:57.309+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.312189+0000) 2019-10-13T10:20:57.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:20:57.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.312765+0000) 2019-10-13T10:20:57.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:20:57.309+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.313529+0000) 2019-10-13T10:20:57.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:57.309+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.313736+0000) 2019-10-13T10:20:57.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:20:57.309+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.314004+0000) 2019-10-13T10:20:57.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:20:57.309+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.314935+0000) 2019-10-13T10:20:57.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:57.309+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.315166+0000) 2019-10-13T10:20:57.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:20:57.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:20:57.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:20:57.314+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.320642+0000) 2019-10-13T10:20:58.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:20:58.581+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.587243+0000) 2019-10-13T10:20:58.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:20:58.589+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.592873+0000) 2019-10-13T10:21:01.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:01.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:01.885150+0000) 2019-10-13T10:21:02.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:02.073+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.076818+0000) 2019-10-13T10:21:02.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:02.317+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.320360+0000) 2019-10-13T10:21:02.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:02.314+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.321150+0000) 2019-10-13T10:21:02.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:02.317+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.320881+0000) 2019-10-13T10:21:02.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:02.317+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.321352+0000) 2019-10-13T10:21:02.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:02.317+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.321909+0000) 2019-10-13T10:21:02.324 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:02.317+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.322764+0000) 2019-10-13T10:21:02.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:02.321+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.323611+0000) 2019-10-13T10:21:02.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:02.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:02.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:02.326+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.330737+0000) 2019-10-13T10:21:03.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:03.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.074743+0000) 2019-10-13T10:21:04.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:04.878+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.878434+0000) 2019-10-13T10:21:05.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:05.101+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:05.105227+0000) 2019-10-13T10:21:06.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:06.321+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:06.324081+0000) 2019-10-13T10:21:07.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:07.325+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.328289+0000) 2019-10-13T10:21:07.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:07.325+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.328529+0000) 2019-10-13T10:21:07.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:07.325+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.330006+0000) 2019-10-13T10:21:07.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:07.325+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.330253+0000) 2019-10-13T10:21:07.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:07.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.330538+0000) 2019-10-13T10:21:07.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:07.325+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.330865+0000) 2019-10-13T10:21:07.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:07.329+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.331764+0000) 2019-10-13T10:21:07.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:07.329+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:07.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:07.338+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.338534+0000) 2019-10-13T10:21:08.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:08.585+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.587481+0000) 2019-10-13T10:21:08.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:08.589+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.593036+0000) 2019-10-13T10:21:11.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:11.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:11.885345+0000) 2019-10-13T10:21:12.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:12.073+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.077017+0000) 2019-10-13T10:21:12.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:12.333+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.335891+0000) 2019-10-13T10:21:12.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:12.333+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.337476+0000) 2019-10-13T10:21:12.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:12.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.338287+0000) 2019-10-13T10:21:12.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:12.333+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.338553+0000) 2019-10-13T10:21:12.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:12.333+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.339053+0000) 2019-10-13T10:21:12.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:12.337+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.339426+0000) 2019-10-13T10:21:12.341 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:12.337+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.339945+0000) 2019-10-13T10:21:12.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:12.337+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:12.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:12.342+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.344566+0000) 2019-10-13T10:21:13.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:13.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.074911+0000) 2019-10-13T10:21:14.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:14.881+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.884924+0000) 2019-10-13T10:21:15.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:15.101+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:15.105430+0000) 2019-10-13T10:21:16.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:16.321+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:16.324278+0000) 2019-10-13T10:21:17.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:17.341+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.344957+0000) 2019-10-13T10:21:17.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:17.341+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.345241+0000) 2019-10-13T10:21:17.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:17.341+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.346144+0000) 2019-10-13T10:21:17.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:17.341+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.346744+0000) 2019-10-13T10:21:17.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:17.341+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.347046+0000) 2019-10-13T10:21:17.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:17.345+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.348092+0000) 2019-10-13T10:21:17.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:17.345+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.348661+0000) 2019-10-13T10:21:17.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:17.345+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:17.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:17.353+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.354301+0000) 2019-10-13T10:21:18.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:18.585+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.587682+0000) 2019-10-13T10:21:18.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:18.589+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.593216+0000) 2019-10-13T10:21:21.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:21.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:21.885574+0000) 2019-10-13T10:21:22.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:22.077+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.077263+0000) 2019-10-13T10:21:22.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:22.349+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.353083+0000) 2019-10-13T10:21:22.355 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:22.349+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.353631+0000) 2019-10-13T10:21:22.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:22.353+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.354719+0000) 2019-10-13T10:21:22.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:22.353+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.355384+0000) 2019-10-13T10:21:22.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:22.353+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.355716+0000) 2019-10-13T10:21:22.357 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:22.353+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.356137+0000) 2019-10-13T10:21:22.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:22.353+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.356369+0000) 2019-10-13T10:21:22.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:22.353+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.357131+0000) 2019-10-13T10:21:22.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:22.353+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:23.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:23.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.075115+0000) 2019-10-13T10:21:24.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:24.889+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.891374+0000) 2019-10-13T10:21:25.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:25.101+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:25.105634+0000) 2019-10-13T10:21:26.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:26.320+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:26.324471+0000) 2019-10-13T10:21:26.429 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:21:26.429 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:26.433 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:21:26.434 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:27.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:27.357+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.360991+0000) 2019-10-13T10:21:27.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:27.356+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.361835+0000) 2019-10-13T10:21:27.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:27.356+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.362044+0000) 2019-10-13T10:21:27.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:27.356+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.362455+0000) 2019-10-13T10:21:27.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:27.360+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.363529+0000) 2019-10-13T10:21:27.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:27.360+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.363661+0000) 2019-10-13T10:21:27.365 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:27.360+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.364021+0000) 2019-10-13T10:21:27.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:27.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:27.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:27.365+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.367127+0000) 2019-10-13T10:21:28.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:28.584+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.587895+0000) 2019-10-13T10:21:28.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:28.588+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.593470+0000) 2019-10-13T10:21:31.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:31.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:31.885775+0000) 2019-10-13T10:21:32.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:32.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.077472+0000) 2019-10-13T10:21:32.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:32.364+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.368738+0000) 2019-10-13T10:21:32.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:32.364+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.369038+0000) 2019-10-13T10:21:32.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:32.364+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.369903+0000) 2019-10-13T10:21:32.371 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:32.364+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.370575+0000) 2019-10-13T10:21:32.372 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:32.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.371343+0000) 2019-10-13T10:21:32.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:32.368+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.371937+0000) 2019-10-13T10:21:32.373 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:32.368+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.372274+0000) 2019-10-13T10:21:32.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:32.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:32.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:32.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.375734+0000) 2019-10-13T10:21:33.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:33.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.075326+0000) 2019-10-13T10:21:34.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:34.901+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.902638+0000) 2019-10-13T10:21:35.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:35.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:35.105838+0000) 2019-10-13T10:21:36.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:36.320+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:36.324684+0000) 2019-10-13T10:21:37.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:37.372+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.377104+0000) 2019-10-13T10:21:37.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:37.373+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.377496+0000) 2019-10-13T10:21:37.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:37.372+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.377836+0000) 2019-10-13T10:21:37.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:37.372+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.378698+0000) 2019-10-13T10:21:37.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:37.372+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.378861+0000) 2019-10-13T10:21:37.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:37.372+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.379220+0000) 2019-10-13T10:21:37.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:37.376+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.379828+0000) 2019-10-13T10:21:37.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:37.376+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:37.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:37.381+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.383207+0000) 2019-10-13T10:21:38.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:38.584+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.588095+0000) 2019-10-13T10:21:38.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:38.588+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.593699+0000) 2019-10-13T10:21:41.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:41.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:41.885972+0000) 2019-10-13T10:21:42.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:42.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.077680+0000) 2019-10-13T10:21:42.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:42.380+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.384696+0000) 2019-10-13T10:21:42.386 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:42.380+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.385226+0000) 2019-10-13T10:21:42.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:42.380+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.385802+0000) 2019-10-13T10:21:42.388 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:42.380+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.387010+0000) 2019-10-13T10:21:42.388 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:42.385+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.387506+0000) 2019-10-13T10:21:42.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:42.384+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.387747+0000) 2019-10-13T10:21:42.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:42.384+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.388525+0000) 2019-10-13T10:21:42.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:42.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:42.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:42.389+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.392645+0000) 2019-10-13T10:21:43.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:43.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.075500+0000) 2019-10-13T10:21:44.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:44.905+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.909085+0000) 2019-10-13T10:21:45.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:45.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:45.106053+0000) 2019-10-13T10:21:46.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:46.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:46.324899+0000) 2019-10-13T10:21:47.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:47.388+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.392373+0000) 2019-10-13T10:21:47.394 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:47.388+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.393014+0000) 2019-10-13T10:21:47.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:47.388+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.393533+0000) 2019-10-13T10:21:47.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:47.388+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.394266+0000) 2019-10-13T10:21:47.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:47.388+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.394917+0000) 2019-10-13T10:21:47.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:47.392+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.395583+0000) 2019-10-13T10:21:47.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:47.393+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.395973+0000) 2019-10-13T10:21:47.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:47.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:47.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:47.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.401167+0000) 2019-10-13T10:21:48.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:48.584+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.588297+0000) 2019-10-13T10:21:48.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:48.592+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.593928+0000) 2019-10-13T10:21:51.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:51.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:51.886151+0000) 2019-10-13T10:21:52.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:52.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.077901+0000) 2019-10-13T10:21:52.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:52.396+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.401235+0000) 2019-10-13T10:21:52.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:52.396+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.401563+0000) 2019-10-13T10:21:52.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:52.396+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.402224+0000) 2019-10-13T10:21:52.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:52.396+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.402589+0000) 2019-10-13T10:21:52.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:52.396+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.403047+0000) 2019-10-13T10:21:52.405 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:52.400+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.404000+0000) 2019-10-13T10:21:52.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:52.401+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.404597+0000) 2019-10-13T10:21:52.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:52.400+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:52.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:52.409+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.410833+0000) 2019-10-13T10:21:53.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:53.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.075688+0000) 2019-10-13T10:21:54.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:54.917+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.918722+0000) 2019-10-13T10:21:55.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:55.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:55.106254+0000) 2019-10-13T10:21:56.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:56.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:56.325091+0000) 2019-10-13T10:21:56.560 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:21:56.561 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:56.564 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:21:56.564 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:57.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:57.404+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.408893+0000) 2019-10-13T10:21:57.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:21:57.404+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.409470+0000) 2019-10-13T10:21:57.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:21:57.409+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.413704+0000) 2019-10-13T10:21:57.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:21:57.404+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.410541+0000) 2019-10-13T10:21:57.417 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:57.404+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.411135+0000) 2019-10-13T10:21:57.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:21:57.408+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.411575+0000) 2019-10-13T10:21:57.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:21:57.408+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.412254+0000) 2019-10-13T10:21:57.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:21:57.408+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97910 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:21:57.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:21:57.413+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.416669+0000) 2019-10-13T10:21:58.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:21:58.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.588513+0000) 2019-10-13T10:21:58.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:21:58.588+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.594129+0000) 2019-10-13T10:22:01.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:01.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:01.886397+0000) 2019-10-13T10:22:02.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:02.072+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.078109+0000) 2019-10-13T10:22:02.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:02.412+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.417283+0000) 2019-10-13T10:22:02.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:02.412+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.418179+0000) 2019-10-13T10:22:02.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:02.412+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.418729+0000) 2019-10-13T10:22:02.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:02.416+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.419591+0000) 2019-10-13T10:22:02.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:02.416+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.420029+0000) 2019-10-13T10:22:02.421 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:02.416+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.420420+0000) 2019-10-13T10:22:02.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:02.417+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.420932+0000) 2019-10-13T10:22:02.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:02.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:02.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:02.421+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.422774+0000) 2019-10-13T10:22:03.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:03.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.075859+0000) 2019-10-13T10:22:04.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:04.925+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.929803+0000) 2019-10-13T10:22:05.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:05.100+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:05.106453+0000) 2019-10-13T10:22:06.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:06.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:06.325310+0000) 2019-10-13T10:22:07.426 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:07.420+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.424746+0000) 2019-10-13T10:22:07.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:07.420+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.425602+0000) 2019-10-13T10:22:07.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:07.420+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.426095+0000) 2019-10-13T10:22:07.427 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:07.425+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.426453+0000) 2019-10-13T10:22:07.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:07.420+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.426921+0000) 2019-10-13T10:22:07.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:07.424+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.427599+0000) 2019-10-13T10:22:07.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:07.424+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.427944+0000) 2019-10-13T10:22:07.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:07.424+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 97977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:07.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:07.429+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.431796+0000) 2019-10-13T10:22:08.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:08.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.588716+0000) 2019-10-13T10:22:08.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:08.592+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.594287+0000) 2019-10-13T10:22:11.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:11.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:11.886594+0000) 2019-10-13T10:22:12.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:12.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.078310+0000) 2019-10-13T10:22:12.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:12.427+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.433407+0000) 2019-10-13T10:22:12.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:12.427+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.433927+0000) 2019-10-13T10:22:12.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:12.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.434542+0000) 2019-10-13T10:22:12.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:12.427+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.433526+0000) 2019-10-13T10:22:12.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:12.427+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.434919+0000) 2019-10-13T10:22:12.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:12.431+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.435454+0000) 2019-10-13T10:22:12.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:12.427+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.434675+0000) 2019-10-13T10:22:12.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:12.440+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.442316+0000) 2019-10-13T10:22:12.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:12.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:13.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:13.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.076029+0000) 2019-10-13T10:22:14.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:14.932+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.936321+0000) 2019-10-13T10:22:15.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:15.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:15.106681+0000) 2019-10-13T10:22:16.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:16.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:16.325503+0000) 2019-10-13T10:22:17.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:17.436+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.440840+0000) 2019-10-13T10:22:17.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:17.435+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.441318+0000) 2019-10-13T10:22:17.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:17.435+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.441966+0000) 2019-10-13T10:22:17.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:17.435+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.442133+0000) 2019-10-13T10:22:17.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:17.435+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.442471+0000) 2019-10-13T10:22:17.444 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:17.435+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.443052+0000) 2019-10-13T10:22:17.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:17.439+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.443656+0000) 2019-10-13T10:22:17.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:17.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:17.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:17.448+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.450956+0000) 2019-10-13T10:22:18.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:18.587+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.588915+0000) 2019-10-13T10:22:18.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:18.591+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.594513+0000) 2019-10-13T10:22:21.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:21.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:21.886800+0000) 2019-10-13T10:22:22.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:22.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.078494+0000) 2019-10-13T10:22:22.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:22.443+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.449925+0000) 2019-10-13T10:22:22.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:22.443+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.450131+0000) 2019-10-13T10:22:22.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:22.443+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.450833+0000) 2019-10-13T10:22:22.452 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:22.443+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.451280+0000) 2019-10-13T10:22:22.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:22.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.451951+0000) 2019-10-13T10:22:22.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:22.447+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.452159+0000) 2019-10-13T10:22:22.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:22.447+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.452836+0000) 2019-10-13T10:22:22.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:22.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:22.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:22.456+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.458666+0000) 2019-10-13T10:22:23.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:23.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.076200+0000) 2019-10-13T10:22:24.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:24.940+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.942836+0000) 2019-10-13T10:22:25.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:25.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:25.106834+0000) 2019-10-13T10:22:26.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:26.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:26.325732+0000) 2019-10-13T10:22:26.666 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:22:26.666 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:26.670 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:22:26.670 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:27.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:27.451+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.459082+0000) 2019-10-13T10:22:27.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:27.455+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.459653+0000) 2019-10-13T10:22:27.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:27.455+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.460033+0000) 2019-10-13T10:22:27.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:27.455+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.460770+0000) 2019-10-13T10:22:27.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:27.455+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.461149+0000) 2019-10-13T10:22:27.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:27.456+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.461461+0000) 2019-10-13T10:22:27.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:27.455+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.461693+0000) 2019-10-13T10:22:27.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:27.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:27.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:27.460+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.464337+0000) 2019-10-13T10:22:28.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:28.587+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.589146+0000) 2019-10-13T10:22:28.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:28.591+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.594708+0000) 2019-10-13T10:22:31.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:31.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:31.887043+0000) 2019-10-13T10:22:32.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:32.071+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.078687+0000) 2019-10-13T10:22:32.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:32.459+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.466378+0000) 2019-10-13T10:22:32.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:32.459+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.466705+0000) 2019-10-13T10:22:32.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:32.463+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.467501+0000) 2019-10-13T10:22:32.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:32.463+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.467703+0000) 2019-10-13T10:22:32.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:32.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.468246+0000) 2019-10-13T10:22:32.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:32.463+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.469040+0000) 2019-10-13T10:22:32.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:32.463+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.469336+0000) 2019-10-13T10:22:32.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:32.463+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:32.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:32.472+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.476614+0000) 2019-10-13T10:22:33.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:33.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.076376+0000) 2019-10-13T10:22:34.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:34.948+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.949446+0000) 2019-10-13T10:22:35.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:35.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:35.107043+0000) 2019-10-13T10:22:36.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:36.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:36.325929+0000) 2019-10-13T10:22:37.475 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:37.467+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.473364+0000) 2019-10-13T10:22:37.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:37.467+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.473697+0000) 2019-10-13T10:22:37.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:37.472+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.476528+0000) 2019-10-13T10:22:37.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:37.467+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.474409+0000) 2019-10-13T10:22:37.482 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:37.467+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.474679+0000) 2019-10-13T10:22:37.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:37.467+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.475219+0000) 2019-10-13T10:22:37.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:37.471+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.476096+0000) 2019-10-13T10:22:37.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:37.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:37.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:37.480+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.484820+0000) 2019-10-13T10:22:38.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:38.587+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.589374+0000) 2019-10-13T10:22:38.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:38.591+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.594908+0000) 2019-10-13T10:22:41.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:41.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:41.887252+0000) 2019-10-13T10:22:42.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:42.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.078889+0000) 2019-10-13T10:22:42.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:42.479+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.481745+0000) 2019-10-13T10:22:42.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:42.479+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.482287+0000) 2019-10-13T10:22:42.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:42.479+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.482536+0000) 2019-10-13T10:22:42.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:42.479+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.483177+0000) 2019-10-13T10:22:42.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:42.483+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.483551+0000) 2019-10-13T10:22:42.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:42.483+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.483988+0000) 2019-10-13T10:22:42.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:42.480+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.484502+0000) 2019-10-13T10:22:42.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:42.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:42.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:42.492+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.496300+0000) 2019-10-13T10:22:43.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:43.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.076569+0000) 2019-10-13T10:22:44.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:44.952+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.956070+0000) 2019-10-13T10:22:45.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:45.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:45.107271+0000) 2019-10-13T10:22:46.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:46.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:46.326123+0000) 2019-10-13T10:22:47.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:47.487+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.489428+0000) 2019-10-13T10:22:47.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:47.487+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.489916+0000) 2019-10-13T10:22:47.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:47.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.491439+0000) 2019-10-13T10:22:47.493 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:47.488+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.491755+0000) 2019-10-13T10:22:47.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:47.491+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.492023+0000) 2019-10-13T10:22:47.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:47.491+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.492314+0000) 2019-10-13T10:22:47.494 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:47.491+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.493452+0000) 2019-10-13T10:22:47.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:47.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:47.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:47.496+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.499408+0000) 2019-10-13T10:22:48.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:48.587+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.589569+0000) 2019-10-13T10:22:48.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:48.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.595133+0000) 2019-10-13T10:22:51.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:51.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:51.887532+0000) 2019-10-13T10:22:52.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:52.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.079156+0000) 2019-10-13T10:22:52.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:52.495+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.498602+0000) 2019-10-13T10:22:52.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:52.495+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.498700+0000) 2019-10-13T10:22:52.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:52.496+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.499439+0000) 2019-10-13T10:22:52.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:52.499+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.499687+0000) 2019-10-13T10:22:52.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:52.499+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.500789+0000) 2019-10-13T10:22:52.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:52.499+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.500557+0000) 2019-10-13T10:22:52.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:52.499+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.501958+0000) 2019-10-13T10:22:52.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:52.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:52.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:52.504+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.508665+0000) 2019-10-13T10:22:53.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:53.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.076738+0000) 2019-10-13T10:22:54.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:54.964+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.967157+0000) 2019-10-13T10:22:55.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:55.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:55.107463+0000) 2019-10-13T10:22:56.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:56.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:56.326312+0000) 2019-10-13T10:22:56.806 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:22:56.806 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:56.810 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:22:56.810 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:57.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:22:57.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.507118+0000) 2019-10-13T10:22:57.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:22:57.506+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.507754+0000) 2019-10-13T10:22:57.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:22:57.506+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.508015+0000) 2019-10-13T10:22:57.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:22:57.506+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.509010+0000) 2019-10-13T10:22:57.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:57.506+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.509282+0000) 2019-10-13T10:22:57.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:22:57.506+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.509574+0000) 2019-10-13T10:22:57.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:57.506+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.511161+0000) 2019-10-13T10:22:57.513 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:22:57.510+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:22:57.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:22:57.512+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.517493+0000) 2019-10-13T10:22:58.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:22:58.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.589723+0000) 2019-10-13T10:22:58.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:22:58.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.595356+0000) 2019-10-13T10:23:01.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:01.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:01.887769+0000) 2019-10-13T10:23:02.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:02.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.079332+0000) 2019-10-13T10:23:02.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:02.514+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.515652+0000) 2019-10-13T10:23:02.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:02.514+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.516209+0000) 2019-10-13T10:23:02.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:02.514+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.517101+0000) 2019-10-13T10:23:02.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:02.514+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.517842+0000) 2019-10-13T10:23:02.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:02.516+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.518383+0000) 2019-10-13T10:23:02.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:02.514+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.518660+0000) 2019-10-13T10:23:02.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:02.514+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.519145+0000) 2019-10-13T10:23:02.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:02.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:02.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:02.520+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.525956+0000) 2019-10-13T10:23:03.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:03.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.076942+0000) 2019-10-13T10:23:04.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:04.976+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.978537+0000) 2019-10-13T10:23:05.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:05.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:05.107710+0000) 2019-10-13T10:23:06.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:06.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:06.326513+0000) 2019-10-13T10:23:07.524 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:07.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.523091+0000) 2019-10-13T10:23:07.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:07.522+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.523678+0000) 2019-10-13T10:23:07.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:07.522+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.524507+0000) 2019-10-13T10:23:07.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:07.522+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.524931+0000) 2019-10-13T10:23:07.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:07.522+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.525519+0000) 2019-10-13T10:23:07.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:07.519+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.525722+0000) 2019-10-13T10:23:07.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:07.522+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.527145+0000) 2019-10-13T10:23:07.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:07.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:07.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:07.527+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.532439+0000) 2019-10-13T10:23:08.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:08.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.589922+0000) 2019-10-13T10:23:08.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:08.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.595509+0000) 2019-10-13T10:23:11.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:11.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:11.887977+0000) 2019-10-13T10:23:12.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:12.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.079569+0000) 2019-10-13T10:23:12.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:12.527+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.531370+0000) 2019-10-13T10:23:12.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:12.530+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.531751+0000) 2019-10-13T10:23:12.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:12.530+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.532462+0000) 2019-10-13T10:23:12.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:12.530+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.532838+0000) 2019-10-13T10:23:12.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:12.530+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.533193+0000) 2019-10-13T10:23:12.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:12.530+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.534221+0000) 2019-10-13T10:23:12.535 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:12.530+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.534753+0000) 2019-10-13T10:23:12.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:12.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:12.541 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:12.535+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.540267+0000) 2019-10-13T10:23:13.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:13.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.077147+0000) 2019-10-13T10:23:14.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:14.983+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.985130+0000) 2019-10-13T10:23:15.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:15.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:15.107914+0000) 2019-10-13T10:23:16.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:16.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:16.326713+0000) 2019-10-13T10:23:17.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:17.538+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.540467+0000) 2019-10-13T10:23:17.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:17.538+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.540708+0000) 2019-10-13T10:23:17.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:17.535+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.541236+0000) 2019-10-13T10:23:17.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:17.538+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.542028+0000) 2019-10-13T10:23:17.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:17.538+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.542561+0000) 2019-10-13T10:23:17.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:17.538+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.542783+0000) 2019-10-13T10:23:17.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:17.538+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.543098+0000) 2019-10-13T10:23:17.545 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:17.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:17.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:17.543+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.546438+0000) 2019-10-13T10:23:18.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:18.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.590122+0000) 2019-10-13T10:23:18.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:18.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.595713+0000) 2019-10-13T10:23:21.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:21.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:21.888221+0000) 2019-10-13T10:23:22.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:22.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.079739+0000) 2019-10-13T10:23:22.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:22.543+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.548485+0000) 2019-10-13T10:23:22.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:22.546+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.548952+0000) 2019-10-13T10:23:22.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:22.546+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.549427+0000) 2019-10-13T10:23:22.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:22.546+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.549853+0000) 2019-10-13T10:23:22.551 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:22.546+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.550278+0000) 2019-10-13T10:23:22.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:22.546+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.550874+0000) 2019-10-13T10:23:22.552 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:22.550+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.551685+0000) 2019-10-13T10:23:22.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:22.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:22.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:22.551+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.557992+0000) 2019-10-13T10:23:23.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:23.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.077317+0000) 2019-10-13T10:23:24.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:24.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.991707+0000) 2019-10-13T10:23:25.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:25.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:25.108159+0000) 2019-10-13T10:23:26.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:26.322+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:26.326860+0000) 2019-10-13T10:23:26.928 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:23:26.928 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:26.933 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:23:26.933 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:27.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:27.554+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.556301+0000) 2019-10-13T10:23:27.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:27.554+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.556694+0000) 2019-10-13T10:23:27.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:27.554+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.557534+0000) 2019-10-13T10:23:27.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:27.554+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.558267+0000) 2019-10-13T10:23:27.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:27.554+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.559051+0000) 2019-10-13T10:23:27.560 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:27.555+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.559429+0000) 2019-10-13T10:23:27.561 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:27.558+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.560571+0000) 2019-10-13T10:23:27.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:27.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:27.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:27.559+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.565025+0000) 2019-10-13T10:23:28.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:28.586+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.590317+0000) 2019-10-13T10:23:28.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:28.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.595939+0000) 2019-10-13T10:23:31.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:31.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:31.888483+0000) 2019-10-13T10:23:32.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:32.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.079956+0000) 2019-10-13T10:23:32.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:32.562+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.564195+0000) 2019-10-13T10:23:32.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:32.562+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.565208+0000) 2019-10-13T10:23:32.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:32.562+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.565827+0000) 2019-10-13T10:23:32.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:32.563+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.566478+0000) 2019-10-13T10:23:32.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:32.562+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.566694+0000) 2019-10-13T10:23:32.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:32.566+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.567791+0000) 2019-10-13T10:23:32.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:32.566+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.568548+0000) 2019-10-13T10:23:32.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:32.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:32.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:32.567+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.573402+0000) 2019-10-13T10:23:33.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:33.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.077488+0000) 2019-10-13T10:23:34.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:34.995+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.998113+0000) 2019-10-13T10:23:35.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:35.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:35.108381+0000) 2019-10-13T10:23:36.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:36.325+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:36.327051+0000) 2019-10-13T10:23:37.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:37.569+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.572577+0000) 2019-10-13T10:23:37.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:37.569+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.573006+0000) 2019-10-13T10:23:37.575 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:37.567+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.573637+0000) 2019-10-13T10:23:37.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:37.569+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.573427+0000) 2019-10-13T10:23:37.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:37.569+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.574371+0000) 2019-10-13T10:23:37.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:37.569+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.574581+0000) 2019-10-13T10:23:37.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:37.569+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.574965+0000) 2019-10-13T10:23:37.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:37.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:37.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:37.575+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.581828+0000) 2019-10-13T10:23:38.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:38.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.590524+0000) 2019-10-13T10:23:38.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:38.593+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.596139+0000) 2019-10-13T10:23:41.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:41.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:41.888702+0000) 2019-10-13T10:23:42.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:42.077+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.080185+0000) 2019-10-13T10:23:42.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:42.577+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.580628+0000) 2019-10-13T10:23:42.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:42.577+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.581133+0000) 2019-10-13T10:23:42.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:42.575+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.581606+0000) 2019-10-13T10:23:42.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:42.577+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.581857+0000) 2019-10-13T10:23:42.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:42.577+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.582218+0000) 2019-10-13T10:23:42.584 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:42.577+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.582914+0000) 2019-10-13T10:23:42.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:42.581+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.583487+0000) 2019-10-13T10:23:42.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:42.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:42.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:42.587+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.590329+0000) 2019-10-13T10:23:43.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:43.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.077652+0000) 2019-10-13T10:23:45.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:45.007+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:45.008088+0000) 2019-10-13T10:23:45.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:45.105+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:45.108625+0000) 2019-10-13T10:23:46.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:46.321+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:46.327248+0000) 2019-10-13T10:23:47.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:47.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.588565+0000) 2019-10-13T10:23:47.590 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:47.585+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.589085+0000) 2019-10-13T10:23:47.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:47.585+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.590119+0000) 2019-10-13T10:23:47.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:47.585+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.590511+0000) 2019-10-13T10:23:47.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:47.585+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.590847+0000) 2019-10-13T10:23:47.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:47.585+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.591172+0000) 2019-10-13T10:23:47.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:47.587+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.591753+0000) 2019-10-13T10:23:47.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:47.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:47.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:47.595+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.598880+0000) 2019-10-13T10:23:48.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:48.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.590736+0000) 2019-10-13T10:23:48.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:48.593+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.596365+0000) 2019-10-13T10:23:51.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:51.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:51.888859+0000) 2019-10-13T10:23:52.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:52.077+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.080383+0000) 2019-10-13T10:23:52.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:52.593+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.596334+0000) 2019-10-13T10:23:52.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:52.593+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.597029+0000) 2019-10-13T10:23:52.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:52.593+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.597409+0000) 2019-10-13T10:23:52.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:52.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.598441+0000) 2019-10-13T10:23:52.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:52.593+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.598216+0000) 2019-10-13T10:23:52.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:52.595+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.599434+0000) 2019-10-13T10:23:52.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:52.597+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.599973+0000) 2019-10-13T10:23:52.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:52.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:52.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:52.599+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.604890+0000) 2019-10-13T10:23:53.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:53.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.077809+0000) 2019-10-13T10:23:55.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:55.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:55.018914+0000) 2019-10-13T10:23:55.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:55.105+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:55.108823+0000) 2019-10-13T10:23:56.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:56.325+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:56.327496+0000) 2019-10-13T10:23:57.069 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:23:57.069 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:57.073 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:23:57.073 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:57.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:57.601+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.604873+0000) 2019-10-13T10:23:57.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:23:57.601+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.605651+0000) 2019-10-13T10:23:57.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:23:57.601+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.606038+0000) 2019-10-13T10:23:57.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:23:57.601+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.606378+0000) 2019-10-13T10:23:57.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:23:57.601+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.606927+0000) 2019-10-13T10:23:57.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:57.601+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.607155+0000) 2019-10-13T10:23:57.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:23:57.599+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.604547+0000) 2019-10-13T10:23:57.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:23:57.605+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98726 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:23:57.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:23:57.607+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.614097+0000) 2019-10-13T10:23:58.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:23:58.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.590913+0000) 2019-10-13T10:23:58.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:23:58.593+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.596621+0000) 2019-10-13T10:24:01.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:01.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:01.889057+0000) 2019-10-13T10:24:02.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:02.077+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.080589+0000) 2019-10-13T10:24:02.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:02.609+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.612380+0000) 2019-10-13T10:24:02.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:02.609+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.612704+0000) 2019-10-13T10:24:02.614 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:02.606+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.613522+0000) 2019-10-13T10:24:02.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:02.609+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.614276+0000) 2019-10-13T10:24:02.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:02.609+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.614790+0000) 2019-10-13T10:24:02.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:02.609+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.614850+0000) 2019-10-13T10:24:02.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:02.613+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.616172+0000) 2019-10-13T10:24:02.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:02.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:02.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:02.614+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.621777+0000) 2019-10-13T10:24:03.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:03.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.077984+0000) 2019-10-13T10:24:05.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:05.018+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:05.025284+0000) 2019-10-13T10:24:05.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:05.105+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:05.109039+0000) 2019-10-13T10:24:06.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:06.325+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:06.327688+0000) 2019-10-13T10:24:07.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:07.617+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.620285+0000) 2019-10-13T10:24:07.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:07.617+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.621006+0000) 2019-10-13T10:24:07.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:07.617+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.622361+0000) 2019-10-13T10:24:07.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:07.618+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.622858+0000) 2019-10-13T10:24:07.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:07.617+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.622545+0000) 2019-10-13T10:24:07.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:07.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.623075+0000) 2019-10-13T10:24:07.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:07.621+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.623880+0000) 2019-10-13T10:24:07.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:07.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:07.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:07.622+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.628426+0000) 2019-10-13T10:24:08.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:08.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.591124+0000) 2019-10-13T10:24:08.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:08.593+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.596817+0000) 2019-10-13T10:24:11.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:11.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:11.889275+0000) 2019-10-13T10:24:12.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:12.077+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.080784+0000) 2019-10-13T10:24:12.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:12.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.627850+0000) 2019-10-13T10:24:12.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:12.625+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.628349+0000) 2019-10-13T10:24:12.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:12.622+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.629140+0000) 2019-10-13T10:24:12.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:12.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.630094+0000) 2019-10-13T10:24:12.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:12.625+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.630538+0000) 2019-10-13T10:24:12.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:12.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.631296+0000) 2019-10-13T10:24:12.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:12.629+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.631473+0000) 2019-10-13T10:24:12.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:12.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:12.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:12.630+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.637426+0000) 2019-10-13T10:24:13.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:13.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.078177+0000) 2019-10-13T10:24:15.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:15.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:15.031605+0000) 2019-10-13T10:24:15.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:15.105+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:15.109234+0000) 2019-10-13T10:24:16.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:16.325+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:16.327920+0000) 2019-10-13T10:24:17.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:17.633+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.635422+0000) 2019-10-13T10:24:17.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:17.633+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.636024+0000) 2019-10-13T10:24:17.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:17.633+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.636573+0000) 2019-10-13T10:24:17.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:17.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.637359+0000) 2019-10-13T10:24:17.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:17.633+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.638141+0000) 2019-10-13T10:24:17.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:17.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.638455+0000) 2019-10-13T10:24:17.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:17.633+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.638702+0000) 2019-10-13T10:24:17.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:17.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:17.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:17.642+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.648151+0000) 2019-10-13T10:24:18.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:18.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.591335+0000) 2019-10-13T10:24:18.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:18.593+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.597059+0000) 2019-10-13T10:24:21.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:21.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:21.889500+0000) 2019-10-13T10:24:22.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:22.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.080999+0000) 2019-10-13T10:24:22.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:22.640+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.643994+0000) 2019-10-13T10:24:22.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:22.640+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.645802+0000) 2019-10-13T10:24:22.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:22.640+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.645998+0000) 2019-10-13T10:24:22.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:22.640+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.647083+0000) 2019-10-13T10:24:22.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:22.642+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.647983+0000) 2019-10-13T10:24:22.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:22.644+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.647447+0000) 2019-10-13T10:24:22.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:22.644+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.648207+0000) 2019-10-13T10:24:22.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:22.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:22.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:22.646+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.653410+0000) 2019-10-13T10:24:23.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:23.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.078357+0000) 2019-10-13T10:24:25.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:25.030+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:25.038007+0000) 2019-10-13T10:24:25.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:25.108+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:25.109436+0000) 2019-10-13T10:24:26.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:26.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:26.328144+0000) 2019-10-13T10:24:27.179 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:24:27.180 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:27.184 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:24:27.184 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:27.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:27.650+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.654468+0000) 2019-10-13T10:24:27.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:27.648+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.653452+0000) 2019-10-13T10:24:27.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:27.648+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.654133+0000) 2019-10-13T10:24:27.674 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:27.648+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.654829+0000) 2019-10-13T10:24:27.674 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:27.648+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.655273+0000) 2019-10-13T10:24:27.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:27.652+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.656173+0000) 2019-10-13T10:24:27.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:27.652+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.655896+0000) 2019-10-13T10:24:27.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:27.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98928 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:27.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:27.654+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.661856+0000) 2019-10-13T10:24:28.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:28.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.591515+0000) 2019-10-13T10:24:28.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:28.596+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.597263+0000) 2019-10-13T10:24:31.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:31.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:31.889698+0000) 2019-10-13T10:24:32.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:32.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.081201+0000) 2019-10-13T10:24:32.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:32.656+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.660754+0000) 2019-10-13T10:24:32.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:32.656+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.661504+0000) 2019-10-13T10:24:32.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:32.658+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.662527+0000) 2019-10-13T10:24:32.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:32.656+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.662738+0000) 2019-10-13T10:24:32.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:32.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.663207+0000) 2019-10-13T10:24:32.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:32.656+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.663268+0000) 2019-10-13T10:24:32.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:32.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.663885+0000) 2019-10-13T10:24:32.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:32.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:32.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:32.662+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.668720+0000) 2019-10-13T10:24:33.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:33.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.078529+0000) 2019-10-13T10:24:35.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:35.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:35.044419+0000) 2019-10-13T10:24:35.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:35.104+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:35.109625+0000) 2019-10-13T10:24:36.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:36.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:36.328362+0000) 2019-10-13T10:24:37.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:37.664+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.668777+0000) 2019-10-13T10:24:37.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:37.664+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.669939+0000) 2019-10-13T10:24:37.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:37.664+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.670436+0000) 2019-10-13T10:24:37.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:37.666+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.670807+0000) 2019-10-13T10:24:37.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:37.668+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.671457+0000) 2019-10-13T10:24:37.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:37.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.671579+0000) 2019-10-13T10:24:37.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:37.668+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.672178+0000) 2019-10-13T10:24:37.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:37.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 98997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:37.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:37.670+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.678010+0000) 2019-10-13T10:24:38.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:38.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.591699+0000) 2019-10-13T10:24:38.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:38.596+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.597441+0000) 2019-10-13T10:24:41.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:41.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:41.889900+0000) 2019-10-13T10:24:42.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:42.080+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.081433+0000) 2019-10-13T10:24:42.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:42.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.679552+0000) 2019-10-13T10:24:42.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:42.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.676865+0000) 2019-10-13T10:24:42.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:42.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.677397+0000) 2019-10-13T10:24:42.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:42.672+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.678210+0000) 2019-10-13T10:24:42.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:42.672+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.678527+0000) 2019-10-13T10:24:42.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:42.676+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.679763+0000) 2019-10-13T10:24:42.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:42.676+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.680039+0000) 2019-10-13T10:24:42.686 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:42.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:42.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:42.678+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.684109+0000) 2019-10-13T10:24:43.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:43.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.078691+0000) 2019-10-13T10:24:45.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:45.050+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:45.053445+0000) 2019-10-13T10:24:45.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:45.104+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:45.109780+0000) 2019-10-13T10:24:46.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:46.324+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:46.328577+0000) 2019-10-13T10:24:47.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:47.680+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.685212+0000) 2019-10-13T10:24:47.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:47.682+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.685671+0000) 2019-10-13T10:24:47.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:47.680+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.686873+0000) 2019-10-13T10:24:47.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:47.680+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.687103+0000) 2019-10-13T10:24:47.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:47.684+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.687684+0000) 2019-10-13T10:24:47.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:47.684+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.688022+0000) 2019-10-13T10:24:47.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:47.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.688765+0000) 2019-10-13T10:24:47.691 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:47.684+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:47.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:47.686+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.692689+0000) 2019-10-13T10:24:48.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:48.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.591873+0000) 2019-10-13T10:24:48.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:48.592+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.597595+0000) 2019-10-13T10:24:51.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:51.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:51.890063+0000) 2019-10-13T10:24:52.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:52.080+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.081638+0000) 2019-10-13T10:24:52.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:52.694+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.696361+0000) 2019-10-13T10:24:52.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:52.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.693960+0000) 2019-10-13T10:24:52.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:52.688+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.694839+0000) 2019-10-13T10:24:52.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:52.692+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.695801+0000) 2019-10-13T10:24:52.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:52.692+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.695960+0000) 2019-10-13T10:24:52.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:52.692+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.696618+0000) 2019-10-13T10:24:52.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:52.692+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.697177+0000) 2019-10-13T10:24:52.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:52.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:52.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:52.702+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.703539+0000) 2019-10-13T10:24:53.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:53.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.078860+0000) 2019-10-13T10:24:55.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:55.062+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:55.063364+0000) 2019-10-13T10:24:55.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:55.104+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:55.109986+0000) 2019-10-13T10:24:56.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:56.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:56.328789+0000) 2019-10-13T10:24:57.240 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:24:57.241 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:57.244 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:24:57.244 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:57.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:24:57.696+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.702682+0000) 2019-10-13T10:24:57.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:24:57.701+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.703096+0000) 2019-10-13T10:24:57.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:57.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.703697+0000) 2019-10-13T10:24:57.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:57.700+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.704078+0000) 2019-10-13T10:24:57.706 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:24:57.700+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.704475+0000) 2019-10-13T10:24:57.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:24:57.700+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.704891+0000) 2019-10-13T10:24:57.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:24:57.700+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.705251+0000) 2019-10-13T10:24:57.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:24:57.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99132 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:24:57.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:24:57.705+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.709061+0000) 2019-10-13T10:24:58.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:24:58.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.592055+0000) 2019-10-13T10:24:58.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:24:58.596+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.597776+0000) 2019-10-13T10:25:01.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:01.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:01.890274+0000) 2019-10-13T10:25:02.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:02.076+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.081833+0000) 2019-10-13T10:25:02.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:02.709+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.711298+0000) 2019-10-13T10:25:02.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:02.708+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.711531+0000) 2019-10-13T10:25:02.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:02.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.711929+0000) 2019-10-13T10:25:02.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:02.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.712659+0000) 2019-10-13T10:25:02.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:02.708+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.712200+0000) 2019-10-13T10:25:02.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:02.708+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.713095+0000) 2019-10-13T10:25:02.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:02.708+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.713170+0000) 2019-10-13T10:25:02.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:02.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:02.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:02.717+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.719755+0000) 2019-10-13T10:25:03.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:03.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.079052+0000) 2019-10-13T10:25:05.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:05.073+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:05.074460+0000) 2019-10-13T10:25:05.111 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:05.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:05.110182+0000) 2019-10-13T10:25:06.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:06.327+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:06.328981+0000) 2019-10-13T10:25:07.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:07.711+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.718923+0000) 2019-10-13T10:25:07.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:07.715+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.719162+0000) 2019-10-13T10:25:07.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:07.715+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.720310+0000) 2019-10-13T10:25:07.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:07.715+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.721413+0000) 2019-10-13T10:25:07.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:07.717+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.721795+0000) 2019-10-13T10:25:07.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:07.715+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.722126+0000) 2019-10-13T10:25:07.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:07.715+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.722446+0000) 2019-10-13T10:25:07.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:07.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:07.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:07.725+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.729952+0000) 2019-10-13T10:25:08.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:08.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.592250+0000) 2019-10-13T10:25:08.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:08.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.597957+0000) 2019-10-13T10:25:11.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:11.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:11.890527+0000) 2019-10-13T10:25:12.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:12.079+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.082035+0000) 2019-10-13T10:25:12.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:12.723+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.726976+0000) 2019-10-13T10:25:12.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:12.725+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.727714+0000) 2019-10-13T10:25:12.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:12.723+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.728027+0000) 2019-10-13T10:25:12.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:12.723+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.729381+0000) 2019-10-13T10:25:12.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:12.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.729673+0000) 2019-10-13T10:25:12.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:12.723+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.729994+0000) 2019-10-13T10:25:12.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:12.723+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.730271+0000) 2019-10-13T10:25:12.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:12.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:12.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:12.733+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.735616+0000) 2019-10-13T10:25:13.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:13.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.079228+0000) 2019-10-13T10:25:15.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:15.077+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:15.080944+0000) 2019-10-13T10:25:15.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:15.103+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:15.110383+0000) 2019-10-13T10:25:16.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:16.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:16.329168+0000) 2019-10-13T10:25:17.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:17.727+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.735030+0000) 2019-10-13T10:25:17.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:17.733+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.735999+0000) 2019-10-13T10:25:17.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:17.731+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.736253+0000) 2019-10-13T10:25:17.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:17.731+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.736570+0000) 2019-10-13T10:25:17.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:17.731+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.736916+0000) 2019-10-13T10:25:17.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:17.731+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.738600+0000) 2019-10-13T10:25:17.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:17.731+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.738934+0000) 2019-10-13T10:25:17.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:17.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:17.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:17.737+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.742031+0000) 2019-10-13T10:25:18.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:18.587+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.592448+0000) 2019-10-13T10:25:18.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:18.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.598155+0000) 2019-10-13T10:25:21.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:21.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:21.890738+0000) 2019-10-13T10:25:22.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:22.075+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.082228+0000) 2019-10-13T10:25:22.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:22.739+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.743573+0000) 2019-10-13T10:25:22.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:22.739+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.743949+0000) 2019-10-13T10:25:22.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:22.739+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.744384+0000) 2019-10-13T10:25:22.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:22.739+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.745226+0000) 2019-10-13T10:25:22.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:22.739+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.745910+0000) 2019-10-13T10:25:22.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:22.745+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.746412+0000) 2019-10-13T10:25:22.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:22.739+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.747192+0000) 2019-10-13T10:25:22.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:22.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:22.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:22.749+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.751172+0000) 2019-10-13T10:25:23.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:23.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:23.079402+0000) 2019-10-13T10:25:25.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:25.085+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:25.087500+0000) 2019-10-13T10:25:25.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:25.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:25.110585+0000) 2019-10-13T10:25:26.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:26.323+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:26.329345+0000) 2019-10-13T10:25:27.372 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:25:27.372 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:27.376 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:25:27.376 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:27.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:27.749+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.752040+0000) 2019-10-13T10:25:27.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:27.747+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.752466+0000) 2019-10-13T10:25:27.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:27.747+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.753036+0000) 2019-10-13T10:25:27.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:27.747+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.753606+0000) 2019-10-13T10:25:27.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:27.747+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.754425+0000) 2019-10-13T10:25:27.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:27.747+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.754686+0000) 2019-10-13T10:25:27.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:27.747+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.755212+0000) 2019-10-13T10:25:27.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:27.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:27.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:27.753+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.757762+0000) 2019-10-13T10:25:28.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:28.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:28.592628+0000) 2019-10-13T10:25:28.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:28.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:28.598332+0000) 2019-10-13T10:25:31.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:31.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:31.890934+0000) 2019-10-13T10:25:32.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:32.079+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.082428+0000) 2019-10-13T10:25:32.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:32.755+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.759773+0000) 2019-10-13T10:25:32.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:32.755+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.760022+0000) 2019-10-13T10:25:32.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:32.755+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.760902+0000) 2019-10-13T10:25:32.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:32.755+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.761592+0000) 2019-10-13T10:25:32.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:32.755+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.761905+0000) 2019-10-13T10:25:32.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:32.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.762768+0000) 2019-10-13T10:25:32.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:32.759+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.763360+0000) 2019-10-13T10:25:32.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:32.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:32.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:32.765+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.766957+0000) 2019-10-13T10:25:33.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:33.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:33.079573+0000) 2019-10-13T10:25:35.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:35.089+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:35.093244+0000) 2019-10-13T10:25:35.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:35.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:35.110781+0000) 2019-10-13T10:25:36.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:36.327+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:36.329559+0000) 2019-10-13T10:25:37.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:37.763+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.767116+0000) 2019-10-13T10:25:37.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:37.767+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.767806+0000) 2019-10-13T10:25:37.769 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:37.767+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.768173+0000) 2019-10-13T10:25:37.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:37.767+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.769416+0000) 2019-10-13T10:25:37.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:37.769+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.770501+0000) 2019-10-13T10:25:37.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:37.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.769982+0000) 2019-10-13T10:25:37.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:37.767+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.769729+0000) 2019-10-13T10:25:37.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:37.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:37.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:37.773+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.775548+0000) 2019-10-13T10:25:38.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:38.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.592810+0000) 2019-10-13T10:25:38.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:38.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.598523+0000) 2019-10-13T10:25:41.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:41.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:41.891144+0000) 2019-10-13T10:25:42.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:42.079+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.082658+0000) 2019-10-13T10:25:42.777 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:42.775+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.775689+0000) 2019-10-13T10:25:42.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:42.775+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.776363+0000) 2019-10-13T10:25:42.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:42.775+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.776605+0000) 2019-10-13T10:25:42.781 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:42.777+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.778705+0000) 2019-10-13T10:25:42.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:42.775+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.777441+0000) 2019-10-13T10:25:42.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:42.775+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.778009+0000) 2019-10-13T10:25:42.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:42.775+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.778454+0000) 2019-10-13T10:25:42.782 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:42.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:42.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:42.781+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.784459+0000) 2019-10-13T10:25:43.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:43.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.079770+0000) 2019-10-13T10:25:45.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:45.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:45.099698+0000) 2019-10-13T10:25:45.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:45.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:45.110979+0000) 2019-10-13T10:25:46.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:46.327+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:46.329785+0000) 2019-10-13T10:25:47.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:47.781+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.783953+0000) 2019-10-13T10:25:47.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:47.782+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.783734+0000) 2019-10-13T10:25:47.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:47.782+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.784542+0000) 2019-10-13T10:25:47.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:47.782+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.784778+0000) 2019-10-13T10:25:47.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:47.782+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.786041+0000) 2019-10-13T10:25:47.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:47.782+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.786349+0000) 2019-10-13T10:25:47.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:47.782+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.787075+0000) 2019-10-13T10:25:47.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:47.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:47.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:47.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.793539+0000) 2019-10-13T10:25:48.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:48.590+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.592987+0000) 2019-10-13T10:25:48.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:48.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.598724+0000) 2019-10-13T10:25:51.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:51.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:51.891397+0000) 2019-10-13T10:25:52.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:52.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.082858+0000) 2019-10-13T10:25:52.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:52.790+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.791485+0000) 2019-10-13T10:25:52.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:52.790+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.792398+0000) 2019-10-13T10:25:52.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:52.790+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.792942+0000) 2019-10-13T10:25:52.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:52.790+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.793315+0000) 2019-10-13T10:25:52.795 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:52.788+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.793709+0000) 2019-10-13T10:25:52.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:52.790+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.794248+0000) 2019-10-13T10:25:52.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:52.790+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.795011+0000) 2019-10-13T10:25:52.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:52.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:52.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:52.800+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.802379+0000) 2019-10-13T10:25:53.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:53.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.079953+0000) 2019-10-13T10:25:55.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:55.106+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:55.111193+0000) 2019-10-13T10:25:55.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:55.108+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:55.110483+0000) 2019-10-13T10:25:56.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:56.326+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:56.330024+0000) 2019-10-13T10:25:57.479 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:25:57.479 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:57.483 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:25:57.483 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:57.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:57.794+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.798921+0000) 2019-10-13T10:25:57.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:25:57.798+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.799392+0000) 2019-10-13T10:25:57.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:25:57.798+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.799992+0000) 2019-10-13T10:25:57.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:25:57.796+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.800732+0000) 2019-10-13T10:25:57.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:57.798+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.800313+0000) 2019-10-13T10:25:57.803 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:25:57.798+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.801278+0000) 2019-10-13T10:25:57.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:25:57.798+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.802628+0000) 2019-10-13T10:25:57.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:25:57.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:25:57.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:25:57.804+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.807847+0000) 2019-10-13T10:25:58.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:25:58.590+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.593116+0000) 2019-10-13T10:25:58.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:25:58.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.598918+0000) 2019-10-13T10:26:01.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:01.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:01.891603+0000) 2019-10-13T10:26:02.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:02.082+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.083057+0000) 2019-10-13T10:26:02.809 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:02.804+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.808388+0000) 2019-10-13T10:26:02.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:02.806+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.808241+0000) 2019-10-13T10:26:02.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:02.806+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.809338+0000) 2019-10-13T10:26:02.811 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:02.806+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.809773+0000) 2019-10-13T10:26:02.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:02.806+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.810113+0000) 2019-10-13T10:26:02.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:02.810+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.811363+0000) 2019-10-13T10:26:02.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:02.810+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.812045+0000) 2019-10-13T10:26:02.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:02.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:02.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:02.812+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.816533+0000) 2019-10-13T10:26:03.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:03.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.080083+0000) 2019-10-13T10:26:05.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:05.110+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:05.111398+0000) 2019-10-13T10:26:05.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:05.116+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:05.117004+0000) 2019-10-13T10:26:06.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:06.326+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:06.330216+0000) 2019-10-13T10:26:07.818 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:07.814+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.817314+0000) 2019-10-13T10:26:07.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:07.814+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.817680+0000) 2019-10-13T10:26:07.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:07.814+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.818020+0000) 2019-10-13T10:26:07.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:07.816+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.818879+0000) 2019-10-13T10:26:07.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:07.818+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.819777+0000) 2019-10-13T10:26:07.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:07.818+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.820032+0000) 2019-10-13T10:26:07.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:07.818+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.820446+0000) 2019-10-13T10:26:07.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:07.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:07.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:07.824+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.828384+0000) 2019-10-13T10:26:08.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:08.590+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.593258+0000) 2019-10-13T10:26:08.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:08.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.599094+0000) 2019-10-13T10:26:11.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:11.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:11.891824+0000) 2019-10-13T10:26:12.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:12.078+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.083252+0000) 2019-10-13T10:26:12.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:12.822+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.824833+0000) 2019-10-13T10:26:12.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:12.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.825654+0000) 2019-10-13T10:26:12.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:12.822+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.826128+0000) 2019-10-13T10:26:12.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:12.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.826875+0000) 2019-10-13T10:26:12.829 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:12.824+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.827031+0000) 2019-10-13T10:26:12.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:12.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.827743+0000) 2019-10-13T10:26:12.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:12.826+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.828559+0000) 2019-10-13T10:26:12.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:12.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:12.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:12.828+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.833432+0000) 2019-10-13T10:26:13.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:13.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.080251+0000) 2019-10-13T10:26:15.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:15.110+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:15.111611+0000) 2019-10-13T10:26:15.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:15.120+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:15.123475+0000) 2019-10-13T10:26:16.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:16.326+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:16.330446+0000) 2019-10-13T10:26:17.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:17.830+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.832975+0000) 2019-10-13T10:26:17.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:17.830+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.834091+0000) 2019-10-13T10:26:17.835 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:17.832+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.834403+0000) 2019-10-13T10:26:17.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:17.830+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.834606+0000) 2019-10-13T10:26:17.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:17.830+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.835066+0000) 2019-10-13T10:26:17.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:17.834+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.835379+0000) 2019-10-13T10:26:17.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:17.834+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.835765+0000) 2019-10-13T10:26:17.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:17.834+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:17.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:17.836+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.840491+0000) 2019-10-13T10:26:18.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:18.590+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.593451+0000) 2019-10-13T10:26:18.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:18.594+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.599265+0000) 2019-10-13T10:26:21.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:21.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:21.892076+0000) 2019-10-13T10:26:22.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:22.082+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.083488+0000) 2019-10-13T10:26:22.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:22.836+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.840206+0000) 2019-10-13T10:26:22.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:22.838+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.840707+0000) 2019-10-13T10:26:22.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:22.838+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.841434+0000) 2019-10-13T10:26:22.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:22.838+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.841753+0000) 2019-10-13T10:26:22.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:22.838+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.842769+0000) 2019-10-13T10:26:22.844 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:22.838+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.843150+0000) 2019-10-13T10:26:22.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:22.842+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.843784+0000) 2019-10-13T10:26:22.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:22.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:22.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:22.844+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.847874+0000) 2019-10-13T10:26:23.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:23.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.080430+0000) 2019-10-13T10:26:25.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:25.110+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:25.111860+0000) 2019-10-13T10:26:25.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:25.132+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:25.134897+0000) 2019-10-13T10:26:26.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:26.326+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:26.330669+0000) 2019-10-13T10:26:27.591 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:26:27.591 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:27.594 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:26:27.594 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:27.849 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:27.846+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.848435+0000) 2019-10-13T10:26:27.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:27.846+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.848874+0000) 2019-10-13T10:26:27.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:27.846+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.849972+0000) 2019-10-13T10:26:27.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:27.848+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.851051+0000) 2019-10-13T10:26:27.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:27.846+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.850571+0000) 2019-10-13T10:26:27.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:27.846+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.850707+0000) 2019-10-13T10:26:27.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:27.850+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.851697+0000) 2019-10-13T10:26:27.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:27.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:27.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:27.852+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.857224+0000) 2019-10-13T10:26:28.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:28.590+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.593649+0000) 2019-10-13T10:26:28.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:28.598+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.599473+0000) 2019-10-13T10:26:31.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:31.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:31.892282+0000) 2019-10-13T10:26:32.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:32.081+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.083694+0000) 2019-10-13T10:26:32.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:32.853+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.856337+0000) 2019-10-13T10:26:32.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:32.853+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.857955+0000) 2019-10-13T10:26:32.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:32.853+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.858503+0000) 2019-10-13T10:26:32.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:32.853+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.858828+0000) 2019-10-13T10:26:32.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:32.857+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.859600+0000) 2019-10-13T10:26:32.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:32.857+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.860131+0000) 2019-10-13T10:26:32.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:32.856+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.861060+0000) 2019-10-13T10:26:32.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:32.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:32.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:32.860+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.863779+0000) 2019-10-13T10:26:33.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:33.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.080592+0000) 2019-10-13T10:26:35.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:35.109+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:35.112118+0000) 2019-10-13T10:26:35.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:35.136+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:35.141509+0000) 2019-10-13T10:26:36.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:36.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:36.330856+0000) 2019-10-13T10:26:37.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:37.861+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.864575+0000) 2019-10-13T10:26:37.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:37.861+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.865548+0000) 2019-10-13T10:26:37.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:37.861+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.865778+0000) 2019-10-13T10:26:37.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:37.861+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.867108+0000) 2019-10-13T10:26:37.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:37.865+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.867437+0000) 2019-10-13T10:26:37.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:37.865+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.867802+0000) 2019-10-13T10:26:37.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:37.864+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.868225+0000) 2019-10-13T10:26:37.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:37.865+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:37.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:37.868+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.872740+0000) 2019-10-13T10:26:38.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:38.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.593842+0000) 2019-10-13T10:26:38.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:38.597+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.599694+0000) 2019-10-13T10:26:41.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:41.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:41.892505+0000) 2019-10-13T10:26:42.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:42.081+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.083891+0000) 2019-10-13T10:26:42.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:42.869+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.873192+0000) 2019-10-13T10:26:42.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:42.869+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.873832+0000) 2019-10-13T10:26:42.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:42.869+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.874513+0000) 2019-10-13T10:26:42.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:42.869+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.875049+0000) 2019-10-13T10:26:42.876 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:42.873+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.875599+0000) 2019-10-13T10:26:42.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:42.873+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.875873+0000) 2019-10-13T10:26:42.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:42.872+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.877029+0000) 2019-10-13T10:26:42.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:42.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:42.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:42.880+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.882631+0000) 2019-10-13T10:26:43.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:43.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.080795+0000) 2019-10-13T10:26:45.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:45.109+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:45.112310+0000) 2019-10-13T10:26:45.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:45.144+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:45.148166+0000) 2019-10-13T10:26:46.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:46.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:46.331046+0000) 2019-10-13T10:26:47.883 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:47.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.881402+0000) 2019-10-13T10:26:47.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:47.877+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.882256+0000) 2019-10-13T10:26:47.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:47.877+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.882807+0000) 2019-10-13T10:26:47.885 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:47.881+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.883377+0000) 2019-10-13T10:26:47.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:47.881+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.883819+0000) 2019-10-13T10:26:47.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:47.881+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.884143+0000) 2019-10-13T10:26:47.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:47.879+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.884682+0000) 2019-10-13T10:26:47.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:47.881+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:47.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:47.883+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.888890+0000) 2019-10-13T10:26:48.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:48.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.594024+0000) 2019-10-13T10:26:48.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:48.597+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.599870+0000) 2019-10-13T10:26:51.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:51.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:51.892704+0000) 2019-10-13T10:26:52.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:52.081+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.084039+0000) 2019-10-13T10:26:52.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:52.885+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.889021+0000) 2019-10-13T10:26:52.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:52.885+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.890416+0000) 2019-10-13T10:26:52.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:52.885+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.890646+0000) 2019-10-13T10:26:52.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:52.887+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.891307+0000) 2019-10-13T10:26:52.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:52.889+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.891781+0000) 2019-10-13T10:26:52.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:52.889+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.892148+0000) 2019-10-13T10:26:52.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:52.889+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.892625+0000) 2019-10-13T10:26:52.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:52.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:52.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:52.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.899217+0000) 2019-10-13T10:26:53.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:53.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.080971+0000) 2019-10-13T10:26:55.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:55.109+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:55.112539+0000) 2019-10-13T10:26:55.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:55.151+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:55.154580+0000) 2019-10-13T10:26:56.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:56.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:56.331273+0000) 2019-10-13T10:26:57.709 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:26:57.709 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:57.713 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:26:57.713 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:57.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:26:57.893+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.898481+0000) 2019-10-13T10:26:57.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:57.893+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.898883+0000) 2019-10-13T10:26:57.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:26:57.897+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.899635+0000) 2019-10-13T10:26:57.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:57.897+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.900080+0000) 2019-10-13T10:26:57.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:26:57.897+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.900384+0000) 2019-10-13T10:26:57.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:26:57.895+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.901870+0000) 2019-10-13T10:26:57.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:26:57.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.902534+0000) 2019-10-13T10:26:57.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:26:57.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:26:57.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:26:57.903+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.908393+0000) 2019-10-13T10:26:58.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:26:58.589+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.594201+0000) 2019-10-13T10:26:58.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:26:58.597+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.600046+0000) 2019-10-13T10:27:01.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:01.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:01.892910+0000) 2019-10-13T10:27:02.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:02.081+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.084267+0000) 2019-10-13T10:27:02.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:02.905+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.907581+0000) 2019-10-13T10:27:02.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:02.905+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.908194+0000) 2019-10-13T10:27:02.909 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:02.905+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.908522+0000) 2019-10-13T10:27:02.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:02.905+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.909226+0000) 2019-10-13T10:27:02.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:02.905+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.910277+0000) 2019-10-13T10:27:02.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:02.907+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.910745+0000) 2019-10-13T10:27:02.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:02.905+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.911126+0000) 2019-10-13T10:27:02.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:02.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 99965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:02.917 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:02.911+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.915832+0000) 2019-10-13T10:27:03.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:03.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.081146+0000) 2019-10-13T10:27:05.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:05.109+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:05.112744+0000) 2019-10-13T10:27:05.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:05.155+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:05.161511+0000) 2019-10-13T10:27:06.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:06.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:06.331525+0000) 2019-10-13T10:27:07.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:07.909+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.914587+0000) 2019-10-13T10:27:07.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:07.913+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.915777+0000) 2019-10-13T10:27:07.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:07.913+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.916469+0000) 2019-10-13T10:27:07.918 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:07.913+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.916651+0000) 2019-10-13T10:27:07.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:07.913+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.917220+0000) 2019-10-13T10:27:07.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:07.913+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.917433+0000) 2019-10-13T10:27:07.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:07.915+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.918576+0000) 2019-10-13T10:27:07.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:07.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:07.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:07.919+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.924329+0000) 2019-10-13T10:27:08.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:08.593+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.594410+0000) 2019-10-13T10:27:08.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:08.597+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.600170+0000) 2019-10-13T10:27:11.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:11.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:11.893128+0000) 2019-10-13T10:27:12.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:12.081+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.084470+0000) 2019-10-13T10:27:12.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:12.917+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.922810+0000) 2019-10-13T10:27:12.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:12.917+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.923069+0000) 2019-10-13T10:27:12.927 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:12.921+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.924040+0000) 2019-10-13T10:27:12.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:12.921+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.924658+0000) 2019-10-13T10:27:12.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:12.921+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.924870+0000) 2019-10-13T10:27:12.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:12.921+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.925072+0000) 2019-10-13T10:27:12.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:12.919+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.925570+0000) 2019-10-13T10:27:12.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:12.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:12.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:12.927+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.932061+0000) 2019-10-13T10:27:13.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:13.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.081335+0000) 2019-10-13T10:27:15.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:15.108+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:15.112945+0000) 2019-10-13T10:27:15.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:15.163+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:15.168624+0000) 2019-10-13T10:27:16.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:16.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:16.331750+0000) 2019-10-13T10:27:17.932 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:17.924+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.930760+0000) 2019-10-13T10:27:17.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:17.928+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.931900+0000) 2019-10-13T10:27:17.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:17.928+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.932085+0000) 2019-10-13T10:27:17.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:17.928+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.933100+0000) 2019-10-13T10:27:17.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:17.928+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.933430+0000) 2019-10-13T10:27:17.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:17.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.934312+0000) 2019-10-13T10:27:17.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:17.931+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.934518+0000) 2019-10-13T10:27:17.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:17.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:17.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:17.935+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.938817+0000) 2019-10-13T10:27:18.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:18.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.594590+0000) 2019-10-13T10:27:18.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:18.596+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.600352+0000) 2019-10-13T10:27:21.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:21.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:21.893355+0000) 2019-10-13T10:27:22.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:22.080+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.084671+0000) 2019-10-13T10:27:22.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:22.932+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.939081+0000) 2019-10-13T10:27:22.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:22.935+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.940551+0000) 2019-10-13T10:27:22.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:22.936+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.941087+0000) 2019-10-13T10:27:22.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:22.936+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.941603+0000) 2019-10-13T10:27:22.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:22.936+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.941991+0000) 2019-10-13T10:27:22.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:22.936+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.942440+0000) 2019-10-13T10:27:22.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:22.936+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.942920+0000) 2019-10-13T10:27:22.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:22.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:22.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:22.943+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.947628+0000) 2019-10-13T10:27:23.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:23.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.081546+0000) 2019-10-13T10:27:25.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:25.112+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:25.113176+0000) 2019-10-13T10:27:25.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:25.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:25.175907+0000) 2019-10-13T10:27:26.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:26.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:26.331945+0000) 2019-10-13T10:27:27.842 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:27:27.842 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:27.846 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:27:27.847 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:27.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:27.944+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.947610+0000) 2019-10-13T10:27:27.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:27.944+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.948007+0000) 2019-10-13T10:27:27.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:27.944+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.948599+0000) 2019-10-13T10:27:27.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:27.943+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.949023+0000) 2019-10-13T10:27:27.952 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:27.944+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.949608+0000) 2019-10-13T10:27:27.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:27.944+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.950234+0000) 2019-10-13T10:27:27.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:27.944+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.950573+0000) 2019-10-13T10:27:27.953 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:27.944+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100132 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:27.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:27.951+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.956624+0000) 2019-10-13T10:27:28.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:28.592+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.594767+0000) 2019-10-13T10:27:28.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:28.596+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.600518+0000) 2019-10-13T10:27:31.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:31.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:31.893583+0000) 2019-10-13T10:27:32.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:32.080+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.084949+0000) 2019-10-13T10:27:32.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:32.951+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.954349+0000) 2019-10-13T10:27:32.956 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:32.948+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.954761+0000) 2019-10-13T10:27:32.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:32.948+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.954977+0000) 2019-10-13T10:27:32.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:32.948+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.955133+0000) 2019-10-13T10:27:32.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:32.952+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.955906+0000) 2019-10-13T10:27:32.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:32.952+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.956118+0000) 2019-10-13T10:27:32.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:32.952+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.956086+0000) 2019-10-13T10:27:32.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:32.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:32.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:32.959+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.962851+0000) 2019-10-13T10:27:33.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:33.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.081742+0000) 2019-10-13T10:27:35.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:35.108+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:35.113400+0000) 2019-10-13T10:27:35.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:35.179+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:35.182951+0000) 2019-10-13T10:27:36.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:36.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:36.332152+0000) 2019-10-13T10:27:37.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:37.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.961525+0000) 2019-10-13T10:27:37.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:37.956+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.961767+0000) 2019-10-13T10:27:37.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:37.956+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.962000+0000) 2019-10-13T10:27:37.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:37.956+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.962136+0000) 2019-10-13T10:27:37.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:37.956+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.962823+0000) 2019-10-13T10:27:37.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:37.956+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.962953+0000) 2019-10-13T10:27:37.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:37.956+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.963069+0000) 2019-10-13T10:27:37.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:37.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:37.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:37.967+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.970347+0000) 2019-10-13T10:27:38.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:38.592+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.594943+0000) 2019-10-13T10:27:38.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:38.600+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.600702+0000) 2019-10-13T10:27:41.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:41.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:41.893815+0000) 2019-10-13T10:27:42.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:42.080+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.085162+0000) 2019-10-13T10:27:42.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:42.964+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.969409+0000) 2019-10-13T10:27:42.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:42.964+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.969764+0000) 2019-10-13T10:27:42.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:42.964+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.971167+0000) 2019-10-13T10:27:42.972 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:42.968+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.971367+0000) 2019-10-13T10:27:42.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:42.966+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.972073+0000) 2019-10-13T10:27:42.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:42.968+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.972596+0000) 2019-10-13T10:27:42.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:42.968+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.972849+0000) 2019-10-13T10:27:42.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:42.970+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.974699+0000) 2019-10-13T10:27:42.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:42.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:43.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:43.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.081934+0000) 2019-10-13T10:27:45.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:45.108+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:45.113601+0000) 2019-10-13T10:27:45.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:45.186+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:45.189972+0000) 2019-10-13T10:27:46.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:46.332+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:46.332355+0000) 2019-10-13T10:27:47.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:47.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.977838+0000) 2019-10-13T10:27:47.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:47.972+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.977934+0000) 2019-10-13T10:27:47.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:47.972+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.978106+0000) 2019-10-13T10:27:47.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:47.976+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.980245+0000) 2019-10-13T10:27:47.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:47.976+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.980375+0000) 2019-10-13T10:27:47.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:47.976+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.980686+0000) 2019-10-13T10:27:47.982 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:47.976+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.980980+0000) 2019-10-13T10:27:47.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:47.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100264 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:47.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:47.982+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.989653+0000) 2019-10-13T10:27:48.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:48.588+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.595118+0000) 2019-10-13T10:27:48.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:48.600+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.600908+0000) 2019-10-13T10:27:51.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:51.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:51.893966+0000) 2019-10-13T10:27:52.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:52.084+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.085358+0000) 2019-10-13T10:27:52.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:52.980+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.985786+0000) 2019-10-13T10:27:52.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:52.980+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.985981+0000) 2019-10-13T10:27:52.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:52.982+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.986916+0000) 2019-10-13T10:27:52.988 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:52.980+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.986831+0000) 2019-10-13T10:27:52.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:52.980+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.987159+0000) 2019-10-13T10:27:52.989 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:52.980+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.987193+0000) 2019-10-13T10:27:52.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:52.980+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.986328+0000) 2019-10-13T10:27:52.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:52.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:52.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:52.990+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.994201+0000) 2019-10-13T10:27:53.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:53.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.082111+0000) 2019-10-13T10:27:55.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:55.108+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:55.113797+0000) 2019-10-13T10:27:55.198 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:55.194+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:55.196943+0000) 2019-10-13T10:27:56.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:56.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:56.332546+0000) 2019-10-13T10:27:57.974 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:27:57.974 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:58.000 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:27:58.001 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:58.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:27:57.987+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.995040+0000) 2019-10-13T10:27:58.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:27:57.990+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.994639+0000) 2019-10-13T10:27:58.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:57.987+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.995197+0000) 2019-10-13T10:27:58.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:27:57.991+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.995848+0000) 2019-10-13T10:27:58.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:27:57.991+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.996351+0000) 2019-10-13T10:27:58.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:27:57.991+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.996445+0000) 2019-10-13T10:27:58.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:57.991+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.997114+0000) 2019-10-13T10:27:58.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:27:57.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:27:58.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:27:57.998+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.005472+0000) 2019-10-13T10:27:58.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:27:58.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.595299+0000) 2019-10-13T10:27:58.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:27:58.595+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.601080+0000) 2019-10-13T10:28:01.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:01.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:01.894169+0000) 2019-10-13T10:28:02.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:02.083+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:02.085563+0000) 2019-10-13T10:28:03.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:02.995+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.001679+0000) 2019-10-13T10:28:03.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:02.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.002243+0000) 2019-10-13T10:28:03.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:02.998+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.002839+0000) 2019-10-13T10:28:03.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:02.995+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.003188+0000) 2019-10-13T10:28:03.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:02.999+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.003827+0000) 2019-10-13T10:28:03.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:02.999+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.004678+0000) 2019-10-13T10:28:03.006 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:02.999+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.005022+0000) 2019-10-13T10:28:03.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:02.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:03.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:03.006+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.010843+0000) 2019-10-13T10:28:03.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:03.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.082275+0000) 2019-10-13T10:28:05.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:05.107+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:05.114032+0000) 2019-10-13T10:28:05.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:05.202+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:05.203943+0000) 2019-10-13T10:28:06.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:06.331+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:06.332733+0000) 2019-10-13T10:28:08.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:08.003+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.008826+0000) 2019-10-13T10:28:08.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:08.003+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.009464+0000) 2019-10-13T10:28:08.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:08.003+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.010546+0000) 2019-10-13T10:28:08.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:08.006+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.010981+0000) 2019-10-13T10:28:08.013 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:08.003+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.011131+0000) 2019-10-13T10:28:08.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:08.007+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.011556+0000) 2019-10-13T10:28:08.014 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:08.007+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.012025+0000) 2019-10-13T10:28:08.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:08.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:08.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:08.014+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.020866+0000) 2019-10-13T10:28:08.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:08.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.595536+0000) 2019-10-13T10:28:08.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:08.599+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.601256+0000) 2019-10-13T10:28:11.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:11.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:11.894401+0000) 2019-10-13T10:28:12.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:12.079+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:12.085764+0000) 2019-10-13T10:28:13.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:13.011+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.017788+0000) 2019-10-13T10:28:13.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:13.011+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.018212+0000) 2019-10-13T10:28:13.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:13.011+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.018551+0000) 2019-10-13T10:28:13.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:13.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.020000+0000) 2019-10-13T10:28:13.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:13.015+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.019903+0000) 2019-10-13T10:28:13.022 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:13.015+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.020411+0000) 2019-10-13T10:28:13.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:13.015+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.020722+0000) 2019-10-13T10:28:13.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:13.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:13.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:13.022+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.026998+0000) 2019-10-13T10:28:13.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:13.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.082462+0000) 2019-10-13T10:28:15.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:15.111+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:15.114256+0000) 2019-10-13T10:28:15.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:15.210+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:15.211108+0000) 2019-10-13T10:28:16.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:16.331+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:16.332920+0000) 2019-10-13T10:28:18.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:18.019+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.026021+0000) 2019-10-13T10:28:18.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:18.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.027349+0000) 2019-10-13T10:28:18.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:18.023+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.027885+0000) 2019-10-13T10:28:18.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:18.023+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.028271+0000) 2019-10-13T10:28:18.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:18.023+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.028836+0000) 2019-10-13T10:28:18.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:18.022+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.028964+0000) 2019-10-13T10:28:18.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:18.023+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.029706+0000) 2019-10-13T10:28:18.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:18.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:18.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:18.030+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.036572+0000) 2019-10-13T10:28:18.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:18.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.595740+0000) 2019-10-13T10:28:18.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:18.599+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.601430+0000) 2019-10-13T10:28:21.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:21.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:21.894570+0000) 2019-10-13T10:28:22.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:22.083+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:22.085965+0000) 2019-10-13T10:28:23.036 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:23.027+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.034883+0000) 2019-10-13T10:28:23.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:23.027+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.035016+0000) 2019-10-13T10:28:23.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:23.031+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.035556+0000) 2019-10-13T10:28:23.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:23.030+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.036240+0000) 2019-10-13T10:28:23.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:23.031+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.036608+0000) 2019-10-13T10:28:23.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:23.031+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.036858+0000) 2019-10-13T10:28:23.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:23.031+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.038274+0000) 2019-10-13T10:28:23.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:23.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:23.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:23.038+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.043423+0000) 2019-10-13T10:28:23.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:23.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.082630+0000) 2019-10-13T10:28:25.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:25.111+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:25.114461+0000) 2019-10-13T10:28:25.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:25.214+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:25.218180+0000) 2019-10-13T10:28:26.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:26.331+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:26.333110+0000) 2019-10-13T10:28:28.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:28.038+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.043059+0000) 2019-10-13T10:28:28.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:28.039+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.043388+0000) 2019-10-13T10:28:28.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:28.039+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.044187+0000) 2019-10-13T10:28:28.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:28.039+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.044147+0000) 2019-10-13T10:28:28.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:28.039+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.044898+0000) 2019-10-13T10:28:28.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:28.039+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.045634+0000) 2019-10-13T10:28:28.047 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:28.039+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.046079+0000) 2019-10-13T10:28:28.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:28.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:28.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:28.046+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.053511+0000) 2019-10-13T10:28:28.117 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:28:28.117 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:28.121 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:28:28.121 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:28.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:28.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.595935+0000) 2019-10-13T10:28:28.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:28.599+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.601603+0000) 2019-10-13T10:28:31.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:31.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:31.894766+0000) 2019-10-13T10:28:32.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:32.083+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:32.086163+0000) 2019-10-13T10:28:33.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:33.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.050954+0000) 2019-10-13T10:28:33.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:33.051+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.051772+0000) 2019-10-13T10:28:33.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:33.051+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.052146+0000) 2019-10-13T10:28:33.054 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:33.051+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.052497+0000) 2019-10-13T10:28:33.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:33.051+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.053781+0000) 2019-10-13T10:28:33.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:33.051+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.054149+0000) 2019-10-13T10:28:33.055 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:33.051+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.054754+0000) 2019-10-13T10:28:33.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:33.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:33.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:33.058+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.060190+0000) 2019-10-13T10:28:33.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:33.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.082796+0000) 2019-10-13T10:28:35.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:35.111+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:35.114656+0000) 2019-10-13T10:28:35.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:35.222+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:35.225225+0000) 2019-10-13T10:28:36.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:36.331+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:36.333305+0000) 2019-10-13T10:28:38.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:38.059+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.060830+0000) 2019-10-13T10:28:38.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:38.059+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.061206+0000) 2019-10-13T10:28:38.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:38.057+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.061709+0000) 2019-10-13T10:28:38.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:38.059+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.062272+0000) 2019-10-13T10:28:38.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:38.059+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.062318+0000) 2019-10-13T10:28:38.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:38.059+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.062602+0000) 2019-10-13T10:28:38.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:38.063+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.063527+0000) 2019-10-13T10:28:38.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:38.063+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100594 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:38.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:38.065+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.066661+0000) 2019-10-13T10:28:38.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:38.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.596113+0000) 2019-10-13T10:28:38.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:38.599+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.601776+0000) 2019-10-13T10:28:41.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:41.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:41.894966+0000) 2019-10-13T10:28:42.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:42.082+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:42.086378+0000) 2019-10-13T10:28:43.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:43.066+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.068507+0000) 2019-10-13T10:28:43.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:43.065+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.068758+0000) 2019-10-13T10:28:43.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:43.066+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.069110+0000) 2019-10-13T10:28:43.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:43.066+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.070147+0000) 2019-10-13T10:28:43.072 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:43.066+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.070822+0000) 2019-10-13T10:28:43.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:43.066+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.071159+0000) 2019-10-13T10:28:43.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:43.070+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.071529+0000) 2019-10-13T10:28:43.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:43.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:43.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:43.077+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.079402+0000) 2019-10-13T10:28:43.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:43.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.082912+0000) 2019-10-13T10:28:45.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:45.110+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:45.114858+0000) 2019-10-13T10:28:45.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:45.229+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:45.232294+0000) 2019-10-13T10:28:46.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:46.330+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:46.333497+0000) 2019-10-13T10:28:48.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:48.074+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.077390+0000) 2019-10-13T10:28:48.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:48.073+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.077732+0000) 2019-10-13T10:28:48.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:48.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.077937+0000) 2019-10-13T10:28:48.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:48.074+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.078565+0000) 2019-10-13T10:28:48.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:48.074+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.078997+0000) 2019-10-13T10:28:48.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:48.078+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.080217+0000) 2019-10-13T10:28:48.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:48.078+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.079975+0000) 2019-10-13T10:28:48.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:48.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:48.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:48.081+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.085805+0000) 2019-10-13T10:28:48.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:48.594+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.596317+0000) 2019-10-13T10:28:48.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:48.598+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.601946+0000) 2019-10-13T10:28:51.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:51.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:51.895189+0000) 2019-10-13T10:28:52.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:52.082+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:52.086577+0000) 2019-10-13T10:28:53.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:53.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.083076+0000) 2019-10-13T10:28:53.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:53.082+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.086132+0000) 2019-10-13T10:28:53.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:53.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.086588+0000) 2019-10-13T10:28:53.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:53.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.087536+0000) 2019-10-13T10:28:53.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:53.086+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.087945+0000) 2019-10-13T10:28:53.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:53.085+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.088506+0000) 2019-10-13T10:28:53.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:53.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.088800+0000) 2019-10-13T10:28:53.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:53.086+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.089405+0000) 2019-10-13T10:28:53.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:53.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:53.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:53.093+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.095444+0000) 2019-10-13T10:28:55.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:55.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:55.115096+0000) 2019-10-13T10:28:55.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:55.237+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:55.239425+0000) 2019-10-13T10:28:56.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:56.330+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:56.333710+0000) 2019-10-13T10:28:58.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:28:58.090+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.094389+0000) 2019-10-13T10:28:58.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:28:58.094+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.096039+0000) 2019-10-13T10:28:58.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:58.094+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.096438+0000) 2019-10-13T10:28:58.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:58.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.096891+0000) 2019-10-13T10:28:58.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:28:58.094+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.097311+0000) 2019-10-13T10:28:58.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:28:58.093+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.097781+0000) 2019-10-13T10:28:58.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:28:58.094+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.097975+0000) 2019-10-13T10:28:58.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:28:58.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100726 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:28:58.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:28:58.101+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.104639+0000) 2019-10-13T10:28:58.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:28:58.594+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.596513+0000) 2019-10-13T10:28:58.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:28:58.598+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.602120+0000) 2019-10-13T10:28:58.622 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:28:58.622 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:58.626 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:28:58.626 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:01.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:01.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:01.895443+0000) 2019-10-13T10:29:02.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:02.082+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:02.086775+0000) 2019-10-13T10:29:03.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:03.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.083276+0000) 2019-10-13T10:29:03.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:03.102+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.103891+0000) 2019-10-13T10:29:03.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:03.102+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.103931+0000) 2019-10-13T10:29:03.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:03.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.104486+0000) 2019-10-13T10:29:03.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:03.102+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.105552+0000) 2019-10-13T10:29:03.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:03.101+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.105910+0000) 2019-10-13T10:29:03.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:03.102+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.106151+0000) 2019-10-13T10:29:03.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:03.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.106748+0000) 2019-10-13T10:29:03.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:03.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:03.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:03.109+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.112797+0000) 2019-10-13T10:29:05.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:05.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:05.115349+0000) 2019-10-13T10:29:05.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:05.245+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:05.248560+0000) 2019-10-13T10:29:06.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:06.330+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:06.333906+0000) 2019-10-13T10:29:08.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:08.106+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.110978+0000) 2019-10-13T10:29:08.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:08.110+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.111739+0000) 2019-10-13T10:29:08.113 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:08.110+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.111932+0000) 2019-10-13T10:29:08.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:08.110+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.112118+0000) 2019-10-13T10:29:08.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:08.109+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.113689+0000) 2019-10-13T10:29:08.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:08.110+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.114098+0000) 2019-10-13T10:29:08.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:08.110+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.114661+0000) 2019-10-13T10:29:08.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:08.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:08.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:08.117+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.119828+0000) 2019-10-13T10:29:08.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:08.594+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.596727+0000) 2019-10-13T10:29:08.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:08.598+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.602323+0000) 2019-10-13T10:29:11.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:11.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:11.895662+0000) 2019-10-13T10:29:12.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:12.086+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:12.086950+0000) 2019-10-13T10:29:13.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:13.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.083483+0000) 2019-10-13T10:29:13.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:13.118+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.119460+0000) 2019-10-13T10:29:13.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:13.118+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.120074+0000) 2019-10-13T10:29:13.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:13.118+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.120530+0000) 2019-10-13T10:29:13.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:13.118+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.121026+0000) 2019-10-13T10:29:13.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:13.118+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.122322+0000) 2019-10-13T10:29:13.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:13.121+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.122900+0000) 2019-10-13T10:29:13.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:13.118+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.123224+0000) 2019-10-13T10:29:13.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:13.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:13.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:13.125+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.127170+0000) 2019-10-13T10:29:15.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:15.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:15.115566+0000) 2019-10-13T10:29:15.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:15.253+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:15.255030+0000) 2019-10-13T10:29:16.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:16.330+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:16.334098+0000) 2019-10-13T10:29:18.129 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:18.126+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.127699+0000) 2019-10-13T10:29:18.129 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:18.126+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.128153+0000) 2019-10-13T10:29:18.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:18.126+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.128872+0000) 2019-10-13T10:29:18.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:18.126+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.129149+0000) 2019-10-13T10:29:18.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:18.126+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.129908+0000) 2019-10-13T10:29:18.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:18.129+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.130453+0000) 2019-10-13T10:29:18.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:18.126+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.131122+0000) 2019-10-13T10:29:18.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:18.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:18.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:18.133+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.136400+0000) 2019-10-13T10:29:18.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:18.594+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.596939+0000) 2019-10-13T10:29:18.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:18.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.602502+0000) 2019-10-13T10:29:21.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:21.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:21.895913+0000) 2019-10-13T10:29:22.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:22.086+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:22.087153+0000) 2019-10-13T10:29:23.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:23.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.083716+0000) 2019-10-13T10:29:23.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:23.134+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.136258+0000) 2019-10-13T10:29:23.138 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:23.134+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.136781+0000) 2019-10-13T10:29:23.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:23.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.137189+0000) 2019-10-13T10:29:23.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:23.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.137563+0000) 2019-10-13T10:29:23.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:23.137+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.138542+0000) 2019-10-13T10:29:23.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:23.134+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.138270+0000) 2019-10-13T10:29:23.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:23.134+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.138859+0000) 2019-10-13T10:29:23.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:23.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:23.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:23.141+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.145606+0000) 2019-10-13T10:29:25.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:25.113+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:25.115770+0000) 2019-10-13T10:29:25.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:25.257+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:25.261535+0000) 2019-10-13T10:29:26.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:26.333+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:26.334296+0000) 2019-10-13T10:29:28.145 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:28.141+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.144000+0000) 2019-10-13T10:29:28.146 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:28.141+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.144628+0000) 2019-10-13T10:29:28.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:28.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.145307+0000) 2019-10-13T10:29:28.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:28.141+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.145560+0000) 2019-10-13T10:29:28.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:28.141+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.145866+0000) 2019-10-13T10:29:28.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:28.141+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.146091+0000) 2019-10-13T10:29:28.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:28.141+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.146774+0000) 2019-10-13T10:29:28.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:28.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:28.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:28.149+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.153950+0000) 2019-10-13T10:29:28.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:28.593+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.597120+0000) 2019-10-13T10:29:28.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:28.601+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.602699+0000) 2019-10-13T10:29:28.687 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:29:28.687 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:28.690 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:29:28.691 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:31.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:31.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:31.896148+0000) 2019-10-13T10:29:32.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:32.085+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:32.087413+0000) 2019-10-13T10:29:33.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:33.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.083971+0000) 2019-10-13T10:29:33.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:33.149+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.153149+0000) 2019-10-13T10:29:33.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:33.149+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.153531+0000) 2019-10-13T10:29:33.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:33.149+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.154158+0000) 2019-10-13T10:29:33.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:33.152+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.155158+0000) 2019-10-13T10:29:33.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:33.153+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.155369+0000) 2019-10-13T10:29:33.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:33.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.155549+0000) 2019-10-13T10:29:33.157 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:33.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.156120+0000) 2019-10-13T10:29:33.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:33.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:33.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:33.160+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.162813+0000) 2019-10-13T10:29:35.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:35.113+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:35.116004+0000) 2019-10-13T10:29:35.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:35.264+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:35.267918+0000) 2019-10-13T10:29:36.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:36.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:36.334485+0000) 2019-10-13T10:29:38.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:38.156+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.161106+0000) 2019-10-13T10:29:38.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:38.157+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.161542+0000) 2019-10-13T10:29:38.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:38.157+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.162248+0000) 2019-10-13T10:29:38.163 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:38.157+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.162569+0000) 2019-10-13T10:29:38.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:38.157+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.163229+0000) 2019-10-13T10:29:38.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:38.161+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.163810+0000) 2019-10-13T10:29:38.165 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:38.161+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.164392+0000) 2019-10-13T10:29:38.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:38.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 100989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:38.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:38.168+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.172867+0000) 2019-10-13T10:29:38.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:38.593+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.597330+0000) 2019-10-13T10:29:38.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:38.601+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.602910+0000) 2019-10-13T10:29:41.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:41.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:41.896374+0000) 2019-10-13T10:29:42.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:42.085+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:42.087614+0000) 2019-10-13T10:29:43.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:43.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.084170+0000) 2019-10-13T10:29:43.171 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:43.165+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.169892+0000) 2019-10-13T10:29:43.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:43.165+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.170402+0000) 2019-10-13T10:29:43.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:43.165+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.169905+0000) 2019-10-13T10:29:43.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:43.165+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.170715+0000) 2019-10-13T10:29:43.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:43.165+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.171257+0000) 2019-10-13T10:29:43.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:43.169+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.172453+0000) 2019-10-13T10:29:43.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:43.168+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.172819+0000) 2019-10-13T10:29:43.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:43.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:43.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:43.176+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.180642+0000) 2019-10-13T10:29:45.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:45.113+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:45.116217+0000) 2019-10-13T10:29:45.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:45.272+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:45.274245+0000) 2019-10-13T10:29:46.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:46.333+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:46.334681+0000) 2019-10-13T10:29:48.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:48.173+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.178043+0000) 2019-10-13T10:29:48.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:48.173+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.178794+0000) 2019-10-13T10:29:48.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:48.176+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.179197+0000) 2019-10-13T10:29:48.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:48.177+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.179503+0000) 2019-10-13T10:29:48.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:48.177+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.179859+0000) 2019-10-13T10:29:48.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:48.177+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.180910+0000) 2019-10-13T10:29:48.182 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:48.177+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.181489+0000) 2019-10-13T10:29:48.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:48.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:48.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:48.184+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.186438+0000) 2019-10-13T10:29:48.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:48.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.597507+0000) 2019-10-13T10:29:48.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:48.601+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.603084+0000) 2019-10-13T10:29:51.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:51.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:51.896611+0000) 2019-10-13T10:29:52.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:52.085+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:52.087825+0000) 2019-10-13T10:29:53.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:53.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.084407+0000) 2019-10-13T10:29:53.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:53.181+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.186523+0000) 2019-10-13T10:29:53.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:53.181+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.186646+0000) 2019-10-13T10:29:53.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:53.181+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.187085+0000) 2019-10-13T10:29:53.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:53.185+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.187425+0000) 2019-10-13T10:29:53.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:53.185+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.187959+0000) 2019-10-13T10:29:53.189 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:53.185+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.188680+0000) 2019-10-13T10:29:53.190 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:53.184+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.189532+0000) 2019-10-13T10:29:53.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:53.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:53.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:53.192+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.194443+0000) 2019-10-13T10:29:55.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:55.113+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:55.116450+0000) 2019-10-13T10:29:55.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:55.276+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:55.280582+0000) 2019-10-13T10:29:56.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:56.333+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:56.334873+0000) 2019-10-13T10:29:58.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:29:58.189+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.193596+0000) 2019-10-13T10:29:58.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:29:58.189+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.194137+0000) 2019-10-13T10:29:58.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:58.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.194761+0000) 2019-10-13T10:29:58.196 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:29:58.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.195088+0000) 2019-10-13T10:29:58.197 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:29:58.192+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.195880+0000) 2019-10-13T10:29:58.197 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:29:58.193+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.196469+0000) 2019-10-13T10:29:58.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:58.193+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.197464+0000) 2019-10-13T10:29:58.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:29:58.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:29:58.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:29:58.200+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.204764+0000) 2019-10-13T10:29:58.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:29:58.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.597686+0000) 2019-10-13T10:29:58.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:29:58.597+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.603259+0000) 2019-10-13T10:29:58.819 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:29:58.820 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:58.823 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:29:58.824 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:01.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:01.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:01.896850+0000) 2019-10-13T10:30:02.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:02.085+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:02.088038+0000) 2019-10-13T10:30:03.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:03.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.084601+0000) 2019-10-13T10:30:03.204 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:03.197+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.203043+0000) 2019-10-13T10:30:03.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:03.201+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.203712+0000) 2019-10-13T10:30:03.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:03.201+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.204124+0000) 2019-10-13T10:30:03.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:03.201+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.204484+0000) 2019-10-13T10:30:03.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:03.200+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.204937+0000) 2019-10-13T10:30:03.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:03.201+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.205194+0000) 2019-10-13T10:30:03.207 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:03.201+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.206449+0000) 2019-10-13T10:30:03.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:03.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:03.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:03.208+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.212149+0000) 2019-10-13T10:30:05.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:05.113+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:05.116682+0000) 2019-10-13T10:30:05.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:05.284+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:05.286432+0000) 2019-10-13T10:30:06.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:06.329+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:06.335063+0000) 2019-10-13T10:30:08.213 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:08.209+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.211669+0000) 2019-10-13T10:30:08.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:08.209+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.212212+0000) 2019-10-13T10:30:08.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:08.209+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.213059+0000) 2019-10-13T10:30:08.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:08.209+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.212773+0000) 2019-10-13T10:30:08.215 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:08.209+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.213705+0000) 2019-10-13T10:30:08.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:08.212+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.214416+0000) 2019-10-13T10:30:08.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:08.209+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.214682+0000) 2019-10-13T10:30:08.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:08.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:08.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:08.216+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.219987+0000) 2019-10-13T10:30:08.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:08.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.597888+0000) 2019-10-13T10:30:08.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:08.601+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.603472+0000) 2019-10-13T10:30:11.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:11.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:11.897085+0000) 2019-10-13T10:30:12.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:12.084+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:12.088262+0000) 2019-10-13T10:30:13.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:13.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.084795+0000) 2019-10-13T10:30:13.221 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:13.216+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.220048+0000) 2019-10-13T10:30:13.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:13.216+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.220494+0000) 2019-10-13T10:30:13.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:13.216+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.221763+0000) 2019-10-13T10:30:13.223 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:13.216+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.222008+0000) 2019-10-13T10:30:13.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:13.216+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.222652+0000) 2019-10-13T10:30:13.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:13.216+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.223134+0000) 2019-10-13T10:30:13.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:13.220+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.223807+0000) 2019-10-13T10:30:13.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:13.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101220 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:13.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:13.224+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.226309+0000) 2019-10-13T10:30:15.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:15.112+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:15.116927+0000) 2019-10-13T10:30:15.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:15.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:15.292798+0000) 2019-10-13T10:30:16.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:16.328+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:16.335262+0000) 2019-10-13T10:30:18.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:18.224+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.228057+0000) 2019-10-13T10:30:18.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:18.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.229407+0000) 2019-10-13T10:30:18.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:18.224+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.229948+0000) 2019-10-13T10:30:18.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:18.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.231004+0000) 2019-10-13T10:30:18.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:18.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.230175+0000) 2019-10-13T10:30:18.233 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:18.228+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.231876+0000) 2019-10-13T10:30:18.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:18.228+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.232206+0000) 2019-10-13T10:30:18.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:18.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:18.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:18.236+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.238977+0000) 2019-10-13T10:30:18.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:18.596+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.598065+0000) 2019-10-13T10:30:18.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:18.600+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.603683+0000) 2019-10-13T10:30:21.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:21.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:21.897301+0000) 2019-10-13T10:30:22.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:22.084+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:22.088526+0000) 2019-10-13T10:30:23.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:23.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.084996+0000) 2019-10-13T10:30:23.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:23.232+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.236521+0000) 2019-10-13T10:30:23.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:23.232+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.237346+0000) 2019-10-13T10:30:23.239 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:23.232+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.237785+0000) 2019-10-13T10:30:23.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:23.232+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.238546+0000) 2019-10-13T10:30:23.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:23.232+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.239253+0000) 2019-10-13T10:30:23.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:23.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.239502+0000) 2019-10-13T10:30:23.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:23.236+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.241018+0000) 2019-10-13T10:30:23.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:23.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:23.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:23.240+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.244626+0000) 2019-10-13T10:30:25.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:25.116+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:25.117152+0000) 2019-10-13T10:30:25.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:25.296+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:25.301600+0000) 2019-10-13T10:30:26.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:26.332+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:26.335556+0000) 2019-10-13T10:30:28.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:28.240+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.245360+0000) 2019-10-13T10:30:28.247 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:28.239+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.245679+0000) 2019-10-13T10:30:28.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:28.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.246096+0000) 2019-10-13T10:30:28.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:28.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.246552+0000) 2019-10-13T10:30:28.249 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:28.240+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.247045+0000) 2019-10-13T10:30:28.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:28.244+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.247293+0000) 2019-10-13T10:30:28.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:28.244+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.247737+0000) 2019-10-13T10:30:28.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:28.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:28.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:28.251+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.254477+0000) 2019-10-13T10:30:28.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:28.592+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.598247+0000) 2019-10-13T10:30:28.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:28.600+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.603870+0000) 2019-10-13T10:30:28.930 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:30:28.930 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:28.940 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:30:28.940 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:31.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:31.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:31.897521+0000) 2019-10-13T10:30:32.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:32.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:32.088732+0000) 2019-10-13T10:30:33.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:33.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.085219+0000) 2019-10-13T10:30:33.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:33.248+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.253419+0000) 2019-10-13T10:30:33.255 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:33.248+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.253815+0000) 2019-10-13T10:30:33.256 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:33.251+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.255178+0000) 2019-10-13T10:30:33.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:33.252+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.255482+0000) 2019-10-13T10:30:33.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:33.252+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.255876+0000) 2019-10-13T10:30:33.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:33.252+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.256312+0000) 2019-10-13T10:30:33.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:33.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.256357+0000) 2019-10-13T10:30:33.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:33.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:33.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:33.255+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.261197+0000) 2019-10-13T10:30:35.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:35.112+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:35.117348+0000) 2019-10-13T10:30:35.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:35.307+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:35.308014+0000) 2019-10-13T10:30:36.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:36.332+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:36.335754+0000) 2019-10-13T10:30:38.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:38.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.261436+0000) 2019-10-13T10:30:38.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:38.256+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.262127+0000) 2019-10-13T10:30:38.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:38.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.262515+0000) 2019-10-13T10:30:38.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:38.256+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.262886+0000) 2019-10-13T10:30:38.265 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:38.259+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.264024+0000) 2019-10-13T10:30:38.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:38.260+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.264202+0000) 2019-10-13T10:30:38.266 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:38.260+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.264796+0000) 2019-10-13T10:30:38.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:38.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:38.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:38.267+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.270827+0000) 2019-10-13T10:30:38.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:38.592+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.598424+0000) 2019-10-13T10:30:38.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:38.600+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.604067+0000) 2019-10-13T10:30:41.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:41.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:41.897730+0000) 2019-10-13T10:30:42.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:42.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:42.088969+0000) 2019-10-13T10:30:43.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:43.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.085413+0000) 2019-10-13T10:30:43.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:43.264+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.269142+0000) 2019-10-13T10:30:43.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:43.264+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.269843+0000) 2019-10-13T10:30:43.271 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:43.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.269770+0000) 2019-10-13T10:30:43.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:43.264+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.270422+0000) 2019-10-13T10:30:43.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:43.264+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.270688+0000) 2019-10-13T10:30:43.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:43.267+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.271640+0000) 2019-10-13T10:30:43.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:43.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.272090+0000) 2019-10-13T10:30:43.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:43.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:43.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:43.275+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.278679+0000) 2019-10-13T10:30:45.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:45.116+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:45.117582+0000) 2019-10-13T10:30:45.316 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:45.311+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:45.314552+0000) 2019-10-13T10:30:46.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:46.332+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:46.335984+0000) 2019-10-13T10:30:48.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:48.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.276682+0000) 2019-10-13T10:30:48.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:48.272+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.277172+0000) 2019-10-13T10:30:48.279 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:48.271+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.277833+0000) 2019-10-13T10:30:48.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:48.272+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.278276+0000) 2019-10-13T10:30:48.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:48.272+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.278602+0000) 2019-10-13T10:30:48.280 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:48.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.279410+0000) 2019-10-13T10:30:48.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:48.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.279193+0000) 2019-10-13T10:30:48.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:48.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:48.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:48.279+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.285575+0000) 2019-10-13T10:30:48.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:48.596+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.598601+0000) 2019-10-13T10:30:48.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:48.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.604241+0000) 2019-10-13T10:30:51.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:51.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:51.897922+0000) 2019-10-13T10:30:52.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:52.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:52.089173+0000) 2019-10-13T10:30:53.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:53.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.085618+0000) 2019-10-13T10:30:53.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:53.279+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.285058+0000) 2019-10-13T10:30:53.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:53.279+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.285320+0000) 2019-10-13T10:30:53.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:53.279+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.286099+0000) 2019-10-13T10:30:53.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:53.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.285962+0000) 2019-10-13T10:30:53.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:53.279+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.286541+0000) 2019-10-13T10:30:53.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:53.287+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.294121+0000) 2019-10-13T10:30:53.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:53.279+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.286970+0000) 2019-10-13T10:30:53.312 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:53.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.287264+0000) 2019-10-13T10:30:53.313 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:53.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:55.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:55.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:55.117788+0000) 2019-10-13T10:30:55.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:55.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:55.321211+0000) 2019-10-13T10:30:56.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:56.331+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:56.336183+0000) 2019-10-13T10:30:58.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:30:58.287+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.293309+0000) 2019-10-13T10:30:58.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:30:58.287+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.293610+0000) 2019-10-13T10:30:58.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:58.287+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.294215+0000) 2019-10-13T10:30:58.296 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:30:58.287+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.294597+0000) 2019-10-13T10:30:58.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:58.287+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.294486+0000) 2019-10-13T10:30:58.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:30:58.287+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.295108+0000) 2019-10-13T10:30:58.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:30:58.291+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.295460+0000) 2019-10-13T10:30:58.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:30:58.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:30:58.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:30:58.299+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.302807+0000) 2019-10-13T10:30:58.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:30:58.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.598779+0000) 2019-10-13T10:30:58.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:30:58.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.604426+0000) 2019-10-13T10:30:59.067 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:30:59.067 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:59.070 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:30:59.071 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:01.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:01.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:01.898130+0000) 2019-10-13T10:31:02.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:02.083+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:02.089334+0000) 2019-10-13T10:31:03.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:03.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.085851+0000) 2019-10-13T10:31:03.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:03.295+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.301326+0000) 2019-10-13T10:31:03.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:03.295+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.301868+0000) 2019-10-13T10:31:03.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:03.295+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.302240+0000) 2019-10-13T10:31:03.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:03.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.302930+0000) 2019-10-13T10:31:03.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:03.295+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.302944+0000) 2019-10-13T10:31:03.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:03.299+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.303883+0000) 2019-10-13T10:31:03.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:03.299+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.304286+0000) 2019-10-13T10:31:03.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:03.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101549 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:03.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:03.303+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.309495+0000) 2019-10-13T10:31:05.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:05.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:05.118008+0000) 2019-10-13T10:31:05.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:05.323+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:05.327592+0000) 2019-10-13T10:31:06.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:06.335+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:06.336398+0000) 2019-10-13T10:31:08.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:08.303+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.308618+0000) 2019-10-13T10:31:08.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:08.303+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.309024+0000) 2019-10-13T10:31:08.311 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:08.303+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.309751+0000) 2019-10-13T10:31:08.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:08.303+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.310710+0000) 2019-10-13T10:31:08.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:08.303+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.311019+0000) 2019-10-13T10:31:08.312 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:08.307+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.311487+0000) 2019-10-13T10:31:08.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:08.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.312228+0000) 2019-10-13T10:31:08.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:08.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:08.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:08.311+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.316966+0000) 2019-10-13T10:31:08.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:08.591+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.598956+0000) 2019-10-13T10:31:08.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:08.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.604628+0000) 2019-10-13T10:31:11.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:11.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:11.898362+0000) 2019-10-13T10:31:12.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:12.087+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:12.089534+0000) 2019-10-13T10:31:13.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:13.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.086045+0000) 2019-10-13T10:31:13.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:13.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.317371+0000) 2019-10-13T10:31:13.320 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:13.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.318508+0000) 2019-10-13T10:31:13.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:13.311+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.318852+0000) 2019-10-13T10:31:13.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:13.315+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.319298+0000) 2019-10-13T10:31:13.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:13.315+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.319524+0000) 2019-10-13T10:31:13.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:13.315+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.320061+0000) 2019-10-13T10:31:13.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:13.315+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.320354+0000) 2019-10-13T10:31:13.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:13.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:13.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:13.319+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.323421+0000) 2019-10-13T10:31:15.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:15.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:15.118229+0000) 2019-10-13T10:31:15.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:15.331+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:15.333942+0000) 2019-10-13T10:31:16.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:16.335+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:16.336593+0000) 2019-10-13T10:31:18.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:18.319+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.324600+0000) 2019-10-13T10:31:18.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:18.319+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.325047+0000) 2019-10-13T10:31:18.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:18.319+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.325526+0000) 2019-10-13T10:31:18.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:18.319+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.325892+0000) 2019-10-13T10:31:18.328 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:18.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.326993+0000) 2019-10-13T10:31:18.329 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:18.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.327928+0000) 2019-10-13T10:31:18.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:18.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.327625+0000) 2019-10-13T10:31:18.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:18.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:18.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:18.331+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.334549+0000) 2019-10-13T10:31:18.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:18.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.599136+0000) 2019-10-13T10:31:18.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:18.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.604828+0000) 2019-10-13T10:31:21.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:21.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:21.898562+0000) 2019-10-13T10:31:22.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:22.087+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:22.089735+0000) 2019-10-13T10:31:23.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:23.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.086272+0000) 2019-10-13T10:31:23.334 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:23.326+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.332710+0000) 2019-10-13T10:31:23.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:23.331+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.333270+0000) 2019-10-13T10:31:23.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:23.331+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.333544+0000) 2019-10-13T10:31:23.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:23.331+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.334042+0000) 2019-10-13T10:31:23.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:23.335+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.335273+0000) 2019-10-13T10:31:23.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:23.335+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.335905+0000) 2019-10-13T10:31:23.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:23.335+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.336441+0000) 2019-10-13T10:31:23.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:23.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:23.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:23.334+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.341727+0000) 2019-10-13T10:31:25.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:25.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:25.118432+0000) 2019-10-13T10:31:25.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:25.338+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:25.340457+0000) 2019-10-13T10:31:26.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:26.335+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:26.336809+0000) 2019-10-13T10:31:28.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:28.339+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.341141+0000) 2019-10-13T10:31:28.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:28.339+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.341681+0000) 2019-10-13T10:31:28.343 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:28.339+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.341977+0000) 2019-10-13T10:31:28.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:28.339+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.343170+0000) 2019-10-13T10:31:28.344 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:28.343+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.343438+0000) 2019-10-13T10:31:28.345 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:28.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.343825+0000) 2019-10-13T10:31:28.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:28.343+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.344303+0000) 2019-10-13T10:31:28.347 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:28.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:28.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:28.342+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.349401+0000) 2019-10-13T10:31:28.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:28.599+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.599340+0000) 2019-10-13T10:31:28.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:28.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.605003+0000) 2019-10-13T10:31:29.206 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:31:29.207 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:29.210 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:31:29.210 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:31.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:31.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:31.898762+0000) 2019-10-13T10:31:32.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:32.087+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:32.089944+0000) 2019-10-13T10:31:33.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:33.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.086471+0000) 2019-10-13T10:31:33.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:33.347+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.350259+0000) 2019-10-13T10:31:33.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:33.347+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.350495+0000) 2019-10-13T10:31:33.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:33.351+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.351480+0000) 2019-10-13T10:31:33.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:33.351+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.352481+0000) 2019-10-13T10:31:33.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:33.351+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.352677+0000) 2019-10-13T10:31:33.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:33.351+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.352971+0000) 2019-10-13T10:31:33.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:33.346+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.353519+0000) 2019-10-13T10:31:33.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:33.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:33.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:33.354+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.358984+0000) 2019-10-13T10:31:35.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:35.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:35.118624+0000) 2019-10-13T10:31:35.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:35.350+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:35.351988+0000) 2019-10-13T10:31:36.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:36.334+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:36.336960+0000) 2019-10-13T10:31:38.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:38.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.358577+0000) 2019-10-13T10:31:38.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:38.358+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.359595+0000) 2019-10-13T10:31:38.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:38.358+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.360171+0000) 2019-10-13T10:31:38.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:38.358+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.360320+0000) 2019-10-13T10:31:38.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:38.354+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.360879+0000) 2019-10-13T10:31:38.362 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:38.358+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.361697+0000) 2019-10-13T10:31:38.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:38.358+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.362195+0000) 2019-10-13T10:31:38.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:38.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:38.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:38.362+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.366457+0000) 2019-10-13T10:31:38.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:38.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.599525+0000) 2019-10-13T10:31:38.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:38.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.605199+0000) 2019-10-13T10:31:41.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:41.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:41.898981+0000) 2019-10-13T10:31:42.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:42.086+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:42.090147+0000) 2019-10-13T10:31:43.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:43.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.086678+0000) 2019-10-13T10:31:43.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:43.366+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.367911+0000) 2019-10-13T10:31:43.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:43.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.368447+0000) 2019-10-13T10:31:43.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:43.366+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.368702+0000) 2019-10-13T10:31:43.370 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:43.366+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.368456+0000) 2019-10-13T10:31:43.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:43.362+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.369241+0000) 2019-10-13T10:31:43.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:43.366+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.369745+0000) 2019-10-13T10:31:43.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:43.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.369371+0000) 2019-10-13T10:31:43.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:43.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:43.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:43.370+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.375061+0000) 2019-10-13T10:31:45.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:45.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:45.118825+0000) 2019-10-13T10:31:45.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:45.354+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:45.358412+0000) 2019-10-13T10:31:46.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:46.334+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:46.337158+0000) 2019-10-13T10:31:48.376 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:48.370+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.375062+0000) 2019-10-13T10:31:48.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:48.374+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.376317+0000) 2019-10-13T10:31:48.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:48.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.376841+0000) 2019-10-13T10:31:48.378 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:48.370+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.377140+0000) 2019-10-13T10:31:48.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:48.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.377470+0000) 2019-10-13T10:31:48.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:48.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.377974+0000) 2019-10-13T10:31:48.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:48.374+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.378481+0000) 2019-10-13T10:31:48.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:48.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:48.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:48.378+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.382326+0000) 2019-10-13T10:31:48.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:48.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.599720+0000) 2019-10-13T10:31:48.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:48.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.605371+0000) 2019-10-13T10:31:51.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:51.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:51.899187+0000) 2019-10-13T10:31:52.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:52.086+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:52.090349+0000) 2019-10-13T10:31:53.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:53.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.086883+0000) 2019-10-13T10:31:53.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:53.378+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.383153+0000) 2019-10-13T10:31:53.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:53.378+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.384157+0000) 2019-10-13T10:31:53.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:53.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.384261+0000) 2019-10-13T10:31:53.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:53.382+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.384705+0000) 2019-10-13T10:31:53.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:53.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.385656+0000) 2019-10-13T10:31:53.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:53.382+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.386073+0000) 2019-10-13T10:31:53.387 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:53.382+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.386382+0000) 2019-10-13T10:31:53.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:53.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:53.396 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:53.390+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.395437+0000) 2019-10-13T10:31:55.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:55.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:55.119022+0000) 2019-10-13T10:31:55.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:55.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:55.369189+0000) 2019-10-13T10:31:56.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:56.334+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:56.337380+0000) 2019-10-13T10:31:58.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:58.390+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.391839+0000) 2019-10-13T10:31:58.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:58.390+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.392679+0000) 2019-10-13T10:31:58.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:31:58.390+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.392524+0000) 2019-10-13T10:31:58.394 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:31:58.390+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.392972+0000) 2019-10-13T10:31:58.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:31:58.390+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.393430+0000) 2019-10-13T10:31:58.395 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:31:58.390+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.394450+0000) 2019-10-13T10:31:58.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:31:58.394+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.396344+0000) 2019-10-13T10:31:58.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:31:58.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:31:58.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:31:58.394+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.401679+0000) 2019-10-13T10:31:58.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:31:58.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.599914+0000) 2019-10-13T10:31:58.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:31:58.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.605573+0000) 2019-10-13T10:31:59.305 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:31:59.305 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:59.309 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:31:59.309 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:01.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:01.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:01.899390+0000) 2019-10-13T10:32:02.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:02.090+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:02.090547+0000) 2019-10-13T10:32:03.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:03.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.087099+0000) 2019-10-13T10:32:03.402 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:03.398+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.401251+0000) 2019-10-13T10:32:03.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:03.398+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.401628+0000) 2019-10-13T10:32:03.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:03.398+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.402038+0000) 2019-10-13T10:32:03.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:03.398+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.402449+0000) 2019-10-13T10:32:03.404 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:03.402+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.403286+0000) 2019-10-13T10:32:03.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:03.402+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.403560+0000) 2019-10-13T10:32:03.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:03.402+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.403998+0000) 2019-10-13T10:32:03.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:03.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:03.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:03.402+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.409249+0000) 2019-10-13T10:32:05.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:05.114+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:05.119240+0000) 2019-10-13T10:32:05.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:05.374+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:05.375727+0000) 2019-10-13T10:32:06.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:06.334+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:06.337602+0000) 2019-10-13T10:32:08.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:08.406+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.408466+0000) 2019-10-13T10:32:08.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:08.406+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.409018+0000) 2019-10-13T10:32:08.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:08.406+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.410467+0000) 2019-10-13T10:32:08.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:08.406+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.411056+0000) 2019-10-13T10:32:08.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:08.406+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.411253+0000) 2019-10-13T10:32:08.413 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:08.410+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.412385+0000) 2019-10-13T10:32:08.414 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:08.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.412716+0000) 2019-10-13T10:32:08.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:08.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 101986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:08.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:08.410+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.416308+0000) 2019-10-13T10:32:08.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:08.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.600113+0000) 2019-10-13T10:32:08.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:08.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.605794+0000) 2019-10-13T10:32:11.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:11.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:11.899592+0000) 2019-10-13T10:32:12.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:12.086+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:12.090752+0000) 2019-10-13T10:32:13.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:13.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.087294+0000) 2019-10-13T10:32:13.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:13.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.418524+0000) 2019-10-13T10:32:13.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:13.414+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.418733+0000) 2019-10-13T10:32:13.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:13.418+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.419896+0000) 2019-10-13T10:32:13.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:13.418+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.419663+0000) 2019-10-13T10:32:13.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:13.418+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.420869+0000) 2019-10-13T10:32:13.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:13.418+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.421205+0000) 2019-10-13T10:32:13.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:13.418+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.421666+0000) 2019-10-13T10:32:13.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:13.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102019 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:13.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:13.426+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.429147+0000) 2019-10-13T10:32:15.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:15.118+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:15.119472+0000) 2019-10-13T10:32:15.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:15.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:15.382680+0000) 2019-10-13T10:32:16.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:16.334+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:16.337803+0000) 2019-10-13T10:32:18.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:18.422+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.426952+0000) 2019-10-13T10:32:18.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:18.426+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.427289+0000) 2019-10-13T10:32:18.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:18.426+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.428056+0000) 2019-10-13T10:32:18.430 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:18.426+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.428877+0000) 2019-10-13T10:32:18.431 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:18.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.429791+0000) 2019-10-13T10:32:18.431 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:18.426+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.430479+0000) 2019-10-13T10:32:18.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:18.429+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.431687+0000) 2019-10-13T10:32:18.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:18.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102052 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:18.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:18.433+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.435806+0000) 2019-10-13T10:32:18.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:18.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.600333+0000) 2019-10-13T10:32:18.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:18.602+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.605974+0000) 2019-10-13T10:32:21.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:21.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:21.899786+0000) 2019-10-13T10:32:22.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:22.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:22.090957+0000) 2019-10-13T10:32:23.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:23.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.087487+0000) 2019-10-13T10:32:23.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:23.429+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.434741+0000) 2019-10-13T10:32:23.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:23.433+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.436444+0000) 2019-10-13T10:32:23.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:23.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.436943+0000) 2019-10-13T10:32:23.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:23.433+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.437399+0000) 2019-10-13T10:32:23.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:23.433+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.437572+0000) 2019-10-13T10:32:23.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:23.433+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.438774+0000) 2019-10-13T10:32:23.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:23.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102085 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:23.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:23.437+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.439925+0000) 2019-10-13T10:32:23.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:23.441+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.444736+0000) 2019-10-13T10:32:25.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:25.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:25.119686+0000) 2019-10-13T10:32:25.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:25.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:25.389302+0000) 2019-10-13T10:32:26.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:26.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:26.338023+0000) 2019-10-13T10:32:28.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:28.437+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.442620+0000) 2019-10-13T10:32:28.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:28.441+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.443487+0000) 2019-10-13T10:32:28.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:28.441+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.444401+0000) 2019-10-13T10:32:28.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:28.441+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.444867+0000) 2019-10-13T10:32:28.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:28.441+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.445229+0000) 2019-10-13T10:32:28.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:28.441+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.446265+0000) 2019-10-13T10:32:28.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:28.445+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.446494+0000) 2019-10-13T10:32:28.449 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:28.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:28.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:28.449+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.454020+0000) 2019-10-13T10:32:28.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:28.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.600480+0000) 2019-10-13T10:32:28.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:28.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.606179+0000) 2019-10-13T10:32:29.444 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:32:29.444 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:29.448 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:32:29.448 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:31.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:31.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:31.899988+0000) 2019-10-13T10:32:32.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:32.085+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:32.091118+0000) 2019-10-13T10:32:33.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:33.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.087683+0000) 2019-10-13T10:32:33.453 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:33.449+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.451999+0000) 2019-10-13T10:32:33.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:33.449+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.452382+0000) 2019-10-13T10:32:33.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:33.449+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.452767+0000) 2019-10-13T10:32:33.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:33.449+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.453754+0000) 2019-10-13T10:32:33.455 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:33.449+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.453924+0000) 2019-10-13T10:32:33.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:33.449+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.454208+0000) 2019-10-13T10:32:33.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:33.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.454788+0000) 2019-10-13T10:32:33.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:33.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:33.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:33.457+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.459486+0000) 2019-10-13T10:32:35.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:35.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:35.119888+0000) 2019-10-13T10:32:35.396 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:35.393+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:35.395215+0000) 2019-10-13T10:32:36.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:36.333+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:36.338253+0000) 2019-10-13T10:32:38.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:38.457+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.460925+0000) 2019-10-13T10:32:38.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:38.457+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.461328+0000) 2019-10-13T10:32:38.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:38.457+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.461973+0000) 2019-10-13T10:32:38.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:38.457+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.462298+0000) 2019-10-13T10:32:38.464 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:38.457+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.462569+0000) 2019-10-13T10:32:38.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:38.461+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.463550+0000) 2019-10-13T10:32:38.465 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:38.461+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.464014+0000) 2019-10-13T10:32:38.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:38.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:38.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:38.469+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.472386+0000) 2019-10-13T10:32:38.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:38.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.600681+0000) 2019-10-13T10:32:38.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:38.601+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.606369+0000) 2019-10-13T10:32:41.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:41.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:41.900196+0000) 2019-10-13T10:32:42.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:42.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:42.091378+0000) 2019-10-13T10:32:43.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:43.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.087882+0000) 2019-10-13T10:32:43.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:43.465+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.468522+0000) 2019-10-13T10:32:43.470 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:43.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.469352+0000) 2019-10-13T10:32:43.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:43.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.469941+0000) 2019-10-13T10:32:43.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:43.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.470950+0000) 2019-10-13T10:32:43.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:43.465+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.470972+0000) 2019-10-13T10:32:43.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:43.469+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.471747+0000) 2019-10-13T10:32:43.473 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:43.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.471226+0000) 2019-10-13T10:32:43.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:43.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:43.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:43.473+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.476033+0000) 2019-10-13T10:32:45.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:45.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:45.120091+0000) 2019-10-13T10:32:45.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:45.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:45.401553+0000) 2019-10-13T10:32:46.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:46.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:46.338444+0000) 2019-10-13T10:32:48.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:48.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.477094+0000) 2019-10-13T10:32:48.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:48.473+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.477526+0000) 2019-10-13T10:32:48.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:48.473+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.477673+0000) 2019-10-13T10:32:48.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:48.473+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.478047+0000) 2019-10-13T10:32:48.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:48.477+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.478741+0000) 2019-10-13T10:32:48.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:48.477+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.479828+0000) 2019-10-13T10:32:48.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:48.477+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.480373+0000) 2019-10-13T10:32:48.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:48.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:48.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:48.485+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.486258+0000) 2019-10-13T10:32:48.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:48.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.600903+0000) 2019-10-13T10:32:48.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:48.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.606572+0000) 2019-10-13T10:32:51.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:51.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:51.900408+0000) 2019-10-13T10:32:52.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:52.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:52.091582+0000) 2019-10-13T10:32:53.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:53.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.088081+0000) 2019-10-13T10:32:53.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:53.481+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.485452+0000) 2019-10-13T10:32:53.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:53.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.486215+0000) 2019-10-13T10:32:53.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:53.481+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.486481+0000) 2019-10-13T10:32:53.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:53.481+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.486894+0000) 2019-10-13T10:32:53.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:53.481+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.487198+0000) 2019-10-13T10:32:53.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:53.485+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.487637+0000) 2019-10-13T10:32:53.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:53.485+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.488098+0000) 2019-10-13T10:32:53.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:53.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:53.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:53.493+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.494202+0000) 2019-10-13T10:32:55.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:55.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:55.120293+0000) 2019-10-13T10:32:55.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:55.405+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:55.407920+0000) 2019-10-13T10:32:56.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:56.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:56.338636+0000) 2019-10-13T10:32:58.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:32:58.489+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.494826+0000) 2019-10-13T10:32:58.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:32:58.493+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.495162+0000) 2019-10-13T10:32:58.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:58.493+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.495700+0000) 2019-10-13T10:32:58.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:32:58.493+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.495914+0000) 2019-10-13T10:32:58.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:32:58.493+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.496271+0000) 2019-10-13T10:32:58.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:58.493+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.496659+0000) 2019-10-13T10:32:58.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:32:58.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.497409+0000) 2019-10-13T10:32:58.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:32:58.493+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:32:58.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:32:58.497+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.501690+0000) 2019-10-13T10:32:58.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:32:58.597+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.601078+0000) 2019-10-13T10:32:58.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:32:58.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.606806+0000) 2019-10-13T10:32:59.579 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:32:59.580 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:59.583 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:32:59.584 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:01.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:01.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:01.900615+0000) 2019-10-13T10:33:02.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:02.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:02.091790+0000) 2019-10-13T10:33:03.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:03.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.088307+0000) 2019-10-13T10:33:03.503 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:03.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.502410+0000) 2019-10-13T10:33:03.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:03.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.503356+0000) 2019-10-13T10:33:03.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:03.500+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.503461+0000) 2019-10-13T10:33:03.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:03.501+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.504054+0000) 2019-10-13T10:33:03.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:03.500+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.504513+0000) 2019-10-13T10:33:03.506 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:03.500+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.504656+0000) 2019-10-13T10:33:03.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:03.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.505048+0000) 2019-10-13T10:33:03.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:03.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:03.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:03.509+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.511157+0000) 2019-10-13T10:33:05.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:05.116+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:05.120524+0000) 2019-10-13T10:33:05.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:05.413+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:05.414400+0000) 2019-10-13T10:33:06.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:06.332+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:06.338862+0000) 2019-10-13T10:33:08.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:08.504+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.510075+0000) 2019-10-13T10:33:08.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:08.509+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.510872+0000) 2019-10-13T10:33:08.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:08.508+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.512229+0000) 2019-10-13T10:33:08.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:08.508+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.512464+0000) 2019-10-13T10:33:08.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:08.508+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.513100+0000) 2019-10-13T10:33:08.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:08.508+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.513474+0000) 2019-10-13T10:33:08.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:08.508+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.513735+0000) 2019-10-13T10:33:08.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:08.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:08.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:08.517+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.518676+0000) 2019-10-13T10:33:08.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:08.600+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.601296+0000) 2019-10-13T10:33:08.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:08.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.607006+0000) 2019-10-13T10:33:11.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:11.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:11.900817+0000) 2019-10-13T10:33:12.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:12.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:12.092033+0000) 2019-10-13T10:33:13.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:13.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.088526+0000) 2019-10-13T10:33:13.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:13.512+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.518406+0000) 2019-10-13T10:33:13.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:13.512+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.518957+0000) 2019-10-13T10:33:13.521 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:13.516+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.519568+0000) 2019-10-13T10:33:13.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:13.516+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.520322+0000) 2019-10-13T10:33:13.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:13.516+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.520355+0000) 2019-10-13T10:33:13.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:13.516+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.520947+0000) 2019-10-13T10:33:13.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:13.516+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.521927+0000) 2019-10-13T10:33:13.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:13.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:13.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:13.524+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.527302+0000) 2019-10-13T10:33:15.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:15.116+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:15.120735+0000) 2019-10-13T10:33:15.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:15.416+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:15.420846+0000) 2019-10-13T10:33:16.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:16.336+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:16.339060+0000) 2019-10-13T10:33:18.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:18.520+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.527105+0000) 2019-10-13T10:33:18.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:18.524+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.527614+0000) 2019-10-13T10:33:18.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:18.524+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.527729+0000) 2019-10-13T10:33:18.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:18.524+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.528331+0000) 2019-10-13T10:33:18.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:18.524+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.528676+0000) 2019-10-13T10:33:18.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:18.524+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.529753+0000) 2019-10-13T10:33:18.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:18.524+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.530200+0000) 2019-10-13T10:33:18.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:18.528+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:18.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:18.528+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.533010+0000) 2019-10-13T10:33:18.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:18.596+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.601495+0000) 2019-10-13T10:33:18.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:18.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.607186+0000) 2019-10-13T10:33:21.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:21.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:21.901016+0000) 2019-10-13T10:33:22.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:22.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:22.092253+0000) 2019-10-13T10:33:23.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:23.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.088753+0000) 2019-10-13T10:33:23.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:23.528+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.534887+0000) 2019-10-13T10:33:23.537 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:23.532+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.535583+0000) 2019-10-13T10:33:23.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:23.532+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.535998+0000) 2019-10-13T10:33:23.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:23.532+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.536396+0000) 2019-10-13T10:33:23.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:23.532+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.537028+0000) 2019-10-13T10:33:23.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:23.532+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.537516+0000) 2019-10-13T10:33:23.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:23.532+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.537970+0000) 2019-10-13T10:33:23.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:23.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:23.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:23.540+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.545538+0000) 2019-10-13T10:33:25.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:25.116+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:25.120960+0000) 2019-10-13T10:33:25.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:25.424+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:25.427307+0000) 2019-10-13T10:33:26.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:26.336+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:26.339257+0000) 2019-10-13T10:33:28.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:28.540+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.543133+0000) 2019-10-13T10:33:28.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:28.540+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.543609+0000) 2019-10-13T10:33:28.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:28.540+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.544429+0000) 2019-10-13T10:33:28.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:28.540+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.544734+0000) 2019-10-13T10:33:28.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:28.540+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.545921+0000) 2019-10-13T10:33:28.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:28.540+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.546175+0000) 2019-10-13T10:33:28.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:28.544+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.546899+0000) 2019-10-13T10:33:28.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:28.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:28.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:28.548+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.552658+0000) 2019-10-13T10:33:28.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:28.596+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.601695+0000) 2019-10-13T10:33:28.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:28.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.607387+0000) 2019-10-13T10:33:29.691 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:33:29.691 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:29.695 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:33:29.695 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:31.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:31.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:31.901237+0000) 2019-10-13T10:33:32.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:32.092+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:32.092451+0000) 2019-10-13T10:33:33.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:33.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.089004+0000) 2019-10-13T10:33:33.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:33.548+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.552518+0000) 2019-10-13T10:33:33.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:33.548+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.553543+0000) 2019-10-13T10:33:33.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:33.548+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.554386+0000) 2019-10-13T10:33:33.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:33.548+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.554845+0000) 2019-10-13T10:33:33.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:33.552+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.556180+0000) 2019-10-13T10:33:33.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:33.552+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.556101+0000) 2019-10-13T10:33:33.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:33.552+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.556546+0000) 2019-10-13T10:33:33.559 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:33.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:33.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:33.556+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.561442+0000) 2019-10-13T10:33:35.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:35.120+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:35.121160+0000) 2019-10-13T10:33:35.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:35.432+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:35.437045+0000) 2019-10-13T10:33:36.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:36.336+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:36.339503+0000) 2019-10-13T10:33:38.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:38.556+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.562052+0000) 2019-10-13T10:33:38.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:38.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.562445+0000) 2019-10-13T10:33:38.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:38.556+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.562766+0000) 2019-10-13T10:33:38.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:38.556+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.563187+0000) 2019-10-13T10:33:38.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:38.560+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.563553+0000) 2019-10-13T10:33:38.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:38.560+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.564489+0000) 2019-10-13T10:33:38.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:38.560+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.564857+0000) 2019-10-13T10:33:38.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:38.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:38.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:38.568+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.571388+0000) 2019-10-13T10:33:38.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:38.596+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.601869+0000) 2019-10-13T10:33:38.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:38.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.607511+0000) 2019-10-13T10:33:41.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:41.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:41.901490+0000) 2019-10-13T10:33:42.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:42.088+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:42.092652+0000) 2019-10-13T10:33:43.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:43.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.089230+0000) 2019-10-13T10:33:43.571 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:43.564+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.569529+0000) 2019-10-13T10:33:43.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:43.564+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.570086+0000) 2019-10-13T10:33:43.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:43.564+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.570495+0000) 2019-10-13T10:33:43.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:43.564+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.571028+0000) 2019-10-13T10:33:43.573 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:43.568+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.571728+0000) 2019-10-13T10:33:43.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:43.568+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.571975+0000) 2019-10-13T10:33:43.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:43.568+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.572743+0000) 2019-10-13T10:33:43.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:43.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:43.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:43.572+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.577628+0000) 2019-10-13T10:33:45.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:45.120+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:45.121361+0000) 2019-10-13T10:33:45.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:45.440+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:45.443679+0000) 2019-10-13T10:33:46.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:46.336+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:46.339702+0000) 2019-10-13T10:33:48.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:48.571+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.578111+0000) 2019-10-13T10:33:48.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:48.571+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.578571+0000) 2019-10-13T10:33:48.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:48.571+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.579112+0000) 2019-10-13T10:33:48.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:48.575+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.579533+0000) 2019-10-13T10:33:48.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:48.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.579964+0000) 2019-10-13T10:33:48.582 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:48.576+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.580869+0000) 2019-10-13T10:33:48.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:48.575+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.581650+0000) 2019-10-13T10:33:48.584 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:48.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:48.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:48.584+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.586611+0000) 2019-10-13T10:33:48.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:48.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.602076+0000) 2019-10-13T10:33:48.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:48.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.607670+0000) 2019-10-13T10:33:51.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:51.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:51.901691+0000) 2019-10-13T10:33:52.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:52.091+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:52.092869+0000) 2019-10-13T10:33:53.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:53.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.089437+0000) 2019-10-13T10:33:53.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:53.579+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.585833+0000) 2019-10-13T10:33:53.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:53.579+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.586307+0000) 2019-10-13T10:33:53.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:53.584+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.586997+0000) 2019-10-13T10:33:53.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:53.583+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.587456+0000) 2019-10-13T10:33:53.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:53.583+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.588413+0000) 2019-10-13T10:33:53.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:53.583+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.588622+0000) 2019-10-13T10:33:53.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:53.583+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.588945+0000) 2019-10-13T10:33:53.592 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:53.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:53.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:53.592+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.594632+0000) 2019-10-13T10:33:55.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:55.115+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:55.121509+0000) 2019-10-13T10:33:55.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:55.452+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:55.454601+0000) 2019-10-13T10:33:56.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:56.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:56.339907+0000) 2019-10-13T10:33:58.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:33:58.587+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.594931+0000) 2019-10-13T10:33:58.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:33:58.591+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.595829+0000) 2019-10-13T10:33:58.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:58.591+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.596760+0000) 2019-10-13T10:33:58.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:33:58.591+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.597271+0000) 2019-10-13T10:33:58.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:33:58.592+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.597663+0000) 2019-10-13T10:33:58.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:33:58.591+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.598041+0000) 2019-10-13T10:33:58.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:58.591+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.598412+0000) 2019-10-13T10:33:58.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:33:58.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:33:58.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:33:58.600+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.603003+0000) 2019-10-13T10:33:58.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:33:58.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.602180+0000) 2019-10-13T10:33:58.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:33:58.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.607790+0000) 2019-10-13T10:33:59.827 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:33:59.827 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:59.831 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:33:59.831 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:01.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:01.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:01.901891+0000) 2019-10-13T10:34:02.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:02.091+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:02.093022+0000) 2019-10-13T10:34:03.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:03.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.089670+0000) 2019-10-13T10:34:03.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:03.595+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.603126+0000) 2019-10-13T10:34:03.605 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:03.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.603976+0000) 2019-10-13T10:34:03.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:03.599+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.604331+0000) 2019-10-13T10:34:03.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:03.599+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.604977+0000) 2019-10-13T10:34:03.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:03.599+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.605546+0000) 2019-10-13T10:34:03.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:03.599+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.605328+0000) 2019-10-13T10:34:03.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:03.599+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.605996+0000) 2019-10-13T10:34:03.609 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:03.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:03.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:03.608+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.612313+0000) 2019-10-13T10:34:05.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:05.119+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:05.121672+0000) 2019-10-13T10:34:05.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:05.456+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:05.460940+0000) 2019-10-13T10:34:06.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:06.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:06.340089+0000) 2019-10-13T10:34:08.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:08.595+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.602365+0000) 2019-10-13T10:34:08.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:08.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.607960+0000) 2019-10-13T10:34:08.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:08.607+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.611504+0000) 2019-10-13T10:34:08.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:08.607+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.612961+0000) 2019-10-13T10:34:08.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:08.607+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.613279+0000) 2019-10-13T10:34:08.615 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:08.608+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.613877+0000) 2019-10-13T10:34:08.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:08.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.614253+0000) 2019-10-13T10:34:08.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:08.607+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.614628+0000) 2019-10-13T10:34:08.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:08.611+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.615286+0000) 2019-10-13T10:34:08.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:08.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:08.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:08.616+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.621515+0000) 2019-10-13T10:34:11.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:11.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:11.902084+0000) 2019-10-13T10:34:12.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:12.087+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:12.093247+0000) 2019-10-13T10:34:13.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:13.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.089864+0000) 2019-10-13T10:34:13.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:13.615+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.620654+0000) 2019-10-13T10:34:13.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:13.615+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.621344+0000) 2019-10-13T10:34:13.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:13.615+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.621785+0000) 2019-10-13T10:34:13.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:13.615+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.622195+0000) 2019-10-13T10:34:13.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:13.615+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.622663+0000) 2019-10-13T10:34:13.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:13.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.623533+0000) 2019-10-13T10:34:13.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:13.615+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.623079+0000) 2019-10-13T10:34:13.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:13.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:13.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:13.627+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.630489+0000) 2019-10-13T10:34:15.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:15.119+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:15.121907+0000) 2019-10-13T10:34:15.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:15.467+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:15.469720+0000) 2019-10-13T10:34:16.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:16.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:16.340301+0000) 2019-10-13T10:34:18.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:18.599+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.602591+0000) 2019-10-13T10:34:18.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:18.607+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.608172+0000) 2019-10-13T10:34:18.630 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:18.627+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.628693+0000) 2019-10-13T10:34:18.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:18.627+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.629216+0000) 2019-10-13T10:34:18.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:18.627+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.629692+0000) 2019-10-13T10:34:18.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:18.627+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.630099+0000) 2019-10-13T10:34:18.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:18.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.631450+0000) 2019-10-13T10:34:18.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:18.627+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.632420+0000) 2019-10-13T10:34:18.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:18.631+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.632469+0000) 2019-10-13T10:34:18.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:18.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:18.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:18.631+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.636919+0000) 2019-10-13T10:34:21.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:21.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:21.902324+0000) 2019-10-13T10:34:22.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:22.091+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:22.093473+0000) 2019-10-13T10:34:23.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:23.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.090107+0000) 2019-10-13T10:34:23.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:23.635+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.637339+0000) 2019-10-13T10:34:23.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:23.635+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.637739+0000) 2019-10-13T10:34:23.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:23.635+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.638080+0000) 2019-10-13T10:34:23.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:23.635+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.638775+0000) 2019-10-13T10:34:23.640 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:23.639+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.639746+0000) 2019-10-13T10:34:23.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:23.639+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.640187+0000) 2019-10-13T10:34:23.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:23.635+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.640912+0000) 2019-10-13T10:34:23.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:23.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:23.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:23.639+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.645894+0000) 2019-10-13T10:34:25.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:25.119+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.122128+0000) 2019-10-13T10:34:25.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:25.475+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.476128+0000) 2019-10-13T10:34:26.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:26.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:26.340495+0000) 2019-10-13T10:34:28.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:28.599+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.602745+0000) 2019-10-13T10:34:28.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:28.607+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.608371+0000) 2019-10-13T10:34:28.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:28.643+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.645850+0000) 2019-10-13T10:34:28.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:28.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.645402+0000) 2019-10-13T10:34:28.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:28.647+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.647551+0000) 2019-10-13T10:34:28.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:28.643+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.647967+0000) 2019-10-13T10:34:28.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:28.647+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.648134+0000) 2019-10-13T10:34:28.650 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:28.647+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.648531+0000) 2019-10-13T10:34:28.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:28.647+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.648950+0000) 2019-10-13T10:34:28.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:28.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:28.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:28.647+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.653222+0000) 2019-10-13T10:34:29.946 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:34:29.947 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:29.950 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:34:29.950 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:31.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:31.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:31.902540+0000) 2019-10-13T10:34:32.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:32.090+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:32.093693+0000) 2019-10-13T10:34:33.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:33.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.090389+0000) 2019-10-13T10:34:33.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:33.647+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.653854+0000) 2019-10-13T10:34:33.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:33.650+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.654578+0000) 2019-10-13T10:34:33.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:33.654+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.655547+0000) 2019-10-13T10:34:33.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:33.654+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.655817+0000) 2019-10-13T10:34:33.658 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:33.654+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.656693+0000) 2019-10-13T10:34:33.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:33.654+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.656153+0000) 2019-10-13T10:34:33.659 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:33.654+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.657667+0000) 2019-10-13T10:34:33.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:33.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:33.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:33.655+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.661179+0000) 2019-10-13T10:34:35.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:35.118+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.122333+0000) 2019-10-13T10:34:35.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:35.479+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.482550+0000) 2019-10-13T10:34:36.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:36.338+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:36.340695+0000) 2019-10-13T10:34:38.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:38.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.602942+0000) 2019-10-13T10:34:38.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:38.606+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.608599+0000) 2019-10-13T10:34:38.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:38.658+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.662774+0000) 2019-10-13T10:34:38.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:38.658+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.663263+0000) 2019-10-13T10:34:38.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:38.662+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.663862+0000) 2019-10-13T10:34:38.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:38.662+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.664326+0000) 2019-10-13T10:34:38.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:38.659+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.664841+0000) 2019-10-13T10:34:38.666 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:38.662+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.665272+0000) 2019-10-13T10:34:38.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:38.662+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.666510+0000) 2019-10-13T10:34:38.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:38.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 102989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:38.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:38.671+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.674487+0000) 2019-10-13T10:34:41.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:41.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:41.902757+0000) 2019-10-13T10:34:42.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:42.090+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:42.093919+0000) 2019-10-13T10:34:43.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:43.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.090620+0000) 2019-10-13T10:34:43.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:43.667+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.671093+0000) 2019-10-13T10:34:43.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:43.666+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.670288+0000) 2019-10-13T10:34:43.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:43.670+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.671731+0000) 2019-10-13T10:34:43.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:43.670+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.672859+0000) 2019-10-13T10:34:43.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:43.670+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.672971+0000) 2019-10-13T10:34:43.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:43.670+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.673632+0000) 2019-10-13T10:34:43.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:43.670+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.674013+0000) 2019-10-13T10:34:43.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:43.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:43.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:43.679+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:43.683436+0000) 2019-10-13T10:34:45.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:45.118+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.122532+0000) 2019-10-13T10:34:45.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:45.487+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.489014+0000) 2019-10-13T10:34:46.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:46.338+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:46.340888+0000) 2019-10-13T10:34:48.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:48.598+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.603134+0000) 2019-10-13T10:34:48.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:48.606+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.608810+0000) 2019-10-13T10:34:48.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:48.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.679783+0000) 2019-10-13T10:34:48.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:48.678+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.680363+0000) 2019-10-13T10:34:48.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:48.678+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.681009+0000) 2019-10-13T10:34:48.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:48.675+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.681380+0000) 2019-10-13T10:34:48.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:48.678+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.681602+0000) 2019-10-13T10:34:48.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:48.678+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.682191+0000) 2019-10-13T10:34:48.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:48.678+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.682524+0000) 2019-10-13T10:34:48.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:48.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:48.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:48.679+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.685634+0000) 2019-10-13T10:34:51.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:51.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:51.902958+0000) 2019-10-13T10:34:52.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:52.090+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:52.094124+0000) 2019-10-13T10:34:53.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:53.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.090823+0000) 2019-10-13T10:34:53.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:53.686+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.688416+0000) 2019-10-13T10:34:53.690 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:53.686+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.688626+0000) 2019-10-13T10:34:53.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:53.686+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.688993+0000) 2019-10-13T10:34:53.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:53.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.689528+0000) 2019-10-13T10:34:53.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:53.686+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.689762+0000) 2019-10-13T10:34:53.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:53.686+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.690250+0000) 2019-10-13T10:34:53.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:53.686+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.690751+0000) 2019-10-13T10:34:53.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:53.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103090 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:53.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:53.691+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:53.694450+0000) 2019-10-13T10:34:55.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:55.118+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.122730+0000) 2019-10-13T10:34:55.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:55.495+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.495443+0000) 2019-10-13T10:34:56.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:56.338+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:56.341095+0000) 2019-10-13T10:34:58.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:58.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.603349+0000) 2019-10-13T10:34:58.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:58.606+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.609039+0000) 2019-10-13T10:34:58.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:34:58.694+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.696154+0000) 2019-10-13T10:34:58.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:34:58.691+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.697151+0000) 2019-10-13T10:34:58.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:34:58.694+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.697465+0000) 2019-10-13T10:34:58.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:34:58.694+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.698637+0000) 2019-10-13T10:34:58.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:34:58.694+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.698650+0000) 2019-10-13T10:34:58.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:34:58.698+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.699544+0000) 2019-10-13T10:34:58.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:34:58.698+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.700051+0000) 2019-10-13T10:34:58.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:34:58.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:34:58.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:34:58.699+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.704554+0000) 2019-10-13T10:35:00.061 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:35:00.062 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:00.066 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:35:00.066 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:01.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:01.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:01.903214+0000) 2019-10-13T10:35:02.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:02.090+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:02.094341+0000) 2019-10-13T10:35:03.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:03.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.091039+0000) 2019-10-13T10:35:03.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:03.699+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.704816+0000) 2019-10-13T10:35:03.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:03.702+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.705143+0000) 2019-10-13T10:35:03.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:03.702+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.705497+0000) 2019-10-13T10:35:03.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:03.702+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.706110+0000) 2019-10-13T10:35:03.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:03.702+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.706177+0000) 2019-10-13T10:35:03.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:03.706+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.707426+0000) 2019-10-13T10:35:03.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:03.706+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.707732+0000) 2019-10-13T10:35:03.710 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:03.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:03.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:03.707+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:03.713844+0000) 2019-10-13T10:35:05.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:05.122+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.122931+0000) 2019-10-13T10:35:05.503 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:05.498+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.501958+0000) 2019-10-13T10:35:06.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:06.338+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:06.341286+0000) 2019-10-13T10:35:08.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:08.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.603550+0000) 2019-10-13T10:35:08.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:08.606+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.609232+0000) 2019-10-13T10:35:08.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:08.710+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.712655+0000) 2019-10-13T10:35:08.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:08.710+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.713015+0000) 2019-10-13T10:35:08.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:08.710+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.713329+0000) 2019-10-13T10:35:08.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:08.710+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.714206+0000) 2019-10-13T10:35:08.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:08.710+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.714535+0000) 2019-10-13T10:35:08.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:08.710+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.715334+0000) 2019-10-13T10:35:08.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:08.710+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.715039+0000) 2019-10-13T10:35:08.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:08.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:08.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:08.718+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.722215+0000) 2019-10-13T10:35:11.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:11.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:11.903417+0000) 2019-10-13T10:35:12.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:12.094+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:12.094545+0000) 2019-10-13T10:35:13.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:13.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.091276+0000) 2019-10-13T10:35:13.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:13.718+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.720667+0000) 2019-10-13T10:35:13.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:13.718+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.721336+0000) 2019-10-13T10:35:13.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:13.714+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.721718+0000) 2019-10-13T10:35:13.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:13.718+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.721938+0000) 2019-10-13T10:35:13.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:13.722+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.723377+0000) 2019-10-13T10:35:13.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:13.722+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.723548+0000) 2019-10-13T10:35:13.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:13.722+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.723930+0000) 2019-10-13T10:35:13.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:13.722+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:13.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:13.722+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:13.729177+0000) 2019-10-13T10:35:15.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:15.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.123083+0000) 2019-10-13T10:35:15.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:15.506+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.508421+0000) 2019-10-13T10:35:16.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:16.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:16.341478+0000) 2019-10-13T10:35:18.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:18.601+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.603789+0000) 2019-10-13T10:35:18.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:18.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.609422+0000) 2019-10-13T10:35:18.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:18.725+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.728507+0000) 2019-10-13T10:35:18.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:18.725+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.729160+0000) 2019-10-13T10:35:18.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:18.725+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.729603+0000) 2019-10-13T10:35:18.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:18.725+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.730206+0000) 2019-10-13T10:35:18.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:18.726+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.730564+0000) 2019-10-13T10:35:18.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:18.729+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.731288+0000) 2019-10-13T10:35:18.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:18.729+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.731793+0000) 2019-10-13T10:35:18.734 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:18.729+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:18.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:18.730+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.737987+0000) 2019-10-13T10:35:21.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:21.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:21.903608+0000) 2019-10-13T10:35:22.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:22.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:22.094689+0000) 2019-10-13T10:35:23.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:23.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.091496+0000) 2019-10-13T10:35:23.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:23.733+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.736256+0000) 2019-10-13T10:35:23.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:23.733+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.736818+0000) 2019-10-13T10:35:23.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:23.730+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.737735+0000) 2019-10-13T10:35:23.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:23.733+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.737903+0000) 2019-10-13T10:35:23.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:23.733+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.738502+0000) 2019-10-13T10:35:23.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:23.733+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.739048+0000) 2019-10-13T10:35:23.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:23.733+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.739198+0000) 2019-10-13T10:35:23.742 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:23.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:23.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:23.738+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:23.745961+0000) 2019-10-13T10:35:25.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:25.117+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.123252+0000) 2019-10-13T10:35:25.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:25.514+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.514742+0000) 2019-10-13T10:35:26.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:26.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:26.341669+0000) 2019-10-13T10:35:28.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:28.601+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.603993+0000) 2019-10-13T10:35:28.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:28.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.609618+0000) 2019-10-13T10:35:28.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:28.741+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.745040+0000) 2019-10-13T10:35:28.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:28.741+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.745767+0000) 2019-10-13T10:35:28.747 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:28.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.746231+0000) 2019-10-13T10:35:28.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:28.741+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.746534+0000) 2019-10-13T10:35:28.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:28.741+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.747034+0000) 2019-10-13T10:35:28.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:28.742+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.747561+0000) 2019-10-13T10:35:28.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:28.745+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.747890+0000) 2019-10-13T10:35:28.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:28.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:28.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:28.750+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.756091+0000) 2019-10-13T10:35:30.187 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:35:30.187 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:30.191 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:35:30.191 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:31.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:31.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:31.903812+0000) 2019-10-13T10:35:32.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:32.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:32.094893+0000) 2019-10-13T10:35:33.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:33.090+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.091718+0000) 2019-10-13T10:35:33.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:33.749+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.753492+0000) 2019-10-13T10:35:33.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:33.749+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.754153+0000) 2019-10-13T10:35:33.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:33.749+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.754566+0000) 2019-10-13T10:35:33.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:33.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.754837+0000) 2019-10-13T10:35:33.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:33.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.755371+0000) 2019-10-13T10:35:33.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:33.750+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.756078+0000) 2019-10-13T10:35:33.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:33.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.756381+0000) 2019-10-13T10:35:33.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:33.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:33.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:33.758+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:33.763097+0000) 2019-10-13T10:35:35.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:35.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.123510+0000) 2019-10-13T10:35:35.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:35.514+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.521142+0000) 2019-10-13T10:35:36.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:36.341+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:36.341870+0000) 2019-10-13T10:35:38.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:38.601+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.604223+0000) 2019-10-13T10:35:38.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:38.605+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.609815+0000) 2019-10-13T10:35:38.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:38.757+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.762173+0000) 2019-10-13T10:35:38.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:38.757+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.762586+0000) 2019-10-13T10:35:38.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:38.757+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.763171+0000) 2019-10-13T10:35:38.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:38.761+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.764152+0000) 2019-10-13T10:35:38.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:38.761+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.764930+0000) 2019-10-13T10:35:38.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:38.761+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.765390+0000) 2019-10-13T10:35:38.767 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:38.758+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.766050+0000) 2019-10-13T10:35:38.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:38.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:38.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:38.766+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.771503+0000) 2019-10-13T10:35:41.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:41.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:41.904008+0000) 2019-10-13T10:35:42.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:42.089+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:42.095094+0000) 2019-10-13T10:35:43.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:43.090+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.091959+0000) 2019-10-13T10:35:43.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:43.765+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.770543+0000) 2019-10-13T10:35:43.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:43.765+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.771153+0000) 2019-10-13T10:35:43.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:43.769+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.771703+0000) 2019-10-13T10:35:43.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:43.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.772104+0000) 2019-10-13T10:35:43.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:43.769+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.772540+0000) 2019-10-13T10:35:43.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:43.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.773603+0000) 2019-10-13T10:35:43.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:43.769+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.774072+0000) 2019-10-13T10:35:43.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:43.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:43.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:43.774+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:43.779364+0000) 2019-10-13T10:35:45.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:45.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.123676+0000) 2019-10-13T10:35:45.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:45.526+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.527792+0000) 2019-10-13T10:35:46.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:46.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:46.342072+0000) 2019-10-13T10:35:48.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:48.601+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.604409+0000) 2019-10-13T10:35:48.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:48.609+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.610007+0000) 2019-10-13T10:35:48.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:48.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.779901+0000) 2019-10-13T10:35:48.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:48.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.780208+0000) 2019-10-13T10:35:48.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:48.774+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.780730+0000) 2019-10-13T10:35:48.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:48.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.780977+0000) 2019-10-13T10:35:48.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:48.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.781889+0000) 2019-10-13T10:35:48.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:48.777+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.782806+0000) 2019-10-13T10:35:48.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:48.777+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.783253+0000) 2019-10-13T10:35:48.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:48.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:48.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:48.782+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.789031+0000) 2019-10-13T10:35:51.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:51.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:51.904206+0000) 2019-10-13T10:35:52.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:52.093+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:52.095295+0000) 2019-10-13T10:35:53.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:53.090+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.092151+0000) 2019-10-13T10:35:53.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:53.785+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.787351+0000) 2019-10-13T10:35:53.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:53.786+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.788125+0000) 2019-10-13T10:35:53.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:53.785+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.789163+0000) 2019-10-13T10:35:53.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:53.785+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.789644+0000) 2019-10-13T10:35:53.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:53.785+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.790027+0000) 2019-10-13T10:35:53.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:53.785+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.790627+0000) 2019-10-13T10:35:53.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:53.785+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.791142+0000) 2019-10-13T10:35:53.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:53.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:53.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:53.798+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:53.798672+0000) 2019-10-13T10:35:55.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:55.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.123873+0000) 2019-10-13T10:35:55.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:55.534+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.534384+0000) 2019-10-13T10:35:56.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:56.337+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:56.342292+0000) 2019-10-13T10:35:58.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:58.600+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.604623+0000) 2019-10-13T10:35:58.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:58.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.610215+0000) 2019-10-13T10:35:58.798 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:35:58.794+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.797306+0000) 2019-10-13T10:35:58.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:35:58.792+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.797906+0000) 2019-10-13T10:35:58.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:35:58.792+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.798447+0000) 2019-10-13T10:35:58.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:35:58.796+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.799413+0000) 2019-10-13T10:35:58.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:35:58.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.800031+0000) 2019-10-13T10:35:58.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:35:58.796+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.800666+0000) 2019-10-13T10:35:58.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:35:58.796+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.801638+0000) 2019-10-13T10:35:58.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:35:58.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:35:58.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:35:58.806+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.807913+0000) 2019-10-13T10:36:00.309 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:36:00.310 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:00.314 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:36:00.314 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:01.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:01.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:01.904404+0000) 2019-10-13T10:36:02.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:02.092+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:02.095521+0000) 2019-10-13T10:36:03.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:03.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.092344+0000) 2019-10-13T10:36:03.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:03.800+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.805809+0000) 2019-10-13T10:36:03.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:03.800+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.806809+0000) 2019-10-13T10:36:03.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:03.804+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.807352+0000) 2019-10-13T10:36:03.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:03.804+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.807492+0000) 2019-10-13T10:36:03.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:03.804+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.808838+0000) 2019-10-13T10:36:03.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:03.804+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.808959+0000) 2019-10-13T10:36:03.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:03.805+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.809527+0000) 2019-10-13T10:36:03.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:03.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:03.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:03.817+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:03.818398+0000) 2019-10-13T10:36:05.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:05.120+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.124094+0000) 2019-10-13T10:36:05.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:05.537+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.540830+0000) 2019-10-13T10:36:06.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:06.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:06.342522+0000) 2019-10-13T10:36:08.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:08.600+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.604836+0000) 2019-10-13T10:36:08.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:08.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.610441+0000) 2019-10-13T10:36:08.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:08.808+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.814425+0000) 2019-10-13T10:36:08.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:08.808+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.814968+0000) 2019-10-13T10:36:08.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:08.808+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.815259+0000) 2019-10-13T10:36:08.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:08.812+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.815823+0000) 2019-10-13T10:36:08.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:08.813+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.816035+0000) 2019-10-13T10:36:08.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:08.812+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.816395+0000) 2019-10-13T10:36:08.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:08.812+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.817588+0000) 2019-10-13T10:36:08.820 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:08.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103594 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:08.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:08.821+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.825186+0000) 2019-10-13T10:36:11.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:11.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:11.904655+0000) 2019-10-13T10:36:12.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:12.092+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:12.095774+0000) 2019-10-13T10:36:13.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:13.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.092543+0000) 2019-10-13T10:36:13.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:13.816+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.822493+0000) 2019-10-13T10:36:13.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:13.820+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.823496+0000) 2019-10-13T10:36:13.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:13.820+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.824342+0000) 2019-10-13T10:36:13.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:13.820+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.825249+0000) 2019-10-13T10:36:13.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:13.820+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.825261+0000) 2019-10-13T10:36:13.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:13.821+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.825978+0000) 2019-10-13T10:36:13.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:13.820+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.826436+0000) 2019-10-13T10:36:13.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:13.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:13.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:13.825+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:13.829906+0000) 2019-10-13T10:36:15.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:15.120+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.124297+0000) 2019-10-13T10:36:15.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:15.545+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.547381+0000) 2019-10-13T10:36:16.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:16.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:16.342775+0000) 2019-10-13T10:36:18.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:18.600+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.605020+0000) 2019-10-13T10:36:18.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:18.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.610588+0000) 2019-10-13T10:36:18.832 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:18.824+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.831006+0000) 2019-10-13T10:36:18.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:18.828+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.831499+0000) 2019-10-13T10:36:18.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:18.828+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.832225+0000) 2019-10-13T10:36:18.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:18.828+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.833171+0000) 2019-10-13T10:36:18.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:18.828+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.833389+0000) 2019-10-13T10:36:18.835 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:18.833+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.834360+0000) 2019-10-13T10:36:18.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:18.828+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.833994+0000) 2019-10-13T10:36:18.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:18.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103662 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:18.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:18.837+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.841294+0000) 2019-10-13T10:36:21.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:21.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:21.904897+0000) 2019-10-13T10:36:22.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:22.092+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:22.095982+0000) 2019-10-13T10:36:23.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:23.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.092757+0000) 2019-10-13T10:36:23.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:23.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.839896+0000) 2019-10-13T10:36:23.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:23.836+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.840067+0000) 2019-10-13T10:36:23.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:23.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.840785+0000) 2019-10-13T10:36:23.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:23.836+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.841217+0000) 2019-10-13T10:36:23.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:23.836+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.841893+0000) 2019-10-13T10:36:23.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:23.836+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.842833+0000) 2019-10-13T10:36:23.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:23.841+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.843366+0000) 2019-10-13T10:36:23.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:23.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:23.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:23.845+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:23.847828+0000) 2019-10-13T10:36:25.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:25.120+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.124497+0000) 2019-10-13T10:36:25.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:25.549+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.553591+0000) 2019-10-13T10:36:26.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:26.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:26.342990+0000) 2019-10-13T10:36:28.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:28.604+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.605220+0000) 2019-10-13T10:36:28.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:28.608+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.610776+0000) 2019-10-13T10:36:28.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:28.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.848197+0000) 2019-10-13T10:36:28.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:28.845+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.849328+0000) 2019-10-13T10:36:28.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:28.844+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.849509+0000) 2019-10-13T10:36:28.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:28.844+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.850111+0000) 2019-10-13T10:36:28.852 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:28.844+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.849989+0000) 2019-10-13T10:36:28.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:28.844+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.850595+0000) 2019-10-13T10:36:28.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:28.844+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.851216+0000) 2019-10-13T10:36:28.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:28.848+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:28.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:28.853+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.854901+0000) 2019-10-13T10:36:30.426 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:36:30.426 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:30.430 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:36:30.430 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:31.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:31.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:31.905105+0000) 2019-10-13T10:36:32.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:32.092+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:32.096185+0000) 2019-10-13T10:36:33.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:33.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.092977+0000) 2019-10-13T10:36:33.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:33.852+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.856507+0000) 2019-10-13T10:36:33.858 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:33.852+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.857161+0000) 2019-10-13T10:36:33.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:33.852+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.857832+0000) 2019-10-13T10:36:33.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:33.852+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.858014+0000) 2019-10-13T10:36:33.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:33.857+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.859282+0000) 2019-10-13T10:36:33.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:33.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.859582+0000) 2019-10-13T10:36:33.861 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:33.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.859984+0000) 2019-10-13T10:36:33.862 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:33.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:33.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:33.861+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:33.864957+0000) 2019-10-13T10:36:35.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:35.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.124736+0000) 2019-10-13T10:36:35.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:35.557+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.560315+0000) 2019-10-13T10:36:36.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:36.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:36.343175+0000) 2019-10-13T10:36:38.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:38.600+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.605411+0000) 2019-10-13T10:36:38.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:38.604+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.610967+0000) 2019-10-13T10:36:38.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:38.860+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.863996+0000) 2019-10-13T10:36:38.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:38.860+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.865035+0000) 2019-10-13T10:36:38.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:38.860+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.865400+0000) 2019-10-13T10:36:38.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:38.861+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.865992+0000) 2019-10-13T10:36:38.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:38.860+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.866750+0000) 2019-10-13T10:36:38.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:38.860+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.866862+0000) 2019-10-13T10:36:38.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:38.864+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.868489+0000) 2019-10-13T10:36:38.870 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:38.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:38.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:38.873+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.876440+0000) 2019-10-13T10:36:41.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:41.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:41.905339+0000) 2019-10-13T10:36:42.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:42.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:42.096418+0000) 2019-10-13T10:36:43.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:43.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.093169+0000) 2019-10-13T10:36:43.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:43.867+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.873036+0000) 2019-10-13T10:36:43.875 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:43.867+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.873828+0000) 2019-10-13T10:36:43.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:43.867+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.874276+0000) 2019-10-13T10:36:43.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:43.873+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.874518+0000) 2019-10-13T10:36:43.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:43.867+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.874810+0000) 2019-10-13T10:36:43.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:43.871+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.875684+0000) 2019-10-13T10:36:43.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:43.871+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.876262+0000) 2019-10-13T10:36:43.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:43.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:43.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:43.881+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:43.882351+0000) 2019-10-13T10:36:45.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:45.119+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.124911+0000) 2019-10-13T10:36:45.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:45.565+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.566726+0000) 2019-10-13T10:36:46.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:46.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:46.343398+0000) 2019-10-13T10:36:48.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:48.603+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.605609+0000) 2019-10-13T10:36:48.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:48.603+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.611196+0000) 2019-10-13T10:36:48.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:48.875+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.881981+0000) 2019-10-13T10:36:48.884 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:48.875+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.882684+0000) 2019-10-13T10:36:48.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:48.875+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.883145+0000) 2019-10-13T10:36:48.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:48.875+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.883283+0000) 2019-10-13T10:36:48.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:48.881+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.884348+0000) 2019-10-13T10:36:48.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:48.879+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.885061+0000) 2019-10-13T10:36:48.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:48.879+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.885748+0000) 2019-10-13T10:36:48.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:48.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:48.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:48.889+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.890194+0000) 2019-10-13T10:36:51.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:51.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:51.905573+0000) 2019-10-13T10:36:52.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:52.091+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:52.096643+0000) 2019-10-13T10:36:53.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:53.093+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.093375+0000) 2019-10-13T10:36:53.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:53.883+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.889348+0000) 2019-10-13T10:36:53.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:53.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.891400+0000) 2019-10-13T10:36:53.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:53.887+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.891845+0000) 2019-10-13T10:36:53.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:53.887+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.891553+0000) 2019-10-13T10:36:53.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:53.887+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.892291+0000) 2019-10-13T10:36:53.894 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:53.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.892769+0000) 2019-10-13T10:36:53.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:53.889+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.893613+0000) 2019-10-13T10:36:53.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:53.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:53.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:53.893+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:53.896522+0000) 2019-10-13T10:36:55.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:55.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.125113+0000) 2019-10-13T10:36:55.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:55.576+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.578130+0000) 2019-10-13T10:36:56.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:56.339+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:56.343607+0000) 2019-10-13T10:36:58.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:58.603+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.605808+0000) 2019-10-13T10:36:58.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:58.607+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.611426+0000) 2019-10-13T10:36:58.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:36:58.891+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.897773+0000) 2019-10-13T10:36:58.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:36:58.891+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.898042+0000) 2019-10-13T10:36:58.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:36:58.896+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.898758+0000) 2019-10-13T10:36:58.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:36:58.891+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.898476+0000) 2019-10-13T10:36:58.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:36:58.895+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.900366+0000) 2019-10-13T10:36:58.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:36:58.895+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.900670+0000) 2019-10-13T10:36:58.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:36:58.895+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.901228+0000) 2019-10-13T10:36:58.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:36:58.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:36:58.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:36:58.900+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.905795+0000) 2019-10-13T10:37:00.548 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:37:00.549 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:00.552 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:37:00.552 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:01.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:01.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:01.905779+0000) 2019-10-13T10:37:02.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:02.091+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:02.096844+0000) 2019-10-13T10:37:03.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:03.092+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.093568+0000) 2019-10-13T10:37:03.908 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:03.899+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.907230+0000) 2019-10-13T10:37:03.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:03.904+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.908038+0000) 2019-10-13T10:37:03.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:03.903+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.908220+0000) 2019-10-13T10:37:03.910 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:03.903+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.908518+0000) 2019-10-13T10:37:03.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:03.903+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.909400+0000) 2019-10-13T10:37:03.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:03.903+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.909802+0000) 2019-10-13T10:37:03.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:03.903+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.910320+0000) 2019-10-13T10:37:03.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:03.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 103971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:03.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:03.912+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:03.917230+0000) 2019-10-13T10:37:05.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:05.119+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.125317+0000) 2019-10-13T10:37:05.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:05.580+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.584681+0000) 2019-10-13T10:37:06.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:06.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:06.343812+0000) 2019-10-13T10:37:08.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:08.603+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.606032+0000) 2019-10-13T10:37:08.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:08.607+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.611644+0000) 2019-10-13T10:37:08.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:08.911+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.915493+0000) 2019-10-13T10:37:08.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:08.911+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.916162+0000) 2019-10-13T10:37:08.918 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:08.911+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.916745+0000) 2019-10-13T10:37:08.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:08.911+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.916931+0000) 2019-10-13T10:37:08.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:08.912+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.917722+0000) 2019-10-13T10:37:08.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:08.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.918168+0000) 2019-10-13T10:37:08.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:08.911+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.918238+0000) 2019-10-13T10:37:08.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:08.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:08.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:08.920+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.922556+0000) 2019-10-13T10:37:11.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:11.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:11.905998+0000) 2019-10-13T10:37:12.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:12.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:12.097077+0000) 2019-10-13T10:37:13.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:13.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.093766+0000) 2019-10-13T10:37:13.924 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:13.919+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.922687+0000) 2019-10-13T10:37:13.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:13.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.922921+0000) 2019-10-13T10:37:13.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:13.920+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.924806+0000) 2019-10-13T10:37:13.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:13.923+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.924609+0000) 2019-10-13T10:37:13.926 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:13.923+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.925241+0000) 2019-10-13T10:37:13.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:13.923+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.925828+0000) 2019-10-13T10:37:13.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:13.923+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.926412+0000) 2019-10-13T10:37:13.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:13.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:13.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:13.928+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:13.930531+0000) 2019-10-13T10:37:15.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:15.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.125520+0000) 2019-10-13T10:37:15.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:15.588+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.591196+0000) 2019-10-13T10:37:16.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:16.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:16.344006+0000) 2019-10-13T10:37:18.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:18.603+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.606263+0000) 2019-10-13T10:37:18.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:18.611+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.611869+0000) 2019-10-13T10:37:18.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:18.931+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.932255+0000) 2019-10-13T10:37:18.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:18.931+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.932659+0000) 2019-10-13T10:37:18.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:18.931+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.933030+0000) 2019-10-13T10:37:18.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:18.931+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.933335+0000) 2019-10-13T10:37:18.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:18.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.933621+0000) 2019-10-13T10:37:18.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:18.931+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.934405+0000) 2019-10-13T10:37:18.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:18.931+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.934651+0000) 2019-10-13T10:37:18.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:18.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:18.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:18.936+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.938235+0000) 2019-10-13T10:37:21.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:21.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:21.906200+0000) 2019-10-13T10:37:22.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:22.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:22.097285+0000) 2019-10-13T10:37:23.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:23.092+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.093958+0000) 2019-10-13T10:37:23.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:23.939+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.939870+0000) 2019-10-13T10:37:23.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:23.939+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.940910+0000) 2019-10-13T10:37:23.942 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:23.939+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.941261+0000) 2019-10-13T10:37:23.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:23.939+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.941685+0000) 2019-10-13T10:37:23.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:23.939+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.942274+0000) 2019-10-13T10:37:23.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:23.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.943025+0000) 2019-10-13T10:37:23.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:23.940+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.943515+0000) 2019-10-13T10:37:23.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:23.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:23.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:23.944+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:23.946938+0000) 2019-10-13T10:37:25.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:25.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.125718+0000) 2019-10-13T10:37:25.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:25.600+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.602371+0000) 2019-10-13T10:37:26.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:26.342+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:26.344220+0000) 2019-10-13T10:37:28.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:28.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.606487+0000) 2019-10-13T10:37:28.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:28.610+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.612083+0000) 2019-10-13T10:37:28.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:28.946+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.947891+0000) 2019-10-13T10:37:28.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:28.946+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.949287+0000) 2019-10-13T10:37:28.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:28.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.949634+0000) 2019-10-13T10:37:28.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:28.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.950372+0000) 2019-10-13T10:37:28.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:28.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.950820+0000) 2019-10-13T10:37:28.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:28.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.951067+0000) 2019-10-13T10:37:28.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:28.948+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.952471+0000) 2019-10-13T10:37:28.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:28.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:28.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:28.952+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.958015+0000) 2019-10-13T10:37:30.657 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:37:30.657 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:30.661 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:37:30.661 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:31.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:31.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:31.906397+0000) 2019-10-13T10:37:32.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:32.094+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:32.097491+0000) 2019-10-13T10:37:33.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:33.092+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.094194+0000) 2019-10-13T10:37:33.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:33.954+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.957051+0000) 2019-10-13T10:37:33.959 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:33.954+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.957304+0000) 2019-10-13T10:37:33.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:33.954+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.957754+0000) 2019-10-13T10:37:33.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:33.954+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.958658+0000) 2019-10-13T10:37:33.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:33.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.959164+0000) 2019-10-13T10:37:33.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:33.958+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.959444+0000) 2019-10-13T10:37:33.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:33.956+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.960250+0000) 2019-10-13T10:37:33.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:33.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:33.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:33.964+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:33.968710+0000) 2019-10-13T10:37:35.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:35.122+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.125942+0000) 2019-10-13T10:37:35.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:35.604+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.609034+0000) 2019-10-13T10:37:36.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:36.342+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:36.344456+0000) 2019-10-13T10:37:38.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:38.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.606691+0000) 2019-10-13T10:37:38.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:38.610+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.612275+0000) 2019-10-13T10:37:38.967 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:38.962+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.965475+0000) 2019-10-13T10:37:38.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:38.962+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.965749+0000) 2019-10-13T10:37:38.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:38.962+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.966000+0000) 2019-10-13T10:37:38.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:38.962+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.966355+0000) 2019-10-13T10:37:38.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:38.962+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.966724+0000) 2019-10-13T10:37:38.969 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:38.966+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.967559+0000) 2019-10-13T10:37:38.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:38.964+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.967974+0000) 2019-10-13T10:37:38.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:38.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:38.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:38.972+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.977037+0000) 2019-10-13T10:37:41.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:41.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:41.906628+0000) 2019-10-13T10:37:42.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:42.094+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:42.097710+0000) 2019-10-13T10:37:43.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:43.092+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.094410+0000) 2019-10-13T10:37:43.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:43.970+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.974101+0000) 2019-10-13T10:37:43.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:43.970+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.974731+0000) 2019-10-13T10:37:43.976 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:43.970+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.975242+0000) 2019-10-13T10:37:43.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:43.974+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.975515+0000) 2019-10-13T10:37:43.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:43.974+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.976218+0000) 2019-10-13T10:37:43.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:43.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.976989+0000) 2019-10-13T10:37:43.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:43.972+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.977613+0000) 2019-10-13T10:37:43.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:43.974+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:43.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:43.980+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:43.984554+0000) 2019-10-13T10:37:45.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:45.122+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.126141+0000) 2019-10-13T10:37:45.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:45.612+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.615424+0000) 2019-10-13T10:37:46.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:46.342+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:46.344661+0000) 2019-10-13T10:37:48.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:48.606+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.606891+0000) 2019-10-13T10:37:48.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:48.610+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.612498+0000) 2019-10-13T10:37:48.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:48.978+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.982337+0000) 2019-10-13T10:37:48.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:48.978+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.982588+0000) 2019-10-13T10:37:48.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:48.978+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.983001+0000) 2019-10-13T10:37:48.985 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:48.982+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.983403+0000) 2019-10-13T10:37:48.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:48.982+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.983914+0000) 2019-10-13T10:37:48.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:48.982+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.985423+0000) 2019-10-13T10:37:48.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:48.984+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.986235+0000) 2019-10-13T10:37:48.988 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:48.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:48.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:48.988+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.992148+0000) 2019-10-13T10:37:51.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:51.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:51.906860+0000) 2019-10-13T10:37:52.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:52.094+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:52.097935+0000) 2019-10-13T10:37:53.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:53.091+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.094612+0000) 2019-10-13T10:37:53.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:53.986+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.990284+0000) 2019-10-13T10:37:53.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:53.986+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.990736+0000) 2019-10-13T10:37:53.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:53.990+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.991788+0000) 2019-10-13T10:37:53.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:53.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.992587+0000) 2019-10-13T10:37:53.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:53.990+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.993697+0000) 2019-10-13T10:37:53.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:53.991+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.994184+0000) 2019-10-13T10:37:53.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:53.990+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.994404+0000) 2019-10-13T10:37:53.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:53.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:53.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:53.995+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:53.998477+0000) 2019-10-13T10:37:55.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:55.122+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.126344+0000) 2019-10-13T10:37:55.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:55.615+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.621766+0000) 2019-10-13T10:37:56.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:56.342+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:56.344854+0000) 2019-10-13T10:37:58.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:58.606+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.607090+0000) 2019-10-13T10:37:58.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:58.610+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.612725+0000) 2019-10-13T10:37:59.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:37:58.995+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.999300+0000) 2019-10-13T10:37:59.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:37:58.998+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.999607+0000) 2019-10-13T10:37:59.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:37:58.998+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.000175+0000) 2019-10-13T10:37:59.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:37:58.998+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.000419+0000) 2019-10-13T10:37:59.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:37:58.998+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.000623+0000) 2019-10-13T10:37:59.004 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:37:58.998+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.002619+0000) 2019-10-13T10:37:59.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:37:58.998+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.002831+0000) 2019-10-13T10:37:59.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:37:59.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:37:59.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:37:59.003+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.008037+0000) 2019-10-13T10:38:00.778 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:38:00.778 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:00.782 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:38:00.782 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:01.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:01.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:01.907061+0000) 2019-10-13T10:38:02.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:02.094+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:02.098133+0000) 2019-10-13T10:38:03.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:03.091+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:03.094806+0000) 2019-10-13T10:38:04.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:04.006+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.008393+0000) 2019-10-13T10:38:04.010 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:04.006+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.008773+0000) 2019-10-13T10:38:04.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:04.006+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.009650+0000) 2019-10-13T10:38:04.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:04.006+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.010116+0000) 2019-10-13T10:38:04.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:04.007+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.010500+0000) 2019-10-13T10:38:04.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:04.006+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.011149+0000) 2019-10-13T10:38:04.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:04.010+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.011754+0000) 2019-10-13T10:38:04.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:04.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:04.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:04.011+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.018002+0000) 2019-10-13T10:38:05.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:05.122+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.126569+0000) 2019-10-13T10:38:05.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:05.627+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.628125+0000) 2019-10-13T10:38:06.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:06.342+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:06.345052+0000) 2019-10-13T10:38:08.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:08.602+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:08.607282+0000) 2019-10-13T10:38:08.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:08.610+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:08.612952+0000) 2019-10-13T10:38:09.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:09.014+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.015745+0000) 2019-10-13T10:38:09.017 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:09.014+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.016154+0000) 2019-10-13T10:38:09.018 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:09.014+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.017089+0000) 2019-10-13T10:38:09.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:09.014+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.017466+0000) 2019-10-13T10:38:09.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:09.014+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.018353+0000) 2019-10-13T10:38:09.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:09.014+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.018658+0000) 2019-10-13T10:38:09.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:09.011+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.016639+0000) 2019-10-13T10:38:09.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:09.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:09.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:09.023+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.027316+0000) 2019-10-13T10:38:11.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:11.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:11.907259+0000) 2019-10-13T10:38:12.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:12.097+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:12.098338+0000) 2019-10-13T10:38:13.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:13.091+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:13.095002+0000) 2019-10-13T10:38:14.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:14.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.023957+0000) 2019-10-13T10:38:14.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:14.019+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.024293+0000) 2019-10-13T10:38:14.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:14.021+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.024876+0000) 2019-10-13T10:38:14.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:14.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.025224+0000) 2019-10-13T10:38:14.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:14.021+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.026796+0000) 2019-10-13T10:38:14.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:14.021+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.027231+0000) 2019-10-13T10:38:14.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:14.025+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.027671+0000) 2019-10-13T10:38:14.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:14.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:14.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:14.027+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.033662+0000) 2019-10-13T10:38:15.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:15.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.126774+0000) 2019-10-13T10:38:15.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:15.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.634693+0000) 2019-10-13T10:38:16.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:16.341+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:16.345247+0000) 2019-10-13T10:38:18.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:18.605+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:18.607520+0000) 2019-10-13T10:38:18.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:18.609+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:18.613148+0000) 2019-10-13T10:38:19.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:19.029+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.033454+0000) 2019-10-13T10:38:19.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:19.029+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.033929+0000) 2019-10-13T10:38:19.036 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:19.029+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.034351+0000) 2019-10-13T10:38:19.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:19.029+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.034864+0000) 2019-10-13T10:38:19.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:19.031+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.035637+0000) 2019-10-13T10:38:19.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:19.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.036351+0000) 2019-10-13T10:38:19.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:19.033+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.036633+0000) 2019-10-13T10:38:19.038 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:19.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:19.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:19.035+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.041717+0000) 2019-10-13T10:38:21.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:21.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:21.907505+0000) 2019-10-13T10:38:22.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:22.093+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:22.098535+0000) 2019-10-13T10:38:23.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:23.091+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:23.095202+0000) 2019-10-13T10:38:24.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:24.037+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.040803+0000) 2019-10-13T10:38:24.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:24.037+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.041147+0000) 2019-10-13T10:38:24.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:24.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.042023+0000) 2019-10-13T10:38:24.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:24.039+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.042288+0000) 2019-10-13T10:38:24.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:24.037+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.042532+0000) 2019-10-13T10:38:24.044 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:24.037+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.043252+0000) 2019-10-13T10:38:24.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:24.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.043541+0000) 2019-10-13T10:38:24.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:24.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:24.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:24.043+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.048329+0000) 2019-10-13T10:38:25.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:25.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.127020+0000) 2019-10-13T10:38:25.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:25.639+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.641084+0000) 2019-10-13T10:38:26.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:26.341+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:26.345466+0000) 2019-10-13T10:38:28.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:28.605+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:28.607715+0000) 2019-10-13T10:38:28.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:28.609+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:28.613333+0000) 2019-10-13T10:38:29.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:29.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.049945+0000) 2019-10-13T10:38:29.051 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:29.045+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.050147+0000) 2019-10-13T10:38:29.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:29.047+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.050563+0000) 2019-10-13T10:38:29.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:29.045+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.051171+0000) 2019-10-13T10:38:29.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:29.049+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.051513+0000) 2019-10-13T10:38:29.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:29.049+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.051832+0000) 2019-10-13T10:38:29.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:29.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.052152+0000) 2019-10-13T10:38:29.056 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:29.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:29.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:29.055+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.058969+0000) 2019-10-13T10:38:30.888 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:38:30.888 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:30.892 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:38:30.892 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:31.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:31.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:31.907741+0000) 2019-10-13T10:38:32.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:32.097+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:32.098742+0000) 2019-10-13T10:38:33.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:33.095+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:33.095417+0000) 2019-10-13T10:38:34.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:34.053+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.058712+0000) 2019-10-13T10:38:34.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:34.053+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.058789+0000) 2019-10-13T10:38:34.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:34.053+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.059145+0000) 2019-10-13T10:38:34.061 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:34.057+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.059859+0000) 2019-10-13T10:38:34.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:34.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.060938+0000) 2019-10-13T10:38:34.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:34.055+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.061341+0000) 2019-10-13T10:38:34.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:34.057+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.061692+0000) 2019-10-13T10:38:34.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:34.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:34.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:34.059+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.064416+0000) 2019-10-13T10:38:35.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:35.121+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.127221+0000) 2019-10-13T10:38:35.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:35.647+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.649777+0000) 2019-10-13T10:38:36.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:36.341+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:36.345636+0000) 2019-10-13T10:38:38.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:38.605+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:38.607933+0000) 2019-10-13T10:38:38.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:38.609+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:38.613547+0000) 2019-10-13T10:38:39.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:39.061+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.066163+0000) 2019-10-13T10:38:39.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:39.061+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.066809+0000) 2019-10-13T10:38:39.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:39.061+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.066917+0000) 2019-10-13T10:38:39.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:39.063+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.069004+0000) 2019-10-13T10:38:39.070 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:39.065+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.069345+0000) 2019-10-13T10:38:39.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:39.065+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.069909+0000) 2019-10-13T10:38:39.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:39.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.069626+0000) 2019-10-13T10:38:39.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:39.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:39.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:39.071+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.076998+0000) 2019-10-13T10:38:41.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:41.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:41.907924+0000) 2019-10-13T10:38:42.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:42.097+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:42.098948+0000) 2019-10-13T10:38:43.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:43.091+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:43.095642+0000) 2019-10-13T10:38:44.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:44.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.075641+0000) 2019-10-13T10:38:44.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:44.073+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.076479+0000) 2019-10-13T10:38:44.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:44.073+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.077166+0000) 2019-10-13T10:38:44.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:44.073+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.077807+0000) 2019-10-13T10:38:44.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:44.073+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.078408+0000) 2019-10-13T10:38:44.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:44.073+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.078524+0000) 2019-10-13T10:38:44.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:44.075+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.078913+0000) 2019-10-13T10:38:44.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:44.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104660 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:44.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:44.079+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.085695+0000) 2019-10-13T10:38:45.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:45.125+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.127419+0000) 2019-10-13T10:38:45.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:45.654+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.656134+0000) 2019-10-13T10:38:46.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:46.345+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:46.345805+0000) 2019-10-13T10:38:48.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:48.605+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:48.608139+0000) 2019-10-13T10:38:48.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:48.609+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:48.613751+0000) 2019-10-13T10:38:49.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:49.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.084385+0000) 2019-10-13T10:38:49.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:49.081+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.084730+0000) 2019-10-13T10:38:49.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:49.081+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.085039+0000) 2019-10-13T10:38:49.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:49.081+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.085563+0000) 2019-10-13T10:38:49.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:49.081+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.085822+0000) 2019-10-13T10:38:49.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:49.081+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.086322+0000) 2019-10-13T10:38:49.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:49.082+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.087836+0000) 2019-10-13T10:38:49.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:49.085+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:49.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:49.090+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.096151+0000) 2019-10-13T10:38:51.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:51.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:51.908125+0000) 2019-10-13T10:38:52.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:52.097+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:52.099183+0000) 2019-10-13T10:38:53.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:53.094+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:53.095844+0000) 2019-10-13T10:38:54.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:54.088+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.091535+0000) 2019-10-13T10:38:54.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:54.088+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.092871+0000) 2019-10-13T10:38:54.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:54.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.093602+0000) 2019-10-13T10:38:54.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:54.088+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.093939+0000) 2019-10-13T10:38:54.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:54.088+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.095136+0000) 2019-10-13T10:38:54.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:54.090+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.095519+0000) 2019-10-13T10:38:54.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:54.092+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.095719+0000) 2019-10-13T10:38:54.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:54.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:54.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:54.094+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.101005+0000) 2019-10-13T10:38:55.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:55.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.127605+0000) 2019-10-13T10:38:55.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:55.658+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.662596+0000) 2019-10-13T10:38:56.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:56.344+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:56.345973+0000) 2019-10-13T10:38:58.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:58.604+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:58.608346+0000) 2019-10-13T10:38:58.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:58.608+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:58.613943+0000) 2019-10-13T10:38:59.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:38:59.096+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.101794+0000) 2019-10-13T10:38:59.104 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:38:59.096+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.102843+0000) 2019-10-13T10:38:59.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:38:59.100+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.103336+0000) 2019-10-13T10:38:59.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:38:59.098+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.104109+0000) 2019-10-13T10:38:59.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:38:59.100+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.103650+0000) 2019-10-13T10:38:59.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:38:59.100+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.104414+0000) 2019-10-13T10:38:59.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:38:59.100+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.105071+0000) 2019-10-13T10:38:59.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:38:59.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:38:59.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:38:59.106+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.110684+0000) 2019-10-13T10:39:00.999 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:39:00.999 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:01.002 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:39:01.003 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:01.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:01.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:01.908339+0000) 2019-10-13T10:39:02.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:02.096+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:02.099386+0000) 2019-10-13T10:39:03.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:03.090+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:03.096062+0000) 2019-10-13T10:39:04.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:04.104+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.109969+0000) 2019-10-13T10:39:04.112 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:04.104+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.110985+0000) 2019-10-13T10:39:04.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:04.108+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.111635+0000) 2019-10-13T10:39:04.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:04.108+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.111905+0000) 2019-10-13T10:39:04.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:04.106+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.112487+0000) 2019-10-13T10:39:04.114 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:04.108+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.112811+0000) 2019-10-13T10:39:04.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:04.108+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.113543+0000) 2019-10-13T10:39:04.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:04.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104798 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:04.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:04.114+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.119755+0000) 2019-10-13T10:39:05.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:05.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.127810+0000) 2019-10-13T10:39:05.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:05.666+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.669238+0000) 2019-10-13T10:39:06.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:06.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:06.346186+0000) 2019-10-13T10:39:08.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:08.604+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:08.608549+0000) 2019-10-13T10:39:08.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:08.608+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:08.614151+0000) 2019-10-13T10:39:09.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:09.112+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.117652+0000) 2019-10-13T10:39:09.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:09.112+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.118383+0000) 2019-10-13T10:39:09.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:09.116+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.119388+0000) 2019-10-13T10:39:09.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:09.116+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.119769+0000) 2019-10-13T10:39:09.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:09.116+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.120318+0000) 2019-10-13T10:39:09.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:09.116+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.120835+0000) 2019-10-13T10:39:09.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:09.114+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.121644+0000) 2019-10-13T10:39:09.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:09.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:09.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:09.122+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.128315+0000) 2019-10-13T10:39:11.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:11.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:11.908537+0000) 2019-10-13T10:39:12.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:12.096+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:12.099554+0000) 2019-10-13T10:39:13.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:13.094+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:13.096283+0000) 2019-10-13T10:39:14.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:14.124+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.127375+0000) 2019-10-13T10:39:14.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:14.124+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.127957+0000) 2019-10-13T10:39:14.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:14.124+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.128563+0000) 2019-10-13T10:39:14.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:14.122+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.129274+0000) 2019-10-13T10:39:14.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:14.124+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.129457+0000) 2019-10-13T10:39:14.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:14.124+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.129754+0000) 2019-10-13T10:39:14.131 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:14.124+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.130386+0000) 2019-10-13T10:39:14.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:14.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104867 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:14.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:14.126+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.134051+0000) 2019-10-13T10:39:15.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:15.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.128008+0000) 2019-10-13T10:39:15.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:15.674+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.675686+0000) 2019-10-13T10:39:16.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:16.344+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:16.346411+0000) 2019-10-13T10:39:18.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:18.604+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:18.608749+0000) 2019-10-13T10:39:18.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:18.608+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:18.614349+0000) 2019-10-13T10:39:19.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:19.128+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.134229+0000) 2019-10-13T10:39:19.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:19.130+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.135286+0000) 2019-10-13T10:39:19.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:19.132+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.135802+0000) 2019-10-13T10:39:19.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:19.132+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.136208+0000) 2019-10-13T10:39:19.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:19.132+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.136722+0000) 2019-10-13T10:39:19.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:19.132+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.137876+0000) 2019-10-13T10:39:19.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:19.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.138221+0000) 2019-10-13T10:39:19.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:19.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:19.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:19.138+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.143408+0000) 2019-10-13T10:39:21.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:21.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:21.908735+0000) 2019-10-13T10:39:22.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:22.096+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:22.099754+0000) 2019-10-13T10:39:23.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:23.094+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:23.096515+0000) 2019-10-13T10:39:24.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:24.138+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.143759+0000) 2019-10-13T10:39:24.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:24.140+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.144127+0000) 2019-10-13T10:39:24.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:24.140+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.144478+0000) 2019-10-13T10:39:24.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:24.140+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.144981+0000) 2019-10-13T10:39:24.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:24.140+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.145853+0000) 2019-10-13T10:39:24.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:24.140+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.146243+0000) 2019-10-13T10:39:24.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:24.140+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.146647+0000) 2019-10-13T10:39:24.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:24.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:24.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:24.146+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.154075+0000) 2019-10-13T10:39:25.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:25.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.128212+0000) 2019-10-13T10:39:25.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:25.678+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.682047+0000) 2019-10-13T10:39:26.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:26.340+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:26.346610+0000) 2019-10-13T10:39:28.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:28.608+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:28.608979+0000) 2019-10-13T10:39:28.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:28.608+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:28.614543+0000) 2019-10-13T10:39:29.153 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:29.148+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.151495+0000) 2019-10-13T10:39:29.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:29.148+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.152608+0000) 2019-10-13T10:39:29.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:29.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.153128+0000) 2019-10-13T10:39:29.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:29.148+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.153510+0000) 2019-10-13T10:39:29.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:29.148+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.153844+0000) 2019-10-13T10:39:29.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:29.148+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.154063+0000) 2019-10-13T10:39:29.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:29.150+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.154817+0000) 2019-10-13T10:39:29.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:29.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 104973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:29.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:29.154+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.161318+0000) 2019-10-13T10:39:31.111 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:39:31.111 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:31.115 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:39:31.115 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:31.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:31.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:31.908951+0000) 2019-10-13T10:39:32.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:32.096+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:32.099963+0000) 2019-10-13T10:39:33.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:33.094+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:33.096702+0000) 2019-10-13T10:39:34.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:34.156+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.159863+0000) 2019-10-13T10:39:34.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:34.156+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.160271+0000) 2019-10-13T10:39:34.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:34.156+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.160712+0000) 2019-10-13T10:39:34.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:34.156+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.161269+0000) 2019-10-13T10:39:34.164 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:34.162+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.163145+0000) 2019-10-13T10:39:34.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:34.160+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.164238+0000) 2019-10-13T10:39:34.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:34.160+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.164559+0000) 2019-10-13T10:39:34.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:34.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:34.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:34.170+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.173565+0000) 2019-10-13T10:39:35.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:35.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.128404+0000) 2019-10-13T10:39:35.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:35.686+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.688060+0000) 2019-10-13T10:39:36.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:36.344+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:36.346801+0000) 2019-10-13T10:39:38.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:38.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:38.609209+0000) 2019-10-13T10:39:38.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:38.611+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:38.614754+0000) 2019-10-13T10:39:39.172 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:39.163+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.168897+0000) 2019-10-13T10:39:39.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:39.163+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.169230+0000) 2019-10-13T10:39:39.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:39.163+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.170598+0000) 2019-10-13T10:39:39.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:39.167+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.171470+0000) 2019-10-13T10:39:39.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:39.167+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.171888+0000) 2019-10-13T10:39:39.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:39.170+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.172446+0000) 2019-10-13T10:39:39.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:39.167+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.173261+0000) 2019-10-13T10:39:39.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:39.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:39.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:39.174+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.176075+0000) 2019-10-13T10:39:41.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:41.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:41.909116+0000) 2019-10-13T10:39:42.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:42.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:42.100185+0000) 2019-10-13T10:39:43.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:43.093+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:43.096895+0000) 2019-10-13T10:39:44.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:44.171+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.177162+0000) 2019-10-13T10:39:44.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:44.173+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.177582+0000) 2019-10-13T10:39:44.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:44.171+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.178243+0000) 2019-10-13T10:39:44.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:44.171+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.179015+0000) 2019-10-13T10:39:44.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:44.175+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.179525+0000) 2019-10-13T10:39:44.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:44.175+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.180393+0000) 2019-10-13T10:39:44.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:44.175+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.181000+0000) 2019-10-13T10:39:44.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:44.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:44.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:44.181+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.182907+0000) 2019-10-13T10:39:45.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:45.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.128586+0000) 2019-10-13T10:39:45.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:45.693+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.695421+0000) 2019-10-13T10:39:46.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:46.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:46.346988+0000) 2019-10-13T10:39:48.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:48.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:48.609425+0000) 2019-10-13T10:39:48.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:48.607+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:48.614945+0000) 2019-10-13T10:39:49.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:49.181+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.185818+0000) 2019-10-13T10:39:49.187 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:49.179+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.186267+0000) 2019-10-13T10:39:49.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:49.179+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.186836+0000) 2019-10-13T10:39:49.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:49.183+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.187717+0000) 2019-10-13T10:39:49.189 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:49.183+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.188338+0000) 2019-10-13T10:39:49.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:49.183+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.189276+0000) 2019-10-13T10:39:49.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:49.183+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.189717+0000) 2019-10-13T10:39:49.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:49.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:49.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:49.193+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.194927+0000) 2019-10-13T10:39:51.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:51.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:51.909328+0000) 2019-10-13T10:39:52.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:52.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:52.100386+0000) 2019-10-13T10:39:53.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:53.093+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:53.097091+0000) 2019-10-13T10:39:54.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:54.187+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.194582+0000) 2019-10-13T10:39:54.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:54.191+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.195378+0000) 2019-10-13T10:39:54.197 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:54.191+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.195853+0000) 2019-10-13T10:39:54.197 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:54.193+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.196438+0000) 2019-10-13T10:39:54.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:54.191+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.196867+0000) 2019-10-13T10:39:54.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:54.191+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.198045+0000) 2019-10-13T10:39:54.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:54.191+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.198375+0000) 2019-10-13T10:39:54.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:54.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:54.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:54.201+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.205994+0000) 2019-10-13T10:39:55.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:55.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.128784+0000) 2019-10-13T10:39:55.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:55.701+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.701842+0000) 2019-10-13T10:39:56.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:56.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:56.347164+0000) 2019-10-13T10:39:58.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:58.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:58.609654+0000) 2019-10-13T10:39:58.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:58.611+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:58.615159+0000) 2019-10-13T10:39:59.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:39:59.199+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.203847+0000) 2019-10-13T10:39:59.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:39:59.199+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.204578+0000) 2019-10-13T10:39:59.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:39:59.199+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.205685+0000) 2019-10-13T10:39:59.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:39:59.199+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.205935+0000) 2019-10-13T10:39:59.207 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:39:59.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.206616+0000) 2019-10-13T10:39:59.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:39:59.199+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.206988+0000) 2019-10-13T10:39:59.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:39:59.203+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.207518+0000) 2019-10-13T10:39:59.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:39:59.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:39:59.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:39:59.205+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.210006+0000) 2019-10-13T10:40:01.246 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:40:01.246 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:01.250 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:40:01.250 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:01.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:01.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:01.909576+0000) 2019-10-13T10:40:02.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:02.095+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:02.100634+0000) 2019-10-13T10:40:03.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:03.093+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:03.097289+0000) 2019-10-13T10:40:04.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:04.207+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.211690+0000) 2019-10-13T10:40:04.213 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:04.207+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.212355+0000) 2019-10-13T10:40:04.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:04.207+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.212966+0000) 2019-10-13T10:40:04.214 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:04.207+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.213298+0000) 2019-10-13T10:40:04.215 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:04.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.213632+0000) 2019-10-13T10:40:04.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:04.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.214511+0000) 2019-10-13T10:40:04.216 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:04.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.215740+0000) 2019-10-13T10:40:04.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:04.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:04.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:04.217+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.221562+0000) 2019-10-13T10:40:05.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:05.123+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.128980+0000) 2019-10-13T10:40:05.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:05.709+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.711793+0000) 2019-10-13T10:40:06.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:06.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:06.347403+0000) 2019-10-13T10:40:08.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:08.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:08.609857+0000) 2019-10-13T10:40:08.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:08.615+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:08.615363+0000) 2019-10-13T10:40:09.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:09.219+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.220319+0000) 2019-10-13T10:40:09.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:09.219+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.220674+0000) 2019-10-13T10:40:09.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:09.219+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.221474+0000) 2019-10-13T10:40:09.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:09.219+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.222757+0000) 2019-10-13T10:40:09.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:09.219+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.223034+0000) 2019-10-13T10:40:09.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:09.223+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.223451+0000) 2019-10-13T10:40:09.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:09.221+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.224473+0000) 2019-10-13T10:40:09.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:09.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:09.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:09.225+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.228842+0000) 2019-10-13T10:40:11.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:11.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:11.909827+0000) 2019-10-13T10:40:12.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:12.099+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:12.100862+0000) 2019-10-13T10:40:13.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:13.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:13.097484+0000) 2019-10-13T10:40:14.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:14.227+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.229416+0000) 2019-10-13T10:40:14.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:14.225+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.229979+0000) 2019-10-13T10:40:14.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:14.227+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.230453+0000) 2019-10-13T10:40:14.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:14.227+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.230691+0000) 2019-10-13T10:40:14.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:14.227+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.230980+0000) 2019-10-13T10:40:14.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:14.227+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.231205+0000) 2019-10-13T10:40:14.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:14.231+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.232183+0000) 2019-10-13T10:40:14.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:14.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105281 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:14.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:14.237+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.238271+0000) 2019-10-13T10:40:15.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:15.127+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.129156+0000) 2019-10-13T10:40:15.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:15.717+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.718476+0000) 2019-10-13T10:40:16.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:16.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:16.347577+0000) 2019-10-13T10:40:18.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:18.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:18.610055+0000) 2019-10-13T10:40:18.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:18.615+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:18.615561+0000) 2019-10-13T10:40:19.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:19.235+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.237773+0000) 2019-10-13T10:40:19.239 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:19.233+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.237886+0000) 2019-10-13T10:40:19.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:19.235+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.238521+0000) 2019-10-13T10:40:19.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:19.235+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.238885+0000) 2019-10-13T10:40:19.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:19.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.240080+0000) 2019-10-13T10:40:19.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:19.239+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.240356+0000) 2019-10-13T10:40:19.245 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:19.239+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.241288+0000) 2019-10-13T10:40:19.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:19.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:19.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:19.241+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.246017+0000) 2019-10-13T10:40:21.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:21.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:21.910060+0000) 2019-10-13T10:40:22.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:22.098+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:22.101092+0000) 2019-10-13T10:40:23.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:23.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:23.097685+0000) 2019-10-13T10:40:24.247 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:24.241+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.245718+0000) 2019-10-13T10:40:24.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:24.242+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.246151+0000) 2019-10-13T10:40:24.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:24.242+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.246505+0000) 2019-10-13T10:40:24.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:24.242+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.246879+0000) 2019-10-13T10:40:24.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:24.246+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.248123+0000) 2019-10-13T10:40:24.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:24.246+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.249352+0000) 2019-10-13T10:40:24.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:24.246+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.249802+0000) 2019-10-13T10:40:24.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:24.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:24.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:24.253+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.255082+0000) 2019-10-13T10:40:25.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:25.126+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.129327+0000) 2019-10-13T10:40:25.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:25.721+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.724131+0000) 2019-10-13T10:40:26.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:26.346+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:26.347746+0000) 2019-10-13T10:40:28.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:28.606+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:28.610249+0000) 2019-10-13T10:40:28.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:28.614+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:28.615756+0000) 2019-10-13T10:40:29.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:29.250+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.254863+0000) 2019-10-13T10:40:29.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:29.250+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.255221+0000) 2019-10-13T10:40:29.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:29.254+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.255860+0000) 2019-10-13T10:40:29.259 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:29.253+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.257023+0000) 2019-10-13T10:40:29.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:29.254+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.257452+0000) 2019-10-13T10:40:29.260 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:29.254+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.257675+0000) 2019-10-13T10:40:29.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:29.254+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.258493+0000) 2019-10-13T10:40:29.261 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:29.258+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:29.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:29.265+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.266321+0000) 2019-10-13T10:40:31.383 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:40:31.383 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:31.387 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:40:31.387 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:31.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:31.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:31.910300+0000) 2019-10-13T10:40:32.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:32.098+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:32.101299+0000) 2019-10-13T10:40:33.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:33.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:33.097882+0000) 2019-10-13T10:40:34.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:34.262+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.263526+0000) 2019-10-13T10:40:34.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:34.262+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.265126+0000) 2019-10-13T10:40:34.267 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:34.261+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.265935+0000) 2019-10-13T10:40:34.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:34.262+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.266234+0000) 2019-10-13T10:40:34.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:34.262+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.266497+0000) 2019-10-13T10:40:34.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:34.262+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.266851+0000) 2019-10-13T10:40:34.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:34.262+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.267189+0000) 2019-10-13T10:40:34.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:34.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:34.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:34.269+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.270461+0000) 2019-10-13T10:40:35.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:35.126+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.129502+0000) 2019-10-13T10:40:35.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:35.729+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.733255+0000) 2019-10-13T10:40:36.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:36.346+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:36.347932+0000) 2019-10-13T10:40:38.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:38.606+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:38.610451+0000) 2019-10-13T10:40:38.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:38.614+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:38.615964+0000) 2019-10-13T10:40:39.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:39.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.273216+0000) 2019-10-13T10:40:39.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:39.270+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.273764+0000) 2019-10-13T10:40:39.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:39.270+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.274350+0000) 2019-10-13T10:40:39.276 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:39.272+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.274737+0000) 2019-10-13T10:40:39.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:39.270+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.275053+0000) 2019-10-13T10:40:39.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:39.270+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.275272+0000) 2019-10-13T10:40:39.277 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:39.274+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.275520+0000) 2019-10-13T10:40:39.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:39.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:39.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:39.280+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.283409+0000) 2019-10-13T10:40:41.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:41.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:41.910508+0000) 2019-10-13T10:40:42.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:42.098+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:42.101509+0000) 2019-10-13T10:40:43.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:43.096+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:43.098078+0000) 2019-10-13T10:40:44.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:44.278+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.280913+0000) 2019-10-13T10:40:44.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:44.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.282869+0000) 2019-10-13T10:40:44.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:44.278+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.283206+0000) 2019-10-13T10:40:44.284 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:44.282+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.283587+0000) 2019-10-13T10:40:44.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:44.280+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.284201+0000) 2019-10-13T10:40:44.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:44.282+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.284663+0000) 2019-10-13T10:40:44.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:44.282+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.285312+0000) 2019-10-13T10:40:44.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:44.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:44.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:44.288+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.291771+0000) 2019-10-13T10:40:45.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:45.126+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.129680+0000) 2019-10-13T10:40:45.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:45.736+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.739689+0000) 2019-10-13T10:40:46.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:46.346+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:46.348126+0000) 2019-10-13T10:40:48.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:48.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:48.610678+0000) 2019-10-13T10:40:48.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:48.614+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:48.616193+0000) 2019-10-13T10:40:49.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:49.286+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.289430+0000) 2019-10-13T10:40:49.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:49.286+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.289892+0000) 2019-10-13T10:40:49.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:49.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.290294+0000) 2019-10-13T10:40:49.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:49.286+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.290748+0000) 2019-10-13T10:40:49.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:49.290+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.291440+0000) 2019-10-13T10:40:49.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:49.290+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.291693+0000) 2019-10-13T10:40:49.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:49.290+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.293467+0000) 2019-10-13T10:40:49.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:49.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105524 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:49.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:49.296+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.299255+0000) 2019-10-13T10:40:51.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:51.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:51.910714+0000) 2019-10-13T10:40:52.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:52.098+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:52.101725+0000) 2019-10-13T10:40:53.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:53.096+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:53.098335+0000) 2019-10-13T10:40:54.299 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:54.294+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.297956+0000) 2019-10-13T10:40:54.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:54.296+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.299363+0000) 2019-10-13T10:40:54.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:54.298+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.299626+0000) 2019-10-13T10:40:54.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:54.298+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.300223+0000) 2019-10-13T10:40:54.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:54.298+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.300842+0000) 2019-10-13T10:40:54.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:54.298+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.301106+0000) 2019-10-13T10:40:54.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:54.298+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.301367+0000) 2019-10-13T10:40:54.304 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:54.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:54.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:54.300+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.304997+0000) 2019-10-13T10:40:55.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:55.126+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.129856+0000) 2019-10-13T10:40:55.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:55.744+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.746179+0000) 2019-10-13T10:40:56.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:56.346+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:56.348307+0000) 2019-10-13T10:40:58.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:58.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:58.610878+0000) 2019-10-13T10:40:58.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:58.614+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:58.616407+0000) 2019-10-13T10:40:59.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:40:59.302+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.306409+0000) 2019-10-13T10:40:59.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:40:59.302+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.306937+0000) 2019-10-13T10:40:59.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:40:59.306+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.307947+0000) 2019-10-13T10:40:59.309 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:40:59.306+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.308317+0000) 2019-10-13T10:40:59.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:40:59.304+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.309300+0000) 2019-10-13T10:40:59.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:40:59.306+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.309968+0000) 2019-10-13T10:40:59.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:40:59.306+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.310752+0000) 2019-10-13T10:40:59.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:40:59.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:40:59.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:40:59.312+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.317176+0000) 2019-10-13T10:41:01.517 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:41:01.517 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:01.521 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:41:01.521 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:01.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:01.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:01.910922+0000) 2019-10-13T10:41:02.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:02.098+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:02.101941+0000) 2019-10-13T10:41:03.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:03.096+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:03.098534+0000) 2019-10-13T10:41:04.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:04.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.314868+0000) 2019-10-13T10:41:04.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:04.312+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.315204+0000) 2019-10-13T10:41:04.317 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:04.314+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.315459+0000) 2019-10-13T10:41:04.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:04.314+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.316261+0000) 2019-10-13T10:41:04.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:04.314+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.316912+0000) 2019-10-13T10:41:04.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:04.314+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.317351+0000) 2019-10-13T10:41:04.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:04.314+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.318127+0000) 2019-10-13T10:41:04.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:04.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:04.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:04.320+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.324842+0000) 2019-10-13T10:41:05.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:05.125+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.130043+0000) 2019-10-13T10:41:05.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:05.748+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.752640+0000) 2019-10-13T10:41:06.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:06.345+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:06.348482+0000) 2019-10-13T10:41:08.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:08.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:08.611083+0000) 2019-10-13T10:41:08.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:08.613+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:08.616601+0000) 2019-10-13T10:41:09.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:09.317+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.323048+0000) 2019-10-13T10:41:09.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:09.321+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.323353+0000) 2019-10-13T10:41:09.325 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:09.321+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.324123+0000) 2019-10-13T10:41:09.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:09.321+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.325088+0000) 2019-10-13T10:41:09.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:09.320+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.325577+0000) 2019-10-13T10:41:09.328 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:09.321+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.326384+0000) 2019-10-13T10:41:09.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:09.321+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.326688+0000) 2019-10-13T10:41:09.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:09.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:09.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:09.332+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.334597+0000) 2019-10-13T10:41:11.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:11.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:11.911157+0000) 2019-10-13T10:41:12.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:12.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:12.102150+0000) 2019-10-13T10:41:13.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:13.096+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:13.098733+0000) 2019-10-13T10:41:14.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:14.329+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.332445+0000) 2019-10-13T10:41:14.334 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:14.328+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.332579+0000) 2019-10-13T10:41:14.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:14.329+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.333309+0000) 2019-10-13T10:41:14.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:14.329+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.333793+0000) 2019-10-13T10:41:14.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:14.329+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.334771+0000) 2019-10-13T10:41:14.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:14.329+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.335283+0000) 2019-10-13T10:41:14.336 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:14.333+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.335815+0000) 2019-10-13T10:41:14.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:14.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:14.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:14.336+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.340748+0000) 2019-10-13T10:41:15.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:15.125+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.130222+0000) 2019-10-13T10:41:15.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:15.760+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.763184+0000) 2019-10-13T10:41:16.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:16.345+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:16.348657+0000) 2019-10-13T10:41:18.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:18.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:18.611278+0000) 2019-10-13T10:41:18.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:18.613+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:18.616798+0000) 2019-10-13T10:41:19.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:19.337+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.340524+0000) 2019-10-13T10:41:19.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:19.336+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.340948+0000) 2019-10-13T10:41:19.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:19.337+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.341220+0000) 2019-10-13T10:41:19.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:19.337+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.342385+0000) 2019-10-13T10:41:19.344 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:19.337+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.341608+0000) 2019-10-13T10:41:19.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:19.337+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.342779+0000) 2019-10-13T10:41:19.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:19.337+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.343224+0000) 2019-10-13T10:41:19.347 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:19.341+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:19.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:19.344+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.349054+0000) 2019-10-13T10:41:21.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:21.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:21.911378+0000) 2019-10-13T10:41:22.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:22.097+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:22.102382+0000) 2019-10-13T10:41:23.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:23.096+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:23.098945+0000) 2019-10-13T10:41:24.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:24.345+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.349853+0000) 2019-10-13T10:41:24.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:24.345+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.350261+0000) 2019-10-13T10:41:24.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:24.345+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.350691+0000) 2019-10-13T10:41:24.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:24.345+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.351047+0000) 2019-10-13T10:41:24.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:24.348+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.352421+0000) 2019-10-13T10:41:24.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:24.349+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.351995+0000) 2019-10-13T10:41:24.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:24.349+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.352700+0000) 2019-10-13T10:41:24.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:24.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:24.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:24.356+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.360249+0000) 2019-10-13T10:41:25.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:25.125+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.130403+0000) 2019-10-13T10:41:25.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:25.768+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.769507+0000) 2019-10-13T10:41:26.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:26.345+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:26.348825+0000) 2019-10-13T10:41:28.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:28.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:28.611508+0000) 2019-10-13T10:41:28.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:28.613+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:28.617002+0000) 2019-10-13T10:41:29.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:29.353+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.357971+0000) 2019-10-13T10:41:29.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:29.353+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.358161+0000) 2019-10-13T10:41:29.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:29.356+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.358617+0000) 2019-10-13T10:41:29.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:29.353+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.359038+0000) 2019-10-13T10:41:29.361 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:29.357+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.359489+0000) 2019-10-13T10:41:29.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:29.357+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.359900+0000) 2019-10-13T10:41:29.363 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:29.357+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.361040+0000) 2019-10-13T10:41:29.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:29.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105797 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:29.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:29.364+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.368929+0000) 2019-10-13T10:41:31.590 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:41:31.590 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:31.594 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:41:31.594 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:31.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:31.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:31.911574+0000) 2019-10-13T10:41:32.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:32.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:32.102605+0000) 2019-10-13T10:41:33.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:33.095+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:33.099123+0000) 2019-10-13T10:41:34.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:34.361+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.366823+0000) 2019-10-13T10:41:34.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:34.365+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.367390+0000) 2019-10-13T10:41:34.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:34.365+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.367849+0000) 2019-10-13T10:41:34.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:34.365+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.368401+0000) 2019-10-13T10:41:34.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:34.363+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.368688+0000) 2019-10-13T10:41:34.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:34.365+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.369142+0000) 2019-10-13T10:41:34.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:34.365+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.370729+0000) 2019-10-13T10:41:34.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:34.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:34.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:34.371+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.376140+0000) 2019-10-13T10:41:35.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:35.125+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.130580+0000) 2019-10-13T10:41:35.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:35.771+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.775437+0000) 2019-10-13T10:41:36.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:36.345+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:36.348974+0000) 2019-10-13T10:41:38.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:38.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:38.611708+0000) 2019-10-13T10:41:38.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:38.613+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:38.617225+0000) 2019-10-13T10:41:39.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:39.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.374571+0000) 2019-10-13T10:41:39.376 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:39.369+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.374973+0000) 2019-10-13T10:41:39.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:39.373+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.375542+0000) 2019-10-13T10:41:39.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:39.373+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.376095+0000) 2019-10-13T10:41:39.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:39.373+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.376654+0000) 2019-10-13T10:41:39.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:39.373+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.377993+0000) 2019-10-13T10:41:39.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:39.373+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.378337+0000) 2019-10-13T10:41:39.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:39.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:39.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:39.379+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.382789+0000) 2019-10-13T10:41:41.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:41.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:41.911785+0000) 2019-10-13T10:41:42.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:42.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:42.102806+0000) 2019-10-13T10:41:43.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:43.095+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:43.099348+0000) 2019-10-13T10:41:44.384 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:44.377+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.382845+0000) 2019-10-13T10:41:44.384 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:44.379+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.383476+0000) 2019-10-13T10:41:44.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:44.381+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.383917+0000) 2019-10-13T10:41:44.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:44.381+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.384044+0000) 2019-10-13T10:41:44.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:44.381+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.384439+0000) 2019-10-13T10:41:44.386 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:44.381+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.385373+0000) 2019-10-13T10:41:44.387 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:44.381+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.386089+0000) 2019-10-13T10:41:44.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:44.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:44.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:44.387+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.393098+0000) 2019-10-13T10:41:45.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:45.129+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.130774+0000) 2019-10-13T10:41:45.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:45.779+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.781822+0000) 2019-10-13T10:41:46.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:46.349+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:46.349171+0000) 2019-10-13T10:41:48.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:48.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:48.611918+0000) 2019-10-13T10:41:48.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:48.617+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:48.617422+0000) 2019-10-13T10:41:49.393 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:49.388+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.391725+0000) 2019-10-13T10:41:49.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:49.388+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.392401+0000) 2019-10-13T10:41:49.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:49.388+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.392097+0000) 2019-10-13T10:41:49.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:49.387+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.392837+0000) 2019-10-13T10:41:49.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:49.388+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.394016+0000) 2019-10-13T10:41:49.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:49.388+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.394499+0000) 2019-10-13T10:41:49.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:49.388+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.394922+0000) 2019-10-13T10:41:49.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:49.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105933 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:49.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:49.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.399638+0000) 2019-10-13T10:41:51.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:51.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:51.912009+0000) 2019-10-13T10:41:52.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:52.100+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:52.103011+0000) 2019-10-13T10:41:53.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:53.095+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:53.099564+0000) 2019-10-13T10:41:54.401 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:54.396+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.399984+0000) 2019-10-13T10:41:54.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:54.395+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.400645+0000) 2019-10-13T10:41:54.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:54.396+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.401204+0000) 2019-10-13T10:41:54.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:54.396+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.401596+0000) 2019-10-13T10:41:54.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:54.396+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.401830+0000) 2019-10-13T10:41:54.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:54.396+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.402369+0000) 2019-10-13T10:41:54.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:54.400+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.403351+0000) 2019-10-13T10:41:54.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:54.400+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 105967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:54.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:54.407+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.410194+0000) 2019-10-13T10:41:55.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:55.124+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.130931+0000) 2019-10-13T10:41:55.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:55.783+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.788599+0000) 2019-10-13T10:41:56.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:56.344+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:56.349383+0000) 2019-10-13T10:41:58.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:58.608+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:58.612118+0000) 2019-10-13T10:41:58.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:58.616+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:58.617632+0000) 2019-10-13T10:41:59.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:41:59.404+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.408166+0000) 2019-10-13T10:41:59.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:41:59.404+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.408819+0000) 2019-10-13T10:41:59.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:41:59.404+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.409467+0000) 2019-10-13T10:41:59.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:41:59.404+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.409802+0000) 2019-10-13T10:41:59.411 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:41:59.403+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.410081+0000) 2019-10-13T10:41:59.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:41:59.404+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.410532+0000) 2019-10-13T10:41:59.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:41:59.404+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.411064+0000) 2019-10-13T10:41:59.415 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:41:59.408+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:41:59.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:41:59.415+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.418872+0000) 2019-10-13T10:42:01.814 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:42:01.814 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:01.817 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:42:01.817 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:01.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:01.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:01.912220+0000) 2019-10-13T10:42:02.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:02.096+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:02.103234+0000) 2019-10-13T10:42:03.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:03.099+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:03.099721+0000) 2019-10-13T10:42:04.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:04.412+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.417511+0000) 2019-10-13T10:42:04.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:04.415+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.418233+0000) 2019-10-13T10:42:04.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:04.412+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.418627+0000) 2019-10-13T10:42:04.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:04.412+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.419032+0000) 2019-10-13T10:42:04.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:04.416+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.419794+0000) 2019-10-13T10:42:04.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:04.416+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.420484+0000) 2019-10-13T10:42:04.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:04.416+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.420954+0000) 2019-10-13T10:42:04.424 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:04.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:04.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:04.423+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.428399+0000) 2019-10-13T10:42:05.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:05.128+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.131141+0000) 2019-10-13T10:42:05.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:05.791+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.795410+0000) 2019-10-13T10:42:06.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:06.348+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:06.349576+0000) 2019-10-13T10:42:08.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:08.608+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:08.612338+0000) 2019-10-13T10:42:08.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:08.612+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:08.617818+0000) 2019-10-13T10:42:09.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:09.420+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.426924+0000) 2019-10-13T10:42:09.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:09.423+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.427365+0000) 2019-10-13T10:42:09.429 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:09.424+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.427694+0000) 2019-10-13T10:42:09.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:09.424+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.428206+0000) 2019-10-13T10:42:09.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:09.424+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.428747+0000) 2019-10-13T10:42:09.430 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:09.424+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.429247+0000) 2019-10-13T10:42:09.431 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:09.424+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.429790+0000) 2019-10-13T10:42:09.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:09.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:09.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:09.431+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.435120+0000) 2019-10-13T10:42:11.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:11.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:11.912433+0000) 2019-10-13T10:42:12.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:12.100+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:12.103471+0000) 2019-10-13T10:42:13.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:13.099+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:13.099899+0000) 2019-10-13T10:42:14.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:14.432+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.435967+0000) 2019-10-13T10:42:14.437 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:14.432+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.436349+0000) 2019-10-13T10:42:14.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:14.432+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.436708+0000) 2019-10-13T10:42:14.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:14.432+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.437131+0000) 2019-10-13T10:42:14.439 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:14.432+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.437779+0000) 2019-10-13T10:42:14.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:14.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.438296+0000) 2019-10-13T10:42:14.440 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:14.432+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.438553+0000) 2019-10-13T10:42:14.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:14.436+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:14.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:14.439+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.445330+0000) 2019-10-13T10:42:15.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:15.128+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.131336+0000) 2019-10-13T10:42:15.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:15.799+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.801802+0000) 2019-10-13T10:42:16.351 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:16.344+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:16.349766+0000) 2019-10-13T10:42:18.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:18.608+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:18.612546+0000) 2019-10-13T10:42:18.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:18.612+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:18.618040+0000) 2019-10-13T10:42:19.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:19.440+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.444208+0000) 2019-10-13T10:42:19.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:19.439+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.444634+0000) 2019-10-13T10:42:19.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:19.440+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.445286+0000) 2019-10-13T10:42:19.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:19.440+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.445929+0000) 2019-10-13T10:42:19.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:19.440+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.446848+0000) 2019-10-13T10:42:19.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:19.440+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.447200+0000) 2019-10-13T10:42:19.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:19.444+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.447887+0000) 2019-10-13T10:42:19.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:19.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:19.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:19.447+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.452513+0000) 2019-10-13T10:42:21.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:21.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:21.912668+0000) 2019-10-13T10:42:22.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:22.100+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:22.103672+0000) 2019-10-13T10:42:23.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:23.095+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:23.100074+0000) 2019-10-13T10:42:24.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:24.448+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.452428+0000) 2019-10-13T10:42:24.455 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:24.448+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.454327+0000) 2019-10-13T10:42:24.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:24.448+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.454659+0000) 2019-10-13T10:42:24.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:24.448+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.454999+0000) 2019-10-13T10:42:24.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:24.451+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.455429+0000) 2019-10-13T10:42:24.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:24.452+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.456218+0000) 2019-10-13T10:42:24.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:24.452+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.456688+0000) 2019-10-13T10:42:24.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:24.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:24.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:24.455+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.460563+0000) 2019-10-13T10:42:25.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:25.128+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.131505+0000) 2019-10-13T10:42:25.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:25.811+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.813025+0000) 2019-10-13T10:42:26.351 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:26.348+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:26.349957+0000) 2019-10-13T10:42:28.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:28.608+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:28.612774+0000) 2019-10-13T10:42:28.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:28.612+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:28.618237+0000) 2019-10-13T10:42:29.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:29.456+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.460567+0000) 2019-10-13T10:42:29.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:29.456+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.461616+0000) 2019-10-13T10:42:29.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:29.456+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.462148+0000) 2019-10-13T10:42:29.463 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:29.456+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.462295+0000) 2019-10-13T10:42:29.464 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:29.456+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.462634+0000) 2019-10-13T10:42:29.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:29.456+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.463218+0000) 2019-10-13T10:42:29.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:29.458+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.463810+0000) 2019-10-13T10:42:29.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:29.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106205 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:29.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:29.466+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.470753+0000) 2019-10-13T10:42:31.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:31.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:31.912882+0000) 2019-10-13T10:42:31.920 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:42:31.921 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:31.924 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:42:31.924 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:32.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:32.100+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:32.103849+0000) 2019-10-13T10:42:33.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:33.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:33.100269+0000) 2019-10-13T10:42:34.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:34.463+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.470109+0000) 2019-10-13T10:42:34.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:34.463+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.470707+0000) 2019-10-13T10:42:34.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:34.467+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.471640+0000) 2019-10-13T10:42:34.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:34.467+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.471587+0000) 2019-10-13T10:42:34.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:34.467+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.471904+0000) 2019-10-13T10:42:34.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:34.467+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.473044+0000) 2019-10-13T10:42:34.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:34.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.473769+0000) 2019-10-13T10:42:34.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:34.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:34.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:34.474+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.479287+0000) 2019-10-13T10:42:35.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:35.127+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.131646+0000) 2019-10-13T10:42:35.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:35.822+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.824742+0000) 2019-10-13T10:42:36.351 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:36.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:36.350193+0000) 2019-10-13T10:42:38.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:38.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:38.613004+0000) 2019-10-13T10:42:38.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:38.611+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:38.618436+0000) 2019-10-13T10:42:39.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:39.475+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.479634+0000) 2019-10-13T10:42:39.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:39.475+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.480118+0000) 2019-10-13T10:42:39.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:39.475+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.480387+0000) 2019-10-13T10:42:39.482 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:39.475+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.480877+0000) 2019-10-13T10:42:39.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:39.475+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.481743+0000) 2019-10-13T10:42:39.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:39.478+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.482790+0000) 2019-10-13T10:42:39.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:39.475+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.482916+0000) 2019-10-13T10:42:39.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:39.479+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:39.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:39.482+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.489032+0000) 2019-10-13T10:42:41.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:41.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:41.913078+0000) 2019-10-13T10:42:42.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:42.099+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:42.104084+0000) 2019-10-13T10:42:43.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:43.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:43.100482+0000) 2019-10-13T10:42:44.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:44.483+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.487503+0000) 2019-10-13T10:42:44.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:44.483+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.487986+0000) 2019-10-13T10:42:44.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:44.483+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.488432+0000) 2019-10-13T10:42:44.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:44.483+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.488865+0000) 2019-10-13T10:42:44.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:44.483+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.489346+0000) 2019-10-13T10:42:44.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:44.486+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.491141+0000) 2019-10-13T10:42:44.492 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:44.483+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.491269+0000) 2019-10-13T10:42:44.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:44.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106306 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:44.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:44.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.497239+0000) 2019-10-13T10:42:45.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:45.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.131841+0000) 2019-10-13T10:42:45.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:45.834+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.835465+0000) 2019-10-13T10:42:46.351 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:46.343+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:46.350383+0000) 2019-10-13T10:42:48.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:48.607+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:48.613220+0000) 2019-10-13T10:42:48.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:48.615+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:48.618636+0000) 2019-10-13T10:42:49.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:49.491+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.496448+0000) 2019-10-13T10:42:49.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:49.491+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.497036+0000) 2019-10-13T10:42:49.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:49.491+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.497361+0000) 2019-10-13T10:42:49.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:49.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.497867+0000) 2019-10-13T10:42:49.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:49.491+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.498526+0000) 2019-10-13T10:42:49.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:49.494+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.499576+0000) 2019-10-13T10:42:49.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:49.495+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.499891+0000) 2019-10-13T10:42:49.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:49.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:49.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:49.498+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.505087+0000) 2019-10-13T10:42:51.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:51.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:51.913294+0000) 2019-10-13T10:42:52.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:52.099+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:52.104302+0000) 2019-10-13T10:42:53.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:53.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:53.100692+0000) 2019-10-13T10:42:54.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:54.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.504896+0000) 2019-10-13T10:42:54.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:54.499+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.505723+0000) 2019-10-13T10:42:54.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:54.499+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.506132+0000) 2019-10-13T10:42:54.508 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:54.499+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.506888+0000) 2019-10-13T10:42:54.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:54.503+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.507437+0000) 2019-10-13T10:42:54.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:54.503+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.507673+0000) 2019-10-13T10:42:54.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:54.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.508328+0000) 2019-10-13T10:42:54.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:54.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:54.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:54.510+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.514770+0000) 2019-10-13T10:42:55.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:55.127+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.132037+0000) 2019-10-13T10:42:55.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:55.846+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.846907+0000) 2019-10-13T10:42:56.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:56.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:56.350595+0000) 2019-10-13T10:42:58.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:58.611+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:58.613417+0000) 2019-10-13T10:42:58.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:58.615+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:58.618833+0000) 2019-10-13T10:42:59.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:42:59.507+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.513572+0000) 2019-10-13T10:42:59.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:42:59.507+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.513874+0000) 2019-10-13T10:42:59.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:42:59.507+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.514508+0000) 2019-10-13T10:42:59.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:42:59.507+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.514780+0000) 2019-10-13T10:42:59.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:42:59.511+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.515750+0000) 2019-10-13T10:42:59.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:42:59.511+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.516310+0000) 2019-10-13T10:42:59.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:42:59.510+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.516733+0000) 2019-10-13T10:42:59.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:42:59.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:42:59.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:42:59.514+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.521833+0000) 2019-10-13T10:43:01.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:01.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:01.913538+0000) 2019-10-13T10:43:02.015 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:43:02.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:02.019 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:43:02.019 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:02.105 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:02.103+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:02.104506+0000) 2019-10-13T10:43:03.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:03.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:03.100921+0000) 2019-10-13T10:43:04.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:04.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.521174+0000) 2019-10-13T10:43:04.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:04.514+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.521931+0000) 2019-10-13T10:43:04.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:04.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.521678+0000) 2019-10-13T10:43:04.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:04.519+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.522375+0000) 2019-10-13T10:43:04.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:04.519+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.523238+0000) 2019-10-13T10:43:04.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:04.523+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.524112+0000) 2019-10-13T10:43:04.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:04.523+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.524591+0000) 2019-10-13T10:43:04.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:04.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106442 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:04.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:04.526+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.531151+0000) 2019-10-13T10:43:05.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:05.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:05.132204+0000) 2019-10-13T10:43:05.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:05.850+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:05.853267+0000) 2019-10-13T10:43:06.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:06.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.350788+0000) 2019-10-13T10:43:08.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:08.611+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:08.613587+0000) 2019-10-13T10:43:08.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:08.615+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:08.619089+0000) 2019-10-13T10:43:09.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:09.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.529891+0000) 2019-10-13T10:43:09.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:09.527+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.529984+0000) 2019-10-13T10:43:09.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:09.527+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.531213+0000) 2019-10-13T10:43:09.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:09.526+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.532115+0000) 2019-10-13T10:43:09.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:09.531+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.532414+0000) 2019-10-13T10:43:09.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:09.531+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.532810+0000) 2019-10-13T10:43:09.534 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:09.531+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.533341+0000) 2019-10-13T10:43:09.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:09.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:09.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:09.530+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.537880+0000) 2019-10-13T10:43:11.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:11.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.913760+0000) 2019-10-13T10:43:12.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:12.103+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:12.104683+0000) 2019-10-13T10:43:13.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:13.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:13.101117+0000) 2019-10-13T10:43:14.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:14.539+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.539286+0000) 2019-10-13T10:43:14.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:14.539+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.539443+0000) 2019-10-13T10:43:14.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:14.539+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.540331+0000) 2019-10-13T10:43:14.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:14.534+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.541012+0000) 2019-10-13T10:43:14.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:14.539+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.541176+0000) 2019-10-13T10:43:14.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:14.539+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.541811+0000) 2019-10-13T10:43:14.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:14.539+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.542081+0000) 2019-10-13T10:43:14.545 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:14.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:14.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:14.550+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.550440+0000) 2019-10-13T10:43:15.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:15.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:15.132355+0000) 2019-10-13T10:43:15.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:15.862+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:15.864900+0000) 2019-10-13T10:43:16.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:16.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.350964+0000) 2019-10-13T10:43:18.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:18.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:18.613827+0000) 2019-10-13T10:43:18.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:18.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:18.619338+0000) 2019-10-13T10:43:19.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:19.546+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.546950+0000) 2019-10-13T10:43:19.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:19.546+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.547657+0000) 2019-10-13T10:43:19.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:19.546+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.548612+0000) 2019-10-13T10:43:19.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:19.546+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.549108+0000) 2019-10-13T10:43:19.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:19.546+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.548807+0000) 2019-10-13T10:43:19.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:19.546+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.549986+0000) 2019-10-13T10:43:19.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:19.546+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.550726+0000) 2019-10-13T10:43:19.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:19.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:19.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:19.554+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.554433+0000) 2019-10-13T10:43:21.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:21.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.914003+0000) 2019-10-13T10:43:22.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:22.102+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:22.104874+0000) 2019-10-13T10:43:23.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:23.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:23.101314+0000) 2019-10-13T10:43:24.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:24.554+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.555689+0000) 2019-10-13T10:43:24.558 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:24.554+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.556580+0000) 2019-10-13T10:43:24.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:24.554+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.557009+0000) 2019-10-13T10:43:24.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:24.554+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.557411+0000) 2019-10-13T10:43:24.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:24.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.557823+0000) 2019-10-13T10:43:24.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:24.554+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.558019+0000) 2019-10-13T10:43:24.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:24.554+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.558583+0000) 2019-10-13T10:43:24.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:24.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:24.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:24.561+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.562838+0000) 2019-10-13T10:43:25.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:25.130+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:25.132543+0000) 2019-10-13T10:43:25.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:25.869+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:25.871302+0000) 2019-10-13T10:43:26.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:26.350+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.351143+0000) 2019-10-13T10:43:28.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:28.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:28.614033+0000) 2019-10-13T10:43:28.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:28.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:28.619596+0000) 2019-10-13T10:43:29.564 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:29.562+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.563298+0000) 2019-10-13T10:43:29.565 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:29.561+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.564059+0000) 2019-10-13T10:43:29.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:29.562+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.564789+0000) 2019-10-13T10:43:29.566 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:29.562+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.565604+0000) 2019-10-13T10:43:29.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:29.562+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.565747+0000) 2019-10-13T10:43:29.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:29.562+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.566530+0000) 2019-10-13T10:43:29.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:29.562+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.567174+0000) 2019-10-13T10:43:29.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:29.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:29.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:29.569+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.572539+0000) 2019-10-13T10:43:31.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:31.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.914271+0000) 2019-10-13T10:43:32.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:32.102+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:32.105105+0000) 2019-10-13T10:43:32.127 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:43:32.127 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:32.131 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:43:32.131 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:33.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:33.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:33.101538+0000) 2019-10-13T10:43:34.573 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:34.569+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.572421+0000) 2019-10-13T10:43:34.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:34.570+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.572622+0000) 2019-10-13T10:43:34.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:34.570+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.573256+0000) 2019-10-13T10:43:34.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:34.570+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.573420+0000) 2019-10-13T10:43:34.575 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:34.570+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.573924+0000) 2019-10-13T10:43:34.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:34.570+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.573669+0000) 2019-10-13T10:43:34.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:34.570+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.574655+0000) 2019-10-13T10:43:34.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:34.574+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:34.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:34.577+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.579405+0000) 2019-10-13T10:43:35.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:35.130+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:35.132748+0000) 2019-10-13T10:43:35.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:35.873+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:35.877909+0000) 2019-10-13T10:43:36.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:36.350+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.351346+0000) 2019-10-13T10:43:38.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:38.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:38.614241+0000) 2019-10-13T10:43:38.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:38.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:38.619818+0000) 2019-10-13T10:43:39.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:39.578+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.580591+0000) 2019-10-13T10:43:39.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:39.578+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.580928+0000) 2019-10-13T10:43:39.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:39.581+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.582590+0000) 2019-10-13T10:43:39.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:39.578+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.582867+0000) 2019-10-13T10:43:39.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:39.582+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.583376+0000) 2019-10-13T10:43:39.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:39.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.583592+0000) 2019-10-13T10:43:39.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:39.582+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.583850+0000) 2019-10-13T10:43:39.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:39.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:39.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:39.585+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.588942+0000) 2019-10-13T10:43:41.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:41.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.914479+0000) 2019-10-13T10:43:42.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:42.102+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:42.105298+0000) 2019-10-13T10:43:43.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:43.101+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:43.101757+0000) 2019-10-13T10:43:44.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:44.585+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.589349+0000) 2019-10-13T10:43:44.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:44.586+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.589807+0000) 2019-10-13T10:43:44.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:44.586+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.590133+0000) 2019-10-13T10:43:44.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:44.586+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.590409+0000) 2019-10-13T10:43:44.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:44.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.591003+0000) 2019-10-13T10:43:44.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:44.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.592181+0000) 2019-10-13T10:43:44.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:44.590+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.592560+0000) 2019-10-13T10:43:44.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:44.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:44.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:44.593+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.597734+0000) 2019-10-13T10:43:45.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:45.130+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.132920+0000) 2019-10-13T10:43:45.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:45.881+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.884325+0000) 2019-10-13T10:43:46.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:46.350+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.351549+0000) 2019-10-13T10:43:48.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:48.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:48.614437+0000) 2019-10-13T10:43:48.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:48.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:48.620012+0000) 2019-10-13T10:43:49.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:49.594+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.597852+0000) 2019-10-13T10:43:49.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:49.594+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.598522+0000) 2019-10-13T10:43:49.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:49.594+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.598901+0000) 2019-10-13T10:43:49.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:49.597+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.599760+0000) 2019-10-13T10:43:49.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:49.594+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.599232+0000) 2019-10-13T10:43:49.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:49.598+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.600565+0000) 2019-10-13T10:43:49.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:49.598+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.600952+0000) 2019-10-13T10:43:49.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:49.602+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106749 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:49.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:49.605+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:49.607044+0000) 2019-10-13T10:43:51.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:51.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.914708+0000) 2019-10-13T10:43:52.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:52.102+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:52.105506+0000) 2019-10-13T10:43:53.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:53.097+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:53.101956+0000) 2019-10-13T10:43:54.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:54.602+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.606610+0000) 2019-10-13T10:43:54.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:54.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.606967+0000) 2019-10-13T10:43:54.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:54.606+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.608924+0000) 2019-10-13T10:43:54.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:54.605+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.609323+0000) 2019-10-13T10:43:54.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:54.606+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.609546+0000) 2019-10-13T10:43:54.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:54.606+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.609917+0000) 2019-10-13T10:43:54.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:54.606+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.610661+0000) 2019-10-13T10:43:54.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:54.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:54.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:54.617+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.618373+0000) 2019-10-13T10:43:55.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:55.130+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.133095+0000) 2019-10-13T10:43:55.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:55.889+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.891147+0000) 2019-10-13T10:43:56.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:56.350+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.351742+0000) 2019-10-13T10:43:58.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:58.610+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:58.614661+0000) 2019-10-13T10:43:58.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:58.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:58.620235+0000) 2019-10-13T10:43:59.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:43:59.610+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.614700+0000) 2019-10-13T10:43:59.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:43:59.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.616076+0000) 2019-10-13T10:43:59.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:43:59.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.616168+0000) 2019-10-13T10:43:59.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:43:59.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.616865+0000) 2019-10-13T10:43:59.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:43:59.614+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.617234+0000) 2019-10-13T10:43:59.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:43:59.617+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.618715+0000) 2019-10-13T10:43:59.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:43:59.614+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.619037+0000) 2019-10-13T10:43:59.621 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:43:59.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106816 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:43:59.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:43:59.621+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:59.622271+0000) 2019-10-13T10:44:01.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:01.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.914911+0000) 2019-10-13T10:44:02.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:02.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:02.105737+0000) 2019-10-13T10:44:02.246 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:44:02.246 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:02.250 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:44:02.250 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:03.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:03.101+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:03.102179+0000) 2019-10-13T10:44:04.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:04.621+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.623855+0000) 2019-10-13T10:44:04.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:04.621+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.624534+0000) 2019-10-13T10:44:04.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:04.621+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.624873+0000) 2019-10-13T10:44:04.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:04.621+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.625206+0000) 2019-10-13T10:44:04.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:04.621+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.625612+0000) 2019-10-13T10:44:04.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:04.621+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.625863+0000) 2019-10-13T10:44:04.627 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:04.621+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.626458+0000) 2019-10-13T10:44:04.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:04.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:04.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:04.629+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.630426+0000) 2019-10-13T10:44:05.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:05.129+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.133275+0000) 2019-10-13T10:44:05.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:05.897+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.897576+0000) 2019-10-13T10:44:06.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:06.349+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.351953+0000) 2019-10-13T10:44:08.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:08.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:08.614862+0000) 2019-10-13T10:44:08.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:08.617+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:08.620427+0000) 2019-10-13T10:44:09.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:09.629+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.631887+0000) 2019-10-13T10:44:09.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:09.629+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.632499+0000) 2019-10-13T10:44:09.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:09.629+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.632751+0000) 2019-10-13T10:44:09.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:09.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.633422+0000) 2019-10-13T10:44:09.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:09.629+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.633949+0000) 2019-10-13T10:44:09.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:09.629+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.634413+0000) 2019-10-13T10:44:09.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:09.629+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.634703+0000) 2019-10-13T10:44:09.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:09.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:09.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:09.641+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:09.642663+0000) 2019-10-13T10:44:11.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:11.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.915108+0000) 2019-10-13T10:44:12.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:12.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:12.105902+0000) 2019-10-13T10:44:13.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:13.101+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:13.102401+0000) 2019-10-13T10:44:14.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:14.637+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.640768+0000) 2019-10-13T10:44:14.642 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:14.637+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.641114+0000) 2019-10-13T10:44:14.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:14.637+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.641862+0000) 2019-10-13T10:44:14.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:14.637+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.642220+0000) 2019-10-13T10:44:14.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:14.637+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.642533+0000) 2019-10-13T10:44:14.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:14.641+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.643874+0000) 2019-10-13T10:44:14.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:14.641+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.645389+0000) 2019-10-13T10:44:14.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:14.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:14.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:14.645+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.647790+0000) 2019-10-13T10:44:15.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:15.129+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.133474+0000) 2019-10-13T10:44:15.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:15.901+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.904113+0000) 2019-10-13T10:44:16.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:16.349+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.352156+0000) 2019-10-13T10:44:18.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:18.609+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:18.615095+0000) 2019-10-13T10:44:18.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:18.617+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:18.620618+0000) 2019-10-13T10:44:19.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:19.645+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.649705+0000) 2019-10-13T10:44:19.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:19.645+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.650325+0000) 2019-10-13T10:44:19.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:19.645+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.650739+0000) 2019-10-13T10:44:19.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:19.645+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.651026+0000) 2019-10-13T10:44:19.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:19.648+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.651627+0000) 2019-10-13T10:44:19.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:19.649+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.652275+0000) 2019-10-13T10:44:19.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:19.649+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.652756+0000) 2019-10-13T10:44:19.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:19.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:19.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:19.660+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:19.663615+0000) 2019-10-13T10:44:21.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:21.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.915336+0000) 2019-10-13T10:44:22.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:22.101+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:22.106124+0000) 2019-10-13T10:44:23.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:23.100+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:23.102603+0000) 2019-10-13T10:44:24.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:24.657+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.659457+0000) 2019-10-13T10:44:24.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:24.657+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.659742+0000) 2019-10-13T10:44:24.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:24.656+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.658419+0000) 2019-10-13T10:44:24.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:24.657+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.660333+0000) 2019-10-13T10:44:24.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:24.657+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.660417+0000) 2019-10-13T10:44:24.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:24.657+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.660760+0000) 2019-10-13T10:44:24.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:24.657+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.661397+0000) 2019-10-13T10:44:24.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:24.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 106985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:24.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:24.660+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.665118+0000) 2019-10-13T10:44:25.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:25.129+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.133664+0000) 2019-10-13T10:44:25.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:25.908+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.910566+0000) 2019-10-13T10:44:26.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:26.349+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.352359+0000) 2019-10-13T10:44:28.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:28.613+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:28.615348+0000) 2019-10-13T10:44:28.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:28.617+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:28.620851+0000) 2019-10-13T10:44:29.668 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:29.661+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.667009+0000) 2019-10-13T10:44:29.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:29.665+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.667436+0000) 2019-10-13T10:44:29.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:29.665+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.667994+0000) 2019-10-13T10:44:29.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:29.665+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.668374+0000) 2019-10-13T10:44:29.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:29.664+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.669122+0000) 2019-10-13T10:44:29.671 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:29.665+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.669602+0000) 2019-10-13T10:44:29.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:29.665+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.669681+0000) 2019-10-13T10:44:29.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:29.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:29.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:29.672+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:29.675485+0000) 2019-10-13T10:44:31.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:31.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.915540+0000) 2019-10-13T10:44:32.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:32.105+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:32.106324+0000) 2019-10-13T10:44:32.389 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:44:32.390 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:32.395 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:44:32.395 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:33.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:33.100+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:33.102801+0000) 2019-10-13T10:44:34.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:34.669+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.674754+0000) 2019-10-13T10:44:34.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:34.669+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.675119+0000) 2019-10-13T10:44:34.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:34.672+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.675725+0000) 2019-10-13T10:44:34.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:34.673+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.676598+0000) 2019-10-13T10:44:34.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:34.673+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.676795+0000) 2019-10-13T10:44:34.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:34.673+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.677279+0000) 2019-10-13T10:44:34.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:34.673+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.679040+0000) 2019-10-13T10:44:34.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:34.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:34.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:34.680+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.682615+0000) 2019-10-13T10:44:35.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:35.129+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.133836+0000) 2019-10-13T10:44:35.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:35.916+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.916972+0000) 2019-10-13T10:44:36.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:36.349+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.352553+0000) 2019-10-13T10:44:38.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:38.613+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:38.615577+0000) 2019-10-13T10:44:38.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:38.617+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:38.621051+0000) 2019-10-13T10:44:39.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:39.677+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.682784+0000) 2019-10-13T10:44:39.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:39.681+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.683818+0000) 2019-10-13T10:44:39.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:39.681+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.684589+0000) 2019-10-13T10:44:39.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:39.681+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.684452+0000) 2019-10-13T10:44:39.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:39.681+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.685442+0000) 2019-10-13T10:44:39.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:39.684+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.686501+0000) 2019-10-13T10:44:39.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:39.685+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.687379+0000) 2019-10-13T10:44:39.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:39.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:39.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:39.688+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:39.691450+0000) 2019-10-13T10:44:41.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:41.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.915774+0000) 2019-10-13T10:44:42.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:42.105+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:42.106526+0000) 2019-10-13T10:44:43.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:43.100+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:43.103010+0000) 2019-10-13T10:44:44.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:44.688+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.691338+0000) 2019-10-13T10:44:44.693 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:44.688+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.692279+0000) 2019-10-13T10:44:44.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:44.688+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.692505+0000) 2019-10-13T10:44:44.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:44.688+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.692766+0000) 2019-10-13T10:44:44.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:44.688+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.693546+0000) 2019-10-13T10:44:44.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:44.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.694686+0000) 2019-10-13T10:44:44.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:44.688+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.694986+0000) 2019-10-13T10:44:44.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:44.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:44.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:44.696+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.700711+0000) 2019-10-13T10:44:45.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:45.128+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.134011+0000) 2019-10-13T10:44:45.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:45.920+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.923497+0000) 2019-10-13T10:44:46.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:46.348+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.352699+0000) 2019-10-13T10:44:48.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:48.612+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:48.615779+0000) 2019-10-13T10:44:48.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:48.616+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:48.621272+0000) 2019-10-13T10:44:49.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:49.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.699976+0000) 2019-10-13T10:44:49.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:49.696+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.700334+0000) 2019-10-13T10:44:49.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:49.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.701981+0000) 2019-10-13T10:44:49.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:49.696+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.702385+0000) 2019-10-13T10:44:49.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:49.696+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.702834+0000) 2019-10-13T10:44:49.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:49.696+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.703267+0000) 2019-10-13T10:44:49.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:49.696+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.701542+0000) 2019-10-13T10:44:49.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:49.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:49.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:49.708+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:49.710305+0000) 2019-10-13T10:44:51.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:51.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.916026+0000) 2019-10-13T10:44:52.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:52.104+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:52.106727+0000) 2019-10-13T10:44:53.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:53.100+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:53.103200+0000) 2019-10-13T10:44:54.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:54.704+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.709658+0000) 2019-10-13T10:44:54.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:54.704+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.710409+0000) 2019-10-13T10:44:54.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:54.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.711076+0000) 2019-10-13T10:44:54.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:54.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.711488+0000) 2019-10-13T10:44:54.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:54.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.711921+0000) 2019-10-13T10:44:54.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:54.708+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.712164+0000) 2019-10-13T10:44:54.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:54.708+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.713078+0000) 2019-10-13T10:44:54.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:54.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:54.716 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:54.712+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.714957+0000) 2019-10-13T10:44:55.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:55.132+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.134188+0000) 2019-10-13T10:44:55.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:55.928+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.930024+0000) 2019-10-13T10:44:56.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:56.348+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.352904+0000) 2019-10-13T10:44:58.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:58.612+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:58.615976+0000) 2019-10-13T10:44:58.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:58.620+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:58.621465+0000) 2019-10-13T10:44:59.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:44:59.712+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.717297+0000) 2019-10-13T10:44:59.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:44:59.712+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.717781+0000) 2019-10-13T10:44:59.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:44:59.712+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.719047+0000) 2019-10-13T10:44:59.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:44:59.716+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.719675+0000) 2019-10-13T10:44:59.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:44:59.716+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.720151+0000) 2019-10-13T10:44:59.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:44:59.716+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.720525+0000) 2019-10-13T10:44:59.722 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:44:59.716+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.721419+0000) 2019-10-13T10:44:59.723 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:44:59.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:44:59.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:44:59.720+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:59.724979+0000) 2019-10-13T10:45:01.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:01.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.916245+0000) 2019-10-13T10:45:02.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:02.100+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:02.106930+0000) 2019-10-13T10:45:02.530 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:45:02.530 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:02.534 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:45:02.534 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:03.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:03.100+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:03.103408+0000) 2019-10-13T10:45:04.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:04.720+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.726489+0000) 2019-10-13T10:45:04.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:04.720+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.726724+0000) 2019-10-13T10:45:04.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:04.724+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.727474+0000) 2019-10-13T10:45:04.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:04.724+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.728072+0000) 2019-10-13T10:45:04.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:04.724+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.728553+0000) 2019-10-13T10:45:04.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:04.724+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.729144+0000) 2019-10-13T10:45:04.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:04.724+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.729469+0000) 2019-10-13T10:45:04.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:04.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:04.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:04.732+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.737319+0000) 2019-10-13T10:45:05.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:05.132+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.134383+0000) 2019-10-13T10:45:05.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:05.932+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.936653+0000) 2019-10-13T10:45:06.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:06.352+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.353098+0000) 2019-10-13T10:45:08.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:08.612+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:08.616175+0000) 2019-10-13T10:45:08.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:08.620+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:08.621656+0000) 2019-10-13T10:45:09.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:09.728+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.734176+0000) 2019-10-13T10:45:09.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:09.732+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.734586+0000) 2019-10-13T10:45:09.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:09.728+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.734935+0000) 2019-10-13T10:45:09.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:09.732+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.735591+0000) 2019-10-13T10:45:09.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:09.732+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.735943+0000) 2019-10-13T10:45:09.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:09.732+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.736123+0000) 2019-10-13T10:45:09.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:09.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.736752+0000) 2019-10-13T10:45:09.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:09.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:09.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:09.740+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:09.743204+0000) 2019-10-13T10:45:11.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:11.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.916463+0000) 2019-10-13T10:45:12.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:12.104+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:12.107145+0000) 2019-10-13T10:45:13.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:13.099+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:13.103648+0000) 2019-10-13T10:45:14.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:14.736+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.741825+0000) 2019-10-13T10:45:14.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:14.736+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.742609+0000) 2019-10-13T10:45:14.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:14.736+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.743131+0000) 2019-10-13T10:45:14.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:14.740+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.743963+0000) 2019-10-13T10:45:14.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:14.739+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.744558+0000) 2019-10-13T10:45:14.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:14.740+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.744964+0000) 2019-10-13T10:45:14.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:14.740+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.745264+0000) 2019-10-13T10:45:14.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:14.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:14.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:14.743+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.749891+0000) 2019-10-13T10:45:15.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:15.128+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.134559+0000) 2019-10-13T10:45:15.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:15.939+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.942938+0000) 2019-10-13T10:45:16.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:16.352+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.353325+0000) 2019-10-13T10:45:18.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:18.612+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:18.616336+0000) 2019-10-13T10:45:18.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:18.620+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:18.621876+0000) 2019-10-13T10:45:19.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:19.744+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.749897+0000) 2019-10-13T10:45:19.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:19.744+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.750566+0000) 2019-10-13T10:45:19.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:19.744+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.750822+0000) 2019-10-13T10:45:19.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:19.748+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.752193+0000) 2019-10-13T10:45:19.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:19.747+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.752554+0000) 2019-10-13T10:45:19.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:19.748+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.753150+0000) 2019-10-13T10:45:19.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:19.748+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.752803+0000) 2019-10-13T10:45:19.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:19.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:19.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:19.755+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:19.761565+0000) 2019-10-13T10:45:21.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:21.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.916701+0000) 2019-10-13T10:45:22.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:22.104+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:22.107377+0000) 2019-10-13T10:45:23.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:23.099+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:23.103842+0000) 2019-10-13T10:45:24.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:24.752+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.758670+0000) 2019-10-13T10:45:24.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:24.755+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.759385+0000) 2019-10-13T10:45:24.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:24.756+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.759297+0000) 2019-10-13T10:45:24.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:24.756+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.760141+0000) 2019-10-13T10:45:24.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:24.756+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.760824+0000) 2019-10-13T10:45:24.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:24.756+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.761086+0000) 2019-10-13T10:45:24.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:24.756+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.761424+0000) 2019-10-13T10:45:24.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:24.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:24.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:24.763+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.768916+0000) 2019-10-13T10:45:25.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:25.132+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.134732+0000) 2019-10-13T10:45:25.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:25.947+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.949319+0000) 2019-10-13T10:45:26.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:26.352+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.353581+0000) 2019-10-13T10:45:28.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:28.615+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:28.616537+0000) 2019-10-13T10:45:28.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:28.619+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:28.622090+0000) 2019-10-13T10:45:29.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:29.763+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.767651+0000) 2019-10-13T10:45:29.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:29.763+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.768219+0000) 2019-10-13T10:45:29.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:29.763+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.768279+0000) 2019-10-13T10:45:29.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:29.763+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.769513+0000) 2019-10-13T10:45:29.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:29.763+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.769678+0000) 2019-10-13T10:45:29.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:29.767+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.770992+0000) 2019-10-13T10:45:29.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:29.763+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.771231+0000) 2019-10-13T10:45:29.774 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:29.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:29.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:29.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:29.776197+0000) 2019-10-13T10:45:31.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:31.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.916909+0000) 2019-10-13T10:45:32.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:32.103+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:32.107581+0000) 2019-10-13T10:45:32.595 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:45:32.595 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:32.599 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:45:32.599 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:33.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:33.103+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:33.104037+0000) 2019-10-13T10:45:34.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:34.771+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.776959+0000) 2019-10-13T10:45:34.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:34.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.777266+0000) 2019-10-13T10:45:34.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:34.771+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.777933+0000) 2019-10-13T10:45:34.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:34.771+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.777990+0000) 2019-10-13T10:45:34.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:34.771+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.778867+0000) 2019-10-13T10:45:34.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:34.771+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.779207+0000) 2019-10-13T10:45:34.781 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:34.775+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.780107+0000) 2019-10-13T10:45:34.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:34.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:34.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:34.779+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.785972+0000) 2019-10-13T10:45:35.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:35.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.134916+0000) 2019-10-13T10:45:35.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:35.955+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.955713+0000) 2019-10-13T10:45:36.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:36.347+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.353782+0000) 2019-10-13T10:45:38.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:38.615+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:38.616753+0000) 2019-10-13T10:45:38.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:38.619+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:38.622290+0000) 2019-10-13T10:45:39.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:39.779+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.786010+0000) 2019-10-13T10:45:39.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:39.779+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.786305+0000) 2019-10-13T10:45:39.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:39.779+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.786661+0000) 2019-10-13T10:45:39.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:39.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.787981+0000) 2019-10-13T10:45:39.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:39.783+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.788622+0000) 2019-10-13T10:45:39.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:39.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.788417+0000) 2019-10-13T10:45:39.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:39.783+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.789225+0000) 2019-10-13T10:45:39.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:39.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:39.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:39.787+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:39.793556+0000) 2019-10-13T10:45:41.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:41.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.917110+0000) 2019-10-13T10:45:42.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:42.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:42.107787+0000) 2019-10-13T10:45:43.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:43.103+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:43.104230+0000) 2019-10-13T10:45:44.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:44.791+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.795142+0000) 2019-10-13T10:45:44.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:44.791+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.795756+0000) 2019-10-13T10:45:44.797 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:44.791+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.796812+0000) 2019-10-13T10:45:44.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:44.791+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.797114+0000) 2019-10-13T10:45:44.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:44.791+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.797838+0000) 2019-10-13T10:45:44.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:44.791+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.798589+0000) 2019-10-13T10:45:44.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:44.791+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.798998+0000) 2019-10-13T10:45:44.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:44.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:44.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:44.799+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.803903+0000) 2019-10-13T10:45:45.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:45.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.135086+0000) 2019-10-13T10:45:45.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:45.959+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.962653+0000) 2019-10-13T10:45:46.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:46.351+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.353974+0000) 2019-10-13T10:45:48.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:48.615+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:48.616952+0000) 2019-10-13T10:45:48.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:48.619+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:48.622490+0000) 2019-10-13T10:45:49.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:49.799+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.804470+0000) 2019-10-13T10:45:49.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:49.799+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.804775+0000) 2019-10-13T10:45:49.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:49.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.805386+0000) 2019-10-13T10:45:49.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:49.799+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.806021+0000) 2019-10-13T10:45:49.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:49.799+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.806764+0000) 2019-10-13T10:45:49.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:49.799+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.806449+0000) 2019-10-13T10:45:49.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:49.803+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.807284+0000) 2019-10-13T10:45:49.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:49.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:49.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:49.803+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:49.810040+0000) 2019-10-13T10:45:51.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:51.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.917326+0000) 2019-10-13T10:45:52.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:52.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:52.108008+0000) 2019-10-13T10:45:53.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:53.103+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:53.104422+0000) 2019-10-13T10:45:54.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:54.807+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.812549+0000) 2019-10-13T10:45:54.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:54.807+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.813156+0000) 2019-10-13T10:45:54.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:54.807+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.813576+0000) 2019-10-13T10:45:54.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:54.807+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.813901+0000) 2019-10-13T10:45:54.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:54.807+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.814846+0000) 2019-10-13T10:45:54.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:54.811+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.816283+0000) 2019-10-13T10:45:54.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:54.811+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.815817+0000) 2019-10-13T10:45:54.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:54.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:54.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:54.815+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.821646+0000) 2019-10-13T10:45:55.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:55.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.135331+0000) 2019-10-13T10:45:55.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:55.967+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.970854+0000) 2019-10-13T10:45:56.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:56.351+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.354214+0000) 2019-10-13T10:45:58.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:58.615+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:58.617149+0000) 2019-10-13T10:45:58.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:58.619+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:58.622686+0000) 2019-10-13T10:45:59.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:45:59.819+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.820957+0000) 2019-10-13T10:45:59.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:45:59.819+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.822088+0000) 2019-10-13T10:45:59.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:45:59.819+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.822437+0000) 2019-10-13T10:45:59.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:45:59.823+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.823389+0000) 2019-10-13T10:45:59.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:45:59.823+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.823733+0000) 2019-10-13T10:45:59.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:45:59.823+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.824348+0000) 2019-10-13T10:45:59.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:45:59.823+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.824916+0000) 2019-10-13T10:45:59.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:45:59.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:45:59.833 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:45:59.827+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:59.830839+0000) 2019-10-13T10:46:01.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:01.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.917527+0000) 2019-10-13T10:46:02.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:02.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:02.108210+0000) 2019-10-13T10:46:02.652 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:46:02.652 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:02.656 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:46:02.656 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:03.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:03.103+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:03.104619+0000) 2019-10-13T10:46:04.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:04.827+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.829272+0000) 2019-10-13T10:46:04.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:04.827+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.829758+0000) 2019-10-13T10:46:04.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:04.827+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.830402+0000) 2019-10-13T10:46:04.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:04.827+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.831099+0000) 2019-10-13T10:46:04.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:04.831+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.831448+0000) 2019-10-13T10:46:04.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:04.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.830839+0000) 2019-10-13T10:46:04.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:04.831+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.831991+0000) 2019-10-13T10:46:04.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:04.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:04.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:04.835+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.840390+0000) 2019-10-13T10:46:05.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:05.135+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.135539+0000) 2019-10-13T10:46:05.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:05.971+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.977242+0000) 2019-10-13T10:46:06.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:06.351+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.354452+0000) 2019-10-13T10:46:08.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:08.615+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:08.617307+0000) 2019-10-13T10:46:08.624 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:08.619+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:08.622891+0000) 2019-10-13T10:46:09.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:09.830+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.837903+0000) 2019-10-13T10:46:09.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:09.835+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.839199+0000) 2019-10-13T10:46:09.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:09.839+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.839876+0000) 2019-10-13T10:46:09.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:09.839+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.840261+0000) 2019-10-13T10:46:09.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:09.839+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.840852+0000) 2019-10-13T10:46:09.843 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:09.839+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.841016+0000) 2019-10-13T10:46:09.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:09.839+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.841368+0000) 2019-10-13T10:46:09.844 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:09.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:09.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:09.838+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:09.845923+0000) 2019-10-13T10:46:11.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:11.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.917726+0000) 2019-10-13T10:46:12.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:12.106+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:12.108411+0000) 2019-10-13T10:46:13.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:13.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:13.104837+0000) 2019-10-13T10:46:14.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:14.842+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.848341+0000) 2019-10-13T10:46:14.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:14.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.847381+0000) 2019-10-13T10:46:14.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:14.846+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.847592+0000) 2019-10-13T10:46:14.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:14.846+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.847878+0000) 2019-10-13T10:46:14.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:14.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.848853+0000) 2019-10-13T10:46:14.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:14.846+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.849019+0000) 2019-10-13T10:46:14.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:14.846+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.849312+0000) 2019-10-13T10:46:14.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:14.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:14.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:14.850+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.856540+0000) 2019-10-13T10:46:15.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:15.134+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.135752+0000) 2019-10-13T10:46:15.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:15.982+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.983803+0000) 2019-10-13T10:46:16.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:16.350+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.354653+0000) 2019-10-13T10:46:18.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:18.614+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:18.617527+0000) 2019-10-13T10:46:18.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:18.618+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:18.623087+0000) 2019-10-13T10:46:19.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:19.854+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.855778+0000) 2019-10-13T10:46:19.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:19.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.857165+0000) 2019-10-13T10:46:19.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:19.854+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.857645+0000) 2019-10-13T10:46:19.859 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:19.854+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.858003+0000) 2019-10-13T10:46:19.859 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:19.854+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.858498+0000) 2019-10-13T10:46:19.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:19.854+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.858879+0000) 2019-10-13T10:46:19.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:19.854+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.860017+0000) 2019-10-13T10:46:19.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:19.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:19.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:19.862+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:19.867237+0000) 2019-10-13T10:46:21.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:21.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.917930+0000) 2019-10-13T10:46:22.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:22.106+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:22.108628+0000) 2019-10-13T10:46:23.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:23.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:23.105067+0000) 2019-10-13T10:46:24.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:24.862+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.863972+0000) 2019-10-13T10:46:24.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:24.862+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.864410+0000) 2019-10-13T10:46:24.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:24.858+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.865056+0000) 2019-10-13T10:46:24.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:24.862+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.866293+0000) 2019-10-13T10:46:24.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:24.862+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.866749+0000) 2019-10-13T10:46:24.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:24.862+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.867149+0000) 2019-10-13T10:46:24.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:24.866+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.868143+0000) 2019-10-13T10:46:24.870 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:24.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:24.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:24.866+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.871545+0000) 2019-10-13T10:46:25.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:25.134+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.135984+0000) 2019-10-13T10:46:25.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:25.986+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.990253+0000) 2019-10-13T10:46:26.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:26.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.354905+0000) 2019-10-13T10:46:28.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:28.614+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:28.617723+0000) 2019-10-13T10:46:28.624 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:28.622+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:28.623349+0000) 2019-10-13T10:46:29.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:29.870+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.872871+0000) 2019-10-13T10:46:29.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:29.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.873579+0000) 2019-10-13T10:46:29.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:29.870+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.874065+0000) 2019-10-13T10:46:29.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:29.870+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.875161+0000) 2019-10-13T10:46:29.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:29.870+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.874676+0000) 2019-10-13T10:46:29.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:29.874+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.875846+0000) 2019-10-13T10:46:29.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:29.874+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.876263+0000) 2019-10-13T10:46:29.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:29.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:29.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:29.878+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:29.884284+0000) 2019-10-13T10:46:31.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:31.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.918160+0000) 2019-10-13T10:46:32.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:32.106+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:32.108848+0000) 2019-10-13T10:46:32.725 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:46:32.726 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:32.729 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:46:32.729 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:33.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:33.098+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:33.105256+0000) 2019-10-13T10:46:34.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:34.878+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.881202+0000) 2019-10-13T10:46:34.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:34.878+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.881647+0000) 2019-10-13T10:46:34.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:34.878+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.882924+0000) 2019-10-13T10:46:34.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:34.882+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.883406+0000) 2019-10-13T10:46:34.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:34.878+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.883990+0000) 2019-10-13T10:46:34.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:34.882+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.884611+0000) 2019-10-13T10:46:34.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:34.882+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.884853+0000) 2019-10-13T10:46:34.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:34.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:34.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:34.882+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.887796+0000) 2019-10-13T10:46:35.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:35.134+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.136162+0000) 2019-10-13T10:46:35.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:35.994+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.996754+0000) 2019-10-13T10:46:36.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:36.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.355100+0000) 2019-10-13T10:46:38.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:38.614+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:38.617919+0000) 2019-10-13T10:46:38.624 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:38.622+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:38.623577+0000) 2019-10-13T10:46:39.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:39.886+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.889273+0000) 2019-10-13T10:46:39.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:39.886+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.890211+0000) 2019-10-13T10:46:39.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:39.886+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.890571+0000) 2019-10-13T10:46:39.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:39.886+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.890880+0000) 2019-10-13T10:46:39.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:39.890+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.891413+0000) 2019-10-13T10:46:39.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:39.886+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.892919+0000) 2019-10-13T10:46:39.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:39.890+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.893640+0000) 2019-10-13T10:46:39.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:39.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:39.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:39.894+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:39.899213+0000) 2019-10-13T10:46:41.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:41.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.918358+0000) 2019-10-13T10:46:42.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:42.106+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:42.109062+0000) 2019-10-13T10:46:43.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:43.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:43.105458+0000) 2019-10-13T10:46:44.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:44.894+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.898244+0000) 2019-10-13T10:46:44.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:44.894+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.898769+0000) 2019-10-13T10:46:44.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:44.894+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.899153+0000) 2019-10-13T10:46:44.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:44.898+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.899591+0000) 2019-10-13T10:46:44.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:44.898+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.900466+0000) 2019-10-13T10:46:44.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:44.898+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.901659+0000) 2019-10-13T10:46:44.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:44.898+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.902491+0000) 2019-10-13T10:46:44.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:44.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:44.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:44.902+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.906638+0000) 2019-10-13T10:46:45.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:45.134+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.136344+0000) 2019-10-13T10:46:46.004 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:46.002+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.003053+0000) 2019-10-13T10:46:46.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:46.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.355297+0000) 2019-10-13T10:46:48.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:48.614+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:48.618150+0000) 2019-10-13T10:46:48.624 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:48.622+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:48.623772+0000) 2019-10-13T10:46:49.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:49.902+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.906249+0000) 2019-10-13T10:46:49.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:49.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.907205+0000) 2019-10-13T10:46:49.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:49.906+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.907871+0000) 2019-10-13T10:46:49.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:49.906+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.908205+0000) 2019-10-13T10:46:49.909 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:49.906+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.908616+0000) 2019-10-13T10:46:49.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:49.902+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.908959+0000) 2019-10-13T10:46:49.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:49.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.909925+0000) 2019-10-13T10:46:49.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:49.910+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 107971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:49.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:49.910+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:49.915597+0000) 2019-10-13T10:46:51.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:51.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.918556+0000) 2019-10-13T10:46:52.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:52.106+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:52.109293+0000) 2019-10-13T10:46:53.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:53.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:53.105660+0000) 2019-10-13T10:46:54.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:54.910+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.915269+0000) 2019-10-13T10:46:54.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:54.914+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.915765+0000) 2019-10-13T10:46:54.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:54.914+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.916024+0000) 2019-10-13T10:46:54.918 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:54.914+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.917127+0000) 2019-10-13T10:46:54.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:54.910+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.917570+0000) 2019-10-13T10:46:54.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:54.914+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.917818+0000) 2019-10-13T10:46:54.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:54.914+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.918846+0000) 2019-10-13T10:46:54.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:54.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:54.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:54.918+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.925578+0000) 2019-10-13T10:46:55.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:55.134+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.136565+0000) 2019-10-13T10:46:56.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:56.014+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.014497+0000) 2019-10-13T10:46:56.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:56.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.355526+0000) 2019-10-13T10:46:58.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:58.613+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:58.618367+0000) 2019-10-13T10:46:58.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:58.621+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:58.623986+0000) 2019-10-13T10:46:59.925 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:46:59.921+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.923443+0000) 2019-10-13T10:46:59.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:46:59.921+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.924421+0000) 2019-10-13T10:46:59.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:46:59.921+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.925392+0000) 2019-10-13T10:46:59.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:46:59.921+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.925919+0000) 2019-10-13T10:46:59.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:46:59.921+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.926237+0000) 2019-10-13T10:46:59.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:46:59.921+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.926712+0000) 2019-10-13T10:46:59.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:46:59.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.925825+0000) 2019-10-13T10:46:59.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:46:59.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:46:59.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:46:59.930+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:59.933086+0000) 2019-10-13T10:47:01.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:01.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.918791+0000) 2019-10-13T10:47:02.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:02.105+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:02.109451+0000) 2019-10-13T10:47:02.790 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:47:02.790 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:02.794 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:47:02.794 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:03.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:03.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:03.105859+0000) 2019-10-13T10:47:04.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:04.933+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.935680+0000) 2019-10-13T10:47:04.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:04.929+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.933472+0000) 2019-10-13T10:47:04.942 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:04.929+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.933698+0000) 2019-10-13T10:47:04.942 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:04.929+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.934620+0000) 2019-10-13T10:47:04.942 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:04.929+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.934964+0000) 2019-10-13T10:47:04.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:04.937+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.940659+0000) 2019-10-13T10:47:04.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:04.933+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.936116+0000) 2019-10-13T10:47:04.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:04.933+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.937288+0000) 2019-10-13T10:47:04.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:04.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:05.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:05.133+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.136753+0000) 2019-10-13T10:47:06.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:06.017+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.020892+0000) 2019-10-13T10:47:06.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:06.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.355742+0000) 2019-10-13T10:47:08.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:08.617+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:08.618564+0000) 2019-10-13T10:47:08.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:08.621+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:08.624191+0000) 2019-10-13T10:47:09.943 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:09.937+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.940975+0000) 2019-10-13T10:47:09.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:09.937+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.941429+0000) 2019-10-13T10:47:09.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:09.937+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.941990+0000) 2019-10-13T10:47:09.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:09.941+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.942778+0000) 2019-10-13T10:47:09.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:09.937+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.942859+0000) 2019-10-13T10:47:09.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:09.937+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.943110+0000) 2019-10-13T10:47:09.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:09.941+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.943691+0000) 2019-10-13T10:47:09.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:09.941+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:09.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:09.949+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:09.950749+0000) 2019-10-13T10:47:11.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:11.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.918988+0000) 2019-10-13T10:47:12.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:12.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:12.109683+0000) 2019-10-13T10:47:13.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:13.101+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:13.106089+0000) 2019-10-13T10:47:14.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:14.945+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.949397+0000) 2019-10-13T10:47:14.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:14.945+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.949546+0000) 2019-10-13T10:47:14.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:14.945+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.949942+0000) 2019-10-13T10:47:14.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:14.945+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.951167+0000) 2019-10-13T10:47:14.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:14.949+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.952126+0000) 2019-10-13T10:47:14.953 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:14.949+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.952708+0000) 2019-10-13T10:47:14.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:14.949+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.953151+0000) 2019-10-13T10:47:14.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:14.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:14.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:14.953+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.956455+0000) 2019-10-13T10:47:15.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:15.133+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.136962+0000) 2019-10-13T10:47:16.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:16.025+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.027522+0000) 2019-10-13T10:47:16.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:16.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.355998+0000) 2019-10-13T10:47:18.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:18.613+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:18.618763+0000) 2019-10-13T10:47:18.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:18.621+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:18.624455+0000) 2019-10-13T10:47:19.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:19.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.957003+0000) 2019-10-13T10:47:19.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:19.953+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.957693+0000) 2019-10-13T10:47:19.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:19.953+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.957968+0000) 2019-10-13T10:47:19.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:19.953+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.958394+0000) 2019-10-13T10:47:19.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:19.953+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.958642+0000) 2019-10-13T10:47:19.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:19.957+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.960118+0000) 2019-10-13T10:47:19.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:19.957+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.960574+0000) 2019-10-13T10:47:19.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:19.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:19.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:19.965+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:19.966697+0000) 2019-10-13T10:47:21.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:21.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.919215+0000) 2019-10-13T10:47:22.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:22.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:22.109895+0000) 2019-10-13T10:47:23.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:23.105+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:23.106367+0000) 2019-10-13T10:47:24.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:24.961+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.965867+0000) 2019-10-13T10:47:24.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:24.961+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.966281+0000) 2019-10-13T10:47:24.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:24.961+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.966645+0000) 2019-10-13T10:47:24.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:24.961+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.967228+0000) 2019-10-13T10:47:24.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:24.965+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.968016+0000) 2019-10-13T10:47:24.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:24.965+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.969271+0000) 2019-10-13T10:47:24.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:24.965+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.970184+0000) 2019-10-13T10:47:24.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:24.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:24.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:24.973+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.975224+0000) 2019-10-13T10:47:25.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:25.133+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.137135+0000) 2019-10-13T10:47:26.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:26.033+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.033920+0000) 2019-10-13T10:47:26.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:26.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.356148+0000) 2019-10-13T10:47:28.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:28.617+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:28.618959+0000) 2019-10-13T10:47:28.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:28.621+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:28.624648+0000) 2019-10-13T10:47:29.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:29.969+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.973782+0000) 2019-10-13T10:47:29.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:29.969+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.974894+0000) 2019-10-13T10:47:29.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:29.973+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.975439+0000) 2019-10-13T10:47:29.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:29.973+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.975878+0000) 2019-10-13T10:47:29.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:29.973+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.976664+0000) 2019-10-13T10:47:29.978 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:29.973+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.977221+0000) 2019-10-13T10:47:29.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:29.973+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.978091+0000) 2019-10-13T10:47:29.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:29.977+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:29.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:29.977+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:29.981811+0000) 2019-10-13T10:47:31.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:31.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.919412+0000) 2019-10-13T10:47:32.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:32.105+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:32.110146+0000) 2019-10-13T10:47:32.855 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:47:32.855 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:32.859 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:47:32.859 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:33.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:33.105+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:33.106580+0000) 2019-10-13T10:47:34.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:34.981+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.983896+0000) 2019-10-13T10:47:34.986 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:34.981+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.984245+0000) 2019-10-13T10:47:34.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:34.981+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.984732+0000) 2019-10-13T10:47:34.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:34.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.985635+0000) 2019-10-13T10:47:34.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:34.981+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.985991+0000) 2019-10-13T10:47:34.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:34.981+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.986420+0000) 2019-10-13T10:47:34.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:34.981+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.986552+0000) 2019-10-13T10:47:34.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:34.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:34.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:34.989+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.994061+0000) 2019-10-13T10:47:35.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:35.133+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.137330+0000) 2019-10-13T10:47:36.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:36.037+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.040344+0000) 2019-10-13T10:47:36.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:36.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.356387+0000) 2019-10-13T10:47:38.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:38.617+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:38.619142+0000) 2019-10-13T10:47:38.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:38.621+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:38.624853+0000) 2019-10-13T10:47:39.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:39.989+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.990917+0000) 2019-10-13T10:47:39.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:39.988+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.992427+0000) 2019-10-13T10:47:39.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:39.988+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.993591+0000) 2019-10-13T10:47:39.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:39.988+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.994114+0000) 2019-10-13T10:47:39.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:39.988+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.994621+0000) 2019-10-13T10:47:39.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:39.988+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.994791+0000) 2019-10-13T10:47:39.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:39.992+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:39.995463+0000) 2019-10-13T10:47:39.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:39.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:40.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:39.997+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.000659+0000) 2019-10-13T10:47:41.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:41.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.919577+0000) 2019-10-13T10:47:42.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:42.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:42.110381+0000) 2019-10-13T10:47:43.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:43.105+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:43.106770+0000) 2019-10-13T10:47:45.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:44.997+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:44.999675+0000) 2019-10-13T10:47:45.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:44.996+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.000813+0000) 2019-10-13T10:47:45.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:44.996+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.001143+0000) 2019-10-13T10:47:45.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:44.996+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.001391+0000) 2019-10-13T10:47:45.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:44.996+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.001921+0000) 2019-10-13T10:47:45.003 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:44.996+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.002226+0000) 2019-10-13T10:47:45.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:44.996+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.002734+0000) 2019-10-13T10:47:45.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:45.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:45.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:45.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.007474+0000) 2019-10-13T10:47:45.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:45.136+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.137515+0000) 2019-10-13T10:47:46.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:46.045+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.046766+0000) 2019-10-13T10:47:46.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:46.352+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.356614+0000) 2019-10-13T10:47:48.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:48.616+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:48.619367+0000) 2019-10-13T10:47:48.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:48.620+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:48.625061+0000) 2019-10-13T10:47:50.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:50.004+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.008004+0000) 2019-10-13T10:47:50.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:50.004+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.008760+0000) 2019-10-13T10:47:50.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:50.005+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.009284+0000) 2019-10-13T10:47:50.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:50.004+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.009646+0000) 2019-10-13T10:47:50.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:50.004+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.010197+0000) 2019-10-13T10:47:50.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:50.004+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.010799+0000) 2019-10-13T10:47:50.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:50.008+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.011695+0000) 2019-10-13T10:47:50.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:50.008+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:50.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:50.013+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.016591+0000) 2019-10-13T10:47:51.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:51.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.919781+0000) 2019-10-13T10:47:52.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:52.104+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:52.110617+0000) 2019-10-13T10:47:53.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:53.105+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:53.106974+0000) 2019-10-13T10:47:55.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:55.012+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.016996+0000) 2019-10-13T10:47:55.019 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:55.012+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.017926+0000) 2019-10-13T10:47:55.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:47:55.012+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.018388+0000) 2019-10-13T10:47:55.020 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:47:55.016+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.019342+0000) 2019-10-13T10:47:55.021 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:55.016+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.020065+0000) 2019-10-13T10:47:55.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:47:55.017+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.020442+0000) 2019-10-13T10:47:55.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:55.016+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.021140+0000) 2019-10-13T10:47:55.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:47:55.016+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:47:55.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:55.021+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.024199+0000) 2019-10-13T10:47:55.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:47:55.136+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.137699+0000) 2019-10-13T10:47:56.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:47:56.049+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.053263+0000) 2019-10-13T10:47:56.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:47:56.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.356834+0000) 2019-10-13T10:47:58.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:47:58.616+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:58.619573+0000) 2019-10-13T10:47:58.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:47:58.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:58.625255+0000) 2019-10-13T10:48:00.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:00.020+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.026315+0000) 2019-10-13T10:48:00.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:00.024+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.027463+0000) 2019-10-13T10:48:00.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:00.024+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.027595+0000) 2019-10-13T10:48:00.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:00.024+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.028124+0000) 2019-10-13T10:48:00.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:00.024+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.028494+0000) 2019-10-13T10:48:00.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:00.024+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.029175+0000) 2019-10-13T10:48:00.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:00.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.029967+0000) 2019-10-13T10:48:00.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:00.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108447 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:00.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:00.028+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.034095+0000) 2019-10-13T10:48:01.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:01.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.920006+0000) 2019-10-13T10:48:02.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:02.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:02.110838+0000) 2019-10-13T10:48:02.988 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:48:02.988 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:02.992 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:48:02.992 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:03.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:03.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:03.107189+0000) 2019-10-13T10:48:05.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:05.028+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.034651+0000) 2019-10-13T10:48:05.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:05.028+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.034902+0000) 2019-10-13T10:48:05.036 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:05.032+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.035474+0000) 2019-10-13T10:48:05.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:05.032+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.036395+0000) 2019-10-13T10:48:05.037 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:05.032+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.036695+0000) 2019-10-13T10:48:05.038 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:05.032+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.037874+0000) 2019-10-13T10:48:05.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:05.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.038298+0000) 2019-10-13T10:48:05.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:05.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:05.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:05.040+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.043698+0000) 2019-10-13T10:48:05.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:05.136+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.137855+0000) 2019-10-13T10:48:06.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:06.056+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.059656+0000) 2019-10-13T10:48:06.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:06.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.357039+0000) 2019-10-13T10:48:08.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:08.616+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:08.619779+0000) 2019-10-13T10:48:08.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:08.620+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:08.625448+0000) 2019-10-13T10:48:10.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:10.040+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.044167+0000) 2019-10-13T10:48:10.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:10.040+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.044729+0000) 2019-10-13T10:48:10.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:10.040+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.045107+0000) 2019-10-13T10:48:10.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:10.040+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.045389+0000) 2019-10-13T10:48:10.047 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:10.040+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.046328+0000) 2019-10-13T10:48:10.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:10.044+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.046995+0000) 2019-10-13T10:48:10.048 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:10.040+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.047052+0000) 2019-10-13T10:48:10.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:10.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:10.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:10.048+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.052713+0000) 2019-10-13T10:48:11.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:11.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.920206+0000) 2019-10-13T10:48:12.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:12.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:12.111045+0000) 2019-10-13T10:48:13.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:13.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:13.107407+0000) 2019-10-13T10:48:15.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:15.048+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.051798+0000) 2019-10-13T10:48:15.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:15.048+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.052952+0000) 2019-10-13T10:48:15.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:15.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.053199+0000) 2019-10-13T10:48:15.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:15.048+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.053843+0000) 2019-10-13T10:48:15.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:15.048+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.054323+0000) 2019-10-13T10:48:15.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:15.048+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.054557+0000) 2019-10-13T10:48:15.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:15.048+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.055205+0000) 2019-10-13T10:48:15.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:15.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:15.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:15.056+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.061419+0000) 2019-10-13T10:48:15.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:15.136+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.138057+0000) 2019-10-13T10:48:16.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:16.064+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.066088+0000) 2019-10-13T10:48:16.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:16.352+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.357291+0000) 2019-10-13T10:48:18.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:18.616+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:18.620013+0000) 2019-10-13T10:48:18.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:18.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:18.625641+0000) 2019-10-13T10:48:20.062 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:20.056+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.060691+0000) 2019-10-13T10:48:20.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:20.056+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.061606+0000) 2019-10-13T10:48:20.063 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:20.056+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.061832+0000) 2019-10-13T10:48:20.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:20.056+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.062344+0000) 2019-10-13T10:48:20.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:20.060+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.062847+0000) 2019-10-13T10:48:20.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:20.056+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.063107+0000) 2019-10-13T10:48:20.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:20.060+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.064280+0000) 2019-10-13T10:48:20.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:20.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:20.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:20.064+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.069579+0000) 2019-10-13T10:48:21.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:21.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.920410+0000) 2019-10-13T10:48:22.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:22.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:22.111281+0000) 2019-10-13T10:48:23.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:23.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:23.107632+0000) 2019-10-13T10:48:25.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:25.063+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.068438+0000) 2019-10-13T10:48:25.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:25.063+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.069510+0000) 2019-10-13T10:48:25.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:25.068+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.070389+0000) 2019-10-13T10:48:25.072 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:25.067+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.071286+0000) 2019-10-13T10:48:25.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:25.067+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.071809+0000) 2019-10-13T10:48:25.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:25.067+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.072150+0000) 2019-10-13T10:48:25.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:25.067+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.072811+0000) 2019-10-13T10:48:25.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:25.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108616 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:25.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:25.072+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.077146+0000) 2019-10-13T10:48:25.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:25.135+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.138254+0000) 2019-10-13T10:48:26.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:26.068+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.072441+0000) 2019-10-13T10:48:26.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:26.355+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.357487+0000) 2019-10-13T10:48:28.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:28.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:28.620277+0000) 2019-10-13T10:48:28.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:28.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:28.625850+0000) 2019-10-13T10:48:30.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:30.071+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.076746+0000) 2019-10-13T10:48:30.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:30.071+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.077366+0000) 2019-10-13T10:48:30.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:30.071+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.077777+0000) 2019-10-13T10:48:30.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:30.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.079041+0000) 2019-10-13T10:48:30.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:30.071+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.079269+0000) 2019-10-13T10:48:30.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:30.075+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.079569+0000) 2019-10-13T10:48:30.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:30.075+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.080305+0000) 2019-10-13T10:48:30.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:30.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:30.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:30.084+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.089474+0000) 2019-10-13T10:48:31.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:31.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.920643+0000) 2019-10-13T10:48:32.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:32.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:32.111523+0000) 2019-10-13T10:48:33.098 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:48:33.099 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:33.102 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:48:33.103 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:33.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:33.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:33.107817+0000) 2019-10-13T10:48:35.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:35.080+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.085191+0000) 2019-10-13T10:48:35.086 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:35.079+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.085894+0000) 2019-10-13T10:48:35.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:35.079+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.086450+0000) 2019-10-13T10:48:35.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:35.079+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.087172+0000) 2019-10-13T10:48:35.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:35.083+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.088210+0000) 2019-10-13T10:48:35.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:35.083+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.088693+0000) 2019-10-13T10:48:35.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:35.083+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.089409+0000) 2019-10-13T10:48:35.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:35.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:35.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:35.092+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.096246+0000) 2019-10-13T10:48:35.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:35.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.138431+0000) 2019-10-13T10:48:36.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:36.076+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.079335+0000) 2019-10-13T10:48:36.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:36.351+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.357683+0000) 2019-10-13T10:48:38.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:38.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:38.620484+0000) 2019-10-13T10:48:38.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:38.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:38.626051+0000) 2019-10-13T10:48:40.095 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:40.087+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.094013+0000) 2019-10-13T10:48:40.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:40.087+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.094285+0000) 2019-10-13T10:48:40.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:40.087+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.095222+0000) 2019-10-13T10:48:40.097 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:40.091+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.096004+0000) 2019-10-13T10:48:40.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:40.091+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.096618+0000) 2019-10-13T10:48:40.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:40.092+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.096918+0000) 2019-10-13T10:48:40.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:40.091+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.097244+0000) 2019-10-13T10:48:40.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:40.096+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.101794+0000) 2019-10-13T10:48:40.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:40.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:41.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:41.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.920876+0000) 2019-10-13T10:48:42.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:42.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:42.111719+0000) 2019-10-13T10:48:43.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:43.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:43.108027+0000) 2019-10-13T10:48:45.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:45.095+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.103127+0000) 2019-10-13T10:48:45.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:45.100+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.103970+0000) 2019-10-13T10:48:45.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:45.099+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.104386+0000) 2019-10-13T10:48:45.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:45.099+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.105111+0000) 2019-10-13T10:48:45.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:45.099+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.105505+0000) 2019-10-13T10:48:45.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:45.099+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.105793+0000) 2019-10-13T10:48:45.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:45.099+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.106237+0000) 2019-10-13T10:48:45.115 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:45.108+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.113504+0000) 2019-10-13T10:48:45.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:45.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:45.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:45.131+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.138640+0000) 2019-10-13T10:48:46.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:46.088+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.090881+0000) 2019-10-13T10:48:46.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:46.351+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.357904+0000) 2019-10-13T10:48:48.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:48.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:48.620707+0000) 2019-10-13T10:48:48.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:48.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:48.626276+0000) 2019-10-13T10:48:50.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:50.107+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.112549+0000) 2019-10-13T10:48:50.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:50.107+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.113296+0000) 2019-10-13T10:48:50.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:50.107+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.113464+0000) 2019-10-13T10:48:50.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:50.112+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.114480+0000) 2019-10-13T10:48:50.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:50.107+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.114937+0000) 2019-10-13T10:48:50.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:50.107+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.115254+0000) 2019-10-13T10:48:50.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:50.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.115961+0000) 2019-10-13T10:48:50.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:50.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:50.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:50.120+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.122480+0000) 2019-10-13T10:48:51.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:51.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.921087+0000) 2019-10-13T10:48:52.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:52.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:52.111921+0000) 2019-10-13T10:48:53.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:53.104+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:53.108230+0000) 2019-10-13T10:48:55.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:55.119+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.120398+0000) 2019-10-13T10:48:55.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:48:55.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.121196+0000) 2019-10-13T10:48:55.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:55.119+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.121681+0000) 2019-10-13T10:48:55.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:55.119+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.122235+0000) 2019-10-13T10:48:55.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:55.119+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.123012+0000) 2019-10-13T10:48:55.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:48:55.123+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.123600+0000) 2019-10-13T10:48:55.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:48:55.123+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.123804+0000) 2019-10-13T10:48:55.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:48:55.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:48:55.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:55.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.131650+0000) 2019-10-13T10:48:55.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:48:55.135+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.138791+0000) 2019-10-13T10:48:56.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:48:56.095+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.097318+0000) 2019-10-13T10:48:56.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:48:56.355+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.358096+0000) 2019-10-13T10:48:58.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:48:58.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:58.620918+0000) 2019-10-13T10:48:58.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:48:58.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:58.626489+0000) 2019-10-13T10:49:00.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:00.127+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.130268+0000) 2019-10-13T10:49:00.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:00.127+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.130660+0000) 2019-10-13T10:49:00.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:00.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.129901+0000) 2019-10-13T10:49:00.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:00.127+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.131069+0000) 2019-10-13T10:49:00.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:00.131+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.131369+0000) 2019-10-13T10:49:00.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:00.131+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.132067+0000) 2019-10-13T10:49:00.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:00.131+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.132562+0000) 2019-10-13T10:49:00.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:00.131+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:00.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:00.131+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.135521+0000) 2019-10-13T10:49:01.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:01.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.921302+0000) 2019-10-13T10:49:02.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:02.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:02.112145+0000) 2019-10-13T10:49:03.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:03.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:03.108430+0000) 2019-10-13T10:49:03.193 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:49:03.194 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:03.198 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:49:03.198 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:05.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:05.135+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.138712+0000) 2019-10-13T10:49:05.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:05.135+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.139186+0000) 2019-10-13T10:49:05.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:05.139+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.139566+0000) 2019-10-13T10:49:05.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:05.139+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.140243+0000) 2019-10-13T10:49:05.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:05.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.141187+0000) 2019-10-13T10:49:05.142 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:05.139+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.141020+0000) 2019-10-13T10:49:05.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:05.139+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.142093+0000) 2019-10-13T10:49:05.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:05.143+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:05.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:05.143+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.148159+0000) 2019-10-13T10:49:06.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:06.107+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.109008+0000) 2019-10-13T10:49:06.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:06.355+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.358254+0000) 2019-10-13T10:49:08.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:08.618+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:08.621121+0000) 2019-10-13T10:49:08.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:08.622+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:08.626635+0000) 2019-10-13T10:49:10.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:10.142+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.146753+0000) 2019-10-13T10:49:10.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:10.146+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.147687+0000) 2019-10-13T10:49:10.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:10.146+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.148018+0000) 2019-10-13T10:49:10.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:10.146+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.148695+0000) 2019-10-13T10:49:10.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:10.146+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.149389+0000) 2019-10-13T10:49:10.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:10.143+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.150005+0000) 2019-10-13T10:49:10.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:10.146+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.150505+0000) 2019-10-13T10:49:10.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:10.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:10.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:10.151+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.155744+0000) 2019-10-13T10:49:11.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:11.886+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.889417+0000) 2019-10-13T10:49:11.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:11.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.921503+0000) 2019-10-13T10:49:12.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:12.110+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:12.112361+0000) 2019-10-13T10:49:13.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:13.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:13.108618+0000) 2019-10-13T10:49:15.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:15.154+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.155477+0000) 2019-10-13T10:49:15.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:15.154+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.156111+0000) 2019-10-13T10:49:15.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:15.151+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.156966+0000) 2019-10-13T10:49:15.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:15.154+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.157399+0000) 2019-10-13T10:49:15.159 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:15.154+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.157803+0000) 2019-10-13T10:49:15.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:15.154+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.158169+0000) 2019-10-13T10:49:15.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:15.154+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.158548+0000) 2019-10-13T10:49:15.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:15.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:15.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:15.163+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.167839+0000) 2019-10-13T10:49:16.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:16.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.120717+0000) 2019-10-13T10:49:16.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:16.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.358472+0000) 2019-10-13T10:49:18.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:18.618+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:18.621357+0000) 2019-10-13T10:49:18.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:18.622+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:18.626855+0000) 2019-10-13T10:49:20.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:20.162+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.164097+0000) 2019-10-13T10:49:20.166 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:20.162+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.164941+0000) 2019-10-13T10:49:20.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:20.162+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.165184+0000) 2019-10-13T10:49:20.167 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:20.162+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.165559+0000) 2019-10-13T10:49:20.168 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:20.163+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.166497+0000) 2019-10-13T10:49:20.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:20.162+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.166974+0000) 2019-10-13T10:49:20.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:20.166+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.167563+0000) 2019-10-13T10:49:20.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:20.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 108989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:20.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:20.171+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.176019+0000) 2019-10-13T10:49:21.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:21.886+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.889614+0000) 2019-10-13T10:49:21.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:21.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.921697+0000) 2019-10-13T10:49:22.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:22.110+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:22.112549+0000) 2019-10-13T10:49:23.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:23.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:23.108817+0000) 2019-10-13T10:49:25.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:25.170+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.172692+0000) 2019-10-13T10:49:25.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:25.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.173042+0000) 2019-10-13T10:49:25.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:25.170+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.173992+0000) 2019-10-13T10:49:25.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:25.170+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.174309+0000) 2019-10-13T10:49:25.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:25.170+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.174611+0000) 2019-10-13T10:49:25.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:25.170+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.175235+0000) 2019-10-13T10:49:25.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:25.174+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.176537+0000) 2019-10-13T10:49:25.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:25.174+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:25.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:25.179+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.183434+0000) 2019-10-13T10:49:26.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:26.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.132444+0000) 2019-10-13T10:49:26.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:26.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.358694+0000) 2019-10-13T10:49:28.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:28.618+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:28.621589+0000) 2019-10-13T10:49:28.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:28.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:28.627050+0000) 2019-10-13T10:49:30.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:30.178+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.180890+0000) 2019-10-13T10:49:30.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:30.178+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.181254+0000) 2019-10-13T10:49:30.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:30.178+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.181641+0000) 2019-10-13T10:49:30.183 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:30.179+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.182371+0000) 2019-10-13T10:49:30.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:30.178+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.182122+0000) 2019-10-13T10:49:30.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:30.178+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.182964+0000) 2019-10-13T10:49:30.184 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:30.182+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.183960+0000) 2019-10-13T10:49:30.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:30.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:30.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:30.183+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.189620+0000) 2019-10-13T10:49:31.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:31.886+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.889790+0000) 2019-10-13T10:49:31.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:31.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.921915+0000) 2019-10-13T10:49:32.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:32.110+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:32.112769+0000) 2019-10-13T10:49:33.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:33.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:33.109003+0000) 2019-10-13T10:49:33.321 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:49:33.321 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:33.325 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:49:33.325 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:35.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:35.186+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.189239+0000) 2019-10-13T10:49:35.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:35.186+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.189694+0000) 2019-10-13T10:49:35.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:35.186+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.190073+0000) 2019-10-13T10:49:35.192 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:35.190+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.191401+0000) 2019-10-13T10:49:35.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:35.190+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.191542+0000) 2019-10-13T10:49:35.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:35.190+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.191822+0000) 2019-10-13T10:49:35.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:35.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.192415+0000) 2019-10-13T10:49:35.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:35.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:35.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:35.191+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.196657+0000) 2019-10-13T10:49:36.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:36.139+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.142319+0000) 2019-10-13T10:49:36.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:36.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.358890+0000) 2019-10-13T10:49:38.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:38.618+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:38.621790+0000) 2019-10-13T10:49:38.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:38.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:38.627242+0000) 2019-10-13T10:49:40.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:40.194+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.197645+0000) 2019-10-13T10:49:40.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:40.194+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.197769+0000) 2019-10-13T10:49:40.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:40.194+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.198408+0000) 2019-10-13T10:49:40.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:40.198+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.199465+0000) 2019-10-13T10:49:40.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:40.195+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.199225+0000) 2019-10-13T10:49:40.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:40.198+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.199753+0000) 2019-10-13T10:49:40.201 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:40.198+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.200415+0000) 2019-10-13T10:49:40.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:40.198+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109125 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:40.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:40.203+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.207335+0000) 2019-10-13T10:49:41.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:41.886+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.889930+0000) 2019-10-13T10:49:41.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:41.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.922128+0000) 2019-10-13T10:49:42.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:42.110+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:42.112969+0000) 2019-10-13T10:49:43.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:43.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:43.109196+0000) 2019-10-13T10:49:45.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:45.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.205400+0000) 2019-10-13T10:49:45.207 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:45.202+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.205693+0000) 2019-10-13T10:49:45.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:45.202+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.207019+0000) 2019-10-13T10:49:45.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:45.202+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.207171+0000) 2019-10-13T10:49:45.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:45.206+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.207588+0000) 2019-10-13T10:49:45.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:45.206+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.208683+0000) 2019-10-13T10:49:45.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:45.206+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.208975+0000) 2019-10-13T10:49:45.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:45.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:45.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:45.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.217207+0000) 2019-10-13T10:49:46.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:46.147+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.148674+0000) 2019-10-13T10:49:46.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:46.354+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.359078+0000) 2019-10-13T10:49:48.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:48.618+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:48.622012+0000) 2019-10-13T10:49:48.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:48.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:48.627471+0000) 2019-10-13T10:49:50.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:50.210+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.213606+0000) 2019-10-13T10:49:50.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:50.210+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.214688+0000) 2019-10-13T10:49:50.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:50.210+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.215242+0000) 2019-10-13T10:49:50.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:50.214+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.215501+0000) 2019-10-13T10:49:50.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:50.211+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.215814+0000) 2019-10-13T10:49:50.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:50.214+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.216350+0000) 2019-10-13T10:49:50.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:50.214+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.217300+0000) 2019-10-13T10:49:50.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:50.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:50.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:50.215+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.221906+0000) 2019-10-13T10:49:51.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:51.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.890136+0000) 2019-10-13T10:49:51.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:51.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.922365+0000) 2019-10-13T10:49:52.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:52.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:52.113162+0000) 2019-10-13T10:49:53.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:53.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:53.109405+0000) 2019-10-13T10:49:55.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:49:55.217+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.222172+0000) 2019-10-13T10:49:55.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:55.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.222639+0000) 2019-10-13T10:49:55.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:55.221+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.223358+0000) 2019-10-13T10:49:55.225 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:49:55.221+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.224328+0000) 2019-10-13T10:49:55.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:55.221+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.225104+0000) 2019-10-13T10:49:55.226 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:49:55.218+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.225568+0000) 2019-10-13T10:49:55.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:49:55.221+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.225916+0000) 2019-10-13T10:49:55.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:49:55.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:49:55.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:55.226+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.231180+0000) 2019-10-13T10:49:56.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:49:56.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.154987+0000) 2019-10-13T10:49:56.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:49:56.353+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.359270+0000) 2019-10-13T10:49:58.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:49:58.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:58.622229+0000) 2019-10-13T10:49:58.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:49:58.625+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:58.627660+0000) 2019-10-13T10:50:00.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:00.225+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.231223+0000) 2019-10-13T10:50:00.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:00.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.231738+0000) 2019-10-13T10:50:00.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:00.229+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.232373+0000) 2019-10-13T10:50:00.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:00.229+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.233196+0000) 2019-10-13T10:50:00.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:00.229+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.233396+0000) 2019-10-13T10:50:00.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:00.229+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.233921+0000) 2019-10-13T10:50:00.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:00.229+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.234438+0000) 2019-10-13T10:50:00.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:00.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:00.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:00.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.239013+0000) 2019-10-13T10:50:01.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:01.885+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.890341+0000) 2019-10-13T10:50:01.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:01.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.922590+0000) 2019-10-13T10:50:02.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:02.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:02.113397+0000) 2019-10-13T10:50:03.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:03.102+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:03.109614+0000) 2019-10-13T10:50:03.455 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:50:03.455 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:03.459 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:50:03.459 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:05.241 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:05.237+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.239761+0000) 2019-10-13T10:50:05.241 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:05.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.240597+0000) 2019-10-13T10:50:05.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:05.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.240948+0000) 2019-10-13T10:50:05.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:05.237+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.240205+0000) 2019-10-13T10:50:05.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:05.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.241292+0000) 2019-10-13T10:50:05.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:05.237+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.242735+0000) 2019-10-13T10:50:05.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:05.237+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.243172+0000) 2019-10-13T10:50:05.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:05.241+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:05.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:05.242+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.247809+0000) 2019-10-13T10:50:06.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:06.158+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.161479+0000) 2019-10-13T10:50:06.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:06.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.359474+0000) 2019-10-13T10:50:08.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:08.617+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:08.622436+0000) 2019-10-13T10:50:08.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:08.625+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:08.627889+0000) 2019-10-13T10:50:10.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:10.245+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.247686+0000) 2019-10-13T10:50:10.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:10.242+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.248181+0000) 2019-10-13T10:50:10.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:10.245+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.248515+0000) 2019-10-13T10:50:10.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:10.245+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.248829+0000) 2019-10-13T10:50:10.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:10.245+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.249194+0000) 2019-10-13T10:50:10.251 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:10.245+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.249879+0000) 2019-10-13T10:50:10.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:10.245+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.250345+0000) 2019-10-13T10:50:10.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:10.249+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:10.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:10.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.258199+0000) 2019-10-13T10:50:11.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:11.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.890537+0000) 2019-10-13T10:50:11.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:11.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.922845+0000) 2019-10-13T10:50:12.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:12.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:12.113614+0000) 2019-10-13T10:50:13.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:13.106+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:13.109797+0000) 2019-10-13T10:50:15.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:15.253+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.256933+0000) 2019-10-13T10:50:15.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:15.253+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.257996+0000) 2019-10-13T10:50:15.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:15.253+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.258215+0000) 2019-10-13T10:50:15.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:15.254+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.258928+0000) 2019-10-13T10:50:15.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:15.253+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.259202+0000) 2019-10-13T10:50:15.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:15.257+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.259549+0000) 2019-10-13T10:50:15.261 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:15.257+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.260281+0000) 2019-10-13T10:50:15.263 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:15.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:15.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:15.258+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.264196+0000) 2019-10-13T10:50:16.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:16.166+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.168060+0000) 2019-10-13T10:50:16.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:16.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.359680+0000) 2019-10-13T10:50:18.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:18.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:18.622664+0000) 2019-10-13T10:50:18.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:18.625+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:18.628084+0000) 2019-10-13T10:50:20.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:20.261+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.266427+0000) 2019-10-13T10:50:20.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:20.261+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.267089+0000) 2019-10-13T10:50:20.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:20.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.267541+0000) 2019-10-13T10:50:20.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:20.265+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.267946+0000) 2019-10-13T10:50:20.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:20.265+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.269005+0000) 2019-10-13T10:50:20.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:20.265+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.269285+0000) 2019-10-13T10:50:20.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:20.265+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.269617+0000) 2019-10-13T10:50:20.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:20.269+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109398 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:20.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:20.270+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.274416+0000) 2019-10-13T10:50:21.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:21.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.890751+0000) 2019-10-13T10:50:21.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:21.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.923049+0000) 2019-10-13T10:50:22.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:22.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:22.113814+0000) 2019-10-13T10:50:23.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:23.106+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:23.110020+0000) 2019-10-13T10:50:25.277 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:25.273+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.275868+0000) 2019-10-13T10:50:25.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:25.273+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.276292+0000) 2019-10-13T10:50:25.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:25.273+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.277125+0000) 2019-10-13T10:50:25.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:25.273+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.277413+0000) 2019-10-13T10:50:25.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:25.273+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.278264+0000) 2019-10-13T10:50:25.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:25.273+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.278066+0000) 2019-10-13T10:50:25.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:25.274+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.278555+0000) 2019-10-13T10:50:25.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:25.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109431 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:25.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:25.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.282278+0000) 2019-10-13T10:50:26.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:26.174+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.174429+0000) 2019-10-13T10:50:26.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:26.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.359887+0000) 2019-10-13T10:50:28.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:28.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:28.622856+0000) 2019-10-13T10:50:28.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:28.625+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:28.628275+0000) 2019-10-13T10:50:30.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:30.281+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.284942+0000) 2019-10-13T10:50:30.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:30.281+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.285056+0000) 2019-10-13T10:50:30.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:30.281+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.285772+0000) 2019-10-13T10:50:30.287 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:30.281+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.286072+0000) 2019-10-13T10:50:30.288 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:30.282+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.286926+0000) 2019-10-13T10:50:30.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:30.281+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.286535+0000) 2019-10-13T10:50:30.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:30.285+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.287746+0000) 2019-10-13T10:50:30.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:30.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:30.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:30.286+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.292709+0000) 2019-10-13T10:50:31.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:31.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.890948+0000) 2019-10-13T10:50:31.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:31.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.923251+0000) 2019-10-13T10:50:32.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:32.109+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:32.114009+0000) 2019-10-13T10:50:33.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:33.106+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:33.110216+0000) 2019-10-13T10:50:33.812 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:50:33.812 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:33.816 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:50:33.816 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:35.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:35.288+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.292436+0000) 2019-10-13T10:50:35.294 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:35.288+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.292771+0000) 2019-10-13T10:50:35.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:35.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.293286+0000) 2019-10-13T10:50:35.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:35.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.293667+0000) 2019-10-13T10:50:35.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:35.288+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.293974+0000) 2019-10-13T10:50:35.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:35.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.294850+0000) 2019-10-13T10:50:35.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:35.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.296337+0000) 2019-10-13T10:50:35.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:35.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:35.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:35.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.302814+0000) 2019-10-13T10:50:36.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:36.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.180702+0000) 2019-10-13T10:50:36.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:36.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.360097+0000) 2019-10-13T10:50:38.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:38.616+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:38.623072+0000) 2019-10-13T10:50:38.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:38.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:38.628468+0000) 2019-10-13T10:50:40.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:40.296+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.300347+0000) 2019-10-13T10:50:40.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:40.298+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.301824+0000) 2019-10-13T10:50:40.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:40.296+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.302114+0000) 2019-10-13T10:50:40.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:40.296+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.302593+0000) 2019-10-13T10:50:40.304 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:40.296+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.302976+0000) 2019-10-13T10:50:40.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:40.300+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.303493+0000) 2019-10-13T10:50:40.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:40.300+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.304189+0000) 2019-10-13T10:50:40.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:40.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:40.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:40.306+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.307324+0000) 2019-10-13T10:50:41.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:41.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.891172+0000) 2019-10-13T10:50:41.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:41.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.923460+0000) 2019-10-13T10:50:42.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:42.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:42.114211+0000) 2019-10-13T10:50:43.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:43.110+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:43.110407+0000) 2019-10-13T10:50:45.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:45.304+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.309386+0000) 2019-10-13T10:50:45.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:45.306+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.309812+0000) 2019-10-13T10:50:45.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:45.304+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.310628+0000) 2019-10-13T10:50:45.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:45.304+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.311071+0000) 2019-10-13T10:50:45.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:45.304+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.311117+0000) 2019-10-13T10:50:45.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:45.308+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.311888+0000) 2019-10-13T10:50:45.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:45.308+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.312974+0000) 2019-10-13T10:50:45.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:45.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:45.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:45.318+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.318488+0000) 2019-10-13T10:50:46.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:46.185+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.186993+0000) 2019-10-13T10:50:46.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:46.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.360313+0000) 2019-10-13T10:50:48.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:48.620+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:48.623323+0000) 2019-10-13T10:50:48.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:48.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:48.628670+0000) 2019-10-13T10:50:50.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:50.312+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.318248+0000) 2019-10-13T10:50:50.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:50.312+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.318796+0000) 2019-10-13T10:50:50.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:50.316+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.319995+0000) 2019-10-13T10:50:50.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:50.316+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.320109+0000) 2019-10-13T10:50:50.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:50.316+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.320423+0000) 2019-10-13T10:50:50.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:50.317+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.321524+0000) 2019-10-13T10:50:50.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:50.316+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.321896+0000) 2019-10-13T10:50:50.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:50.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:50.328 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:50.325+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.327869+0000) 2019-10-13T10:50:51.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:51.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.891381+0000) 2019-10-13T10:50:51.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:51.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.923677+0000) 2019-10-13T10:50:52.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:52.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:52.114468+0000) 2019-10-13T10:50:53.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:53.109+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:53.110621+0000) 2019-10-13T10:50:55.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:55.320+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.326614+0000) 2019-10-13T10:50:55.328 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:55.324+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.327416+0000) 2019-10-13T10:50:55.328 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:50:55.324+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.327877+0000) 2019-10-13T10:50:55.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:55.324+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.328210+0000) 2019-10-13T10:50:55.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:50:55.324+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.328927+0000) 2019-10-13T10:50:55.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:50:55.324+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.329131+0000) 2019-10-13T10:50:55.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:50:55.329+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.330426+0000) 2019-10-13T10:50:55.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:50:55.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:50:55.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:55.333+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.335461+0000) 2019-10-13T10:50:56.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:50:56.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.193357+0000) 2019-10-13T10:50:56.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:50:56.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.360479+0000) 2019-10-13T10:50:58.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:50:58.620+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:58.623537+0000) 2019-10-13T10:50:58.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:50:58.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:58.628826+0000) 2019-10-13T10:51:00.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:00.332+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.335731+0000) 2019-10-13T10:51:00.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:00.332+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.336107+0000) 2019-10-13T10:51:00.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:00.332+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.336343+0000) 2019-10-13T10:51:00.338 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:00.332+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.337223+0000) 2019-10-13T10:51:00.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:00.332+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.337732+0000) 2019-10-13T10:51:00.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:00.332+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.338392+0000) 2019-10-13T10:51:00.340 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:00.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.339168+0000) 2019-10-13T10:51:00.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:00.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:00.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:00.345+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.348353+0000) 2019-10-13T10:51:01.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:01.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.891587+0000) 2019-10-13T10:51:01.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:01.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.923912+0000) 2019-10-13T10:51:02.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:02.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:02.114671+0000) 2019-10-13T10:51:03.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:03.109+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:03.110850+0000) 2019-10-13T10:51:03.917 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:51:03.918 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:03.922 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:51:03.922 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:05.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:05.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.344907+0000) 2019-10-13T10:51:05.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:05.341+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.345293+0000) 2019-10-13T10:51:05.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:05.340+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.345921+0000) 2019-10-13T10:51:05.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:05.340+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.346613+0000) 2019-10-13T10:51:05.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:05.344+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.347350+0000) 2019-10-13T10:51:05.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:05.344+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.347560+0000) 2019-10-13T10:51:05.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:05.344+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.348007+0000) 2019-10-13T10:51:05.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:05.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:05.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:05.353+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.355680+0000) 2019-10-13T10:51:06.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:06.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.199931+0000) 2019-10-13T10:51:06.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:06.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.360663+0000) 2019-10-13T10:51:08.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:08.620+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:08.623768+0000) 2019-10-13T10:51:08.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:08.624+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:08.629036+0000) 2019-10-13T10:51:10.355 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:10.348+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.353621+0000) 2019-10-13T10:51:10.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:10.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.354622+0000) 2019-10-13T10:51:10.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:10.348+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.354206+0000) 2019-10-13T10:51:10.356 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:10.348+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.354920+0000) 2019-10-13T10:51:10.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:10.352+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.355538+0000) 2019-10-13T10:51:10.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:10.352+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.355370+0000) 2019-10-13T10:51:10.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:10.352+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.356101+0000) 2019-10-13T10:51:10.359 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:10.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:10.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:10.361+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.364914+0000) 2019-10-13T10:51:11.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:11.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.891795+0000) 2019-10-13T10:51:11.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:11.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.924126+0000) 2019-10-13T10:51:12.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:12.108+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:12.114898+0000) 2019-10-13T10:51:13.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:13.109+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:13.111045+0000) 2019-10-13T10:51:15.362 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:15.356+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.360945+0000) 2019-10-13T10:51:15.363 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:15.357+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.361381+0000) 2019-10-13T10:51:15.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:15.356+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.361678+0000) 2019-10-13T10:51:15.363 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:15.356+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.362160+0000) 2019-10-13T10:51:15.364 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:15.356+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.363027+0000) 2019-10-13T10:51:15.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:15.360+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.363902+0000) 2019-10-13T10:51:15.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:15.360+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.364314+0000) 2019-10-13T10:51:15.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:15.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:15.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:15.365+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.369454+0000) 2019-10-13T10:51:16.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:16.205+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.206301+0000) 2019-10-13T10:51:16.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:16.356+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.360856+0000) 2019-10-13T10:51:18.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:18.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:18.623970+0000) 2019-10-13T10:51:18.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:18.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:18.629264+0000) 2019-10-13T10:51:20.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:20.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.370222+0000) 2019-10-13T10:51:20.372 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:20.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.370475+0000) 2019-10-13T10:51:20.373 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:20.363+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.370975+0000) 2019-10-13T10:51:20.373 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:20.363+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.371157+0000) 2019-10-13T10:51:20.373 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:20.367+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.372057+0000) 2019-10-13T10:51:20.374 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:20.367+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.373073+0000) 2019-10-13T10:51:20.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:20.367+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.374060+0000) 2019-10-13T10:51:20.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:20.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:20.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:20.377+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.380551+0000) 2019-10-13T10:51:21.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:21.887+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.892039+0000) 2019-10-13T10:51:21.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:21.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.924347+0000) 2019-10-13T10:51:22.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:22.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:22.115055+0000) 2019-10-13T10:51:23.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:23.109+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:23.111243+0000) 2019-10-13T10:51:25.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:25.375+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.379756+0000) 2019-10-13T10:51:25.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:25.375+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.380450+0000) 2019-10-13T10:51:25.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:25.375+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.381048+0000) 2019-10-13T10:51:25.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:25.375+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.381526+0000) 2019-10-13T10:51:25.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:25.377+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.381831+0000) 2019-10-13T10:51:25.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:25.375+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.382457+0000) 2019-10-13T10:51:25.383 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:25.375+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.382680+0000) 2019-10-13T10:51:25.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:25.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:25.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:25.389+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.390213+0000) 2019-10-13T10:51:26.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:26.209+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.212692+0000) 2019-10-13T10:51:26.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:26.355+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.361039+0000) 2019-10-13T10:51:28.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:28.623+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:28.624166+0000) 2019-10-13T10:51:28.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:28.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:28.629454+0000) 2019-10-13T10:51:30.389 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:30.385+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.387769+0000) 2019-10-13T10:51:30.389 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:30.383+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.388385+0000) 2019-10-13T10:51:30.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:30.383+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.389168+0000) 2019-10-13T10:51:30.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:30.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.389656+0000) 2019-10-13T10:51:30.391 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:30.383+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.390184+0000) 2019-10-13T10:51:30.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:30.383+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.390572+0000) 2019-10-13T10:51:30.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:30.383+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.390976+0000) 2019-10-13T10:51:30.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:30.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:30.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:30.393+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.397867+0000) 2019-10-13T10:51:31.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:31.887+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.892255+0000) 2019-10-13T10:51:31.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:31.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.924549+0000) 2019-10-13T10:51:32.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:32.107+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:32.115230+0000) 2019-10-13T10:51:33.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:33.109+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:33.111433+0000) 2019-10-13T10:51:33.985 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:51:33.985 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:33.989 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:51:33.989 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:35.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:35.391+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.397261+0000) 2019-10-13T10:51:35.399 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:35.397+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.398202+0000) 2019-10-13T10:51:35.399 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:35.391+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.398420+0000) 2019-10-13T10:51:35.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:35.395+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.399500+0000) 2019-10-13T10:51:35.401 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:35.395+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.399503+0000) 2019-10-13T10:51:35.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:35.395+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.399824+0000) 2019-10-13T10:51:35.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:35.395+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.400827+0000) 2019-10-13T10:51:35.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:35.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:35.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:35.405+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.407157+0000) 2019-10-13T10:51:36.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:36.217+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.219102+0000) 2019-10-13T10:51:36.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:36.359+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.361205+0000) 2019-10-13T10:51:38.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:38.619+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:38.624387+0000) 2019-10-13T10:51:38.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:38.623+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:38.629665+0000) 2019-10-13T10:51:40.408 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:40.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.406685+0000) 2019-10-13T10:51:40.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:40.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.406909+0000) 2019-10-13T10:51:40.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:40.403+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.408468+0000) 2019-10-13T10:51:40.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:40.405+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.409036+0000) 2019-10-13T10:51:40.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:40.403+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.409550+0000) 2019-10-13T10:51:40.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:40.403+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.409854+0000) 2019-10-13T10:51:40.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:40.403+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.410442+0000) 2019-10-13T10:51:40.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:40.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:40.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:40.413+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.415688+0000) 2019-10-13T10:51:41.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:41.887+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.892472+0000) 2019-10-13T10:51:41.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:41.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.924748+0000) 2019-10-13T10:51:42.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:42.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:42.115460+0000) 2019-10-13T10:51:43.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:43.108+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:43.111643+0000) 2019-10-13T10:51:45.417 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:45.411+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.415749+0000) 2019-10-13T10:51:45.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:45.411+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.415591+0000) 2019-10-13T10:51:45.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:45.412+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.416937+0000) 2019-10-13T10:51:45.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:45.411+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.416539+0000) 2019-10-13T10:51:45.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:45.411+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.417289+0000) 2019-10-13T10:51:45.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:45.411+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.417768+0000) 2019-10-13T10:51:45.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:45.411+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.418611+0000) 2019-10-13T10:51:45.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:45.415+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 109970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:45.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:45.424+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.426398+0000) 2019-10-13T10:51:46.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:46.224+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:46.225403+0000) 2019-10-13T10:51:46.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:46.355+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:46.361420+0000) 2019-10-13T10:51:48.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:48.623+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:48.624617+0000) 2019-10-13T10:51:48.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:48.627+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:48.629853+0000) 2019-10-13T10:51:50.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:50.423+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.423675+0000) 2019-10-13T10:51:50.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:50.423+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.423968+0000) 2019-10-13T10:51:50.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:50.423+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.424719+0000) 2019-10-13T10:51:50.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:50.423+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.425106+0000) 2019-10-13T10:51:50.427 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:50.420+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.425801+0000) 2019-10-13T10:51:50.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:50.423+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.426121+0000) 2019-10-13T10:51:50.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:50.423+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.426657+0000) 2019-10-13T10:51:50.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:50.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:50.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:50.428+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.432952+0000) 2019-10-13T10:51:51.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:51.891+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.892703+0000) 2019-10-13T10:51:51.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:51.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.924947+0000) 2019-10-13T10:51:52.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:52.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.115650+0000) 2019-10-13T10:51:53.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:53.108+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:53.111837+0000) 2019-10-13T10:51:55.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:55.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.432014+0000) 2019-10-13T10:51:55.434 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:51:55.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.433279+0000) 2019-10-13T10:51:55.434 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:55.431+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.433581+0000) 2019-10-13T10:51:55.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:55.431+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.434397+0000) 2019-10-13T10:51:55.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:51:55.431+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.434098+0000) 2019-10-13T10:51:55.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:51:55.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.434826+0000) 2019-10-13T10:51:55.436 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:51:55.435+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.435973+0000) 2019-10-13T10:51:55.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:51:55.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:51:55.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:55.436+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.442003+0000) 2019-10-13T10:51:56.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:51:56.228+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:56.231838+0000) 2019-10-13T10:51:56.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:51:56.359+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:56.361604+0000) 2019-10-13T10:51:58.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:51:58.623+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:58.624847+0000) 2019-10-13T10:51:58.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:51:58.627+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:58.630046+0000) 2019-10-13T10:52:00.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:00.439+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.441782+0000) 2019-10-13T10:52:00.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:00.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.442717+0000) 2019-10-13T10:52:00.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:00.440+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.443447+0000) 2019-10-13T10:52:00.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:00.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.443621+0000) 2019-10-13T10:52:00.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:00.443+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.444149+0000) 2019-10-13T10:52:00.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:00.443+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.444248+0000) 2019-10-13T10:52:00.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:00.443+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.445068+0000) 2019-10-13T10:52:00.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:00.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:00.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:00.452+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.454300+0000) 2019-10-13T10:52:01.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:01.891+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.892919+0000) 2019-10-13T10:52:01.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:01.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.925151+0000) 2019-10-13T10:52:02.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:02.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.115871+0000) 2019-10-13T10:52:03.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:03.108+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:03.112028+0000) 2019-10-13T10:52:04.125 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:52:04.125 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:04.128 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:52:04.129 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:05.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:05.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.453252+0000) 2019-10-13T10:52:05.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:05.446+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.450559+0000) 2019-10-13T10:52:05.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:05.446+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.450939+0000) 2019-10-13T10:52:05.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:05.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.451983+0000) 2019-10-13T10:52:05.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:05.450+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.452096+0000) 2019-10-13T10:52:05.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:05.450+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.453773+0000) 2019-10-13T10:52:05.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:05.450+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.454558+0000) 2019-10-13T10:52:05.461 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:05.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:05.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:05.456+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.460758+0000) 2019-10-13T10:52:06.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:06.236+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:06.238187+0000) 2019-10-13T10:52:06.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:06.358+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:06.361796+0000) 2019-10-13T10:52:08.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:08.622+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:08.625046+0000) 2019-10-13T10:52:08.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:08.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:08.630241+0000) 2019-10-13T10:52:10.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:10.458+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.460119+0000) 2019-10-13T10:52:10.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:10.458+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.460807+0000) 2019-10-13T10:52:10.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:10.456+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.461350+0000) 2019-10-13T10:52:10.462 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:10.458+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.461883+0000) 2019-10-13T10:52:10.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:10.458+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.462298+0000) 2019-10-13T10:52:10.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:10.458+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.462628+0000) 2019-10-13T10:52:10.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:10.458+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.463013+0000) 2019-10-13T10:52:10.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:10.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:10.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:10.464+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.469513+0000) 2019-10-13T10:52:11.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:11.890+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.893151+0000) 2019-10-13T10:52:11.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:11.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.925394+0000) 2019-10-13T10:52:12.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:12.114+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.116106+0000) 2019-10-13T10:52:13.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:13.108+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:13.112226+0000) 2019-10-13T10:52:15.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:15.466+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.468293+0000) 2019-10-13T10:52:15.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:15.466+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.468966+0000) 2019-10-13T10:52:15.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:15.466+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.469659+0000) 2019-10-13T10:52:15.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:15.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.470596+0000) 2019-10-13T10:52:15.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:15.466+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.471153+0000) 2019-10-13T10:52:15.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:15.468+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.471849+0000) 2019-10-13T10:52:15.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:15.472+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.476314+0000) 2019-10-13T10:52:15.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:15.470+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.472375+0000) 2019-10-13T10:52:15.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:15.470+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:16.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:16.244+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:16.244773+0000) 2019-10-13T10:52:16.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:16.358+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:16.361990+0000) 2019-10-13T10:52:18.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:18.622+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:18.625200+0000) 2019-10-13T10:52:18.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:18.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:18.630436+0000) 2019-10-13T10:52:20.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:20.472+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.477485+0000) 2019-10-13T10:52:20.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:20.474+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.478133+0000) 2019-10-13T10:52:20.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:20.474+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.478478+0000) 2019-10-13T10:52:20.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:20.474+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.479105+0000) 2019-10-13T10:52:20.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:20.478+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.479391+0000) 2019-10-13T10:52:20.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:20.478+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.480717+0000) 2019-10-13T10:52:20.482 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:20.478+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.481308+0000) 2019-10-13T10:52:20.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:20.482+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:20.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:20.484+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.486530+0000) 2019-10-13T10:52:21.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:21.890+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.893356+0000) 2019-10-13T10:52:21.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:21.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.925593+0000) 2019-10-13T10:52:22.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:22.114+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.116308+0000) 2019-10-13T10:52:23.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:23.112+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:23.112419+0000) 2019-10-13T10:52:25.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:25.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.486907+0000) 2019-10-13T10:52:25.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:25.486+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.487808+0000) 2019-10-13T10:52:25.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:25.484+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.488323+0000) 2019-10-13T10:52:25.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:25.486+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.488961+0000) 2019-10-13T10:52:25.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:25.486+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.489054+0000) 2019-10-13T10:52:25.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:25.486+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.489439+0000) 2019-10-13T10:52:25.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:25.486+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.489720+0000) 2019-10-13T10:52:25.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:25.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:25.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:25.496+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.499240+0000) 2019-10-13T10:52:26.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:26.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:26.251369+0000) 2019-10-13T10:52:26.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:26.358+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:26.362166+0000) 2019-10-13T10:52:28.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:28.622+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:28.625471+0000) 2019-10-13T10:52:28.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:28.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:28.630660+0000) 2019-10-13T10:52:30.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:30.494+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.495856+0000) 2019-10-13T10:52:30.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:30.492+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.496530+0000) 2019-10-13T10:52:30.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:30.494+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.496694+0000) 2019-10-13T10:52:30.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:30.494+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.497310+0000) 2019-10-13T10:52:30.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:30.494+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.497224+0000) 2019-10-13T10:52:30.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:30.494+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.497661+0000) 2019-10-13T10:52:30.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:30.494+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.498202+0000) 2019-10-13T10:52:30.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:30.498+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:30.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:30.500+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.505839+0000) 2019-10-13T10:52:31.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:31.890+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.893531+0000) 2019-10-13T10:52:31.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:31.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.925818+0000) 2019-10-13T10:52:32.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:32.114+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.116511+0000) 2019-10-13T10:52:33.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:33.112+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:33.112618+0000) 2019-10-13T10:52:34.240 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:52:34.240 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:34.244 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:52:34.244 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:35.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:35.502+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.504187+0000) 2019-10-13T10:52:35.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:35.502+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.504995+0000) 2019-10-13T10:52:35.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:35.502+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.505398+0000) 2019-10-13T10:52:35.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:35.502+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.505729+0000) 2019-10-13T10:52:35.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:35.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.506654+0000) 2019-10-13T10:52:35.509 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:35.502+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.506344+0000) 2019-10-13T10:52:35.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:35.506+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.507469+0000) 2019-10-13T10:52:35.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:35.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:35.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:35.508+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.512104+0000) 2019-10-13T10:52:36.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:36.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.257541+0000) 2019-10-13T10:52:36.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:36.358+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.362366+0000) 2019-10-13T10:52:38.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:38.622+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:38.625671+0000) 2019-10-13T10:52:38.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:38.626+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:38.630859+0000) 2019-10-13T10:52:40.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:40.510+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.513136+0000) 2019-10-13T10:52:40.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:40.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.513563+0000) 2019-10-13T10:52:40.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:40.510+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.513970+0000) 2019-10-13T10:52:40.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:40.511+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.514740+0000) 2019-10-13T10:52:40.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:40.510+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.514606+0000) 2019-10-13T10:52:40.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:40.514+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.515463+0000) 2019-10-13T10:52:40.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:40.514+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.516132+0000) 2019-10-13T10:52:40.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:40.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:40.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:40.515+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:40.522104+0000) 2019-10-13T10:52:41.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:41.890+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.893710+0000) 2019-10-13T10:52:41.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:41.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.925997+0000) 2019-10-13T10:52:42.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:42.114+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.116728+0000) 2019-10-13T10:52:43.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:43.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:43.112811+0000) 2019-10-13T10:52:45.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:45.518+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.522084+0000) 2019-10-13T10:52:45.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:45.518+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.522660+0000) 2019-10-13T10:52:45.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:45.518+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.523044+0000) 2019-10-13T10:52:45.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:45.522+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.523350+0000) 2019-10-13T10:52:45.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:45.522+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.523670+0000) 2019-10-13T10:52:45.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:45.522+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.524978+0000) 2019-10-13T10:52:45.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:45.519+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.525499+0000) 2019-10-13T10:52:45.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:45.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:45.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:45.527+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.531184+0000) 2019-10-13T10:52:46.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:46.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.263982+0000) 2019-10-13T10:52:46.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:46.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.362539+0000) 2019-10-13T10:52:48.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:48.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:48.625906+0000) 2019-10-13T10:52:48.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:48.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:48.631063+0000) 2019-10-13T10:52:50.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:50.525+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.531073+0000) 2019-10-13T10:52:50.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:50.529+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.531798+0000) 2019-10-13T10:52:50.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:50.529+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.532143+0000) 2019-10-13T10:52:50.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:50.529+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.532583+0000) 2019-10-13T10:52:50.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:50.529+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.533089+0000) 2019-10-13T10:52:50.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:50.527+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.533839+0000) 2019-10-13T10:52:50.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:50.529+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.534756+0000) 2019-10-13T10:52:50.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:50.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:50.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:50.539+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:50.542563+0000) 2019-10-13T10:52:51.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:51.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.893947+0000) 2019-10-13T10:52:51.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:51.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.926234+0000) 2019-10-13T10:52:52.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:52.113+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.116925+0000) 2019-10-13T10:52:53.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:53.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:53.112997+0000) 2019-10-13T10:52:55.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:55.537+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.540476+0000) 2019-10-13T10:52:55.542 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:52:55.537+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.541112+0000) 2019-10-13T10:52:55.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:52:55.537+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.541792+0000) 2019-10-13T10:52:55.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:52:55.539+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.542215+0000) 2019-10-13T10:52:55.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:55.537+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.542608+0000) 2019-10-13T10:52:55.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:52:55.537+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.543062+0000) 2019-10-13T10:52:55.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:55.541+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.543578+0000) 2019-10-13T10:52:55.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:52:55.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:52:55.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:55.547+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.550225+0000) 2019-10-13T10:52:56.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:52:56.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.270402+0000) 2019-10-13T10:52:56.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:52:56.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.362728+0000) 2019-10-13T10:52:58.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:52:58.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:58.626143+0000) 2019-10-13T10:52:58.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:52:58.625+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:58.631269+0000) 2019-10-13T10:53:00.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:00.545+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.549358+0000) 2019-10-13T10:53:00.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:00.545+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.550320+0000) 2019-10-13T10:53:00.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:00.547+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.550734+0000) 2019-10-13T10:53:00.552 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:00.545+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.551222+0000) 2019-10-13T10:53:00.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:00.549+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.551644+0000) 2019-10-13T10:53:00.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:00.549+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.551979+0000) 2019-10-13T10:53:00.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:00.549+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.552513+0000) 2019-10-13T10:53:00.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:00.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:00.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:00.551+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:00.556361+0000) 2019-10-13T10:53:01.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:01.893+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.894156+0000) 2019-10-13T10:53:01.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:01.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.926441+0000) 2019-10-13T10:53:02.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:02.113+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.117126+0000) 2019-10-13T10:53:03.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:03.111+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:03.113191+0000) 2019-10-13T10:53:04.369 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:53:04.369 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:04.373 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:53:04.373 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:05.559 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:05.553+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.558306+0000) 2019-10-13T10:53:05.560 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:05.555+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.559014+0000) 2019-10-13T10:53:05.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:05.557+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.559640+0000) 2019-10-13T10:53:05.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:05.557+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.559767+0000) 2019-10-13T10:53:05.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:05.557+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.560586+0000) 2019-10-13T10:53:05.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:05.557+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.561141+0000) 2019-10-13T10:53:05.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:05.557+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.562038+0000) 2019-10-13T10:53:05.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:05.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:05.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:05.563+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.566887+0000) 2019-10-13T10:53:06.278 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:06.271+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.277215+0000) 2019-10-13T10:53:06.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:06.361+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.362866+0000) 2019-10-13T10:53:08.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:08.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:08.626345+0000) 2019-10-13T10:53:08.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:08.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:08.631505+0000) 2019-10-13T10:53:10.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:10.561+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.567200+0000) 2019-10-13T10:53:10.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:10.565+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.567536+0000) 2019-10-13T10:53:10.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:10.565+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.568780+0000) 2019-10-13T10:53:10.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:10.565+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.569571+0000) 2019-10-13T10:53:10.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:10.563+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.569930+0000) 2019-10-13T10:53:10.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:10.565+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.570446+0000) 2019-10-13T10:53:10.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:10.565+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.571168+0000) 2019-10-13T10:53:10.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:10.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:10.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:10.571+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:10.577581+0000) 2019-10-13T10:53:11.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:11.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.894341+0000) 2019-10-13T10:53:11.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:11.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.926677+0000) 2019-10-13T10:53:12.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:12.113+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.117330+0000) 2019-10-13T10:53:13.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:13.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:13.113387+0000) 2019-10-13T10:53:15.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:15.573+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.576095+0000) 2019-10-13T10:53:15.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:15.573+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.576867+0000) 2019-10-13T10:53:15.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:15.573+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.577437+0000) 2019-10-13T10:53:15.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:15.573+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.577902+0000) 2019-10-13T10:53:15.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:15.573+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.578473+0000) 2019-10-13T10:53:15.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:15.575+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.579000+0000) 2019-10-13T10:53:15.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:15.573+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.579220+0000) 2019-10-13T10:53:15.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:15.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:15.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:15.579+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.585821+0000) 2019-10-13T10:53:16.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:16.283+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.283647+0000) 2019-10-13T10:53:16.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:16.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.363049+0000) 2019-10-13T10:53:18.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:18.625+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:18.626550+0000) 2019-10-13T10:53:18.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:18.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:18.631703+0000) 2019-10-13T10:53:20.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:20.581+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.585338+0000) 2019-10-13T10:53:20.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:20.581+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.585989+0000) 2019-10-13T10:53:20.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:20.581+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.586530+0000) 2019-10-13T10:53:20.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:20.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.587949+0000) 2019-10-13T10:53:20.589 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:20.583+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.588381+0000) 2019-10-13T10:53:20.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:20.585+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.588626+0000) 2019-10-13T10:53:20.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:20.585+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.589066+0000) 2019-10-13T10:53:20.592 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:20.585+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:20.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:20.587+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:20.593607+0000) 2019-10-13T10:53:21.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:21.889+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.894543+0000) 2019-10-13T10:53:21.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:21.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.926865+0000) 2019-10-13T10:53:22.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:22.113+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.117510+0000) 2019-10-13T10:53:23.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:23.107+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:23.113599+0000) 2019-10-13T10:53:25.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:25.589+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.594295+0000) 2019-10-13T10:53:25.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:25.589+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.595019+0000) 2019-10-13T10:53:25.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:25.593+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.595338+0000) 2019-10-13T10:53:25.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:25.593+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.596307+0000) 2019-10-13T10:53:25.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:25.591+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.596578+0000) 2019-10-13T10:53:25.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:25.593+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.597385+0000) 2019-10-13T10:53:25.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:25.593+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.597778+0000) 2019-10-13T10:53:25.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:25.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:25.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:25.599+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.602649+0000) 2019-10-13T10:53:26.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:26.287+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.289860+0000) 2019-10-13T10:53:26.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:26.357+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.363247+0000) 2019-10-13T10:53:28.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:28.621+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:28.626749+0000) 2019-10-13T10:53:28.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:28.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:28.631917+0000) 2019-10-13T10:53:30.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:30.596+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.603158+0000) 2019-10-13T10:53:30.605 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:30.599+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.604338+0000) 2019-10-13T10:53:30.605 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:30.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.604833+0000) 2019-10-13T10:53:30.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:30.600+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.605233+0000) 2019-10-13T10:53:30.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:30.600+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.606060+0000) 2019-10-13T10:53:30.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:30.600+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.606532+0000) 2019-10-13T10:53:30.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:30.600+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.606942+0000) 2019-10-13T10:53:30.609 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:30.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:30.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:30.607+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:30.611998+0000) 2019-10-13T10:53:31.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:31.892+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.894741+0000) 2019-10-13T10:53:31.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:31.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.927064+0000) 2019-10-13T10:53:32.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:32.116+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.117679+0000) 2019-10-13T10:53:33.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:33.106+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:33.113809+0000) 2019-10-13T10:53:34.509 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:53:34.509 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:34.514 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:53:34.514 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:35.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:35.608+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.612462+0000) 2019-10-13T10:53:35.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:35.608+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.613095+0000) 2019-10-13T10:53:35.615 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:35.606+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.613434+0000) 2019-10-13T10:53:35.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:35.608+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.613637+0000) 2019-10-13T10:53:35.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:35.608+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.613941+0000) 2019-10-13T10:53:35.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:35.608+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.614246+0000) 2019-10-13T10:53:35.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:35.608+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.615119+0000) 2019-10-13T10:53:35.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:35.612+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:35.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:35.614+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.618563+0000) 2019-10-13T10:53:36.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:36.294+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.296571+0000) 2019-10-13T10:53:36.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:36.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.363418+0000) 2019-10-13T10:53:38.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:38.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:38.626957+0000) 2019-10-13T10:53:38.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:38.628+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:38.632146+0000) 2019-10-13T10:53:40.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:40.614+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.620456+0000) 2019-10-13T10:53:40.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:40.616+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.620882+0000) 2019-10-13T10:53:40.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:40.616+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.621348+0000) 2019-10-13T10:53:40.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:40.616+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.621761+0000) 2019-10-13T10:53:40.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:40.616+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.622261+0000) 2019-10-13T10:53:40.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:40.616+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.622467+0000) 2019-10-13T10:53:40.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:40.620+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.623748+0000) 2019-10-13T10:53:40.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:40.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:40.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:40.626+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:40.630554+0000) 2019-10-13T10:53:41.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:41.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.894945+0000) 2019-10-13T10:53:41.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:41.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.927243+0000) 2019-10-13T10:53:42.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:42.112+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.117899+0000) 2019-10-13T10:53:43.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:43.110+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:43.114000+0000) 2019-10-13T10:53:45.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:45.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.629946+0000) 2019-10-13T10:53:45.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:45.624+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.630509+0000) 2019-10-13T10:53:45.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:45.624+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.631041+0000) 2019-10-13T10:53:45.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:45.628+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.632217+0000) 2019-10-13T10:53:45.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:45.626+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.632690+0000) 2019-10-13T10:53:45.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:45.628+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.633242+0000) 2019-10-13T10:53:45.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:45.628+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.633543+0000) 2019-10-13T10:53:45.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:45.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:45.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:45.634+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.638622+0000) 2019-10-13T10:53:46.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:46.302+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.302878+0000) 2019-10-13T10:53:46.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:46.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.363566+0000) 2019-10-13T10:53:48.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:48.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:48.627180+0000) 2019-10-13T10:53:48.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:48.628+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:48.632378+0000) 2019-10-13T10:53:50.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:50.632+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.639033+0000) 2019-10-13T10:53:50.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:50.636+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.639397+0000) 2019-10-13T10:53:50.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:50.636+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.640068+0000) 2019-10-13T10:53:50.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:50.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.640351+0000) 2019-10-13T10:53:50.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:50.636+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.641105+0000) 2019-10-13T10:53:50.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:50.636+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.642051+0000) 2019-10-13T10:53:50.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:50.638+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.642573+0000) 2019-10-13T10:53:50.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:50.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110815 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:50.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:50.642+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:50.647770+0000) 2019-10-13T10:53:51.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:51.888+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.895165+0000) 2019-10-13T10:53:51.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:51.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.927477+0000) 2019-10-13T10:53:52.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:52.116+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.118083+0000) 2019-10-13T10:53:53.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:53.110+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:53.114180+0000) 2019-10-13T10:53:55.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:55.644+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.647645+0000) 2019-10-13T10:53:55.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:55.644+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.648039+0000) 2019-10-13T10:53:55.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:53:55.644+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.649380+0000) 2019-10-13T10:53:55.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:53:55.644+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.649600+0000) 2019-10-13T10:53:55.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:53:55.644+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.649993+0000) 2019-10-13T10:53:55.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:55.644+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.650673+0000) 2019-10-13T10:53:55.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:53:55.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.651263+0000) 2019-10-13T10:53:55.654 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:53:55.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:53:55.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:55.650+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.656851+0000) 2019-10-13T10:53:56.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:53:56.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.309284+0000) 2019-10-13T10:53:56.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:53:56.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.363743+0000) 2019-10-13T10:53:58.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:53:58.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:58.627378+0000) 2019-10-13T10:53:58.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:53:58.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:58.632569+0000) 2019-10-13T10:54:00.658 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:00.652+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.657070+0000) 2019-10-13T10:54:00.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:00.652+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.657433+0000) 2019-10-13T10:54:00.659 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:00.652+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.658037+0000) 2019-10-13T10:54:00.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:00.654+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.658247+0000) 2019-10-13T10:54:00.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:00.652+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.658544+0000) 2019-10-13T10:54:00.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:00.656+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.659386+0000) 2019-10-13T10:54:00.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:00.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.661176+0000) 2019-10-13T10:54:00.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:00.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:00.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:00.662+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:00.666846+0000) 2019-10-13T10:54:01.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:01.892+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.895377+0000) 2019-10-13T10:54:01.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:01.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.927670+0000) 2019-10-13T10:54:02.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:02.112+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.118279+0000) 2019-10-13T10:54:03.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:03.110+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:03.114370+0000) 2019-10-13T10:54:04.634 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:54:04.634 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:04.638 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:54:04.638 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:05.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:05.660+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.664837+0000) 2019-10-13T10:54:05.666 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:05.660+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.665538+0000) 2019-10-13T10:54:05.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:05.658+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.665991+0000) 2019-10-13T10:54:05.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:05.660+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.666276+0000) 2019-10-13T10:54:05.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:05.660+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.666698+0000) 2019-10-13T10:54:05.668 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:05.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.667396+0000) 2019-10-13T10:54:05.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:05.664+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.667762+0000) 2019-10-13T10:54:05.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:05.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:05.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:05.670+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.674749+0000) 2019-10-13T10:54:06.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:06.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.315675+0000) 2019-10-13T10:54:06.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:06.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.363937+0000) 2019-10-13T10:54:08.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:08.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:08.627583+0000) 2019-10-13T10:54:08.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:08.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:08.632768+0000) 2019-10-13T10:54:10.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:10.668+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.673820+0000) 2019-10-13T10:54:10.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:10.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.674238+0000) 2019-10-13T10:54:10.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:10.668+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.675003+0000) 2019-10-13T10:54:10.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:10.672+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.675519+0000) 2019-10-13T10:54:10.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:10.670+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.676149+0000) 2019-10-13T10:54:10.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:10.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.676568+0000) 2019-10-13T10:54:10.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:10.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.677235+0000) 2019-10-13T10:54:10.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:10.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:10.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:10.678+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:10.685219+0000) 2019-10-13T10:54:11.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:11.892+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.895580+0000) 2019-10-13T10:54:11.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:11.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.927901+0000) 2019-10-13T10:54:12.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:12.116+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.118506+0000) 2019-10-13T10:54:13.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:13.110+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:13.114564+0000) 2019-10-13T10:54:15.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:15.679+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.683356+0000) 2019-10-13T10:54:15.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:15.679+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.683666+0000) 2019-10-13T10:54:15.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:15.679+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.684135+0000) 2019-10-13T10:54:15.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:15.679+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.684761+0000) 2019-10-13T10:54:15.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:15.678+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.684911+0000) 2019-10-13T10:54:15.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:15.679+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.685247+0000) 2019-10-13T10:54:15.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:15.679+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.685541+0000) 2019-10-13T10:54:15.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:15.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 110983 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:15.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:15.686+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.692933+0000) 2019-10-13T10:54:16.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:16.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.321970+0000) 2019-10-13T10:54:16.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:16.359+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.364177+0000) 2019-10-13T10:54:18.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:18.623+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:18.627806+0000) 2019-10-13T10:54:18.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:18.627+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:18.632965+0000) 2019-10-13T10:54:20.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:20.683+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.690642+0000) 2019-10-13T10:54:20.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:20.683+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.690897+0000) 2019-10-13T10:54:20.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:20.687+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.692033+0000) 2019-10-13T10:54:20.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:20.687+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.692424+0000) 2019-10-13T10:54:20.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:20.687+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.692629+0000) 2019-10-13T10:54:20.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:20.690+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.693920+0000) 2019-10-13T10:54:20.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:20.687+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.694512+0000) 2019-10-13T10:54:20.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:20.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:20.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:20.694+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:20.698026+0000) 2019-10-13T10:54:21.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:21.891+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.895781+0000) 2019-10-13T10:54:21.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:21.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.928110+0000) 2019-10-13T10:54:22.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:22.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.118734+0000) 2019-10-13T10:54:23.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:23.114+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:23.114757+0000) 2019-10-13T10:54:25.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:25.695+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.699376+0000) 2019-10-13T10:54:25.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:25.695+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.699820+0000) 2019-10-13T10:54:25.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:25.695+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.701153+0000) 2019-10-13T10:54:25.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:25.698+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.701743+0000) 2019-10-13T10:54:25.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:25.695+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.702601+0000) 2019-10-13T10:54:25.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:25.695+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.702360+0000) 2019-10-13T10:54:25.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:25.695+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.702964+0000) 2019-10-13T10:54:25.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:25.699+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:25.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:25.706+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.709781+0000) 2019-10-13T10:54:26.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:26.330+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.333296+0000) 2019-10-13T10:54:26.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:26.359+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.364352+0000) 2019-10-13T10:54:28.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:28.627+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:28.627999+0000) 2019-10-13T10:54:28.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:28.631+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:28.633161+0000) 2019-10-13T10:54:30.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:30.703+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.709235+0000) 2019-10-13T10:54:30.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:30.705+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.710096+0000) 2019-10-13T10:54:30.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:30.703+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.709996+0000) 2019-10-13T10:54:30.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:30.703+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.710549+0000) 2019-10-13T10:54:30.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:30.707+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.711283+0000) 2019-10-13T10:54:30.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:30.707+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.712105+0000) 2019-10-13T10:54:30.713 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:30.707+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.712753+0000) 2019-10-13T10:54:30.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:30.707+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:30.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:30.713+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:30.718083+0000) 2019-10-13T10:54:31.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:31.891+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.896010+0000) 2019-10-13T10:54:31.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:31.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.928355+0000) 2019-10-13T10:54:32.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:32.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.118958+0000) 2019-10-13T10:54:33.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:33.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:33.114955+0000) 2019-10-13T10:54:34.743 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:54:34.743 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:34.748 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:54:34.748 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:35.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:35.711+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.717525+0000) 2019-10-13T10:54:35.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:35.711+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.718197+0000) 2019-10-13T10:54:35.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:35.717+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.718615+0000) 2019-10-13T10:54:35.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:35.711+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.718877+0000) 2019-10-13T10:54:35.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:35.715+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.719441+0000) 2019-10-13T10:54:35.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:35.715+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.719878+0000) 2019-10-13T10:54:35.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:35.715+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.720239+0000) 2019-10-13T10:54:35.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:35.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:35.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:35.721+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.723844+0000) 2019-10-13T10:54:36.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:36.337+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.339907+0000) 2019-10-13T10:54:36.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:36.359+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.364523+0000) 2019-10-13T10:54:38.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:38.623+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:38.628211+0000) 2019-10-13T10:54:38.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:38.631+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:38.633361+0000) 2019-10-13T10:54:40.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:40.723+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.727257+0000) 2019-10-13T10:54:40.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:40.723+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.728011+0000) 2019-10-13T10:54:40.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:40.723+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.728244+0000) 2019-10-13T10:54:40.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:40.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.728560+0000) 2019-10-13T10:54:40.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:40.725+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.728993+0000) 2019-10-13T10:54:40.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:40.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.729923+0000) 2019-10-13T10:54:40.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:40.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.730525+0000) 2019-10-13T10:54:40.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:40.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111150 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:40.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:40.733+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:40.736866+0000) 2019-10-13T10:54:41.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:41.891+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.896229+0000) 2019-10-13T10:54:41.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:41.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.928593+0000) 2019-10-13T10:54:42.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:42.111+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.119159+0000) 2019-10-13T10:54:43.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:43.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:43.115188+0000) 2019-10-13T10:54:45.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:45.735+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.735297+0000) 2019-10-13T10:54:45.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:45.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.735950+0000) 2019-10-13T10:54:45.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:45.735+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.735937+0000) 2019-10-13T10:54:45.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:45.735+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.736267+0000) 2019-10-13T10:54:45.738 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:45.735+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.737612+0000) 2019-10-13T10:54:45.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:45.737+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.738419+0000) 2019-10-13T10:54:45.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:45.735+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.738649+0000) 2019-10-13T10:54:45.742 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:45.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:45.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:45.741+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.744699+0000) 2019-10-13T10:54:46.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:46.345+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.346383+0000) 2019-10-13T10:54:46.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:46.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.364708+0000) 2019-10-13T10:54:48.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:48.627+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:48.628411+0000) 2019-10-13T10:54:48.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:48.631+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:48.633595+0000) 2019-10-13T10:54:50.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:50.743+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.744321+0000) 2019-10-13T10:54:50.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:50.743+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.744681+0000) 2019-10-13T10:54:50.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:50.743+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.745685+0000) 2019-10-13T10:54:50.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:50.741+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.746110+0000) 2019-10-13T10:54:50.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:50.743+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.746519+0000) 2019-10-13T10:54:50.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:50.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.747159+0000) 2019-10-13T10:54:50.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:50.743+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.747025+0000) 2019-10-13T10:54:50.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:50.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111217 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:50.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:50.749+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:50.752985+0000) 2019-10-13T10:54:51.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:51.895+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.896436+0000) 2019-10-13T10:54:51.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:51.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.928804+0000) 2019-10-13T10:54:52.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:52.119+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.119357+0000) 2019-10-13T10:54:53.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:53.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:53.115388+0000) 2019-10-13T10:54:55.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:55.751+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.753371+0000) 2019-10-13T10:54:55.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:54:55.751+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.753668+0000) 2019-10-13T10:54:55.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:55.751+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.754230+0000) 2019-10-13T10:54:55.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:54:55.753+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.755006+0000) 2019-10-13T10:54:55.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:54:55.755+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.755537+0000) 2019-10-13T10:54:55.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:54:55.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.756045+0000) 2019-10-13T10:54:55.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:55.755+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.756780+0000) 2019-10-13T10:54:55.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:54:55.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:54:55.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:55.757+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.761660+0000) 2019-10-13T10:54:56.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:54:56.349+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.352875+0000) 2019-10-13T10:54:56.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:54:56.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.364885+0000) 2019-10-13T10:54:58.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:54:58.626+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:58.628639+0000) 2019-10-13T10:54:58.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:54:58.630+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:58.633822+0000) 2019-10-13T10:55:00.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:00.758+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.761618+0000) 2019-10-13T10:55:00.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:00.758+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.763088+0000) 2019-10-13T10:55:00.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:00.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.763917+0000) 2019-10-13T10:55:00.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:00.761+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.764216+0000) 2019-10-13T10:55:00.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:00.762+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.764783+0000) 2019-10-13T10:55:00.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:00.762+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.765112+0000) 2019-10-13T10:55:00.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:00.762+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.765394+0000) 2019-10-13T10:55:00.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:00.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:00.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:00.765+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:00.770069+0000) 2019-10-13T10:55:01.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:01.894+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.896635+0000) 2019-10-13T10:55:01.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:01.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.929034+0000) 2019-10-13T10:55:02.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:02.118+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.119561+0000) 2019-10-13T10:55:03.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:03.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:03.115590+0000) 2019-10-13T10:55:04.879 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:55:04.880 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:04.883 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:55:04.883 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:05.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:05.766+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.770715+0000) 2019-10-13T10:55:05.772 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:05.766+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.771207+0000) 2019-10-13T10:55:05.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:05.770+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.771787+0000) 2019-10-13T10:55:05.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:05.769+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.772213+0000) 2019-10-13T10:55:05.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:05.770+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.772639+0000) 2019-10-13T10:55:05.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:05.770+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.773158+0000) 2019-10-13T10:55:05.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:05.770+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.773471+0000) 2019-10-13T10:55:05.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:05.774+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:05.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:05.773+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.777360+0000) 2019-10-13T10:55:06.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:06.357+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.359383+0000) 2019-10-13T10:55:06.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:06.362+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.365093+0000) 2019-10-13T10:55:08.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:08.626+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:08.628836+0000) 2019-10-13T10:55:08.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:08.630+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:08.634022+0000) 2019-10-13T10:55:10.780 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:10.778+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.779393+0000) 2019-10-13T10:55:10.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:10.778+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.780763+0000) 2019-10-13T10:55:10.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:10.778+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.780929+0000) 2019-10-13T10:55:10.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:10.778+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.781194+0000) 2019-10-13T10:55:10.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:10.781+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.782230+0000) 2019-10-13T10:55:10.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:10.778+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.782466+0000) 2019-10-13T10:55:10.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:10.778+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.782799+0000) 2019-10-13T10:55:10.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:10.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:10.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:10.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:10.791587+0000) 2019-10-13T10:55:11.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:11.894+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.896865+0000) 2019-10-13T10:55:11.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:11.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.929270+0000) 2019-10-13T10:55:12.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:12.118+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.119740+0000) 2019-10-13T10:55:13.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:13.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:13.115847+0000) 2019-10-13T10:55:15.789 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:15.786+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.787879+0000) 2019-10-13T10:55:15.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:15.786+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.788489+0000) 2019-10-13T10:55:15.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:15.786+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.789279+0000) 2019-10-13T10:55:15.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:15.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.789913+0000) 2019-10-13T10:55:15.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:15.789+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.790770+0000) 2019-10-13T10:55:15.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:15.786+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.791139+0000) 2019-10-13T10:55:15.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:15.790+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.791526+0000) 2019-10-13T10:55:15.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:15.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:15.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:15.797+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.798230+0000) 2019-10-13T10:55:16.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:16.362+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.365275+0000) 2019-10-13T10:55:16.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:16.365+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.365810+0000) 2019-10-13T10:55:18.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:18.626+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:18.629010+0000) 2019-10-13T10:55:18.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:18.630+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:18.634218+0000) 2019-10-13T10:55:20.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:20.794+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.797104+0000) 2019-10-13T10:55:20.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:20.794+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.797982+0000) 2019-10-13T10:55:20.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:20.794+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.798268+0000) 2019-10-13T10:55:20.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:20.797+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.799161+0000) 2019-10-13T10:55:20.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:20.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.799465+0000) 2019-10-13T10:55:20.801 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:20.798+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.800220+0000) 2019-10-13T10:55:20.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:20.798+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.800680+0000) 2019-10-13T10:55:20.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:20.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:20.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:20.801+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:20.805415+0000) 2019-10-13T10:55:21.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:21.894+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.897081+0000) 2019-10-13T10:55:21.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:21.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.929469+0000) 2019-10-13T10:55:22.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:22.118+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.119936+0000) 2019-10-13T10:55:23.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:23.113+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:23.116046+0000) 2019-10-13T10:55:25.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:25.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.805639+0000) 2019-10-13T10:55:25.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:25.802+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.806225+0000) 2019-10-13T10:55:25.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:25.802+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.806770+0000) 2019-10-13T10:55:25.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:25.802+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.807195+0000) 2019-10-13T10:55:25.808 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:25.804+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.807722+0000) 2019-10-13T10:55:25.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:25.806+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.808904+0000) 2019-10-13T10:55:25.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:25.806+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.809255+0000) 2019-10-13T10:55:25.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:25.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:25.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:25.812+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.814612+0000) 2019-10-13T10:55:26.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:26.362+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.365488+0000) 2019-10-13T10:55:26.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:26.368+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.372341+0000) 2019-10-13T10:55:28.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:28.626+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:28.629216+0000) 2019-10-13T10:55:28.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:28.630+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:28.634418+0000) 2019-10-13T10:55:30.816 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:30.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.814873+0000) 2019-10-13T10:55:30.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:30.812+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.815512+0000) 2019-10-13T10:55:30.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:30.814+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.816088+0000) 2019-10-13T10:55:30.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:30.814+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.817115+0000) 2019-10-13T10:55:30.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:30.814+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.817545+0000) 2019-10-13T10:55:30.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:30.814+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.818445+0000) 2019-10-13T10:55:30.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:30.814+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.818561+0000) 2019-10-13T10:55:30.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:30.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:30.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:30.824+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:30.827021+0000) 2019-10-13T10:55:31.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:31.894+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.897287+0000) 2019-10-13T10:55:31.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:31.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.929673+0000) 2019-10-13T10:55:32.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:32.118+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.120127+0000) 2019-10-13T10:55:33.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:33.112+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:33.116269+0000) 2019-10-13T10:55:35.015 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:55:35.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:35.019 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:55:35.019 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:35.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:35.822+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.824983+0000) 2019-10-13T10:55:35.826 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:35.822+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.825526+0000) 2019-10-13T10:55:35.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:35.820+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.826080+0000) 2019-10-13T10:55:35.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:35.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.826507+0000) 2019-10-13T10:55:35.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:35.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.827101+0000) 2019-10-13T10:55:35.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:35.822+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.826606+0000) 2019-10-13T10:55:35.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:35.826+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.827462+0000) 2019-10-13T10:55:35.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:35.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:35.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:35.828+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.833706+0000) 2019-10-13T10:55:36.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:36.362+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.365667+0000) 2019-10-13T10:55:36.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:36.376+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.378651+0000) 2019-10-13T10:55:38.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:38.626+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:38.629381+0000) 2019-10-13T10:55:38.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:38.630+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:38.634605+0000) 2019-10-13T10:55:40.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:40.830+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.834190+0000) 2019-10-13T10:55:40.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:40.830+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.834515+0000) 2019-10-13T10:55:40.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:40.830+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.835091+0000) 2019-10-13T10:55:40.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:40.834+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.835633+0000) 2019-10-13T10:55:40.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:40.834+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.836032+0000) 2019-10-13T10:55:40.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:40.834+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.836444+0000) 2019-10-13T10:55:40.838 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:40.832+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.837389+0000) 2019-10-13T10:55:40.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:40.838+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:40.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:40.840+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:40.843320+0000) 2019-10-13T10:55:41.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:41.893+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.897496+0000) 2019-10-13T10:55:41.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:41.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.929871+0000) 2019-10-13T10:55:42.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:42.117+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.120322+0000) 2019-10-13T10:55:43.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:43.112+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:43.116514+0000) 2019-10-13T10:55:45.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:45.841+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.843662+0000) 2019-10-13T10:55:45.845 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:45.841+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.843841+0000) 2019-10-13T10:55:45.846 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:45.841+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.844304+0000) 2019-10-13T10:55:45.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:45.841+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.844954+0000) 2019-10-13T10:55:45.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:45.841+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.845326+0000) 2019-10-13T10:55:45.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:45.840+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.846000+0000) 2019-10-13T10:55:45.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:45.841+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.846253+0000) 2019-10-13T10:55:45.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:45.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:45.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:45.852+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.855045+0000) 2019-10-13T10:55:46.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:46.361+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.365848+0000) 2019-10-13T10:55:46.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:46.380+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.385059+0000) 2019-10-13T10:55:48.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:48.625+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:48.629553+0000) 2019-10-13T10:55:48.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:48.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:48.634804+0000) 2019-10-13T10:55:50.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:50.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.851573+0000) 2019-10-13T10:55:50.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:50.849+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.852370+0000) 2019-10-13T10:55:50.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:50.849+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.852468+0000) 2019-10-13T10:55:50.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:50.849+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.853151+0000) 2019-10-13T10:55:50.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:50.849+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.854310+0000) 2019-10-13T10:55:50.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:50.849+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.854662+0000) 2019-10-13T10:55:50.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:50.848+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.854088+0000) 2019-10-13T10:55:50.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:50.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:50.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:50.856+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:50.859538+0000) 2019-10-13T10:55:51.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:51.893+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.897695+0000) 2019-10-13T10:55:51.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:51.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.930027+0000) 2019-10-13T10:55:52.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:52.117+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.120549+0000) 2019-10-13T10:55:53.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:53.116+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:53.116709+0000) 2019-10-13T10:55:55.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:55:55.856+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.860843+0000) 2019-10-13T10:55:55.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:55.857+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.861099+0000) 2019-10-13T10:55:55.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:55.857+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.861413+0000) 2019-10-13T10:55:55.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:55:55.857+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.862122+0000) 2019-10-13T10:55:55.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:55.857+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.862939+0000) 2019-10-13T10:55:55.864 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:55:55.861+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.863535+0000) 2019-10-13T10:55:55.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:55:55.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.863393+0000) 2019-10-13T10:55:55.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:55:55.861+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:55:55.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:55.864+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.867954+0000) 2019-10-13T10:55:56.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:55:56.365+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.366055+0000) 2019-10-13T10:55:56.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:55:56.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.391511+0000) 2019-10-13T10:55:58.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:55:58.625+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:58.629747+0000) 2019-10-13T10:55:58.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:55:58.629+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:58.635000+0000) 2019-10-13T10:56:00.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:00.865+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.869588+0000) 2019-10-13T10:56:00.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:00.868+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.870764+0000) 2019-10-13T10:56:00.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:00.865+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.871074+0000) 2019-10-13T10:56:00.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:00.865+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.871263+0000) 2019-10-13T10:56:00.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:00.869+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.871573+0000) 2019-10-13T10:56:00.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:00.869+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.871953+0000) 2019-10-13T10:56:00.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:00.869+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.872512+0000) 2019-10-13T10:56:00.875 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:00.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:00.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:00.872+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:00.876294+0000) 2019-10-13T10:56:01.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:01.897+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.897896+0000) 2019-10-13T10:56:01.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:01.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.930238+0000) 2019-10-13T10:56:02.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:02.117+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.120769+0000) 2019-10-13T10:56:03.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:03.116+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:03.116917+0000) 2019-10-13T10:56:05.152 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:56:05.153 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:05.156 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:56:05.156 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:05.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:05.873+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.877844+0000) 2019-10-13T10:56:05.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:05.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.878361+0000) 2019-10-13T10:56:05.880 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:05.877+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.879809+0000) 2019-10-13T10:56:05.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:05.876+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.880459+0000) 2019-10-13T10:56:05.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:05.877+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.880728+0000) 2019-10-13T10:56:05.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:05.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.881104+0000) 2019-10-13T10:56:05.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:05.877+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.881605+0000) 2019-10-13T10:56:05.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:05.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:05.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:05.880+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.885084+0000) 2019-10-13T10:56:06.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:06.365+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.366261+0000) 2019-10-13T10:56:06.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:06.396+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.397768+0000) 2019-10-13T10:56:08.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:08.629+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:08.629923+0000) 2019-10-13T10:56:08.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:08.633+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:08.635197+0000) 2019-10-13T10:56:10.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:10.881+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.885735+0000) 2019-10-13T10:56:10.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:10.881+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.887210+0000) 2019-10-13T10:56:10.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:10.885+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.887576+0000) 2019-10-13T10:56:10.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:10.884+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.888815+0000) 2019-10-13T10:56:10.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:10.885+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.888455+0000) 2019-10-13T10:56:10.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:10.885+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.889350+0000) 2019-10-13T10:56:10.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:10.885+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.889623+0000) 2019-10-13T10:56:10.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:10.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:10.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:10.888+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:10.892010+0000) 2019-10-13T10:56:11.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:11.893+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.898091+0000) 2019-10-13T10:56:11.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:11.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.930448+0000) 2019-10-13T10:56:12.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:12.117+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.121002+0000) 2019-10-13T10:56:13.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:13.116+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:13.117109+0000) 2019-10-13T10:56:15.896 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:15.889+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.894844+0000) 2019-10-13T10:56:15.897 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:15.893+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.896330+0000) 2019-10-13T10:56:15.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:15.893+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.896637+0000) 2019-10-13T10:56:15.898 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:15.893+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.896951+0000) 2019-10-13T10:56:15.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:15.893+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.897547+0000) 2019-10-13T10:56:15.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:15.893+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.897831+0000) 2019-10-13T10:56:15.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:15.896+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.898629+0000) 2019-10-13T10:56:15.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:15.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:15.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:15.896+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.900959+0000) 2019-10-13T10:56:16.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:16.361+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.366488+0000) 2019-10-13T10:56:16.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:16.400+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.404613+0000) 2019-10-13T10:56:18.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:18.625+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:18.630090+0000) 2019-10-13T10:56:18.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:18.633+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:18.635394+0000) 2019-10-13T10:56:20.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:20.897+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.902315+0000) 2019-10-13T10:56:20.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:20.901+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.903942+0000) 2019-10-13T10:56:20.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:20.901+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.904755+0000) 2019-10-13T10:56:20.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:20.901+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.904986+0000) 2019-10-13T10:56:20.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:20.899+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.905885+0000) 2019-10-13T10:56:20.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:20.901+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.906150+0000) 2019-10-13T10:56:20.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:20.901+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.906552+0000) 2019-10-13T10:56:20.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:20.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:20.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:20.903+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:20.909856+0000) 2019-10-13T10:56:21.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:21.897+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.898288+0000) 2019-10-13T10:56:21.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:21.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.930683+0000) 2019-10-13T10:56:22.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:22.117+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.121172+0000) 2019-10-13T10:56:23.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:23.115+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:23.117332+0000) 2019-10-13T10:56:25.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:25.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.911875+0000) 2019-10-13T10:56:25.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:25.908+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.912295+0000) 2019-10-13T10:56:25.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:25.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.913784+0000) 2019-10-13T10:56:25.915 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:25.908+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.914224+0000) 2019-10-13T10:56:25.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:25.908+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.914546+0000) 2019-10-13T10:56:25.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:25.911+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.915038+0000) 2019-10-13T10:56:25.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:25.912+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.915324+0000) 2019-10-13T10:56:25.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:25.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:25.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:25.915+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.920891+0000) 2019-10-13T10:56:26.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:26.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.366669+0000) 2019-10-13T10:56:26.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:26.411+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.415154+0000) 2019-10-13T10:56:28.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:28.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:28.630243+0000) 2019-10-13T10:56:28.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:28.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:28.635584+0000) 2019-10-13T10:56:30.922 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:30.916+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.921042+0000) 2019-10-13T10:56:30.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:30.916+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.921948+0000) 2019-10-13T10:56:30.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:30.916+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.922245+0000) 2019-10-13T10:56:30.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:30.916+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.922780+0000) 2019-10-13T10:56:30.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:30.916+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.922927+0000) 2019-10-13T10:56:30.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:30.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.923521+0000) 2019-10-13T10:56:30.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:30.920+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.923626+0000) 2019-10-13T10:56:30.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:30.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:30.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:30.923+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:30.928636+0000) 2019-10-13T10:56:31.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:31.896+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.898497+0000) 2019-10-13T10:56:31.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:31.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.930900+0000) 2019-10-13T10:56:32.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:32.116+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.121368+0000) 2019-10-13T10:56:33.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:33.115+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:33.117529+0000) 2019-10-13T10:56:35.233 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:56:35.233 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:35.236 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:56:35.237 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:35.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:35.924+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.930617+0000) 2019-10-13T10:56:35.932 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:35.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.930986+0000) 2019-10-13T10:56:35.933 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:35.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.932094+0000) 2019-10-13T10:56:35.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:35.928+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.932392+0000) 2019-10-13T10:56:35.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:35.928+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.933659+0000) 2019-10-13T10:56:35.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:35.928+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.933019+0000) 2019-10-13T10:56:35.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:35.927+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.933925+0000) 2019-10-13T10:56:35.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:35.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:35.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:35.935+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.938398+0000) 2019-10-13T10:56:36.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:36.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.366845+0000) 2019-10-13T10:56:36.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:36.415+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.421587+0000) 2019-10-13T10:56:38.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:38.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:38.630480+0000) 2019-10-13T10:56:38.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:38.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:38.635825+0000) 2019-10-13T10:56:40.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:40.935+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.939430+0000) 2019-10-13T10:56:40.941 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:40.936+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.939720+0000) 2019-10-13T10:56:40.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:40.936+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.940059+0000) 2019-10-13T10:56:40.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:40.936+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.940416+0000) 2019-10-13T10:56:40.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:40.936+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.940984+0000) 2019-10-13T10:56:40.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:40.936+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.941806+0000) 2019-10-13T10:56:40.943 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:40.936+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.942077+0000) 2019-10-13T10:56:40.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:40.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:40.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:40.947+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:40.950318+0000) 2019-10-13T10:56:41.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:41.896+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.898674+0000) 2019-10-13T10:56:41.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:41.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.931113+0000) 2019-10-13T10:56:42.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:42.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.121575+0000) 2019-10-13T10:56:43.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:43.115+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:43.117733+0000) 2019-10-13T10:56:45.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:45.944+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.948221+0000) 2019-10-13T10:56:45.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:45.944+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.949060+0000) 2019-10-13T10:56:45.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:45.944+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.949657+0000) 2019-10-13T10:56:45.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:45.944+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.949256+0000) 2019-10-13T10:56:45.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:45.944+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.950459+0000) 2019-10-13T10:56:45.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:45.947+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.951357+0000) 2019-10-13T10:56:45.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:45.948+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.952165+0000) 2019-10-13T10:56:45.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:45.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 111987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:45.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:45.951+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.957142+0000) 2019-10-13T10:56:46.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:46.364+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.367024+0000) 2019-10-13T10:56:46.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:46.423+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.428089+0000) 2019-10-13T10:56:48.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:48.628+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:48.630670+0000) 2019-10-13T10:56:48.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:48.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:48.636053+0000) 2019-10-13T10:56:50.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:50.952+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.956064+0000) 2019-10-13T10:56:50.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:50.952+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.957010+0000) 2019-10-13T10:56:50.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:50.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.957797+0000) 2019-10-13T10:56:50.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:50.952+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.958217+0000) 2019-10-13T10:56:50.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:50.955+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.958481+0000) 2019-10-13T10:56:50.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:50.952+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.958622+0000) 2019-10-13T10:56:50.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:50.952+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.958794+0000) 2019-10-13T10:56:50.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:50.956+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:50.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:50.959+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:50.965013+0000) 2019-10-13T10:56:51.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:51.892+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.898851+0000) 2019-10-13T10:56:51.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:51.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.931341+0000) 2019-10-13T10:56:52.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:52.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.121783+0000) 2019-10-13T10:56:53.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:53.111+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:53.117941+0000) 2019-10-13T10:56:55.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:56:55.960+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.966131+0000) 2019-10-13T10:56:55.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:55.960+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.966309+0000) 2019-10-13T10:56:55.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:55.960+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.966918+0000) 2019-10-13T10:56:55.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:56:55.960+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.967205+0000) 2019-10-13T10:56:55.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:56:55.964+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.967625+0000) 2019-10-13T10:56:55.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:55.964+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.968116+0000) 2019-10-13T10:56:55.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:56:55.963+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.968659+0000) 2019-10-13T10:56:55.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:55.963+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.968768+0000) 2019-10-13T10:56:55.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:56:55.964+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:56:56.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:56:56.360+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.367185+0000) 2019-10-13T10:56:56.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:56:56.431+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.434817+0000) 2019-10-13T10:56:58.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:56:58.624+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:58.630866+0000) 2019-10-13T10:56:58.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:56:58.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:58.636259+0000) 2019-10-13T10:57:00.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:00.968+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.974609+0000) 2019-10-13T10:57:00.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:00.968+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.974943+0000) 2019-10-13T10:57:00.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:00.972+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.975753+0000) 2019-10-13T10:57:00.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:00.972+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.976558+0000) 2019-10-13T10:57:00.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:00.972+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.977000+0000) 2019-10-13T10:57:00.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:00.971+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.977537+0000) 2019-10-13T10:57:00.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:00.972+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.977290+0000) 2019-10-13T10:57:00.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:00.971+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:00.978008+0000) 2019-10-13T10:57:00.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:00.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:01.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:01.896+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.899062+0000) 2019-10-13T10:57:01.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:01.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.931562+0000) 2019-10-13T10:57:02.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:02.116+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.122000+0000) 2019-10-13T10:57:03.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:03.115+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:03.118184+0000) 2019-10-13T10:57:05.509 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:57:05.509 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:05.513 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:57:05.513 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:05.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:05.976+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.982699+0000) 2019-10-13T10:57:05.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:05.980+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.983888+0000) 2019-10-13T10:57:05.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:05.980+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.984158+0000) 2019-10-13T10:57:05.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:05.979+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.984779+0000) 2019-10-13T10:57:05.986 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:05.980+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.985200+0000) 2019-10-13T10:57:05.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:05.980+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.985663+0000) 2019-10-13T10:57:05.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:05.980+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.985872+0000) 2019-10-13T10:57:05.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:05.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:05.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:05.983+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.989509+0000) 2019-10-13T10:57:06.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:06.364+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.367404+0000) 2019-10-13T10:57:06.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:06.443+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.446433+0000) 2019-10-13T10:57:08.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:08.628+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:08.631062+0000) 2019-10-13T10:57:08.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:08.632+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:08.636470+0000) 2019-10-13T10:57:10.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:10.987+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.992559+0000) 2019-10-13T10:57:10.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:10.987+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.992930+0000) 2019-10-13T10:57:10.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:10.987+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.993308+0000) 2019-10-13T10:57:10.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:10.987+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.993702+0000) 2019-10-13T10:57:10.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:10.991+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.994504+0000) 2019-10-13T10:57:10.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:10.987+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.994884+0000) 2019-10-13T10:57:10.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:10.991+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:10.995732+0000) 2019-10-13T10:57:10.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:10.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112154 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:11.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:10.995+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.000389+0000) 2019-10-13T10:57:11.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:11.895+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.899243+0000) 2019-10-13T10:57:11.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:11.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.931765+0000) 2019-10-13T10:57:12.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:12.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.122199+0000) 2019-10-13T10:57:13.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:13.115+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:13.118375+0000) 2019-10-13T10:57:16.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:15.995+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.001699+0000) 2019-10-13T10:57:16.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:15.995+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.002067+0000) 2019-10-13T10:57:16.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:15.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.002448+0000) 2019-10-13T10:57:16.004 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:15.999+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.003693+0000) 2019-10-13T10:57:16.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:15.999+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.004769+0000) 2019-10-13T10:57:16.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:15.998+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.005128+0000) 2019-10-13T10:57:16.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:15.999+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.005595+0000) 2019-10-13T10:57:16.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:15.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:16.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:16.006+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.010317+0000) 2019-10-13T10:57:16.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:16.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.367545+0000) 2019-10-13T10:57:16.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:16.450+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.452816+0000) 2019-10-13T10:57:18.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:18.627+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:18.631341+0000) 2019-10-13T10:57:18.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:18.631+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:18.636700+0000) 2019-10-13T10:57:21.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:21.003+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.010006+0000) 2019-10-13T10:57:21.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:21.003+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.010964+0000) 2019-10-13T10:57:21.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:21.007+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.012059+0000) 2019-10-13T10:57:21.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:21.007+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.012329+0000) 2019-10-13T10:57:21.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:21.007+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.012690+0000) 2019-10-13T10:57:21.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:21.007+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.013305+0000) 2019-10-13T10:57:21.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:21.006+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.013782+0000) 2019-10-13T10:57:21.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:21.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:21.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:21.014+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.020972+0000) 2019-10-13T10:57:21.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:21.895+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.899423+0000) 2019-10-13T10:57:21.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:21.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.932022+0000) 2019-10-13T10:57:22.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:22.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.122435+0000) 2019-10-13T10:57:23.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:23.114+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:23.118574+0000) 2019-10-13T10:57:26.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:26.014+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.019928+0000) 2019-10-13T10:57:26.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:26.015+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.020231+0000) 2019-10-13T10:57:26.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:26.015+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.020790+0000) 2019-10-13T10:57:26.022 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:26.015+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.021392+0000) 2019-10-13T10:57:26.023 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:26.015+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.021698+0000) 2019-10-13T10:57:26.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:26.015+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.022705+0000) 2019-10-13T10:57:26.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:26.015+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.023081+0000) 2019-10-13T10:57:26.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:26.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:26.030 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:26.022+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.028994+0000) 2019-10-13T10:57:26.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:26.367+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.367715+0000) 2019-10-13T10:57:26.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:26.458+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.459127+0000) 2019-10-13T10:57:28.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:28.627+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:28.631533+0000) 2019-10-13T10:57:28.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:28.635+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:28.636927+0000) 2019-10-13T10:57:31.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:31.023+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.028741+0000) 2019-10-13T10:57:31.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:31.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.029466+0000) 2019-10-13T10:57:31.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:31.023+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.029747+0000) 2019-10-13T10:57:31.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:31.026+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.030307+0000) 2019-10-13T10:57:31.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:31.023+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.030170+0000) 2019-10-13T10:57:31.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:31.023+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.030697+0000) 2019-10-13T10:57:31.032 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:31.023+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.031219+0000) 2019-10-13T10:57:31.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:31.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:31.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:31.034+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.038812+0000) 2019-10-13T10:57:31.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:31.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.899603+0000) 2019-10-13T10:57:31.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:31.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.932235+0000) 2019-10-13T10:57:32.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:32.115+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.122661+0000) 2019-10-13T10:57:33.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:33.118+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:33.118777+0000) 2019-10-13T10:57:35.687 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:57:35.687 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:35.691 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:57:35.691 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:36.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:36.031+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.037612+0000) 2019-10-13T10:57:36.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:36.031+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.038186+0000) 2019-10-13T10:57:36.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:36.034+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.039121+0000) 2019-10-13T10:57:36.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:36.031+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.039269+0000) 2019-10-13T10:57:36.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:36.035+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.039629+0000) 2019-10-13T10:57:36.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:36.035+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.040276+0000) 2019-10-13T10:57:36.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:36.035+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.040582+0000) 2019-10-13T10:57:36.043 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:36.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:36.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:36.038+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.045429+0000) 2019-10-13T10:57:36.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:36.367+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.367901+0000) 2019-10-13T10:57:36.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:36.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.465496+0000) 2019-10-13T10:57:38.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:38.631+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:38.631726+0000) 2019-10-13T10:57:38.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:38.631+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:38.637151+0000) 2019-10-13T10:57:41.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:41.043+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.046719+0000) 2019-10-13T10:57:41.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:41.043+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.046377+0000) 2019-10-13T10:57:41.049 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:41.043+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.047088+0000) 2019-10-13T10:57:41.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:41.047+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.047588+0000) 2019-10-13T10:57:41.050 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:41.047+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.048657+0000) 2019-10-13T10:57:41.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:41.042+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.048767+0000) 2019-10-13T10:57:41.051 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:41.047+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.049182+0000) 2019-10-13T10:57:41.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:41.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:41.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:41.046+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.052361+0000) 2019-10-13T10:57:41.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:41.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.899782+0000) 2019-10-13T10:57:41.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:41.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.932435+0000) 2019-10-13T10:57:42.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:42.119+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.122891+0000) 2019-10-13T10:57:43.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:43.118+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:43.118997+0000) 2019-10-13T10:57:46.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:46.051+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.053611+0000) 2019-10-13T10:57:46.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:46.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.053868+0000) 2019-10-13T10:57:46.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:46.051+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.055131+0000) 2019-10-13T10:57:46.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:46.055+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.056025+0000) 2019-10-13T10:57:46.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:46.055+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.056654+0000) 2019-10-13T10:57:46.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:46.055+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.057069+0000) 2019-10-13T10:57:46.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:46.055+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.057622+0000) 2019-10-13T10:57:46.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:46.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:46.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:46.058+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.062312+0000) 2019-10-13T10:57:46.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:46.367+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.368092+0000) 2019-10-13T10:57:46.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:46.466+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.471858+0000) 2019-10-13T10:57:48.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:48.631+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:48.631880+0000) 2019-10-13T10:57:48.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:48.635+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:48.637346+0000) 2019-10-13T10:57:51.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:51.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.063476+0000) 2019-10-13T10:57:51.065 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:51.063+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.064047+0000) 2019-10-13T10:57:51.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:51.063+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.064478+0000) 2019-10-13T10:57:51.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:51.058+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.065274+0000) 2019-10-13T10:57:51.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:51.063+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.065893+0000) 2019-10-13T10:57:51.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:51.063+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.066033+0000) 2019-10-13T10:57:51.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:51.063+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.066802+0000) 2019-10-13T10:57:51.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:51.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:51.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:51.066+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.072801+0000) 2019-10-13T10:57:51.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:51.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.899964+0000) 2019-10-13T10:57:51.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:51.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.932655+0000) 2019-10-13T10:57:52.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:52.119+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.123087+0000) 2019-10-13T10:57:53.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:53.118+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:53.119223+0000) 2019-10-13T10:57:56.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:56.070+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.072556+0000) 2019-10-13T10:57:56.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:56.070+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.073034+0000) 2019-10-13T10:57:56.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:57:56.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.073840+0000) 2019-10-13T10:57:56.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:57:56.070+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.074298+0000) 2019-10-13T10:57:56.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:57:56.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.075130+0000) 2019-10-13T10:57:56.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:57:56.070+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.075258+0000) 2019-10-13T10:57:56.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:56.074+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.075833+0000) 2019-10-13T10:57:56.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:57:56.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:57:56.083 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:56.078+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.082933+0000) 2019-10-13T10:57:56.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:57:56.366+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.368269+0000) 2019-10-13T10:57:56.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:57:56.474+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.478227+0000) 2019-10-13T10:57:58.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:57:58.630+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:58.632085+0000) 2019-10-13T10:57:58.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:57:58.634+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:58.637562+0000) 2019-10-13T10:58:01.083 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:01.078+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.081566+0000) 2019-10-13T10:58:01.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:01.078+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.082004+0000) 2019-10-13T10:58:01.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:01.078+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.082654+0000) 2019-10-13T10:58:01.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:01.082+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.083857+0000) 2019-10-13T10:58:01.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:01.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.083770+0000) 2019-10-13T10:58:01.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:01.082+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.084070+0000) 2019-10-13T10:58:01.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:01.082+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.084784+0000) 2019-10-13T10:58:01.087 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:01.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:01.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:01.086+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.090776+0000) 2019-10-13T10:58:01.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:01.898+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.900137+0000) 2019-10-13T10:58:01.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:01.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.932839+0000) 2019-10-13T10:58:02.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:02.118+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.123262+0000) 2019-10-13T10:58:03.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:03.118+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:03.119456+0000) 2019-10-13T10:58:05.790 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:58:05.790 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:05.793 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:58:05.794 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:06.092 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:06.086+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.090861+0000) 2019-10-13T10:58:06.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:06.090+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.091671+0000) 2019-10-13T10:58:06.093 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:06.086+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.091260+0000) 2019-10-13T10:58:06.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:06.090+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.092022+0000) 2019-10-13T10:58:06.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:06.090+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.092356+0000) 2019-10-13T10:58:06.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:06.090+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.092366+0000) 2019-10-13T10:58:06.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:06.090+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.093240+0000) 2019-10-13T10:58:06.096 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:06.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:06.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:06.098+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.101219+0000) 2019-10-13T10:58:06.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:06.366+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.368480+0000) 2019-10-13T10:58:06.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:06.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.484859+0000) 2019-10-13T10:58:08.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:08.630+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:08.632308+0000) 2019-10-13T10:58:08.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:08.634+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:08.637791+0000) 2019-10-13T10:58:11.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:11.094+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.098630+0000) 2019-10-13T10:58:11.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:11.098+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.099638+0000) 2019-10-13T10:58:11.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:11.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.100056+0000) 2019-10-13T10:58:11.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:11.097+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.100508+0000) 2019-10-13T10:58:11.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:11.098+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.100828+0000) 2019-10-13T10:58:11.102 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:11.098+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.101240+0000) 2019-10-13T10:58:11.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:11.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.102053+0000) 2019-10-13T10:58:11.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:11.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:11.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:11.101+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.105569+0000) 2019-10-13T10:58:11.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:11.898+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.900321+0000) 2019-10-13T10:58:11.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:11.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.933030+0000) 2019-10-13T10:58:12.124 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:12.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.123453+0000) 2019-10-13T10:58:13.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:13.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:13.119663+0000) 2019-10-13T10:58:16.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:16.105+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.107085+0000) 2019-10-13T10:58:16.108 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:16.106+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.107710+0000) 2019-10-13T10:58:16.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:16.106+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.108301+0000) 2019-10-13T10:58:16.110 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:16.106+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.108747+0000) 2019-10-13T10:58:16.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:16.106+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.108947+0000) 2019-10-13T10:58:16.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:16.106+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.109356+0000) 2019-10-13T10:58:16.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:16.106+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.110043+0000) 2019-10-13T10:58:16.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:16.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:16.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:16.113+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.115478+0000) 2019-10-13T10:58:16.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:16.366+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.368660+0000) 2019-10-13T10:58:16.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:16.489+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.490760+0000) 2019-10-13T10:58:18.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:18.630+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:18.632509+0000) 2019-10-13T10:58:18.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:18.634+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:18.637989+0000) 2019-10-13T10:58:21.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:21.114+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.116063+0000) 2019-10-13T10:58:21.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:21.114+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.115803+0000) 2019-10-13T10:58:21.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:21.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.118071+0000) 2019-10-13T10:58:21.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:21.114+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.117997+0000) 2019-10-13T10:58:21.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:21.114+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.118365+0000) 2019-10-13T10:58:21.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:21.114+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.118739+0000) 2019-10-13T10:58:21.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:21.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:21.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:21.113+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.117107+0000) 2019-10-13T10:58:21.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:21.125+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.127845+0000) 2019-10-13T10:58:21.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:21.898+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.900503+0000) 2019-10-13T10:58:21.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:21.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.933262+0000) 2019-10-13T10:58:22.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:22.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.123656+0000) 2019-10-13T10:58:23.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:23.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:23.119858+0000) 2019-10-13T10:58:26.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:26.122+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.125553+0000) 2019-10-13T10:58:26.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:26.121+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.125866+0000) 2019-10-13T10:58:26.128 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:26.122+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.126349+0000) 2019-10-13T10:58:26.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:26.122+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.126541+0000) 2019-10-13T10:58:26.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:26.122+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.126774+0000) 2019-10-13T10:58:26.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:26.126+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.127539+0000) 2019-10-13T10:58:26.129 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:26.126+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.128353+0000) 2019-10-13T10:58:26.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:26.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:26.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:26.133+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.134713+0000) 2019-10-13T10:58:26.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:26.366+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.368846+0000) 2019-10-13T10:58:26.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:26.493+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.497120+0000) 2019-10-13T10:58:28.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:28.630+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:28.632711+0000) 2019-10-13T10:58:28.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:28.634+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:28.638183+0000) 2019-10-13T10:58:31.135 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:31.130+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.134095+0000) 2019-10-13T10:58:31.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:31.130+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.134544+0000) 2019-10-13T10:58:31.136 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:31.130+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.134894+0000) 2019-10-13T10:58:31.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:31.133+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.135756+0000) 2019-10-13T10:58:31.137 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:31.134+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.136557+0000) 2019-10-13T10:58:31.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:31.134+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.137289+0000) 2019-10-13T10:58:31.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:31.134+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.138010+0000) 2019-10-13T10:58:31.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:31.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:31.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:31.141+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.144144+0000) 2019-10-13T10:58:31.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:31.898+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.900703+0000) 2019-10-13T10:58:31.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:31.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.933475+0000) 2019-10-13T10:58:32.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:32.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.123850+0000) 2019-10-13T10:58:33.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:33.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:33.120072+0000) 2019-10-13T10:58:35.928 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:58:35.928 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:35.932 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:58:35.932 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:36.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:36.142+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.144257+0000) 2019-10-13T10:58:36.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:36.142+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.144674+0000) 2019-10-13T10:58:36.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:36.142+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.145069+0000) 2019-10-13T10:58:36.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:36.142+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.145694+0000) 2019-10-13T10:58:36.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:36.142+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.145851+0000) 2019-10-13T10:58:36.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:36.145+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.146839+0000) 2019-10-13T10:58:36.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:36.146+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.147513+0000) 2019-10-13T10:58:36.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:36.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:36.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:36.153+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.154541+0000) 2019-10-13T10:58:36.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:36.366+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.369064+0000) 2019-10-13T10:58:36.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:36.505+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.508712+0000) 2019-10-13T10:58:38.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:38.629+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:38.632911+0000) 2019-10-13T10:58:38.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:38.637+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:38.638415+0000) 2019-10-13T10:58:41.154 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:41.149+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.152866+0000) 2019-10-13T10:58:41.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:41.149+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.153586+0000) 2019-10-13T10:58:41.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:41.149+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.154000+0000) 2019-10-13T10:58:41.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:41.149+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.154366+0000) 2019-10-13T10:58:41.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:41.149+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.155029+0000) 2019-10-13T10:58:41.156 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:41.153+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.155528+0000) 2019-10-13T10:58:41.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:41.153+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.156313+0000) 2019-10-13T10:58:41.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:41.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:41.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:41.157+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.162118+0000) 2019-10-13T10:58:41.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:41.897+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.900906+0000) 2019-10-13T10:58:41.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:41.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.933657+0000) 2019-10-13T10:58:42.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:42.121+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.124032+0000) 2019-10-13T10:58:43.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:43.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:43.120271+0000) 2019-10-13T10:58:46.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:46.157+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.162020+0000) 2019-10-13T10:58:46.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:46.157+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.162637+0000) 2019-10-13T10:58:46.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:46.161+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.163579+0000) 2019-10-13T10:58:46.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:46.161+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.163717+0000) 2019-10-13T10:58:46.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:46.161+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.164585+0000) 2019-10-13T10:58:46.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:46.161+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.164919+0000) 2019-10-13T10:58:46.166 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:46.161+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.165301+0000) 2019-10-13T10:58:46.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:46.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:46.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:46.169+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.170326+0000) 2019-10-13T10:58:46.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:46.365+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.369246+0000) 2019-10-13T10:58:46.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:46.513+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.515154+0000) 2019-10-13T10:58:48.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:48.629+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:48.633140+0000) 2019-10-13T10:58:48.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:48.637+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:48.638633+0000) 2019-10-13T10:58:51.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:51.169+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.171435+0000) 2019-10-13T10:58:51.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:51.169+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.172303+0000) 2019-10-13T10:58:51.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:51.169+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.172793+0000) 2019-10-13T10:58:51.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:51.169+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.172925+0000) 2019-10-13T10:58:51.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:51.169+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.173542+0000) 2019-10-13T10:58:51.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:51.169+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.174127+0000) 2019-10-13T10:58:51.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:51.169+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.174855+0000) 2019-10-13T10:58:51.178 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:51.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:51.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:51.177+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.180288+0000) 2019-10-13T10:58:51.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:51.897+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.901116+0000) 2019-10-13T10:58:51.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:51.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.933868+0000) 2019-10-13T10:58:52.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:52.121+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.124215+0000) 2019-10-13T10:58:53.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:53.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:53.120517+0000) 2019-10-13T10:58:56.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:56.177+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.180323+0000) 2019-10-13T10:58:56.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:58:56.177+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.181377+0000) 2019-10-13T10:58:56.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:58:56.177+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.181568+0000) 2019-10-13T10:58:56.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:56.177+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.181904+0000) 2019-10-13T10:58:56.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:56.177+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.182335+0000) 2019-10-13T10:58:56.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:58:56.177+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.182795+0000) 2019-10-13T10:58:56.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:58:56.181+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.183558+0000) 2019-10-13T10:58:56.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:58:56.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:58:56.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:56.189+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.190945+0000) 2019-10-13T10:58:56.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:58:56.365+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.369426+0000) 2019-10-13T10:58:56.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:58:56.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.522965+0000) 2019-10-13T10:58:58.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:58:58.629+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:58.633346+0000) 2019-10-13T10:58:58.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:58:58.637+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:58.638863+0000) 2019-10-13T10:59:01.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:01.185+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.189107+0000) 2019-10-13T10:59:01.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:01.185+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.190381+0000) 2019-10-13T10:59:01.191 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:01.185+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.190723+0000) 2019-10-13T10:59:01.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:01.189+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.191206+0000) 2019-10-13T10:59:01.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:01.189+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.191875+0000) 2019-10-13T10:59:01.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:01.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.192184+0000) 2019-10-13T10:59:01.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:01.189+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.192585+0000) 2019-10-13T10:59:01.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:01.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:01.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:01.197+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.198655+0000) 2019-10-13T10:59:01.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:01.897+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.901278+0000) 2019-10-13T10:59:01.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:01.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.934083+0000) 2019-10-13T10:59:02.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:02.121+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.124389+0000) 2019-10-13T10:59:03.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:03.117+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:03.120715+0000) 2019-10-13T10:59:06.053 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:59:06.053 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:06.060 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:59:06.060 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:06.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:06.193+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.198665+0000) 2019-10-13T10:59:06.201 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:06.193+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.199251+0000) 2019-10-13T10:59:06.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:06.196+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.199897+0000) 2019-10-13T10:59:06.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:06.197+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.200781+0000) 2019-10-13T10:59:06.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:06.197+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.201370+0000) 2019-10-13T10:59:06.203 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:06.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.201890+0000) 2019-10-13T10:59:06.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:06.197+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.202127+0000) 2019-10-13T10:59:06.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:06.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:06.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:06.204+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.208077+0000) 2019-10-13T10:59:06.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:06.365+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.369641+0000) 2019-10-13T10:59:06.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:06.528+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.529295+0000) 2019-10-13T10:59:08.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:08.633+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:08.633545+0000) 2019-10-13T10:59:08.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:08.633+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:08.639059+0000) 2019-10-13T10:59:11.207 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:11.204+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.206610+0000) 2019-10-13T10:59:11.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:11.201+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.207043+0000) 2019-10-13T10:59:11.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:11.205+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.207570+0000) 2019-10-13T10:59:11.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:11.205+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.208544+0000) 2019-10-13T10:59:11.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:11.205+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.209599+0000) 2019-10-13T10:59:11.211 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:11.205+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.209796+0000) 2019-10-13T10:59:11.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:11.205+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.210055+0000) 2019-10-13T10:59:11.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:11.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:11.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:11.208+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.212424+0000) 2019-10-13T10:59:11.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:11.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.901459+0000) 2019-10-13T10:59:11.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:11.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.934264+0000) 2019-10-13T10:59:12.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:12.121+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.124569+0000) 2019-10-13T10:59:13.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:13.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:13.120913+0000) 2019-10-13T10:59:16.216 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:16.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.215477+0000) 2019-10-13T10:59:16.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:16.213+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.216019+0000) 2019-10-13T10:59:16.217 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:16.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.216712+0000) 2019-10-13T10:59:16.218 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:16.212+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.217228+0000) 2019-10-13T10:59:16.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:16.213+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.217563+0000) 2019-10-13T10:59:16.218 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:16.213+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.217813+0000) 2019-10-13T10:59:16.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:16.213+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.218423+0000) 2019-10-13T10:59:16.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:16.216+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.220679+0000) 2019-10-13T10:59:16.222 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:16.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 112980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:16.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:16.369+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.369819+0000) 2019-10-13T10:59:16.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:16.532+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.535693+0000) 2019-10-13T10:59:18.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:18.629+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:18.633776+0000) 2019-10-13T10:59:18.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:18.633+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:18.639255+0000) 2019-10-13T10:59:21.226 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:21.220+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.224466+0000) 2019-10-13T10:59:21.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:21.220+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.224424+0000) 2019-10-13T10:59:21.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:21.220+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.225546+0000) 2019-10-13T10:59:21.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:21.220+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.226014+0000) 2019-10-13T10:59:21.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:21.220+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.226541+0000) 2019-10-13T10:59:21.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:21.224+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.227843+0000) 2019-10-13T10:59:21.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:21.224+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.228073+0000) 2019-10-13T10:59:21.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:21.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:21.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:21.232+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.236231+0000) 2019-10-13T10:59:21.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:21.900+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.901636+0000) 2019-10-13T10:59:21.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:21.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.934480+0000) 2019-10-13T10:59:22.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:22.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.124773+0000) 2019-10-13T10:59:23.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:23.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:23.121109+0000) 2019-10-13T10:59:26.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:26.228+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.232475+0000) 2019-10-13T10:59:26.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:26.228+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.233313+0000) 2019-10-13T10:59:26.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:26.228+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.233898+0000) 2019-10-13T10:59:26.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:26.228+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.234242+0000) 2019-10-13T10:59:26.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:26.228+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.234566+0000) 2019-10-13T10:59:26.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:26.228+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.234787+0000) 2019-10-13T10:59:26.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:26.228+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.235106+0000) 2019-10-13T10:59:26.239 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:26.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:26.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:26.240+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.244148+0000) 2019-10-13T10:59:26.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:26.368+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.370026+0000) 2019-10-13T10:59:26.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:26.540+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.542218+0000) 2019-10-13T10:59:28.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:28.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:28.634010+0000) 2019-10-13T10:59:28.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:28.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:28.639527+0000) 2019-10-13T10:59:31.242 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:31.236+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.241005+0000) 2019-10-13T10:59:31.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:31.236+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.241408+0000) 2019-10-13T10:59:31.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:31.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.242249+0000) 2019-10-13T10:59:31.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:31.236+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.242400+0000) 2019-10-13T10:59:31.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:31.236+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.242723+0000) 2019-10-13T10:59:31.244 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:31.240+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.243153+0000) 2019-10-13T10:59:31.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:31.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.243963+0000) 2019-10-13T10:59:31.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:31.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:31.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:31.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.248956+0000) 2019-10-13T10:59:31.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:31.896+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.901828+0000) 2019-10-13T10:59:31.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:31.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.934659+0000) 2019-10-13T10:59:32.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:32.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.124967+0000) 2019-10-13T10:59:33.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:33.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:33.121339+0000) 2019-10-13T10:59:36.151 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T10:59:36.151 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:36.155 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T10:59:36.155 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:36.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:36.244+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.249514+0000) 2019-10-13T10:59:36.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:36.244+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.250173+0000) 2019-10-13T10:59:36.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:36.244+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.250666+0000) 2019-10-13T10:59:36.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:36.244+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.251158+0000) 2019-10-13T10:59:36.253 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:36.248+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.251635+0000) 2019-10-13T10:59:36.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:36.248+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.252985+0000) 2019-10-13T10:59:36.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:36.248+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.253453+0000) 2019-10-13T10:59:36.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:36.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:36.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:36.256+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.261183+0000) 2019-10-13T10:59:36.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:36.364+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.370222+0000) 2019-10-13T10:59:36.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:36.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.548918+0000) 2019-10-13T10:59:38.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:38.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:38.634237+0000) 2019-10-13T10:59:38.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:38.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:38.639741+0000) 2019-10-13T10:59:41.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:41.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.257898+0000) 2019-10-13T10:59:41.259 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:41.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.258404+0000) 2019-10-13T10:59:41.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:41.256+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.258933+0000) 2019-10-13T10:59:41.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:41.256+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.259394+0000) 2019-10-13T10:59:41.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:41.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.259276+0000) 2019-10-13T10:59:41.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:41.256+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.259993+0000) 2019-10-13T10:59:41.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:41.256+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.260683+0000) 2019-10-13T10:59:41.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:41.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:41.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:41.264+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.267709+0000) 2019-10-13T10:59:41.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:41.900+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.902042+0000) 2019-10-13T10:59:41.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:41.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.934863+0000) 2019-10-13T10:59:42.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:42.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.125204+0000) 2019-10-13T10:59:43.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:43.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:43.121542+0000) 2019-10-13T10:59:46.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:46.260+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.266938+0000) 2019-10-13T10:59:46.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:46.264+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.268036+0000) 2019-10-13T10:59:46.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:46.264+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.268494+0000) 2019-10-13T10:59:46.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:46.264+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.268653+0000) 2019-10-13T10:59:46.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:46.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.269138+0000) 2019-10-13T10:59:46.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:46.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.269592+0000) 2019-10-13T10:59:46.271 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:46.264+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.270611+0000) 2019-10-13T10:59:46.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:46.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:46.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:46.272+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.276052+0000) 2019-10-13T10:59:46.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:46.368+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.370398+0000) 2019-10-13T10:59:46.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:46.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.555334+0000) 2019-10-13T10:59:48.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:48.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:48.634400+0000) 2019-10-13T10:59:48.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:48.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:48.639954+0000) 2019-10-13T10:59:51.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:51.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.276111+0000) 2019-10-13T10:59:51.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:51.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.276813+0000) 2019-10-13T10:59:51.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:51.272+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.276895+0000) 2019-10-13T10:59:51.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:51.272+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.277636+0000) 2019-10-13T10:59:51.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:51.272+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.277891+0000) 2019-10-13T10:59:51.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:51.276+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.279104+0000) 2019-10-13T10:59:51.280 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:51.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.279705+0000) 2019-10-13T10:59:51.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:51.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:51.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:51.280+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.283478+0000) 2019-10-13T10:59:51.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:51.900+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.902241+0000) 2019-10-13T10:59:51.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:51.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.935052+0000) 2019-10-13T10:59:52.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:52.120+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.125415+0000) 2019-10-13T10:59:53.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:53.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:53.121739+0000) 2019-10-13T10:59:56.286 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T10:59:56.280+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.285107+0000) 2019-10-13T10:59:56.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:56.280+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.285497+0000) 2019-10-13T10:59:56.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:56.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.286359+0000) 2019-10-13T10:59:56.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:56.280+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.286612+0000) 2019-10-13T10:59:56.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T10:59:56.280+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.287019+0000) 2019-10-13T10:59:56.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T10:59:56.284+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.287506+0000) 2019-10-13T10:59:56.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T10:59:56.284+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.288023+0000) 2019-10-13T10:59:56.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T10:59:56.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T10:59:56.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:56.288+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.292439+0000) 2019-10-13T10:59:56.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T10:59:56.364+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.370536+0000) 2019-10-13T10:59:56.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T10:59:56.556+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.561793+0000) 2019-10-13T10:59:58.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T10:59:58.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:58.634628+0000) 2019-10-13T10:59:58.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T10:59:58.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:58.640147+0000) 2019-10-13T11:00:01.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:01.288+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.292109+0000) 2019-10-13T11:00:01.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:01.288+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.293398+0000) 2019-10-13T11:00:01.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:01.288+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.294006+0000) 2019-10-13T11:00:01.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:01.288+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.294828+0000) 2019-10-13T11:00:01.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:01.292+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.295314+0000) 2019-10-13T11:00:01.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:01.292+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.295834+0000) 2019-10-13T11:00:01.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:01.292+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.296314+0000) 2019-10-13T11:00:01.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:01.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:01.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:01.296+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.300981+0000) 2019-10-13T11:00:01.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:01.900+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.902422+0000) 2019-10-13T11:00:01.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:01.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.935275+0000) 2019-10-13T11:00:02.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:02.124+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.125588+0000) 2019-10-13T11:00:03.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:03.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:03.121939+0000) 2019-10-13T11:00:06.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:00:06.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:06.202 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:00:06.203 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:06.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:06.295+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.300405+0000) 2019-10-13T11:00:06.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:06.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.301033+0000) 2019-10-13T11:00:06.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:06.295+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.300940+0000) 2019-10-13T11:00:06.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:06.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.301302+0000) 2019-10-13T11:00:06.303 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:06.295+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.301685+0000) 2019-10-13T11:00:06.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:06.295+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.302103+0000) 2019-10-13T11:00:06.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:06.295+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.302584+0000) 2019-10-13T11:00:06.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:06.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:06.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:06.303+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.307546+0000) 2019-10-13T11:00:06.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:06.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.370726+0000) 2019-10-13T11:00:06.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:06.563+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.568239+0000) 2019-10-13T11:00:08.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:08.631+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:08.634875+0000) 2019-10-13T11:00:08.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:08.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:08.640353+0000) 2019-10-13T11:00:11.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:11.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.309528+0000) 2019-10-13T11:00:11.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:11.303+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.309729+0000) 2019-10-13T11:00:11.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:11.303+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.309946+0000) 2019-10-13T11:00:11.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:11.303+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.310161+0000) 2019-10-13T11:00:11.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:11.303+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.310935+0000) 2019-10-13T11:00:11.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:11.307+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.311629+0000) 2019-10-13T11:00:11.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:11.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.312341+0000) 2019-10-13T11:00:11.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:11.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113339 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:11.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:11.315+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.321226+0000) 2019-10-13T11:00:11.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:11.895+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.902598+0000) 2019-10-13T11:00:11.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:11.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.935527+0000) 2019-10-13T11:00:12.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:12.119+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:12.125794+0000) 2019-10-13T11:00:13.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:13.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.122174+0000) 2019-10-13T11:00:16.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:16.311+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.316897+0000) 2019-10-13T11:00:16.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:16.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.317270+0000) 2019-10-13T11:00:16.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:16.311+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.318092+0000) 2019-10-13T11:00:16.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:16.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.318992+0000) 2019-10-13T11:00:16.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:16.315+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.320086+0000) 2019-10-13T11:00:16.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:16.315+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.320559+0000) 2019-10-13T11:00:16.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:16.315+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.321219+0000) 2019-10-13T11:00:16.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:16.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:16.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:16.319+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.323885+0000) 2019-10-13T11:00:16.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:16.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.370914+0000) 2019-10-13T11:00:16.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:16.571+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.574217+0000) 2019-10-13T11:00:18.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:18.627+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.635120+0000) 2019-10-13T11:00:18.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:18.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.640554+0000) 2019-10-13T11:00:21.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:21.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.325717+0000) 2019-10-13T11:00:21.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:21.319+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.326587+0000) 2019-10-13T11:00:21.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:21.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.327354+0000) 2019-10-13T11:00:21.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:21.323+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.327794+0000) 2019-10-13T11:00:21.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:21.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.327782+0000) 2019-10-13T11:00:21.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:21.323+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.328128+0000) 2019-10-13T11:00:21.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:21.323+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.328626+0000) 2019-10-13T11:00:21.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:21.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:21.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:21.327+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.333343+0000) 2019-10-13T11:00:21.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:21.895+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.902769+0000) 2019-10-13T11:00:21.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:21.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.935754+0000) 2019-10-13T11:00:22.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:22.119+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:22.125976+0000) 2019-10-13T11:00:23.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:23.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.122397+0000) 2019-10-13T11:00:26.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:26.331+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.335225+0000) 2019-10-13T11:00:26.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:26.331+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.335484+0000) 2019-10-13T11:00:26.337 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:26.331+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.335851+0000) 2019-10-13T11:00:26.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:26.331+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.335917+0000) 2019-10-13T11:00:26.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:26.331+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.336606+0000) 2019-10-13T11:00:26.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:26.331+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.337177+0000) 2019-10-13T11:00:26.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:26.331+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.337494+0000) 2019-10-13T11:00:26.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:26.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:26.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:26.335+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.342076+0000) 2019-10-13T11:00:26.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:26.363+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.371116+0000) 2019-10-13T11:00:26.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:26.575+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.580502+0000) 2019-10-13T11:00:28.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:28.631+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.635345+0000) 2019-10-13T11:00:28.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:28.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.640748+0000) 2019-10-13T11:00:31.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:31.339+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.343241+0000) 2019-10-13T11:00:31.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:31.339+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.343982+0000) 2019-10-13T11:00:31.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:31.339+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.344507+0000) 2019-10-13T11:00:31.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:31.339+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.344740+0000) 2019-10-13T11:00:31.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:31.339+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.345316+0000) 2019-10-13T11:00:31.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:31.339+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.345920+0000) 2019-10-13T11:00:31.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:31.339+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.346352+0000) 2019-10-13T11:00:31.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:31.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:31.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:31.347+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.352033+0000) 2019-10-13T11:00:31.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:31.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.902943+0000) 2019-10-13T11:00:31.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:31.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.935987+0000) 2019-10-13T11:00:32.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:32.123+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:32.126178+0000) 2019-10-13T11:00:33.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:33.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.122641+0000) 2019-10-13T11:00:36.321 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:00:36.321 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:36.325 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:00:36.325 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:36.352 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:36.351+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.351358+0000) 2019-10-13T11:00:36.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:36.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.351443+0000) 2019-10-13T11:00:36.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:36.351+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.352101+0000) 2019-10-13T11:00:36.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:36.351+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.352377+0000) 2019-10-13T11:00:36.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:36.351+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.353157+0000) 2019-10-13T11:00:36.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:36.347+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.353462+0000) 2019-10-13T11:00:36.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:36.351+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.353870+0000) 2019-10-13T11:00:36.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:36.355+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:36.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:36.351+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.357901+0000) 2019-10-13T11:00:36.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:36.367+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.371270+0000) 2019-10-13T11:00:36.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:36.591+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.592204+0000) 2019-10-13T11:00:38.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:38.635+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.635563+0000) 2019-10-13T11:00:38.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:38.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.640946+0000) 2019-10-13T11:00:41.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:41.359+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.360570+0000) 2019-10-13T11:00:41.362 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:41.359+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.360897+0000) 2019-10-13T11:00:41.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:41.359+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.361270+0000) 2019-10-13T11:00:41.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:41.359+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.361630+0000) 2019-10-13T11:00:41.363 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:41.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.362415+0000) 2019-10-13T11:00:41.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:41.363+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.363359+0000) 2019-10-13T11:00:41.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:41.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.363941+0000) 2019-10-13T11:00:41.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:41.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113533 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:41.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:41.367+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.370523+0000) 2019-10-13T11:00:41.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:41.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.903121+0000) 2019-10-13T11:00:41.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:41.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.936192+0000) 2019-10-13T11:00:42.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:42.123+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:42.126351+0000) 2019-10-13T11:00:43.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:43.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.122892+0000) 2019-10-13T11:00:46.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:46.367+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.369537+0000) 2019-10-13T11:00:46.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:46.367+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.369972+0000) 2019-10-13T11:00:46.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:46.367+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.370163+0000) 2019-10-13T11:00:46.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:46.367+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.370909+0000) 2019-10-13T11:00:46.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:46.371+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.371408+0000) 2019-10-13T11:00:46.373 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:46.371+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.372126+0000) 2019-10-13T11:00:46.373 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:46.367+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.372693+0000) 2019-10-13T11:00:46.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:46.371+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.373327+0000) 2019-10-13T11:00:46.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:46.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:46.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:46.375+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.378844+0000) 2019-10-13T11:00:46.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:46.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.598573+0000) 2019-10-13T11:00:48.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:48.634+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.635758+0000) 2019-10-13T11:00:48.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:48.638+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.641185+0000) 2019-10-13T11:00:51.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:51.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.378087+0000) 2019-10-13T11:00:51.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:51.374+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.378671+0000) 2019-10-13T11:00:51.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:51.378+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.379829+0000) 2019-10-13T11:00:51.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:51.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.380026+0000) 2019-10-13T11:00:51.381 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:51.375+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.380445+0000) 2019-10-13T11:00:51.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:51.378+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.381212+0000) 2019-10-13T11:00:51.382 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:51.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.381437+0000) 2019-10-13T11:00:51.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:51.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:51.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:51.383+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.389159+0000) 2019-10-13T11:00:51.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:51.902+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.903356+0000) 2019-10-13T11:00:51.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:51.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.936398+0000) 2019-10-13T11:00:52.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:52.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:52.126525+0000) 2019-10-13T11:00:53.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:53.119+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.123081+0000) 2019-10-13T11:00:56.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:56.370+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.371587+0000) 2019-10-13T11:00:56.390 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:00:56.386+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.388836+0000) 2019-10-13T11:00:56.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:00:56.383+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.389295+0000) 2019-10-13T11:00:56.391 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:00:56.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.390090+0000) 2019-10-13T11:00:56.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:56.386+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.390125+0000) 2019-10-13T11:00:56.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:00:56.386+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.390851+0000) 2019-10-13T11:00:56.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:56.390+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.391452+0000) 2019-10-13T11:00:56.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:00:56.390+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.391918+0000) 2019-10-13T11:00:56.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:00:56.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:00:56.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:56.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.399274+0000) 2019-10-13T11:00:56.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:00:56.603+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.605065+0000) 2019-10-13T11:00:58.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:00:58.634+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.635979+0000) 2019-10-13T11:00:58.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:00:58.638+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.641382+0000) 2019-10-13T11:01:01.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:01.394+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.397377+0000) 2019-10-13T11:01:01.399 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:01.390+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.397731+0000) 2019-10-13T11:01:01.399 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:01.394+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.398232+0000) 2019-10-13T11:01:01.399 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:01.394+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.398550+0000) 2019-10-13T11:01:01.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:01.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.399824+0000) 2019-10-13T11:01:01.402 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:01.398+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.401149+0000) 2019-10-13T11:01:01.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:01.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.401901+0000) 2019-10-13T11:01:01.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:01.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:01.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:01.402+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.406738+0000) 2019-10-13T11:01:01.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:01.902+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.903532+0000) 2019-10-13T11:01:01.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:01.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.936590+0000) 2019-10-13T11:01:02.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:02.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:02.126702+0000) 2019-10-13T11:01:03.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:03.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.123289+0000) 2019-10-13T11:01:06.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:06.370+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.371780+0000) 2019-10-13T11:01:06.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:06.402+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.406386+0000) 2019-10-13T11:01:06.407 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:06.402+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.406931+0000) 2019-10-13T11:01:06.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:06.402+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.408083+0000) 2019-10-13T11:01:06.409 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:06.406+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.408629+0000) 2019-10-13T11:01:06.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:06.406+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.409076+0000) 2019-10-13T11:01:06.410 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:06.406+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.409417+0000) 2019-10-13T11:01:06.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:06.406+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.410255+0000) 2019-10-13T11:01:06.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:06.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:06.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:06.410+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.416895+0000) 2019-10-13T11:01:06.454 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:01:06.454 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:06.458 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:01:06.458 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:06.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:06.610+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.611559+0000) 2019-10-13T11:01:08.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:08.634+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.636198+0000) 2019-10-13T11:01:08.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:08.638+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.641590+0000) 2019-10-13T11:01:11.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:11.414+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.415671+0000) 2019-10-13T11:01:11.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:11.414+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.416302+0000) 2019-10-13T11:01:11.418 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:11.414+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.417057+0000) 2019-10-13T11:01:11.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:11.410+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.417786+0000) 2019-10-13T11:01:11.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:11.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.418194+0000) 2019-10-13T11:01:11.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:11.414+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.418755+0000) 2019-10-13T11:01:11.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:11.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.419341+0000) 2019-10-13T11:01:11.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:11.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:11.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:11.418+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.423731+0000) 2019-10-13T11:01:11.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:11.902+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.903753+0000) 2019-10-13T11:01:11.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:11.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.936775+0000) 2019-10-13T11:01:12.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:12.122+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.126852+0000) 2019-10-13T11:01:13.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:13.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.123505+0000) 2019-10-13T11:01:16.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:16.370+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.371999+0000) 2019-10-13T11:01:16.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:16.422+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.424408+0000) 2019-10-13T11:01:16.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:16.422+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.424815+0000) 2019-10-13T11:01:16.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:16.422+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.425567+0000) 2019-10-13T11:01:16.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:16.422+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.426276+0000) 2019-10-13T11:01:16.427 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:16.422+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.426800+0000) 2019-10-13T11:01:16.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:16.422+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.426917+0000) 2019-10-13T11:01:16.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:16.422+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.427161+0000) 2019-10-13T11:01:16.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:16.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:16.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:16.426+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.433232+0000) 2019-10-13T11:01:16.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:16.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:16.618194+0000) 2019-10-13T11:01:18.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:18.634+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.636428+0000) 2019-10-13T11:01:18.642 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:18.638+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.641789+0000) 2019-10-13T11:01:21.434 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:21.430+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.433368+0000) 2019-10-13T11:01:21.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:21.430+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.433437+0000) 2019-10-13T11:01:21.435 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:21.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.433843+0000) 2019-10-13T11:01:21.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:21.430+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.434592+0000) 2019-10-13T11:01:21.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:21.430+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.435258+0000) 2019-10-13T11:01:21.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:21.430+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.435626+0000) 2019-10-13T11:01:21.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:21.434+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.435744+0000) 2019-10-13T11:01:21.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:21.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:21.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:21.434+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.441349+0000) 2019-10-13T11:01:21.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:21.902+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.903884+0000) 2019-10-13T11:01:21.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:21.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.936977+0000) 2019-10-13T11:01:22.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:22.126+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.127086+0000) 2019-10-13T11:01:23.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:23.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.123694+0000) 2019-10-13T11:01:26.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:26.370+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.372196+0000) 2019-10-13T11:01:26.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:26.434+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.441018+0000) 2019-10-13T11:01:26.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:26.438+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.442457+0000) 2019-10-13T11:01:26.444 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:26.442+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.443499+0000) 2019-10-13T11:01:26.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:26.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.443720+0000) 2019-10-13T11:01:26.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:26.442+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.444040+0000) 2019-10-13T11:01:26.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:26.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.444436+0000) 2019-10-13T11:01:26.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:26.442+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.444662+0000) 2019-10-13T11:01:26.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:26.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:26.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:26.442+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.448807+0000) 2019-10-13T11:01:26.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:26.618+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:26.624622+0000) 2019-10-13T11:01:28.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:28.634+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.636617+0000) 2019-10-13T11:01:28.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:28.638+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.641992+0000) 2019-10-13T11:01:31.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:31.446+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.450610+0000) 2019-10-13T11:01:31.452 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:31.450+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.451297+0000) 2019-10-13T11:01:31.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:31.446+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.451764+0000) 2019-10-13T11:01:31.453 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:31.450+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.452056+0000) 2019-10-13T11:01:31.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:31.450+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.452669+0000) 2019-10-13T11:01:31.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:31.450+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.452999+0000) 2019-10-13T11:01:31.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:31.450+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.453682+0000) 2019-10-13T11:01:31.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:31.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:31.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:31.454+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.459532+0000) 2019-10-13T11:01:31.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:31.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.904067+0000) 2019-10-13T11:01:31.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:31.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.937178+0000) 2019-10-13T11:01:32.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:32.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.127261+0000) 2019-10-13T11:01:33.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:33.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.123920+0000) 2019-10-13T11:01:36.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:36.369+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.372388+0000) 2019-10-13T11:01:36.460 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:36.453+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.458947+0000) 2019-10-13T11:01:36.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:36.457+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.460284+0000) 2019-10-13T11:01:36.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:36.454+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.460759+0000) 2019-10-13T11:01:36.462 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:36.457+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.460565+0000) 2019-10-13T11:01:36.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:36.457+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.461144+0000) 2019-10-13T11:01:36.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:36.457+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.461443+0000) 2019-10-13T11:01:36.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:36.457+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.462227+0000) 2019-10-13T11:01:36.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:36.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113883 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:36.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:36.458+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.466126+0000) 2019-10-13T11:01:36.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:36.630+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:36.631185+0000) 2019-10-13T11:01:36.735 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:01:36.735 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:36.739 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:01:36.739 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:38.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:38.633+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.636818+0000) 2019-10-13T11:01:38.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:38.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.642234+0000) 2019-10-13T11:01:41.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:41.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.467851+0000) 2019-10-13T11:01:41.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:41.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.468523+0000) 2019-10-13T11:01:41.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:41.462+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.468989+0000) 2019-10-13T11:01:41.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:41.465+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.469477+0000) 2019-10-13T11:01:41.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:41.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.470337+0000) 2019-10-13T11:01:41.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:41.465+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.470964+0000) 2019-10-13T11:01:41.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:41.465+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.470654+0000) 2019-10-13T11:01:41.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:41.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:41.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:41.470+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.476235+0000) 2019-10-13T11:01:41.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:41.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.904235+0000) 2019-10-13T11:01:41.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:41.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.937376+0000) 2019-10-13T11:01:42.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:42.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.127464+0000) 2019-10-13T11:01:43.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:43.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.124114+0000) 2019-10-13T11:01:46.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:46.369+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.372532+0000) 2019-10-13T11:01:46.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:46.474+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.477447+0000) 2019-10-13T11:01:46.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:46.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.478023+0000) 2019-10-13T11:01:46.479 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:46.473+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.478489+0000) 2019-10-13T11:01:46.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:46.473+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.478734+0000) 2019-10-13T11:01:46.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:46.473+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.479072+0000) 2019-10-13T11:01:46.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:46.477+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.479452+0000) 2019-10-13T11:01:46.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:46.477+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.480126+0000) 2019-10-13T11:01:46.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:46.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:46.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:46.486+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.486738+0000) 2019-10-13T11:01:46.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:46.634+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:46.637715+0000) 2019-10-13T11:01:48.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:48.633+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.637014+0000) 2019-10-13T11:01:48.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:48.637+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.642428+0000) 2019-10-13T11:01:51.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:51.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.486070+0000) 2019-10-13T11:01:51.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:51.481+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.486447+0000) 2019-10-13T11:01:51.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:51.481+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.487246+0000) 2019-10-13T11:01:51.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:51.485+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.488246+0000) 2019-10-13T11:01:51.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:51.485+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.488126+0000) 2019-10-13T11:01:51.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:51.486+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.488984+0000) 2019-10-13T11:01:51.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:51.485+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.489437+0000) 2019-10-13T11:01:51.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:51.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 113979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:51.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:51.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.493683+0000) 2019-10-13T11:01:51.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:51.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.904411+0000) 2019-10-13T11:01:51.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:51.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.937558+0000) 2019-10-13T11:01:52.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:52.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.127634+0000) 2019-10-13T11:01:53.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:53.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.124309+0000) 2019-10-13T11:01:56.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:56.369+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.372766+0000) 2019-10-13T11:01:56.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:01:56.489+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.495080+0000) 2019-10-13T11:01:56.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:01:56.493+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.495438+0000) 2019-10-13T11:01:56.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:01:56.493+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.496089+0000) 2019-10-13T11:01:56.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:01:56.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.496270+0000) 2019-10-13T11:01:56.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:01:56.493+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.496543+0000) 2019-10-13T11:01:56.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:56.493+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.496959+0000) 2019-10-13T11:01:56.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:56.493+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.497378+0000) 2019-10-13T11:01:56.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:01:56.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:01:56.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:56.501+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.505071+0000) 2019-10-13T11:01:56.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:01:56.641+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:56.644092+0000) 2019-10-13T11:01:58.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:01:58.633+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.637213+0000) 2019-10-13T11:01:58.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:01:58.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.642626+0000) 2019-10-13T11:02:01.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:01.501+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.503764+0000) 2019-10-13T11:02:01.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:01.501+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.504173+0000) 2019-10-13T11:02:01.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:01.501+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.504615+0000) 2019-10-13T11:02:01.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:01.501+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.504845+0000) 2019-10-13T11:02:01.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:01.501+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.505222+0000) 2019-10-13T11:02:01.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:01.501+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.506011+0000) 2019-10-13T11:02:01.507 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:01.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.506541+0000) 2019-10-13T11:02:01.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:01.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:01.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:01.513+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.514965+0000) 2019-10-13T11:02:01.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:01.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.904605+0000) 2019-10-13T11:02:01.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:01.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.937728+0000) 2019-10-13T11:02:02.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:02.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.127818+0000) 2019-10-13T11:02:03.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:03.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.124504+0000) 2019-10-13T11:02:06.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:06.369+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.372951+0000) 2019-10-13T11:02:06.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:06.509+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.513453+0000) 2019-10-13T11:02:06.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:06.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.513849+0000) 2019-10-13T11:02:06.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:06.509+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.514216+0000) 2019-10-13T11:02:06.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:06.509+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.514764+0000) 2019-10-13T11:02:06.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:06.509+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.515124+0000) 2019-10-13T11:02:06.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:06.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.515454+0000) 2019-10-13T11:02:06.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:06.513+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.516292+0000) 2019-10-13T11:02:06.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:06.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:06.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:06.521+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.524815+0000) 2019-10-13T11:02:06.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:06.649+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:06.650365+0000) 2019-10-13T11:02:06.874 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:02:06.883 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:06.894 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:02:06.895 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:08.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:08.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.637416+0000) 2019-10-13T11:02:08.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:08.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.642857+0000) 2019-10-13T11:02:11.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:11.517+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.522207+0000) 2019-10-13T11:02:11.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:11.517+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.522585+0000) 2019-10-13T11:02:11.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:11.521+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.523529+0000) 2019-10-13T11:02:11.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:11.521+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.524482+0000) 2019-10-13T11:02:11.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:11.521+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.524536+0000) 2019-10-13T11:02:11.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:11.521+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.523902+0000) 2019-10-13T11:02:11.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:11.521+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.525572+0000) 2019-10-13T11:02:11.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:11.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:11.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:11.529+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.530995+0000) 2019-10-13T11:02:11.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:11.901+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.904782+0000) 2019-10-13T11:02:11.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:11.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.937918+0000) 2019-10-13T11:02:12.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:12.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.127992+0000) 2019-10-13T11:02:13.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:13.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.124732+0000) 2019-10-13T11:02:16.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:16.372+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.373152+0000) 2019-10-13T11:02:16.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:16.524+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.531272+0000) 2019-10-13T11:02:16.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:16.528+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.531842+0000) 2019-10-13T11:02:16.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:16.529+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.532300+0000) 2019-10-13T11:02:16.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:16.528+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.532544+0000) 2019-10-13T11:02:16.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:16.528+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.532892+0000) 2019-10-13T11:02:16.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:16.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.533804+0000) 2019-10-13T11:02:16.535 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:16.528+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.533855+0000) 2019-10-13T11:02:16.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:16.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:16.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:16.537+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.539822+0000) 2019-10-13T11:02:16.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:16.653+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:16.656835+0000) 2019-10-13T11:02:18.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:18.636+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.637661+0000) 2019-10-13T11:02:18.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:18.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.643078+0000) 2019-10-13T11:02:21.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:21.537+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.540402+0000) 2019-10-13T11:02:21.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:21.536+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.541368+0000) 2019-10-13T11:02:21.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:21.536+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.542189+0000) 2019-10-13T11:02:21.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:21.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.542284+0000) 2019-10-13T11:02:21.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:21.536+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.542589+0000) 2019-10-13T11:02:21.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:21.540+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.543340+0000) 2019-10-13T11:02:21.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:21.540+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.543908+0000) 2019-10-13T11:02:21.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:21.540+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:21.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:21.549+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.553376+0000) 2019-10-13T11:02:21.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:21.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.904957+0000) 2019-10-13T11:02:21.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:21.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.938105+0000) 2019-10-13T11:02:22.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:22.124+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.128198+0000) 2019-10-13T11:02:23.126 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:23.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.124929+0000) 2019-10-13T11:02:26.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:26.372+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.373352+0000) 2019-10-13T11:02:26.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:26.544+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.550520+0000) 2019-10-13T11:02:26.552 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:26.544+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.551063+0000) 2019-10-13T11:02:26.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:26.548+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.551538+0000) 2019-10-13T11:02:26.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:26.548+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.552094+0000) 2019-10-13T11:02:26.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:26.548+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.552704+0000) 2019-10-13T11:02:26.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:26.549+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.552908+0000) 2019-10-13T11:02:26.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:26.548+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.553552+0000) 2019-10-13T11:02:26.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:26.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:26.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:26.557+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.558899+0000) 2019-10-13T11:02:26.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:26.657+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:26.660747+0000) 2019-10-13T11:02:28.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:28.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.637879+0000) 2019-10-13T11:02:28.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:28.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.643277+0000) 2019-10-13T11:02:31.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:31.552+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.558559+0000) 2019-10-13T11:02:31.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:31.552+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.559142+0000) 2019-10-13T11:02:31.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:31.556+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.559642+0000) 2019-10-13T11:02:31.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:31.556+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.560147+0000) 2019-10-13T11:02:31.562 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:31.556+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.561094+0000) 2019-10-13T11:02:31.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:31.556+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.561436+0000) 2019-10-13T11:02:31.563 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:31.557+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.562083+0000) 2019-10-13T11:02:31.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:31.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:31.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:31.565+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.568853+0000) 2019-10-13T11:02:31.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:31.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.905171+0000) 2019-10-13T11:02:31.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:31.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.938283+0000) 2019-10-13T11:02:32.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:32.124+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.128410+0000) 2019-10-13T11:02:33.126 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:33.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.125144+0000) 2019-10-13T11:02:36.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:36.368+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.373539+0000) 2019-10-13T11:02:36.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:36.565+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.568029+0000) 2019-10-13T11:02:36.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:36.564+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.568518+0000) 2019-10-13T11:02:36.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:36.564+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.568985+0000) 2019-10-13T11:02:36.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:36.564+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.569343+0000) 2019-10-13T11:02:36.571 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:36.564+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.570086+0000) 2019-10-13T11:02:36.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:36.564+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.570324+0000) 2019-10-13T11:02:36.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:36.564+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.571132+0000) 2019-10-13T11:02:36.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:36.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:36.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:36.577+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.579530+0000) 2019-10-13T11:02:36.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:36.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:36.667193+0000) 2019-10-13T11:02:37.033 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:02:37.033 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:37.036 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:02:37.037 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:38.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:38.636+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.638082+0000) 2019-10-13T11:02:38.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:38.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.643493+0000) 2019-10-13T11:02:41.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:41.572+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.576896+0000) 2019-10-13T11:02:41.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:41.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.577997+0000) 2019-10-13T11:02:41.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:41.572+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.578517+0000) 2019-10-13T11:02:41.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:41.577+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.579111+0000) 2019-10-13T11:02:41.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:41.576+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.579372+0000) 2019-10-13T11:02:41.581 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:41.576+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.579810+0000) 2019-10-13T11:02:41.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:41.576+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.580092+0000) 2019-10-13T11:02:41.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:41.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:41.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:41.585+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.587113+0000) 2019-10-13T11:02:41.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:41.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.905374+0000) 2019-10-13T11:02:41.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:41.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.938470+0000) 2019-10-13T11:02:42.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:42.124+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.128609+0000) 2019-10-13T11:02:43.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:43.121+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.125341+0000) 2019-10-13T11:02:46.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:46.368+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.373740+0000) 2019-10-13T11:02:46.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:46.580+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.585972+0000) 2019-10-13T11:02:46.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:46.580+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.586456+0000) 2019-10-13T11:02:46.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:46.585+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.587358+0000) 2019-10-13T11:02:46.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:46.584+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.587862+0000) 2019-10-13T11:02:46.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:46.584+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.588248+0000) 2019-10-13T11:02:46.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:46.584+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.588600+0000) 2019-10-13T11:02:46.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:46.584+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.588913+0000) 2019-10-13T11:02:46.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:46.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:46.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:46.597+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.598386+0000) 2019-10-13T11:02:46.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:46.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:46.678418+0000) 2019-10-13T11:02:48.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:48.636+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.638278+0000) 2019-10-13T11:02:48.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:48.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.643691+0000) 2019-10-13T11:02:51.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:51.588+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.594342+0000) 2019-10-13T11:02:51.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:51.588+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.595052+0000) 2019-10-13T11:02:51.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:51.588+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.595029+0000) 2019-10-13T11:02:51.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:51.592+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.596546+0000) 2019-10-13T11:02:51.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:51.592+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.597003+0000) 2019-10-13T11:02:51.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:51.592+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.597355+0000) 2019-10-13T11:02:51.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:51.592+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.597831+0000) 2019-10-13T11:02:51.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:51.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:51.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:51.600+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.603018+0000) 2019-10-13T11:02:51.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:51.900+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.905555+0000) 2019-10-13T11:02:51.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:51.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.938651+0000) 2019-10-13T11:02:52.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:52.124+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.128790+0000) 2019-10-13T11:02:53.126 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:53.124+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.125536+0000) 2019-10-13T11:02:56.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:56.372+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.373939+0000) 2019-10-13T11:02:56.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:56.596+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.601846+0000) 2019-10-13T11:02:56.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:02:56.596+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.602751+0000) 2019-10-13T11:02:56.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:02:56.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.603678+0000) 2019-10-13T11:02:56.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:02:56.600+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.603893+0000) 2019-10-13T11:02:56.605 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:02:56.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.604855+0000) 2019-10-13T11:02:56.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:02:56.600+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.605244+0000) 2019-10-13T11:02:56.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:56.600+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.605832+0000) 2019-10-13T11:02:56.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:02:56.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:02:56.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:56.608+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.612323+0000) 2019-10-13T11:02:56.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:02:56.680+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:56.684994+0000) 2019-10-13T11:02:58.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:02:58.632+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.638471+0000) 2019-10-13T11:02:58.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:02:58.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.643888+0000) 2019-10-13T11:03:01.613 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:01.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.612463+0000) 2019-10-13T11:03:01.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:01.607+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.612822+0000) 2019-10-13T11:03:01.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:01.607+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.613044+0000) 2019-10-13T11:03:01.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:01.607+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.613486+0000) 2019-10-13T11:03:01.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:01.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.614360+0000) 2019-10-13T11:03:01.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:01.612+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.615317+0000) 2019-10-13T11:03:01.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:01.611+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.615742+0000) 2019-10-13T11:03:01.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:01.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:01.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:01.620+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.624111+0000) 2019-10-13T11:03:01.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:01.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.905731+0000) 2019-10-13T11:03:01.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:01.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.938863+0000) 2019-10-13T11:03:02.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:02.127+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.128926+0000) 2019-10-13T11:03:03.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:03.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.125764+0000) 2019-10-13T11:03:06.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:06.367+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.374160+0000) 2019-10-13T11:03:06.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:06.615+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.620987+0000) 2019-10-13T11:03:06.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:06.615+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.621356+0000) 2019-10-13T11:03:06.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:06.615+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.621808+0000) 2019-10-13T11:03:06.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:06.615+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.623093+0000) 2019-10-13T11:03:06.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:06.619+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.623701+0000) 2019-10-13T11:03:06.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:06.619+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.624155+0000) 2019-10-13T11:03:06.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:06.620+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.624473+0000) 2019-10-13T11:03:06.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:06.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:06.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:06.628+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.631673+0000) 2019-10-13T11:03:06.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:06.688+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:06.691317+0000) 2019-10-13T11:03:07.258 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:03:07.259 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:07.262 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:03:07.263 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:08.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:08.631+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.638672+0000) 2019-10-13T11:03:08.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:08.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.644110+0000) 2019-10-13T11:03:11.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:11.627+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.630932+0000) 2019-10-13T11:03:11.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:11.627+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.631254+0000) 2019-10-13T11:03:11.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:11.627+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.631710+0000) 2019-10-13T11:03:11.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:11.627+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.632499+0000) 2019-10-13T11:03:11.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:11.627+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.632839+0000) 2019-10-13T11:03:11.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:11.628+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.633515+0000) 2019-10-13T11:03:11.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:11.627+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.633547+0000) 2019-10-13T11:03:11.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:11.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:11.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:11.636+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.641133+0000) 2019-10-13T11:03:11.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:11.899+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.905862+0000) 2019-10-13T11:03:11.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:11.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.939079+0000) 2019-10-13T11:03:12.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:12.123+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.129107+0000) 2019-10-13T11:03:13.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:13.120+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.125950+0000) 2019-10-13T11:03:16.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:16.371+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.374351+0000) 2019-10-13T11:03:16.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:16.635+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.641255+0000) 2019-10-13T11:03:16.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:16.636+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.641995+0000) 2019-10-13T11:03:16.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:16.635+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.641777+0000) 2019-10-13T11:03:16.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:16.635+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.642328+0000) 2019-10-13T11:03:16.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:16.635+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.642796+0000) 2019-10-13T11:03:16.644 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:16.639+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.643346+0000) 2019-10-13T11:03:16.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:16.639+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.643937+0000) 2019-10-13T11:03:16.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:16.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:16.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:16.648+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.650434+0000) 2019-10-13T11:03:16.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:16.692+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:16.698041+0000) 2019-10-13T11:03:18.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:18.635+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.638888+0000) 2019-10-13T11:03:18.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:18.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.644307+0000) 2019-10-13T11:03:21.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:21.643+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.649844+0000) 2019-10-13T11:03:21.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:21.648+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.650284+0000) 2019-10-13T11:03:21.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:21.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.650876+0000) 2019-10-13T11:03:21.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:21.647+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.652034+0000) 2019-10-13T11:03:21.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:21.647+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.652423+0000) 2019-10-13T11:03:21.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:21.647+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.652959+0000) 2019-10-13T11:03:21.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:21.647+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.653490+0000) 2019-10-13T11:03:21.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:21.651+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:21.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:21.656+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.660784+0000) 2019-10-13T11:03:21.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:21.903+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.906030+0000) 2019-10-13T11:03:21.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:21.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.939291+0000) 2019-10-13T11:03:22.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:22.127+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.129318+0000) 2019-10-13T11:03:23.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:23.124+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.126166+0000) 2019-10-13T11:03:26.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:26.371+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.374563+0000) 2019-10-13T11:03:26.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:26.656+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.659154+0000) 2019-10-13T11:03:26.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:26.655+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.659822+0000) 2019-10-13T11:03:26.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:26.655+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.660492+0000) 2019-10-13T11:03:26.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:26.655+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.661288+0000) 2019-10-13T11:03:26.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:26.655+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.661976+0000) 2019-10-13T11:03:26.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:26.655+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.662228+0000) 2019-10-13T11:03:26.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:26.655+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.662521+0000) 2019-10-13T11:03:26.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:26.659+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:26.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:26.664+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.667954+0000) 2019-10-13T11:03:26.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:26.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:26.704441+0000) 2019-10-13T11:03:28.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:28.635+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.639120+0000) 2019-10-13T11:03:28.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:28.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.644499+0000) 2019-10-13T11:03:31.669 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:31.667+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.667904+0000) 2019-10-13T11:03:31.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:31.667+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.668489+0000) 2019-10-13T11:03:31.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:31.667+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.669264+0000) 2019-10-13T11:03:31.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:31.664+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.669514+0000) 2019-10-13T11:03:31.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:31.667+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.670592+0000) 2019-10-13T11:03:31.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:31.667+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.671089+0000) 2019-10-13T11:03:31.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:31.671+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.671500+0000) 2019-10-13T11:03:31.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:31.671+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:31.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:31.676+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.679264+0000) 2019-10-13T11:03:31.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:31.903+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.906227+0000) 2019-10-13T11:03:31.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:31.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.939525+0000) 2019-10-13T11:03:32.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:32.127+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.129523+0000) 2019-10-13T11:03:33.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:33.124+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.126371+0000) 2019-10-13T11:03:36.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:36.371+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.374773+0000) 2019-10-13T11:03:36.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:36.675+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.677457+0000) 2019-10-13T11:03:36.679 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:36.675+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.677873+0000) 2019-10-13T11:03:36.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:36.675+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.678823+0000) 2019-10-13T11:03:36.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:36.675+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.679140+0000) 2019-10-13T11:03:36.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:36.679+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.680065+0000) 2019-10-13T11:03:36.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:36.676+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.680401+0000) 2019-10-13T11:03:36.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:36.679+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.681151+0000) 2019-10-13T11:03:36.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:36.683+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:36.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:36.684+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.687972+0000) 2019-10-13T11:03:36.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:36.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:36.710601+0000) 2019-10-13T11:03:37.390 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:03:37.390 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:37.394 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:03:37.394 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:38.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:38.639+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.639356+0000) 2019-10-13T11:03:38.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:38.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.644690+0000) 2019-10-13T11:03:41.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:41.683+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.686800+0000) 2019-10-13T11:03:41.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:41.687+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.687701+0000) 2019-10-13T11:03:41.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:41.687+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.688058+0000) 2019-10-13T11:03:41.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:41.687+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.688377+0000) 2019-10-13T11:03:41.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:41.684+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.688725+0000) 2019-10-13T11:03:41.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:41.687+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.689352+0000) 2019-10-13T11:03:41.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:41.687+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.689909+0000) 2019-10-13T11:03:41.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:41.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:41.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:41.688+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.693706+0000) 2019-10-13T11:03:41.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:41.903+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.906395+0000) 2019-10-13T11:03:41.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:41.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.939750+0000) 2019-10-13T11:03:42.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:42.127+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.129699+0000) 2019-10-13T11:03:43.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:43.124+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.126571+0000) 2019-10-13T11:03:46.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:46.374+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.374995+0000) 2019-10-13T11:03:46.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:46.694+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.696105+0000) 2019-10-13T11:03:46.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:46.694+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.697170+0000) 2019-10-13T11:03:46.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:46.691+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.697803+0000) 2019-10-13T11:03:46.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:46.694+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.698416+0000) 2019-10-13T11:03:46.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:46.694+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.699051+0000) 2019-10-13T11:03:46.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:46.694+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.699087+0000) 2019-10-13T11:03:46.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:46.698+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.699534+0000) 2019-10-13T11:03:46.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:46.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:46.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:46.699+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.702708+0000) 2019-10-13T11:03:46.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:46.711+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:46.716995+0000) 2019-10-13T11:03:48.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:48.638+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.639592+0000) 2019-10-13T11:03:48.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:48.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.644882+0000) 2019-10-13T11:03:51.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:51.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.706416+0000) 2019-10-13T11:03:51.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:51.702+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.706846+0000) 2019-10-13T11:03:51.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:51.706+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.707756+0000) 2019-10-13T11:03:51.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:51.703+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.708068+0000) 2019-10-13T11:03:51.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:51.706+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.708948+0000) 2019-10-13T11:03:51.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:51.706+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.709307+0000) 2019-10-13T11:03:51.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:51.706+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.709672+0000) 2019-10-13T11:03:51.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:51.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:51.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:51.711+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.714569+0000) 2019-10-13T11:03:51.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:51.902+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.906575+0000) 2019-10-13T11:03:51.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:51.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.939937+0000) 2019-10-13T11:03:52.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:52.126+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.129893+0000) 2019-10-13T11:03:53.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:53.123+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.126765+0000) 2019-10-13T11:03:56.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:56.374+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.375243+0000) 2019-10-13T11:03:56.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:03:56.710+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.715272+0000) 2019-10-13T11:03:56.717 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:03:56.714+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.715733+0000) 2019-10-13T11:03:56.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:56.714+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.715963+0000) 2019-10-13T11:03:56.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:03:56.711+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.716434+0000) 2019-10-13T11:03:56.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:03:56.714+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.717355+0000) 2019-10-13T11:03:56.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:03:56.714+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.718261+0000) 2019-10-13T11:03:56.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:56.714+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.718877+0000) 2019-10-13T11:03:56.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:03:56.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:03:56.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:56.723+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.726592+0000) 2019-10-13T11:03:56.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:03:56.723+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:56.726682+0000) 2019-10-13T11:03:58.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:03:58.638+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.639815+0000) 2019-10-13T11:03:58.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:03:58.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.645079+0000) 2019-10-13T11:04:01.726 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:01.722+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.724835+0000) 2019-10-13T11:04:01.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:01.722+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.725675+0000) 2019-10-13T11:04:01.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:01.722+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.726863+0000) 2019-10-13T11:04:01.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:01.723+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.727372+0000) 2019-10-13T11:04:01.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:01.726+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.728085+0000) 2019-10-13T11:04:01.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:01.726+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.728411+0000) 2019-10-13T11:04:01.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:01.726+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.728771+0000) 2019-10-13T11:04:01.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:01.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:01.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:01.727+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.731686+0000) 2019-10-13T11:04:01.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:01.906+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.906793+0000) 2019-10-13T11:04:01.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:01.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.940141+0000) 2019-10-13T11:04:02.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:02.126+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.130071+0000) 2019-10-13T11:04:03.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:03.123+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.127002+0000) 2019-10-13T11:04:06.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:06.374+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.375492+0000) 2019-10-13T11:04:06.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:06.723+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.726887+0000) 2019-10-13T11:04:06.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:06.730+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.732937+0000) 2019-10-13T11:04:06.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:06.727+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.734039+0000) 2019-10-13T11:04:06.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:06.730+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.734331+0000) 2019-10-13T11:04:06.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:06.730+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.734895+0000) 2019-10-13T11:04:06.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:06.734+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.735294+0000) 2019-10-13T11:04:06.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:06.734+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.735712+0000) 2019-10-13T11:04:06.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:06.734+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.736653+0000) 2019-10-13T11:04:06.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:06.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:06.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:06.739+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:06.742864+0000) 2019-10-13T11:04:07.510 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:04:07.510 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:07.514 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:04:07.514 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:08.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:08.638+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.640043+0000) 2019-10-13T11:04:08.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:08.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.645268+0000) 2019-10-13T11:04:11.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:11.739+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.742438+0000) 2019-10-13T11:04:11.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:11.738+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.743109+0000) 2019-10-13T11:04:11.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:11.742+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.743525+0000) 2019-10-13T11:04:11.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:11.742+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.744069+0000) 2019-10-13T11:04:11.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:11.742+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.744209+0000) 2019-10-13T11:04:11.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:11.742+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.744497+0000) 2019-10-13T11:04:11.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:11.742+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.745356+0000) 2019-10-13T11:04:11.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:11.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114883 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:11.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:11.747+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.750588+0000) 2019-10-13T11:04:11.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:11.906+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.906996+0000) 2019-10-13T11:04:11.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:11.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.940354+0000) 2019-10-13T11:04:12.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:12.126+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.130253+0000) 2019-10-13T11:04:13.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:13.123+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.127178+0000) 2019-10-13T11:04:16.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:16.374+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.375711+0000) 2019-10-13T11:04:16.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:16.731+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.733172+0000) 2019-10-13T11:04:16.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:16.747+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.750704+0000) 2019-10-13T11:04:16.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:16.746+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.751007+0000) 2019-10-13T11:04:16.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:16.750+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.751518+0000) 2019-10-13T11:04:16.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:16.750+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.752002+0000) 2019-10-13T11:04:16.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:16.750+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.752465+0000) 2019-10-13T11:04:16.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:16.750+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.753199+0000) 2019-10-13T11:04:16.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:16.750+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.754201+0000) 2019-10-13T11:04:16.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:16.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:16.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:16.751+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:16.758088+0000) 2019-10-13T11:04:18.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:18.638+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.640259+0000) 2019-10-13T11:04:18.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:18.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.645502+0000) 2019-10-13T11:04:21.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:21.758+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.760080+0000) 2019-10-13T11:04:21.762 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:21.755+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.760959+0000) 2019-10-13T11:04:21.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:21.758+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.761291+0000) 2019-10-13T11:04:21.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:21.758+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.761603+0000) 2019-10-13T11:04:21.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:21.758+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.762205+0000) 2019-10-13T11:04:21.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:21.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.763783+0000) 2019-10-13T11:04:21.765 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:21.762+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.764038+0000) 2019-10-13T11:04:21.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:21.762+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114949 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:21.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:21.763+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.770097+0000) 2019-10-13T11:04:21.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:21.906+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.907197+0000) 2019-10-13T11:04:21.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:21.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.940539+0000) 2019-10-13T11:04:22.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:22.130+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.130438+0000) 2019-10-13T11:04:23.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:23.127+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.127406+0000) 2019-10-13T11:04:26.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:26.374+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.375952+0000) 2019-10-13T11:04:26.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:26.735+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.739613+0000) 2019-10-13T11:04:26.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:26.766+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.769012+0000) 2019-10-13T11:04:26.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:26.766+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.770290+0000) 2019-10-13T11:04:26.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:26.766+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.770543+0000) 2019-10-13T11:04:26.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:26.766+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.771200+0000) 2019-10-13T11:04:26.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:26.767+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.771752+0000) 2019-10-13T11:04:26.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:26.770+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.771913+0000) 2019-10-13T11:04:26.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:26.770+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.772523+0000) 2019-10-13T11:04:26.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:26.770+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 114981 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:26.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:26.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:26.775758+0000) 2019-10-13T11:04:28.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:28.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.640524+0000) 2019-10-13T11:04:28.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:28.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.645731+0000) 2019-10-13T11:04:31.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:31.773+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.778265+0000) 2019-10-13T11:04:31.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:31.773+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.779065+0000) 2019-10-13T11:04:31.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:31.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.780631+0000) 2019-10-13T11:04:31.782 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:31.777+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.780835+0000) 2019-10-13T11:04:31.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:31.777+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.781135+0000) 2019-10-13T11:04:31.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:31.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.781764+0000) 2019-10-13T11:04:31.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:31.779+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.782457+0000) 2019-10-13T11:04:31.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:31.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:31.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:31.783+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.786304+0000) 2019-10-13T11:04:31.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:31.905+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.907380+0000) 2019-10-13T11:04:31.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:31.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.940719+0000) 2019-10-13T11:04:32.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:32.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.130672+0000) 2019-10-13T11:04:33.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:33.127+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.127627+0000) 2019-10-13T11:04:36.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:36.373+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.376191+0000) 2019-10-13T11:04:36.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:36.739+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.745976+0000) 2019-10-13T11:04:36.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:36.785+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.788980+0000) 2019-10-13T11:04:36.790 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:36.783+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.789274+0000) 2019-10-13T11:04:36.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:36.785+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.790017+0000) 2019-10-13T11:04:36.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:36.785+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.789732+0000) 2019-10-13T11:04:36.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:36.785+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.790118+0000) 2019-10-13T11:04:36.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:36.785+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.790470+0000) 2019-10-13T11:04:36.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:36.789+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.791482+0000) 2019-10-13T11:04:36.793 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:36.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:36.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:36.791+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:36.794452+0000) 2019-10-13T11:04:37.608 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:04:37.609 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:37.612 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:04:37.612 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:38.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:38.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.640742+0000) 2019-10-13T11:04:38.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:38.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.645931+0000) 2019-10-13T11:04:41.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:41.793+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.796826+0000) 2019-10-13T11:04:41.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:41.793+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.797068+0000) 2019-10-13T11:04:41.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:41.793+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.797890+0000) 2019-10-13T11:04:41.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:41.793+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.798737+0000) 2019-10-13T11:04:41.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:41.794+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.799550+0000) 2019-10-13T11:04:41.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:41.797+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.799984+0000) 2019-10-13T11:04:41.801 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:41.797+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.800430+0000) 2019-10-13T11:04:41.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:41.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:41.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:41.798+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.805654+0000) 2019-10-13T11:04:41.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:41.905+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.907572+0000) 2019-10-13T11:04:41.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:41.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.940901+0000) 2019-10-13T11:04:42.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:42.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.130876+0000) 2019-10-13T11:04:43.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:43.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.127847+0000) 2019-10-13T11:04:46.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:46.373+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.376413+0000) 2019-10-13T11:04:46.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:46.750+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.752312+0000) 2019-10-13T11:04:46.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:46.801+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.804891+0000) 2019-10-13T11:04:46.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:46.801+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.805452+0000) 2019-10-13T11:04:46.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:46.801+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.806006+0000) 2019-10-13T11:04:46.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:46.801+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.806690+0000) 2019-10-13T11:04:46.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:46.805+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.807385+0000) 2019-10-13T11:04:46.809 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:46.802+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.808390+0000) 2019-10-13T11:04:46.809 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:46.805+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.808676+0000) 2019-10-13T11:04:46.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:46.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:46.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:46.810+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:46.814598+0000) 2019-10-13T11:04:48.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:48.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.640961+0000) 2019-10-13T11:04:48.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:48.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.646134+0000) 2019-10-13T11:04:51.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:51.813+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.815365+0000) 2019-10-13T11:04:51.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:51.813+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.816193+0000) 2019-10-13T11:04:51.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:51.813+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.816513+0000) 2019-10-13T11:04:51.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:51.813+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.816927+0000) 2019-10-13T11:04:51.819 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:51.813+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.817452+0000) 2019-10-13T11:04:51.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:51.813+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.818576+0000) 2019-10-13T11:04:51.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:51.814+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.818985+0000) 2019-10-13T11:04:51.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:51.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:51.825 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:51.818+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.824013+0000) 2019-10-13T11:04:51.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:51.905+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.907766+0000) 2019-10-13T11:04:51.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:51.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.941086+0000) 2019-10-13T11:04:52.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:52.129+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.131085+0000) 2019-10-13T11:04:53.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:53.126+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.128072+0000) 2019-10-13T11:04:56.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:56.373+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.376637+0000) 2019-10-13T11:04:56.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:56.754+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.758652+0000) 2019-10-13T11:04:56.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:56.821+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.824930+0000) 2019-10-13T11:04:56.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:04:56.821+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.825562+0000) 2019-10-13T11:04:56.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:04:56.821+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.825830+0000) 2019-10-13T11:04:56.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:04:56.821+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.827151+0000) 2019-10-13T11:04:56.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:04:56.822+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.827582+0000) 2019-10-13T11:04:56.829 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:04:56.825+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.827778+0000) 2019-10-13T11:04:56.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:56.825+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.828135+0000) 2019-10-13T11:04:56.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:04:56.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115176 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:04:56.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:04:56.826+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:56.833258+0000) 2019-10-13T11:04:58.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:04:58.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.641166+0000) 2019-10-13T11:04:58.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:04:58.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.646365+0000) 2019-10-13T11:05:01.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:01.829+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.834586+0000) 2019-10-13T11:05:01.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:01.829+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.835001+0000) 2019-10-13T11:05:01.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:01.833+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.835394+0000) 2019-10-13T11:05:01.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:01.833+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.835892+0000) 2019-10-13T11:05:01.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:01.830+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.836538+0000) 2019-10-13T11:05:01.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:01.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.836699+0000) 2019-10-13T11:05:01.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:01.833+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.837556+0000) 2019-10-13T11:05:01.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:01.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:01.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:01.838+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.844457+0000) 2019-10-13T11:05:01.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:01.905+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.907952+0000) 2019-10-13T11:05:01.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:01.941+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.941272+0000) 2019-10-13T11:05:02.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:02.125+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.131259+0000) 2019-10-13T11:05:03.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:03.122+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.128276+0000) 2019-10-13T11:05:06.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:06.373+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.376862+0000) 2019-10-13T11:05:06.766 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:06.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.765010+0000) 2019-10-13T11:05:06.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:06.837+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.843165+0000) 2019-10-13T11:05:06.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:06.841+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.844351+0000) 2019-10-13T11:05:06.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:06.841+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.844689+0000) 2019-10-13T11:05:06.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:06.838+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.845060+0000) 2019-10-13T11:05:06.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:06.841+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.845506+0000) 2019-10-13T11:05:06.847 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:06.841+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.845884+0000) 2019-10-13T11:05:06.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:06.841+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.846639+0000) 2019-10-13T11:05:06.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:06.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115241 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:06.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:06.842+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:06.849731+0000) 2019-10-13T11:05:07.749 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:05:07.749 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:07.753 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:05:07.753 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:08.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:08.637+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.641361+0000) 2019-10-13T11:05:08.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:08.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.646602+0000) 2019-10-13T11:05:11.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:11.848+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.851372+0000) 2019-10-13T11:05:11.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:11.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.852447+0000) 2019-10-13T11:05:11.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:11.848+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.852960+0000) 2019-10-13T11:05:11.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:11.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.853579+0000) 2019-10-13T11:05:11.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:11.848+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.853864+0000) 2019-10-13T11:05:11.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:11.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.854521+0000) 2019-10-13T11:05:11.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:11.850+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.854835+0000) 2019-10-13T11:05:11.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:11.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:11.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:11.854+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.861792+0000) 2019-10-13T11:05:11.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:11.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.908146+0000) 2019-10-13T11:05:11.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:11.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.941489+0000) 2019-10-13T11:05:12.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:12.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.131464+0000) 2019-10-13T11:05:13.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:13.126+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.128479+0000) 2019-10-13T11:05:16.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:16.372+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.377060+0000) 2019-10-13T11:05:16.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:16.770+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.771427+0000) 2019-10-13T11:05:16.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:16.856+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.860386+0000) 2019-10-13T11:05:16.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:16.856+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.860901+0000) 2019-10-13T11:05:16.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:16.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.861530+0000) 2019-10-13T11:05:16.863 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:16.856+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.861961+0000) 2019-10-13T11:05:16.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:16.858+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.862617+0000) 2019-10-13T11:05:16.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:16.860+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.863373+0000) 2019-10-13T11:05:16.864 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:16.860+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.863926+0000) 2019-10-13T11:05:16.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:16.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115306 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:16.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:16.862+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:16.870060+0000) 2019-10-13T11:05:18.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:18.640+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.641558+0000) 2019-10-13T11:05:18.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:18.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.646817+0000) 2019-10-13T11:05:21.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:21.866+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.869892+0000) 2019-10-13T11:05:21.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:21.864+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.870520+0000) 2019-10-13T11:05:21.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:21.864+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.871143+0000) 2019-10-13T11:05:21.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:21.868+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.871654+0000) 2019-10-13T11:05:21.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:21.868+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.872050+0000) 2019-10-13T11:05:21.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:21.868+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.872414+0000) 2019-10-13T11:05:21.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:21.868+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.873644+0000) 2019-10-13T11:05:21.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:21.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:21.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:21.874+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.879832+0000) 2019-10-13T11:05:21.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:21.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.908349+0000) 2019-10-13T11:05:21.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:21.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.941684+0000) 2019-10-13T11:05:22.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:22.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.131643+0000) 2019-10-13T11:05:23.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:23.126+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.128675+0000) 2019-10-13T11:05:26.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:26.376+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.377282+0000) 2019-10-13T11:05:26.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:26.774+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.777838+0000) 2019-10-13T11:05:26.880 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:26.872+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.879066+0000) 2019-10-13T11:05:26.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:26.876+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.879887+0000) 2019-10-13T11:05:26.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:26.876+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.880290+0000) 2019-10-13T11:05:26.882 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:26.876+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.881134+0000) 2019-10-13T11:05:26.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:26.878+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.881841+0000) 2019-10-13T11:05:26.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:26.876+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.881376+0000) 2019-10-13T11:05:26.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:26.876+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.882105+0000) 2019-10-13T11:05:26.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:26.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:26.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:26.886+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:26.889246+0000) 2019-10-13T11:05:28.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:28.636+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641747+0000) 2019-10-13T11:05:28.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:28.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.647005+0000) 2019-10-13T11:05:31.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:31.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.888444+0000) 2019-10-13T11:05:31.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:31.884+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.888971+0000) 2019-10-13T11:05:31.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:31.884+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.890082+0000) 2019-10-13T11:05:31.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:31.884+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.890186+0000) 2019-10-13T11:05:31.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:31.890+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.890832+0000) 2019-10-13T11:05:31.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:31.888+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.891489+0000) 2019-10-13T11:05:31.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:31.888+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.892202+0000) 2019-10-13T11:05:31.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:31.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:31.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:31.894+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.897602+0000) 2019-10-13T11:05:31.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:31.904+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.908491+0000) 2019-10-13T11:05:31.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:31.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.941866+0000) 2019-10-13T11:05:32.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:32.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.131782+0000) 2019-10-13T11:05:33.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:33.126+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.128873+0000) 2019-10-13T11:05:36.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:36.376+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.377493+0000) 2019-10-13T11:05:36.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:36.781+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.784550+0000) 2019-10-13T11:05:36.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:36.892+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.898184+0000) 2019-10-13T11:05:36.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:36.892+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.898569+0000) 2019-10-13T11:05:36.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:36.892+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.898817+0000) 2019-10-13T11:05:36.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:36.897+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.899415+0000) 2019-10-13T11:05:36.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:36.892+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.899189+0000) 2019-10-13T11:05:36.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:36.896+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.900055+0000) 2019-10-13T11:05:36.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:36.896+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.901419+0000) 2019-10-13T11:05:36.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:36.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:36.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:36.901+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:36.903433+0000) 2019-10-13T11:05:37.860 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:05:37.861 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:37.864 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:05:37.865 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:38.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:38.640+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.641979+0000) 2019-10-13T11:05:38.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:38.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.647205+0000) 2019-10-13T11:05:41.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:41.900+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.906892+0000) 2019-10-13T11:05:41.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:41.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.907116+0000) 2019-10-13T11:05:41.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:41.904+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.907530+0000) 2019-10-13T11:05:41.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:41.904+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.907924+0000) 2019-10-13T11:05:41.909 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:41.904+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.908328+0000) 2019-10-13T11:05:41.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:41.905+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.909011+0000) 2019-10-13T11:05:41.911 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:41.904+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.910291+0000) 2019-10-13T11:05:41.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:41.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:41.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:41.909+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.912851+0000) 2019-10-13T11:05:41.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:41.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.942032+0000) 2019-10-13T11:05:42.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:42.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.131972+0000) 2019-10-13T11:05:43.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:43.125+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.129082+0000) 2019-10-13T11:05:46.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:46.376+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.377724+0000) 2019-10-13T11:05:46.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:46.789+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.790959+0000) 2019-10-13T11:05:46.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:46.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.914825+0000) 2019-10-13T11:05:46.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:46.913+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.915241+0000) 2019-10-13T11:05:46.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:46.912+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.915541+0000) 2019-10-13T11:05:46.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:46.912+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.915937+0000) 2019-10-13T11:05:46.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:46.912+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.916358+0000) 2019-10-13T11:05:46.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:46.912+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.917318+0000) 2019-10-13T11:05:46.918 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:46.912+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.917213+0000) 2019-10-13T11:05:46.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:46.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:46.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:46.921+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:46.925043+0000) 2019-10-13T11:05:48.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:48.640+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.642210+0000) 2019-10-13T11:05:48.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:48.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.647457+0000) 2019-10-13T11:05:48.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:48.656+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.659145+0000) 2019-10-13T11:05:51.925 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:51.920+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.924011+0000) 2019-10-13T11:05:51.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:51.920+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.924604+0000) 2019-10-13T11:05:51.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:51.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.925189+0000) 2019-10-13T11:05:51.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:51.920+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.925725+0000) 2019-10-13T11:05:51.927 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:51.920+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.925908+0000) 2019-10-13T11:05:51.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:51.920+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.926499+0000) 2019-10-13T11:05:51.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:51.924+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.927563+0000) 2019-10-13T11:05:51.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:51.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:51.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:51.929+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.932959+0000) 2019-10-13T11:05:51.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:51.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.942170+0000) 2019-10-13T11:05:52.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:52.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.132155+0000) 2019-10-13T11:05:53.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:53.125+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.129223+0000) 2019-10-13T11:05:56.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:56.371+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.377897+0000) 2019-10-13T11:05:56.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:56.793+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.797490+0000) 2019-10-13T11:05:56.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:05:56.927+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.933314+0000) 2019-10-13T11:05:56.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:05:56.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.934081+0000) 2019-10-13T11:05:56.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:05:56.931+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.935517+0000) 2019-10-13T11:05:56.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:56.931+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.935860+0000) 2019-10-13T11:05:56.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:56.931+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.936874+0000) 2019-10-13T11:05:56.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:56.931+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.937303+0000) 2019-10-13T11:05:56.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:05:56.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:05:56.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:05:56.933+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.936647+0000) 2019-10-13T11:05:56.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:05:56.937+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:56.941935+0000) 2019-10-13T11:05:58.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:05:58.639+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.642440+0000) 2019-10-13T11:05:58.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:05:58.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.647652+0000) 2019-10-13T11:05:58.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:05:58.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.659358+0000) 2019-10-13T11:06:01.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:01.935+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.942489+0000) 2019-10-13T11:06:01.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:01.939+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.944198+0000) 2019-10-13T11:06:01.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:01.941+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.943953+0000) 2019-10-13T11:06:01.946 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:01.939+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.944560+0000) 2019-10-13T11:06:01.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:01.939+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.945067+0000) 2019-10-13T11:06:01.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:01.939+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.945519+0000) 2019-10-13T11:06:01.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:01.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.946298+0000) 2019-10-13T11:06:01.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:01.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:01.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:01.949+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.951528+0000) 2019-10-13T11:06:02.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:02.131+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.132341+0000) 2019-10-13T11:06:03.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:03.125+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.129416+0000) 2019-10-13T11:06:06.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:06.375+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.378123+0000) 2019-10-13T11:06:06.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:06.801+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.803877+0000) 2019-10-13T11:06:06.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:06.949+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.952122+0000) 2019-10-13T11:06:06.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:06.947+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.952964+0000) 2019-10-13T11:06:06.955 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:06.947+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.953993+0000) 2019-10-13T11:06:06.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:06.947+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.954023+0000) 2019-10-13T11:06:06.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:06.947+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.954321+0000) 2019-10-13T11:06:06.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:06.947+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.954611+0000) 2019-10-13T11:06:06.956 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:06.947+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.955239+0000) 2019-10-13T11:06:06.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:06.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:06.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:06.961+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:06.963359+0000) 2019-10-13T11:06:07.971 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:06:07.971 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:07.975 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:06:07.975 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:08.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:08.639+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.642643+0000) 2019-10-13T11:06:08.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:08.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.647877+0000) 2019-10-13T11:06:08.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:08.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.659553+0000) 2019-10-13T11:06:08.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:08.687+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.692796+0000) 2019-10-13T11:06:11.962 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:11.955+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.960949+0000) 2019-10-13T11:06:11.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:11.957+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.961748+0000) 2019-10-13T11:06:11.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:11.955+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.962979+0000) 2019-10-13T11:06:11.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:11.955+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.963076+0000) 2019-10-13T11:06:11.964 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:11.959+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.963451+0000) 2019-10-13T11:06:11.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:11.959+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.964191+0000) 2019-10-13T11:06:11.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:11.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.964479+0000) 2019-10-13T11:06:11.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:11.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115660 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:11.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:11.969+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.970162+0000) 2019-10-13T11:06:12.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:12.127+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.132520+0000) 2019-10-13T11:06:13.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:13.125+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.129643+0000) 2019-10-13T11:06:16.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:16.375+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.378318+0000) 2019-10-13T11:06:16.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:16.809+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.810277+0000) 2019-10-13T11:06:16.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:16.963+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.970001+0000) 2019-10-13T11:06:16.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:16.963+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.970172+0000) 2019-10-13T11:06:16.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:16.963+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.971242+0000) 2019-10-13T11:06:16.972 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:16.967+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.971500+0000) 2019-10-13T11:06:16.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:16.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.972256+0000) 2019-10-13T11:06:16.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:16.967+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.971888+0000) 2019-10-13T11:06:16.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:16.967+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.972841+0000) 2019-10-13T11:06:16.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:16.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:16.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:16.977+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:16.978958+0000) 2019-10-13T11:06:18.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:18.639+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.642864+0000) 2019-10-13T11:06:18.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:18.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.648072+0000) 2019-10-13T11:06:18.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:18.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.659755+0000) 2019-10-13T11:06:18.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:18.687+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.693001+0000) 2019-10-13T11:06:21.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:21.975+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.979681+0000) 2019-10-13T11:06:21.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:21.975+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.979996+0000) 2019-10-13T11:06:21.982 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:21.975+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.981225+0000) 2019-10-13T11:06:21.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:21.975+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.981800+0000) 2019-10-13T11:06:21.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:21.975+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.981893+0000) 2019-10-13T11:06:21.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:21.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.982611+0000) 2019-10-13T11:06:21.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:21.975+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.983217+0000) 2019-10-13T11:06:21.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:21.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:21.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:21.985+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.988050+0000) 2019-10-13T11:06:22.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:22.131+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.132746+0000) 2019-10-13T11:06:23.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:23.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.129892+0000) 2019-10-13T11:06:26.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:26.375+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.378528+0000) 2019-10-13T11:06:26.818 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:26.813+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.816790+0000) 2019-10-13T11:06:26.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:26.987+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.988460+0000) 2019-10-13T11:06:26.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:26.987+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.989104+0000) 2019-10-13T11:06:26.990 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:26.987+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.989506+0000) 2019-10-13T11:06:26.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:26.987+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.990078+0000) 2019-10-13T11:06:26.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:26.987+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.990499+0000) 2019-10-13T11:06:26.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:26.989+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.990547+0000) 2019-10-13T11:06:26.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:26.991+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.991609+0000) 2019-10-13T11:06:26.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:26.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115762 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:26.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:26.997+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:26.998671+0000) 2019-10-13T11:06:28.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:28.639+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.643093+0000) 2019-10-13T11:06:28.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:28.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.648227+0000) 2019-10-13T11:06:28.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:28.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.659985+0000) 2019-10-13T11:06:28.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:28.691+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.693225+0000) 2019-10-13T11:06:32.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:31.999+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.999407+0000) 2019-10-13T11:06:32.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:31.999+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.999579+0000) 2019-10-13T11:06:32.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:31.999+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.000560+0000) 2019-10-13T11:06:32.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:31.999+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.000533+0000) 2019-10-13T11:06:32.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:31.996+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.000787+0000) 2019-10-13T11:06:32.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:31.999+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.001129+0000) 2019-10-13T11:06:32.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:31.999+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.001857+0000) 2019-10-13T11:06:32.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:32.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:32.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:32.004+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.007310+0000) 2019-10-13T11:06:32.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:32.131+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.132933+0000) 2019-10-13T11:06:33.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:33.124+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.130119+0000) 2019-10-13T11:06:36.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:36.375+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:36.378739+0000) 2019-10-13T11:06:36.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:36.824+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:36.828607+0000) 2019-10-13T11:06:37.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:37.004+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.007500+0000) 2019-10-13T11:06:37.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:37.007+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.008423+0000) 2019-10-13T11:06:37.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:37.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.009118+0000) 2019-10-13T11:06:37.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:37.007+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.010066+0000) 2019-10-13T11:06:37.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:37.007+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.010532+0000) 2019-10-13T11:06:37.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:37.007+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.011035+0000) 2019-10-13T11:06:37.012 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:37.007+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.010935+0000) 2019-10-13T11:06:37.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:37.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:37.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:37.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.015969+0000) 2019-10-13T11:06:38.101 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:06:38.102 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:38.105 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:06:38.105 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:38.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:38.642+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.643345+0000) 2019-10-13T11:06:38.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:38.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.648423+0000) 2019-10-13T11:06:38.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:38.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.660208+0000) 2019-10-13T11:06:38.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:38.690+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.693442+0000) 2019-10-13T11:06:42.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:42.014+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.016850+0000) 2019-10-13T11:06:42.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:42.014+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.017623+0000) 2019-10-13T11:06:42.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:42.014+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.018019+0000) 2019-10-13T11:06:42.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:42.016+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.018413+0000) 2019-10-13T11:06:42.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:42.018+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.019424+0000) 2019-10-13T11:06:42.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:42.018+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.019631+0000) 2019-10-13T11:06:42.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:42.018+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.020079+0000) 2019-10-13T11:06:42.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:42.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:42.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:42.020+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.025211+0000) 2019-10-13T11:06:42.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:42.130+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.133113+0000) 2019-10-13T11:06:43.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:43.128+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.130368+0000) 2019-10-13T11:06:46.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:46.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:46.378931+0000) 2019-10-13T11:06:46.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:46.832+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:46.834895+0000) 2019-10-13T11:06:47.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:47.022+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.026299+0000) 2019-10-13T11:06:47.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:47.022+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.026779+0000) 2019-10-13T11:06:47.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:47.026+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.028098+0000) 2019-10-13T11:06:47.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:47.026+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.028102+0000) 2019-10-13T11:06:47.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:47.026+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.028502+0000) 2019-10-13T11:06:47.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:47.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.028966+0000) 2019-10-13T11:06:47.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:47.026+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.029285+0000) 2019-10-13T11:06:47.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:47.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:47.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:47.032+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.037680+0000) 2019-10-13T11:06:48.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:48.642+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.643582+0000) 2019-10-13T11:06:48.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:48.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.648618+0000) 2019-10-13T11:06:48.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:48.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.660408+0000) 2019-10-13T11:06:48.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:48.690+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.693654+0000) 2019-10-13T11:06:52.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:52.030+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.034873+0000) 2019-10-13T11:06:52.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:52.030+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.035138+0000) 2019-10-13T11:06:52.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:52.034+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.035873+0000) 2019-10-13T11:06:52.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:52.034+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.036219+0000) 2019-10-13T11:06:52.038 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:52.034+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.036926+0000) 2019-10-13T11:06:52.038 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:52.034+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.037331+0000) 2019-10-13T11:06:52.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:52.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.038643+0000) 2019-10-13T11:06:52.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:52.038+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:52.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:52.040+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.043332+0000) 2019-10-13T11:06:52.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:52.130+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.133292+0000) 2019-10-13T11:06:53.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:53.128+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.130585+0000) 2019-10-13T11:06:56.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:56.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:56.379126+0000) 2019-10-13T11:06:56.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:56.840+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:56.842840+0000) 2019-10-13T11:06:57.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:06:57.042+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.043536+0000) 2019-10-13T11:06:57.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:06:57.042+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.043916+0000) 2019-10-13T11:06:57.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:57.042+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.044501+0000) 2019-10-13T11:06:57.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:57.042+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.045423+0000) 2019-10-13T11:06:57.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:06:57.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.045815+0000) 2019-10-13T11:06:57.047 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:57.042+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.046566+0000) 2019-10-13T11:06:57.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:57.042+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.046564+0000) 2019-10-13T11:06:57.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:06:57.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115960 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:06:57.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:06:57.048+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.053335+0000) 2019-10-13T11:06:58.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:06:58.642+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.643783+0000) 2019-10-13T11:06:58.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:06:58.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.648841+0000) 2019-10-13T11:06:58.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:06:58.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.660607+0000) 2019-10-13T11:06:58.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:06:58.690+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.693854+0000) 2019-10-13T11:07:02.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:02.050+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.052819+0000) 2019-10-13T11:07:02.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:02.050+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.053217+0000) 2019-10-13T11:07:02.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:02.050+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.053836+0000) 2019-10-13T11:07:02.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:02.050+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.054301+0000) 2019-10-13T11:07:02.056 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:02.050+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.055140+0000) 2019-10-13T11:07:02.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:02.052+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.055465+0000) 2019-10-13T11:07:02.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:02.054+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.055986+0000) 2019-10-13T11:07:02.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:02.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 115991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:02.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:02.060+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.062792+0000) 2019-10-13T11:07:02.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:02.130+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.133507+0000) 2019-10-13T11:07:03.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:03.128+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.130788+0000) 2019-10-13T11:07:06.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:06.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:06.379352+0000) 2019-10-13T11:07:06.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:06.848+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:06.849252+0000) 2019-10-13T11:07:07.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:07.058+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.062090+0000) 2019-10-13T11:07:07.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:07.060+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.062830+0000) 2019-10-13T11:07:07.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:07.058+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.062595+0000) 2019-10-13T11:07:07.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:07.062+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.063405+0000) 2019-10-13T11:07:07.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:07.062+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.063490+0000) 2019-10-13T11:07:07.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:07.062+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.064186+0000) 2019-10-13T11:07:07.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:07.062+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.064999+0000) 2019-10-13T11:07:07.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:07.062+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116025 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:07.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:07.068+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.070500+0000) 2019-10-13T11:07:08.336 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:07:08.337 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:08.341 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:07:08.341 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:08.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:08.642+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.643986+0000) 2019-10-13T11:07:08.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:08.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.649040+0000) 2019-10-13T11:07:08.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:08.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.660835+0000) 2019-10-13T11:07:08.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:08.690+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.694054+0000) 2019-10-13T11:07:12.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:12.066+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.069504+0000) 2019-10-13T11:07:12.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:12.066+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.070411+0000) 2019-10-13T11:07:12.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:12.066+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.071188+0000) 2019-10-13T11:07:12.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:12.070+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.071647+0000) 2019-10-13T11:07:12.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:12.068+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.072228+0000) 2019-10-13T11:07:12.074 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:12.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.073149+0000) 2019-10-13T11:07:12.075 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:12.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.073346+0000) 2019-10-13T11:07:12.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:12.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:12.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:12.076+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.079152+0000) 2019-10-13T11:07:12.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:12.130+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.133687+0000) 2019-10-13T11:07:13.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:13.128+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.130984+0000) 2019-10-13T11:07:16.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:16.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:16.379574+0000) 2019-10-13T11:07:16.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:16.856+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:16.859663+0000) 2019-10-13T11:07:17.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:17.078+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.079581+0000) 2019-10-13T11:07:17.081 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:17.074+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.079276+0000) 2019-10-13T11:07:17.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:17.076+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.080016+0000) 2019-10-13T11:07:17.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:17.078+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.080276+0000) 2019-10-13T11:07:17.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:17.078+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.080682+0000) 2019-10-13T11:07:17.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:17.078+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.081219+0000) 2019-10-13T11:07:17.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:17.078+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.081891+0000) 2019-10-13T11:07:17.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:17.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:17.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:17.084+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.088298+0000) 2019-10-13T11:07:18.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:18.642+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.644195+0000) 2019-10-13T11:07:18.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:18.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.649262+0000) 2019-10-13T11:07:18.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:18.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.661030+0000) 2019-10-13T11:07:18.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:18.690+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.694289+0000) 2019-10-13T11:07:22.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:22.085+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.087671+0000) 2019-10-13T11:07:22.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:22.085+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.088266+0000) 2019-10-13T11:07:22.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:22.085+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.088828+0000) 2019-10-13T11:07:22.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:22.085+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.089640+0000) 2019-10-13T11:07:22.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:22.085+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.090140+0000) 2019-10-13T11:07:22.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:22.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.090662+0000) 2019-10-13T11:07:22.092 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:22.085+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.090373+0000) 2019-10-13T11:07:22.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:22.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:22.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:22.092+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.097610+0000) 2019-10-13T11:07:22.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:22.129+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.133870+0000) 2019-10-13T11:07:23.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:23.128+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.131176+0000) 2019-10-13T11:07:26.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:26.377+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:26.379796+0000) 2019-10-13T11:07:26.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:26.859+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:26.865962+0000) 2019-10-13T11:07:27.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:27.093+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.096934+0000) 2019-10-13T11:07:27.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:27.093+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.097886+0000) 2019-10-13T11:07:27.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:27.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.098525+0000) 2019-10-13T11:07:27.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:27.093+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.099121+0000) 2019-10-13T11:07:27.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:27.095+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.099663+0000) 2019-10-13T11:07:27.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:27.097+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.100511+0000) 2019-10-13T11:07:27.102 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:27.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.101043+0000) 2019-10-13T11:07:27.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:27.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:27.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:27.103+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.106620+0000) 2019-10-13T11:07:28.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:28.641+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.644423+0000) 2019-10-13T11:07:28.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:28.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.649475+0000) 2019-10-13T11:07:28.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:28.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.661217+0000) 2019-10-13T11:07:28.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:28.689+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.694508+0000) 2019-10-13T11:07:32.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:32.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.106743+0000) 2019-10-13T11:07:32.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:32.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.106861+0000) 2019-10-13T11:07:32.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:32.105+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.107638+0000) 2019-10-13T11:07:32.109 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:32.105+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.108039+0000) 2019-10-13T11:07:32.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:32.105+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.108353+0000) 2019-10-13T11:07:32.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:32.103+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.108820+0000) 2019-10-13T11:07:32.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:32.105+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.109522+0000) 2019-10-13T11:07:32.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:32.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:32.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:32.111+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.115477+0000) 2019-10-13T11:07:32.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:32.129+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.134104+0000) 2019-10-13T11:07:33.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:33.127+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.131369+0000) 2019-10-13T11:07:36.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:36.377+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:36.380025+0000) 2019-10-13T11:07:36.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:36.871+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:36.872373+0000) 2019-10-13T11:07:37.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:37.113+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.116209+0000) 2019-10-13T11:07:37.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:37.113+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.116841+0000) 2019-10-13T11:07:37.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:37.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.117640+0000) 2019-10-13T11:07:37.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:37.113+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.118051+0000) 2019-10-13T11:07:37.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:37.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.118922+0000) 2019-10-13T11:07:37.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:37.117+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.119397+0000) 2019-10-13T11:07:37.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:37.115+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.120192+0000) 2019-10-13T11:07:37.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:37.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116224 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:37.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:37.119+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.123777+0000) 2019-10-13T11:07:38.456 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:07:38.456 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:38.459 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:07:38.460 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:38.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:38.641+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.644629+0000) 2019-10-13T11:07:38.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:38.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.649671+0000) 2019-10-13T11:07:38.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:38.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.661459+0000) 2019-10-13T11:07:38.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:38.689+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.694742+0000) 2019-10-13T11:07:42.125 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:42.121+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.124389+0000) 2019-10-13T11:07:42.126 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:42.121+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.125436+0000) 2019-10-13T11:07:42.126 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:42.121+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.125884+0000) 2019-10-13T11:07:42.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:42.121+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.126558+0000) 2019-10-13T11:07:42.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:42.121+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.126788+0000) 2019-10-13T11:07:42.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:42.125+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.127625+0000) 2019-10-13T11:07:42.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:42.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.128137+0000) 2019-10-13T11:07:42.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:42.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:42.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:42.127+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.132665+0000) 2019-10-13T11:07:42.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:42.129+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.134237+0000) 2019-10-13T11:07:43.132 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:43.127+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.131629+0000) 2019-10-13T11:07:46.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:46.377+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:46.380232+0000) 2019-10-13T11:07:46.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:46.875+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:46.878886+0000) 2019-10-13T11:07:47.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:47.129+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.134775+0000) 2019-10-13T11:07:47.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:47.133+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.135804+0000) 2019-10-13T11:07:47.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:47.133+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.136375+0000) 2019-10-13T11:07:47.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:47.133+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.136842+0000) 2019-10-13T11:07:47.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:47.133+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.137098+0000) 2019-10-13T11:07:47.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:47.133+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.137653+0000) 2019-10-13T11:07:47.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:47.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.138502+0000) 2019-10-13T11:07:47.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:47.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:47.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:47.139+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.145372+0000) 2019-10-13T11:07:48.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:48.641+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.644818+0000) 2019-10-13T11:07:48.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:48.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.649882+0000) 2019-10-13T11:07:48.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:48.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.661667+0000) 2019-10-13T11:07:48.696 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:48.689+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.694978+0000) 2019-10-13T11:07:52.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:52.129+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.134429+0000) 2019-10-13T11:07:52.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:52.141+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.143965+0000) 2019-10-13T11:07:52.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:52.141+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.144670+0000) 2019-10-13T11:07:52.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:52.141+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.145785+0000) 2019-10-13T11:07:52.147 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:52.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.146072+0000) 2019-10-13T11:07:52.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:52.141+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.146828+0000) 2019-10-13T11:07:52.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:52.141+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.146911+0000) 2019-10-13T11:07:52.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:52.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.147992+0000) 2019-10-13T11:07:52.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:52.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:52.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:52.147+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.151830+0000) 2019-10-13T11:07:53.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:53.131+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.131849+0000) 2019-10-13T11:07:56.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:56.377+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:56.380456+0000) 2019-10-13T11:07:56.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:56.883+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:56.889617+0000) 2019-10-13T11:07:57.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:57.149+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.153878+0000) 2019-10-13T11:07:57.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:07:57.149+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.154186+0000) 2019-10-13T11:07:57.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:07:57.151+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.154504+0000) 2019-10-13T11:07:57.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:57.153+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.155754+0000) 2019-10-13T11:07:57.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:07:57.153+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.156243+0000) 2019-10-13T11:07:57.157 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:57.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.156808+0000) 2019-10-13T11:07:57.158 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:57.153+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.157440+0000) 2019-10-13T11:07:57.160 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:07:57.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:07:57.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:07:57.155+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.161695+0000) 2019-10-13T11:07:58.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:07:58.641+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.645015+0000) 2019-10-13T11:07:58.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:07:58.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.650078+0000) 2019-10-13T11:07:58.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:07:58.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.661896+0000) 2019-10-13T11:07:58.696 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:07:58.689+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.695204+0000) 2019-10-13T11:08:02.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:02.133+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.134664+0000) 2019-10-13T11:08:02.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:02.159+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.162441+0000) 2019-10-13T11:08:02.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:02.161+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.163676+0000) 2019-10-13T11:08:02.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:02.161+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.164060+0000) 2019-10-13T11:08:02.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:02.161+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.164808+0000) 2019-10-13T11:08:02.166 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:02.161+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.165271+0000) 2019-10-13T11:08:02.166 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:02.161+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.165726+0000) 2019-10-13T11:08:02.167 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:02.161+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.166885+0000) 2019-10-13T11:08:02.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:02.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:02.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:02.167+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.173983+0000) 2019-10-13T11:08:03.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:03.127+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.131997+0000) 2019-10-13T11:08:06.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:06.376+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:06.380669+0000) 2019-10-13T11:08:06.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:06.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:06.896116+0000) 2019-10-13T11:08:07.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:07.168+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.172866+0000) 2019-10-13T11:08:07.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:07.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.173155+0000) 2019-10-13T11:08:07.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:07.168+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.174004+0000) 2019-10-13T11:08:07.175 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:07.168+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.174501+0000) 2019-10-13T11:08:07.176 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:07.168+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.174848+0000) 2019-10-13T11:08:07.176 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:07.168+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.175256+0000) 2019-10-13T11:08:07.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:07.172+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.175810+0000) 2019-10-13T11:08:07.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:07.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:07.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:07.175+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.179965+0000) 2019-10-13T11:08:08.592 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:08:08.592 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:08.596 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:08:08.596 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:08.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:08.640+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.645206+0000) 2019-10-13T11:08:08.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:08.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.650282+0000) 2019-10-13T11:08:08.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:08.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.662131+0000) 2019-10-13T11:08:08.696 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:08.692+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.695405+0000) 2019-10-13T11:08:12.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:12.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.134865+0000) 2019-10-13T11:08:12.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:12.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.182085+0000) 2019-10-13T11:08:12.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:12.176+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.182760+0000) 2019-10-13T11:08:12.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:12.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.182840+0000) 2019-10-13T11:08:12.184 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:12.180+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.183807+0000) 2019-10-13T11:08:12.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:12.179+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.184249+0000) 2019-10-13T11:08:12.185 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:12.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.184582+0000) 2019-10-13T11:08:12.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:12.180+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.184859+0000) 2019-10-13T11:08:12.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:12.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:12.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:12.187+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.191974+0000) 2019-10-13T11:08:13.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:13.131+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.132166+0000) 2019-10-13T11:08:16.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:16.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:16.380895+0000) 2019-10-13T11:08:16.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:16.899+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:16.902690+0000) 2019-10-13T11:08:17.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:17.184+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.190832+0000) 2019-10-13T11:08:17.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:17.187+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.191684+0000) 2019-10-13T11:08:17.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:17.188+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.192376+0000) 2019-10-13T11:08:17.194 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:17.188+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.193200+0000) 2019-10-13T11:08:17.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:17.188+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.193922+0000) 2019-10-13T11:08:17.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:17.188+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.194942+0000) 2019-10-13T11:08:17.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:17.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.195267+0000) 2019-10-13T11:08:17.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:17.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:17.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:17.199+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.202187+0000) 2019-10-13T11:08:18.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:18.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.645429+0000) 2019-10-13T11:08:18.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:18.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.650476+0000) 2019-10-13T11:08:18.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:18.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.662363+0000) 2019-10-13T11:08:18.696 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:18.692+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.695611+0000) 2019-10-13T11:08:22.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:22.132+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.135071+0000) 2019-10-13T11:08:22.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:22.196+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.201564+0000) 2019-10-13T11:08:22.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:22.196+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.202023+0000) 2019-10-13T11:08:22.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:22.196+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.202460+0000) 2019-10-13T11:08:22.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:22.196+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.202974+0000) 2019-10-13T11:08:22.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:22.200+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.203389+0000) 2019-10-13T11:08:22.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:22.198+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.203782+0000) 2019-10-13T11:08:22.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:22.200+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.204171+0000) 2019-10-13T11:08:22.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:22.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:22.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:22.206+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.211451+0000) 2019-10-13T11:08:23.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:23.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.132340+0000) 2019-10-13T11:08:26.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:26.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:26.381095+0000) 2019-10-13T11:08:26.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:26.906+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:26.909046+0000) 2019-10-13T11:08:27.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:27.204+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.209857+0000) 2019-10-13T11:08:27.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:27.204+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.210524+0000) 2019-10-13T11:08:27.212 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:27.204+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.211153+0000) 2019-10-13T11:08:27.212 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:27.208+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.211612+0000) 2019-10-13T11:08:27.213 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:27.208+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.212233+0000) 2019-10-13T11:08:27.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:27.208+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.212400+0000) 2019-10-13T11:08:27.214 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:27.206+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.212783+0000) 2019-10-13T11:08:27.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:27.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:27.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:27.210+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.216547+0000) 2019-10-13T11:08:28.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:28.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.645656+0000) 2019-10-13T11:08:28.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:28.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.650704+0000) 2019-10-13T11:08:28.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:28.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.662593+0000) 2019-10-13T11:08:28.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:28.692+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.695839+0000) 2019-10-13T11:08:32.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:32.128+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.135270+0000) 2019-10-13T11:08:32.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:32.212+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.218553+0000) 2019-10-13T11:08:32.221 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:32.216+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.219522+0000) 2019-10-13T11:08:32.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:32.216+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.220251+0000) 2019-10-13T11:08:32.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:32.216+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.220732+0000) 2019-10-13T11:08:32.222 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:32.214+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.221185+0000) 2019-10-13T11:08:32.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:32.216+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.221419+0000) 2019-10-13T11:08:32.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:32.216+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.221964+0000) 2019-10-13T11:08:32.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:32.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:32.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:32.222+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.229159+0000) 2019-10-13T11:08:33.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:33.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:33.132529+0000) 2019-10-13T11:08:36.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:36.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:36.381289+0000) 2019-10-13T11:08:36.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:36.918+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:36.920400+0000) 2019-10-13T11:08:37.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:37.224+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.228261+0000) 2019-10-13T11:08:37.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:37.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.229154+0000) 2019-10-13T11:08:37.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:37.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.229877+0000) 2019-10-13T11:08:37.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:37.224+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.230434+0000) 2019-10-13T11:08:37.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:37.224+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.230565+0000) 2019-10-13T11:08:37.232 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:37.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.231060+0000) 2019-10-13T11:08:37.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:37.228+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.231430+0000) 2019-10-13T11:08:37.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:37.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:37.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:37.234+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.238398+0000) 2019-10-13T11:08:38.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:38.640+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.645854+0000) 2019-10-13T11:08:38.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:38.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.650907+0000) 2019-10-13T11:08:38.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:38.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.662787+0000) 2019-10-13T11:08:38.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:38.692+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.696062+0000) 2019-10-13T11:08:38.720 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:08:38.720 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:38.724 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:08:38.724 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:42.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:42.132+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.135535+0000) 2019-10-13T11:08:42.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:42.232+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.238115+0000) 2019-10-13T11:08:42.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:42.232+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.238610+0000) 2019-10-13T11:08:42.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:42.232+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.238933+0000) 2019-10-13T11:08:42.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:42.236+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.239741+0000) 2019-10-13T11:08:42.241 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:42.236+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.239587+0000) 2019-10-13T11:08:42.243 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:42.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.241030+0000) 2019-10-13T11:08:42.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:42.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.240449+0000) 2019-10-13T11:08:42.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:42.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:42.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:42.242+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.248055+0000) 2019-10-13T11:08:43.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:43.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:43.132706+0000) 2019-10-13T11:08:46.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:46.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:46.381477+0000) 2019-10-13T11:08:46.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:46.926+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:46.932202+0000) 2019-10-13T11:08:47.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:47.240+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.246681+0000) 2019-10-13T11:08:47.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:47.244+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.248022+0000) 2019-10-13T11:08:47.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:47.244+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.248631+0000) 2019-10-13T11:08:47.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:47.244+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.249086+0000) 2019-10-13T11:08:47.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:47.244+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.249736+0000) 2019-10-13T11:08:47.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:47.244+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.250147+0000) 2019-10-13T11:08:47.252 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:47.246+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.251163+0000) 2019-10-13T11:08:47.252 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:47.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:47.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:47.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.262705+0000) 2019-10-13T11:08:48.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:48.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.646114+0000) 2019-10-13T11:08:48.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:48.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.651109+0000) 2019-10-13T11:08:48.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:48.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.662993+0000) 2019-10-13T11:08:48.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:48.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.696260+0000) 2019-10-13T11:08:52.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:52.131+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.135729+0000) 2019-10-13T11:08:52.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:52.251+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.255638+0000) 2019-10-13T11:08:52.257 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:52.251+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.255912+0000) 2019-10-13T11:08:52.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:52.251+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.256687+0000) 2019-10-13T11:08:52.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:52.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.257899+0000) 2019-10-13T11:08:52.259 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:52.251+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.258532+0000) 2019-10-13T11:08:52.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:52.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.259036+0000) 2019-10-13T11:08:52.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:52.254+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.260383+0000) 2019-10-13T11:08:52.261 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:52.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:52.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:52.262+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.266986+0000) 2019-10-13T11:08:53.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:53.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:53.132882+0000) 2019-10-13T11:08:56.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:56.379+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:56.381699+0000) 2019-10-13T11:08:56.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:56.938+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:56.938533+0000) 2019-10-13T11:08:57.265 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:57.259+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.264212+0000) 2019-10-13T11:08:57.266 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:57.259+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.264796+0000) 2019-10-13T11:08:57.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:57.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.265382+0000) 2019-10-13T11:08:57.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:08:57.259+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.265725+0000) 2019-10-13T11:08:57.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:08:57.263+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.267397+0000) 2019-10-13T11:08:57.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:08:57.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.267818+0000) 2019-10-13T11:08:57.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:57.263+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.268178+0000) 2019-10-13T11:08:57.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:08:57.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:08:57.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:08:57.266+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.272783+0000) 2019-10-13T11:08:58.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:08:58.643+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.646313+0000) 2019-10-13T11:08:58.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:08:58.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.651328+0000) 2019-10-13T11:08:58.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:08:58.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.663202+0000) 2019-10-13T11:08:58.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:08:58.691+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.696475+0000) 2019-10-13T11:09:02.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:02.131+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.135929+0000) 2019-10-13T11:09:02.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:02.267+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.273129+0000) 2019-10-13T11:09:02.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:02.267+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.273429+0000) 2019-10-13T11:09:02.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:02.270+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.273847+0000) 2019-10-13T11:09:02.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:02.267+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.274730+0000) 2019-10-13T11:09:02.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:02.267+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.274890+0000) 2019-10-13T11:09:02.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:02.271+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.275720+0000) 2019-10-13T11:09:02.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:02.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.276554+0000) 2019-10-13T11:09:02.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:02.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:02.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:02.282+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.286214+0000) 2019-10-13T11:09:03.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:03.126+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:03.133050+0000) 2019-10-13T11:09:06.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:06.379+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:06.381936+0000) 2019-10-13T11:09:06.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:06.942+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:06.944923+0000) 2019-10-13T11:09:07.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:07.275+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.281015+0000) 2019-10-13T11:09:07.283 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:07.275+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.282463+0000) 2019-10-13T11:09:07.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:07.275+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.283030+0000) 2019-10-13T11:09:07.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:07.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.283703+0000) 2019-10-13T11:09:07.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:07.279+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.284034+0000) 2019-10-13T11:09:07.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:07.279+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.284276+0000) 2019-10-13T11:09:07.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:07.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.285515+0000) 2019-10-13T11:09:07.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:07.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:07.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:07.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.291321+0000) 2019-10-13T11:09:08.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:08.643+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.646522+0000) 2019-10-13T11:09:08.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:08.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.651560+0000) 2019-10-13T11:09:08.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:08.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.663415+0000) 2019-10-13T11:09:08.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:08.691+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.696714+0000) 2019-10-13T11:09:08.815 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:09:08.815 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:08.819 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:09:08.819 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:12.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:12.135+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.136146+0000) 2019-10-13T11:09:12.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:12.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.290551+0000) 2019-10-13T11:09:12.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:12.287+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.291375+0000) 2019-10-13T11:09:12.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:12.287+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.291819+0000) 2019-10-13T11:09:12.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:12.287+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.292264+0000) 2019-10-13T11:09:12.294 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:12.287+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.292401+0000) 2019-10-13T11:09:12.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:12.287+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.292730+0000) 2019-10-13T11:09:12.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:12.287+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.294729+0000) 2019-10-13T11:09:12.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:12.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:12.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:12.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.300072+0000) 2019-10-13T11:09:13.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:13.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:13.133225+0000) 2019-10-13T11:09:16.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:16.379+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:16.382136+0000) 2019-10-13T11:09:16.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:16.949+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:16.951400+0000) 2019-10-13T11:09:17.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:17.297+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.299650+0000) 2019-10-13T11:09:17.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:17.295+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.300250+0000) 2019-10-13T11:09:17.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:17.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.301058+0000) 2019-10-13T11:09:17.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:17.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.301658+0000) 2019-10-13T11:09:17.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:17.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.301738+0000) 2019-10-13T11:09:17.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:17.295+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.302868+0000) 2019-10-13T11:09:17.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:17.299+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.303660+0000) 2019-10-13T11:09:17.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:17.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:17.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:17.305+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.308610+0000) 2019-10-13T11:09:18.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:18.643+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.646770+0000) 2019-10-13T11:09:18.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:18.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.651752+0000) 2019-10-13T11:09:18.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:18.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.663614+0000) 2019-10-13T11:09:18.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:18.695+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.696936+0000) 2019-10-13T11:09:22.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:22.135+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.136380+0000) 2019-10-13T11:09:22.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:22.307+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.309431+0000) 2019-10-13T11:09:22.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:22.307+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.310317+0000) 2019-10-13T11:09:22.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:22.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.310817+0000) 2019-10-13T11:09:22.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:22.309+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.311754+0000) 2019-10-13T11:09:22.313 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:22.311+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.312129+0000) 2019-10-13T11:09:22.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:22.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.312538+0000) 2019-10-13T11:09:22.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:22.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.313582+0000) 2019-10-13T11:09:22.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:22.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:22.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:22.313+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.316651+0000) 2019-10-13T11:09:23.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:23.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:23.133436+0000) 2019-10-13T11:09:26.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:26.379+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:26.382330+0000) 2019-10-13T11:09:26.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:26.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:26.957719+0000) 2019-10-13T11:09:27.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:27.315+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.318126+0000) 2019-10-13T11:09:27.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:27.319+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.319361+0000) 2019-10-13T11:09:27.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:27.319+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.320002+0000) 2019-10-13T11:09:27.321 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:27.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.320500+0000) 2019-10-13T11:09:27.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:27.319+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.320783+0000) 2019-10-13T11:09:27.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:27.317+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.321698+0000) 2019-10-13T11:09:27.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:27.319+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.322115+0000) 2019-10-13T11:09:27.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:27.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 116966 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:27.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:27.329+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.330292+0000) 2019-10-13T11:09:28.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:28.643+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.647003+0000) 2019-10-13T11:09:28.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:28.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.651957+0000) 2019-10-13T11:09:28.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:28.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.663810+0000) 2019-10-13T11:09:28.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:28.695+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.697151+0000) 2019-10-13T11:09:32.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:32.135+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.136580+0000) 2019-10-13T11:09:32.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:32.325+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.327049+0000) 2019-10-13T11:09:32.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:32.327+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.327823+0000) 2019-10-13T11:09:32.329 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:32.327+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.328088+0000) 2019-10-13T11:09:32.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:32.327+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.329307+0000) 2019-10-13T11:09:32.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:32.327+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.329538+0000) 2019-10-13T11:09:32.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:32.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.329844+0000) 2019-10-13T11:09:32.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:32.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.330608+0000) 2019-10-13T11:09:32.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:32.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:32.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:32.333+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.335024+0000) 2019-10-13T11:09:33.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:33.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:33.133604+0000) 2019-10-13T11:09:36.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:36.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:36.382540+0000) 2019-10-13T11:09:36.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:36.961+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:36.964284+0000) 2019-10-13T11:09:37.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:37.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.336538+0000) 2019-10-13T11:09:37.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:37.333+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.337310+0000) 2019-10-13T11:09:37.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:37.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.337045+0000) 2019-10-13T11:09:37.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:37.334+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.338218+0000) 2019-10-13T11:09:37.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:37.334+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.339021+0000) 2019-10-13T11:09:37.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:37.338+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.339720+0000) 2019-10-13T11:09:37.341 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:37.338+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.340158+0000) 2019-10-13T11:09:37.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:37.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:37.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:37.341+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.345155+0000) 2019-10-13T11:09:38.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:38.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.647205+0000) 2019-10-13T11:09:38.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:38.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.652152+0000) 2019-10-13T11:09:38.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:38.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.664012+0000) 2019-10-13T11:09:38.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:38.694+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.697383+0000) 2019-10-13T11:09:38.955 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:09:38.955 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:38.959 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:09:38.959 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:42.138 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:42.134+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.136813+0000) 2019-10-13T11:09:42.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:42.342+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.346306+0000) 2019-10-13T11:09:42.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:42.342+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.346745+0000) 2019-10-13T11:09:42.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:42.345+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.347107+0000) 2019-10-13T11:09:42.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:42.346+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.347807+0000) 2019-10-13T11:09:42.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:42.346+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.348587+0000) 2019-10-13T11:09:42.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:42.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.348272+0000) 2019-10-13T11:09:42.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:42.346+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.349177+0000) 2019-10-13T11:09:42.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:42.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:42.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:42.353+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:42.356521+0000) 2019-10-13T11:09:43.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:43.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.133769+0000) 2019-10-13T11:09:46.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:46.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:46.382757+0000) 2019-10-13T11:09:46.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:46.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:46.970784+0000) 2019-10-13T11:09:47.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:47.353+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.355079+0000) 2019-10-13T11:09:47.356 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:47.354+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.355452+0000) 2019-10-13T11:09:47.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:47.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.355856+0000) 2019-10-13T11:09:47.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:47.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.356375+0000) 2019-10-13T11:09:47.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:47.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.357440+0000) 2019-10-13T11:09:47.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:47.354+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.358009+0000) 2019-10-13T11:09:47.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:47.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.358498+0000) 2019-10-13T11:09:47.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:47.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:47.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:47.361+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.365126+0000) 2019-10-13T11:09:48.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:48.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.647441+0000) 2019-10-13T11:09:48.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:48.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.652384+0000) 2019-10-13T11:09:48.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:48.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.664209+0000) 2019-10-13T11:09:48.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:48.694+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.697584+0000) 2019-10-13T11:09:52.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:52.134+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.137055+0000) 2019-10-13T11:09:52.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:52.362+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.364401+0000) 2019-10-13T11:09:52.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:52.362+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.364446+0000) 2019-10-13T11:09:52.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:52.362+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.365538+0000) 2019-10-13T11:09:52.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:52.362+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.366339+0000) 2019-10-13T11:09:52.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:52.365+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.366608+0000) 2019-10-13T11:09:52.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:52.362+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.367007+0000) 2019-10-13T11:09:52.368 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:52.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.367799+0000) 2019-10-13T11:09:52.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:52.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:52.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:52.373+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:52.374174+0000) 2019-10-13T11:09:53.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:53.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.133934+0000) 2019-10-13T11:09:56.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:56.378+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:56.382969+0000) 2019-10-13T11:09:56.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:56.973+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:56.977030+0000) 2019-10-13T11:09:57.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:09:57.369+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.373894+0000) 2019-10-13T11:09:57.375 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:57.370+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.374147+0000) 2019-10-13T11:09:57.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:09:57.370+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.374559+0000) 2019-10-13T11:09:57.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:57.370+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.375077+0000) 2019-10-13T11:09:57.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:57.374+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.376186+0000) 2019-10-13T11:09:57.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:57.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.376543+0000) 2019-10-13T11:09:57.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:09:57.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.376973+0000) 2019-10-13T11:09:57.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:09:57.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:09:57.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:09:57.377+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.379455+0000) 2019-10-13T11:09:58.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:09:58.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.647646+0000) 2019-10-13T11:09:58.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:09:58.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.652579+0000) 2019-10-13T11:09:58.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:09:58.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.664409+0000) 2019-10-13T11:09:58.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:09:58.694+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.697784+0000) 2019-10-13T11:10:02.138 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:02.134+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.137274+0000) 2019-10-13T11:10:02.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:02.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.383124+0000) 2019-10-13T11:10:02.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:02.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.383713+0000) 2019-10-13T11:10:02.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:02.382+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.384114+0000) 2019-10-13T11:10:02.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:02.382+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.384837+0000) 2019-10-13T11:10:02.386 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:02.381+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.385213+0000) 2019-10-13T11:10:02.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:02.382+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.385572+0000) 2019-10-13T11:10:02.387 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:02.382+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.386500+0000) 2019-10-13T11:10:02.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:02.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:02.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:02.389+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:02.391971+0000) 2019-10-13T11:10:03.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:03.129+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.134099+0000) 2019-10-13T11:10:06.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:06.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:06.383220+0000) 2019-10-13T11:10:06.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:06.985+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:06.988715+0000) 2019-10-13T11:10:07.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:07.390+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.391754+0000) 2019-10-13T11:10:07.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:07.390+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.392441+0000) 2019-10-13T11:10:07.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:07.390+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.392957+0000) 2019-10-13T11:10:07.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:07.390+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.393207+0000) 2019-10-13T11:10:07.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:07.389+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.393804+0000) 2019-10-13T11:10:07.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:07.390+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.394513+0000) 2019-10-13T11:10:07.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:07.390+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.394762+0000) 2019-10-13T11:10:07.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:07.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:07.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:07.397+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.400237+0000) 2019-10-13T11:10:08.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:08.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.647842+0000) 2019-10-13T11:10:08.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:08.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.652773+0000) 2019-10-13T11:10:08.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:08.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.664632+0000) 2019-10-13T11:10:08.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:08.694+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.698012+0000) 2019-10-13T11:10:09.087 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:10:09.087 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:09.091 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:10:09.091 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:12.138 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:12.134+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.137472+0000) 2019-10-13T11:10:12.402 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:12.398+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.401445+0000) 2019-10-13T11:10:12.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:12.398+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.402103+0000) 2019-10-13T11:10:12.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:12.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.402768+0000) 2019-10-13T11:10:12.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:12.400+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.403066+0000) 2019-10-13T11:10:12.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:12.402+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.403498+0000) 2019-10-13T11:10:12.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:12.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.404033+0000) 2019-10-13T11:10:12.405 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:12.402+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.404462+0000) 2019-10-13T11:10:12.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:12.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:12.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:12.404+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:12.407481+0000) 2019-10-13T11:10:13.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:13.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.134318+0000) 2019-10-13T11:10:16.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:16.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:16.383410+0000) 2019-10-13T11:10:16.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:16.992+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:16.995169+0000) 2019-10-13T11:10:17.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:17.404+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.409896+0000) 2019-10-13T11:10:17.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:17.405+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.411108+0000) 2019-10-13T11:10:17.412 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:17.409+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.411449+0000) 2019-10-13T11:10:17.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:17.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.411977+0000) 2019-10-13T11:10:17.413 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:17.409+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.412284+0000) 2019-10-13T11:10:17.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:17.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.412738+0000) 2019-10-13T11:10:17.414 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:17.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.413590+0000) 2019-10-13T11:10:17.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:17.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:17.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:17.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.416959+0000) 2019-10-13T11:10:18.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:18.645+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.648044+0000) 2019-10-13T11:10:18.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:18.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.652973+0000) 2019-10-13T11:10:18.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:18.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.664858+0000) 2019-10-13T11:10:18.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:18.693+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.698207+0000) 2019-10-13T11:10:22.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:22.133+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.137667+0000) 2019-10-13T11:10:22.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:22.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.419731+0000) 2019-10-13T11:10:22.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:22.417+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.420295+0000) 2019-10-13T11:10:22.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:22.416+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.421780+0000) 2019-10-13T11:10:22.429 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:22.417+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.421326+0000) 2019-10-13T11:10:22.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:22.417+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.422054+0000) 2019-10-13T11:10:22.429 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:22.417+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.422326+0000) 2019-10-13T11:10:22.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:22.417+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.422781+0000) 2019-10-13T11:10:22.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:22.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:22.432 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:22.424+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:22.427243+0000) 2019-10-13T11:10:23.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:23.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.134522+0000) 2019-10-13T11:10:26.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:26.381+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:26.383605+0000) 2019-10-13T11:10:27.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:26.996+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.001890+0000) 2019-10-13T11:10:27.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:27.425+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.429781+0000) 2019-10-13T11:10:27.431 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:27.425+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.430357+0000) 2019-10-13T11:10:27.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:27.425+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.430649+0000) 2019-10-13T11:10:27.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:27.425+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.430998+0000) 2019-10-13T11:10:27.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:27.428+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.432702+0000) 2019-10-13T11:10:27.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:27.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.433004+0000) 2019-10-13T11:10:27.434 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:27.429+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.433442+0000) 2019-10-13T11:10:27.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:27.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:27.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:27.432+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.436664+0000) 2019-10-13T11:10:28.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:28.645+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.648301+0000) 2019-10-13T11:10:28.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:28.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.653212+0000) 2019-10-13T11:10:28.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:28.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.665052+0000) 2019-10-13T11:10:28.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:28.693+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.698403+0000) 2019-10-13T11:10:32.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:32.133+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.137860+0000) 2019-10-13T11:10:32.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:32.437+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.439298+0000) 2019-10-13T11:10:32.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:32.437+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.440244+0000) 2019-10-13T11:10:32.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:32.437+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.440436+0000) 2019-10-13T11:10:32.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:32.437+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.440850+0000) 2019-10-13T11:10:32.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:32.437+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.441494+0000) 2019-10-13T11:10:32.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:32.436+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.441917+0000) 2019-10-13T11:10:32.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:32.437+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.442144+0000) 2019-10-13T11:10:32.445 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:32.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:32.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:32.444+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:32.449819+0000) 2019-10-13T11:10:33.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:33.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.134712+0000) 2019-10-13T11:10:36.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:36.381+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:36.383848+0000) 2019-10-13T11:10:37.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:37.008+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.011238+0000) 2019-10-13T11:10:37.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:37.441+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.447133+0000) 2019-10-13T11:10:37.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:37.444+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.447935+0000) 2019-10-13T11:10:37.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:37.445+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.448366+0000) 2019-10-13T11:10:37.450 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:37.445+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.449307+0000) 2019-10-13T11:10:37.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:37.445+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.450535+0000) 2019-10-13T11:10:37.451 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:37.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.450330+0000) 2019-10-13T11:10:37.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:37.449+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.451343+0000) 2019-10-13T11:10:37.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:37.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:37.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:37.452+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.456802+0000) 2019-10-13T11:10:38.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:38.645+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.648506+0000) 2019-10-13T11:10:38.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:38.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.653455+0000) 2019-10-13T11:10:38.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:38.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.665253+0000) 2019-10-13T11:10:38.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:38.693+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.698600+0000) 2019-10-13T11:10:39.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:10:39.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:39.203 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:10:39.203 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:42.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:42.133+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.138073+0000) 2019-10-13T11:10:42.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:42.452+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.457173+0000) 2019-10-13T11:10:42.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:42.453+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.457582+0000) 2019-10-13T11:10:42.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:42.453+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.457990+0000) 2019-10-13T11:10:42.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:42.453+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.458384+0000) 2019-10-13T11:10:42.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:42.453+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.458990+0000) 2019-10-13T11:10:42.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:42.457+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.459940+0000) 2019-10-13T11:10:42.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:42.457+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.460842+0000) 2019-10-13T11:10:42.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:42.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:42.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:42.460+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:42.466022+0000) 2019-10-13T11:10:43.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:43.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.134913+0000) 2019-10-13T11:10:46.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:46.381+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:46.384058+0000) 2019-10-13T11:10:47.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:47.016+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.017655+0000) 2019-10-13T11:10:47.467 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:47.461+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.466508+0000) 2019-10-13T11:10:47.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:47.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.466957+0000) 2019-10-13T11:10:47.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:47.465+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.467407+0000) 2019-10-13T11:10:47.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:47.465+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.467659+0000) 2019-10-13T11:10:47.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:47.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.467906+0000) 2019-10-13T11:10:47.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:47.465+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.468306+0000) 2019-10-13T11:10:47.470 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:47.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.469438+0000) 2019-10-13T11:10:47.472 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:47.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:47.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:47.472+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.474886+0000) 2019-10-13T11:10:48.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:48.645+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.648709+0000) 2019-10-13T11:10:48.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:48.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.653656+0000) 2019-10-13T11:10:48.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:48.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.665452+0000) 2019-10-13T11:10:48.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:48.693+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.698806+0000) 2019-10-13T11:10:52.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:52.133+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.138290+0000) 2019-10-13T11:10:52.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:52.469+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.475083+0000) 2019-10-13T11:10:52.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:52.473+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.475887+0000) 2019-10-13T11:10:52.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:52.472+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.476638+0000) 2019-10-13T11:10:52.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:52.473+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.476906+0000) 2019-10-13T11:10:52.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:52.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.477334+0000) 2019-10-13T11:10:52.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:52.473+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.478277+0000) 2019-10-13T11:10:52.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:52.473+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.478447+0000) 2019-10-13T11:10:52.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:52.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:52.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:52.480+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:52.485546+0000) 2019-10-13T11:10:53.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:53.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.135086+0000) 2019-10-13T11:10:56.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:56.381+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:56.384252+0000) 2019-10-13T11:10:57.030 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:57.024+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.028972+0000) 2019-10-13T11:10:57.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:10:57.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.484183+0000) 2019-10-13T11:10:57.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:57.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.485620+0000) 2019-10-13T11:10:57.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:57.481+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.485888+0000) 2019-10-13T11:10:57.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:10:57.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.486190+0000) 2019-10-13T11:10:57.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:10:57.484+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.487689+0000) 2019-10-13T11:10:57.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:57.481+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.486883+0000) 2019-10-13T11:10:57.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:57.481+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.487242+0000) 2019-10-13T11:10:57.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:10:57.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:10:57.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:10:57.492+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.494439+0000) 2019-10-13T11:10:58.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:10:58.645+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.648908+0000) 2019-10-13T11:10:58.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:10:58.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.653850+0000) 2019-10-13T11:10:58.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:10:58.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.665645+0000) 2019-10-13T11:10:58.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:10:58.693+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.698999+0000) 2019-10-13T11:11:02.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:02.136+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.138493+0000) 2019-10-13T11:11:02.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:02.488+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.493205+0000) 2019-10-13T11:11:02.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:02.488+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.493500+0000) 2019-10-13T11:11:02.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:02.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.494195+0000) 2019-10-13T11:11:02.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:02.488+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.495256+0000) 2019-10-13T11:11:02.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:02.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.496223+0000) 2019-10-13T11:11:02.497 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:02.492+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.495823+0000) 2019-10-13T11:11:02.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:02.492+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.496685+0000) 2019-10-13T11:11:02.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:02.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:02.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:02.504+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:02.506574+0000) 2019-10-13T11:11:03.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:03.132+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.135264+0000) 2019-10-13T11:11:06.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:06.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:06.384480+0000) 2019-10-13T11:11:07.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:07.036+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.040665+0000) 2019-10-13T11:11:07.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:07.496+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.502906+0000) 2019-10-13T11:11:07.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:07.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.503174+0000) 2019-10-13T11:11:07.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:07.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.504040+0000) 2019-10-13T11:11:07.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:07.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.504556+0000) 2019-10-13T11:11:07.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:07.500+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.506221+0000) 2019-10-13T11:11:07.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:07.500+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.506602+0000) 2019-10-13T11:11:07.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:07.503+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.507337+0000) 2019-10-13T11:11:07.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:07.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:07.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:07.507+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.510351+0000) 2019-10-13T11:11:08.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:08.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.649105+0000) 2019-10-13T11:11:08.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:08.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.654045+0000) 2019-10-13T11:11:08.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:08.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.665841+0000) 2019-10-13T11:11:08.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:08.692+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.699203+0000) 2019-10-13T11:11:09.335 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:11:09.335 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:09.339 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:11:09.339 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:12.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:12.136+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.138696+0000) 2019-10-13T11:11:12.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:12.508+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.512397+0000) 2019-10-13T11:11:12.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:12.508+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.512773+0000) 2019-10-13T11:11:12.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:12.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.513535+0000) 2019-10-13T11:11:12.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:12.507+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.513920+0000) 2019-10-13T11:11:12.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:12.508+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.514178+0000) 2019-10-13T11:11:12.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:12.508+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.514531+0000) 2019-10-13T11:11:12.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:12.508+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.514766+0000) 2019-10-13T11:11:12.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:12.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:12.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:12.515+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:12.519290+0000) 2019-10-13T11:11:13.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:13.131+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.135458+0000) 2019-10-13T11:11:16.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:16.380+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:16.384671+0000) 2019-10-13T11:11:17.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:17.043+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.047650+0000) 2019-10-13T11:11:17.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:17.516+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.521366+0000) 2019-10-13T11:11:17.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:17.516+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.521982+0000) 2019-10-13T11:11:17.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:17.516+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.521743+0000) 2019-10-13T11:11:17.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:17.516+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.522643+0000) 2019-10-13T11:11:17.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:17.519+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.522956+0000) 2019-10-13T11:11:17.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:17.520+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.523426+0000) 2019-10-13T11:11:17.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:17.520+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.523979+0000) 2019-10-13T11:11:17.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:17.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117714 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:17.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:17.523+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.528979+0000) 2019-10-13T11:11:18.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:18.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.649322+0000) 2019-10-13T11:11:18.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:18.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.654246+0000) 2019-10-13T11:11:18.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:18.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.666035+0000) 2019-10-13T11:11:18.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:18.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.699422+0000) 2019-10-13T11:11:22.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:22.136+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.138890+0000) 2019-10-13T11:11:22.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:22.524+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.530120+0000) 2019-10-13T11:11:22.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:22.524+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.530481+0000) 2019-10-13T11:11:22.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:22.527+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.531565+0000) 2019-10-13T11:11:22.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:22.528+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.531918+0000) 2019-10-13T11:11:22.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:22.528+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.532490+0000) 2019-10-13T11:11:22.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:22.528+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.533068+0000) 2019-10-13T11:11:22.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:22.528+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.533070+0000) 2019-10-13T11:11:22.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:22.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:22.541 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:22.535+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:22.540560+0000) 2019-10-13T11:11:23.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:23.131+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.135627+0000) 2019-10-13T11:11:26.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:26.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:26.384901+0000) 2019-10-13T11:11:27.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:27.051+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.054571+0000) 2019-10-13T11:11:27.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:27.532+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.539160+0000) 2019-10-13T11:11:27.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:27.535+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.540116+0000) 2019-10-13T11:11:27.541 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:27.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.540563+0000) 2019-10-13T11:11:27.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:27.536+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.541663+0000) 2019-10-13T11:11:27.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:27.536+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.542123+0000) 2019-10-13T11:11:27.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:27.536+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.542586+0000) 2019-10-13T11:11:27.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:27.536+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.543218+0000) 2019-10-13T11:11:27.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:27.540+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:27.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:27.543+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.548295+0000) 2019-10-13T11:11:28.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:28.644+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.649555+0000) 2019-10-13T11:11:28.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:28.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.654450+0000) 2019-10-13T11:11:28.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:28.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.666226+0000) 2019-10-13T11:11:28.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:28.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.699631+0000) 2019-10-13T11:11:32.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:32.132+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.139042+0000) 2019-10-13T11:11:32.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:32.544+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.548950+0000) 2019-10-13T11:11:32.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:32.544+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.549339+0000) 2019-10-13T11:11:32.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:32.544+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.550059+0000) 2019-10-13T11:11:32.551 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:32.544+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.550868+0000) 2019-10-13T11:11:32.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:32.547+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.551368+0000) 2019-10-13T11:11:32.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:32.548+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.552168+0000) 2019-10-13T11:11:32.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:32.548+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.552962+0000) 2019-10-13T11:11:32.554 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:32.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117816 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:32.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:32.551+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:32.556217+0000) 2019-10-13T11:11:33.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:33.131+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.135818+0000) 2019-10-13T11:11:36.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:36.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:36.385102+0000) 2019-10-13T11:11:37.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:37.059+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.061529+0000) 2019-10-13T11:11:37.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:37.552+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.557053+0000) 2019-10-13T11:11:37.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:37.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.557329+0000) 2019-10-13T11:11:37.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:37.552+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.559092+0000) 2019-10-13T11:11:37.561 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:37.556+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.559593+0000) 2019-10-13T11:11:37.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:37.556+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.559939+0000) 2019-10-13T11:11:37.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:37.556+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.560424+0000) 2019-10-13T11:11:37.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:37.555+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.560953+0000) 2019-10-13T11:11:37.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:37.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:37.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:37.563+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.567721+0000) 2019-10-13T11:11:38.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:38.648+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.649757+0000) 2019-10-13T11:11:38.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:38.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.654672+0000) 2019-10-13T11:11:38.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:38.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.666417+0000) 2019-10-13T11:11:38.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:38.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.699864+0000) 2019-10-13T11:11:39.449 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:11:39.450 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:39.453 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:11:39.453 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:42.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:42.132+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.139233+0000) 2019-10-13T11:11:42.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:42.560+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.566111+0000) 2019-10-13T11:11:42.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:42.560+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.567246+0000) 2019-10-13T11:11:42.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:42.564+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.567649+0000) 2019-10-13T11:11:42.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:42.563+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.567876+0000) 2019-10-13T11:11:42.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:42.564+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.568594+0000) 2019-10-13T11:11:42.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:42.564+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.569236+0000) 2019-10-13T11:11:42.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:42.564+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.570007+0000) 2019-10-13T11:11:42.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:42.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:42.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:42.571+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:42.574826+0000) 2019-10-13T11:11:43.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:43.135+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.135990+0000) 2019-10-13T11:11:46.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:46.379+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:46.385302+0000) 2019-10-13T11:11:47.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:47.067+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.068467+0000) 2019-10-13T11:11:47.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:47.571+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.575515+0000) 2019-10-13T11:11:47.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:47.571+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.576082+0000) 2019-10-13T11:11:47.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:47.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.576359+0000) 2019-10-13T11:11:47.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:47.571+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.576736+0000) 2019-10-13T11:11:47.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:47.571+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.577059+0000) 2019-10-13T11:11:47.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:47.571+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.578219+0000) 2019-10-13T11:11:47.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:47.571+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.578394+0000) 2019-10-13T11:11:47.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:47.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:47.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:47.575+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.581595+0000) 2019-10-13T11:11:48.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:48.647+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.650008+0000) 2019-10-13T11:11:48.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:48.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.654898+0000) 2019-10-13T11:11:48.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:48.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.666619+0000) 2019-10-13T11:11:48.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:48.695+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.700073+0000) 2019-10-13T11:11:52.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:52.135+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.139492+0000) 2019-10-13T11:11:52.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:52.579+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.584251+0000) 2019-10-13T11:11:52.586 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:52.579+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.584955+0000) 2019-10-13T11:11:52.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:52.579+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.586178+0000) 2019-10-13T11:11:52.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:52.583+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.586491+0000) 2019-10-13T11:11:52.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:52.579+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.586852+0000) 2019-10-13T11:11:52.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:52.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.587166+0000) 2019-10-13T11:11:52.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:52.583+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.587498+0000) 2019-10-13T11:11:52.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:52.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:52.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:52.587+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:52.592261+0000) 2019-10-13T11:11:53.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:53.135+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.136161+0000) 2019-10-13T11:11:56.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:56.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:56.385513+0000) 2019-10-13T11:11:57.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:57.071+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.075411+0000) 2019-10-13T11:11:57.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:11:57.587+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.593517+0000) 2019-10-13T11:11:57.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:11:57.587+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.593857+0000) 2019-10-13T11:11:57.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:57.591+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.595295+0000) 2019-10-13T11:11:57.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:57.591+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.595617+0000) 2019-10-13T11:11:57.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:57.591+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.596435+0000) 2019-10-13T11:11:57.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:11:57.591+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.596771+0000) 2019-10-13T11:11:57.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:57.591+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.597988+0000) 2019-10-13T11:11:57.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:11:57.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 117985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:11:57.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:11:57.599+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.603713+0000) 2019-10-13T11:11:58.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:11:58.647+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.650210+0000) 2019-10-13T11:11:58.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:11:58.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.655127+0000) 2019-10-13T11:11:58.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:11:58.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.666768+0000) 2019-10-13T11:11:58.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:11:58.695+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.700282+0000) 2019-10-13T11:12:02.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:02.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.139702+0000) 2019-10-13T11:12:02.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:02.599+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.603464+0000) 2019-10-13T11:12:02.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:02.599+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.605072+0000) 2019-10-13T11:12:02.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:02.599+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.605192+0000) 2019-10-13T11:12:02.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:02.599+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.605593+0000) 2019-10-13T11:12:02.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:02.599+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.606144+0000) 2019-10-13T11:12:02.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:02.599+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.606709+0000) 2019-10-13T11:12:02.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:02.599+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.607054+0000) 2019-10-13T11:12:02.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:02.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:02.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:02.607+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:02.612655+0000) 2019-10-13T11:12:03.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:03.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.136345+0000) 2019-10-13T11:12:06.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:06.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:06.385727+0000) 2019-10-13T11:12:07.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:07.082+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.086279+0000) 2019-10-13T11:12:07.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:07.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.613240+0000) 2019-10-13T11:12:07.615 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:07.607+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.614427+0000) 2019-10-13T11:12:07.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:07.607+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.614990+0000) 2019-10-13T11:12:07.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:07.607+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.614697+0000) 2019-10-13T11:12:07.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:07.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.615920+0000) 2019-10-13T11:12:07.617 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:07.610+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.616239+0000) 2019-10-13T11:12:07.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:07.611+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.616471+0000) 2019-10-13T11:12:07.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:07.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118052 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:07.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:07.614+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.620231+0000) 2019-10-13T11:12:08.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:08.647+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.650407+0000) 2019-10-13T11:12:08.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:08.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.655337+0000) 2019-10-13T11:12:08.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:08.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.666901+0000) 2019-10-13T11:12:08.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:08.695+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.700493+0000) 2019-10-13T11:12:09.757 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:12:09.757 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:09.761 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:12:09.761 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:12.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:12.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.139905+0000) 2019-10-13T11:12:12.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:12.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.622580+0000) 2019-10-13T11:12:12.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:12.615+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.622699+0000) 2019-10-13T11:12:12.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:12.615+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.623070+0000) 2019-10-13T11:12:12.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:12.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.623518+0000) 2019-10-13T11:12:12.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:12.619+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.624888+0000) 2019-10-13T11:12:12.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:12.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.625295+0000) 2019-10-13T11:12:12.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:12.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.625518+0000) 2019-10-13T11:12:12.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:12.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:12.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:12.626+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:12.632466+0000) 2019-10-13T11:12:13.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:13.134+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.136476+0000) 2019-10-13T11:12:16.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:16.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:16.385918+0000) 2019-10-13T11:12:17.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:17.086+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.092797+0000) 2019-10-13T11:12:17.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:17.631+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.631474+0000) 2019-10-13T11:12:17.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:17.631+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.632047+0000) 2019-10-13T11:12:17.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:17.631+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.632710+0000) 2019-10-13T11:12:17.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:17.631+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.633040+0000) 2019-10-13T11:12:17.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:17.631+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.633782+0000) 2019-10-13T11:12:17.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:17.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.634080+0000) 2019-10-13T11:12:17.636 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:17.630+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.635648+0000) 2019-10-13T11:12:17.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:17.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118121 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:17.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:17.638+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.642410+0000) 2019-10-13T11:12:18.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:18.647+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.650612+0000) 2019-10-13T11:12:18.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:18.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.655573+0000) 2019-10-13T11:12:18.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:18.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.667114+0000) 2019-10-13T11:12:18.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:18.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.700662+0000) 2019-10-13T11:12:22.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:22.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.140164+0000) 2019-10-13T11:12:22.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:22.639+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.641873+0000) 2019-10-13T11:12:22.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:22.639+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.641998+0000) 2019-10-13T11:12:22.643 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:22.639+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.642431+0000) 2019-10-13T11:12:22.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:22.639+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.642800+0000) 2019-10-13T11:12:22.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:22.643+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.643709+0000) 2019-10-13T11:12:22.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:22.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.644588+0000) 2019-10-13T11:12:22.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:22.638+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.645319+0000) 2019-10-13T11:12:22.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:22.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:22.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:22.646+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:22.653157+0000) 2019-10-13T11:12:23.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:23.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.136648+0000) 2019-10-13T11:12:26.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:26.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:26.386117+0000) 2019-10-13T11:12:27.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:27.098+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.099177+0000) 2019-10-13T11:12:27.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:27.646+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.650866+0000) 2019-10-13T11:12:27.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:27.650+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.651383+0000) 2019-10-13T11:12:27.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:27.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.652302+0000) 2019-10-13T11:12:27.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:27.650+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.653041+0000) 2019-10-13T11:12:27.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:27.650+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.653442+0000) 2019-10-13T11:12:27.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:27.650+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.653913+0000) 2019-10-13T11:12:27.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:27.650+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.654782+0000) 2019-10-13T11:12:27.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:27.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:27.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:27.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.659248+0000) 2019-10-13T11:12:28.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:28.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.650842+0000) 2019-10-13T11:12:28.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:28.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.655809+0000) 2019-10-13T11:12:28.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:28.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.667339+0000) 2019-10-13T11:12:28.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:28.698+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.700896+0000) 2019-10-13T11:12:32.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:32.138+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.140365+0000) 2019-10-13T11:12:32.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:32.658+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.660231+0000) 2019-10-13T11:12:32.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:32.658+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.660730+0000) 2019-10-13T11:12:32.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:32.658+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.661088+0000) 2019-10-13T11:12:32.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:32.658+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.661605+0000) 2019-10-13T11:12:32.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:32.654+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.661953+0000) 2019-10-13T11:12:32.663 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:32.658+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.662309+0000) 2019-10-13T11:12:32.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:32.658+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.662861+0000) 2019-10-13T11:12:32.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:32.662+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:32.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:32.662+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:32.668827+0000) 2019-10-13T11:12:33.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:33.130+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.136866+0000) 2019-10-13T11:12:36.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:36.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:36.386364+0000) 2019-10-13T11:12:37.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:37.102+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.105558+0000) 2019-10-13T11:12:37.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:37.666+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.669440+0000) 2019-10-13T11:12:37.671 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:37.666+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.670178+0000) 2019-10-13T11:12:37.671 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:37.666+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.669977+0000) 2019-10-13T11:12:37.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:37.666+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.670910+0000) 2019-10-13T11:12:37.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:37.666+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.671114+0000) 2019-10-13T11:12:37.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:37.670+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.671565+0000) 2019-10-13T11:12:37.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:37.666+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.672292+0000) 2019-10-13T11:12:37.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:37.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:37.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:37.674+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.678341+0000) 2019-10-13T11:12:38.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:38.650+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.651018+0000) 2019-10-13T11:12:38.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:38.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.656032+0000) 2019-10-13T11:12:38.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:38.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.667535+0000) 2019-10-13T11:12:38.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:38.698+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.701103+0000) 2019-10-13T11:12:39.871 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:12:39.871 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:39.875 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:12:39.875 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:42.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:42.138+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.140595+0000) 2019-10-13T11:12:42.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:42.674+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.676761+0000) 2019-10-13T11:12:42.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:42.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.678309+0000) 2019-10-13T11:12:42.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:42.674+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.678585+0000) 2019-10-13T11:12:42.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:42.674+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.678952+0000) 2019-10-13T11:12:42.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:42.678+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.679678+0000) 2019-10-13T11:12:42.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:42.678+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.680026+0000) 2019-10-13T11:12:42.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:42.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.680795+0000) 2019-10-13T11:12:42.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:42.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:42.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:42.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:42.684131+0000) 2019-10-13T11:12:43.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:43.134+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.137033+0000) 2019-10-13T11:12:46.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:46.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:46.386601+0000) 2019-10-13T11:12:47.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:47.110+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.112114+0000) 2019-10-13T11:12:47.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:47.682+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.686212+0000) 2019-10-13T11:12:47.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:47.682+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.686761+0000) 2019-10-13T11:12:47.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:47.686+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.687931+0000) 2019-10-13T11:12:47.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:47.682+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.688294+0000) 2019-10-13T11:12:47.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:47.686+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.688854+0000) 2019-10-13T11:12:47.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:47.686+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.688881+0000) 2019-10-13T11:12:47.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:47.686+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.689212+0000) 2019-10-13T11:12:47.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:47.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:47.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:47.690+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.694151+0000) 2019-10-13T11:12:48.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:48.646+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.651186+0000) 2019-10-13T11:12:48.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:48.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.656239+0000) 2019-10-13T11:12:48.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:48.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.667711+0000) 2019-10-13T11:12:48.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:48.698+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.701325+0000) 2019-10-13T11:12:52.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:52.138+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.140814+0000) 2019-10-13T11:12:52.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:52.694+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.695789+0000) 2019-10-13T11:12:52.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:52.694+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.696271+0000) 2019-10-13T11:12:52.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:52.694+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.696989+0000) 2019-10-13T11:12:52.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:52.694+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.697437+0000) 2019-10-13T11:12:52.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:52.694+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.697792+0000) 2019-10-13T11:12:52.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:52.694+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.698835+0000) 2019-10-13T11:12:52.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:52.698+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.699196+0000) 2019-10-13T11:12:52.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:52.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:52.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:52.702+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:52.705803+0000) 2019-10-13T11:12:53.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:53.134+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.137235+0000) 2019-10-13T11:12:56.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:56.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:56.386814+0000) 2019-10-13T11:12:57.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:57.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.122967+0000) 2019-10-13T11:12:57.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:57.702+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.705073+0000) 2019-10-13T11:12:57.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:12:57.702+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.705703+0000) 2019-10-13T11:12:57.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:57.702+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.706947+0000) 2019-10-13T11:12:57.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:57.702+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.706983+0000) 2019-10-13T11:12:57.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:12:57.706+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.707465+0000) 2019-10-13T11:12:57.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:57.706+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.707816+0000) 2019-10-13T11:12:57.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:12:57.706+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.708644+0000) 2019-10-13T11:12:57.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:12:57.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:12:57.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:12:57.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.712612+0000) 2019-10-13T11:12:58.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:12:58.650+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.651338+0000) 2019-10-13T11:12:58.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:12:58.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.656403+0000) 2019-10-13T11:12:58.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:12:58.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.667907+0000) 2019-10-13T11:12:58.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:12:58.698+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.701524+0000) 2019-10-13T11:13:02.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:02.138+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.141014+0000) 2019-10-13T11:13:02.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:02.713+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.714263+0000) 2019-10-13T11:13:02.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:02.710+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.714698+0000) 2019-10-13T11:13:02.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:02.714+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.715543+0000) 2019-10-13T11:13:02.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:02.714+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.715939+0000) 2019-10-13T11:13:02.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:02.714+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.716458+0000) 2019-10-13T11:13:02.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:02.714+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.716830+0000) 2019-10-13T11:13:02.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:02.714+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.717974+0000) 2019-10-13T11:13:02.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:02.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:02.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:02.717+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:02.721625+0000) 2019-10-13T11:13:03.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:03.133+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.137441+0000) 2019-10-13T11:13:06.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:06.382+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:06.387016+0000) 2019-10-13T11:13:07.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:07.125+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.129457+0000) 2019-10-13T11:13:07.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:07.718+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.723132+0000) 2019-10-13T11:13:07.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:07.722+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.723697+0000) 2019-10-13T11:13:07.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:07.722+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.724369+0000) 2019-10-13T11:13:07.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:07.722+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.725059+0000) 2019-10-13T11:13:07.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:07.721+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.725613+0000) 2019-10-13T11:13:07.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:07.722+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.726585+0000) 2019-10-13T11:13:07.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:07.722+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.727147+0000) 2019-10-13T11:13:07.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:07.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:07.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:07.729+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.730674+0000) 2019-10-13T11:13:08.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:08.650+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.651517+0000) 2019-10-13T11:13:08.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:08.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.656596+0000) 2019-10-13T11:13:08.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:08.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.668081+0000) 2019-10-13T11:13:08.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:08.698+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.701705+0000) 2019-10-13T11:13:09.979 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:13:09.980 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:09.984 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:13:09.984 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:12.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:12.137+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.141216+0000) 2019-10-13T11:13:12.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:12.729+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.732114+0000) 2019-10-13T11:13:12.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:12.729+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.732557+0000) 2019-10-13T11:13:12.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:12.729+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.733153+0000) 2019-10-13T11:13:12.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:12.729+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.733550+0000) 2019-10-13T11:13:12.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:12.729+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.734249+0000) 2019-10-13T11:13:12.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:12.729+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.735111+0000) 2019-10-13T11:13:12.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:12.733+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.736009+0000) 2019-10-13T11:13:12.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:12.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:12.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:12.737+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:12.740233+0000) 2019-10-13T11:13:13.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:13.133+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.137623+0000) 2019-10-13T11:13:16.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:16.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:16.387190+0000) 2019-10-13T11:13:17.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:17.133+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.135863+0000) 2019-10-13T11:13:17.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:17.737+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.741734+0000) 2019-10-13T11:13:17.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:17.737+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.742108+0000) 2019-10-13T11:13:17.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:17.737+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.743050+0000) 2019-10-13T11:13:17.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:17.741+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.743714+0000) 2019-10-13T11:13:17.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:17.741+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.743899+0000) 2019-10-13T11:13:17.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:17.741+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.744717+0000) 2019-10-13T11:13:17.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:17.741+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.744872+0000) 2019-10-13T11:13:17.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:17.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:17.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:17.745+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.748652+0000) 2019-10-13T11:13:18.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:18.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.651727+0000) 2019-10-13T11:13:18.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:18.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.656806+0000) 2019-10-13T11:13:18.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:18.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.668260+0000) 2019-10-13T11:13:18.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:18.697+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.701883+0000) 2019-10-13T11:13:22.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:22.137+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.141415+0000) 2019-10-13T11:13:22.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:22.745+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.751192+0000) 2019-10-13T11:13:22.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:22.749+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.751889+0000) 2019-10-13T11:13:22.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:22.749+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.752137+0000) 2019-10-13T11:13:22.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:22.749+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.752634+0000) 2019-10-13T11:13:22.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:22.749+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.753156+0000) 2019-10-13T11:13:22.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:22.749+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.753767+0000) 2019-10-13T11:13:22.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:22.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.754083+0000) 2019-10-13T11:13:22.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:22.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:22.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:22.753+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:22.757381+0000) 2019-10-13T11:13:23.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:23.137+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.137795+0000) 2019-10-13T11:13:26.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:26.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:26.387382+0000) 2019-10-13T11:13:27.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:27.141+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.142459+0000) 2019-10-13T11:13:27.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:27.757+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.760570+0000) 2019-10-13T11:13:27.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:27.757+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.761307+0000) 2019-10-13T11:13:27.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:27.757+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.760858+0000) 2019-10-13T11:13:27.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:27.757+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.761490+0000) 2019-10-13T11:13:27.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:27.757+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.761727+0000) 2019-10-13T11:13:27.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:27.761+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.763298+0000) 2019-10-13T11:13:27.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:27.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.764299+0000) 2019-10-13T11:13:27.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:27.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118595 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:27.775 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:27.769+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.774067+0000) 2019-10-13T11:13:28.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:28.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.651900+0000) 2019-10-13T11:13:28.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:28.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.656973+0000) 2019-10-13T11:13:28.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:28.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.668435+0000) 2019-10-13T11:13:28.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:28.697+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.702091+0000) 2019-10-13T11:13:32.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:32.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.141611+0000) 2019-10-13T11:13:32.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:32.765+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.768853+0000) 2019-10-13T11:13:32.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:32.765+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.769655+0000) 2019-10-13T11:13:32.771 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:32.769+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.770384+0000) 2019-10-13T11:13:32.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:32.765+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.770697+0000) 2019-10-13T11:13:32.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:32.769+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.771380+0000) 2019-10-13T11:13:32.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:32.769+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.772665+0000) 2019-10-13T11:13:32.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:32.769+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.772977+0000) 2019-10-13T11:13:32.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:32.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:32.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:32.777+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:32.780657+0000) 2019-10-13T11:13:33.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:33.133+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.137968+0000) 2019-10-13T11:13:36.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:36.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:36.387610+0000) 2019-10-13T11:13:37.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:37.145+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.148900+0000) 2019-10-13T11:13:37.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:37.773+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.778437+0000) 2019-10-13T11:13:37.780 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:37.777+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.778758+0000) 2019-10-13T11:13:37.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:37.773+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.779082+0000) 2019-10-13T11:13:37.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:37.777+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.779877+0000) 2019-10-13T11:13:37.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:37.777+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.780283+0000) 2019-10-13T11:13:37.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:37.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.780979+0000) 2019-10-13T11:13:37.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:37.777+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.781589+0000) 2019-10-13T11:13:37.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:37.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118664 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:37.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:37.785+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.786917+0000) 2019-10-13T11:13:38.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:38.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.652049+0000) 2019-10-13T11:13:38.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:38.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.657163+0000) 2019-10-13T11:13:38.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:38.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.668615+0000) 2019-10-13T11:13:38.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:38.697+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.702263+0000) 2019-10-13T11:13:40.275 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:13:40.275 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:40.279 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:13:40.279 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:42.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:42.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.141811+0000) 2019-10-13T11:13:42.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:42.785+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.787749+0000) 2019-10-13T11:13:42.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:42.785+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.788658+0000) 2019-10-13T11:13:42.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:42.785+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.789327+0000) 2019-10-13T11:13:42.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:42.785+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.789645+0000) 2019-10-13T11:13:42.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:42.785+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.789989+0000) 2019-10-13T11:13:42.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:42.785+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.790184+0000) 2019-10-13T11:13:42.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:42.785+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.790671+0000) 2019-10-13T11:13:42.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:42.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:42.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:42.793+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:42.794828+0000) 2019-10-13T11:13:43.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:43.137+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.138133+0000) 2019-10-13T11:13:46.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:46.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:46.387858+0000) 2019-10-13T11:13:47.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:47.153+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.155713+0000) 2019-10-13T11:13:47.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:47.793+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.797937+0000) 2019-10-13T11:13:47.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:47.793+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.798399+0000) 2019-10-13T11:13:47.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:47.797+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.798850+0000) 2019-10-13T11:13:47.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:47.793+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.799149+0000) 2019-10-13T11:13:47.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:47.797+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.800254+0000) 2019-10-13T11:13:47.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:47.797+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.800686+0000) 2019-10-13T11:13:47.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:47.797+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.801036+0000) 2019-10-13T11:13:47.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:47.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:47.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:47.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.808819+0000) 2019-10-13T11:13:48.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:48.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.652255+0000) 2019-10-13T11:13:48.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:48.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.657350+0000) 2019-10-13T11:13:48.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:48.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.668793+0000) 2019-10-13T11:13:48.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:48.697+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.702440+0000) 2019-10-13T11:13:52.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:52.137+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.142034+0000) 2019-10-13T11:13:52.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:52.801+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.805896+0000) 2019-10-13T11:13:52.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:52.805+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.806498+0000) 2019-10-13T11:13:52.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:52.801+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.806735+0000) 2019-10-13T11:13:52.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:52.805+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.807579+0000) 2019-10-13T11:13:52.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:52.805+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.808086+0000) 2019-10-13T11:13:52.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:52.805+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.808801+0000) 2019-10-13T11:13:52.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:52.805+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.808949+0000) 2019-10-13T11:13:52.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:52.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:52.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:52.813+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:52.816881+0000) 2019-10-13T11:13:53.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:53.137+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.138348+0000) 2019-10-13T11:13:56.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:56.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:56.388072+0000) 2019-10-13T11:13:57.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:57.160+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.162190+0000) 2019-10-13T11:13:57.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:13:57.812+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.816032+0000) 2019-10-13T11:13:57.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:13:57.812+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.816693+0000) 2019-10-13T11:13:57.818 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:13:57.812+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.817000+0000) 2019-10-13T11:13:57.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:57.812+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.817347+0000) 2019-10-13T11:13:57.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:57.812+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.817665+0000) 2019-10-13T11:13:57.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:57.812+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.817961+0000) 2019-10-13T11:13:57.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:57.812+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.818666+0000) 2019-10-13T11:13:57.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:13:57.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:13:57.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:13:57.820+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.825486+0000) 2019-10-13T11:13:58.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:13:58.648+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.652433+0000) 2019-10-13T11:13:58.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:13:58.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.657562+0000) 2019-10-13T11:13:58.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:13:58.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.668969+0000) 2019-10-13T11:13:58.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:13:58.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.702618+0000) 2019-10-13T11:14:02.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:02.136+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.142236+0000) 2019-10-13T11:14:02.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:02.820+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.825418+0000) 2019-10-13T11:14:02.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:02.820+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.825958+0000) 2019-10-13T11:14:02.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:02.820+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.826765+0000) 2019-10-13T11:14:02.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:02.820+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.826816+0000) 2019-10-13T11:14:02.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:02.824+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.827520+0000) 2019-10-13T11:14:02.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:02.824+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.828468+0000) 2019-10-13T11:14:02.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:02.824+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.829284+0000) 2019-10-13T11:14:02.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:02.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:02.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:02.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:02.834686+0000) 2019-10-13T11:14:03.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:03.136+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.138518+0000) 2019-10-13T11:14:06.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:06.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:06.388294+0000) 2019-10-13T11:14:07.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:07.172+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.173409+0000) 2019-10-13T11:14:07.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:07.828+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.833906+0000) 2019-10-13T11:14:07.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:07.828+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.834607+0000) 2019-10-13T11:14:07.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:07.828+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.834977+0000) 2019-10-13T11:14:07.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:07.832+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.835766+0000) 2019-10-13T11:14:07.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:07.832+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.836441+0000) 2019-10-13T11:14:07.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:07.832+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.836802+0000) 2019-10-13T11:14:07.838 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:07.832+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.837715+0000) 2019-10-13T11:14:07.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:07.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118867 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:07.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:07.840+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.844703+0000) 2019-10-13T11:14:08.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:08.648+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.652613+0000) 2019-10-13T11:14:08.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:08.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.657771+0000) 2019-10-13T11:14:08.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:08.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.669170+0000) 2019-10-13T11:14:08.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:08.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.702799+0000) 2019-10-13T11:14:10.402 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:14:10.403 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:10.406 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:14:10.406 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:12.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:12.136+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.142466+0000) 2019-10-13T11:14:12.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:12.840+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.844039+0000) 2019-10-13T11:14:12.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:12.840+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.844737+0000) 2019-10-13T11:14:12.846 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:12.840+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.845137+0000) 2019-10-13T11:14:12.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:12.840+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.845651+0000) 2019-10-13T11:14:12.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:12.840+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.846013+0000) 2019-10-13T11:14:12.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:12.840+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.846441+0000) 2019-10-13T11:14:12.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:12.840+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.846950+0000) 2019-10-13T11:14:12.851 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:12.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:12.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:12.848+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:12.853648+0000) 2019-10-13T11:14:13.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:13.136+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.138717+0000) 2019-10-13T11:14:16.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:16.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:16.388494+0000) 2019-10-13T11:14:17.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:17.184+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.184845+0000) 2019-10-13T11:14:17.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:17.848+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.853132+0000) 2019-10-13T11:14:17.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:17.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.854003+0000) 2019-10-13T11:14:17.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:17.852+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.855213+0000) 2019-10-13T11:14:17.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:17.848+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.854984+0000) 2019-10-13T11:14:17.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:17.852+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.855654+0000) 2019-10-13T11:14:17.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:17.852+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.855386+0000) 2019-10-13T11:14:17.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:17.852+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.857012+0000) 2019-10-13T11:14:17.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:17.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:17.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:17.856+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.861402+0000) 2019-10-13T11:14:18.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:18.648+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.652787+0000) 2019-10-13T11:14:18.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:18.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.658002+0000) 2019-10-13T11:14:18.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:18.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.669343+0000) 2019-10-13T11:14:18.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:18.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.702978+0000) 2019-10-13T11:14:22.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:22.140+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.142668+0000) 2019-10-13T11:14:22.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:22.856+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.861974+0000) 2019-10-13T11:14:22.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:22.860+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.863696+0000) 2019-10-13T11:14:22.865 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:22.860+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.863901+0000) 2019-10-13T11:14:22.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:22.860+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.864739+0000) 2019-10-13T11:14:22.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:22.860+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.865418+0000) 2019-10-13T11:14:22.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:22.860+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.865626+0000) 2019-10-13T11:14:22.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:22.860+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.866070+0000) 2019-10-13T11:14:22.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:22.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 118969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:22.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:22.864+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.869213+0000) 2019-10-13T11:14:23.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:23.136+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.138877+0000) 2019-10-13T11:14:26.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:26.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:26.388707+0000) 2019-10-13T11:14:27.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:27.192+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.195991+0000) 2019-10-13T11:14:27.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:27.868+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.871625+0000) 2019-10-13T11:14:27.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:27.868+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.872184+0000) 2019-10-13T11:14:27.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:27.868+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.873141+0000) 2019-10-13T11:14:27.874 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:27.868+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.872840+0000) 2019-10-13T11:14:27.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:27.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.873606+0000) 2019-10-13T11:14:27.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:27.868+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.873796+0000) 2019-10-13T11:14:27.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:27.868+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.874814+0000) 2019-10-13T11:14:27.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:27.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:27.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:27.876+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.880887+0000) 2019-10-13T11:14:28.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:28.652+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.652961+0000) 2019-10-13T11:14:28.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:28.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.658183+0000) 2019-10-13T11:14:28.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:28.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.669518+0000) 2019-10-13T11:14:28.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:28.696+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.703159+0000) 2019-10-13T11:14:32.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:32.140+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.142900+0000) 2019-10-13T11:14:32.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:32.876+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.881835+0000) 2019-10-13T11:14:32.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:32.876+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.882078+0000) 2019-10-13T11:14:32.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:32.876+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.882719+0000) 2019-10-13T11:14:32.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:32.876+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.883044+0000) 2019-10-13T11:14:32.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:32.880+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.883805+0000) 2019-10-13T11:14:32.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:32.880+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.883635+0000) 2019-10-13T11:14:32.885 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:32.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.884542+0000) 2019-10-13T11:14:32.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:32.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:32.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:32.888+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.892698+0000) 2019-10-13T11:14:33.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:33.136+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.139046+0000) 2019-10-13T11:14:36.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:36.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:36.388936+0000) 2019-10-13T11:14:37.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:37.204+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.206503+0000) 2019-10-13T11:14:37.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:37.887+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.891461+0000) 2019-10-13T11:14:37.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:37.887+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.891843+0000) 2019-10-13T11:14:37.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:37.887+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.893097+0000) 2019-10-13T11:14:37.894 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:37.887+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.893752+0000) 2019-10-13T11:14:37.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:37.887+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.893475+0000) 2019-10-13T11:14:37.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:37.892+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.894290+0000) 2019-10-13T11:14:37.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:37.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.894735+0000) 2019-10-13T11:14:37.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:37.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:37.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:37.900+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.902549+0000) 2019-10-13T11:14:38.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:38.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.653131+0000) 2019-10-13T11:14:38.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:38.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.658358+0000) 2019-10-13T11:14:38.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:38.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.669693+0000) 2019-10-13T11:14:38.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:38.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.703365+0000) 2019-10-13T11:14:40.507 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:14:40.507 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:40.513 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:14:40.513 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:42.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:42.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.143092+0000) 2019-10-13T11:14:42.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:42.895+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.901514+0000) 2019-10-13T11:14:42.903 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:42.895+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.901923+0000) 2019-10-13T11:14:42.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:42.895+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.902543+0000) 2019-10-13T11:14:42.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:42.895+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.902903+0000) 2019-10-13T11:14:42.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:42.895+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.903151+0000) 2019-10-13T11:14:42.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:42.900+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.903580+0000) 2019-10-13T11:14:42.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:42.899+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.904447+0000) 2019-10-13T11:14:42.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:42.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:42.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:42.908+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.911425+0000) 2019-10-13T11:14:43.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:43.136+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.139276+0000) 2019-10-13T11:14:46.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:46.387+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:46.389122+0000) 2019-10-13T11:14:47.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:47.208+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.212887+0000) 2019-10-13T11:14:47.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:47.903+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.910237+0000) 2019-10-13T11:14:47.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:47.903+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.911173+0000) 2019-10-13T11:14:47.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:47.908+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.911975+0000) 2019-10-13T11:14:47.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:47.907+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.912292+0000) 2019-10-13T11:14:47.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:47.907+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.912795+0000) 2019-10-13T11:14:47.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:47.907+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.913030+0000) 2019-10-13T11:14:47.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:47.907+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.914035+0000) 2019-10-13T11:14:47.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:47.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:47.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:47.916+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.920009+0000) 2019-10-13T11:14:48.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:48.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.653344+0000) 2019-10-13T11:14:48.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:48.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.658537+0000) 2019-10-13T11:14:48.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:48.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.669891+0000) 2019-10-13T11:14:48.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:48.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.703553+0000) 2019-10-13T11:14:52.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:52.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.143299+0000) 2019-10-13T11:14:52.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:52.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.920719+0000) 2019-10-13T11:14:52.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:52.915+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.921285+0000) 2019-10-13T11:14:52.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:52.915+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.921920+0000) 2019-10-13T11:14:52.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:52.919+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.922648+0000) 2019-10-13T11:14:52.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:52.915+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.922979+0000) 2019-10-13T11:14:52.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:52.919+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.923753+0000) 2019-10-13T11:14:52.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:52.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.924054+0000) 2019-10-13T11:14:52.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:52.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:52.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:52.923+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.927759+0000) 2019-10-13T11:14:53.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:53.135+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.139471+0000) 2019-10-13T11:14:56.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:56.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:56.389344+0000) 2019-10-13T11:14:57.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:57.215+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.219357+0000) 2019-10-13T11:14:57.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:57.923+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.929757+0000) 2019-10-13T11:14:57.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:14:57.923+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.930118+0000) 2019-10-13T11:14:57.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:57.923+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.930862+0000) 2019-10-13T11:14:57.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:14:57.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.931351+0000) 2019-10-13T11:14:57.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:14:57.927+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.932363+0000) 2019-10-13T11:14:57.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:57.927+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.932812+0000) 2019-10-13T11:14:57.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:57.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.933420+0000) 2019-10-13T11:14:57.936 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:14:57.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:14:57.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:14:57.935+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.938802+0000) 2019-10-13T11:14:58.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:14:58.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.653553+0000) 2019-10-13T11:14:58.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:14:58.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.658669+0000) 2019-10-13T11:14:58.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:14:58.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.670096+0000) 2019-10-13T11:14:58.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:14:58.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.703758+0000) 2019-10-13T11:15:02.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:02.139+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.143520+0000) 2019-10-13T11:15:02.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:02.931+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.938594+0000) 2019-10-13T11:15:02.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:02.931+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.938916+0000) 2019-10-13T11:15:02.940 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:02.935+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.939797+0000) 2019-10-13T11:15:02.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:02.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.940272+0000) 2019-10-13T11:15:02.942 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:02.935+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.941558+0000) 2019-10-13T11:15:02.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:02.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.942121+0000) 2019-10-13T11:15:02.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:02.939+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.942468+0000) 2019-10-13T11:15:02.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:02.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:02.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:02.943+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.947463+0000) 2019-10-13T11:15:03.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:03.135+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.139702+0000) 2019-10-13T11:15:06.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:06.383+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:06.389537+0000) 2019-10-13T11:15:07.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:07.223+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.225844+0000) 2019-10-13T11:15:07.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:07.943+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.946937+0000) 2019-10-13T11:15:07.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:07.943+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.947639+0000) 2019-10-13T11:15:07.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:07.943+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.948692+0000) 2019-10-13T11:15:07.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:07.943+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.949097+0000) 2019-10-13T11:15:07.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:07.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.949279+0000) 2019-10-13T11:15:07.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:07.943+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.949659+0000) 2019-10-13T11:15:07.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:07.947+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.950779+0000) 2019-10-13T11:15:07.953 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:07.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:07.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:07.951+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.957123+0000) 2019-10-13T11:15:08.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:08.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.653725+0000) 2019-10-13T11:15:08.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:08.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.658871+0000) 2019-10-13T11:15:08.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:08.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.670271+0000) 2019-10-13T11:15:08.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:08.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.703944+0000) 2019-10-13T11:15:10.630 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:15:10.630 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:10.634 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:15:10.634 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:12.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:12.143+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.143722+0000) 2019-10-13T11:15:12.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:12.955+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.956416+0000) 2019-10-13T11:15:12.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:12.955+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.956983+0000) 2019-10-13T11:15:12.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:12.955+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.957310+0000) 2019-10-13T11:15:12.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:12.955+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.958459+0000) 2019-10-13T11:15:12.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:12.955+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.958609+0000) 2019-10-13T11:15:12.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:12.955+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.959082+0000) 2019-10-13T11:15:12.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:12.955+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.959751+0000) 2019-10-13T11:15:12.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:12.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:12.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:12.959+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.965768+0000) 2019-10-13T11:15:13.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:13.139+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.139889+0000) 2019-10-13T11:15:16.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:16.387+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:16.389731+0000) 2019-10-13T11:15:17.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:17.231+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.232382+0000) 2019-10-13T11:15:17.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:17.963+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.965669+0000) 2019-10-13T11:15:17.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:17.963+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.966113+0000) 2019-10-13T11:15:17.967 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:17.963+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.966487+0000) 2019-10-13T11:15:17.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:17.963+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.966886+0000) 2019-10-13T11:15:17.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:17.967+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.967533+0000) 2019-10-13T11:15:17.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:17.967+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.967868+0000) 2019-10-13T11:15:17.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:17.967+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.968185+0000) 2019-10-13T11:15:17.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:17.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:17.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:17.971+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.976779+0000) 2019-10-13T11:15:18.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:18.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.653903+0000) 2019-10-13T11:15:18.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:18.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.659047+0000) 2019-10-13T11:15:18.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:18.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.670447+0000) 2019-10-13T11:15:18.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:18.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.704133+0000) 2019-10-13T11:15:22.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:22.142+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.143958+0000) 2019-10-13T11:15:22.976 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:22.970+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.974580+0000) 2019-10-13T11:15:22.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:22.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.975562+0000) 2019-10-13T11:15:22.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:22.971+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.976304+0000) 2019-10-13T11:15:22.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:22.974+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.976626+0000) 2019-10-13T11:15:22.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:22.974+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.977162+0000) 2019-10-13T11:15:22.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:22.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.977635+0000) 2019-10-13T11:15:22.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:22.974+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.977928+0000) 2019-10-13T11:15:22.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:22.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:22.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:22.975+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.981847+0000) 2019-10-13T11:15:23.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:23.139+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.140054+0000) 2019-10-13T11:15:26.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:26.386+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:26.389923+0000) 2019-10-13T11:15:27.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:27.235+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.238352+0000) 2019-10-13T11:15:27.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:27.982+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.983844+0000) 2019-10-13T11:15:27.985 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:27.982+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.984610+0000) 2019-10-13T11:15:27.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:27.982+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.986442+0000) 2019-10-13T11:15:27.987 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:27.982+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.986429+0000) 2019-10-13T11:15:27.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:27.983+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.987014+0000) 2019-10-13T11:15:27.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:27.982+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.987163+0000) 2019-10-13T11:15:27.989 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:27.986+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.987591+0000) 2019-10-13T11:15:27.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:27.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:27.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:27.987+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.992125+0000) 2019-10-13T11:15:28.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:28.650+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.654082+0000) 2019-10-13T11:15:28.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:28.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.659257+0000) 2019-10-13T11:15:28.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:28.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.670650+0000) 2019-10-13T11:15:28.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:28.702+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.704331+0000) 2019-10-13T11:15:32.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:32.142+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.144195+0000) 2019-10-13T11:15:32.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:32.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.992837+0000) 2019-10-13T11:15:32.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:32.990+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.993766+0000) 2019-10-13T11:15:32.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:32.990+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.994557+0000) 2019-10-13T11:15:32.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:32.990+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.994958+0000) 2019-10-13T11:15:32.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:32.991+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.995899+0000) 2019-10-13T11:15:32.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:32.994+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.996136+0000) 2019-10-13T11:15:32.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:32.994+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.996404+0000) 2019-10-13T11:15:32.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:32.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:33.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:32.995+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.001828+0000) 2019-10-13T11:15:33.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:33.139+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.140225+0000) 2019-10-13T11:15:36.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:36.386+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:36.390140+0000) 2019-10-13T11:15:37.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:37.243+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:37.244721+0000) 2019-10-13T11:15:38.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:37.998+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.002463+0000) 2019-10-13T11:15:38.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:37.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.002756+0000) 2019-10-13T11:15:38.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:37.998+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.003177+0000) 2019-10-13T11:15:38.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:38.002+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.003509+0000) 2019-10-13T11:15:38.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:37.999+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.003968+0000) 2019-10-13T11:15:38.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:38.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.005063+0000) 2019-10-13T11:15:38.006 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:38.002+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.005409+0000) 2019-10-13T11:15:38.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:38.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:38.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:38.007+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.012579+0000) 2019-10-13T11:15:38.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:38.650+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.654262+0000) 2019-10-13T11:15:38.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:38.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.659508+0000) 2019-10-13T11:15:38.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:38.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.670859+0000) 2019-10-13T11:15:38.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:38.702+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.704517+0000) 2019-10-13T11:15:40.679 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:15:40.680 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:40.683 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:15:40.684 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:42.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:42.142+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:42.144388+0000) 2019-10-13T11:15:43.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:43.010+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.012063+0000) 2019-10-13T11:15:43.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:43.007+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.013061+0000) 2019-10-13T11:15:43.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:43.010+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.013360+0000) 2019-10-13T11:15:43.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:43.010+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.013551+0000) 2019-10-13T11:15:43.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:43.010+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.014007+0000) 2019-10-13T11:15:43.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:43.010+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.014401+0000) 2019-10-13T11:15:43.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:43.010+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.014608+0000) 2019-10-13T11:15:43.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:43.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:43.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:43.015+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.019527+0000) 2019-10-13T11:15:43.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:43.135+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.140400+0000) 2019-10-13T11:15:46.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:46.386+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:46.390354+0000) 2019-10-13T11:15:47.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:47.254+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:47.255302+0000) 2019-10-13T11:15:48.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:48.018+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.021043+0000) 2019-10-13T11:15:48.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:48.018+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.021566+0000) 2019-10-13T11:15:48.023 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:48.018+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.022365+0000) 2019-10-13T11:15:48.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:48.018+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.022620+0000) 2019-10-13T11:15:48.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:48.018+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.023237+0000) 2019-10-13T11:15:48.024 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:48.022+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.023689+0000) 2019-10-13T11:15:48.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:48.018+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.024121+0000) 2019-10-13T11:15:48.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:48.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:48.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:48.022+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.028526+0000) 2019-10-13T11:15:48.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:48.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.654431+0000) 2019-10-13T11:15:48.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:48.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.659667+0000) 2019-10-13T11:15:48.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:48.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.671032+0000) 2019-10-13T11:15:48.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:48.702+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.704706+0000) 2019-10-13T11:15:52.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:52.142+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:52.144590+0000) 2019-10-13T11:15:53.030 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:53.026+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.029598+0000) 2019-10-13T11:15:53.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:53.026+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.030695+0000) 2019-10-13T11:15:53.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:53.030+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.031580+0000) 2019-10-13T11:15:53.033 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:53.030+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.032214+0000) 2019-10-13T11:15:53.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:53.026+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.032712+0000) 2019-10-13T11:15:53.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:53.030+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.032976+0000) 2019-10-13T11:15:53.034 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:53.030+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.033518+0000) 2019-10-13T11:15:53.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:53.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:53.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:53.030+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.037823+0000) 2019-10-13T11:15:53.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:53.138+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.140574+0000) 2019-10-13T11:15:56.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:56.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:56.390568+0000) 2019-10-13T11:15:57.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:57.254+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:57.261577+0000) 2019-10-13T11:15:58.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:58.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.039415+0000) 2019-10-13T11:15:58.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:58.038+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.039625+0000) 2019-10-13T11:15:58.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:58.038+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.039921+0000) 2019-10-13T11:15:58.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:15:58.034+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.040290+0000) 2019-10-13T11:15:58.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:15:58.038+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.040746+0000) 2019-10-13T11:15:58.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:58.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.041054+0000) 2019-10-13T11:15:58.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:15:58.038+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.041850+0000) 2019-10-13T11:15:58.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:15:58.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:15:58.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:15:58.042+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.049476+0000) 2019-10-13T11:15:58.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:15:58.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.654643+0000) 2019-10-13T11:15:58.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:15:58.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.659868+0000) 2019-10-13T11:15:58.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:15:58.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.671205+0000) 2019-10-13T11:15:58.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:15:58.702+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.704887+0000) 2019-10-13T11:16:02.145 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:02.142+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:02.144807+0000) 2019-10-13T11:16:03.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:03.042+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.047197+0000) 2019-10-13T11:16:03.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:03.046+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.048329+0000) 2019-10-13T11:16:03.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:03.046+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.049026+0000) 2019-10-13T11:16:03.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:03.042+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.049932+0000) 2019-10-13T11:16:03.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:03.046+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.050082+0000) 2019-10-13T11:16:03.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:03.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.050766+0000) 2019-10-13T11:16:03.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:03.046+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.050959+0000) 2019-10-13T11:16:03.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:03.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:03.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:03.050+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.056531+0000) 2019-10-13T11:16:03.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:03.134+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.140741+0000) 2019-10-13T11:16:06.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:06.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:06.390764+0000) 2019-10-13T11:16:07.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:07.266+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:07.267461+0000) 2019-10-13T11:16:08.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:08.053+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.056143+0000) 2019-10-13T11:16:08.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:08.053+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.057140+0000) 2019-10-13T11:16:08.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:08.053+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.057650+0000) 2019-10-13T11:16:08.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:08.053+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.057688+0000) 2019-10-13T11:16:08.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:08.054+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.058606+0000) 2019-10-13T11:16:08.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:08.053+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.058824+0000) 2019-10-13T11:16:08.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:08.053+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.059141+0000) 2019-10-13T11:16:08.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:08.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:08.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:08.058+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.064616+0000) 2019-10-13T11:16:08.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:08.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.654817+0000) 2019-10-13T11:16:08.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:08.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.660056+0000) 2019-10-13T11:16:08.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:08.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.671383+0000) 2019-10-13T11:16:08.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:08.701+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.705104+0000) 2019-10-13T11:16:10.803 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:16:10.803 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:10.806 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:16:10.807 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:12.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:12.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:12.145014+0000) 2019-10-13T11:16:13.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:13.062+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.066673+0000) 2019-10-13T11:16:13.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:13.061+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.067101+0000) 2019-10-13T11:16:13.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:13.061+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.067229+0000) 2019-10-13T11:16:13.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:13.065+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.067749+0000) 2019-10-13T11:16:13.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:13.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.067856+0000) 2019-10-13T11:16:13.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:13.065+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.069036+0000) 2019-10-13T11:16:13.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:13.065+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.069421+0000) 2019-10-13T11:16:13.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:13.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:13.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:13.070+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.074490+0000) 2019-10-13T11:16:13.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:13.138+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.140875+0000) 2019-10-13T11:16:16.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:16.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:16.390955+0000) 2019-10-13T11:16:17.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:17.270+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:17.273773+0000) 2019-10-13T11:16:18.076 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:18.069+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.075179+0000) 2019-10-13T11:16:18.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:18.070+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.075764+0000) 2019-10-13T11:16:18.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:18.073+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.076271+0000) 2019-10-13T11:16:18.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:18.073+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.076543+0000) 2019-10-13T11:16:18.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:18.073+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.077217+0000) 2019-10-13T11:16:18.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:18.073+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.078089+0000) 2019-10-13T11:16:18.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:18.073+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.078517+0000) 2019-10-13T11:16:18.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:18.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:18.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:18.078+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.085318+0000) 2019-10-13T11:16:18.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:18.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.654998+0000) 2019-10-13T11:16:18.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:18.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.660266+0000) 2019-10-13T11:16:18.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:18.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.671567+0000) 2019-10-13T11:16:18.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:18.701+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.705286+0000) 2019-10-13T11:16:22.146 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:22.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:22.145179+0000) 2019-10-13T11:16:23.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:23.081+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.084328+0000) 2019-10-13T11:16:23.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:23.081+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.084684+0000) 2019-10-13T11:16:23.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:23.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.085085+0000) 2019-10-13T11:16:23.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:23.081+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.085813+0000) 2019-10-13T11:16:23.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:23.081+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.086261+0000) 2019-10-13T11:16:23.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:23.081+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.087158+0000) 2019-10-13T11:16:23.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:23.082+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.087933+0000) 2019-10-13T11:16:23.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:23.085+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:23.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:23.086+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.093239+0000) 2019-10-13T11:16:23.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:23.134+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.141077+0000) 2019-10-13T11:16:26.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:26.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:26.391148+0000) 2019-10-13T11:16:27.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:27.278+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:27.280216+0000) 2019-10-13T11:16:28.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:28.089+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.093324+0000) 2019-10-13T11:16:28.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:28.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.094168+0000) 2019-10-13T11:16:28.095 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:28.089+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.094426+0000) 2019-10-13T11:16:28.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:28.089+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.094995+0000) 2019-10-13T11:16:28.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:28.089+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.095227+0000) 2019-10-13T11:16:28.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:28.090+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.095652+0000) 2019-10-13T11:16:28.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:28.093+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.096898+0000) 2019-10-13T11:16:28.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:28.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:28.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:28.098+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.102497+0000) 2019-10-13T11:16:28.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:28.649+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.655180+0000) 2019-10-13T11:16:28.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:28.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.660471+0000) 2019-10-13T11:16:28.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:28.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.671744+0000) 2019-10-13T11:16:28.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:28.701+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.705492+0000) 2019-10-13T11:16:32.146 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:32.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:32.145379+0000) 2019-10-13T11:16:33.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:33.097+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.102753+0000) 2019-10-13T11:16:33.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:33.101+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.103581+0000) 2019-10-13T11:16:33.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:33.101+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.104263+0000) 2019-10-13T11:16:33.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:33.101+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.104739+0000) 2019-10-13T11:16:33.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:33.102+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.105496+0000) 2019-10-13T11:16:33.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:33.101+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.105896+0000) 2019-10-13T11:16:33.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:33.101+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.106232+0000) 2019-10-13T11:16:33.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:33.110+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.110196+0000) 2019-10-13T11:16:33.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:33.105+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:33.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:33.138+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.141229+0000) 2019-10-13T11:16:36.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:36.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:36.391342+0000) 2019-10-13T11:16:37.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:37.286+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:37.286615+0000) 2019-10-13T11:16:38.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:38.109+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.112127+0000) 2019-10-13T11:16:38.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:38.109+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.113220+0000) 2019-10-13T11:16:38.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:38.109+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.113842+0000) 2019-10-13T11:16:38.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:38.109+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.114576+0000) 2019-10-13T11:16:38.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:38.109+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.114461+0000) 2019-10-13T11:16:38.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:38.109+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.115180+0000) 2019-10-13T11:16:38.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:38.114+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.116308+0000) 2019-10-13T11:16:38.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:38.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:38.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:38.118+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.122065+0000) 2019-10-13T11:16:38.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:38.653+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.655372+0000) 2019-10-13T11:16:38.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:38.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.660646+0000) 2019-10-13T11:16:38.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:38.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.671955+0000) 2019-10-13T11:16:38.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:38.701+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.705680+0000) 2019-10-13T11:16:40.960 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:16:40.960 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:40.964 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:16:40.964 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:42.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:42.141+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:42.145599+0000) 2019-10-13T11:16:43.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:43.117+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.122463+0000) 2019-10-13T11:16:43.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:43.117+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.123070+0000) 2019-10-13T11:16:43.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:43.121+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.123552+0000) 2019-10-13T11:16:43.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:43.121+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.124037+0000) 2019-10-13T11:16:43.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:43.121+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.124300+0000) 2019-10-13T11:16:43.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:43.121+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.125421+0000) 2019-10-13T11:16:43.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:43.121+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.125667+0000) 2019-10-13T11:16:43.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:43.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:43.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:43.129+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.132937+0000) 2019-10-13T11:16:43.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:43.137+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.141390+0000) 2019-10-13T11:16:46.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:46.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:46.391567+0000) 2019-10-13T11:16:47.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:47.289+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:47.293043+0000) 2019-10-13T11:16:48.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:48.128+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.132203+0000) 2019-10-13T11:16:48.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:48.128+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.132532+0000) 2019-10-13T11:16:48.134 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:48.128+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.132820+0000) 2019-10-13T11:16:48.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:48.128+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.133512+0000) 2019-10-13T11:16:48.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:48.133+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.134281+0000) 2019-10-13T11:16:48.135 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:48.128+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.134828+0000) 2019-10-13T11:16:48.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:48.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.135749+0000) 2019-10-13T11:16:48.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:48.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:48.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:48.141+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.142414+0000) 2019-10-13T11:16:48.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:48.652+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.655591+0000) 2019-10-13T11:16:48.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:48.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.660824+0000) 2019-10-13T11:16:48.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:48.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.672131+0000) 2019-10-13T11:16:48.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:48.700+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.705862+0000) 2019-10-13T11:16:52.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:52.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:52.145824+0000) 2019-10-13T11:16:53.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:53.136+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.141021+0000) 2019-10-13T11:16:53.143 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:53.136+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.142401+0000) 2019-10-13T11:16:53.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:53.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.143450+0000) 2019-10-13T11:16:53.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:53.140+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.143837+0000) 2019-10-13T11:16:53.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:53.141+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.144220+0000) 2019-10-13T11:16:53.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:53.140+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.144467+0000) 2019-10-13T11:16:53.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:53.140+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.145362+0000) 2019-10-13T11:16:53.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:53.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:53.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:53.149+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.151879+0000) 2019-10-13T11:16:56.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:56.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:56.391768+0000) 2019-10-13T11:16:57.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:57.297+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:57.299529+0000) 2019-10-13T11:16:58.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:58.144+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.151231+0000) 2019-10-13T11:16:58.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:58.148+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.151556+0000) 2019-10-13T11:16:58.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:58.149+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.152309+0000) 2019-10-13T11:16:58.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:58.148+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.152524+0000) 2019-10-13T11:16:58.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:16:58.148+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.153016+0000) 2019-10-13T11:16:58.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:58.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.153759+0000) 2019-10-13T11:16:58.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:16:58.148+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.154460+0000) 2019-10-13T11:16:58.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:16:58.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:16:58.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:16:58.157+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.161271+0000) 2019-10-13T11:16:58.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:16:58.652+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.655795+0000) 2019-10-13T11:16:58.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:16:58.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.661025+0000) 2019-10-13T11:16:58.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:16:58.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.672323+0000) 2019-10-13T11:16:58.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:16:58.700+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.706080+0000) 2019-10-13T11:16:59.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:16:59.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:59.892049+0000) 2019-10-13T11:17:02.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:02.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:02.146023+0000) 2019-10-13T11:17:03.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:03.156+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.160402+0000) 2019-10-13T11:17:03.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:03.156+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.160812+0000) 2019-10-13T11:17:03.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:03.156+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.161371+0000) 2019-10-13T11:17:03.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:03.161+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.162189+0000) 2019-10-13T11:17:03.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:03.156+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.162606+0000) 2019-10-13T11:17:03.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:03.156+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.162887+0000) 2019-10-13T11:17:03.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:03.160+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.163520+0000) 2019-10-13T11:17:03.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:03.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 119998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:03.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:03.165+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.169731+0000) 2019-10-13T11:17:06.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:06.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:06.391982+0000) 2019-10-13T11:17:07.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:07.305+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:07.306171+0000) 2019-10-13T11:17:08.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:08.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.169968+0000) 2019-10-13T11:17:08.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:08.164+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.170337+0000) 2019-10-13T11:17:08.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:08.169+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.171064+0000) 2019-10-13T11:17:08.173 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:08.164+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.171195+0000) 2019-10-13T11:17:08.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:08.168+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.171976+0000) 2019-10-13T11:17:08.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:08.168+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.172815+0000) 2019-10-13T11:17:08.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:08.168+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.173867+0000) 2019-10-13T11:17:08.178 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:08.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:08.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:08.177+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.178249+0000) 2019-10-13T11:17:08.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:08.652+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.655970+0000) 2019-10-13T11:17:08.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:08.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.661197+0000) 2019-10-13T11:17:08.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:08.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.672522+0000) 2019-10-13T11:17:08.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:08.700+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.706315+0000) 2019-10-13T11:17:09.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:09.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:09.892236+0000) 2019-10-13T11:17:11.092 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:17:11.092 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:11.096 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:17:11.096 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:12.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:12.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:12.146243+0000) 2019-10-13T11:17:13.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:13.177+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.179496+0000) 2019-10-13T11:17:13.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:13.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.179828+0000) 2019-10-13T11:17:13.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:13.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.180635+0000) 2019-10-13T11:17:13.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:13.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.181458+0000) 2019-10-13T11:17:13.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:13.176+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.181670+0000) 2019-10-13T11:17:13.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:13.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.182261+0000) 2019-10-13T11:17:13.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:13.176+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.182711+0000) 2019-10-13T11:17:13.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:13.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:13.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:13.185+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.189840+0000) 2019-10-13T11:17:16.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:16.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:16.392183+0000) 2019-10-13T11:17:17.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:17.309+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:17.312856+0000) 2019-10-13T11:17:18.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:18.184+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.189490+0000) 2019-10-13T11:17:18.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:18.184+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.190238+0000) 2019-10-13T11:17:18.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:18.184+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.190501+0000) 2019-10-13T11:17:18.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:18.189+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.191516+0000) 2019-10-13T11:17:18.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:18.184+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.190830+0000) 2019-10-13T11:17:18.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:18.188+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.191645+0000) 2019-10-13T11:17:18.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:18.188+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.192084+0000) 2019-10-13T11:17:18.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:18.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:18.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:18.197+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.200573+0000) 2019-10-13T11:17:18.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:18.652+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.656147+0000) 2019-10-13T11:17:18.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:18.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.661379+0000) 2019-10-13T11:17:18.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:18.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.672700+0000) 2019-10-13T11:17:18.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:18.700+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.706493+0000) 2019-10-13T11:17:19.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:19.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:19.892437+0000) 2019-10-13T11:17:22.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:22.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:22.146443+0000) 2019-10-13T11:17:23.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:23.192+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.198532+0000) 2019-10-13T11:17:23.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:23.192+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.199207+0000) 2019-10-13T11:17:23.200 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:23.196+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.199417+0000) 2019-10-13T11:17:23.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:23.196+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.200020+0000) 2019-10-13T11:17:23.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:23.197+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.200450+0000) 2019-10-13T11:17:23.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:23.196+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.201214+0000) 2019-10-13T11:17:23.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:23.196+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.201421+0000) 2019-10-13T11:17:23.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:23.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120125 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:23.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:23.201+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.204130+0000) 2019-10-13T11:17:26.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:26.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:26.392379+0000) 2019-10-13T11:17:27.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:27.317+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:27.319557+0000) 2019-10-13T11:17:28.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:28.204+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.207328+0000) 2019-10-13T11:17:28.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:28.204+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.208229+0000) 2019-10-13T11:17:28.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:28.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.209375+0000) 2019-10-13T11:17:28.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:28.204+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.209934+0000) 2019-10-13T11:17:28.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:28.204+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.210288+0000) 2019-10-13T11:17:28.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:28.209+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.210805+0000) 2019-10-13T11:17:28.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:28.204+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.211228+0000) 2019-10-13T11:17:28.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:28.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:28.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:28.213+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.215435+0000) 2019-10-13T11:17:28.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:28.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.656330+0000) 2019-10-13T11:17:28.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:28.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.661559+0000) 2019-10-13T11:17:28.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:28.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.672897+0000) 2019-10-13T11:17:28.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:28.700+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.706669+0000) 2019-10-13T11:17:29.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:29.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:29.892664+0000) 2019-10-13T11:17:32.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:32.143+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:32.146645+0000) 2019-10-13T11:17:33.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:33.211+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.217470+0000) 2019-10-13T11:17:33.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:33.211+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.217880+0000) 2019-10-13T11:17:33.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:33.217+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.219681+0000) 2019-10-13T11:17:33.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:33.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.220019+0000) 2019-10-13T11:17:33.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:33.215+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.220448+0000) 2019-10-13T11:17:33.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:33.215+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.220834+0000) 2019-10-13T11:17:33.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:33.215+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.221838+0000) 2019-10-13T11:17:33.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:33.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:33.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:33.221+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.224350+0000) 2019-10-13T11:17:36.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:36.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:36.392578+0000) 2019-10-13T11:17:37.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:37.324+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:37.326034+0000) 2019-10-13T11:17:38.227 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:38.224+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.226304+0000) 2019-10-13T11:17:38.228 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:38.219+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.227059+0000) 2019-10-13T11:17:38.229 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:38.223+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.228280+0000) 2019-10-13T11:17:38.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:38.223+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.228738+0000) 2019-10-13T11:17:38.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:38.223+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.229080+0000) 2019-10-13T11:17:38.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:38.223+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.229624+0000) 2019-10-13T11:17:38.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:38.223+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.230136+0000) 2019-10-13T11:17:38.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:38.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:38.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:38.232+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.234438+0000) 2019-10-13T11:17:38.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:38.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.656468+0000) 2019-10-13T11:17:38.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:38.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.661734+0000) 2019-10-13T11:17:38.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:38.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.673128+0000) 2019-10-13T11:17:38.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:38.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.706888+0000) 2019-10-13T11:17:39.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:39.888+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:39.892856+0000) 2019-10-13T11:17:41.216 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:17:41.216 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:41.220 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:17:41.220 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:42.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:42.143+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:42.146871+0000) 2019-10-13T11:17:43.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:43.232+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.236088+0000) 2019-10-13T11:17:43.237 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:43.231+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.236446+0000) 2019-10-13T11:17:43.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:43.231+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.236659+0000) 2019-10-13T11:17:43.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:43.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.236859+0000) 2019-10-13T11:17:43.238 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:43.231+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.237233+0000) 2019-10-13T11:17:43.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:43.231+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.238061+0000) 2019-10-13T11:17:43.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:43.231+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.239163+0000) 2019-10-13T11:17:43.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:43.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:43.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:43.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.247273+0000) 2019-10-13T11:17:46.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:46.387+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:46.392769+0000) 2019-10-13T11:17:47.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:47.328+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:47.332813+0000) 2019-10-13T11:17:48.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:48.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.246055+0000) 2019-10-13T11:17:48.247 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:48.244+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.246905+0000) 2019-10-13T11:17:48.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:48.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.246944+0000) 2019-10-13T11:17:48.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:48.243+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.247351+0000) 2019-10-13T11:17:48.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:48.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.247849+0000) 2019-10-13T11:17:48.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:48.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.248761+0000) 2019-10-13T11:17:48.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:48.243+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.248158+0000) 2019-10-13T11:17:48.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:48.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:48.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:48.252+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.255631+0000) 2019-10-13T11:17:48.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:48.655+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.656647+0000) 2019-10-13T11:17:48.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:48.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.661934+0000) 2019-10-13T11:17:48.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:48.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.673320+0000) 2019-10-13T11:17:48.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:48.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.707075+0000) 2019-10-13T11:17:49.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:49.888+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:49.893026+0000) 2019-10-13T11:17:52.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:52.143+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:52.147077+0000) 2019-10-13T11:17:53.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:53.247+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.254557+0000) 2019-10-13T11:17:53.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:53.247+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.254878+0000) 2019-10-13T11:17:53.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:53.252+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.255455+0000) 2019-10-13T11:17:53.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:53.251+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.256151+0000) 2019-10-13T11:17:53.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:53.251+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.256428+0000) 2019-10-13T11:17:53.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:53.251+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.257018+0000) 2019-10-13T11:17:53.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:53.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.258233+0000) 2019-10-13T11:17:53.261 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:53.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:53.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:53.260+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.263912+0000) 2019-10-13T11:17:56.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:56.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:56.392960+0000) 2019-10-13T11:17:57.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:57.340+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:57.342566+0000) 2019-10-13T11:17:58.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:58.259+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.263885+0000) 2019-10-13T11:17:58.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:17:58.259+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.264922+0000) 2019-10-13T11:17:58.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:17:58.259+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.265295+0000) 2019-10-13T11:17:58.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:58.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.265903+0000) 2019-10-13T11:17:58.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:58.259+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.265956+0000) 2019-10-13T11:17:58.267 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:58.264+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.266438+0000) 2019-10-13T11:17:58.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:58.259+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.266944+0000) 2019-10-13T11:17:58.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:17:58.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:17:58.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:17:58.268+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.273368+0000) 2019-10-13T11:17:58.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:17:58.651+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.656821+0000) 2019-10-13T11:17:58.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:17:58.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.662137+0000) 2019-10-13T11:17:58.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:17:58.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.673503+0000) 2019-10-13T11:17:58.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:17:58.699+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.707252+0000) 2019-10-13T11:17:59.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:17:59.892+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.893216+0000) 2019-10-13T11:18:02.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:02.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:02.147333+0000) 2019-10-13T11:18:03.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:03.271+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.273072+0000) 2019-10-13T11:18:03.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:03.271+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.273451+0000) 2019-10-13T11:18:03.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:03.271+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.273890+0000) 2019-10-13T11:18:03.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:03.271+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.275023+0000) 2019-10-13T11:18:03.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:03.275+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.275562+0000) 2019-10-13T11:18:03.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:03.272+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.276941+0000) 2019-10-13T11:18:03.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:03.275+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.277445+0000) 2019-10-13T11:18:03.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:03.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:03.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:03.280+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.282430+0000) 2019-10-13T11:18:06.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:06.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:06.393183+0000) 2019-10-13T11:18:07.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:07.344+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:07.349005+0000) 2019-10-13T11:18:08.283 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:08.279+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.282193+0000) 2019-10-13T11:18:08.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:08.283+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.283576+0000) 2019-10-13T11:18:08.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:08.280+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.284119+0000) 2019-10-13T11:18:08.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:08.283+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.283592+0000) 2019-10-13T11:18:08.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:08.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.284285+0000) 2019-10-13T11:18:08.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:08.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.285171+0000) 2019-10-13T11:18:08.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:08.283+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.285474+0000) 2019-10-13T11:18:08.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:08.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:08.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:08.284+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.289414+0000) 2019-10-13T11:18:08.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:08.655+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.656994+0000) 2019-10-13T11:18:08.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:08.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.662319+0000) 2019-10-13T11:18:08.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:08.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.673678+0000) 2019-10-13T11:18:08.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:08.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.707500+0000) 2019-10-13T11:18:09.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:09.888+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.893455+0000) 2019-10-13T11:18:11.336 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:18:11.336 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:11.340 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:18:11.340 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:12.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:12.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:12.147549+0000) 2019-10-13T11:18:13.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:13.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.291968+0000) 2019-10-13T11:18:13.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:13.291+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.291874+0000) 2019-10-13T11:18:13.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:13.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.293166+0000) 2019-10-13T11:18:13.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:13.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.292668+0000) 2019-10-13T11:18:13.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:13.291+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.292620+0000) 2019-10-13T11:18:13.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:13.291+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.294035+0000) 2019-10-13T11:18:13.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:13.291+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.294403+0000) 2019-10-13T11:18:13.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:13.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:13.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:13.300+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.303889+0000) 2019-10-13T11:18:16.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:16.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:16.393342+0000) 2019-10-13T11:18:17.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:17.356+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:17.359503+0000) 2019-10-13T11:18:18.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:18.296+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.299762+0000) 2019-10-13T11:18:18.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:18.298+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.300622+0000) 2019-10-13T11:18:18.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:18.298+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.301137+0000) 2019-10-13T11:18:18.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:18.298+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.301451+0000) 2019-10-13T11:18:18.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:18.298+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.301987+0000) 2019-10-13T11:18:18.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:18.298+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.302106+0000) 2019-10-13T11:18:18.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:18.298+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.302423+0000) 2019-10-13T11:18:18.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:18.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:18.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:18.304+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.309297+0000) 2019-10-13T11:18:18.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:18.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.657168+0000) 2019-10-13T11:18:18.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:18.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.662491+0000) 2019-10-13T11:18:18.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:18.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.673859+0000) 2019-10-13T11:18:18.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:18.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.707717+0000) 2019-10-13T11:18:19.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:19.892+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.893689+0000) 2019-10-13T11:18:22.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:22.146+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:22.147756+0000) 2019-10-13T11:18:23.309 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:23.306+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.308355+0000) 2019-10-13T11:18:23.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:23.306+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.309012+0000) 2019-10-13T11:18:23.310 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:23.306+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.309148+0000) 2019-10-13T11:18:23.310 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:23.306+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.309633+0000) 2019-10-13T11:18:23.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:23.306+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.310274+0000) 2019-10-13T11:18:23.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:23.308+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.311328+0000) 2019-10-13T11:18:23.312 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:23.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.311770+0000) 2019-10-13T11:18:23.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:23.310+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:23.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:23.312+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.317693+0000) 2019-10-13T11:18:26.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:26.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:26.393564+0000) 2019-10-13T11:18:27.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:27.368+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:27.370834+0000) 2019-10-13T11:18:28.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:28.314+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.318281+0000) 2019-10-13T11:18:28.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:28.314+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.318689+0000) 2019-10-13T11:18:28.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:28.318+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.319293+0000) 2019-10-13T11:18:28.320 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:28.318+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.319964+0000) 2019-10-13T11:18:28.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:28.318+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.320624+0000) 2019-10-13T11:18:28.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:28.316+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.321078+0000) 2019-10-13T11:18:28.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:28.318+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.321753+0000) 2019-10-13T11:18:28.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:28.322+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:28.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:28.324+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.326863+0000) 2019-10-13T11:18:28.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:28.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.657343+0000) 2019-10-13T11:18:28.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:28.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.662668+0000) 2019-10-13T11:18:28.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:28.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.674075+0000) 2019-10-13T11:18:28.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:28.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.707852+0000) 2019-10-13T11:18:29.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:29.888+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.893918+0000) 2019-10-13T11:18:32.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:32.146+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:32.147951+0000) 2019-10-13T11:18:33.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:33.326+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.328079+0000) 2019-10-13T11:18:33.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:33.326+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.329250+0000) 2019-10-13T11:18:33.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:33.326+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.328946+0000) 2019-10-13T11:18:33.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:33.326+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.329961+0000) 2019-10-13T11:18:33.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:33.326+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.330299+0000) 2019-10-13T11:18:33.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:33.326+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.330451+0000) 2019-10-13T11:18:33.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:33.327+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.330838+0000) 2019-10-13T11:18:33.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:33.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:33.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:33.335+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.338484+0000) 2019-10-13T11:18:36.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:36.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:36.393761+0000) 2019-10-13T11:18:37.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:37.379+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:37.382563+0000) 2019-10-13T11:18:38.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:38.335+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.338791+0000) 2019-10-13T11:18:38.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:38.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.338968+0000) 2019-10-13T11:18:38.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:38.338+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.339392+0000) 2019-10-13T11:18:38.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:38.338+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.339966+0000) 2019-10-13T11:18:38.341 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:38.338+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.340484+0000) 2019-10-13T11:18:38.342 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:38.338+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.340913+0000) 2019-10-13T11:18:38.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:38.338+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.341230+0000) 2019-10-13T11:18:38.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:38.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:38.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:38.343+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.349381+0000) 2019-10-13T11:18:38.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:38.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.657515+0000) 2019-10-13T11:18:38.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:38.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.662861+0000) 2019-10-13T11:18:38.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:38.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.674249+0000) 2019-10-13T11:18:38.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:38.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.708046+0000) 2019-10-13T11:18:39.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:39.891+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.894114+0000) 2019-10-13T11:18:41.448 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:18:41.448 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:41.452 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:18:41.452 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:42.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:42.146+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:42.148159+0000) 2019-10-13T11:18:43.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:43.342+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.347186+0000) 2019-10-13T11:18:43.349 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:43.346+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.348044+0000) 2019-10-13T11:18:43.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:43.346+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.348414+0000) 2019-10-13T11:18:43.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:43.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.349396+0000) 2019-10-13T11:18:43.351 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:43.346+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.350205+0000) 2019-10-13T11:18:43.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:43.347+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.350770+0000) 2019-10-13T11:18:43.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:43.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.351889+0000) 2019-10-13T11:18:43.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:43.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:43.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:43.351+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.355198+0000) 2019-10-13T11:18:46.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:46.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:46.393959+0000) 2019-10-13T11:18:47.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:47.391+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:47.393756+0000) 2019-10-13T11:18:48.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:48.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.356514+0000) 2019-10-13T11:18:48.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:48.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.357163+0000) 2019-10-13T11:18:48.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:48.354+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.357559+0000) 2019-10-13T11:18:48.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:48.351+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.358027+0000) 2019-10-13T11:18:48.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:48.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.358441+0000) 2019-10-13T11:18:48.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:48.354+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.358872+0000) 2019-10-13T11:18:48.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:48.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.359611+0000) 2019-10-13T11:18:48.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:48.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:48.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:48.363+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.366455+0000) 2019-10-13T11:18:48.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:48.654+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.657702+0000) 2019-10-13T11:18:48.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:48.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.663030+0000) 2019-10-13T11:18:48.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:48.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.674438+0000) 2019-10-13T11:18:48.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:48.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.708279+0000) 2019-10-13T11:18:49.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:49.891+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.894334+0000) 2019-10-13T11:18:52.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:52.146+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:52.148364+0000) 2019-10-13T11:18:53.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:53.362+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.365406+0000) 2019-10-13T11:18:53.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:53.362+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.366122+0000) 2019-10-13T11:18:53.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:53.362+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.366546+0000) 2019-10-13T11:18:53.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:53.362+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.367118+0000) 2019-10-13T11:18:53.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:53.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.367235+0000) 2019-10-13T11:18:53.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:53.363+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.369334+0000) 2019-10-13T11:18:53.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:53.366+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.369379+0000) 2019-10-13T11:18:53.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:53.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:53.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:53.371+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.375312+0000) 2019-10-13T11:18:56.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:56.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:56.394197+0000) 2019-10-13T11:18:57.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:57.399+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:57.400242+0000) 2019-10-13T11:18:58.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:58.370+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.374459+0000) 2019-10-13T11:18:58.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:18:58.370+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.374685+0000) 2019-10-13T11:18:58.376 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:18:58.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.375333+0000) 2019-10-13T11:18:58.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:58.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.376210+0000) 2019-10-13T11:18:58.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:58.374+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.376633+0000) 2019-10-13T11:18:58.378 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:58.371+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.377095+0000) 2019-10-13T11:18:58.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:58.374+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.378208+0000) 2019-10-13T11:18:58.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:18:58.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:18:58.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:18:58.383+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.387466+0000) 2019-10-13T11:18:58.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:18:58.653+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.657872+0000) 2019-10-13T11:18:58.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:18:58.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.663206+0000) 2019-10-13T11:18:58.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:18:58.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.674613+0000) 2019-10-13T11:18:58.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:18:58.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.708469+0000) 2019-10-13T11:18:59.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:18:59.891+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.894537+0000) 2019-10-13T11:19:02.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:02.145+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:02.148563+0000) 2019-10-13T11:19:03.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:03.381+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.383612+0000) 2019-10-13T11:19:03.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:03.381+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.383986+0000) 2019-10-13T11:19:03.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:03.381+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.385134+0000) 2019-10-13T11:19:03.386 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:03.379+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.385623+0000) 2019-10-13T11:19:03.386 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:03.381+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.385232+0000) 2019-10-13T11:19:03.387 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:03.381+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.386026+0000) 2019-10-13T11:19:03.387 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:03.381+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.386560+0000) 2019-10-13T11:19:03.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:03.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:03.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:03.387+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.393721+0000) 2019-10-13T11:19:06.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:06.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:06.394411+0000) 2019-10-13T11:19:07.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:07.403+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:07.406683+0000) 2019-10-13T11:19:08.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:08.389+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.393251+0000) 2019-10-13T11:19:08.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:08.389+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.393720+0000) 2019-10-13T11:19:08.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:08.389+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.394144+0000) 2019-10-13T11:19:08.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:08.389+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.394719+0000) 2019-10-13T11:19:08.396 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:08.391+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.395239+0000) 2019-10-13T11:19:08.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:08.393+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.395476+0000) 2019-10-13T11:19:08.397 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:08.393+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.396902+0000) 2019-10-13T11:19:08.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:08.393+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:08.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:08.399+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.403157+0000) 2019-10-13T11:19:08.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:08.653+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.658068+0000) 2019-10-13T11:19:08.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:08.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.663381+0000) 2019-10-13T11:19:08.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:08.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.674824+0000) 2019-10-13T11:19:08.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:08.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.708713+0000) 2019-10-13T11:19:09.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:09.891+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.894726+0000) 2019-10-13T11:19:11.555 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:19:11.555 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:11.559 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:19:11.559 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:12.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:12.145+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:12.148759+0000) 2019-10-13T11:19:13.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:13.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.403472+0000) 2019-10-13T11:19:13.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:13.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.403921+0000) 2019-10-13T11:19:13.405 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:13.401+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.404487+0000) 2019-10-13T11:19:13.406 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:13.401+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.404999+0000) 2019-10-13T11:19:13.406 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:13.401+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.405705+0000) 2019-10-13T11:19:13.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:13.401+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.406075+0000) 2019-10-13T11:19:13.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:13.403+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.406986+0000) 2019-10-13T11:19:13.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:13.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120832 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:13.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:13.407+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.411818+0000) 2019-10-13T11:19:16.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:16.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:16.394637+0000) 2019-10-13T11:19:17.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:17.411+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:17.413187+0000) 2019-10-13T11:19:18.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:18.409+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.412223+0000) 2019-10-13T11:19:18.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:18.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.413331+0000) 2019-10-13T11:19:18.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:18.407+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.414073+0000) 2019-10-13T11:19:18.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:18.409+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.413615+0000) 2019-10-13T11:19:18.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:18.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.414636+0000) 2019-10-13T11:19:18.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:18.413+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.415478+0000) 2019-10-13T11:19:18.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:18.413+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.415621+0000) 2019-10-13T11:19:18.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:18.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:18.419 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:18.415+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.419063+0000) 2019-10-13T11:19:18.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:18.653+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.658242+0000) 2019-10-13T11:19:18.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:18.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.663579+0000) 2019-10-13T11:19:18.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:18.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.675030+0000) 2019-10-13T11:19:18.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:18.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.708915+0000) 2019-10-13T11:19:19.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:19.891+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.894936+0000) 2019-10-13T11:19:22.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:22.145+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:22.148958+0000) 2019-10-13T11:19:23.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:23.417+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.421482+0000) 2019-10-13T11:19:23.423 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:23.417+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.422088+0000) 2019-10-13T11:19:23.423 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:23.417+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.422297+0000) 2019-10-13T11:19:23.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:23.417+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.422994+0000) 2019-10-13T11:19:23.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:23.421+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.423381+0000) 2019-10-13T11:19:23.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:23.419+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.424087+0000) 2019-10-13T11:19:23.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:23.421+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.423667+0000) 2019-10-13T11:19:23.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:23.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:23.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:23.423+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.427956+0000) 2019-10-13T11:19:26.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:26.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:26.394833+0000) 2019-10-13T11:19:27.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:27.418+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:27.419705+0000) 2019-10-13T11:19:28.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:28.425+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.430371+0000) 2019-10-13T11:19:28.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:28.429+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.431458+0000) 2019-10-13T11:19:28.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:28.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.431935+0000) 2019-10-13T11:19:28.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:28.429+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.432412+0000) 2019-10-13T11:19:28.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:28.426+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.433066+0000) 2019-10-13T11:19:28.434 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:28.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.433652+0000) 2019-10-13T11:19:28.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:28.429+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.433835+0000) 2019-10-13T11:19:28.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:28.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:28.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:28.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.437330+0000) 2019-10-13T11:19:28.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:28.657+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.658417+0000) 2019-10-13T11:19:28.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:28.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.663786+0000) 2019-10-13T11:19:28.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:28.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.675226+0000) 2019-10-13T11:19:28.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:28.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.709093+0000) 2019-10-13T11:19:29.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:29.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.895172+0000) 2019-10-13T11:19:32.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:32.145+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:32.149154+0000) 2019-10-13T11:19:33.440 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:33.433+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.439206+0000) 2019-10-13T11:19:33.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:33.434+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.439914+0000) 2019-10-13T11:19:33.441 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:33.437+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.440248+0000) 2019-10-13T11:19:33.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:33.437+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.440775+0000) 2019-10-13T11:19:33.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:33.437+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.440436+0000) 2019-10-13T11:19:33.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:33.437+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.441986+0000) 2019-10-13T11:19:33.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:33.437+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.442259+0000) 2019-10-13T11:19:33.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:33.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:33.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:33.442+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.447668+0000) 2019-10-13T11:19:36.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:36.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:36.395049+0000) 2019-10-13T11:19:37.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:37.422+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:37.426122+0000) 2019-10-13T11:19:38.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:38.445+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.448413+0000) 2019-10-13T11:19:38.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:38.445+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.448972+0000) 2019-10-13T11:19:38.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:38.445+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.449858+0000) 2019-10-13T11:19:38.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:38.445+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.450754+0000) 2019-10-13T11:19:38.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:38.446+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.451072+0000) 2019-10-13T11:19:38.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:38.445+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.451133+0000) 2019-10-13T11:19:38.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:38.449+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.452233+0000) 2019-10-13T11:19:38.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:38.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 120994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:38.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:38.450+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.455616+0000) 2019-10-13T11:19:38.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:38.653+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.658598+0000) 2019-10-13T11:19:38.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:38.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.663957+0000) 2019-10-13T11:19:38.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:38.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.675399+0000) 2019-10-13T11:19:38.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:38.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.709295+0000) 2019-10-13T11:19:39.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:39.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.895412+0000) 2019-10-13T11:19:41.712 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:19:41.712 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:41.716 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:19:41.716 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:42.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:42.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:42.149370+0000) 2019-10-13T11:19:43.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:43.452+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.457790+0000) 2019-10-13T11:19:43.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:43.452+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.458453+0000) 2019-10-13T11:19:43.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:43.454+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.459137+0000) 2019-10-13T11:19:43.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:43.456+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.459760+0000) 2019-10-13T11:19:43.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:43.456+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.460056+0000) 2019-10-13T11:19:43.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:43.456+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.460549+0000) 2019-10-13T11:19:43.462 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:43.456+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.460961+0000) 2019-10-13T11:19:43.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:43.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:43.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:43.462+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.466793+0000) 2019-10-13T11:19:46.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:46.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:46.395358+0000) 2019-10-13T11:19:47.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:47.434+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:47.437311+0000) 2019-10-13T11:19:48.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:48.460+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.466730+0000) 2019-10-13T11:19:48.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:48.462+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.467196+0000) 2019-10-13T11:19:48.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:48.464+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.468438+0000) 2019-10-13T11:19:48.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:48.464+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.468536+0000) 2019-10-13T11:19:48.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:48.464+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.469231+0000) 2019-10-13T11:19:48.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:48.464+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.470025+0000) 2019-10-13T11:19:48.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:48.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.470517+0000) 2019-10-13T11:19:48.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:48.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:48.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:48.474+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.479685+0000) 2019-10-13T11:19:48.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:48.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.658775+0000) 2019-10-13T11:19:48.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:48.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.664139+0000) 2019-10-13T11:19:48.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:48.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.675571+0000) 2019-10-13T11:19:48.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:48.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.709438+0000) 2019-10-13T11:19:49.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:49.890+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.895601+0000) 2019-10-13T11:19:52.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:52.148+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:52.149575+0000) 2019-10-13T11:19:53.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:53.470+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.476963+0000) 2019-10-13T11:19:53.478 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:53.472+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.477537+0000) 2019-10-13T11:19:53.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:53.472+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.477982+0000) 2019-10-13T11:19:53.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:53.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.478478+0000) 2019-10-13T11:19:53.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:53.472+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.479007+0000) 2019-10-13T11:19:53.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:53.476+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.479476+0000) 2019-10-13T11:19:53.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:53.476+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.480136+0000) 2019-10-13T11:19:53.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:53.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:53.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:53.482+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.486680+0000) 2019-10-13T11:19:56.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:56.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:56.395573+0000) 2019-10-13T11:19:57.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:57.442+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:57.443815+0000) 2019-10-13T11:19:58.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:58.482+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.486295+0000) 2019-10-13T11:19:58.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:58.480+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.486959+0000) 2019-10-13T11:19:58.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:58.484+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.487456+0000) 2019-10-13T11:19:58.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:19:58.484+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.488050+0000) 2019-10-13T11:19:58.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:58.484+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.488737+0000) 2019-10-13T11:19:58.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:19:58.484+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.489083+0000) 2019-10-13T11:19:58.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:58.484+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.490268+0000) 2019-10-13T11:19:58.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:19:58.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:19:58.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:19:58.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.496712+0000) 2019-10-13T11:19:58.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:19:58.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.658949+0000) 2019-10-13T11:19:58.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:19:58.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.664311+0000) 2019-10-13T11:19:58.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:19:58.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.675746+0000) 2019-10-13T11:19:58.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:19:58.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.709624+0000) 2019-10-13T11:19:59.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:19:59.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.895793+0000) 2019-10-13T11:20:02.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:02.148+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:02.149801+0000) 2019-10-13T11:20:03.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:03.492+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.495435+0000) 2019-10-13T11:20:03.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:03.490+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.496870+0000) 2019-10-13T11:20:03.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:03.492+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.496945+0000) 2019-10-13T11:20:03.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:03.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.497354+0000) 2019-10-13T11:20:03.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:03.492+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.497828+0000) 2019-10-13T11:20:03.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:03.492+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.498261+0000) 2019-10-13T11:20:03.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:03.492+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.498600+0000) 2019-10-13T11:20:03.503 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:03.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:03.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:03.502+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.507498+0000) 2019-10-13T11:20:06.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:06.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:06.395764+0000) 2019-10-13T11:20:07.451 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:07.446+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:07.450269+0000) 2019-10-13T11:20:08.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:08.500+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.506008+0000) 2019-10-13T11:20:08.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:08.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.506369+0000) 2019-10-13T11:20:08.508 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:08.500+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.507103+0000) 2019-10-13T11:20:08.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:08.504+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.507445+0000) 2019-10-13T11:20:08.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:08.504+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.508371+0000) 2019-10-13T11:20:08.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:08.504+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.508793+0000) 2019-10-13T11:20:08.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:08.504+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.509557+0000) 2019-10-13T11:20:08.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:08.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:08.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:08.506+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.513253+0000) 2019-10-13T11:20:08.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:08.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.659125+0000) 2019-10-13T11:20:08.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:08.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.664444+0000) 2019-10-13T11:20:08.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:08.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.675936+0000) 2019-10-13T11:20:08.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:08.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.709808+0000) 2019-10-13T11:20:09.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:09.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.896010+0000) 2019-10-13T11:20:11.759 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:20:11.759 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:11.763 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:20:11.763 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:12.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:12.144+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:12.150004+0000) 2019-10-13T11:20:13.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:13.508+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.514924+0000) 2019-10-13T11:20:13.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:13.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.515806+0000) 2019-10-13T11:20:13.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:13.512+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.516238+0000) 2019-10-13T11:20:13.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:13.512+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.516813+0000) 2019-10-13T11:20:13.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:13.512+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.516993+0000) 2019-10-13T11:20:13.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:13.512+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.517648+0000) 2019-10-13T11:20:13.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:13.514+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.518372+0000) 2019-10-13T11:20:13.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:13.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:13.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:13.518+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.522308+0000) 2019-10-13T11:20:16.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:16.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:16.395972+0000) 2019-10-13T11:20:17.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:17.454+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:17.456815+0000) 2019-10-13T11:20:18.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:18.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.524407+0000) 2019-10-13T11:20:18.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:18.520+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.524867+0000) 2019-10-13T11:20:18.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:18.520+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.525384+0000) 2019-10-13T11:20:18.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:18.520+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.525639+0000) 2019-10-13T11:20:18.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:18.520+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.526026+0000) 2019-10-13T11:20:18.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:18.520+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.526552+0000) 2019-10-13T11:20:18.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:18.526+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.527448+0000) 2019-10-13T11:20:18.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:18.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:18.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:18.534+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.534391+0000) 2019-10-13T11:20:18.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:18.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.659328+0000) 2019-10-13T11:20:18.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:18.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.664630+0000) 2019-10-13T11:20:18.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:18.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.676118+0000) 2019-10-13T11:20:18.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:18.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.709997+0000) 2019-10-13T11:20:19.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:19.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.896230+0000) 2019-10-13T11:20:22.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:22.148+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:22.150208+0000) 2019-10-13T11:20:23.535 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:23.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.534494+0000) 2019-10-13T11:20:23.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:23.528+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.534764+0000) 2019-10-13T11:20:23.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:23.528+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.535121+0000) 2019-10-13T11:20:23.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:23.532+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.535524+0000) 2019-10-13T11:20:23.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:23.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.536285+0000) 2019-10-13T11:20:23.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:23.532+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.537091+0000) 2019-10-13T11:20:23.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:23.532+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.537644+0000) 2019-10-13T11:20:23.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:23.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:23.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:23.541+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.543632+0000) 2019-10-13T11:20:26.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:26.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:26.396167+0000) 2019-10-13T11:20:27.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:27.465+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:27.468457+0000) 2019-10-13T11:20:28.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:28.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.542842+0000) 2019-10-13T11:20:28.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:28.539+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.543669+0000) 2019-10-13T11:20:28.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:28.539+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.543957+0000) 2019-10-13T11:20:28.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:28.539+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.544288+0000) 2019-10-13T11:20:28.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:28.539+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.544776+0000) 2019-10-13T11:20:28.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:28.539+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.545000+0000) 2019-10-13T11:20:28.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:28.539+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.545374+0000) 2019-10-13T11:20:28.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:28.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:28.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:28.549+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.551307+0000) 2019-10-13T11:20:28.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:28.655+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.659474+0000) 2019-10-13T11:20:28.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:28.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.664837+0000) 2019-10-13T11:20:28.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:28.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.676306+0000) 2019-10-13T11:20:28.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:28.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.710190+0000) 2019-10-13T11:20:29.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:29.893+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.896386+0000) 2019-10-13T11:20:32.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:32.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:32.150397+0000) 2019-10-13T11:20:33.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:33.547+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.552436+0000) 2019-10-13T11:20:33.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:33.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.553684+0000) 2019-10-13T11:20:33.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:33.547+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.553783+0000) 2019-10-13T11:20:33.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:33.547+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.554638+0000) 2019-10-13T11:20:33.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:33.547+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.554088+0000) 2019-10-13T11:20:33.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:33.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.555297+0000) 2019-10-13T11:20:33.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:33.551+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.555854+0000) 2019-10-13T11:20:33.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:33.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:33.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:33.557+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.561118+0000) 2019-10-13T11:20:36.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:36.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:36.396323+0000) 2019-10-13T11:20:37.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:37.473+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:37.474906+0000) 2019-10-13T11:20:38.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:38.555+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.562282+0000) 2019-10-13T11:20:38.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:38.555+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.562714+0000) 2019-10-13T11:20:38.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:38.561+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.563322+0000) 2019-10-13T11:20:38.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:38.559+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.563749+0000) 2019-10-13T11:20:38.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:38.559+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.563935+0000) 2019-10-13T11:20:38.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:38.559+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.564284+0000) 2019-10-13T11:20:38.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:38.559+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.565977+0000) 2019-10-13T11:20:38.569 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:38.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121383 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:38.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:38.569+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.573753+0000) 2019-10-13T11:20:38.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:38.655+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.659653+0000) 2019-10-13T11:20:38.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:38.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.665018+0000) 2019-10-13T11:20:38.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:38.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.676481+0000) 2019-10-13T11:20:38.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:38.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.710402+0000) 2019-10-13T11:20:39.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:39.893+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.896571+0000) 2019-10-13T11:20:41.898 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:20:41.898 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:41.901 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:20:41.902 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:42.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:42.143+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:42.150605+0000) 2019-10-13T11:20:43.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:43.569+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.571839+0000) 2019-10-13T11:20:43.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:43.567+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.572120+0000) 2019-10-13T11:20:43.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:43.567+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.572500+0000) 2019-10-13T11:20:43.574 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:43.567+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.573291+0000) 2019-10-13T11:20:43.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:43.567+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.573787+0000) 2019-10-13T11:20:43.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:43.567+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.574106+0000) 2019-10-13T11:20:43.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:43.567+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.574575+0000) 2019-10-13T11:20:43.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:43.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:43.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:43.577+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.579796+0000) 2019-10-13T11:20:46.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:46.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:46.396551+0000) 2019-10-13T11:20:47.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:47.481+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:47.485158+0000) 2019-10-13T11:20:48.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:48.575+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.581174+0000) 2019-10-13T11:20:48.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:48.575+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.580932+0000) 2019-10-13T11:20:48.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:48.575+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.581956+0000) 2019-10-13T11:20:48.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:48.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.582397+0000) 2019-10-13T11:20:48.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:48.575+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.581666+0000) 2019-10-13T11:20:48.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:48.581+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.583163+0000) 2019-10-13T11:20:48.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:48.579+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.584429+0000) 2019-10-13T11:20:48.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:48.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:48.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:48.589+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.590708+0000) 2019-10-13T11:20:48.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:48.659+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.659835+0000) 2019-10-13T11:20:48.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:48.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.665192+0000) 2019-10-13T11:20:48.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:48.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.676659+0000) 2019-10-13T11:20:48.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:48.703+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.710594+0000) 2019-10-13T11:20:49.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:49.893+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.896793+0000) 2019-10-13T11:20:52.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:52.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:52.150810+0000) 2019-10-13T11:20:53.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:53.583+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.589609+0000) 2019-10-13T11:20:53.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:53.589+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.590122+0000) 2019-10-13T11:20:53.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:53.583+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.590990+0000) 2019-10-13T11:20:53.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:53.583+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.591176+0000) 2019-10-13T11:20:53.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:53.587+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.591829+0000) 2019-10-13T11:20:53.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:53.587+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.592279+0000) 2019-10-13T11:20:53.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:53.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.593684+0000) 2019-10-13T11:20:53.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:53.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:53.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:53.597+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.598454+0000) 2019-10-13T11:20:56.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:56.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:56.396759+0000) 2019-10-13T11:20:57.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:57.489+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:57.491636+0000) 2019-10-13T11:20:58.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:20:58.599+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.599399+0000) 2019-10-13T11:20:58.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:58.599+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.599943+0000) 2019-10-13T11:20:58.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:58.599+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.600478+0000) 2019-10-13T11:20:58.602 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:58.597+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.601804+0000) 2019-10-13T11:20:58.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:20:58.599+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.602337+0000) 2019-10-13T11:20:58.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:58.599+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.602214+0000) 2019-10-13T11:20:58.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:58.599+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.602584+0000) 2019-10-13T11:20:58.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:20:58.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:20:58.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:20:58.605+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.608034+0000) 2019-10-13T11:20:58.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:20:58.659+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.660039+0000) 2019-10-13T11:20:58.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:20:58.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.665371+0000) 2019-10-13T11:20:58.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:20:58.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.676834+0000) 2019-10-13T11:20:58.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:20:58.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.710776+0000) 2019-10-13T11:20:59.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:20:59.893+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.896985+0000) 2019-10-13T11:21:02.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:02.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:02.151009+0000) 2019-10-13T11:21:03.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:03.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.607855+0000) 2019-10-13T11:21:03.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:03.607+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.608559+0000) 2019-10-13T11:21:03.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:03.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.609427+0000) 2019-10-13T11:21:03.611 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:03.609+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.610534+0000) 2019-10-13T11:21:03.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:03.607+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.610746+0000) 2019-10-13T11:21:03.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:03.607+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.610916+0000) 2019-10-13T11:21:03.612 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:03.607+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.610953+0000) 2019-10-13T11:21:03.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:03.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:03.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:03.613+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.615985+0000) 2019-10-13T11:21:06.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:06.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:06.396978+0000) 2019-10-13T11:21:07.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:07.497+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:07.498006+0000) 2019-10-13T11:21:08.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:08.615+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.617950+0000) 2019-10-13T11:21:08.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:08.617+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.618836+0000) 2019-10-13T11:21:08.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:08.619+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.619943+0000) 2019-10-13T11:21:08.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:08.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.620439+0000) 2019-10-13T11:21:08.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:08.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.621033+0000) 2019-10-13T11:21:08.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:08.619+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.621644+0000) 2019-10-13T11:21:08.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:08.619+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.622293+0000) 2019-10-13T11:21:08.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:08.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:08.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:08.621+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.625880+0000) 2019-10-13T11:21:08.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:08.659+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.660255+0000) 2019-10-13T11:21:08.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:08.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.665570+0000) 2019-10-13T11:21:08.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:08.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.677012+0000) 2019-10-13T11:21:08.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:08.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.710955+0000) 2019-10-13T11:21:09.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:09.893+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.897179+0000) 2019-10-13T11:21:11.987 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:21:11.987 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:11.991 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:21:11.991 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:12.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:12.146+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:12.151265+0000) 2019-10-13T11:21:13.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:13.626+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.628308+0000) 2019-10-13T11:21:13.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:13.626+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.629156+0000) 2019-10-13T11:21:13.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:13.625+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.629585+0000) 2019-10-13T11:21:13.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:13.626+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.629838+0000) 2019-10-13T11:21:13.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:13.626+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.630136+0000) 2019-10-13T11:21:13.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:13.626+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.630407+0000) 2019-10-13T11:21:13.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:13.630+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.631287+0000) 2019-10-13T11:21:13.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:13.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:13.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:13.633+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.637514+0000) 2019-10-13T11:21:16.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:16.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:16.397199+0000) 2019-10-13T11:21:17.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:17.501+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:17.504567+0000) 2019-10-13T11:21:18.639 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:18.634+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.637914+0000) 2019-10-13T11:21:18.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:18.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.638172+0000) 2019-10-13T11:21:18.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:18.634+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.638888+0000) 2019-10-13T11:21:18.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:18.634+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.639142+0000) 2019-10-13T11:21:18.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:18.638+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.639924+0000) 2019-10-13T11:21:18.641 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:18.636+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.640348+0000) 2019-10-13T11:21:18.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:18.638+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.640688+0000) 2019-10-13T11:21:18.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:18.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121643 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:18.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:18.640+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.646019+0000) 2019-10-13T11:21:18.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:18.658+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.660495+0000) 2019-10-13T11:21:18.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:18.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.665743+0000) 2019-10-13T11:21:18.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:18.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.677223+0000) 2019-10-13T11:21:18.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:18.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.711168+0000) 2019-10-13T11:21:19.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:19.892+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.897370+0000) 2019-10-13T11:21:22.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:22.150+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:22.151537+0000) 2019-10-13T11:21:23.647 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:23.642+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.646612+0000) 2019-10-13T11:21:23.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:23.646+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.647690+0000) 2019-10-13T11:21:23.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:23.646+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.648301+0000) 2019-10-13T11:21:23.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:23.644+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.648811+0000) 2019-10-13T11:21:23.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:23.646+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.648937+0000) 2019-10-13T11:21:23.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:23.646+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.649283+0000) 2019-10-13T11:21:23.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:23.646+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.650125+0000) 2019-10-13T11:21:23.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:23.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:23.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:23.648+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.653777+0000) 2019-10-13T11:21:26.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:26.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:26.397394+0000) 2019-10-13T11:21:27.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:27.508+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:27.510969+0000) 2019-10-13T11:21:28.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:28.654+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.655981+0000) 2019-10-13T11:21:28.657 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:28.654+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.656516+0000) 2019-10-13T11:21:28.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:28.654+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.657114+0000) 2019-10-13T11:21:28.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:28.654+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.657358+0000) 2019-10-13T11:21:28.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:28.652+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.657882+0000) 2019-10-13T11:21:28.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:28.654+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.658881+0000) 2019-10-13T11:21:28.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:28.654+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.658135+0000) 2019-10-13T11:21:28.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:28.658+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.660655+0000) 2019-10-13T11:21:28.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:28.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121708 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:28.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:28.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.665880+0000) 2019-10-13T11:21:28.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:28.664+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.666649+0000) 2019-10-13T11:21:28.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:28.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.677395+0000) 2019-10-13T11:21:28.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:28.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.711397+0000) 2019-10-13T11:21:29.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:29.896+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.897560+0000) 2019-10-13T11:21:32.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:32.150+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:32.151736+0000) 2019-10-13T11:21:33.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:33.660+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.665981+0000) 2019-10-13T11:21:33.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:33.662+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.667130+0000) 2019-10-13T11:21:33.668 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:33.666+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.667730+0000) 2019-10-13T11:21:33.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:33.666+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.667815+0000) 2019-10-13T11:21:33.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:33.666+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.668704+0000) 2019-10-13T11:21:33.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:33.666+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.668954+0000) 2019-10-13T11:21:33.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:33.666+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.669768+0000) 2019-10-13T11:21:33.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:33.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:33.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:33.668+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.673158+0000) 2019-10-13T11:21:36.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:36.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:36.397630+0000) 2019-10-13T11:21:37.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:37.516+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:37.517604+0000) 2019-10-13T11:21:38.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:38.658+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.660853+0000) 2019-10-13T11:21:38.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:38.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.666035+0000) 2019-10-13T11:21:38.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:38.674+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.675560+0000) 2019-10-13T11:21:38.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:38.674+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.675914+0000) 2019-10-13T11:21:38.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:38.674+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.676632+0000) 2019-10-13T11:21:38.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:38.674+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.676957+0000) 2019-10-13T11:21:38.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:38.674+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.677293+0000) 2019-10-13T11:21:38.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:38.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.677555+0000) 2019-10-13T11:21:38.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:38.676+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.678454+0000) 2019-10-13T11:21:38.679 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:38.674+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.679037+0000) 2019-10-13T11:21:38.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:38.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:38.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:38.680+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.684743+0000) 2019-10-13T11:21:38.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:38.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.711531+0000) 2019-10-13T11:21:39.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:39.892+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.897748+0000) 2019-10-13T11:21:42.099 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:21:42.099 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:42.103 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:21:42.103 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:42.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:42.150+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:42.151942+0000) 2019-10-13T11:21:43.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:43.680+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.684598+0000) 2019-10-13T11:21:43.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:43.682+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.685329+0000) 2019-10-13T11:21:43.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:43.682+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.685726+0000) 2019-10-13T11:21:43.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:43.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.686558+0000) 2019-10-13T11:21:43.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:43.682+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.686013+0000) 2019-10-13T11:21:43.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:43.682+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.686934+0000) 2019-10-13T11:21:43.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:43.682+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.686939+0000) 2019-10-13T11:21:43.691 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:43.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:43.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:43.688+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.693753+0000) 2019-10-13T11:21:46.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:46.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:46.397828+0000) 2019-10-13T11:21:47.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:47.520+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:47.524078+0000) 2019-10-13T11:21:48.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:48.658+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.661078+0000) 2019-10-13T11:21:48.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:48.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.666236+0000) 2019-10-13T11:21:48.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:48.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.677748+0000) 2019-10-13T11:21:48.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:48.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.693403+0000) 2019-10-13T11:21:48.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:48.688+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.694026+0000) 2019-10-13T11:21:48.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:48.690+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.694718+0000) 2019-10-13T11:21:48.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:48.690+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.694840+0000) 2019-10-13T11:21:48.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:48.690+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.695218+0000) 2019-10-13T11:21:48.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:48.694+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.695801+0000) 2019-10-13T11:21:48.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:48.694+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.696844+0000) 2019-10-13T11:21:48.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:48.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:48.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:48.700+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.702900+0000) 2019-10-13T11:21:48.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:48.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.711668+0000) 2019-10-13T11:21:49.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:49.896+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.897969+0000) 2019-10-13T11:21:52.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:52.150+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:52.152147+0000) 2019-10-13T11:21:53.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:53.697+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.702999+0000) 2019-10-13T11:21:53.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:53.701+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.703555+0000) 2019-10-13T11:21:53.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:53.701+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.704121+0000) 2019-10-13T11:21:53.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:53.701+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.704606+0000) 2019-10-13T11:21:53.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:53.700+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.704935+0000) 2019-10-13T11:21:53.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:53.701+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.705606+0000) 2019-10-13T11:21:53.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:53.701+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.705344+0000) 2019-10-13T11:21:53.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:53.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:53.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:53.712+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.714744+0000) 2019-10-13T11:21:56.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:56.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:56.398019+0000) 2019-10-13T11:21:57.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:57.532+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:57.535128+0000) 2019-10-13T11:21:58.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:58.657+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.661273+0000) 2019-10-13T11:21:58.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:58.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.666467+0000) 2019-10-13T11:21:58.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:58.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.677948+0000) 2019-10-13T11:21:58.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:58.708+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.712003+0000) 2019-10-13T11:21:58.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:21:58.709+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.712175+0000) 2019-10-13T11:21:58.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:21:58.709+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.713391+0000) 2019-10-13T11:21:58.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:21:58.709+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.713932+0000) 2019-10-13T11:21:58.715 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:21:58.709+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.714775+0000) 2019-10-13T11:21:58.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:21:58.709+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.715151+0000) 2019-10-13T11:21:58.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:21:58.713+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.715873+0000) 2019-10-13T11:21:58.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:21:58.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:21:58.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:21:58.720+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.722314+0000) 2019-10-13T11:21:59.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:21:59.896+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.898191+0000) 2019-10-13T11:22:02.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:02.149+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:02.152347+0000) 2019-10-13T11:22:03.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:03.717+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.721647+0000) 2019-10-13T11:22:03.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:03.717+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.721996+0000) 2019-10-13T11:22:03.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:03.717+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.722919+0000) 2019-10-13T11:22:03.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:03.717+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.723116+0000) 2019-10-13T11:22:03.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:03.720+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.723714+0000) 2019-10-13T11:22:03.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:03.721+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.724606+0000) 2019-10-13T11:22:03.726 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:03.721+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.725411+0000) 2019-10-13T11:22:03.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:03.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:03.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:03.728+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.730521+0000) 2019-10-13T11:22:05.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:05.457+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.462493+0000) 2019-10-13T11:22:06.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:06.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:06.398228+0000) 2019-10-13T11:22:07.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:07.540+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:07.541619+0000) 2019-10-13T11:22:08.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:08.657+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.661472+0000) 2019-10-13T11:22:08.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:08.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.666661+0000) 2019-10-13T11:22:08.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:08.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.678170+0000) 2019-10-13T11:22:08.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:08.725+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.730461+0000) 2019-10-13T11:22:08.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:08.725+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.730944+0000) 2019-10-13T11:22:08.732 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:08.729+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.731907+0000) 2019-10-13T11:22:08.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:08.729+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.732609+0000) 2019-10-13T11:22:08.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:08.729+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.732634+0000) 2019-10-13T11:22:08.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:08.729+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.732960+0000) 2019-10-13T11:22:08.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:08.732+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.734854+0000) 2019-10-13T11:22:08.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:08.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 121968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:08.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:08.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.740376+0000) 2019-10-13T11:22:09.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:09.896+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.898387+0000) 2019-10-13T11:22:12.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:12.149+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:12.152550+0000) 2019-10-13T11:22:12.161 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:22:12.161 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:12.165 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:22:12.165 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:13.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:13.735+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.739937+0000) 2019-10-13T11:22:13.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:13.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.740623+0000) 2019-10-13T11:22:13.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:13.737+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.741191+0000) 2019-10-13T11:22:13.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:13.737+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.741089+0000) 2019-10-13T11:22:13.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:13.737+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.741678+0000) 2019-10-13T11:22:13.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:13.737+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.741838+0000) 2019-10-13T11:22:13.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:13.737+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.742421+0000) 2019-10-13T11:22:13.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:13.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:13.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:13.739+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.745607+0000) 2019-10-13T11:22:15.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:15.461+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.462687+0000) 2019-10-13T11:22:16.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:16.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:16.398416+0000) 2019-10-13T11:22:17.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:17.543+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:17.548081+0000) 2019-10-13T11:22:18.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:18.657+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.661668+0000) 2019-10-13T11:22:18.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:18.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.666855+0000) 2019-10-13T11:22:18.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:18.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.678373+0000) 2019-10-13T11:22:18.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:18.743+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.748266+0000) 2019-10-13T11:22:18.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:18.745+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.748775+0000) 2019-10-13T11:22:18.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:18.745+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.748920+0000) 2019-10-13T11:22:18.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:18.745+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.749252+0000) 2019-10-13T11:22:18.751 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:18.745+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.750154+0000) 2019-10-13T11:22:18.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:18.745+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.750701+0000) 2019-10-13T11:22:18.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:18.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.751611+0000) 2019-10-13T11:22:18.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:18.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:18.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:18.755+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.758543+0000) 2019-10-13T11:22:19.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:19.895+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.898588+0000) 2019-10-13T11:22:22.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:22.149+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:22.152769+0000) 2019-10-13T11:22:23.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:23.755+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.758235+0000) 2019-10-13T11:22:23.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:23.753+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.758682+0000) 2019-10-13T11:22:23.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:23.757+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.759337+0000) 2019-10-13T11:22:23.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:23.757+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.759568+0000) 2019-10-13T11:22:23.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:23.757+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.760257+0000) 2019-10-13T11:22:23.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:23.757+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.760495+0000) 2019-10-13T11:22:23.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:23.757+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.761225+0000) 2019-10-13T11:22:23.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:23.759+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.763715+0000) 2019-10-13T11:22:23.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:23.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:25.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:25.457+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.462887+0000) 2019-10-13T11:22:26.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:26.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:26.398603+0000) 2019-10-13T11:22:27.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:27.555+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:27.557427+0000) 2019-10-13T11:22:28.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:28.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.661887+0000) 2019-10-13T11:22:28.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:28.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.667067+0000) 2019-10-13T11:22:28.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:28.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.678567+0000) 2019-10-13T11:22:28.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:28.761+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.767278+0000) 2019-10-13T11:22:28.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:28.765+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.768037+0000) 2019-10-13T11:22:28.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:28.765+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.768475+0000) 2019-10-13T11:22:28.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:28.765+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.768983+0000) 2019-10-13T11:22:28.771 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:28.763+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.770090+0000) 2019-10-13T11:22:28.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:28.765+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.770376+0000) 2019-10-13T11:22:28.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:28.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.770512+0000) 2019-10-13T11:22:28.774 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:28.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:28.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:28.775+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.779490+0000) 2019-10-13T11:22:29.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:29.895+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.898778+0000) 2019-10-13T11:22:32.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:32.149+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:32.153012+0000) 2019-10-13T11:22:33.777 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:33.773+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.776823+0000) 2019-10-13T11:22:33.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:33.773+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.777309+0000) 2019-10-13T11:22:33.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:33.773+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.778006+0000) 2019-10-13T11:22:33.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:33.773+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.778413+0000) 2019-10-13T11:22:33.780 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:33.775+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.779369+0000) 2019-10-13T11:22:33.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:33.777+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.779756+0000) 2019-10-13T11:22:33.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:33.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.780017+0000) 2019-10-13T11:22:33.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:33.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:33.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:33.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.785351+0000) 2019-10-13T11:22:35.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:35.461+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.463111+0000) 2019-10-13T11:22:36.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:36.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:36.398790+0000) 2019-10-13T11:22:37.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:37.563+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:37.563948+0000) 2019-10-13T11:22:38.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:38.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.662086+0000) 2019-10-13T11:22:38.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:38.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.667296+0000) 2019-10-13T11:22:38.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:38.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.678761+0000) 2019-10-13T11:22:38.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:38.784+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.787368+0000) 2019-10-13T11:22:38.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:38.784+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.787948+0000) 2019-10-13T11:22:38.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:38.784+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.789003+0000) 2019-10-13T11:22:38.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:38.784+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.789468+0000) 2019-10-13T11:22:38.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:38.784+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.789914+0000) 2019-10-13T11:22:38.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:38.784+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.790274+0000) 2019-10-13T11:22:38.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:38.787+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.790213+0000) 2019-10-13T11:22:38.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:38.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:38.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:38.791+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.795084+0000) 2019-10-13T11:22:39.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:39.895+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.898970+0000) 2019-10-13T11:22:42.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:42.148+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:42.153260+0000) 2019-10-13T11:22:42.220 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:22:42.220 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:42.224 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:22:42.224 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:43.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:43.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.795992+0000) 2019-10-13T11:22:43.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:43.792+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.796384+0000) 2019-10-13T11:22:43.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:43.792+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.797375+0000) 2019-10-13T11:22:43.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:43.792+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.797918+0000) 2019-10-13T11:22:43.799 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:43.795+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.798730+0000) 2019-10-13T11:22:43.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:43.796+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.799622+0000) 2019-10-13T11:22:43.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:43.796+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.800227+0000) 2019-10-13T11:22:43.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:43.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:43.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:43.799+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.805554+0000) 2019-10-13T11:22:45.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:45.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.463346+0000) 2019-10-13T11:22:46.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:46.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:46.398982+0000) 2019-10-13T11:22:47.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:47.567+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:47.570450+0000) 2019-10-13T11:22:48.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:48.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.662326+0000) 2019-10-13T11:22:48.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:48.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.667516+0000) 2019-10-13T11:22:48.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:48.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.678953+0000) 2019-10-13T11:22:48.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:48.800+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.806004+0000) 2019-10-13T11:22:48.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:48.800+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.806566+0000) 2019-10-13T11:22:48.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:48.800+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.806854+0000) 2019-10-13T11:22:48.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:48.804+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.807980+0000) 2019-10-13T11:22:48.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:48.804+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.808025+0000) 2019-10-13T11:22:48.809 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:48.803+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.808992+0000) 2019-10-13T11:22:48.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:48.804+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.809252+0000) 2019-10-13T11:22:48.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:48.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:48.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:48.811+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.815790+0000) 2019-10-13T11:22:49.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:49.895+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.899214+0000) 2019-10-13T11:22:52.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:52.152+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:52.153465+0000) 2019-10-13T11:22:53.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:53.808+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.814222+0000) 2019-10-13T11:22:53.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:53.808+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.814619+0000) 2019-10-13T11:22:53.816 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:53.812+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.815655+0000) 2019-10-13T11:22:53.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:53.812+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.816292+0000) 2019-10-13T11:22:53.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:53.812+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.817431+0000) 2019-10-13T11:22:53.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:53.811+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.817609+0000) 2019-10-13T11:22:53.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:53.812+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.817846+0000) 2019-10-13T11:22:53.821 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:53.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:53.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:53.815+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.821526+0000) 2019-10-13T11:22:55.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:55.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.463524+0000) 2019-10-13T11:22:56.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:56.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:56.399178+0000) 2019-10-13T11:22:57.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:57.571+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:57.577111+0000) 2019-10-13T11:22:58.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:58.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.662528+0000) 2019-10-13T11:22:58.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:58.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.667751+0000) 2019-10-13T11:22:58.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:58.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.679187+0000) 2019-10-13T11:22:58.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:22:58.820+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.824377+0000) 2019-10-13T11:22:58.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:22:58.820+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.825181+0000) 2019-10-13T11:22:58.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:22:58.820+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.825507+0000) 2019-10-13T11:22:58.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:58.823+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.826416+0000) 2019-10-13T11:22:58.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:22:58.820+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.825862+0000) 2019-10-13T11:22:58.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:22:58.820+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.827023+0000) 2019-10-13T11:22:58.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:22:58.820+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.826540+0000) 2019-10-13T11:22:58.830 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:22:58.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:22:58.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:22:58.831+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.835328+0000) 2019-10-13T11:22:59.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:22:59.895+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.899411+0000) 2019-10-13T11:23:02.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:02.148+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:02.153666+0000) 2019-10-13T11:23:03.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:03.828+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.832926+0000) 2019-10-13T11:23:03.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:03.828+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.833798+0000) 2019-10-13T11:23:03.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:03.828+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.834402+0000) 2019-10-13T11:23:03.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:03.828+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.834582+0000) 2019-10-13T11:23:03.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:03.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.835346+0000) 2019-10-13T11:23:03.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:03.828+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.834930+0000) 2019-10-13T11:23:03.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:03.832+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.835727+0000) 2019-10-13T11:23:03.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:03.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:03.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:03.835+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.839525+0000) 2019-10-13T11:23:05.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:05.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.463749+0000) 2019-10-13T11:23:06.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:06.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:06.399420+0000) 2019-10-13T11:23:07.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:07.582+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:07.583716+0000) 2019-10-13T11:23:08.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:08.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.662759+0000) 2019-10-13T11:23:08.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:08.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.667957+0000) 2019-10-13T11:23:08.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:08.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.679417+0000) 2019-10-13T11:23:08.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:08.836+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.841734+0000) 2019-10-13T11:23:08.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:08.836+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.842029+0000) 2019-10-13T11:23:08.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:08.838+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.842894+0000) 2019-10-13T11:23:08.844 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:08.836+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.843138+0000) 2019-10-13T11:23:08.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:08.840+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.843596+0000) 2019-10-13T11:23:08.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:08.840+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.844227+0000) 2019-10-13T11:23:08.846 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:08.840+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.844922+0000) 2019-10-13T11:23:08.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:08.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:08.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:08.846+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.852272+0000) 2019-10-13T11:23:09.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:09.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.899623+0000) 2019-10-13T11:23:12.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:12.152+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:12.153878+0000) 2019-10-13T11:23:12.327 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:23:12.327 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:12.331 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:23:12.331 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:13.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:13.844+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.851124+0000) 2019-10-13T11:23:13.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:13.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.851793+0000) 2019-10-13T11:23:13.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:13.848+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.852152+0000) 2019-10-13T11:23:13.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:13.848+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.852369+0000) 2019-10-13T11:23:13.854 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:13.848+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.853060+0000) 2019-10-13T11:23:13.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:13.848+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.853982+0000) 2019-10-13T11:23:13.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:13.848+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.854317+0000) 2019-10-13T11:23:13.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:13.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:13.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:13.858+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.862636+0000) 2019-10-13T11:23:15.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:15.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.463974+0000) 2019-10-13T11:23:16.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:16.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:16.399611+0000) 2019-10-13T11:23:17.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:17.586+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:17.590018+0000) 2019-10-13T11:23:18.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:18.656+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.662979+0000) 2019-10-13T11:23:18.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:18.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.668159+0000) 2019-10-13T11:23:18.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:18.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.679616+0000) 2019-10-13T11:23:18.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:18.856+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.860001+0000) 2019-10-13T11:23:18.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:18.856+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.860762+0000) 2019-10-13T11:23:18.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:18.856+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.861140+0000) 2019-10-13T11:23:18.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:18.856+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.861714+0000) 2019-10-13T11:23:18.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:18.854+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.861977+0000) 2019-10-13T11:23:18.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:18.856+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.862691+0000) 2019-10-13T11:23:18.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:18.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.863215+0000) 2019-10-13T11:23:18.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:18.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:18.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:18.866+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.870499+0000) 2019-10-13T11:23:19.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:19.894+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.899829+0000) 2019-10-13T11:23:22.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:22.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:22.154079+0000) 2019-10-13T11:23:23.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:23.863+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.869245+0000) 2019-10-13T11:23:23.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:23.863+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.870048+0000) 2019-10-13T11:23:23.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:23.866+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.870310+0000) 2019-10-13T11:23:23.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:23.863+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.870699+0000) 2019-10-13T11:23:23.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:23.863+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.871056+0000) 2019-10-13T11:23:23.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:23.867+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.871561+0000) 2019-10-13T11:23:23.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:23.867+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.871818+0000) 2019-10-13T11:23:23.876 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:23.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:23.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:23.874+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.878814+0000) 2019-10-13T11:23:25.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:25.463+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.464173+0000) 2019-10-13T11:23:26.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:26.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:26.399819+0000) 2019-10-13T11:23:27.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:27.594+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:27.596363+0000) 2019-10-13T11:23:28.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:28.655+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.663222+0000) 2019-10-13T11:23:28.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:28.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.668399+0000) 2019-10-13T11:23:28.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:28.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.679818+0000) 2019-10-13T11:23:28.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:28.871+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.878358+0000) 2019-10-13T11:23:28.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:28.871+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.879065+0000) 2019-10-13T11:23:28.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:28.875+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.879964+0000) 2019-10-13T11:23:28.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:28.875+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.880130+0000) 2019-10-13T11:23:28.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:28.874+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.881045+0000) 2019-10-13T11:23:28.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:28.875+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.881045+0000) 2019-10-13T11:23:28.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:28.875+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.881647+0000) 2019-10-13T11:23:28.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:28.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:28.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:28.882+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.886764+0000) 2019-10-13T11:23:29.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:29.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.900059+0000) 2019-10-13T11:23:32.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:32.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:32.154310+0000) 2019-10-13T11:23:33.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:33.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.888984+0000) 2019-10-13T11:23:33.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:33.883+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.889523+0000) 2019-10-13T11:23:33.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:33.883+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.889912+0000) 2019-10-13T11:23:33.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:33.883+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.890258+0000) 2019-10-13T11:23:33.891 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:33.883+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.891067+0000) 2019-10-13T11:23:33.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:33.883+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.890814+0000) 2019-10-13T11:23:33.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:33.887+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.891489+0000) 2019-10-13T11:23:33.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:33.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:33.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:33.894+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.898304+0000) 2019-10-13T11:23:35.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:35.463+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.464379+0000) 2019-10-13T11:23:36.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:36.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:36.400013+0000) 2019-10-13T11:23:37.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:37.598+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:37.602969+0000) 2019-10-13T11:23:38.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:38.659+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.663417+0000) 2019-10-13T11:23:38.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:38.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.668596+0000) 2019-10-13T11:23:38.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:38.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.680015+0000) 2019-10-13T11:23:38.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:38.891+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.898572+0000) 2019-10-13T11:23:38.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:38.891+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.899065+0000) 2019-10-13T11:23:38.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:38.895+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.899634+0000) 2019-10-13T11:23:38.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:38.895+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.900251+0000) 2019-10-13T11:23:38.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:38.894+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.900379+0000) 2019-10-13T11:23:38.903 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:38.895+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.900847+0000) 2019-10-13T11:23:38.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:38.895+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.901408+0000) 2019-10-13T11:23:38.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:38.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:38.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:38.902+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.907352+0000) 2019-10-13T11:23:39.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:39.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.900269+0000) 2019-10-13T11:23:42.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:42.147+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:42.154467+0000) 2019-10-13T11:23:42.478 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:23:42.478 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:42.481 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:23:42.481 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:43.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:43.903+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.908465+0000) 2019-10-13T11:23:43.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:43.903+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.908365+0000) 2019-10-13T11:23:43.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:43.903+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.909136+0000) 2019-10-13T11:23:43.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:43.903+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.909565+0000) 2019-10-13T11:23:43.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:43.903+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.909828+0000) 2019-10-13T11:23:43.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:43.903+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.910689+0000) 2019-10-13T11:23:43.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:43.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.911089+0000) 2019-10-13T11:23:43.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:43.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:43.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:43.910+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.917887+0000) 2019-10-13T11:23:45.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:45.463+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.464583+0000) 2019-10-13T11:23:46.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:46.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:46.400238+0000) 2019-10-13T11:23:47.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:47.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:47.614548+0000) 2019-10-13T11:23:48.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:48.663+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.663618+0000) 2019-10-13T11:23:48.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:48.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.668827+0000) 2019-10-13T11:23:48.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:48.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.680258+0000) 2019-10-13T11:23:48.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:48.911+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.917372+0000) 2019-10-13T11:23:48.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:48.911+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.918210+0000) 2019-10-13T11:23:48.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:48.914+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.918866+0000) 2019-10-13T11:23:48.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:48.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.919282+0000) 2019-10-13T11:23:48.920 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:48.915+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.919404+0000) 2019-10-13T11:23:48.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:48.915+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.919850+0000) 2019-10-13T11:23:48.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:48.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.920875+0000) 2019-10-13T11:23:48.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:48.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:48.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:48.922+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.926726+0000) 2019-10-13T11:23:49.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:49.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.900446+0000) 2019-10-13T11:23:52.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:52.151+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:52.154670+0000) 2019-10-13T11:23:53.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:53.923+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.926796+0000) 2019-10-13T11:23:53.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:53.927+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.927702+0000) 2019-10-13T11:23:53.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:53.927+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.928194+0000) 2019-10-13T11:23:53.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:53.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.928447+0000) 2019-10-13T11:23:53.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:53.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.928589+0000) 2019-10-13T11:23:53.930 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:53.927+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.929008+0000) 2019-10-13T11:23:53.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:53.927+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.929636+0000) 2019-10-13T11:23:53.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:53.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:53.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:53.934+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.935738+0000) 2019-10-13T11:23:55.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:55.463+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.464735+0000) 2019-10-13T11:23:56.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:56.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:56.400445+0000) 2019-10-13T11:23:57.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:57.618+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:57.622069+0000) 2019-10-13T11:23:58.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:58.663+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.663815+0000) 2019-10-13T11:23:58.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:58.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.669043+0000) 2019-10-13T11:23:58.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:58.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.680468+0000) 2019-10-13T11:23:58.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:23:58.935+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.935654+0000) 2019-10-13T11:23:58.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:23:58.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.937037+0000) 2019-10-13T11:23:58.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:23:58.935+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.937364+0000) 2019-10-13T11:23:58.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:23:58.935+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.938105+0000) 2019-10-13T11:23:58.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:23:58.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.938730+0000) 2019-10-13T11:23:58.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:23:58.939+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.939762+0000) 2019-10-13T11:23:58.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:58.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.940199+0000) 2019-10-13T11:23:58.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:23:58.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:23:58.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:23:58.942+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.944191+0000) 2019-10-13T11:23:59.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:23:59.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.900652+0000) 2019-10-13T11:24:02.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:02.151+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:02.154901+0000) 2019-10-13T11:24:03.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:03.942+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.945221+0000) 2019-10-13T11:24:03.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:03.942+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.945431+0000) 2019-10-13T11:24:03.947 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:03.942+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.946214+0000) 2019-10-13T11:24:03.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:03.945+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.947201+0000) 2019-10-13T11:24:03.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:03.946+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.947479+0000) 2019-10-13T11:24:03.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:03.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.948635+0000) 2019-10-13T11:24:03.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:03.946+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.948682+0000) 2019-10-13T11:24:03.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:03.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:03.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:03.953+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.954165+0000) 2019-10-13T11:24:05.466 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:05.462+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.464964+0000) 2019-10-13T11:24:06.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:06.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:06.400635+0000) 2019-10-13T11:24:07.630 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:07.625+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:07.628787+0000) 2019-10-13T11:24:08.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:08.662+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.664038+0000) 2019-10-13T11:24:08.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:08.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.669240+0000) 2019-10-13T11:24:08.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:08.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.680665+0000) 2019-10-13T11:24:08.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:08.950+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.954636+0000) 2019-10-13T11:24:08.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:08.954+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.956751+0000) 2019-10-13T11:24:08.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:08.954+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.956789+0000) 2019-10-13T11:24:08.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:08.954+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.957518+0000) 2019-10-13T11:24:08.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:08.954+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.958289+0000) 2019-10-13T11:24:08.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:08.954+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.958435+0000) 2019-10-13T11:24:08.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:08.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.958117+0000) 2019-10-13T11:24:08.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:08.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122759 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:08.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:08.961+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.963869+0000) 2019-10-13T11:24:09.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:09.897+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.900855+0000) 2019-10-13T11:24:12.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:12.150+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:12.155101+0000) 2019-10-13T11:24:12.616 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:24:12.616 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:12.620 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:24:12.620 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:13.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:13.962+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.963490+0000) 2019-10-13T11:24:13.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:13.962+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.964411+0000) 2019-10-13T11:24:13.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:13.962+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.965211+0000) 2019-10-13T11:24:13.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:13.962+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.965427+0000) 2019-10-13T11:24:13.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:13.961+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.965214+0000) 2019-10-13T11:24:13.967 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:13.962+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.966295+0000) 2019-10-13T11:24:13.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:13.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.966768+0000) 2019-10-13T11:24:13.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:13.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:13.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:13.969+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.973111+0000) 2019-10-13T11:24:15.466 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:15.462+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.465163+0000) 2019-10-13T11:24:16.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:16.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:16.400831+0000) 2019-10-13T11:24:17.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:17.633+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:17.635161+0000) 2019-10-13T11:24:18.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:18.662+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.664249+0000) 2019-10-13T11:24:18.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:18.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.669438+0000) 2019-10-13T11:24:18.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:18.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.680857+0000) 2019-10-13T11:24:18.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:18.970+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.973147+0000) 2019-10-13T11:24:18.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:18.970+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.973374+0000) 2019-10-13T11:24:18.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:18.970+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.974365+0000) 2019-10-13T11:24:18.977 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:18.974+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.975463+0000) 2019-10-13T11:24:18.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:18.974+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.976066+0000) 2019-10-13T11:24:18.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:18.973+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.977038+0000) 2019-10-13T11:24:18.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:18.974+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.977201+0000) 2019-10-13T11:24:18.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:18.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122827 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:18.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:18.981+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.984458+0000) 2019-10-13T11:24:19.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:19.897+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.901062+0000) 2019-10-13T11:24:22.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:22.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:22.155343+0000) 2019-10-13T11:24:23.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:23.978+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.982949+0000) 2019-10-13T11:24:23.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:23.982+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.983682+0000) 2019-10-13T11:24:23.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:23.982+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.984537+0000) 2019-10-13T11:24:23.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:23.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.985511+0000) 2019-10-13T11:24:23.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:23.982+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.985619+0000) 2019-10-13T11:24:23.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:23.982+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.986071+0000) 2019-10-13T11:24:23.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:23.982+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.986792+0000) 2019-10-13T11:24:23.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:23.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:23.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:23.989+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.992845+0000) 2019-10-13T11:24:25.466 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:25.462+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.465374+0000) 2019-10-13T11:24:26.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:26.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:26.401028+0000) 2019-10-13T11:24:27.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:27.637+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:27.641520+0000) 2019-10-13T11:24:28.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:28.662+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.664449+0000) 2019-10-13T11:24:28.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:28.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.669634+0000) 2019-10-13T11:24:28.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:28.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.681084+0000) 2019-10-13T11:24:28.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:28.990+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.992534+0000) 2019-10-13T11:24:28.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:28.990+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.993579+0000) 2019-10-13T11:24:28.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:28.993+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.994263+0000) 2019-10-13T11:24:28.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:28.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.994997+0000) 2019-10-13T11:24:28.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:28.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.995190+0000) 2019-10-13T11:24:28.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:28.994+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.995521+0000) 2019-10-13T11:24:28.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:28.994+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.995998+0000) 2019-10-13T11:24:28.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:28.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:29.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:28.997+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.999791+0000) 2019-10-13T11:24:29.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:29.897+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.901235+0000) 2019-10-13T11:24:32.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:32.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:32.155547+0000) 2019-10-13T11:24:34.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:33.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.001812+0000) 2019-10-13T11:24:34.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:33.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.002138+0000) 2019-10-13T11:24:34.003 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:33.998+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.002602+0000) 2019-10-13T11:24:34.004 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:33.998+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.002753+0000) 2019-10-13T11:24:34.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:34.002+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.004163+0000) 2019-10-13T11:24:34.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:34.002+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.004406+0000) 2019-10-13T11:24:34.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:34.002+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.004999+0000) 2019-10-13T11:24:34.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:34.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:34.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:34.009+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.011233+0000) 2019-10-13T11:24:35.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:35.462+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.465584+0000) 2019-10-13T11:24:36.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:36.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:36.401223+0000) 2019-10-13T11:24:37.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:37.645+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:37.648205+0000) 2019-10-13T11:24:38.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:38.662+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.664683+0000) 2019-10-13T11:24:38.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:38.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.669829+0000) 2019-10-13T11:24:38.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:38.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.681281+0000) 2019-10-13T11:24:39.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:39.006+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.011180+0000) 2019-10-13T11:24:39.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:39.010+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.011571+0000) 2019-10-13T11:24:39.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:39.010+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.012306+0000) 2019-10-13T11:24:39.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:39.010+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.013190+0000) 2019-10-13T11:24:39.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:39.009+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.013672+0000) 2019-10-13T11:24:39.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:39.010+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.013967+0000) 2019-10-13T11:24:39.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:39.010+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.014426+0000) 2019-10-13T11:24:39.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:39.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:39.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:39.017+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.018243+0000) 2019-10-13T11:24:39.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:39.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.901418+0000) 2019-10-13T11:24:42.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:42.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:42.155751+0000) 2019-10-13T11:24:42.730 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:24:42.730 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:42.733 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:24:42.733 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:44.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:44.018+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.020342+0000) 2019-10-13T11:24:44.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:44.017+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.022019+0000) 2019-10-13T11:24:44.023 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:44.018+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.022697+0000) 2019-10-13T11:24:44.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:44.018+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.023084+0000) 2019-10-13T11:24:44.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:44.022+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.023474+0000) 2019-10-13T11:24:44.024 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:44.022+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.023856+0000) 2019-10-13T11:24:44.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:44.022+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.024297+0000) 2019-10-13T11:24:44.027 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:44.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 122998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:44.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:44.025+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.028891+0000) 2019-10-13T11:24:45.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:45.462+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.465796+0000) 2019-10-13T11:24:46.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:46.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:46.401416+0000) 2019-10-13T11:24:47.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:47.653+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:47.655109+0000) 2019-10-13T11:24:48.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:48.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.664918+0000) 2019-10-13T11:24:48.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:48.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.670032+0000) 2019-10-13T11:24:48.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:48.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.681486+0000) 2019-10-13T11:24:49.032 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:49.025+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.030803+0000) 2019-10-13T11:24:49.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:49.029+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.031555+0000) 2019-10-13T11:24:49.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:49.029+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.032329+0000) 2019-10-13T11:24:49.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:49.029+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.032469+0000) 2019-10-13T11:24:49.034 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:49.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.032719+0000) 2019-10-13T11:24:49.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:49.029+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.033256+0000) 2019-10-13T11:24:49.035 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:49.029+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.033459+0000) 2019-10-13T11:24:49.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:49.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:49.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:49.033+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.037502+0000) 2019-10-13T11:24:49.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:49.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.901585+0000) 2019-10-13T11:24:52.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:52.153+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:52.155986+0000) 2019-10-13T11:24:54.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:54.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.039271+0000) 2019-10-13T11:24:54.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:54.037+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.040015+0000) 2019-10-13T11:24:54.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:54.037+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.040472+0000) 2019-10-13T11:24:54.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:54.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.040968+0000) 2019-10-13T11:24:54.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:54.037+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.041086+0000) 2019-10-13T11:24:54.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:54.037+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.042786+0000) 2019-10-13T11:24:54.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:54.041+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.043175+0000) 2019-10-13T11:24:54.046 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:54.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:54.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:54.049+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.051395+0000) 2019-10-13T11:24:55.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:55.461+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.465982+0000) 2019-10-13T11:24:56.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:56.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:56.401612+0000) 2019-10-13T11:24:57.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:57.656+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:57.661396+0000) 2019-10-13T11:24:58.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:58.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.665134+0000) 2019-10-13T11:24:58.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:58.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.670253+0000) 2019-10-13T11:24:58.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:58.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.681680+0000) 2019-10-13T11:24:59.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:24:59.045+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.048420+0000) 2019-10-13T11:24:59.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:24:59.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.049061+0000) 2019-10-13T11:24:59.050 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:24:59.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.049663+0000) 2019-10-13T11:24:59.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:59.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.050296+0000) 2019-10-13T11:24:59.051 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:24:59.045+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.050477+0000) 2019-10-13T11:24:59.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:24:59.045+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.050977+0000) 2019-10-13T11:24:59.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:24:59.049+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.051716+0000) 2019-10-13T11:24:59.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:24:59.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:24:59.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:24:59.052+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.055808+0000) 2019-10-13T11:24:59.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:24:59.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.901754+0000) 2019-10-13T11:25:02.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:02.153+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:02.156191+0000) 2019-10-13T11:25:04.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:04.053+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.057511+0000) 2019-10-13T11:25:04.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:04.053+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.057890+0000) 2019-10-13T11:25:04.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:04.053+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.058550+0000) 2019-10-13T11:25:04.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:04.053+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.058879+0000) 2019-10-13T11:25:04.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:04.056+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.059142+0000) 2019-10-13T11:25:04.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:04.057+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.059980+0000) 2019-10-13T11:25:04.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:04.057+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.060485+0000) 2019-10-13T11:25:04.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:04.060+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.064746+0000) 2019-10-13T11:25:04.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:04.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:05.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:05.465+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.466162+0000) 2019-10-13T11:25:06.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:06.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:06.401804+0000) 2019-10-13T11:25:07.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:07.664+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:07.667386+0000) 2019-10-13T11:25:08.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:08.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.665332+0000) 2019-10-13T11:25:08.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:08.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.670445+0000) 2019-10-13T11:25:08.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:08.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.681876+0000) 2019-10-13T11:25:09.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:09.065+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.067611+0000) 2019-10-13T11:25:09.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:09.065+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.068401+0000) 2019-10-13T11:25:09.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:09.065+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.068763+0000) 2019-10-13T11:25:09.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:09.065+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.069018+0000) 2019-10-13T11:25:09.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:09.065+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.069582+0000) 2019-10-13T11:25:09.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:09.068+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.070202+0000) 2019-10-13T11:25:09.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:09.069+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.071532+0000) 2019-10-13T11:25:09.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:09.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:09.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:09.072+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.077603+0000) 2019-10-13T11:25:09.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:09.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.901928+0000) 2019-10-13T11:25:12.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:12.153+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:12.156346+0000) 2019-10-13T11:25:12.795 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:25:12.796 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:12.799 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:25:12.799 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:14.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:14.073+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.076988+0000) 2019-10-13T11:25:14.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:14.073+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.077644+0000) 2019-10-13T11:25:14.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:14.073+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.078258+0000) 2019-10-13T11:25:14.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:14.073+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.079274+0000) 2019-10-13T11:25:14.080 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:14.077+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.079559+0000) 2019-10-13T11:25:14.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:14.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.080172+0000) 2019-10-13T11:25:14.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:14.077+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.080508+0000) 2019-10-13T11:25:14.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:14.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:14.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:14.084+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.087303+0000) 2019-10-13T11:25:15.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:15.465+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:15.466455+0000) 2019-10-13T11:25:16.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:16.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.402028+0000) 2019-10-13T11:25:17.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:17.668+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:17.674042+0000) 2019-10-13T11:25:18.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:18.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.665537+0000) 2019-10-13T11:25:18.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:18.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.670604+0000) 2019-10-13T11:25:18.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:18.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.682104+0000) 2019-10-13T11:25:19.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:19.081+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.086406+0000) 2019-10-13T11:25:19.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:19.081+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.086691+0000) 2019-10-13T11:25:19.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:19.085+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.088091+0000) 2019-10-13T11:25:19.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:19.085+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.088505+0000) 2019-10-13T11:25:19.090 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:19.085+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.089291+0000) 2019-10-13T11:25:19.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:19.085+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.090120+0000) 2019-10-13T11:25:19.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:19.088+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.090969+0000) 2019-10-13T11:25:19.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:19.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123241 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:19.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:19.092+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.095840+0000) 2019-10-13T11:25:19.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:19.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.902123+0000) 2019-10-13T11:25:22.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:22.153+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:22.156538+0000) 2019-10-13T11:25:24.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:24.093+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.095981+0000) 2019-10-13T11:25:24.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:24.093+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.096405+0000) 2019-10-13T11:25:24.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:24.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.097497+0000) 2019-10-13T11:25:24.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:24.093+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.097791+0000) 2019-10-13T11:25:24.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:24.093+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.098715+0000) 2019-10-13T11:25:24.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:24.097+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.099845+0000) 2019-10-13T11:25:24.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:24.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.100044+0000) 2019-10-13T11:25:24.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:24.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:24.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:24.104+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.106529+0000) 2019-10-13T11:25:25.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:25.465+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:25.466677+0000) 2019-10-13T11:25:26.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:26.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.402252+0000) 2019-10-13T11:25:27.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:27.676+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:27.680438+0000) 2019-10-13T11:25:28.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:28.661+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.665762+0000) 2019-10-13T11:25:28.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:28.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.670795+0000) 2019-10-13T11:25:28.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:28.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.682301+0000) 2019-10-13T11:25:29.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:29.101+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.105840+0000) 2019-10-13T11:25:29.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:29.101+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.106230+0000) 2019-10-13T11:25:29.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:29.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.107039+0000) 2019-10-13T11:25:29.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:29.101+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.106509+0000) 2019-10-13T11:25:29.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:29.105+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.107356+0000) 2019-10-13T11:25:29.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:29.105+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.107912+0000) 2019-10-13T11:25:29.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:29.105+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.108042+0000) 2019-10-13T11:25:29.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:29.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:29.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:29.112+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.115849+0000) 2019-10-13T11:25:29.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:29.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.902337+0000) 2019-10-13T11:25:32.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:32.152+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:32.156739+0000) 2019-10-13T11:25:34.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:34.108+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.114905+0000) 2019-10-13T11:25:34.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:34.112+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.116065+0000) 2019-10-13T11:25:34.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:34.112+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.116948+0000) 2019-10-13T11:25:34.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:34.112+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.117262+0000) 2019-10-13T11:25:34.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:34.112+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.117883+0000) 2019-10-13T11:25:34.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:34.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.118740+0000) 2019-10-13T11:25:34.120 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:34.112+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.119211+0000) 2019-10-13T11:25:34.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:34.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:34.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:34.120+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.125947+0000) 2019-10-13T11:25:35.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:35.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:35.466888+0000) 2019-10-13T11:25:36.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:36.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.402444+0000) 2019-10-13T11:25:37.687 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:37.684+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:37.686891+0000) 2019-10-13T11:25:38.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:38.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.665976+0000) 2019-10-13T11:25:38.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:38.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.670982+0000) 2019-10-13T11:25:38.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:38.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.682519+0000) 2019-10-13T11:25:39.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:39.120+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.125113+0000) 2019-10-13T11:25:39.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:39.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.125956+0000) 2019-10-13T11:25:39.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:39.120+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.126434+0000) 2019-10-13T11:25:39.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:39.124+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.126965+0000) 2019-10-13T11:25:39.128 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:39.120+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.127200+0000) 2019-10-13T11:25:39.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:39.124+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.127704+0000) 2019-10-13T11:25:39.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:39.124+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.127922+0000) 2019-10-13T11:25:39.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:39.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123379 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:39.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:39.132+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.135462+0000) 2019-10-13T11:25:39.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:39.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.902507+0000) 2019-10-13T11:25:42.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:42.152+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:42.156955+0000) 2019-10-13T11:25:42.916 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:25:42.916 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:42.920 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:25:42.920 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:44.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:44.128+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.134102+0000) 2019-10-13T11:25:44.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:44.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.134438+0000) 2019-10-13T11:25:44.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:44.128+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.134955+0000) 2019-10-13T11:25:44.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:44.132+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.135605+0000) 2019-10-13T11:25:44.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:44.132+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.136064+0000) 2019-10-13T11:25:44.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:44.132+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.136873+0000) 2019-10-13T11:25:44.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:44.132+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.137738+0000) 2019-10-13T11:25:44.141 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:44.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:44.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:44.136+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.141595+0000) 2019-10-13T11:25:45.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:45.460+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:45.467090+0000) 2019-10-13T11:25:46.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:46.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.402660+0000) 2019-10-13T11:25:47.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:47.692+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:47.693299+0000) 2019-10-13T11:25:48.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:48.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.666173+0000) 2019-10-13T11:25:48.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:48.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.671177+0000) 2019-10-13T11:25:48.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:48.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.682742+0000) 2019-10-13T11:25:49.144 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:49.140+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.143113+0000) 2019-10-13T11:25:49.144 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:49.140+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.143789+0000) 2019-10-13T11:25:49.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:49.140+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.145236+0000) 2019-10-13T11:25:49.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:49.140+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.146171+0000) 2019-10-13T11:25:49.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:49.144+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.146570+0000) 2019-10-13T11:25:49.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:49.140+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.147241+0000) 2019-10-13T11:25:49.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:49.144+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.147640+0000) 2019-10-13T11:25:49.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:49.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:49.153 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:49.148+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.152428+0000) 2019-10-13T11:25:49.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:49.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.902677+0000) 2019-10-13T11:25:52.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:52.156+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:52.157156+0000) 2019-10-13T11:25:54.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:54.148+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.152546+0000) 2019-10-13T11:25:54.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:54.148+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.153200+0000) 2019-10-13T11:25:54.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:54.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.153899+0000) 2019-10-13T11:25:54.155 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:54.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.154546+0000) 2019-10-13T11:25:54.155 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:54.148+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.154861+0000) 2019-10-13T11:25:54.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:54.152+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.155577+0000) 2019-10-13T11:25:54.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:54.151+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.156227+0000) 2019-10-13T11:25:54.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:54.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:54.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:54.155+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.161618+0000) 2019-10-13T11:25:55.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:55.464+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:55.467345+0000) 2019-10-13T11:25:56.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:56.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.402882+0000) 2019-10-13T11:25:57.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:57.695+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:57.699806+0000) 2019-10-13T11:25:58.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:58.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.666393+0000) 2019-10-13T11:25:58.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:58.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.671397+0000) 2019-10-13T11:25:58.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:58.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.682958+0000) 2019-10-13T11:25:59.162 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:25:59.156+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.161624+0000) 2019-10-13T11:25:59.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:25:59.156+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.162237+0000) 2019-10-13T11:25:59.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:25:59.156+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.162475+0000) 2019-10-13T11:25:59.164 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:59.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.163546+0000) 2019-10-13T11:25:59.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:25:59.160+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.163981+0000) 2019-10-13T11:25:59.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:25:59.160+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.164042+0000) 2019-10-13T11:25:59.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:25:59.160+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.164727+0000) 2019-10-13T11:25:59.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:25:59.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:25:59.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:25:59.167+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.172593+0000) 2019-10-13T11:25:59.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:25:59.899+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.902853+0000) 2019-10-13T11:26:02.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:02.156+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:02.157353+0000) 2019-10-13T11:26:04.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:04.168+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.172250+0000) 2019-10-13T11:26:04.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:04.168+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.172562+0000) 2019-10-13T11:26:04.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:04.168+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.173951+0000) 2019-10-13T11:26:04.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:04.168+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.174339+0000) 2019-10-13T11:26:04.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:04.168+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.174600+0000) 2019-10-13T11:26:04.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:04.171+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.175523+0000) 2019-10-13T11:26:04.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:04.172+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.175788+0000) 2019-10-13T11:26:04.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:04.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:04.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:04.175+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.180452+0000) 2019-10-13T11:26:05.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:05.464+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.467593+0000) 2019-10-13T11:26:06.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:06.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.403118+0000) 2019-10-13T11:26:07.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:07.703+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:07.706210+0000) 2019-10-13T11:26:08.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:08.660+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.666588+0000) 2019-10-13T11:26:08.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:08.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.671588+0000) 2019-10-13T11:26:08.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:08.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.683149+0000) 2019-10-13T11:26:09.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:09.175+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.181244+0000) 2019-10-13T11:26:09.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:09.176+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.181563+0000) 2019-10-13T11:26:09.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:09.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.181810+0000) 2019-10-13T11:26:09.183 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:09.176+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.182219+0000) 2019-10-13T11:26:09.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:09.180+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.184041+0000) 2019-10-13T11:26:09.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:09.180+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.184257+0000) 2019-10-13T11:26:09.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:09.180+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.184977+0000) 2019-10-13T11:26:09.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:09.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123584 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:09.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:09.187+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.191386+0000) 2019-10-13T11:26:09.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:09.899+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:09.903026+0000) 2019-10-13T11:26:12.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:12.156+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:12.157586+0000) 2019-10-13T11:26:13.052 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:26:13.052 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:13.055 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:26:13.056 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:14.192 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:14.183+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.190844+0000) 2019-10-13T11:26:14.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:14.187+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.191487+0000) 2019-10-13T11:26:14.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:14.187+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.192154+0000) 2019-10-13T11:26:14.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:14.187+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.192883+0000) 2019-10-13T11:26:14.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:14.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.193254+0000) 2019-10-13T11:26:14.194 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:14.187+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.194103+0000) 2019-10-13T11:26:14.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:14.187+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.194781+0000) 2019-10-13T11:26:14.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:14.191+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:14.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:14.199+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.202395+0000) 2019-10-13T11:26:15.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:15.463+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.467835+0000) 2019-10-13T11:26:16.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:16.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.403334+0000) 2019-10-13T11:26:17.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:17.711+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:17.712579+0000) 2019-10-13T11:26:18.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:18.659+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.666781+0000) 2019-10-13T11:26:18.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:18.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.671818+0000) 2019-10-13T11:26:18.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:18.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.683348+0000) 2019-10-13T11:26:19.202 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:19.195+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.201155+0000) 2019-10-13T11:26:19.202 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:19.195+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.201561+0000) 2019-10-13T11:26:19.203 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:19.195+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.202202+0000) 2019-10-13T11:26:19.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:19.195+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.202868+0000) 2019-10-13T11:26:19.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:19.199+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.203566+0000) 2019-10-13T11:26:19.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:19.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.204426+0000) 2019-10-13T11:26:19.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:19.199+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.204777+0000) 2019-10-13T11:26:19.208 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:19.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:19.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:19.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.211202+0000) 2019-10-13T11:26:19.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:19.899+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:19.903200+0000) 2019-10-13T11:26:22.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:22.155+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:22.157789+0000) 2019-10-13T11:26:24.211 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:24.203+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.210696+0000) 2019-10-13T11:26:24.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:24.203+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.211240+0000) 2019-10-13T11:26:24.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:24.207+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.211817+0000) 2019-10-13T11:26:24.213 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:24.207+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.212094+0000) 2019-10-13T11:26:24.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:24.207+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.212914+0000) 2019-10-13T11:26:24.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:24.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.213170+0000) 2019-10-13T11:26:24.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:24.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.213603+0000) 2019-10-13T11:26:24.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:24.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:24.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:24.215+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.220458+0000) 2019-10-13T11:26:25.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:25.467+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.468043+0000) 2019-10-13T11:26:26.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:26.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.403533+0000) 2019-10-13T11:26:27.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:27.715+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:27.719159+0000) 2019-10-13T11:26:28.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:28.663+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.666975+0000) 2019-10-13T11:26:28.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:28.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.672010+0000) 2019-10-13T11:26:28.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:28.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.683497+0000) 2019-10-13T11:26:29.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:29.215+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.220880+0000) 2019-10-13T11:26:29.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:29.215+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.220316+0000) 2019-10-13T11:26:29.222 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:29.215+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.221810+0000) 2019-10-13T11:26:29.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:29.215+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.222132+0000) 2019-10-13T11:26:29.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:29.219+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.222644+0000) 2019-10-13T11:26:29.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:29.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.222791+0000) 2019-10-13T11:26:29.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:29.215+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.223248+0000) 2019-10-13T11:26:29.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:29.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:29.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:29.223+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.227101+0000) 2019-10-13T11:26:29.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:29.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:29.903369+0000) 2019-10-13T11:26:32.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:32.155+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:32.157988+0000) 2019-10-13T11:26:34.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:34.223+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.229815+0000) 2019-10-13T11:26:34.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:34.223+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.230188+0000) 2019-10-13T11:26:34.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:34.223+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.231200+0000) 2019-10-13T11:26:34.232 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:34.227+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.231906+0000) 2019-10-13T11:26:34.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:34.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.232354+0000) 2019-10-13T11:26:34.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:34.227+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.232821+0000) 2019-10-13T11:26:34.234 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:34.227+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.233184+0000) 2019-10-13T11:26:34.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:34.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:34.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:34.235+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.239133+0000) 2019-10-13T11:26:35.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:35.467+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.468254+0000) 2019-10-13T11:26:36.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:36.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.403690+0000) 2019-10-13T11:26:37.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:37.719+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:37.725723+0000) 2019-10-13T11:26:38.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:38.663+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.667162+0000) 2019-10-13T11:26:38.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:38.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.672222+0000) 2019-10-13T11:26:38.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:38.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.683698+0000) 2019-10-13T11:26:39.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:39.235+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.239629+0000) 2019-10-13T11:26:39.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:39.235+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.240744+0000) 2019-10-13T11:26:39.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:39.235+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.240976+0000) 2019-10-13T11:26:39.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:39.235+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.241145+0000) 2019-10-13T11:26:39.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:39.235+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.241622+0000) 2019-10-13T11:26:39.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:39.235+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.242005+0000) 2019-10-13T11:26:39.243 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:39.235+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.242478+0000) 2019-10-13T11:26:39.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:39.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:39.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:39.243+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.246976+0000) 2019-10-13T11:26:39.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:39.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:39.903546+0000) 2019-10-13T11:26:42.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:42.155+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:42.158189+0000) 2019-10-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:44.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:44.243+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.249015+0000) 2019-10-13T11:26:44.250 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:44.247+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.249428+0000) 2019-10-13T11:26:44.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:44.247+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.250186+0000) 2019-10-13T11:26:44.251 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:44.247+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.250384+0000) 2019-10-13T11:26:44.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:44.247+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.250824+0000) 2019-10-13T11:26:44.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:44.251+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.251356+0000) 2019-10-13T11:26:44.252 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:44.247+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.250847+0000) 2019-10-13T11:26:44.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:44.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:44.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:44.255+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.258254+0000) 2019-10-13T11:26:45.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:45.467+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.468455+0000) 2019-10-13T11:26:46.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:46.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.403882+0000) 2019-10-13T11:26:47.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:47.730+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:47.732266+0000) 2019-10-13T11:26:48.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:48.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.667363+0000) 2019-10-13T11:26:48.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:48.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.672450+0000) 2019-10-13T11:26:48.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:48.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.683903+0000) 2019-10-13T11:26:49.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:49.255+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.258512+0000) 2019-10-13T11:26:49.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:49.254+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.259430+0000) 2019-10-13T11:26:49.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:49.259+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.259727+0000) 2019-10-13T11:26:49.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:49.259+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.260518+0000) 2019-10-13T11:26:49.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:49.259+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.260777+0000) 2019-10-13T11:26:49.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:49.259+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.261663+0000) 2019-10-13T11:26:49.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:49.259+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.262426+0000) 2019-10-13T11:26:49.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:49.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:49.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:49.262+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.267151+0000) 2019-10-13T11:26:49.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:49.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:49.903734+0000) 2019-10-13T11:26:52.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:52.155+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:52.158390+0000) 2019-10-13T11:26:54.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:54.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.268977+0000) 2019-10-13T11:26:54.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:54.267+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.269183+0000) 2019-10-13T11:26:54.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:54.267+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.269550+0000) 2019-10-13T11:26:54.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:54.267+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.270368+0000) 2019-10-13T11:26:54.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:54.267+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.270680+0000) 2019-10-13T11:26:54.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:54.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.271133+0000) 2019-10-13T11:26:54.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:54.271+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.272076+0000) 2019-10-13T11:26:54.275 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:54.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:54.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:54.270+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.276358+0000) 2019-10-13T11:26:55.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:55.467+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.468675+0000) 2019-10-13T11:26:56.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:56.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.404104+0000) 2019-10-13T11:26:57.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:57.738+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:57.738853+0000) 2019-10-13T11:26:58.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:58.666+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.667579+0000) 2019-10-13T11:26:58.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:58.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.672700+0000) 2019-10-13T11:26:58.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:58.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.684124+0000) 2019-10-13T11:26:59.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:26:59.274+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.278842+0000) 2019-10-13T11:26:59.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:26:59.274+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.279136+0000) 2019-10-13T11:26:59.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:26:59.278+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.279797+0000) 2019-10-13T11:26:59.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:26:59.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.280023+0000) 2019-10-13T11:26:59.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:26:59.278+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.280623+0000) 2019-10-13T11:26:59.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:26:59.278+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.281225+0000) 2019-10-13T11:26:59.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:59.274+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.281497+0000) 2019-10-13T11:26:59.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:26:59.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:26:59.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:26:59.282+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.288444+0000) 2019-10-13T11:26:59.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:26:59.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:59.903926+0000) 2019-10-13T11:27:02.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:02.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:02.158630+0000) 2019-10-13T11:27:04.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:04.282+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.286817+0000) 2019-10-13T11:27:04.289 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:04.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.288151+0000) 2019-10-13T11:27:04.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:04.286+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.288669+0000) 2019-10-13T11:27:04.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:04.286+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.289482+0000) 2019-10-13T11:27:04.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:04.286+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.289791+0000) 2019-10-13T11:27:04.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:04.286+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.290332+0000) 2019-10-13T11:27:04.291 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:04.286+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.290673+0000) 2019-10-13T11:27:04.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:04.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123953 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:04.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:04.290+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.297982+0000) 2019-10-13T11:27:05.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:05.466+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.468890+0000) 2019-10-13T11:27:06.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:06.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.404332+0000) 2019-10-13T11:27:07.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:07.738+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:07.745282+0000) 2019-10-13T11:27:08.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:08.666+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.667784+0000) 2019-10-13T11:27:08.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:08.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.672885+0000) 2019-10-13T11:27:08.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:08.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.684335+0000) 2019-10-13T11:27:09.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:09.294+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.296745+0000) 2019-10-13T11:27:09.298 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:09.294+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.297266+0000) 2019-10-13T11:27:09.298 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:09.294+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.297522+0000) 2019-10-13T11:27:09.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:09.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.298265+0000) 2019-10-13T11:27:09.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:09.294+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.298842+0000) 2019-10-13T11:27:09.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:09.294+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.299217+0000) 2019-10-13T11:27:09.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:09.298+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.299872+0000) 2019-10-13T11:27:09.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:09.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 123992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:09.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:09.302+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.307452+0000) 2019-10-13T11:27:09.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:09.898+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:09.904117+0000) 2019-10-13T11:27:12.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:12.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:12.158827+0000) 2019-10-13T11:27:13.299 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:27:13.299 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:13.302 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:27:13.303 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:14.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:14.302+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.305812+0000) 2019-10-13T11:27:14.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:14.302+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.306304+0000) 2019-10-13T11:27:14.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:14.302+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.307285+0000) 2019-10-13T11:27:14.308 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:14.306+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.307992+0000) 2019-10-13T11:27:14.309 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:14.306+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.308512+0000) 2019-10-13T11:27:14.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:14.306+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.309585+0000) 2019-10-13T11:27:14.310 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:14.306+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.309903+0000) 2019-10-13T11:27:14.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:14.310+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:14.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:14.310+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.314605+0000) 2019-10-13T11:27:15.470 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:15.466+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.469167+0000) 2019-10-13T11:27:16.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:16.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.404545+0000) 2019-10-13T11:27:17.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:17.750+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:17.751801+0000) 2019-10-13T11:27:18.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:18.666+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.667989+0000) 2019-10-13T11:27:18.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:18.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.673046+0000) 2019-10-13T11:27:18.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:18.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.684528+0000) 2019-10-13T11:27:19.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:19.310+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.317294+0000) 2019-10-13T11:27:19.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:19.314+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.318019+0000) 2019-10-13T11:27:19.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:19.314+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.318608+0000) 2019-10-13T11:27:19.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:19.314+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.318709+0000) 2019-10-13T11:27:19.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:19.314+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.319074+0000) 2019-10-13T11:27:19.320 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:19.318+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.319365+0000) 2019-10-13T11:27:19.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:19.318+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.320441+0000) 2019-10-13T11:27:19.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:19.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124059 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:19.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:19.318+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.324093+0000) 2019-10-13T11:27:19.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:19.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:19.904329+0000) 2019-10-13T11:27:22.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:22.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:22.159060+0000) 2019-10-13T11:27:24.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:24.322+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.326309+0000) 2019-10-13T11:27:24.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:24.322+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.326983+0000) 2019-10-13T11:27:24.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:24.326+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.327397+0000) 2019-10-13T11:27:24.328 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:24.326+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.327576+0000) 2019-10-13T11:27:24.329 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:24.326+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.328090+0000) 2019-10-13T11:27:24.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:24.326+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.328758+0000) 2019-10-13T11:27:24.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:24.326+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.328992+0000) 2019-10-13T11:27:24.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:24.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124090 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:24.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:24.330+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.334363+0000) 2019-10-13T11:27:25.470 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:25.466+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.469406+0000) 2019-10-13T11:27:26.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:26.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.404740+0000) 2019-10-13T11:27:27.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:27.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:27.762892+0000) 2019-10-13T11:27:28.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:28.666+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.668188+0000) 2019-10-13T11:27:28.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:28.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.673279+0000) 2019-10-13T11:27:28.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:28.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.684732+0000) 2019-10-13T11:27:29.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:29.334+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.336202+0000) 2019-10-13T11:27:29.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:29.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.336604+0000) 2019-10-13T11:27:29.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:29.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.337135+0000) 2019-10-13T11:27:29.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:29.334+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.337529+0000) 2019-10-13T11:27:29.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:29.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.338173+0000) 2019-10-13T11:27:29.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:29.334+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.338919+0000) 2019-10-13T11:27:29.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:29.338+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.339471+0000) 2019-10-13T11:27:29.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:29.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124128 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:29.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:29.338+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.344463+0000) 2019-10-13T11:27:29.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:29.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:29.904494+0000) 2019-10-13T11:27:32.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:32.154+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:32.159274+0000) 2019-10-13T11:27:34.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:34.342+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.344625+0000) 2019-10-13T11:27:34.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:34.342+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.345236+0000) 2019-10-13T11:27:34.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:34.346+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.346233+0000) 2019-10-13T11:27:34.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:34.342+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.345688+0000) 2019-10-13T11:27:34.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:34.342+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.346797+0000) 2019-10-13T11:27:34.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:34.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.347706+0000) 2019-10-13T11:27:34.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:34.346+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.348416+0000) 2019-10-13T11:27:34.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:34.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:34.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:34.350+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.353250+0000) 2019-10-13T11:27:35.470 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:35.466+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.469634+0000) 2019-10-13T11:27:36.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:36.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.404928+0000) 2019-10-13T11:27:37.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:37.766+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:37.769526+0000) 2019-10-13T11:27:38.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:38.666+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.668389+0000) 2019-10-13T11:27:38.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:38.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.673511+0000) 2019-10-13T11:27:38.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:38.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.684940+0000) 2019-10-13T11:27:39.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:39.354+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.355451+0000) 2019-10-13T11:27:39.357 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:39.354+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.356263+0000) 2019-10-13T11:27:39.357 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:39.354+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.356625+0000) 2019-10-13T11:27:39.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:39.354+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.357025+0000) 2019-10-13T11:27:39.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:39.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.357262+0000) 2019-10-13T11:27:39.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:39.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.357874+0000) 2019-10-13T11:27:39.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:39.358+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.359371+0000) 2019-10-13T11:27:39.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:39.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:39.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:39.362+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.365326+0000) 2019-10-13T11:27:39.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:39.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:39.904669+0000) 2019-10-13T11:27:42.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:42.157+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:42.159508+0000) 2019-10-13T11:27:43.402 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:27:43.402 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:43.406 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:27:43.406 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:44.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:44.361+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.365616+0000) 2019-10-13T11:27:44.368 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:44.361+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.366232+0000) 2019-10-13T11:27:44.368 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:44.361+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.366654+0000) 2019-10-13T11:27:44.369 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:44.365+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.367292+0000) 2019-10-13T11:27:44.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:44.361+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.366044+0000) 2019-10-13T11:27:44.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:44.365+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.368257+0000) 2019-10-13T11:27:44.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:44.365+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.368531+0000) 2019-10-13T11:27:44.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:44.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:44.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:44.369+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.373391+0000) 2019-10-13T11:27:45.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:45.465+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.469853+0000) 2019-10-13T11:27:46.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:46.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.405143+0000) 2019-10-13T11:27:47.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:47.773+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:47.776131+0000) 2019-10-13T11:27:48.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:48.665+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.668623+0000) 2019-10-13T11:27:48.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:48.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.673749+0000) 2019-10-13T11:27:48.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:48.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.685141+0000) 2019-10-13T11:27:49.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:49.369+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.374439+0000) 2019-10-13T11:27:49.375 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:49.369+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.374569+0000) 2019-10-13T11:27:49.376 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:49.369+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.375259+0000) 2019-10-13T11:27:49.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:49.373+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.375933+0000) 2019-10-13T11:27:49.377 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:49.373+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.376362+0000) 2019-10-13T11:27:49.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:49.373+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.377021+0000) 2019-10-13T11:27:49.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:49.373+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.377425+0000) 2019-10-13T11:27:49.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:49.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:49.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:49.381+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.383610+0000) 2019-10-13T11:27:49.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:49.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:49.904842+0000) 2019-10-13T11:27:52.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:52.157+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:52.159708+0000) 2019-10-13T11:27:54.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:54.377+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.383180+0000) 2019-10-13T11:27:54.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:54.381+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.384059+0000) 2019-10-13T11:27:54.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:54.381+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.384257+0000) 2019-10-13T11:27:54.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:54.381+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.384721+0000) 2019-10-13T11:27:54.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:54.381+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.385513+0000) 2019-10-13T11:27:54.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:54.385+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.386618+0000) 2019-10-13T11:27:54.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:54.385+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.387772+0000) 2019-10-13T11:27:54.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:54.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:54.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:54.393+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.396371+0000) 2019-10-13T11:27:55.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:55.465+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.470052+0000) 2019-10-13T11:27:56.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:56.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.405354+0000) 2019-10-13T11:27:57.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:57.781+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:57.782585+0000) 2019-10-13T11:27:58.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:58.665+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.668824+0000) 2019-10-13T11:27:58.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:58.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.673944+0000) 2019-10-13T11:27:58.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:58.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.685340+0000) 2019-10-13T11:27:59.394 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:27:59.389+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.393005+0000) 2019-10-13T11:27:59.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:59.389+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.393723+0000) 2019-10-13T11:27:59.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:27:59.389+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.394010+0000) 2019-10-13T11:27:59.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:27:59.389+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.394693+0000) 2019-10-13T11:27:59.395 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:27:59.389+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.394865+0000) 2019-10-13T11:27:59.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:27:59.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.395210+0000) 2019-10-13T11:27:59.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:27:59.393+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.396103+0000) 2019-10-13T11:27:59.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:27:59.393+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:27:59.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:27:59.401+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.403890+0000) 2019-10-13T11:27:59.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:27:59.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:59.905027+0000) 2019-10-13T11:28:02.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:02.157+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:02.159923+0000) 2019-10-13T11:28:04.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:04.397+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.402739+0000) 2019-10-13T11:28:04.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:04.401+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.403480+0000) 2019-10-13T11:28:04.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:04.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.402949+0000) 2019-10-13T11:28:04.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:04.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.404182+0000) 2019-10-13T11:28:04.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:04.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.404879+0000) 2019-10-13T11:28:04.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:04.401+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.404978+0000) 2019-10-13T11:28:04.406 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:04.401+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.405444+0000) 2019-10-13T11:28:04.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:04.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:04.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:04.409+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.412906+0000) 2019-10-13T11:28:05.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:05.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.470269+0000) 2019-10-13T11:28:06.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:06.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.405571+0000) 2019-10-13T11:28:07.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:07.785+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:07.789068+0000) 2019-10-13T11:28:08.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:08.665+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.669021+0000) 2019-10-13T11:28:08.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:08.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.674126+0000) 2019-10-13T11:28:08.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:08.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.685578+0000) 2019-10-13T11:28:09.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:09.409+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.411883+0000) 2019-10-13T11:28:09.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:09.409+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.412251+0000) 2019-10-13T11:28:09.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:09.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.412885+0000) 2019-10-13T11:28:09.414 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:09.409+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.413225+0000) 2019-10-13T11:28:09.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:09.409+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.413481+0000) 2019-10-13T11:28:09.415 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:09.409+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.414687+0000) 2019-10-13T11:28:09.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:09.413+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.415296+0000) 2019-10-13T11:28:09.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:09.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124399 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:09.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:09.421+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.422386+0000) 2019-10-13T11:28:09.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:09.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:09.905201+0000) 2019-10-13T11:28:12.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:12.157+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:12.160127+0000) 2019-10-13T11:28:13.474 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:28:13.474 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:13.477 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:28:13.478 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:14.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:14.417+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.421064+0000) 2019-10-13T11:28:14.422 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:14.417+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.421419+0000) 2019-10-13T11:28:14.423 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:14.417+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.422021+0000) 2019-10-13T11:28:14.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:14.417+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.422441+0000) 2019-10-13T11:28:14.423 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:14.417+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.422637+0000) 2019-10-13T11:28:14.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:14.421+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.423947+0000) 2019-10-13T11:28:14.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:14.421+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.424257+0000) 2019-10-13T11:28:14.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:14.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:14.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:14.425+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.429620+0000) 2019-10-13T11:28:15.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:15.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.470505+0000) 2019-10-13T11:28:16.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:16.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.405758+0000) 2019-10-13T11:28:17.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:17.793+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:17.795546+0000) 2019-10-13T11:28:18.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:18.669+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.669234+0000) 2019-10-13T11:28:18.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:18.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.674329+0000) 2019-10-13T11:28:18.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:18.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.685769+0000) 2019-10-13T11:28:19.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:19.425+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.430269+0000) 2019-10-13T11:28:19.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:19.425+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.431094+0000) 2019-10-13T11:28:19.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:19.429+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.431508+0000) 2019-10-13T11:28:19.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:19.429+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.432115+0000) 2019-10-13T11:28:19.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:19.429+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.432653+0000) 2019-10-13T11:28:19.434 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:19.429+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.433444+0000) 2019-10-13T11:28:19.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:19.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.434094+0000) 2019-10-13T11:28:19.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:19.433+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:19.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:19.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.440157+0000) 2019-10-13T11:28:19.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:19.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:19.905410+0000) 2019-10-13T11:28:22.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:22.157+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:22.160325+0000) 2019-10-13T11:28:24.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:24.436+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.440595+0000) 2019-10-13T11:28:24.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:24.436+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.440834+0000) 2019-10-13T11:28:24.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:24.436+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.441684+0000) 2019-10-13T11:28:24.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:24.436+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.442223+0000) 2019-10-13T11:28:24.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:24.441+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.442447+0000) 2019-10-13T11:28:24.444 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:24.436+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.442774+0000) 2019-10-13T11:28:24.444 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:24.436+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.443132+0000) 2019-10-13T11:28:24.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:24.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:24.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:24.449+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.453632+0000) 2019-10-13T11:28:25.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:25.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.470709+0000) 2019-10-13T11:28:26.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:26.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.405987+0000) 2019-10-13T11:28:27.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:27.797+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:27.801934+0000) 2019-10-13T11:28:28.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:28.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.669431+0000) 2019-10-13T11:28:28.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:28.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.674525+0000) 2019-10-13T11:28:28.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:28.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.686010+0000) 2019-10-13T11:28:29.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:29.444+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.450145+0000) 2019-10-13T11:28:29.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:29.444+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.450484+0000) 2019-10-13T11:28:29.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:29.444+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.451005+0000) 2019-10-13T11:28:29.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:29.449+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.451398+0000) 2019-10-13T11:28:29.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:29.448+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.452406+0000) 2019-10-13T11:28:29.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:29.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.452684+0000) 2019-10-13T11:28:29.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:29.448+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.453412+0000) 2019-10-13T11:28:29.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:29.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124534 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:29.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:29.453+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.458110+0000) 2019-10-13T11:28:29.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:29.901+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:29.905555+0000) 2019-10-13T11:28:32.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:32.156+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:32.160524+0000) 2019-10-13T11:28:34.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:34.456+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.459884+0000) 2019-10-13T11:28:34.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:34.456+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.460234+0000) 2019-10-13T11:28:34.461 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:34.456+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.461052+0000) 2019-10-13T11:28:34.462 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:34.457+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.461424+0000) 2019-10-13T11:28:34.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:34.456+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.461688+0000) 2019-10-13T11:28:34.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:34.456+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.462101+0000) 2019-10-13T11:28:34.464 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:34.456+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.462599+0000) 2019-10-13T11:28:34.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:34.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:34.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:34.469+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.472689+0000) 2019-10-13T11:28:35.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:35.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.470910+0000) 2019-10-13T11:28:36.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:36.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.406189+0000) 2019-10-13T11:28:37.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:37.804+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:37.808299+0000) 2019-10-13T11:28:38.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:38.664+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.669644+0000) 2019-10-13T11:28:38.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:38.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.674736+0000) 2019-10-13T11:28:38.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:38.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.686201+0000) 2019-10-13T11:28:39.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:39.464+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.469892+0000) 2019-10-13T11:28:39.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:39.464+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.470890+0000) 2019-10-13T11:28:39.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:39.468+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.471516+0000) 2019-10-13T11:28:39.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:39.468+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.471949+0000) 2019-10-13T11:28:39.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:39.468+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.472386+0000) 2019-10-13T11:28:39.473 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:39.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.472769+0000) 2019-10-13T11:28:39.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:39.468+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.473398+0000) 2019-10-13T11:28:39.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:39.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124601 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:39.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:39.480+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.483244+0000) 2019-10-13T11:28:39.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:39.900+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:39.905721+0000) 2019-10-13T11:28:42.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:42.160+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:42.160723+0000) 2019-10-13T11:28:43.539 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:28:43.539 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:43.542 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:28:43.542 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:44.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:44.476+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.480434+0000) 2019-10-13T11:28:44.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:44.476+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.480562+0000) 2019-10-13T11:28:44.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:44.476+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.481008+0000) 2019-10-13T11:28:44.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:44.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.481539+0000) 2019-10-13T11:28:44.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:44.476+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.481993+0000) 2019-10-13T11:28:44.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:44.476+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.483021+0000) 2019-10-13T11:28:44.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:44.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.483336+0000) 2019-10-13T11:28:44.487 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:44.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:44.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:44.484+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.488044+0000) 2019-10-13T11:28:45.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:45.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.471142+0000) 2019-10-13T11:28:46.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:46.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.406415+0000) 2019-10-13T11:28:47.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:47.816+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:47.820115+0000) 2019-10-13T11:28:48.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:48.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.669848+0000) 2019-10-13T11:28:48.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:48.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.674934+0000) 2019-10-13T11:28:48.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:48.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.686353+0000) 2019-10-13T11:28:49.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:49.484+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.488962+0000) 2019-10-13T11:28:49.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:49.484+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.489838+0000) 2019-10-13T11:28:49.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:49.484+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.490226+0000) 2019-10-13T11:28:49.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:49.488+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.491733+0000) 2019-10-13T11:28:49.493 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:49.488+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.492553+0000) 2019-10-13T11:28:49.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:49.488+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.492943+0000) 2019-10-13T11:28:49.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:49.488+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.493699+0000) 2019-10-13T11:28:49.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:49.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:49.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:49.496+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.498550+0000) 2019-10-13T11:28:49.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:49.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:49.905916+0000) 2019-10-13T11:28:52.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:52.156+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:52.160953+0000) 2019-10-13T11:28:54.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:54.496+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.500151+0000) 2019-10-13T11:28:54.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:54.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.500582+0000) 2019-10-13T11:28:54.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:54.496+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.501390+0000) 2019-10-13T11:28:54.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:54.496+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.501826+0000) 2019-10-13T11:28:54.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:54.496+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.502283+0000) 2019-10-13T11:28:54.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:54.496+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.502709+0000) 2019-10-13T11:28:54.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:54.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.503434+0000) 2019-10-13T11:28:54.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:54.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:54.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:54.504+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.509869+0000) 2019-10-13T11:28:55.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:55.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.471346+0000) 2019-10-13T11:28:56.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:56.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.406622+0000) 2019-10-13T11:28:57.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:57.824+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:57.826704+0000) 2019-10-13T11:28:58.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:58.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.670054+0000) 2019-10-13T11:28:58.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:58.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.675157+0000) 2019-10-13T11:28:58.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:58.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.686521+0000) 2019-10-13T11:28:59.510 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:28:59.504+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.509130+0000) 2019-10-13T11:28:59.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:28:59.504+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.509741+0000) 2019-10-13T11:28:59.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:28:59.504+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.510060+0000) 2019-10-13T11:28:59.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:28:59.504+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.510398+0000) 2019-10-13T11:28:59.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:28:59.504+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.510664+0000) 2019-10-13T11:28:59.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:59.508+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.511578+0000) 2019-10-13T11:28:59.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:28:59.508+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.512420+0000) 2019-10-13T11:28:59.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:28:59.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:28:59.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:28:59.516+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.518798+0000) 2019-10-13T11:28:59.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:28:59.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:59.906086+0000) 2019-10-13T11:29:02.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:02.160+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:02.161154+0000) 2019-10-13T11:29:04.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:04.516+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.518840+0000) 2019-10-13T11:29:04.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:04.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.519230+0000) 2019-10-13T11:29:04.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:04.516+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.519653+0000) 2019-10-13T11:29:04.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:04.516+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.520562+0000) 2019-10-13T11:29:04.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:04.516+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.520605+0000) 2019-10-13T11:29:04.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:04.516+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.521043+0000) 2019-10-13T11:29:04.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:04.516+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.521575+0000) 2019-10-13T11:29:04.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:04.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:04.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:04.524+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.527807+0000) 2019-10-13T11:29:05.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:05.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.471526+0000) 2019-10-13T11:29:06.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:06.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.406820+0000) 2019-10-13T11:29:07.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:07.832+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:07.836503+0000) 2019-10-13T11:29:08.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:08.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.670279+0000) 2019-10-13T11:29:08.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:08.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.675347+0000) 2019-10-13T11:29:08.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:08.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.686755+0000) 2019-10-13T11:29:09.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:09.523+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.527893+0000) 2019-10-13T11:29:09.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:09.523+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.529477+0000) 2019-10-13T11:29:09.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:09.523+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.529222+0000) 2019-10-13T11:29:09.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:09.523+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.529756+0000) 2019-10-13T11:29:09.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:09.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.530131+0000) 2019-10-13T11:29:09.532 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:09.523+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.530378+0000) 2019-10-13T11:29:09.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:09.527+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.531479+0000) 2019-10-13T11:29:09.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:09.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124806 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:09.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:09.536+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.538882+0000) 2019-10-13T11:29:09.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:09.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:09.906249+0000) 2019-10-13T11:29:12.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:12.159+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:12.161355+0000) 2019-10-13T11:29:13.679 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:29:13.679 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:13.682 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:29:13.683 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:14.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:14.532+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.537692+0000) 2019-10-13T11:29:14.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:14.531+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.537994+0000) 2019-10-13T11:29:14.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:14.531+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.539029+0000) 2019-10-13T11:29:14.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:14.535+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.539714+0000) 2019-10-13T11:29:14.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:14.535+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.540319+0000) 2019-10-13T11:29:14.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:14.535+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.540765+0000) 2019-10-13T11:29:14.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:14.535+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.541699+0000) 2019-10-13T11:29:14.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:14.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:14.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:14.540+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.545958+0000) 2019-10-13T11:29:15.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:15.467+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.471737+0000) 2019-10-13T11:29:16.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:16.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.407047+0000) 2019-10-13T11:29:17.843 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:17.840+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:17.842819+0000) 2019-10-13T11:29:18.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:18.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.670477+0000) 2019-10-13T11:29:18.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:18.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.675539+0000) 2019-10-13T11:29:18.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:18.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.686950+0000) 2019-10-13T11:29:19.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:19.543+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.549524+0000) 2019-10-13T11:29:19.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:19.543+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.549327+0000) 2019-10-13T11:29:19.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:19.543+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.550455+0000) 2019-10-13T11:29:19.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:19.543+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.550866+0000) 2019-10-13T11:29:19.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:19.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.551162+0000) 2019-10-13T11:29:19.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:19.547+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.551432+0000) 2019-10-13T11:29:19.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:19.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.551960+0000) 2019-10-13T11:29:19.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:19.547+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:19.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:19.552+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.557031+0000) 2019-10-13T11:29:19.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:19.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:19.906420+0000) 2019-10-13T11:29:22.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:22.155+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:22.161591+0000) 2019-10-13T11:29:24.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:24.551+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.557736+0000) 2019-10-13T11:29:24.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:24.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.558970+0000) 2019-10-13T11:29:24.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:24.555+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.559393+0000) 2019-10-13T11:29:24.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:24.555+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.559792+0000) 2019-10-13T11:29:24.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:24.555+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.560149+0000) 2019-10-13T11:29:24.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:24.555+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.561028+0000) 2019-10-13T11:29:24.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:24.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.561683+0000) 2019-10-13T11:29:24.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:24.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:24.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:24.564+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.567013+0000) 2019-10-13T11:29:25.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:25.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.471880+0000) 2019-10-13T11:29:26.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:26.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.407248+0000) 2019-10-13T11:29:27.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:27.844+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:27.849254+0000) 2019-10-13T11:29:28.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:28.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.670692+0000) 2019-10-13T11:29:28.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:28.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.675736+0000) 2019-10-13T11:29:28.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:28.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.687145+0000) 2019-10-13T11:29:29.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:29.563+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.566857+0000) 2019-10-13T11:29:29.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:29.563+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.568013+0000) 2019-10-13T11:29:29.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:29.563+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.568554+0000) 2019-10-13T11:29:29.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:29.563+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.569266+0000) 2019-10-13T11:29:29.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:29.563+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.569684+0000) 2019-10-13T11:29:29.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:29.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.570083+0000) 2019-10-13T11:29:29.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:29.563+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.570994+0000) 2019-10-13T11:29:29.573 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:29.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:29.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:29.572+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.577023+0000) 2019-10-13T11:29:29.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:29.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:29.906593+0000) 2019-10-13T11:29:32.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:32.159+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:32.161823+0000) 2019-10-13T11:29:34.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:34.571+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.576496+0000) 2019-10-13T11:29:34.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:34.571+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.577608+0000) 2019-10-13T11:29:34.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:34.571+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.578211+0000) 2019-10-13T11:29:34.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:34.571+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.578422+0000) 2019-10-13T11:29:34.579 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:34.575+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.578783+0000) 2019-10-13T11:29:34.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:34.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.579101+0000) 2019-10-13T11:29:34.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:34.575+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.579435+0000) 2019-10-13T11:29:34.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:34.575+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.582128+0000) 2019-10-13T11:29:34.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:34.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 124974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:35.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:35.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.472083+0000) 2019-10-13T11:29:36.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:36.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.407547+0000) 2019-10-13T11:29:37.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:37.851+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:37.855818+0000) 2019-10-13T11:29:38.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:38.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.670908+0000) 2019-10-13T11:29:38.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:38.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.675933+0000) 2019-10-13T11:29:38.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:38.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.687342+0000) 2019-10-13T11:29:39.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:39.583+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.586688+0000) 2019-10-13T11:29:39.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:39.587+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.587293+0000) 2019-10-13T11:29:39.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:39.587+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.588074+0000) 2019-10-13T11:29:39.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:39.587+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.588572+0000) 2019-10-13T11:29:39.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:39.587+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.588860+0000) 2019-10-13T11:29:39.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:39.587+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.589137+0000) 2019-10-13T11:29:39.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:39.583+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.589721+0000) 2019-10-13T11:29:39.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:39.587+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.592366+0000) 2019-10-13T11:29:39.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:39.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:39.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:39.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:39.906766+0000) 2019-10-13T11:29:42.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:42.159+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:42.162023+0000) 2019-10-13T11:29:43.818 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:29:43.819 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:43.822 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:29:43.822 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:44.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:44.595+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.595615+0000) 2019-10-13T11:29:44.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:44.595+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.596037+0000) 2019-10-13T11:29:44.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:44.595+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.597141+0000) 2019-10-13T11:29:44.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:44.595+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.597733+0000) 2019-10-13T11:29:44.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:44.595+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.598003+0000) 2019-10-13T11:29:44.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:44.595+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.598605+0000) 2019-10-13T11:29:44.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:44.595+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.598601+0000) 2019-10-13T11:29:44.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:44.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:44.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:44.599+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.603971+0000) 2019-10-13T11:29:45.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:45.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.472267+0000) 2019-10-13T11:29:46.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:46.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.407758+0000) 2019-10-13T11:29:47.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:47.859+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:47.862197+0000) 2019-10-13T11:29:48.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:48.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.671131+0000) 2019-10-13T11:29:48.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:48.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.676124+0000) 2019-10-13T11:29:48.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:48.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.687557+0000) 2019-10-13T11:29:49.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:49.603+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.605344+0000) 2019-10-13T11:29:49.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:49.603+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.605967+0000) 2019-10-13T11:29:49.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:49.603+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.606422+0000) 2019-10-13T11:29:49.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:49.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.607569+0000) 2019-10-13T11:29:49.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:49.607+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.608256+0000) 2019-10-13T11:29:49.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:49.607+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.608483+0000) 2019-10-13T11:29:49.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:49.603+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.608998+0000) 2019-10-13T11:29:49.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:49.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:49.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:49.611+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.615290+0000) 2019-10-13T11:29:49.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:49.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:49.906960+0000) 2019-10-13T11:29:52.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:52.158+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:52.162257+0000) 2019-10-13T11:29:54.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:54.610+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.613850+0000) 2019-10-13T11:29:54.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:54.610+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.614814+0000) 2019-10-13T11:29:54.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:54.614+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.615936+0000) 2019-10-13T11:29:54.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:54.614+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.616761+0000) 2019-10-13T11:29:54.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:54.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.617553+0000) 2019-10-13T11:29:54.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:54.614+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.617791+0000) 2019-10-13T11:29:54.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:54.615+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.618357+0000) 2019-10-13T11:29:54.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:54.618+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:54.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:54.619+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.623467+0000) 2019-10-13T11:29:55.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:55.470+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.472456+0000) 2019-10-13T11:29:56.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:56.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.407971+0000) 2019-10-13T11:29:57.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:57.867+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:57.868615+0000) 2019-10-13T11:29:58.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:58.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.671357+0000) 2019-10-13T11:29:58.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:58.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.676313+0000) 2019-10-13T11:29:58.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:58.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.687740+0000) 2019-10-13T11:29:59.627 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:29:59.622+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.626006+0000) 2019-10-13T11:29:59.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:29:59.622+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.626109+0000) 2019-10-13T11:29:59.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:29:59.622+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.626506+0000) 2019-10-13T11:29:59.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:59.623+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.627032+0000) 2019-10-13T11:29:59.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:29:59.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.628141+0000) 2019-10-13T11:29:59.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:29:59.626+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.628752+0000) 2019-10-13T11:29:59.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:29:59.626+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.629193+0000) 2019-10-13T11:29:59.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:29:59.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:29:59.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:29:59.631+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.634646+0000) 2019-10-13T11:29:59.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:29:59.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:59.907083+0000) 2019-10-13T11:30:02.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:02.158+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:02.162457+0000) 2019-10-13T11:30:04.636 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:04.634+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.635416+0000) 2019-10-13T11:30:04.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:04.634+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.636052+0000) 2019-10-13T11:30:04.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:04.634+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.636840+0000) 2019-10-13T11:30:04.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:04.631+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.637286+0000) 2019-10-13T11:30:04.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:04.634+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.637569+0000) 2019-10-13T11:30:04.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:04.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.637974+0000) 2019-10-13T11:30:04.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:04.634+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.638871+0000) 2019-10-13T11:30:04.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:04.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:04.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:04.643+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.646367+0000) 2019-10-13T11:30:05.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:05.470+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.472684+0000) 2019-10-13T11:30:06.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:06.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.408158+0000) 2019-10-13T11:30:07.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:07.871+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:07.875146+0000) 2019-10-13T11:30:08.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:08.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.671592+0000) 2019-10-13T11:30:08.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:08.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.676520+0000) 2019-10-13T11:30:08.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:08.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.687939+0000) 2019-10-13T11:30:09.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:09.642+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.644488+0000) 2019-10-13T11:30:09.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:09.642+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.645598+0000) 2019-10-13T11:30:09.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:09.642+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.645213+0000) 2019-10-13T11:30:09.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:09.642+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.646741+0000) 2019-10-13T11:30:09.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:09.642+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.646760+0000) 2019-10-13T11:30:09.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:09.642+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.647116+0000) 2019-10-13T11:30:09.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:09.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.647569+0000) 2019-10-13T11:30:09.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:09.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:09.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:09.651+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.654922+0000) 2019-10-13T11:30:09.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:09.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:09.907234+0000) 2019-10-13T11:30:12.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:12.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:12.162658+0000) 2019-10-13T11:30:13.946 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:30:13.946 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:13.949 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:30:13.950 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:14.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:14.650+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.655128+0000) 2019-10-13T11:30:14.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:14.651+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.655811+0000) 2019-10-13T11:30:14.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:14.654+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.656369+0000) 2019-10-13T11:30:14.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:14.654+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.656931+0000) 2019-10-13T11:30:14.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:14.654+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.657285+0000) 2019-10-13T11:30:14.658 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:14.654+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.657736+0000) 2019-10-13T11:30:14.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:14.654+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.658427+0000) 2019-10-13T11:30:14.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:14.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:14.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:14.659+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.662660+0000) 2019-10-13T11:30:15.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:15.470+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.472876+0000) 2019-10-13T11:30:16.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:16.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.408361+0000) 2019-10-13T11:30:17.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:17.879+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:17.881566+0000) 2019-10-13T11:30:18.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:18.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.671790+0000) 2019-10-13T11:30:18.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:18.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.676719+0000) 2019-10-13T11:30:18.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:18.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.688168+0000) 2019-10-13T11:30:19.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:19.662+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.665440+0000) 2019-10-13T11:30:19.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:19.662+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.666072+0000) 2019-10-13T11:30:19.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:19.662+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.666602+0000) 2019-10-13T11:30:19.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:19.663+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.667305+0000) 2019-10-13T11:30:19.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:19.666+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.667858+0000) 2019-10-13T11:30:19.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:19.666+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.667602+0000) 2019-10-13T11:30:19.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:19.666+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.668362+0000) 2019-10-13T11:30:19.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:19.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125281 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:19.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:19.671+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.675962+0000) 2019-10-13T11:30:19.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:19.903+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:19.907437+0000) 2019-10-13T11:30:22.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:22.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:22.162861+0000) 2019-10-13T11:30:24.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:24.670+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.674792+0000) 2019-10-13T11:30:24.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:24.670+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.674965+0000) 2019-10-13T11:30:24.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:24.670+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.675648+0000) 2019-10-13T11:30:24.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:24.674+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.676100+0000) 2019-10-13T11:30:24.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:24.674+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.676374+0000) 2019-10-13T11:30:24.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:24.674+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.676709+0000) 2019-10-13T11:30:24.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:24.674+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.677609+0000) 2019-10-13T11:30:24.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:24.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:24.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:24.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.683782+0000) 2019-10-13T11:30:25.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:25.470+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.473113+0000) 2019-10-13T11:30:26.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:26.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.408552+0000) 2019-10-13T11:30:27.889 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:27.882+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:27.888108+0000) 2019-10-13T11:30:28.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:28.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.671994+0000) 2019-10-13T11:30:28.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:28.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.676920+0000) 2019-10-13T11:30:28.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:28.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.688359+0000) 2019-10-13T11:30:29.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:29.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.683625+0000) 2019-10-13T11:30:29.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:29.682+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.685169+0000) 2019-10-13T11:30:29.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:29.682+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.684864+0000) 2019-10-13T11:30:29.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:29.682+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.685511+0000) 2019-10-13T11:30:29.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:29.682+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.685996+0000) 2019-10-13T11:30:29.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:29.682+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.686478+0000) 2019-10-13T11:30:29.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:29.682+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.686982+0000) 2019-10-13T11:30:29.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:29.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:29.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:29.686+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.690419+0000) 2019-10-13T11:30:29.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:29.902+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:29.907603+0000) 2019-10-13T11:30:32.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:32.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:32.163083+0000) 2019-10-13T11:30:34.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:34.689+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.693413+0000) 2019-10-13T11:30:34.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:34.689+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.693870+0000) 2019-10-13T11:30:34.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:34.689+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.694917+0000) 2019-10-13T11:30:34.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:34.693+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.695421+0000) 2019-10-13T11:30:34.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:34.693+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.696236+0000) 2019-10-13T11:30:34.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:34.693+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.696760+0000) 2019-10-13T11:30:34.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:34.690+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.697332+0000) 2019-10-13T11:30:34.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:34.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:34.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:34.694+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.700984+0000) 2019-10-13T11:30:35.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:35.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.473302+0000) 2019-10-13T11:30:36.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:36.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.408749+0000) 2019-10-13T11:30:37.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:37.890+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:37.894555+0000) 2019-10-13T11:30:38.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:38.669+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.672181+0000) 2019-10-13T11:30:38.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:38.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.677103+0000) 2019-10-13T11:30:38.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:38.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.688578+0000) 2019-10-13T11:30:39.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:39.701+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.703647+0000) 2019-10-13T11:30:39.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:39.698+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.704984+0000) 2019-10-13T11:30:39.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:39.701+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.705253+0000) 2019-10-13T11:30:39.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:39.701+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.706215+0000) 2019-10-13T11:30:39.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:39.701+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.706105+0000) 2019-10-13T11:30:39.707 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:39.701+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.706763+0000) 2019-10-13T11:30:39.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:39.705+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.707290+0000) 2019-10-13T11:30:39.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:39.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:39.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:39.706+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.712242+0000) 2019-10-13T11:30:39.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:39.906+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:39.907776+0000) 2019-10-13T11:30:42.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:42.161+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:42.163331+0000) 2019-10-13T11:30:44.046 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:30:44.046 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:44.050 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:30:44.050 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:44.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:44.709+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.713568+0000) 2019-10-13T11:30:44.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:44.710+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.714179+0000) 2019-10-13T11:30:44.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:44.709+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.714873+0000) 2019-10-13T11:30:44.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:44.709+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.715133+0000) 2019-10-13T11:30:44.717 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:44.713+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.715548+0000) 2019-10-13T11:30:44.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:44.713+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.715810+0000) 2019-10-13T11:30:44.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:44.713+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.716390+0000) 2019-10-13T11:30:44.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:44.717+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:44.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:44.718+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.723751+0000) 2019-10-13T11:30:45.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:45.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.473442+0000) 2019-10-13T11:30:46.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:46.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.408937+0000) 2019-10-13T11:30:47.902 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:47.898+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:47.901023+0000) 2019-10-13T11:30:48.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:48.669+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.672390+0000) 2019-10-13T11:30:48.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:48.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.677312+0000) 2019-10-13T11:30:48.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:48.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.688758+0000) 2019-10-13T11:30:49.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:49.721+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.724574+0000) 2019-10-13T11:30:49.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:49.721+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.725246+0000) 2019-10-13T11:30:49.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:49.721+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.725886+0000) 2019-10-13T11:30:49.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:49.722+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.726625+0000) 2019-10-13T11:30:49.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:49.721+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.726999+0000) 2019-10-13T11:30:49.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:49.721+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.727191+0000) 2019-10-13T11:30:49.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:49.725+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.727683+0000) 2019-10-13T11:30:49.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:49.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:49.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:49.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.731925+0000) 2019-10-13T11:30:49.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:49.906+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:49.907915+0000) 2019-10-13T11:30:52.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:52.161+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:52.163554+0000) 2019-10-13T11:30:54.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:54.729+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.733460+0000) 2019-10-13T11:30:54.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:54.729+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.733928+0000) 2019-10-13T11:30:54.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:54.729+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.734420+0000) 2019-10-13T11:30:54.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:54.729+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.734717+0000) 2019-10-13T11:30:54.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:54.729+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.735053+0000) 2019-10-13T11:30:54.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:54.730+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.735525+0000) 2019-10-13T11:30:54.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:54.733+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.736277+0000) 2019-10-13T11:30:54.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:54.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:54.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:54.738+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.743210+0000) 2019-10-13T11:30:55.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:55.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.473624+0000) 2019-10-13T11:30:56.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:56.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.409132+0000) 2019-10-13T11:30:57.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:57.906+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:57.907465+0000) 2019-10-13T11:30:58.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:58.669+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.672614+0000) 2019-10-13T11:30:58.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:58.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.677488+0000) 2019-10-13T11:30:58.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:58.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.688950+0000) 2019-10-13T11:30:59.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:30:59.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.743617+0000) 2019-10-13T11:30:59.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:30:59.741+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.743903+0000) 2019-10-13T11:30:59.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:30:59.741+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.744796+0000) 2019-10-13T11:30:59.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:59.738+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.745272+0000) 2019-10-13T11:30:59.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:30:59.741+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.745361+0000) 2019-10-13T11:30:59.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:30:59.741+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.745917+0000) 2019-10-13T11:30:59.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:30:59.741+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.747152+0000) 2019-10-13T11:30:59.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:30:59.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:30:59.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:30:59.746+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.751736+0000) 2019-10-13T11:30:59.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:30:59.906+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:59.908103+0000) 2019-10-13T11:31:02.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:02.161+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:02.163821+0000) 2019-10-13T11:31:04.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:04.749+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.753980+0000) 2019-10-13T11:31:04.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:04.749+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.754922+0000) 2019-10-13T11:31:04.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:04.750+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.755690+0000) 2019-10-13T11:31:04.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:04.753+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.755587+0000) 2019-10-13T11:31:04.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:04.753+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.756240+0000) 2019-10-13T11:31:04.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:04.753+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.756485+0000) 2019-10-13T11:31:04.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:04.753+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.756796+0000) 2019-10-13T11:31:04.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:04.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:04.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:04.758+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.764021+0000) 2019-10-13T11:31:05.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:05.473+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.473804+0000) 2019-10-13T11:31:06.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:06.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.409376+0000) 2019-10-13T11:31:07.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:07.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:07.914053+0000) 2019-10-13T11:31:08.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:08.669+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.672846+0000) 2019-10-13T11:31:08.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:08.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.677681+0000) 2019-10-13T11:31:08.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:08.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.689155+0000) 2019-10-13T11:31:09.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:09.761+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.763984+0000) 2019-10-13T11:31:09.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:09.761+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.764778+0000) 2019-10-13T11:31:09.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:09.761+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.765199+0000) 2019-10-13T11:31:09.767 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:09.762+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.766303+0000) 2019-10-13T11:31:09.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:09.761+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.766465+0000) 2019-10-13T11:31:09.767 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:09.761+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.766756+0000) 2019-10-13T11:31:09.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:09.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.767413+0000) 2019-10-13T11:31:09.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:09.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:09.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:09.766+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.771636+0000) 2019-10-13T11:31:09.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:09.906+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:09.908268+0000) 2019-10-13T11:31:12.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:12.161+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:12.164034+0000) 2019-10-13T11:31:14.189 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:31:14.189 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:14.194 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:31:14.194 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:14.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:14.769+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.774726+0000) 2019-10-13T11:31:14.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:14.769+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.775059+0000) 2019-10-13T11:31:14.777 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:14.773+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.775623+0000) 2019-10-13T11:31:14.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:14.773+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.776095+0000) 2019-10-13T11:31:14.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:14.774+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.776623+0000) 2019-10-13T11:31:14.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:14.773+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.777082+0000) 2019-10-13T11:31:14.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:14.773+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.777451+0000) 2019-10-13T11:31:14.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:14.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:14.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:14.782+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.783666+0000) 2019-10-13T11:31:15.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:15.469+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.473997+0000) 2019-10-13T11:31:16.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:16.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.409571+0000) 2019-10-13T11:31:17.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:17.922+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:17.924735+0000) 2019-10-13T11:31:18.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:18.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.673041+0000) 2019-10-13T11:31:18.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:18.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.677846+0000) 2019-10-13T11:31:18.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:18.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.689333+0000) 2019-10-13T11:31:19.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:19.776+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.783213+0000) 2019-10-13T11:31:19.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:19.780+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.783558+0000) 2019-10-13T11:31:19.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:19.780+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.784577+0000) 2019-10-13T11:31:19.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:19.780+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.784248+0000) 2019-10-13T11:31:19.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:19.780+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.785820+0000) 2019-10-13T11:31:19.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:19.780+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.786164+0000) 2019-10-13T11:31:19.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:19.785+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.786562+0000) 2019-10-13T11:31:19.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:19.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:19.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:19.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.793952+0000) 2019-10-13T11:31:19.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:19.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:19.908438+0000) 2019-10-13T11:31:22.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:22.160+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:22.164234+0000) 2019-10-13T11:31:24.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:24.788+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.792410+0000) 2019-10-13T11:31:24.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:24.788+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.792661+0000) 2019-10-13T11:31:24.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:24.789+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.793439+0000) 2019-10-13T11:31:24.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:24.788+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.794670+0000) 2019-10-13T11:31:24.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:24.788+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.795281+0000) 2019-10-13T11:31:24.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:24.792+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.795832+0000) 2019-10-13T11:31:24.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:24.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.796174+0000) 2019-10-13T11:31:24.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:24.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:24.804 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:24.801+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.803844+0000) 2019-10-13T11:31:25.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:25.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.474195+0000) 2019-10-13T11:31:26.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:26.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.409803+0000) 2019-10-13T11:31:27.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:27.933+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:27.936320+0000) 2019-10-13T11:31:28.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:28.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.673231+0000) 2019-10-13T11:31:28.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:28.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.677999+0000) 2019-10-13T11:31:28.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:28.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.689485+0000) 2019-10-13T11:31:29.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:29.796+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.802053+0000) 2019-10-13T11:31:29.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:29.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.802708+0000) 2019-10-13T11:31:29.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:29.800+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.804325+0000) 2019-10-13T11:31:29.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:29.800+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.804685+0000) 2019-10-13T11:31:29.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:29.800+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.804970+0000) 2019-10-13T11:31:29.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:29.805+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.806301+0000) 2019-10-13T11:31:29.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:29.800+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.806555+0000) 2019-10-13T11:31:29.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:29.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:29.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:29.809+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.810363+0000) 2019-10-13T11:31:29.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:29.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:29.908602+0000) 2019-10-13T11:31:32.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:32.160+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:32.164435+0000) 2019-10-13T11:31:34.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:34.808+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.812063+0000) 2019-10-13T11:31:34.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:34.808+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.813014+0000) 2019-10-13T11:31:34.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:34.808+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.814075+0000) 2019-10-13T11:31:34.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:34.808+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.814674+0000) 2019-10-13T11:31:34.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:34.808+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.813755+0000) 2019-10-13T11:31:34.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:34.813+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.815654+0000) 2019-10-13T11:31:34.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:34.812+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.815948+0000) 2019-10-13T11:31:34.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:34.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:34.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:34.821+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.822575+0000) 2019-10-13T11:31:35.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:35.472+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.474372+0000) 2019-10-13T11:31:36.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:36.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.410015+0000) 2019-10-13T11:31:37.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:37.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:37.942839+0000) 2019-10-13T11:31:38.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:38.672+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.673433+0000) 2019-10-13T11:31:38.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:38.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.678196+0000) 2019-10-13T11:31:38.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:38.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.689671+0000) 2019-10-13T11:31:39.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:39.816+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.822101+0000) 2019-10-13T11:31:39.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:39.816+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.822180+0000) 2019-10-13T11:31:39.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:39.821+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.822513+0000) 2019-10-13T11:31:39.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:39.816+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.823089+0000) 2019-10-13T11:31:39.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:39.820+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.823477+0000) 2019-10-13T11:31:39.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:39.820+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.823749+0000) 2019-10-13T11:31:39.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:39.820+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.824327+0000) 2019-10-13T11:31:39.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:39.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:39.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:39.829+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.831394+0000) 2019-10-13T11:31:39.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:39.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:39.908784+0000) 2019-10-13T11:31:42.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:42.164+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:42.164663+0000) 2019-10-13T11:31:44.309 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:31:44.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:44.313 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:31:44.313 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:44.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:44.828+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.831558+0000) 2019-10-13T11:31:44.833 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:44.828+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.832017+0000) 2019-10-13T11:31:44.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:44.828+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.832201+0000) 2019-10-13T11:31:44.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:44.828+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.832767+0000) 2019-10-13T11:31:44.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:44.829+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.833165+0000) 2019-10-13T11:31:44.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:44.828+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.833757+0000) 2019-10-13T11:31:44.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:44.828+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.834034+0000) 2019-10-13T11:31:44.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:44.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:44.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:44.837+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.840926+0000) 2019-10-13T11:31:45.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:45.468+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.474582+0000) 2019-10-13T11:31:46.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:46.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.410209+0000) 2019-10-13T11:31:47.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:47.945+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:47.949277+0000) 2019-10-13T11:31:48.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:48.668+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.673647+0000) 2019-10-13T11:31:48.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:48.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.678394+0000) 2019-10-13T11:31:48.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:48.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.689828+0000) 2019-10-13T11:31:49.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:49.836+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.840887+0000) 2019-10-13T11:31:49.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:49.836+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.840658+0000) 2019-10-13T11:31:49.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:49.836+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.841252+0000) 2019-10-13T11:31:49.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:49.836+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.842297+0000) 2019-10-13T11:31:49.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:49.841+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.842962+0000) 2019-10-13T11:31:49.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:49.836+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.843235+0000) 2019-10-13T11:31:49.844 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:49.840+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.843676+0000) 2019-10-13T11:31:49.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:49.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:49.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:49.845+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.848624+0000) 2019-10-13T11:31:49.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:49.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:49.908957+0000) 2019-10-13T11:31:52.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:52.164+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:52.164904+0000) 2019-10-13T11:31:54.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:54.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.849887+0000) 2019-10-13T11:31:54.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:54.844+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.850186+0000) 2019-10-13T11:31:54.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:54.844+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.851078+0000) 2019-10-13T11:31:54.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:54.848+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.852248+0000) 2019-10-13T11:31:54.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:54.849+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.852667+0000) 2019-10-13T11:31:54.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:54.848+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.852984+0000) 2019-10-13T11:31:54.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:54.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.853175+0000) 2019-10-13T11:31:54.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:54.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:54.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:54.853+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.857610+0000) 2019-10-13T11:31:55.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:55.472+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.474780+0000) 2019-10-13T11:31:56.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:56.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.410402+0000) 2019-10-13T11:31:57.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:57.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:57.955686+0000) 2019-10-13T11:31:58.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:58.672+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.673847+0000) 2019-10-13T11:31:58.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:58.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.678613+0000) 2019-10-13T11:31:58.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:58.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.690045+0000) 2019-10-13T11:31:59.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:31:59.856+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.859412+0000) 2019-10-13T11:31:59.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:31:59.856+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.859616+0000) 2019-10-13T11:31:59.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:31:59.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.860294+0000) 2019-10-13T11:31:59.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:31:59.856+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.860654+0000) 2019-10-13T11:31:59.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:31:59.856+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.861386+0000) 2019-10-13T11:31:59.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:59.857+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.861737+0000) 2019-10-13T11:31:59.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:31:59.856+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.861980+0000) 2019-10-13T11:31:59.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:31:59.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:31:59.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:31:59.865+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.867675+0000) 2019-10-13T11:31:59.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:31:59.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:59.909131+0000) 2019-10-13T11:32:02.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:02.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:02.165127+0000) 2019-10-13T11:32:04.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:04.865+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.869027+0000) 2019-10-13T11:32:04.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:04.863+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.869212+0000) 2019-10-13T11:32:04.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:04.863+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.870013+0000) 2019-10-13T11:32:04.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:04.863+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.871040+0000) 2019-10-13T11:32:04.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:04.863+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.870866+0000) 2019-10-13T11:32:04.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:04.867+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.871538+0000) 2019-10-13T11:32:04.873 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:04.867+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.871953+0000) 2019-10-13T11:32:04.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:04.873+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.876008+0000) 2019-10-13T11:32:04.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:04.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 125984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:05.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:05.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.474994+0000) 2019-10-13T11:32:06.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:06.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.410587+0000) 2019-10-13T11:32:07.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:07.961+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:07.962184+0000) 2019-10-13T11:32:08.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:08.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.674047+0000) 2019-10-13T11:32:08.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:08.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.678823+0000) 2019-10-13T11:32:08.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:08.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.690242+0000) 2019-10-13T11:32:09.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:09.875+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.879027+0000) 2019-10-13T11:32:09.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:09.877+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.880366+0000) 2019-10-13T11:32:09.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:09.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.880578+0000) 2019-10-13T11:32:09.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:09.875+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.881141+0000) 2019-10-13T11:32:09.883 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:09.875+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.881585+0000) 2019-10-13T11:32:09.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:09.875+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.881863+0000) 2019-10-13T11:32:09.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:09.875+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.882812+0000) 2019-10-13T11:32:09.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:09.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:09.889 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:09.885+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.888619+0000) 2019-10-13T11:32:09.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:09.905+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:09.909296+0000) 2019-10-13T11:32:12.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:12.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:12.165326+0000) 2019-10-13T11:32:14.449 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:32:14.450 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:14.453 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:32:14.453 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:14.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:14.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.889723+0000) 2019-10-13T11:32:14.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:14.883+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.890181+0000) 2019-10-13T11:32:14.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:14.883+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.890777+0000) 2019-10-13T11:32:14.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:14.887+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.891640+0000) 2019-10-13T11:32:14.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:14.888+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.891983+0000) 2019-10-13T11:32:14.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:14.887+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.892609+0000) 2019-10-13T11:32:14.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:14.887+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.892639+0000) 2019-10-13T11:32:14.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:14.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:14.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:14.896+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.898750+0000) 2019-10-13T11:32:15.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:15.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.475178+0000) 2019-10-13T11:32:16.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:16.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.410781+0000) 2019-10-13T11:32:17.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:17.964+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:17.968755+0000) 2019-10-13T11:32:18.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:18.667+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.674243+0000) 2019-10-13T11:32:18.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:18.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.679055+0000) 2019-10-13T11:32:18.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:18.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.690440+0000) 2019-10-13T11:32:19.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:19.895+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.899083+0000) 2019-10-13T11:32:19.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:19.895+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.899497+0000) 2019-10-13T11:32:19.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:19.895+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.900167+0000) 2019-10-13T11:32:19.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:19.896+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.900707+0000) 2019-10-13T11:32:19.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:19.895+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.900937+0000) 2019-10-13T11:32:19.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:19.895+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.901554+0000) 2019-10-13T11:32:19.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:19.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.902390+0000) 2019-10-13T11:32:19.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:19.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:19.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:19.904+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.907469+0000) 2019-10-13T11:32:19.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:19.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:19.909443+0000) 2019-10-13T11:32:22.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:22.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:22.165530+0000) 2019-10-13T11:32:24.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:24.903+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.908430+0000) 2019-10-13T11:32:24.910 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:24.903+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.908733+0000) 2019-10-13T11:32:24.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:24.903+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.909223+0000) 2019-10-13T11:32:24.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:24.903+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.910174+0000) 2019-10-13T11:32:24.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:24.903+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.910957+0000) 2019-10-13T11:32:24.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:24.907+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.911429+0000) 2019-10-13T11:32:24.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:24.908+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.911813+0000) 2019-10-13T11:32:24.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:24.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:24.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:24.916+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.918399+0000) 2019-10-13T11:32:25.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:25.471+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.475395+0000) 2019-10-13T11:32:26.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:26.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.410978+0000) 2019-10-13T11:32:27.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:27.972+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:27.975198+0000) 2019-10-13T11:32:28.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:28.671+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.674471+0000) 2019-10-13T11:32:28.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:28.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.679253+0000) 2019-10-13T11:32:28.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:28.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.690638+0000) 2019-10-13T11:32:29.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:29.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.909643+0000) 2019-10-13T11:32:29.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:29.911+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.917624+0000) 2019-10-13T11:32:29.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:29.911+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.918016+0000) 2019-10-13T11:32:29.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:29.911+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.918387+0000) 2019-10-13T11:32:29.920 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:29.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.919035+0000) 2019-10-13T11:32:29.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:29.915+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.920047+0000) 2019-10-13T11:32:29.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:29.915+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.920174+0000) 2019-10-13T11:32:29.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:29.916+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.921191+0000) 2019-10-13T11:32:29.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:29.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:29.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:29.924+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:29.927046+0000) 2019-10-13T11:32:32.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:32.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:32.165757+0000) 2019-10-13T11:32:34.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:34.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.927732+0000) 2019-10-13T11:32:34.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:34.927+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.928144+0000) 2019-10-13T11:32:34.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:34.927+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.928363+0000) 2019-10-13T11:32:34.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:34.927+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.928804+0000) 2019-10-13T11:32:34.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:34.927+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.929136+0000) 2019-10-13T11:32:34.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:34.927+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.929886+0000) 2019-10-13T11:32:34.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:34.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.930438+0000) 2019-10-13T11:32:34.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:34.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:34.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:34.928+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.933802+0000) 2019-10-13T11:32:35.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:35.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.475603+0000) 2019-10-13T11:32:36.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:36.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.411178+0000) 2019-10-13T11:32:37.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:37.976+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:37.981711+0000) 2019-10-13T11:32:38.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:38.671+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.674706+0000) 2019-10-13T11:32:38.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:38.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.679506+0000) 2019-10-13T11:32:38.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:38.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.690815+0000) 2019-10-13T11:32:39.910 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:39.904+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.909868+0000) 2019-10-13T11:32:39.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:39.935+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.935886+0000) 2019-10-13T11:32:39.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:39.932+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.936623+0000) 2019-10-13T11:32:39.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:39.935+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.936907+0000) 2019-10-13T11:32:39.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:39.935+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.937301+0000) 2019-10-13T11:32:39.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:39.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.937893+0000) 2019-10-13T11:32:39.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:39.935+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.938129+0000) 2019-10-13T11:32:39.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:39.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.938384+0000) 2019-10-13T11:32:39.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:39.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126220 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:39.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:39.940+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:39.942187+0000) 2019-10-13T11:32:42.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:42.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:42.165963+0000) 2019-10-13T11:32:44.497 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:32:44.498 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:44.501 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:32:44.501 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:44.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:44.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.945139+0000) 2019-10-13T11:32:44.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:44.942+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.946140+0000) 2019-10-13T11:32:44.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:44.942+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.946332+0000) 2019-10-13T11:32:44.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:44.942+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.946711+0000) 2019-10-13T11:32:44.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:44.944+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.948093+0000) 2019-10-13T11:32:44.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:44.946+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.948404+0000) 2019-10-13T11:32:44.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:44.946+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.948631+0000) 2019-10-13T11:32:44.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:44.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:44.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:44.952+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.954186+0000) 2019-10-13T11:32:45.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:45.474+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.475788+0000) 2019-10-13T11:32:46.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:46.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.411397+0000) 2019-10-13T11:32:47.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:47.984+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:47.988280+0000) 2019-10-13T11:32:48.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:48.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.674908+0000) 2019-10-13T11:32:48.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:48.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.679701+0000) 2019-10-13T11:32:48.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:48.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.691041+0000) 2019-10-13T11:32:49.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:49.908+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.910063+0000) 2019-10-13T11:32:49.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:49.950+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.954676+0000) 2019-10-13T11:32:49.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:49.954+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.955654+0000) 2019-10-13T11:32:49.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:49.954+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.956424+0000) 2019-10-13T11:32:49.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:49.954+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.956798+0000) 2019-10-13T11:32:49.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:49.952+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.957406+0000) 2019-10-13T11:32:49.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:49.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.957534+0000) 2019-10-13T11:32:49.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:49.954+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.957891+0000) 2019-10-13T11:32:49.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:49.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:49.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:49.960+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:49.962612+0000) 2019-10-13T11:32:52.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:52.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:52.166173+0000) 2019-10-13T11:32:54.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:54.962+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.964959+0000) 2019-10-13T11:32:54.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:54.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.965385+0000) 2019-10-13T11:32:54.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:54.962+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.966039+0000) 2019-10-13T11:32:54.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:54.962+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.966093+0000) 2019-10-13T11:32:54.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:54.964+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.966861+0000) 2019-10-13T11:32:54.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:54.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.967276+0000) 2019-10-13T11:32:54.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:54.966+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.967806+0000) 2019-10-13T11:32:54.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:54.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:54.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:54.968+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.971505+0000) 2019-10-13T11:32:55.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:55.474+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.475973+0000) 2019-10-13T11:32:56.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:56.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.411626+0000) 2019-10-13T11:32:57.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:57.992+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:57.994860+0000) 2019-10-13T11:32:58.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:58.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.675083+0000) 2019-10-13T11:32:58.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:58.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.679926+0000) 2019-10-13T11:32:58.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:58.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.691245+0000) 2019-10-13T11:32:59.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:59.908+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.910262+0000) 2019-10-13T11:32:59.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:32:59.970+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.973434+0000) 2019-10-13T11:32:59.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:32:59.970+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.974010+0000) 2019-10-13T11:32:59.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:32:59.970+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.974053+0000) 2019-10-13T11:32:59.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:32:59.972+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.975144+0000) 2019-10-13T11:32:59.976 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:32:59.974+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.975886+0000) 2019-10-13T11:32:59.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:32:59.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.976212+0000) 2019-10-13T11:32:59.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:32:59.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.976965+0000) 2019-10-13T11:32:59.980 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:32:59.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:32:59.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:32:59.980+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:59.983023+0000) 2019-10-13T11:33:02.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:02.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:02.166410+0000) 2019-10-13T11:33:04.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:04.978+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.982619+0000) 2019-10-13T11:33:04.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:04.978+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.983187+0000) 2019-10-13T11:33:04.985 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:04.982+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.983827+0000) 2019-10-13T11:33:04.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:04.982+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.984137+0000) 2019-10-13T11:33:04.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:04.980+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.984713+0000) 2019-10-13T11:33:04.986 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:04.982+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.985005+0000) 2019-10-13T11:33:04.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:04.982+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.985571+0000) 2019-10-13T11:33:04.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:04.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:04.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:04.988+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.992683+0000) 2019-10-13T11:33:05.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:05.474+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.476177+0000) 2019-10-13T11:33:06.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:06.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.411821+0000) 2019-10-13T11:33:08.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:08.000+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.001288+0000) 2019-10-13T11:33:08.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:08.670+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.675273+0000) 2019-10-13T11:33:08.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:08.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.680139+0000) 2019-10-13T11:33:08.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:08.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.691498+0000) 2019-10-13T11:33:09.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:09.907+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.910501+0000) 2019-10-13T11:33:09.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:09.990+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.991895+0000) 2019-10-13T11:33:09.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:09.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.992439+0000) 2019-10-13T11:33:09.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:09.990+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.993359+0000) 2019-10-13T11:33:09.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:09.987+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.993769+0000) 2019-10-13T11:33:09.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:09.990+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.994359+0000) 2019-10-13T11:33:09.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:09.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.995038+0000) 2019-10-13T11:33:09.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:09.994+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:09.995494+0000) 2019-10-13T11:33:09.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:09.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:10.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:09.995+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.001117+0000) 2019-10-13T11:33:12.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:12.162+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:12.166594+0000) 2019-10-13T11:33:14.621 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:33:14.621 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:14.625 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:33:14.625 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:15.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:14.998+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.001695+0000) 2019-10-13T11:33:15.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:14.998+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.002067+0000) 2019-10-13T11:33:15.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:14.998+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.002948+0000) 2019-10-13T11:33:15.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:14.999+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.003452+0000) 2019-10-13T11:33:15.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:15.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.003939+0000) 2019-10-13T11:33:15.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:15.002+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.004035+0000) 2019-10-13T11:33:15.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:15.002+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.004459+0000) 2019-10-13T11:33:15.010 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:15.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:15.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:15.007+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.012051+0000) 2019-10-13T11:33:15.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:15.474+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.476385+0000) 2019-10-13T11:33:16.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:16.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:16.412015+0000) 2019-10-13T11:33:18.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:18.003+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.007751+0000) 2019-10-13T11:33:18.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:18.674+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.675507+0000) 2019-10-13T11:33:18.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:18.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.680344+0000) 2019-10-13T11:33:18.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:18.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.691696+0000) 2019-10-13T11:33:19.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:19.907+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:19.910727+0000) 2019-10-13T11:33:20.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:20.010+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.012019+0000) 2019-10-13T11:33:20.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:20.010+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.012915+0000) 2019-10-13T11:33:20.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:20.010+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.013310+0000) 2019-10-13T11:33:20.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:20.010+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.013780+0000) 2019-10-13T11:33:20.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:20.010+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.014114+0000) 2019-10-13T11:33:20.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:20.011+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.014622+0000) 2019-10-13T11:33:20.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:20.010+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.015082+0000) 2019-10-13T11:33:20.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:20.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:20.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:20.015+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.021319+0000) 2019-10-13T11:33:22.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:22.166+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.166821+0000) 2019-10-13T11:33:25.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:25.018+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.022924+0000) 2019-10-13T11:33:25.024 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:25.022+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.023683+0000) 2019-10-13T11:33:25.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:25.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.024879+0000) 2019-10-13T11:33:25.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:25.022+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.025154+0000) 2019-10-13T11:33:25.026 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:25.022+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.025636+0000) 2019-10-13T11:33:25.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:25.023+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.026467+0000) 2019-10-13T11:33:25.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:25.022+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.027020+0000) 2019-10-13T11:33:25.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:25.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:25.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:25.027+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.030954+0000) 2019-10-13T11:33:25.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:25.474+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.476576+0000) 2019-10-13T11:33:26.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:26.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:26.412212+0000) 2019-10-13T11:33:28.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:28.011+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.014280+0000) 2019-10-13T11:33:28.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:28.673+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.675704+0000) 2019-10-13T11:33:28.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:28.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.680555+0000) 2019-10-13T11:33:28.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:28.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.691921+0000) 2019-10-13T11:33:29.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:29.907+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:29.910917+0000) 2019-10-13T11:33:30.034 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:30.029+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.032938+0000) 2019-10-13T11:33:30.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:30.029+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.033937+0000) 2019-10-13T11:33:30.035 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:30.029+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.033330+0000) 2019-10-13T11:33:30.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:30.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.034154+0000) 2019-10-13T11:33:30.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:30.029+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.034411+0000) 2019-10-13T11:33:30.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:30.031+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.035086+0000) 2019-10-13T11:33:30.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:30.033+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.035790+0000) 2019-10-13T11:33:30.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:30.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:30.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:30.039+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.042620+0000) 2019-10-13T11:33:32.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:32.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.167039+0000) 2019-10-13T11:33:35.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:35.035+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.042003+0000) 2019-10-13T11:33:35.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:35.037+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.042921+0000) 2019-10-13T11:33:35.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:35.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.043273+0000) 2019-10-13T11:33:35.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:35.041+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.043731+0000) 2019-10-13T11:33:35.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:35.041+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.044508+0000) 2019-10-13T11:33:35.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:35.041+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.045404+0000) 2019-10-13T11:33:35.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:35.041+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.045747+0000) 2019-10-13T11:33:35.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:35.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:35.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:35.043+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.049728+0000) 2019-10-13T11:33:35.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:35.473+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.476779+0000) 2019-10-13T11:33:36.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:36.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:36.412435+0000) 2019-10-13T11:33:38.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:38.019+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.020874+0000) 2019-10-13T11:33:38.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:38.673+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.675917+0000) 2019-10-13T11:33:38.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:38.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.680769+0000) 2019-10-13T11:33:38.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:38.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.692112+0000) 2019-10-13T11:33:39.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:39.907+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:39.911114+0000) 2019-10-13T11:33:40.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:40.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.050650+0000) 2019-10-13T11:33:40.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:40.049+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.051554+0000) 2019-10-13T11:33:40.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:40.049+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.052171+0000) 2019-10-13T11:33:40.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:40.047+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.053318+0000) 2019-10-13T11:33:40.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:40.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.053607+0000) 2019-10-13T11:33:40.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:40.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.054090+0000) 2019-10-13T11:33:40.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:40.049+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.054461+0000) 2019-10-13T11:33:40.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:40.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:40.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:40.055+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.059223+0000) 2019-10-13T11:33:42.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:42.161+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.167260+0000) 2019-10-13T11:33:44.710 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:33:44.711 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:44.714 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:33:44.714 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:45.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:45.057+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.060884+0000) 2019-10-13T11:33:45.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:45.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.061542+0000) 2019-10-13T11:33:45.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:45.057+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.062790+0000) 2019-10-13T11:33:45.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:45.059+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.063065+0000) 2019-10-13T11:33:45.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:45.061+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.063359+0000) 2019-10-13T11:33:45.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:45.061+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.064015+0000) 2019-10-13T11:33:45.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:45.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.064690+0000) 2019-10-13T11:33:45.068 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:45.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:45.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:45.063+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.067824+0000) 2019-10-13T11:33:45.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:45.473+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.476962+0000) 2019-10-13T11:33:46.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:46.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:46.412641+0000) 2019-10-13T11:33:48.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:48.023+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.026692+0000) 2019-10-13T11:33:48.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:48.673+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.676181+0000) 2019-10-13T11:33:48.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:48.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.680967+0000) 2019-10-13T11:33:48.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:48.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.692324+0000) 2019-10-13T11:33:49.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:49.911+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:49.911320+0000) 2019-10-13T11:33:50.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:50.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.070015+0000) 2019-10-13T11:33:50.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:50.065+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.070753+0000) 2019-10-13T11:33:50.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:50.065+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.071064+0000) 2019-10-13T11:33:50.072 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:50.069+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.072120+0000) 2019-10-13T11:33:50.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:50.067+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.072506+0000) 2019-10-13T11:33:50.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:50.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.073091+0000) 2019-10-13T11:33:50.074 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:50.069+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.073496+0000) 2019-10-13T11:33:50.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:50.069+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.073943+0000) 2019-10-13T11:33:50.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:50.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:52.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:52.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.167556+0000) 2019-10-13T11:33:55.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:55.077+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.080263+0000) 2019-10-13T11:33:55.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:55.075+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.080582+0000) 2019-10-13T11:33:55.082 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:55.077+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.081332+0000) 2019-10-13T11:33:55.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:55.077+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.081864+0000) 2019-10-13T11:33:55.083 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:55.077+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.082606+0000) 2019-10-13T11:33:55.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:33:55.077+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.082852+0000) 2019-10-13T11:33:55.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:55.081+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.083538+0000) 2019-10-13T11:33:55.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:33:55.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:33:55.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:55.083+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.088565+0000) 2019-10-13T11:33:55.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:33:55.473+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.477144+0000) 2019-10-13T11:33:56.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:33:56.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:56.412835+0000) 2019-10-13T11:33:58.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:33:58.031+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.033095+0000) 2019-10-13T11:33:58.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:33:58.673+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.676376+0000) 2019-10-13T11:33:58.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:33:58.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.681191+0000) 2019-10-13T11:33:58.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:33:58.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.692521+0000) 2019-10-13T11:33:59.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:33:59.911+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:59.911512+0000) 2019-10-13T11:34:00.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:00.085+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.089617+0000) 2019-10-13T11:34:00.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:00.087+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.090929+0000) 2019-10-13T11:34:00.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:00.085+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.091255+0000) 2019-10-13T11:34:00.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:00.089+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.091697+0000) 2019-10-13T11:34:00.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:00.089+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.092236+0000) 2019-10-13T11:34:00.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:00.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.092402+0000) 2019-10-13T11:34:00.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:00.089+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.093373+0000) 2019-10-13T11:34:00.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:00.091+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.095277+0000) 2019-10-13T11:34:00.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:00.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:02.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:02.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.167753+0000) 2019-10-13T11:34:05.101 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:05.097+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.099850+0000) 2019-10-13T11:34:05.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:05.097+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.100250+0000) 2019-10-13T11:34:05.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:05.094+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.100560+0000) 2019-10-13T11:34:05.102 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:05.097+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.101248+0000) 2019-10-13T11:34:05.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:05.097+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.101569+0000) 2019-10-13T11:34:05.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:05.097+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.101881+0000) 2019-10-13T11:34:05.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:05.097+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.103238+0000) 2019-10-13T11:34:05.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:05.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:05.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:05.106+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.110325+0000) 2019-10-13T11:34:05.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:05.477+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.477340+0000) 2019-10-13T11:34:06.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:06.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.413027+0000) 2019-10-13T11:34:08.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:08.034+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.039652+0000) 2019-10-13T11:34:08.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:08.673+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.676621+0000) 2019-10-13T11:34:08.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:08.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.681384+0000) 2019-10-13T11:34:08.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:08.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.692722+0000) 2019-10-13T11:34:09.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:09.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:09.911722+0000) 2019-10-13T11:34:10.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:10.105+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.109019+0000) 2019-10-13T11:34:10.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:10.105+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.110159+0000) 2019-10-13T11:34:10.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:10.105+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.111085+0000) 2019-10-13T11:34:10.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:10.109+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.111367+0000) 2019-10-13T11:34:10.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:10.109+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.111703+0000) 2019-10-13T11:34:10.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:10.109+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.112338+0000) 2019-10-13T11:34:10.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:10.106+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.112937+0000) 2019-10-13T11:34:10.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:10.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:10.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:10.110+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:10.115738+0000) 2019-10-13T11:34:12.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:12.164+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.167956+0000) 2019-10-13T11:34:14.775 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:34:14.775 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:14.779 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:34:14.779 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:15.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:15.112+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.118551+0000) 2019-10-13T11:34:15.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:15.116+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.119651+0000) 2019-10-13T11:34:15.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:15.116+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.119992+0000) 2019-10-13T11:34:15.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:15.116+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.120420+0000) 2019-10-13T11:34:15.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:15.116+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.121292+0000) 2019-10-13T11:34:15.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:15.116+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.121625+0000) 2019-10-13T11:34:15.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:15.118+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.122424+0000) 2019-10-13T11:34:15.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:15.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:15.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:15.126+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.131612+0000) 2019-10-13T11:34:15.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:15.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.477558+0000) 2019-10-13T11:34:16.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:16.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.413249+0000) 2019-10-13T11:34:18.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:18.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.049363+0000) 2019-10-13T11:34:18.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:18.672+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.676851+0000) 2019-10-13T11:34:18.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:18.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.681605+0000) 2019-10-13T11:34:18.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:18.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.692952+0000) 2019-10-13T11:34:19.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:19.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:19.911921+0000) 2019-10-13T11:34:20.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:20.124+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.128882+0000) 2019-10-13T11:34:20.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:20.124+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.129400+0000) 2019-10-13T11:34:20.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:20.124+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.129817+0000) 2019-10-13T11:34:20.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:20.124+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.130621+0000) 2019-10-13T11:34:20.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:20.124+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.131202+0000) 2019-10-13T11:34:20.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:20.128+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.131779+0000) 2019-10-13T11:34:20.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:20.126+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.133011+0000) 2019-10-13T11:34:20.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:20.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126890 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:20.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:20.134+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:20.138713+0000) 2019-10-13T11:34:22.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:22.164+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.168179+0000) 2019-10-13T11:34:25.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:25.132+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.138286+0000) 2019-10-13T11:34:25.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:25.132+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.138780+0000) 2019-10-13T11:34:25.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:25.136+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.139790+0000) 2019-10-13T11:34:25.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:25.136+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.139681+0000) 2019-10-13T11:34:25.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:25.134+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.141007+0000) 2019-10-13T11:34:25.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:25.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.141510+0000) 2019-10-13T11:34:25.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:25.136+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.141978+0000) 2019-10-13T11:34:25.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:25.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:25.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:25.142+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.146498+0000) 2019-10-13T11:34:25.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:25.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.477689+0000) 2019-10-13T11:34:26.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:26.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.413440+0000) 2019-10-13T11:34:28.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:28.054+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.055825+0000) 2019-10-13T11:34:28.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:28.672+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.677046+0000) 2019-10-13T11:34:28.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:28.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.681832+0000) 2019-10-13T11:34:28.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:28.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.693155+0000) 2019-10-13T11:34:29.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:29.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:29.912130+0000) 2019-10-13T11:34:30.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:30.142+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.148715+0000) 2019-10-13T11:34:30.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:30.144+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.149216+0000) 2019-10-13T11:34:30.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:30.144+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.149406+0000) 2019-10-13T11:34:30.151 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:30.144+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.149842+0000) 2019-10-13T11:34:30.151 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:30.144+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.150136+0000) 2019-10-13T11:34:30.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:30.144+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.150823+0000) 2019-10-13T11:34:30.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:30.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.151426+0000) 2019-10-13T11:34:30.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:30.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:30.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:30.150+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:30.156061+0000) 2019-10-13T11:34:32.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:32.164+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.168390+0000) 2019-10-13T11:34:35.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:35.152+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.157521+0000) 2019-10-13T11:34:35.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:35.152+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.158121+0000) 2019-10-13T11:34:35.159 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:35.152+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.158960+0000) 2019-10-13T11:34:35.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:35.152+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.159227+0000) 2019-10-13T11:34:35.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:35.154+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.159846+0000) 2019-10-13T11:34:35.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:35.156+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.160857+0000) 2019-10-13T11:34:35.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:35.156+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.161056+0000) 2019-10-13T11:34:35.164 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:35.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 126989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:35.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:35.158+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.165268+0000) 2019-10-13T11:34:35.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:35.472+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.477870+0000) 2019-10-13T11:34:36.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:36.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.413677+0000) 2019-10-13T11:34:38.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:38.058+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.062188+0000) 2019-10-13T11:34:38.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:38.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.677248+0000) 2019-10-13T11:34:38.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:38.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.682028+0000) 2019-10-13T11:34:38.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:38.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.693356+0000) 2019-10-13T11:34:39.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:39.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:39.912323+0000) 2019-10-13T11:34:40.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:40.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.167355+0000) 2019-10-13T11:34:40.169 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:40.162+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.167824+0000) 2019-10-13T11:34:40.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:40.164+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.168158+0000) 2019-10-13T11:34:40.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:40.164+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.168719+0000) 2019-10-13T11:34:40.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:40.162+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.168602+0000) 2019-10-13T11:34:40.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:40.164+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.169168+0000) 2019-10-13T11:34:40.171 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:40.164+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.170135+0000) 2019-10-13T11:34:40.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:40.164+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:40.170942+0000) 2019-10-13T11:34:40.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:40.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:42.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:42.168+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.168589+0000) 2019-10-13T11:34:44.832 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:34:44.832 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:44.836 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:34:44.836 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:45.178 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:45.172+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.176901+0000) 2019-10-13T11:34:45.178 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:45.172+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.177614+0000) 2019-10-13T11:34:45.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:45.170+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.178008+0000) 2019-10-13T11:34:45.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:45.172+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.178295+0000) 2019-10-13T11:34:45.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:45.172+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.178545+0000) 2019-10-13T11:34:45.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:45.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.179907+0000) 2019-10-13T11:34:45.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:45.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.180257+0000) 2019-10-13T11:34:45.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:45.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127057 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:45.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:45.182+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.186276+0000) 2019-10-13T11:34:45.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:45.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.478078+0000) 2019-10-13T11:34:46.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:46.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.413922+0000) 2019-10-13T11:34:48.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:48.066+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.068521+0000) 2019-10-13T11:34:48.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:48.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.677468+0000) 2019-10-13T11:34:48.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:48.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.682224+0000) 2019-10-13T11:34:48.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:48.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.693567+0000) 2019-10-13T11:34:49.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:49.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:49.912481+0000) 2019-10-13T11:34:50.186 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:50.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.185268+0000) 2019-10-13T11:34:50.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:50.180+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.185862+0000) 2019-10-13T11:34:50.187 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:50.180+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.186503+0000) 2019-10-13T11:34:50.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:50.182+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.186986+0000) 2019-10-13T11:34:50.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:50.184+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.187298+0000) 2019-10-13T11:34:50.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:50.184+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.187965+0000) 2019-10-13T11:34:50.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:50.184+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.188136+0000) 2019-10-13T11:34:50.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:50.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:50.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:50.190+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:50.194409+0000) 2019-10-13T11:34:52.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:52.168+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.168794+0000) 2019-10-13T11:34:55.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:55.187+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.194677+0000) 2019-10-13T11:34:55.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:55.194+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.195440+0000) 2019-10-13T11:34:55.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:55.191+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.196057+0000) 2019-10-13T11:34:55.197 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:55.191+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.196417+0000) 2019-10-13T11:34:55.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:55.191+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.197300+0000) 2019-10-13T11:34:55.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:34:55.191+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.197625+0000) 2019-10-13T11:34:55.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:55.191+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.198188+0000) 2019-10-13T11:34:55.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:34:55.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:34:55.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:55.202+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.205528+0000) 2019-10-13T11:34:55.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:34:55.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.478259+0000) 2019-10-13T11:34:56.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:34:56.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.414135+0000) 2019-10-13T11:34:58.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:34:58.073+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.075075+0000) 2019-10-13T11:34:58.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:34:58.675+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.677669+0000) 2019-10-13T11:34:58.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:34:58.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.682454+0000) 2019-10-13T11:34:58.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:34:58.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.693764+0000) 2019-10-13T11:34:59.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:34:59.909+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:59.912671+0000) 2019-10-13T11:35:00.205 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:00.199+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.203814+0000) 2019-10-13T11:35:00.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:00.199+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.204173+0000) 2019-10-13T11:35:00.206 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:00.199+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.204752+0000) 2019-10-13T11:35:00.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:00.199+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.205230+0000) 2019-10-13T11:35:00.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:00.199+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.206125+0000) 2019-10-13T11:35:00.207 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:00.205+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.206924+0000) 2019-10-13T11:35:00.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:00.199+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.207137+0000) 2019-10-13T11:35:00.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:00.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:00.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:00.213+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:00.216416+0000) 2019-10-13T11:35:02.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:02.167+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.168992+0000) 2019-10-13T11:35:05.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:05.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.212912+0000) 2019-10-13T11:35:05.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:05.209+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.213471+0000) 2019-10-13T11:35:05.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:05.207+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.213941+0000) 2019-10-13T11:35:05.216 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:05.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.214300+0000) 2019-10-13T11:35:05.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:05.207+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.214824+0000) 2019-10-13T11:35:05.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:05.207+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.215222+0000) 2019-10-13T11:35:05.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:05.211+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.215830+0000) 2019-10-13T11:35:05.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:05.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:05.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:05.217+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.221167+0000) 2019-10-13T11:35:05.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:05.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.478494+0000) 2019-10-13T11:35:06.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:06.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.414282+0000) 2019-10-13T11:35:08.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:08.077+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.081452+0000) 2019-10-13T11:35:08.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:08.675+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.677916+0000) 2019-10-13T11:35:08.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:08.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.682647+0000) 2019-10-13T11:35:08.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:08.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.693958+0000) 2019-10-13T11:35:09.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:09.909+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:09.912884+0000) 2019-10-13T11:35:10.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:10.219+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.222673+0000) 2019-10-13T11:35:10.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:10.219+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.223403+0000) 2019-10-13T11:35:10.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:10.219+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.223889+0000) 2019-10-13T11:35:10.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:10.221+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.224178+0000) 2019-10-13T11:35:10.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:10.219+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.224888+0000) 2019-10-13T11:35:10.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:10.219+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.225510+0000) 2019-10-13T11:35:10.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:10.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.225858+0000) 2019-10-13T11:35:10.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:10.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:10.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:10.229+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:10.231967+0000) 2019-10-13T11:35:12.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:12.167+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.169190+0000) 2019-10-13T11:35:14.948 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:35:14.948 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:14.952 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:35:14.952 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:15.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:15.227+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.232912+0000) 2019-10-13T11:35:15.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:15.229+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.233662+0000) 2019-10-13T11:35:15.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:15.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.234142+0000) 2019-10-13T11:35:15.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:15.227+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.234556+0000) 2019-10-13T11:35:15.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:15.227+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.235170+0000) 2019-10-13T11:35:15.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:15.231+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.235560+0000) 2019-10-13T11:35:15.237 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:15.231+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.236332+0000) 2019-10-13T11:35:15.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:15.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:15.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:15.237+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.241425+0000) 2019-10-13T11:35:15.479 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:15.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.478653+0000) 2019-10-13T11:35:16.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:16.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.414476+0000) 2019-10-13T11:35:18.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:18.085+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.087904+0000) 2019-10-13T11:35:18.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:18.675+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.678116+0000) 2019-10-13T11:35:18.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:18.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.682877+0000) 2019-10-13T11:35:18.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:18.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.694208+0000) 2019-10-13T11:35:19.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:19.909+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:19.913076+0000) 2019-10-13T11:35:20.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:20.239+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.242885+0000) 2019-10-13T11:35:20.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:20.239+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.243490+0000) 2019-10-13T11:35:20.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:20.239+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.243828+0000) 2019-10-13T11:35:20.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:20.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.244344+0000) 2019-10-13T11:35:20.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:20.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.245053+0000) 2019-10-13T11:35:20.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:20.239+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.245351+0000) 2019-10-13T11:35:20.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:20.239+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.245890+0000) 2019-10-13T11:35:20.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:20.245+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:20.248281+0000) 2019-10-13T11:35:20.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:20.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:22.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:22.163+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.169398+0000) 2019-10-13T11:35:25.253 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:25.251+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.252440+0000) 2019-10-13T11:35:25.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:25.251+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.252877+0000) 2019-10-13T11:35:25.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:25.251+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.253190+0000) 2019-10-13T11:35:25.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:25.251+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.253986+0000) 2019-10-13T11:35:25.255 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:25.253+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.254425+0000) 2019-10-13T11:35:25.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:25.251+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.254995+0000) 2019-10-13T11:35:25.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:25.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.255466+0000) 2019-10-13T11:35:25.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:25.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127329 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:25.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:25.257+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.261332+0000) 2019-10-13T11:35:25.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:25.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.478835+0000) 2019-10-13T11:35:26.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:26.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.414673+0000) 2019-10-13T11:35:28.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:28.093+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.094250+0000) 2019-10-13T11:35:28.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:28.675+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.678351+0000) 2019-10-13T11:35:28.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:28.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.683071+0000) 2019-10-13T11:35:28.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:28.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.694401+0000) 2019-10-13T11:35:29.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:29.909+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:29.913272+0000) 2019-10-13T11:35:30.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:30.259+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.261052+0000) 2019-10-13T11:35:30.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:30.259+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.261807+0000) 2019-10-13T11:35:30.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:30.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.262358+0000) 2019-10-13T11:35:30.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:30.259+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.262163+0000) 2019-10-13T11:35:30.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:30.259+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.262665+0000) 2019-10-13T11:35:30.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:30.261+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.262963+0000) 2019-10-13T11:35:30.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:30.263+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.263329+0000) 2019-10-13T11:35:30.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:30.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127367 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:30.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:30.269+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:30.271025+0000) 2019-10-13T11:35:32.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:32.167+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.169606+0000) 2019-10-13T11:35:35.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:35.269+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.271141+0000) 2019-10-13T11:35:35.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:35.267+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.271015+0000) 2019-10-13T11:35:35.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:35.271+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.272095+0000) 2019-10-13T11:35:35.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:35.271+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.272674+0000) 2019-10-13T11:35:35.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:35.271+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.273341+0000) 2019-10-13T11:35:35.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:35.271+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.272999+0000) 2019-10-13T11:35:35.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:35.271+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.274367+0000) 2019-10-13T11:35:35.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:35.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:35.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:35.277+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.281031+0000) 2019-10-13T11:35:35.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:35.475+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.479027+0000) 2019-10-13T11:35:36.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:36.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.414899+0000) 2019-10-13T11:35:38.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:38.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.100985+0000) 2019-10-13T11:35:38.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:38.674+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.678552+0000) 2019-10-13T11:35:38.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:38.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.683274+0000) 2019-10-13T11:35:38.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:38.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.694609+0000) 2019-10-13T11:35:39.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:39.913+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:39.913473+0000) 2019-10-13T11:35:40.281 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:40.278+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.280542+0000) 2019-10-13T11:35:40.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:40.278+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.280950+0000) 2019-10-13T11:35:40.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:40.278+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.281938+0000) 2019-10-13T11:35:40.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:40.281+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.282379+0000) 2019-10-13T11:35:40.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:40.282+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.283345+0000) 2019-10-13T11:35:40.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:40.282+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.283362+0000) 2019-10-13T11:35:40.284 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:40.282+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.283579+0000) 2019-10-13T11:35:40.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:40.286+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:40.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:40.285+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:40.289257+0000) 2019-10-13T11:35:42.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:42.166+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.169806+0000) 2019-10-13T11:35:45.078 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:35:45.078 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:45.082 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:35:45.082 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:45.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:45.290+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.291717+0000) 2019-10-13T11:35:45.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:45.289+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.292091+0000) 2019-10-13T11:35:45.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:45.290+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.292390+0000) 2019-10-13T11:35:45.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:45.290+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.293028+0000) 2019-10-13T11:35:45.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:45.290+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.293366+0000) 2019-10-13T11:35:45.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:45.290+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.294569+0000) 2019-10-13T11:35:45.296 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:45.290+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.294545+0000) 2019-10-13T11:35:45.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:45.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:45.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:45.297+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.301570+0000) 2019-10-13T11:35:45.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:45.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.479265+0000) 2019-10-13T11:35:46.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:46.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.415093+0000) 2019-10-13T11:35:48.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:48.105+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.106843+0000) 2019-10-13T11:35:48.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:48.674+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.678791+0000) 2019-10-13T11:35:48.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:48.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.683525+0000) 2019-10-13T11:35:48.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:48.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.694806+0000) 2019-10-13T11:35:49.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:49.913+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:49.913669+0000) 2019-10-13T11:35:50.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:50.298+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.300593+0000) 2019-10-13T11:35:50.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:50.298+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.301003+0000) 2019-10-13T11:35:50.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:50.298+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.302095+0000) 2019-10-13T11:35:50.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:50.298+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.302871+0000) 2019-10-13T11:35:50.304 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:50.301+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.303224+0000) 2019-10-13T11:35:50.304 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:50.302+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.304068+0000) 2019-10-13T11:35:50.305 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:50.302+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.304604+0000) 2019-10-13T11:35:50.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:50.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127503 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:50.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:50.305+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:50.309565+0000) 2019-10-13T11:35:52.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:52.166+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.170049+0000) 2019-10-13T11:35:55.312 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:55.306+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.311090+0000) 2019-10-13T11:35:55.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:55.310+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.312387+0000) 2019-10-13T11:35:55.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:55.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.312560+0000) 2019-10-13T11:35:55.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:55.308+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.312988+0000) 2019-10-13T11:35:55.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:55.310+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.313267+0000) 2019-10-13T11:35:55.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:55.310+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.313636+0000) 2019-10-13T11:35:55.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:35:55.310+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.314586+0000) 2019-10-13T11:35:55.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:35:55.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:35:55.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:55.316+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.318211+0000) 2019-10-13T11:35:55.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:35:55.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.479484+0000) 2019-10-13T11:35:56.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:35:56.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.415279+0000) 2019-10-13T11:35:58.119 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:35:58.116+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.118722+0000) 2019-10-13T11:35:58.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:35:58.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.679026+0000) 2019-10-13T11:35:58.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:35:58.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.683763+0000) 2019-10-13T11:35:58.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:35:58.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.695023+0000) 2019-10-13T11:35:59.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:35:59.908+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:59.913882+0000) 2019-10-13T11:36:00.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:00.318+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.321810+0000) 2019-10-13T11:36:00.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:00.318+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.323124+0000) 2019-10-13T11:36:00.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:00.322+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.323589+0000) 2019-10-13T11:36:00.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:00.320+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.323729+0000) 2019-10-13T11:36:00.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:00.322+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.324215+0000) 2019-10-13T11:36:00.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:00.322+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.324931+0000) 2019-10-13T11:36:00.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:00.322+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.325280+0000) 2019-10-13T11:36:00.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:00.326+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:00.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:00.324+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:00.329976+0000) 2019-10-13T11:36:02.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:02.166+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.170250+0000) 2019-10-13T11:36:05.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:05.330+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.331633+0000) 2019-10-13T11:36:05.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:05.330+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.332264+0000) 2019-10-13T11:36:05.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:05.330+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.332355+0000) 2019-10-13T11:36:05.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:05.330+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.332752+0000) 2019-10-13T11:36:05.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:05.330+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.333150+0000) 2019-10-13T11:36:05.334 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:05.330+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.333542+0000) 2019-10-13T11:36:05.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:05.332+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.334512+0000) 2019-10-13T11:36:05.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:05.334+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127601 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:05.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:05.336+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.341537+0000) 2019-10-13T11:36:05.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:05.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.479659+0000) 2019-10-13T11:36:06.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:06.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.415502+0000) 2019-10-13T11:36:08.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:08.128+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.130572+0000) 2019-10-13T11:36:08.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:08.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.679221+0000) 2019-10-13T11:36:08.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:08.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.683950+0000) 2019-10-13T11:36:08.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:08.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.695245+0000) 2019-10-13T11:36:09.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:09.908+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:09.914075+0000) 2019-10-13T11:36:10.343 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:10.340+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.342448+0000) 2019-10-13T11:36:10.344 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:10.338+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.342616+0000) 2019-10-13T11:36:10.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:10.338+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.343183+0000) 2019-10-13T11:36:10.344 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:10.342+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.343607+0000) 2019-10-13T11:36:10.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:10.342+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.343415+0000) 2019-10-13T11:36:10.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:10.342+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.344381+0000) 2019-10-13T11:36:10.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:10.342+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.344711+0000) 2019-10-13T11:36:10.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:10.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:10.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:10.344+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:10.349047+0000) 2019-10-13T11:36:12.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:12.170+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.170455+0000) 2019-10-13T11:36:15.193 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:36:15.193 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:15.196 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:36:15.197 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:15.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:15.348+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.351224+0000) 2019-10-13T11:36:15.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:15.350+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.351494+0000) 2019-10-13T11:36:15.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:15.350+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.352200+0000) 2019-10-13T11:36:15.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:15.350+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.352553+0000) 2019-10-13T11:36:15.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:15.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.353021+0000) 2019-10-13T11:36:15.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:15.350+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.353180+0000) 2019-10-13T11:36:15.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:15.350+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.353605+0000) 2019-10-13T11:36:15.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:15.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:15.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:15.352+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.357306+0000) 2019-10-13T11:36:15.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:15.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.479849+0000) 2019-10-13T11:36:16.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:16.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.415738+0000) 2019-10-13T11:36:18.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:18.136+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.136829+0000) 2019-10-13T11:36:18.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:18.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.679460+0000) 2019-10-13T11:36:18.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:18.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.684157+0000) 2019-10-13T11:36:18.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:18.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.695474+0000) 2019-10-13T11:36:19.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:19.912+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:19.914316+0000) 2019-10-13T11:36:20.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:20.358+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.359883+0000) 2019-10-13T11:36:20.361 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:20.358+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.360968+0000) 2019-10-13T11:36:20.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:20.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.361358+0000) 2019-10-13T11:36:20.362 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:20.358+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.361820+0000) 2019-10-13T11:36:20.363 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:20.360+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.362325+0000) 2019-10-13T11:36:20.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:20.358+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.362607+0000) 2019-10-13T11:36:20.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:20.358+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.363104+0000) 2019-10-13T11:36:20.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:20.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127706 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:20.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:20.368+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:20.370515+0000) 2019-10-13T11:36:22.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:22.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.170658+0000) 2019-10-13T11:36:25.370 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:25.365+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.369356+0000) 2019-10-13T11:36:25.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:25.365+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.370078+0000) 2019-10-13T11:36:25.371 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:25.365+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.370475+0000) 2019-10-13T11:36:25.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:25.365+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.370664+0000) 2019-10-13T11:36:25.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:25.365+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.371170+0000) 2019-10-13T11:36:25.373 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:25.368+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.372148+0000) 2019-10-13T11:36:25.373 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:25.369+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.372283+0000) 2019-10-13T11:36:25.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:25.373+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:25.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:25.376+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.378286+0000) 2019-10-13T11:36:25.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:25.477+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.480073+0000) 2019-10-13T11:36:26.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:26.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.415966+0000) 2019-10-13T11:36:28.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:28.148+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.148540+0000) 2019-10-13T11:36:28.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:28.677+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.679660+0000) 2019-10-13T11:36:28.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:28.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.684349+0000) 2019-10-13T11:36:28.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:28.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.695665+0000) 2019-10-13T11:36:29.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:29.912+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:29.914513+0000) 2019-10-13T11:36:30.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:30.373+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.378681+0000) 2019-10-13T11:36:30.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:30.373+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.379077+0000) 2019-10-13T11:36:30.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:30.377+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.379963+0000) 2019-10-13T11:36:30.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:30.377+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.380583+0000) 2019-10-13T11:36:30.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:30.377+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.381076+0000) 2019-10-13T11:36:30.383 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:30.377+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.382236+0000) 2019-10-13T11:36:30.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:30.380+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.382679+0000) 2019-10-13T11:36:30.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:30.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:30.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:30.384+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:30.389172+0000) 2019-10-13T11:36:32.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:32.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.170856+0000) 2019-10-13T11:36:35.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:35.384+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.389491+0000) 2019-10-13T11:36:35.391 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:35.385+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.388935+0000) 2019-10-13T11:36:35.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:35.385+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.389979+0000) 2019-10-13T11:36:35.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:35.385+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.390490+0000) 2019-10-13T11:36:35.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:35.385+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.390976+0000) 2019-10-13T11:36:35.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:35.389+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.391447+0000) 2019-10-13T11:36:35.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:35.389+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.391692+0000) 2019-10-13T11:36:35.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:35.389+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:35.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:35.396+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.398150+0000) 2019-10-13T11:36:35.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:35.477+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.480282+0000) 2019-10-13T11:36:36.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:36.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.416173+0000) 2019-10-13T11:36:38.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:38.152+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.154924+0000) 2019-10-13T11:36:38.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:38.677+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.679813+0000) 2019-10-13T11:36:38.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:38.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.684544+0000) 2019-10-13T11:36:38.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:38.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.695879+0000) 2019-10-13T11:36:39.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:39.912+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:39.914726+0000) 2019-10-13T11:36:40.399 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:40.396+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.398359+0000) 2019-10-13T11:36:40.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:40.393+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.399189+0000) 2019-10-13T11:36:40.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:40.397+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.399763+0000) 2019-10-13T11:36:40.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:40.397+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.400073+0000) 2019-10-13T11:36:40.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:40.397+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.400777+0000) 2019-10-13T11:36:40.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:40.397+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.401292+0000) 2019-10-13T11:36:40.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:40.397+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.401976+0000) 2019-10-13T11:36:40.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:40.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:40.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:40.404+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:40.407101+0000) 2019-10-13T11:36:42.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:42.165+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.171078+0000) 2019-10-13T11:36:45.326 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:36:45.326 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:45.330 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:36:45.330 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:45.409 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:45.405+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.407846+0000) 2019-10-13T11:36:45.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:45.404+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.408712+0000) 2019-10-13T11:36:45.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:45.405+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.409152+0000) 2019-10-13T11:36:45.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:45.405+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.410505+0000) 2019-10-13T11:36:45.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:45.405+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.410545+0000) 2019-10-13T11:36:45.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:45.405+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.411177+0000) 2019-10-13T11:36:45.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:45.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.412081+0000) 2019-10-13T11:36:45.415 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:45.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:45.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:45.412+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.416507+0000) 2019-10-13T11:36:45.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:45.477+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.480464+0000) 2019-10-13T11:36:46.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:46.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.416374+0000) 2019-10-13T11:36:48.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:48.155+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.161272+0000) 2019-10-13T11:36:48.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:48.677+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.680032+0000) 2019-10-13T11:36:48.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:48.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.684756+0000) 2019-10-13T11:36:48.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:48.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.696107+0000) 2019-10-13T11:36:49.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:49.911+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:49.914978+0000) 2019-10-13T11:36:50.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:50.411+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.417325+0000) 2019-10-13T11:36:50.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:50.413+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.417940+0000) 2019-10-13T11:36:50.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:50.413+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.418212+0000) 2019-10-13T11:36:50.419 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:50.413+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.418962+0000) 2019-10-13T11:36:50.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:50.417+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.419475+0000) 2019-10-13T11:36:50.420 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:50.417+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.419937+0000) 2019-10-13T11:36:50.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:50.417+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.421018+0000) 2019-10-13T11:36:50.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:50.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:50.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:50.423+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:50.427658+0000) 2019-10-13T11:36:52.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:52.169+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.171279+0000) 2019-10-13T11:36:55.430 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:55.425+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.428506+0000) 2019-10-13T11:36:55.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:55.425+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.428566+0000) 2019-10-13T11:36:55.431 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:55.423+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.429033+0000) 2019-10-13T11:36:55.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:55.425+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.429388+0000) 2019-10-13T11:36:55.431 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:55.425+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.429860+0000) 2019-10-13T11:36:55.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:36:55.425+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.430351+0000) 2019-10-13T11:36:55.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:55.425+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.430842+0000) 2019-10-13T11:36:55.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:55.427+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.433806+0000) 2019-10-13T11:36:55.435 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:36:55.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:36:55.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:36:55.477+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.480666+0000) 2019-10-13T11:36:56.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:36:56.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.416574+0000) 2019-10-13T11:36:58.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:36:58.163+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.167692+0000) 2019-10-13T11:36:58.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:36:58.677+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.680232+0000) 2019-10-13T11:36:58.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:36:58.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.684953+0000) 2019-10-13T11:36:58.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:36:58.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.696307+0000) 2019-10-13T11:36:59.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:36:59.911+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:59.915201+0000) 2019-10-13T11:37:00.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:00.433+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.437272+0000) 2019-10-13T11:37:00.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:00.433+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.438333+0000) 2019-10-13T11:37:00.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:00.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.438606+0000) 2019-10-13T11:37:00.440 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:00.433+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.438949+0000) 2019-10-13T11:37:00.440 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:00.433+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.439247+0000) 2019-10-13T11:37:00.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:00.437+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.440632+0000) 2019-10-13T11:37:00.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:00.437+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.440987+0000) 2019-10-13T11:37:00.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:00.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 127977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:00.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:00.439+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:00.445927+0000) 2019-10-13T11:37:02.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:02.169+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.171531+0000) 2019-10-13T11:37:05.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:05.440+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.447059+0000) 2019-10-13T11:37:05.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:05.444+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.448240+0000) 2019-10-13T11:37:05.449 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:05.444+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.448975+0000) 2019-10-13T11:37:05.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:05.444+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.449217+0000) 2019-10-13T11:37:05.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:05.444+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.450226+0000) 2019-10-13T11:37:05.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:05.444+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.450552+0000) 2019-10-13T11:37:05.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:05.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.451033+0000) 2019-10-13T11:37:05.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:05.448+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:05.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:05.451+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.454219+0000) 2019-10-13T11:37:05.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:05.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.480875+0000) 2019-10-13T11:37:06.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:06.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.416775+0000) 2019-10-13T11:37:08.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:08.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.174197+0000) 2019-10-13T11:37:08.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:08.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.680426+0000) 2019-10-13T11:37:08.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:08.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.685189+0000) 2019-10-13T11:37:08.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:08.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.696526+0000) 2019-10-13T11:37:09.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:09.911+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:09.915429+0000) 2019-10-13T11:37:10.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:10.452+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.457012+0000) 2019-10-13T11:37:10.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:10.452+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.457817+0000) 2019-10-13T11:37:10.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:10.452+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.458428+0000) 2019-10-13T11:37:10.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:10.452+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.458468+0000) 2019-10-13T11:37:10.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:10.455+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.458751+0000) 2019-10-13T11:37:10.460 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:10.452+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.459180+0000) 2019-10-13T11:37:10.461 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:10.456+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.460400+0000) 2019-10-13T11:37:10.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:10.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:10.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:10.459+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:10.465788+0000) 2019-10-13T11:37:12.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:12.168+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.171735+0000) 2019-10-13T11:37:15.446 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:37:15.447 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:15.450 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:37:15.451 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:15.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:15.460+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.466490+0000) 2019-10-13T11:37:15.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:15.460+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.467232+0000) 2019-10-13T11:37:15.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:15.463+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.467551+0000) 2019-10-13T11:37:15.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:15.464+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.467558+0000) 2019-10-13T11:37:15.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:15.464+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.467919+0000) 2019-10-13T11:37:15.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:15.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.468563+0000) 2019-10-13T11:37:15.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:15.464+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.469234+0000) 2019-10-13T11:37:15.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:15.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:15.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:15.471+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.474746+0000) 2019-10-13T11:37:15.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:15.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.480984+0000) 2019-10-13T11:37:16.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:16.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.416966+0000) 2019-10-13T11:37:18.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:18.175+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.180599+0000) 2019-10-13T11:37:18.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:18.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.680623+0000) 2019-10-13T11:37:18.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:18.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.685376+0000) 2019-10-13T11:37:18.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:18.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.696721+0000) 2019-10-13T11:37:19.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:19.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:19.915685+0000) 2019-10-13T11:37:20.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:20.472+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.474687+0000) 2019-10-13T11:37:20.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:20.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.475168+0000) 2019-10-13T11:37:20.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:20.472+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.475550+0000) 2019-10-13T11:37:20.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:20.471+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.476690+0000) 2019-10-13T11:37:20.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:20.472+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.476991+0000) 2019-10-13T11:37:20.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:20.472+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.477146+0000) 2019-10-13T11:37:20.478 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:20.472+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.477424+0000) 2019-10-13T11:37:20.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:20.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:20.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:20.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:20.482711+0000) 2019-10-13T11:37:22.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:22.168+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.171935+0000) 2019-10-13T11:37:25.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:25.480+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.481143+0000) 2019-10-13T11:37:25.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:25.480+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.484232+0000) 2019-10-13T11:37:25.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:25.480+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.485357+0000) 2019-10-13T11:37:25.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:25.480+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.485673+0000) 2019-10-13T11:37:25.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:25.480+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.486259+0000) 2019-10-13T11:37:25.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:25.480+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.486567+0000) 2019-10-13T11:37:25.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:25.480+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.486763+0000) 2019-10-13T11:37:25.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:25.483+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.488111+0000) 2019-10-13T11:37:25.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:25.487+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.491116+0000) 2019-10-13T11:37:25.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:25.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:26.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:26.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.417142+0000) 2019-10-13T11:37:28.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:28.183+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.187117+0000) 2019-10-13T11:37:28.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:28.680+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.680826+0000) 2019-10-13T11:37:28.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:28.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.685568+0000) 2019-10-13T11:37:28.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:28.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.696949+0000) 2019-10-13T11:37:29.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:29.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:29.915882+0000) 2019-10-13T11:37:30.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:30.488+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.493438+0000) 2019-10-13T11:37:30.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:30.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.494124+0000) 2019-10-13T11:37:30.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:30.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.494851+0000) 2019-10-13T11:37:30.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:30.492+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.495476+0000) 2019-10-13T11:37:30.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:30.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.496023+0000) 2019-10-13T11:37:30.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:30.492+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.496638+0000) 2019-10-13T11:37:30.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:30.492+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.497112+0000) 2019-10-13T11:37:30.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:30.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:30.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:30.499+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:30.504109+0000) 2019-10-13T11:37:32.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:32.168+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.172138+0000) 2019-10-13T11:37:35.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:35.480+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.481363+0000) 2019-10-13T11:37:35.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:35.499+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.504989+0000) 2019-10-13T11:37:35.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:35.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.506070+0000) 2019-10-13T11:37:35.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:35.500+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.506595+0000) 2019-10-13T11:37:35.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:35.500+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.506929+0000) 2019-10-13T11:37:35.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:35.504+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.507396+0000) 2019-10-13T11:37:35.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:35.504+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.507788+0000) 2019-10-13T11:37:35.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:35.504+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.508155+0000) 2019-10-13T11:37:35.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:35.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:35.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:35.507+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.511677+0000) 2019-10-13T11:37:36.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:36.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.417344+0000) 2019-10-13T11:37:38.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:38.187+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.193460+0000) 2019-10-13T11:37:38.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:38.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.681026+0000) 2019-10-13T11:37:38.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:38.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.685766+0000) 2019-10-13T11:37:38.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:38.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.697144+0000) 2019-10-13T11:37:39.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:39.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:39.916076+0000) 2019-10-13T11:37:40.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:40.508+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.514201+0000) 2019-10-13T11:37:40.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:40.508+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.514985+0000) 2019-10-13T11:37:40.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:40.511+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.515371+0000) 2019-10-13T11:37:40.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:40.512+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.515874+0000) 2019-10-13T11:37:40.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:40.512+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.516315+0000) 2019-10-13T11:37:40.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:40.512+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.516563+0000) 2019-10-13T11:37:40.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:40.512+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.517220+0000) 2019-10-13T11:37:40.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:40.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:40.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:40.515+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:40.522034+0000) 2019-10-13T11:37:42.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:42.172+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.172337+0000) 2019-10-13T11:37:45.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:45.476+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.481564+0000) 2019-10-13T11:37:45.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:45.516+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.522626+0000) 2019-10-13T11:37:45.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:45.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.524042+0000) 2019-10-13T11:37:45.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:45.518+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.524840+0000) 2019-10-13T11:37:45.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:45.520+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.525000+0000) 2019-10-13T11:37:45.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:45.520+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.525432+0000) 2019-10-13T11:37:45.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:45.520+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.525404+0000) 2019-10-13T11:37:45.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:45.520+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.525623+0000) 2019-10-13T11:37:45.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:45.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128281 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:45.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:45.526+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.531281+0000) 2019-10-13T11:37:45.578 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:37:45.579 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:45.582 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:37:45.582 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:46.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:46.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.417561+0000) 2019-10-13T11:37:48.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:48.198+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.199972+0000) 2019-10-13T11:37:48.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:48.676+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.681218+0000) 2019-10-13T11:37:48.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:48.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.685975+0000) 2019-10-13T11:37:48.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:48.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.697341+0000) 2019-10-13T11:37:49.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:49.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:49.916268+0000) 2019-10-13T11:37:50.535 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:50.527+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.533980+0000) 2019-10-13T11:37:50.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:50.530+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.534370+0000) 2019-10-13T11:37:50.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:50.527+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.535148+0000) 2019-10-13T11:37:50.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:50.531+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.535758+0000) 2019-10-13T11:37:50.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:50.531+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.536414+0000) 2019-10-13T11:37:50.537 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:50.531+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.536872+0000) 2019-10-13T11:37:50.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:50.531+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.537611+0000) 2019-10-13T11:37:50.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:50.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:50.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:50.534+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:50.542109+0000) 2019-10-13T11:37:52.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:52.171+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.172534+0000) 2019-10-13T11:37:55.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:55.479+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.481847+0000) 2019-10-13T11:37:55.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:55.539+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.544518+0000) 2019-10-13T11:37:55.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:55.539+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.545239+0000) 2019-10-13T11:37:55.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:55.539+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.545674+0000) 2019-10-13T11:37:55.547 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:37:55.539+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.546064+0000) 2019-10-13T11:37:55.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:37:55.539+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.546761+0000) 2019-10-13T11:37:55.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:55.539+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.546666+0000) 2019-10-13T11:37:55.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:55.542+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.547145+0000) 2019-10-13T11:37:55.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:37:55.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:37:55.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:55.546+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.553497+0000) 2019-10-13T11:37:56.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:37:56.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.417730+0000) 2019-10-13T11:37:58.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:37:58.202+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.205820+0000) 2019-10-13T11:37:58.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:37:58.679+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.681440+0000) 2019-10-13T11:37:58.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:37:58.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.686179+0000) 2019-10-13T11:37:58.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:37:58.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.697552+0000) 2019-10-13T11:37:59.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:37:59.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:59.916485+0000) 2019-10-13T11:38:00.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:00.547+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.554409+0000) 2019-10-13T11:38:00.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:00.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.555063+0000) 2019-10-13T11:38:00.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:00.551+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.555353+0000) 2019-10-13T11:38:00.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:00.550+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.556271+0000) 2019-10-13T11:38:00.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:00.551+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.556597+0000) 2019-10-13T11:38:00.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:00.551+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.557623+0000) 2019-10-13T11:38:00.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:00.551+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.557928+0000) 2019-10-13T11:38:00.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:00.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:00.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:00.558+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:00.564850+0000) 2019-10-13T11:38:02.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:02.167+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.172774+0000) 2019-10-13T11:38:05.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:05.479+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.482079+0000) 2019-10-13T11:38:05.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:05.559+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.563422+0000) 2019-10-13T11:38:05.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:05.559+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.564267+0000) 2019-10-13T11:38:05.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:05.559+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.565487+0000) 2019-10-13T11:38:05.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:05.559+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.565850+0000) 2019-10-13T11:38:05.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:05.559+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.566250+0000) 2019-10-13T11:38:05.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:05.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.566352+0000) 2019-10-13T11:38:05.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:05.559+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.566690+0000) 2019-10-13T11:38:05.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:05.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:05.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:05.566+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.570733+0000) 2019-10-13T11:38:06.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:06.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.417931+0000) 2019-10-13T11:38:08.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:08.206+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.212191+0000) 2019-10-13T11:38:08.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:08.679+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.681637+0000) 2019-10-13T11:38:08.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:08.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.686405+0000) 2019-10-13T11:38:08.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:08.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.697742+0000) 2019-10-13T11:38:09.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:09.914+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:09.916700+0000) 2019-10-13T11:38:10.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:10.567+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.573552+0000) 2019-10-13T11:38:10.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:10.567+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.573841+0000) 2019-10-13T11:38:10.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:10.567+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.574238+0000) 2019-10-13T11:38:10.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:10.567+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.575279+0000) 2019-10-13T11:38:10.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:10.571+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.576078+0000) 2019-10-13T11:38:10.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:10.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.576190+0000) 2019-10-13T11:38:10.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:10.571+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.576688+0000) 2019-10-13T11:38:10.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:10.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:10.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:10.578+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:10.582861+0000) 2019-10-13T11:38:12.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:12.171+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.172974+0000) 2019-10-13T11:38:15.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:15.479+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.482342+0000) 2019-10-13T11:38:15.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:15.579+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.582898+0000) 2019-10-13T11:38:15.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:15.579+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.583292+0000) 2019-10-13T11:38:15.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:15.579+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.583725+0000) 2019-10-13T11:38:15.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:15.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.584071+0000) 2019-10-13T11:38:15.585 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:15.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.584788+0000) 2019-10-13T11:38:15.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:15.579+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.586126+0000) 2019-10-13T11:38:15.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:15.579+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.586259+0000) 2019-10-13T11:38:15.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:15.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:15.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:15.586+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.591068+0000) 2019-10-13T11:38:15.701 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:38:15.701 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:15.705 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:38:15.705 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:16.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:16.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.418121+0000) 2019-10-13T11:38:18.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:18.218+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.218492+0000) 2019-10-13T11:38:18.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:18.679+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.681840+0000) 2019-10-13T11:38:18.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:18.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.686603+0000) 2019-10-13T11:38:18.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:18.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.697935+0000) 2019-10-13T11:38:19.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:19.910+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:19.916904+0000) 2019-10-13T11:38:20.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:20.591+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.592387+0000) 2019-10-13T11:38:20.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:20.591+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.592797+0000) 2019-10-13T11:38:20.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:20.591+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.593666+0000) 2019-10-13T11:38:20.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:20.591+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.593797+0000) 2019-10-13T11:38:20.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:20.591+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.594291+0000) 2019-10-13T11:38:20.595 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:20.590+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.594587+0000) 2019-10-13T11:38:20.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:20.591+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.595032+0000) 2019-10-13T11:38:20.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:20.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:20.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:20.598+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:20.602630+0000) 2019-10-13T11:38:22.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:22.171+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.173172+0000) 2019-10-13T11:38:25.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:25.479+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.482573+0000) 2019-10-13T11:38:25.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:25.599+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.602363+0000) 2019-10-13T11:38:25.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:25.599+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.602640+0000) 2019-10-13T11:38:25.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:25.603+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.603358+0000) 2019-10-13T11:38:25.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:25.603+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.603724+0000) 2019-10-13T11:38:25.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:25.603+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.603465+0000) 2019-10-13T11:38:25.605 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:25.598+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.604496+0000) 2019-10-13T11:38:25.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:25.603+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.604908+0000) 2019-10-13T11:38:25.609 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:25.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:25.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:25.606+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.610460+0000) 2019-10-13T11:38:26.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:26.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.418290+0000) 2019-10-13T11:38:28.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:28.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.228011+0000) 2019-10-13T11:38:28.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:28.679+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.682125+0000) 2019-10-13T11:38:28.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:28.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.686798+0000) 2019-10-13T11:38:28.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:28.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.698145+0000) 2019-10-13T11:38:29.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:29.914+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:29.917137+0000) 2019-10-13T11:38:30.613 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:30.611+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.612109+0000) 2019-10-13T11:38:30.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:30.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.612405+0000) 2019-10-13T11:38:30.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:30.611+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.612667+0000) 2019-10-13T11:38:30.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:30.611+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.613337+0000) 2019-10-13T11:38:30.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:30.611+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.613599+0000) 2019-10-13T11:38:30.615 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:30.610+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.614051+0000) 2019-10-13T11:38:30.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:30.611+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.614743+0000) 2019-10-13T11:38:30.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:30.615+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:30.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:30.618+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:30.621264+0000) 2019-10-13T11:38:32.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:32.171+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.173375+0000) 2019-10-13T11:38:35.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:35.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.482775+0000) 2019-10-13T11:38:35.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:35.618+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.621390+0000) 2019-10-13T11:38:35.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:35.618+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.621806+0000) 2019-10-13T11:38:35.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:35.618+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.622735+0000) 2019-10-13T11:38:35.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:35.622+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.623493+0000) 2019-10-13T11:38:35.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:35.622+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.624663+0000) 2019-10-13T11:38:35.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:35.622+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.625447+0000) 2019-10-13T11:38:35.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:35.622+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.625791+0000) 2019-10-13T11:38:35.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:35.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:35.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:35.630+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.630769+0000) 2019-10-13T11:38:36.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:36.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.418469+0000) 2019-10-13T11:38:38.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:38.233+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.234475+0000) 2019-10-13T11:38:38.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:38.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.682328+0000) 2019-10-13T11:38:38.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:38.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.687015+0000) 2019-10-13T11:38:38.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:38.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.698337+0000) 2019-10-13T11:38:39.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:39.913+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:39.917362+0000) 2019-10-13T11:38:40.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:40.630+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.632640+0000) 2019-10-13T11:38:40.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:40.630+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.632949+0000) 2019-10-13T11:38:40.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:40.630+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.633396+0000) 2019-10-13T11:38:40.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:40.630+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.633655+0000) 2019-10-13T11:38:40.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:40.633+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.634489+0000) 2019-10-13T11:38:40.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:40.630+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.634223+0000) 2019-10-13T11:38:40.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:40.634+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.635503+0000) 2019-10-13T11:38:40.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:40.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:40.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:40.637+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:40.641136+0000) 2019-10-13T11:38:42.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:42.170+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.173582+0000) 2019-10-13T11:38:45.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:45.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.482972+0000) 2019-10-13T11:38:45.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:45.638+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.641912+0000) 2019-10-13T11:38:45.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:45.638+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.642163+0000) 2019-10-13T11:38:45.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:45.638+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.642745+0000) 2019-10-13T11:38:45.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:45.638+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.643130+0000) 2019-10-13T11:38:45.644 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:45.642+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.644026+0000) 2019-10-13T11:38:45.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:45.641+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.644154+0000) 2019-10-13T11:38:45.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:45.642+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.644873+0000) 2019-10-13T11:38:45.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:45.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:45.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:45.649+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.651706+0000) 2019-10-13T11:38:45.820 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:38:45.821 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:45.824 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:38:45.824 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:46.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:46.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.418663+0000) 2019-10-13T11:38:48.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:48.245+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.246181+0000) 2019-10-13T11:38:48.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:48.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.682506+0000) 2019-10-13T11:38:48.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:48.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.687237+0000) 2019-10-13T11:38:48.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:48.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.698536+0000) 2019-10-13T11:38:49.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:49.913+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:49.917616+0000) 2019-10-13T11:38:50.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:50.650+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.652530+0000) 2019-10-13T11:38:50.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:50.650+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.653016+0000) 2019-10-13T11:38:50.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:50.650+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.653304+0000) 2019-10-13T11:38:50.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:50.653+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.654884+0000) 2019-10-13T11:38:50.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:50.654+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.655498+0000) 2019-10-13T11:38:50.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:50.654+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.655940+0000) 2019-10-13T11:38:50.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:50.654+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.656229+0000) 2019-10-13T11:38:50.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:50.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:50.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:50.661+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:50.664209+0000) 2019-10-13T11:38:52.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:52.170+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.173816+0000) 2019-10-13T11:38:55.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:55.478+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.483170+0000) 2019-10-13T11:38:55.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:55.658+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.661538+0000) 2019-10-13T11:38:55.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:55.657+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.662101+0000) 2019-10-13T11:38:55.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:38:55.658+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.663222+0000) 2019-10-13T11:38:55.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:55.662+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.663634+0000) 2019-10-13T11:38:55.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:38:55.662+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.664246+0000) 2019-10-13T11:38:55.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:55.662+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.664699+0000) 2019-10-13T11:38:55.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:55.662+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.665898+0000) 2019-10-13T11:38:55.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:38:55.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:38:55.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:55.673+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.676002+0000) 2019-10-13T11:38:56.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:38:56.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.418838+0000) 2019-10-13T11:38:58.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:38:58.249+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.252695+0000) 2019-10-13T11:38:58.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:38:58.682+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.682737+0000) 2019-10-13T11:38:58.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:38:58.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.687476+0000) 2019-10-13T11:38:58.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:38:58.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.698729+0000) 2019-10-13T11:38:59.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:38:59.917+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:59.917808+0000) 2019-10-13T11:39:00.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:00.670+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.671888+0000) 2019-10-13T11:39:00.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:00.670+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.672328+0000) 2019-10-13T11:39:00.674 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:00.670+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.673038+0000) 2019-10-13T11:39:00.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:00.670+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.673456+0000) 2019-10-13T11:39:00.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:00.670+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.673722+0000) 2019-10-13T11:39:00.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:00.669+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.674070+0000) 2019-10-13T11:39:00.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:00.670+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.674551+0000) 2019-10-13T11:39:00.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:00.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:00.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:00.681+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:00.683068+0000) 2019-10-13T11:39:02.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:02.170+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.174019+0000) 2019-10-13T11:39:05.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:05.482+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.483374+0000) 2019-10-13T11:39:05.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:05.678+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.681023+0000) 2019-10-13T11:39:05.683 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:05.678+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.682121+0000) 2019-10-13T11:39:05.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:05.678+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.682243+0000) 2019-10-13T11:39:05.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:05.678+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.682640+0000) 2019-10-13T11:39:05.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:05.682+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.683518+0000) 2019-10-13T11:39:05.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:05.682+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.684607+0000) 2019-10-13T11:39:05.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:05.681+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.685076+0000) 2019-10-13T11:39:05.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:05.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:05.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:05.689+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.691272+0000) 2019-10-13T11:39:06.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:06.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.419013+0000) 2019-10-13T11:39:08.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:08.257+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.259183+0000) 2019-10-13T11:39:08.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:08.678+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.682939+0000) 2019-10-13T11:39:08.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:08.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.687676+0000) 2019-10-13T11:39:08.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:08.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.698924+0000) 2019-10-13T11:39:09.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:09.917+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:09.918000+0000) 2019-10-13T11:39:10.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:10.686+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.690838+0000) 2019-10-13T11:39:10.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:10.689+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.691703+0000) 2019-10-13T11:39:10.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:10.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.692109+0000) 2019-10-13T11:39:10.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:10.690+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.692689+0000) 2019-10-13T11:39:10.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:10.690+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.692957+0000) 2019-10-13T11:39:10.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:10.690+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.693308+0000) 2019-10-13T11:39:10.695 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:10.690+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.694104+0000) 2019-10-13T11:39:10.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:10.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:10.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:10.701+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:10.705177+0000) 2019-10-13T11:39:12.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:12.170+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.174220+0000) 2019-10-13T11:39:15.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:15.482+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.483572+0000) 2019-10-13T11:39:15.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:15.698+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.700306+0000) 2019-10-13T11:39:15.701 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:15.698+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.701005+0000) 2019-10-13T11:39:15.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:15.698+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.701456+0000) 2019-10-13T11:39:15.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:15.698+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.702251+0000) 2019-10-13T11:39:15.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:15.701+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.702485+0000) 2019-10-13T11:39:15.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:15.698+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.702794+0000) 2019-10-13T11:39:15.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:15.698+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.703097+0000) 2019-10-13T11:39:15.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:15.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:15.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:15.705+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.709654+0000) 2019-10-13T11:39:15.881 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:39:15.882 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:15.885 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:39:15.885 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:16.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:16.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.419223+0000) 2019-10-13T11:39:18.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:18.265+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.265694+0000) 2019-10-13T11:39:18.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:18.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.683180+0000) 2019-10-13T11:39:18.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:18.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.687868+0000) 2019-10-13T11:39:18.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:18.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.699121+0000) 2019-10-13T11:39:19.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:19.917+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:19.918223+0000) 2019-10-13T11:39:20.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:20.705+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.709817+0000) 2019-10-13T11:39:20.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:20.705+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.711122+0000) 2019-10-13T11:39:20.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:20.709+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.711997+0000) 2019-10-13T11:39:20.713 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:20.709+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.712263+0000) 2019-10-13T11:39:20.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:20.709+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.713112+0000) 2019-10-13T11:39:20.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:20.709+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.713340+0000) 2019-10-13T11:39:20.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:20.709+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.713922+0000) 2019-10-13T11:39:20.717 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:20.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:20.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:20.717+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:20.719178+0000) 2019-10-13T11:39:22.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:22.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.174423+0000) 2019-10-13T11:39:25.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:25.481+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.483777+0000) 2019-10-13T11:39:25.721 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:25.717+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.720288+0000) 2019-10-13T11:39:25.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:25.717+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.719773+0000) 2019-10-13T11:39:25.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:25.717+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.720772+0000) 2019-10-13T11:39:25.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:25.717+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.721422+0000) 2019-10-13T11:39:25.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:25.717+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.722188+0000) 2019-10-13T11:39:25.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:25.717+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.722642+0000) 2019-10-13T11:39:25.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:25.721+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.723612+0000) 2019-10-13T11:39:25.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:25.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128957 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:25.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:25.725+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.727556+0000) 2019-10-13T11:39:26.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:26.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.419416+0000) 2019-10-13T11:39:28.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:28.269+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.272204+0000) 2019-10-13T11:39:28.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:28.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.683376+0000) 2019-10-13T11:39:28.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:28.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.688096+0000) 2019-10-13T11:39:28.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:28.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.699273+0000) 2019-10-13T11:39:29.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:29.917+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:29.918372+0000) 2019-10-13T11:39:30.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:30.725+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.729808+0000) 2019-10-13T11:39:30.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:30.725+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.730091+0000) 2019-10-13T11:39:30.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:30.725+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.730655+0000) 2019-10-13T11:39:30.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:30.729+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.731433+0000) 2019-10-13T11:39:30.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:30.729+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.732248+0000) 2019-10-13T11:39:30.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:30.729+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.732607+0000) 2019-10-13T11:39:30.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:30.729+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.733024+0000) 2019-10-13T11:39:30.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:30.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 128994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:30.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:30.733+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:30.735618+0000) 2019-10-13T11:39:32.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:32.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.174644+0000) 2019-10-13T11:39:35.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:35.481+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.483986+0000) 2019-10-13T11:39:35.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:35.733+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.738663+0000) 2019-10-13T11:39:35.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:35.733+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.739232+0000) 2019-10-13T11:39:35.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:35.737+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.739905+0000) 2019-10-13T11:39:35.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:35.737+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.740566+0000) 2019-10-13T11:39:35.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:35.737+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.740951+0000) 2019-10-13T11:39:35.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:35.736+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.741353+0000) 2019-10-13T11:39:35.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:35.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.741921+0000) 2019-10-13T11:39:35.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:35.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:35.748 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:35.744+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.747897+0000) 2019-10-13T11:39:36.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:36.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.419545+0000) 2019-10-13T11:39:38.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:38.280+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.283982+0000) 2019-10-13T11:39:38.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:38.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.683578+0000) 2019-10-13T11:39:38.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:38.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.688287+0000) 2019-10-13T11:39:38.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:38.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.699499+0000) 2019-10-13T11:39:39.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:39.916+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:39.918576+0000) 2019-10-13T11:39:40.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:40.745+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.749465+0000) 2019-10-13T11:39:40.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:40.745+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.750019+0000) 2019-10-13T11:39:40.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:40.745+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.750605+0000) 2019-10-13T11:39:40.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:40.749+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.751298+0000) 2019-10-13T11:39:40.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:40.748+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.751855+0000) 2019-10-13T11:39:40.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:40.749+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.752167+0000) 2019-10-13T11:39:40.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:40.749+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.752439+0000) 2019-10-13T11:39:40.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:40.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:40.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:40.760+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:40.763573+0000) 2019-10-13T11:39:42.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:42.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.174844+0000) 2019-10-13T11:39:45.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:45.481+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.484226+0000) 2019-10-13T11:39:45.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:45.757+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.759484+0000) 2019-10-13T11:39:45.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:45.757+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.760285+0000) 2019-10-13T11:39:45.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:45.757+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.760611+0000) 2019-10-13T11:39:45.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:45.756+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.760820+0000) 2019-10-13T11:39:45.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:45.757+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.761491+0000) 2019-10-13T11:39:45.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:45.757+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.761795+0000) 2019-10-13T11:39:45.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:45.757+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.762474+0000) 2019-10-13T11:39:45.767 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:45.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129093 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:45.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:45.764+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.769093+0000) 2019-10-13T11:39:46.012 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:39:46.012 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:46.016 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:39:46.016 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:46.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:46.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.419731+0000) 2019-10-13T11:39:48.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:48.288+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.290564+0000) 2019-10-13T11:39:48.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:48.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.683801+0000) 2019-10-13T11:39:48.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:48.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.688485+0000) 2019-10-13T11:39:48.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:48.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.699714+0000) 2019-10-13T11:39:49.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:49.916+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:49.918791+0000) 2019-10-13T11:39:50.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:50.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.770553+0000) 2019-10-13T11:39:50.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:50.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.770503+0000) 2019-10-13T11:39:50.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:50.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.771011+0000) 2019-10-13T11:39:50.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:50.769+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.771393+0000) 2019-10-13T11:39:50.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:50.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.772093+0000) 2019-10-13T11:39:50.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:50.769+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.772442+0000) 2019-10-13T11:39:50.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:50.769+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.773672+0000) 2019-10-13T11:39:50.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:50.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:50.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:50.776+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:50.779622+0000) 2019-10-13T11:39:52.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:52.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.175069+0000) 2019-10-13T11:39:55.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:55.481+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.484427+0000) 2019-10-13T11:39:55.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:55.777+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.780190+0000) 2019-10-13T11:39:55.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:39:55.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.780530+0000) 2019-10-13T11:39:55.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:55.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.780855+0000) 2019-10-13T11:39:55.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:55.776+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.781182+0000) 2019-10-13T11:39:55.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:39:55.777+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.781859+0000) 2019-10-13T11:39:55.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:55.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.782556+0000) 2019-10-13T11:39:55.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:55.777+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.781964+0000) 2019-10-13T11:39:55.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:39:55.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:39:55.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:55.784+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.786282+0000) 2019-10-13T11:39:56.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:39:56.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.419950+0000) 2019-10-13T11:39:58.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:39:58.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.297219+0000) 2019-10-13T11:39:58.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:39:58.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.684002+0000) 2019-10-13T11:39:58.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:39:58.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.688671+0000) 2019-10-13T11:39:58.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:39:58.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.699931+0000) 2019-10-13T11:39:59.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:39:59.916+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:59.919021+0000) 2019-10-13T11:40:00.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:00.784+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.788958+0000) 2019-10-13T11:40:00.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:00.784+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.789610+0000) 2019-10-13T11:40:00.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:00.788+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.790672+0000) 2019-10-13T11:40:00.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:00.784+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.790752+0000) 2019-10-13T11:40:00.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:00.784+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.791184+0000) 2019-10-13T11:40:00.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:00.788+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.791564+0000) 2019-10-13T11:40:00.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:00.788+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.791924+0000) 2019-10-13T11:40:00.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:00.792+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:00.795445+0000) 2019-10-13T11:40:00.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:00.792+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:02.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:02.172+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.175349+0000) 2019-10-13T11:40:05.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:05.480+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.484626+0000) 2019-10-13T11:40:05.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:05.792+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.799252+0000) 2019-10-13T11:40:05.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:05.796+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.799728+0000) 2019-10-13T11:40:05.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:05.796+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.800565+0000) 2019-10-13T11:40:05.801 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:05.796+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.800912+0000) 2019-10-13T11:40:05.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:05.796+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.801228+0000) 2019-10-13T11:40:05.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:05.796+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.801524+0000) 2019-10-13T11:40:05.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:05.796+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.801753+0000) 2019-10-13T11:40:05.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:05.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:05.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:05.804+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.808417+0000) 2019-10-13T11:40:06.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:06.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.420157+0000) 2019-10-13T11:40:08.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:08.300+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.304041+0000) 2019-10-13T11:40:08.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:08.680+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.684208+0000) 2019-10-13T11:40:08.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:08.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.688865+0000) 2019-10-13T11:40:08.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:08.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.700158+0000) 2019-10-13T11:40:09.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:09.916+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:09.919217+0000) 2019-10-13T11:40:10.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:10.804+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.809265+0000) 2019-10-13T11:40:10.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:10.804+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.809701+0000) 2019-10-13T11:40:10.811 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:10.804+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.810368+0000) 2019-10-13T11:40:10.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:10.804+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.810944+0000) 2019-10-13T11:40:10.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:10.808+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.811445+0000) 2019-10-13T11:40:10.813 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:10.808+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.811728+0000) 2019-10-13T11:40:10.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:10.808+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.812029+0000) 2019-10-13T11:40:10.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:10.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:10.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:10.816+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:10.819209+0000) 2019-10-13T11:40:12.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:12.172+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.175580+0000) 2019-10-13T11:40:15.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:15.484+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.484823+0000) 2019-10-13T11:40:15.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:15.816+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.819356+0000) 2019-10-13T11:40:15.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:15.816+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.820106+0000) 2019-10-13T11:40:15.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:15.816+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.820379+0000) 2019-10-13T11:40:15.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:15.816+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.820864+0000) 2019-10-13T11:40:15.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:15.816+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.821246+0000) 2019-10-13T11:40:15.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:15.816+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.821618+0000) 2019-10-13T11:40:15.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:15.816+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.822007+0000) 2019-10-13T11:40:15.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:15.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:15.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:15.824+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.828863+0000) 2019-10-13T11:40:16.120 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:40:16.120 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:16.124 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:40:16.124 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:16.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:16.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.420352+0000) 2019-10-13T11:40:18.311 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:18.308+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.310584+0000) 2019-10-13T11:40:18.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:18.680+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.684404+0000) 2019-10-13T11:40:18.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:18.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.689060+0000) 2019-10-13T11:40:18.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:18.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.700418+0000) 2019-10-13T11:40:19.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:19.916+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:19.919462+0000) 2019-10-13T11:40:20.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:20.824+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.829897+0000) 2019-10-13T11:40:20.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:20.824+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.830326+0000) 2019-10-13T11:40:20.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:20.824+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.830931+0000) 2019-10-13T11:40:20.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:20.824+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.831154+0000) 2019-10-13T11:40:20.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:20.828+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.831777+0000) 2019-10-13T11:40:20.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:20.828+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.832318+0000) 2019-10-13T11:40:20.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:20.828+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.832926+0000) 2019-10-13T11:40:20.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:20.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129333 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:20.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:20.832+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:20.837803+0000) 2019-10-13T11:40:22.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:22.172+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.175808+0000) 2019-10-13T11:40:25.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:25.480+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.485018+0000) 2019-10-13T11:40:25.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:25.836+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.839790+0000) 2019-10-13T11:40:25.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:25.836+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.840258+0000) 2019-10-13T11:40:25.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:25.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.840905+0000) 2019-10-13T11:40:25.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:25.836+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.841281+0000) 2019-10-13T11:40:25.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:25.836+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.841718+0000) 2019-10-13T11:40:25.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:25.836+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.842318+0000) 2019-10-13T11:40:25.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:25.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.842879+0000) 2019-10-13T11:40:25.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:25.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:25.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:25.848+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.850699+0000) 2019-10-13T11:40:26.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:26.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.420603+0000) 2019-10-13T11:40:28.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:28.315+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.316865+0000) 2019-10-13T11:40:28.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:28.684+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.684608+0000) 2019-10-13T11:40:28.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:28.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.689250+0000) 2019-10-13T11:40:28.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:28.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.700632+0000) 2019-10-13T11:40:29.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:29.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:29.919687+0000) 2019-10-13T11:40:30.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:30.844+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.849237+0000) 2019-10-13T11:40:30.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:30.843+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.849737+0000) 2019-10-13T11:40:30.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:30.844+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.850904+0000) 2019-10-13T11:40:30.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:30.844+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.851226+0000) 2019-10-13T11:40:30.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:30.848+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.851532+0000) 2019-10-13T11:40:30.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:30.848+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.851860+0000) 2019-10-13T11:40:30.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:30.848+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.852365+0000) 2019-10-13T11:40:30.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:30.851+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:30.856207+0000) 2019-10-13T11:40:30.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:30.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129401 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:32.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:32.172+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.176032+0000) 2019-10-13T11:40:35.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:35.484+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.485229+0000) 2019-10-13T11:40:35.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:35.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.859483+0000) 2019-10-13T11:40:35.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:35.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.860765+0000) 2019-10-13T11:40:35.862 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:35.856+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.861123+0000) 2019-10-13T11:40:35.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:35.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.861833+0000) 2019-10-13T11:40:35.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:35.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.862143+0000) 2019-10-13T11:40:35.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:35.859+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.862440+0000) 2019-10-13T11:40:35.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:35.860+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.863421+0000) 2019-10-13T11:40:35.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:35.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129432 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:35.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:35.867+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.871495+0000) 2019-10-13T11:40:36.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:36.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.420779+0000) 2019-10-13T11:40:38.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:38.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.323502+0000) 2019-10-13T11:40:38.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:38.680+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.684818+0000) 2019-10-13T11:40:38.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:38.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.689426+0000) 2019-10-13T11:40:38.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:38.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.700850+0000) 2019-10-13T11:40:39.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:39.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:39.919878+0000) 2019-10-13T11:40:40.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:40.864+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.871055+0000) 2019-10-13T11:40:40.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:40.864+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.870968+0000) 2019-10-13T11:40:40.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:40.868+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.871488+0000) 2019-10-13T11:40:40.873 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:40.868+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.871971+0000) 2019-10-13T11:40:40.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:40.868+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.872415+0000) 2019-10-13T11:40:40.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:40.868+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.872888+0000) 2019-10-13T11:40:40.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:40.867+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.873898+0000) 2019-10-13T11:40:40.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:40.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:40.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:40.875+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:40.879840+0000) 2019-10-13T11:40:42.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:42.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.176237+0000) 2019-10-13T11:40:45.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:45.479+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.485430+0000) 2019-10-13T11:40:45.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:45.875+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.880272+0000) 2019-10-13T11:40:45.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:45.875+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.880965+0000) 2019-10-13T11:40:45.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:45.875+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.881330+0000) 2019-10-13T11:40:45.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:45.875+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.881997+0000) 2019-10-13T11:40:45.883 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:45.875+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.882250+0000) 2019-10-13T11:40:45.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:45.875+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.882539+0000) 2019-10-13T11:40:45.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:45.879+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.882942+0000) 2019-10-13T11:40:45.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:45.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:45.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:45.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.890775+0000) 2019-10-13T11:40:46.266 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:40:46.266 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:46.270 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:40:46.270 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:46.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:46.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.420965+0000) 2019-10-13T11:40:48.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:48.327+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.330075+0000) 2019-10-13T11:40:48.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:48.683+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.685004+0000) 2019-10-13T11:40:48.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:48.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.689637+0000) 2019-10-13T11:40:48.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:48.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.701052+0000) 2019-10-13T11:40:49.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:49.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:49.920073+0000) 2019-10-13T11:40:50.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:50.883+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.890515+0000) 2019-10-13T11:40:50.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:50.883+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.891272+0000) 2019-10-13T11:40:50.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:50.887+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.891760+0000) 2019-10-13T11:40:50.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:50.887+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.892190+0000) 2019-10-13T11:40:50.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:50.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.893031+0000) 2019-10-13T11:40:50.895 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:50.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.893361+0000) 2019-10-13T11:40:50.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:50.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.893874+0000) 2019-10-13T11:40:50.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:50.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129537 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:50.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:50.895+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:50.898551+0000) 2019-10-13T11:40:52.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:52.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.176453+0000) 2019-10-13T11:40:55.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:55.483+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.485643+0000) 2019-10-13T11:40:55.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:55.895+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.900767+0000) 2019-10-13T11:40:55.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:40:55.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.901302+0000) 2019-10-13T11:40:55.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:55.895+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.901463+0000) 2019-10-13T11:40:55.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:40:55.895+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.902930+0000) 2019-10-13T11:40:55.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:55.899+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.903676+0000) 2019-10-13T11:40:55.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:55.899+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.904043+0000) 2019-10-13T11:40:55.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:55.899+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.904309+0000) 2019-10-13T11:40:55.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:40:55.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:40:55.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:55.903+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.909688+0000) 2019-10-13T11:40:56.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:40:56.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.421150+0000) 2019-10-13T11:40:58.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:40:58.335+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.336711+0000) 2019-10-13T11:40:58.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:40:58.683+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.685213+0000) 2019-10-13T11:40:58.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:40:58.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.689849+0000) 2019-10-13T11:40:58.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:40:58.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.701316+0000) 2019-10-13T11:40:59.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:40:59.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:59.920264+0000) 2019-10-13T11:41:00.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:00.903+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.910132+0000) 2019-10-13T11:41:00.911 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:00.903+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.910580+0000) 2019-10-13T11:41:00.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:00.907+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.911385+0000) 2019-10-13T11:41:00.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:00.907+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.911907+0000) 2019-10-13T11:41:00.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:00.907+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.912194+0000) 2019-10-13T11:41:00.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:00.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.912520+0000) 2019-10-13T11:41:00.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:00.907+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.913580+0000) 2019-10-13T11:41:00.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:00.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:00.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:00.911+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:00.917417+0000) 2019-10-13T11:41:02.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:02.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.176678+0000) 2019-10-13T11:41:05.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:05.483+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.485845+0000) 2019-10-13T11:41:05.922 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:05.915+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.920577+0000) 2019-10-13T11:41:05.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:05.915+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.920914+0000) 2019-10-13T11:41:05.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:05.915+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.921244+0000) 2019-10-13T11:41:05.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:05.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.922227+0000) 2019-10-13T11:41:05.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:05.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.923028+0000) 2019-10-13T11:41:05.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:05.919+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.923863+0000) 2019-10-13T11:41:05.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:05.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.923878+0000) 2019-10-13T11:41:05.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:05.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:05.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:05.923+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.928069+0000) 2019-10-13T11:41:06.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:06.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.421322+0000) 2019-10-13T11:41:08.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:08.339+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.343218+0000) 2019-10-13T11:41:08.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:08.683+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.685430+0000) 2019-10-13T11:41:08.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:08.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.690044+0000) 2019-10-13T11:41:08.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:08.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.701530+0000) 2019-10-13T11:41:09.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:09.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:09.920464+0000) 2019-10-13T11:41:10.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:10.923+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.929452+0000) 2019-10-13T11:41:10.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:10.923+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.930693+0000) 2019-10-13T11:41:10.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:10.923+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.931085+0000) 2019-10-13T11:41:10.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:10.927+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.931340+0000) 2019-10-13T11:41:10.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:10.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.932376+0000) 2019-10-13T11:41:10.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:10.927+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.932783+0000) 2019-10-13T11:41:10.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:10.927+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.933409+0000) 2019-10-13T11:41:10.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:10.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:10.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:10.931+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:10.936785+0000) 2019-10-13T11:41:12.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:12.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.176873+0000) 2019-10-13T11:41:15.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:15.483+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.486074+0000) 2019-10-13T11:41:15.940 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:15.935+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.938951+0000) 2019-10-13T11:41:15.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:15.939+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.940181+0000) 2019-10-13T11:41:15.941 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:15.939+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.940498+0000) 2019-10-13T11:41:15.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:15.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.941353+0000) 2019-10-13T11:41:15.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:15.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.941748+0000) 2019-10-13T11:41:15.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:15.939+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.942198+0000) 2019-10-13T11:41:15.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:15.943+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.943510+0000) 2019-10-13T11:41:15.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:15.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:15.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:15.947+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.950244+0000) 2019-10-13T11:41:16.381 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:41:16.382 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:16.385 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:41:16.385 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:16.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:16.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.421517+0000) 2019-10-13T11:41:18.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:18.347+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.349684+0000) 2019-10-13T11:41:18.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:18.683+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.685634+0000) 2019-10-13T11:41:18.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:18.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.690242+0000) 2019-10-13T11:41:18.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:18.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.701757+0000) 2019-10-13T11:41:19.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:19.915+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:19.920665+0000) 2019-10-13T11:41:20.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:20.947+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.949018+0000) 2019-10-13T11:41:20.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:20.947+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.949357+0000) 2019-10-13T11:41:20.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:20.947+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.950321+0000) 2019-10-13T11:41:20.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:20.947+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.950797+0000) 2019-10-13T11:41:20.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:20.951+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.951352+0000) 2019-10-13T11:41:20.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:20.946+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.950978+0000) 2019-10-13T11:41:20.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:20.951+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.951798+0000) 2019-10-13T11:41:20.957 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:20.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:20.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:20.954+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:20.958507+0000) 2019-10-13T11:41:22.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:22.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:22.177079+0000) 2019-10-13T11:41:25.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:25.483+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.486269+0000) 2019-10-13T11:41:25.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:25.959+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.959413+0000) 2019-10-13T11:41:25.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:25.959+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.960020+0000) 2019-10-13T11:41:25.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:25.959+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.960330+0000) 2019-10-13T11:41:25.961 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:25.959+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.961122+0000) 2019-10-13T11:41:25.962 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:25.959+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.961756+0000) 2019-10-13T11:41:25.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:25.954+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.962086+0000) 2019-10-13T11:41:25.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:25.959+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.962513+0000) 2019-10-13T11:41:25.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:25.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:25.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:25.962+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.968152+0000) 2019-10-13T11:41:26.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:26.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.421740+0000) 2019-10-13T11:41:28.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:28.354+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.356146+0000) 2019-10-13T11:41:28.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:28.682+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.685831+0000) 2019-10-13T11:41:28.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:28.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.690438+0000) 2019-10-13T11:41:28.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:28.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.701962+0000) 2019-10-13T11:41:29.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:29.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:29.920819+0000) 2019-10-13T11:41:30.970 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:30.966+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.968990+0000) 2019-10-13T11:41:30.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:30.966+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.969792+0000) 2019-10-13T11:41:30.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:30.962+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.970090+0000) 2019-10-13T11:41:30.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:30.966+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.970550+0000) 2019-10-13T11:41:30.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:30.966+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.971225+0000) 2019-10-13T11:41:30.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:30.966+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.971020+0000) 2019-10-13T11:41:30.972 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:30.970+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.972015+0000) 2019-10-13T11:41:30.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:30.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:30.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:30.974+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:30.978956+0000) 2019-10-13T11:41:32.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:32.174+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:32.177286+0000) 2019-10-13T11:41:35.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:35.482+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.486473+0000) 2019-10-13T11:41:35.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:35.974+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.978582+0000) 2019-10-13T11:41:35.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:35.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.979013+0000) 2019-10-13T11:41:35.980 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:35.978+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.979854+0000) 2019-10-13T11:41:35.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:35.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.980055+0000) 2019-10-13T11:41:35.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:35.978+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.980500+0000) 2019-10-13T11:41:35.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:35.978+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.980669+0000) 2019-10-13T11:41:35.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:35.978+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.981432+0000) 2019-10-13T11:41:35.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:35.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:35.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:35.982+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.987332+0000) 2019-10-13T11:41:36.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:36.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.421934+0000) 2019-10-13T11:41:38.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:38.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.362490+0000) 2019-10-13T11:41:38.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:38.682+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.686030+0000) 2019-10-13T11:41:38.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:38.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.690633+0000) 2019-10-13T11:41:38.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:38.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.702172+0000) 2019-10-13T11:41:39.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:39.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:39.921031+0000) 2019-10-13T11:41:40.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:40.986+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.988805+0000) 2019-10-13T11:41:40.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:40.986+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.988997+0000) 2019-10-13T11:41:40.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:40.986+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.989680+0000) 2019-10-13T11:41:40.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:40.986+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.989361+0000) 2019-10-13T11:41:40.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:40.986+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.990607+0000) 2019-10-13T11:41:40.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:40.986+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.990723+0000) 2019-10-13T11:41:40.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:40.990+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.991377+0000) 2019-10-13T11:41:40.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:40.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:40.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:40.990+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:40.997887+0000) 2019-10-13T11:41:42.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:42.174+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:42.177516+0000) 2019-10-13T11:41:45.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:45.482+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.486644+0000) 2019-10-13T11:41:45.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:45.994+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.998808+0000) 2019-10-13T11:41:46.000 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:45.998+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.999422+0000) 2019-10-13T11:41:46.000 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:45.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.999593+0000) 2019-10-13T11:41:46.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:45.998+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.000861+0000) 2019-10-13T11:41:46.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:45.998+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.001208+0000) 2019-10-13T11:41:46.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:45.994+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.001425+0000) 2019-10-13T11:41:46.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:45.998+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.001902+0000) 2019-10-13T11:41:46.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:46.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:46.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:46.002+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.008659+0000) 2019-10-13T11:41:46.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:46.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.422170+0000) 2019-10-13T11:41:46.705 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:41:46.705 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:46.709 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:41:46.709 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:48.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:48.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.369116+0000) 2019-10-13T11:41:48.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:48.682+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.686234+0000) 2019-10-13T11:41:48.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:48.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.690837+0000) 2019-10-13T11:41:48.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:48.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.702371+0000) 2019-10-13T11:41:49.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:49.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:49.921259+0000) 2019-10-13T11:41:51.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:51.006+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.009023+0000) 2019-10-13T11:41:51.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:51.006+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.009803+0000) 2019-10-13T11:41:51.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:51.006+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.009972+0000) 2019-10-13T11:41:51.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:51.006+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.010876+0000) 2019-10-13T11:41:51.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:51.006+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.011240+0000) 2019-10-13T11:41:51.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:51.010+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.011824+0000) 2019-10-13T11:41:51.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:51.006+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.012317+0000) 2019-10-13T11:41:51.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:51.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:51.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:51.010+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.016144+0000) 2019-10-13T11:41:52.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:52.174+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:52.177716+0000) 2019-10-13T11:41:55.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:55.486+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:55.486868+0000) 2019-10-13T11:41:56.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:56.018+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.019475+0000) 2019-10-13T11:41:56.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:56.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.019996+0000) 2019-10-13T11:41:56.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:41:56.018+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.021090+0000) 2019-10-13T11:41:56.022 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:56.018+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.021336+0000) 2019-10-13T11:41:56.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:41:56.018+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.021636+0000) 2019-10-13T11:41:56.023 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:56.018+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.022176+0000) 2019-10-13T11:41:56.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:56.018+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.022588+0000) 2019-10-13T11:41:56.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:41:56.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 129976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:41:56.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:56.022+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.028641+0000) 2019-10-13T11:41:56.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:41:56.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.422347+0000) 2019-10-13T11:41:58.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:41:58.374+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.376018+0000) 2019-10-13T11:41:58.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:41:58.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.686436+0000) 2019-10-13T11:41:58.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:41:58.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.691031+0000) 2019-10-13T11:41:58.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:41:58.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.702601+0000) 2019-10-13T11:41:59.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:41:59.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:59.921458+0000) 2019-10-13T11:42:01.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:01.026+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.029375+0000) 2019-10-13T11:42:01.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:01.026+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.029630+0000) 2019-10-13T11:42:01.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:01.026+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.030157+0000) 2019-10-13T11:42:01.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:01.026+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.030850+0000) 2019-10-13T11:42:01.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:01.026+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.031142+0000) 2019-10-13T11:42:01.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:01.030+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.031975+0000) 2019-10-13T11:42:01.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:01.030+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.032280+0000) 2019-10-13T11:42:01.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:01.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130012 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:01.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:01.038+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.043182+0000) 2019-10-13T11:42:02.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:02.174+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.177912+0000) 2019-10-13T11:42:05.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:05.482+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:05.487076+0000) 2019-10-13T11:42:06.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:06.034+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.038780+0000) 2019-10-13T11:42:06.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:06.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.039859+0000) 2019-10-13T11:42:06.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:06.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.040438+0000) 2019-10-13T11:42:06.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:06.034+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.040700+0000) 2019-10-13T11:42:06.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:06.038+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.041242+0000) 2019-10-13T11:42:06.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:06.038+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.041470+0000) 2019-10-13T11:42:06.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:06.038+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.042534+0000) 2019-10-13T11:42:06.046 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:06.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:06.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:06.046+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.051259+0000) 2019-10-13T11:42:06.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:06.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:06.422550+0000) 2019-10-13T11:42:08.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:08.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.382794+0000) 2019-10-13T11:42:08.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:08.682+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.686631+0000) 2019-10-13T11:42:08.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:08.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.691231+0000) 2019-10-13T11:42:08.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:08.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.702798+0000) 2019-10-13T11:42:09.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:09.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:09.921678+0000) 2019-10-13T11:42:11.050 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:11.045+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.048885+0000) 2019-10-13T11:42:11.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:11.045+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.049288+0000) 2019-10-13T11:42:11.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:11.046+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.049496+0000) 2019-10-13T11:42:11.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:11.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.050642+0000) 2019-10-13T11:42:11.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:11.045+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.050974+0000) 2019-10-13T11:42:11.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:11.045+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.051268+0000) 2019-10-13T11:42:11.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:11.049+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.052614+0000) 2019-10-13T11:42:11.056 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:11.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:11.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:11.058+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.060445+0000) 2019-10-13T11:42:12.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:12.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.178119+0000) 2019-10-13T11:42:15.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:15.485+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:15.487327+0000) 2019-10-13T11:42:16.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:16.053+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.058631+0000) 2019-10-13T11:42:16.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:16.057+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.058961+0000) 2019-10-13T11:42:16.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:16.057+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.059415+0000) 2019-10-13T11:42:16.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:16.057+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.060768+0000) 2019-10-13T11:42:16.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:16.057+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.061375+0000) 2019-10-13T11:42:16.062 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:16.057+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.061620+0000) 2019-10-13T11:42:16.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:16.057+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.062269+0000) 2019-10-13T11:42:16.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:16.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:16.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:16.065+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.069354+0000) 2019-10-13T11:42:16.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:16.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:16.422732+0000) 2019-10-13T11:42:17.024 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:42:17.024 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:17.027 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:42:17.028 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:18.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:18.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.389461+0000) 2019-10-13T11:42:18.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:18.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.686842+0000) 2019-10-13T11:42:18.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:18.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.691429+0000) 2019-10-13T11:42:18.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:18.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.703056+0000) 2019-10-13T11:42:19.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:19.917+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:19.921909+0000) 2019-10-13T11:42:21.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:21.065+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.068878+0000) 2019-10-13T11:42:21.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:21.065+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.069939+0000) 2019-10-13T11:42:21.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:21.065+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.070291+0000) 2019-10-13T11:42:21.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:21.065+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.071132+0000) 2019-10-13T11:42:21.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:21.065+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.071152+0000) 2019-10-13T11:42:21.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:21.069+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.071390+0000) 2019-10-13T11:42:21.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:21.069+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.072205+0000) 2019-10-13T11:42:21.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:21.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:21.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:21.073+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.075784+0000) 2019-10-13T11:42:22.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:22.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.178327+0000) 2019-10-13T11:42:25.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:25.485+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:25.487539+0000) 2019-10-13T11:42:26.079 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:26.073+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.078371+0000) 2019-10-13T11:42:26.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:26.073+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.078744+0000) 2019-10-13T11:42:26.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:26.077+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.079353+0000) 2019-10-13T11:42:26.081 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:26.077+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.080493+0000) 2019-10-13T11:42:26.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:26.077+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.080755+0000) 2019-10-13T11:42:26.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:26.077+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.081281+0000) 2019-10-13T11:42:26.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:26.077+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.081837+0000) 2019-10-13T11:42:26.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:26.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:26.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:26.085+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.087093+0000) 2019-10-13T11:42:26.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:26.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:26.422912+0000) 2019-10-13T11:42:28.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:28.397+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.401299+0000) 2019-10-13T11:42:28.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:28.681+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.687043+0000) 2019-10-13T11:42:28.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:28.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.691639+0000) 2019-10-13T11:42:28.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:28.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.703254+0000) 2019-10-13T11:42:29.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:29.921+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:29.922163+0000) 2019-10-13T11:42:31.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:31.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.086753+0000) 2019-10-13T11:42:31.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:31.081+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.087095+0000) 2019-10-13T11:42:31.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:31.085+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.087596+0000) 2019-10-13T11:42:31.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:31.085+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.089078+0000) 2019-10-13T11:42:31.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:31.085+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.089146+0000) 2019-10-13T11:42:31.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:31.085+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.089858+0000) 2019-10-13T11:42:31.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:31.085+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.090343+0000) 2019-10-13T11:42:31.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:31.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:31.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:31.093+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.095525+0000) 2019-10-13T11:42:32.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:32.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.178556+0000) 2019-10-13T11:42:35.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:35.485+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:35.487733+0000) 2019-10-13T11:42:36.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:36.093+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.097175+0000) 2019-10-13T11:42:36.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:36.093+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.097663+0000) 2019-10-13T11:42:36.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:36.093+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.098055+0000) 2019-10-13T11:42:36.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:36.097+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.098457+0000) 2019-10-13T11:42:36.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:36.093+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.099019+0000) 2019-10-13T11:42:36.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:36.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.099426+0000) 2019-10-13T11:42:36.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:36.097+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.100055+0000) 2019-10-13T11:42:36.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:36.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:36.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:36.105+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.106769+0000) 2019-10-13T11:42:36.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:36.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:36.423093+0000) 2019-10-13T11:42:38.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:38.405+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.407816+0000) 2019-10-13T11:42:38.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:38.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.687252+0000) 2019-10-13T11:42:38.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:38.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.691831+0000) 2019-10-13T11:42:38.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:38.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.703476+0000) 2019-10-13T11:42:39.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:39.921+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:39.922385+0000) 2019-10-13T11:42:41.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:41.105+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.107488+0000) 2019-10-13T11:42:41.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:41.105+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.107942+0000) 2019-10-13T11:42:41.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:41.105+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.108930+0000) 2019-10-13T11:42:41.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:41.105+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.109651+0000) 2019-10-13T11:42:41.111 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:41.105+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.110545+0000) 2019-10-13T11:42:41.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:41.105+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.111040+0000) 2019-10-13T11:42:41.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:41.109+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.112479+0000) 2019-10-13T11:42:41.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:41.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:41.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:41.113+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.115678+0000) 2019-10-13T11:42:42.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:42.173+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.178755+0000) 2019-10-13T11:42:45.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:45.485+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:45.487935+0000) 2019-10-13T11:42:46.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:46.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.117639+0000) 2019-10-13T11:42:46.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:46.113+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.118475+0000) 2019-10-13T11:42:46.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:46.117+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.119665+0000) 2019-10-13T11:42:46.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:46.117+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.120409+0000) 2019-10-13T11:42:46.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:46.117+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.120853+0000) 2019-10-13T11:42:46.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:46.117+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.121646+0000) 2019-10-13T11:42:46.123 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:46.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.122791+0000) 2019-10-13T11:42:46.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:46.121+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.123347+0000) 2019-10-13T11:42:46.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:46.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:46.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:46.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:46.423275+0000) 2019-10-13T11:42:47.068 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:42:47.069 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:47.073 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:42:47.073 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:48.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:48.413+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.414339+0000) 2019-10-13T11:42:48.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:48.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.687510+0000) 2019-10-13T11:42:48.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:48.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.692044+0000) 2019-10-13T11:42:48.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:48.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.703655+0000) 2019-10-13T11:42:49.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:49.921+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:49.922621+0000) 2019-10-13T11:42:51.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:51.121+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.127230+0000) 2019-10-13T11:42:51.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:51.125+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.128078+0000) 2019-10-13T11:42:51.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:51.125+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.128657+0000) 2019-10-13T11:42:51.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:51.125+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.129442+0000) 2019-10-13T11:42:51.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:51.125+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.129995+0000) 2019-10-13T11:42:51.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:51.125+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.130299+0000) 2019-10-13T11:42:51.131 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:51.125+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.130825+0000) 2019-10-13T11:42:51.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:51.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:51.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:51.129+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.133764+0000) 2019-10-13T11:42:52.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:52.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.178958+0000) 2019-10-13T11:42:55.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:55.484+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:55.488159+0000) 2019-10-13T11:42:56.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:56.132+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.137335+0000) 2019-10-13T11:42:56.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:56.132+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.137646+0000) 2019-10-13T11:42:56.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:42:56.132+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.138026+0000) 2019-10-13T11:42:56.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:42:56.132+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.138680+0000) 2019-10-13T11:42:56.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:56.137+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.139236+0000) 2019-10-13T11:42:56.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:56.136+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.139872+0000) 2019-10-13T11:42:56.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:56.136+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.140712+0000) 2019-10-13T11:42:56.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:42:56.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130383 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:42:56.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:56.145+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.148336+0000) 2019-10-13T11:42:56.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:42:56.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:56.423487+0000) 2019-10-13T11:42:58.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:42:58.417+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.420834+0000) 2019-10-13T11:42:58.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:42:58.684+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.687709+0000) 2019-10-13T11:42:58.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:42:58.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.692239+0000) 2019-10-13T11:42:58.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:42:58.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.703868+0000) 2019-10-13T11:42:59.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:42:59.921+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:59.922830+0000) 2019-10-13T11:43:01.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:01.140+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.146472+0000) 2019-10-13T11:43:01.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:01.140+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.147025+0000) 2019-10-13T11:43:01.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:01.144+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.148456+0000) 2019-10-13T11:43:01.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:01.145+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.148861+0000) 2019-10-13T11:43:01.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:01.144+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.149318+0000) 2019-10-13T11:43:01.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:01.144+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.150002+0000) 2019-10-13T11:43:01.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:01.144+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.150369+0000) 2019-10-13T11:43:01.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:01.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:01.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:01.153+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.157773+0000) 2019-10-13T11:43:02.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:02.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.179164+0000) 2019-10-13T11:43:05.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:05.484+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:05.488361+0000) 2019-10-13T11:43:06.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:06.152+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.156563+0000) 2019-10-13T11:43:06.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:06.152+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.157608+0000) 2019-10-13T11:43:06.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:06.152+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.158031+0000) 2019-10-13T11:43:06.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:06.152+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.158451+0000) 2019-10-13T11:43:06.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:06.152+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.158954+0000) 2019-10-13T11:43:06.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:06.157+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.159372+0000) 2019-10-13T11:43:06.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:06.156+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.159751+0000) 2019-10-13T11:43:06.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:06.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:06.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:06.161+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.162570+0000) 2019-10-13T11:43:06.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:06.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:06.423667+0000) 2019-10-13T11:43:08.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:08.425+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.427456+0000) 2019-10-13T11:43:08.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:08.684+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.687931+0000) 2019-10-13T11:43:08.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:08.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.692449+0000) 2019-10-13T11:43:08.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:08.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.704061+0000) 2019-10-13T11:43:09.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:09.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:09.923025+0000) 2019-10-13T11:43:11.167 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:11.160+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.165594+0000) 2019-10-13T11:43:11.167 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:11.160+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.165949+0000) 2019-10-13T11:43:11.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:11.160+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.166870+0000) 2019-10-13T11:43:11.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:11.164+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.167562+0000) 2019-10-13T11:43:11.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:11.164+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.167926+0000) 2019-10-13T11:43:11.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:11.164+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.168242+0000) 2019-10-13T11:43:11.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:11.164+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.169146+0000) 2019-10-13T11:43:11.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:11.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:11.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:11.168+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.172855+0000) 2019-10-13T11:43:12.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:12.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.179378+0000) 2019-10-13T11:43:15.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:15.488+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:15.488631+0000) 2019-10-13T11:43:16.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:16.172+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.175487+0000) 2019-10-13T11:43:16.177 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:16.172+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.175812+0000) 2019-10-13T11:43:16.177 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:16.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.176533+0000) 2019-10-13T11:43:16.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:16.172+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.178076+0000) 2019-10-13T11:43:16.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:16.172+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.178434+0000) 2019-10-13T11:43:16.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:16.172+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.178968+0000) 2019-10-13T11:43:16.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:16.176+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.179587+0000) 2019-10-13T11:43:16.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:16.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130518 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:16.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:16.180+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.183923+0000) 2019-10-13T11:43:16.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:16.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:16.423893+0000) 2019-10-13T11:43:17.217 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:43:17.217 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:17.220 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:43:17.221 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:18.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:18.432+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.433771+0000) 2019-10-13T11:43:18.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:18.684+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.688136+0000) 2019-10-13T11:43:18.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:18.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.692648+0000) 2019-10-13T11:43:18.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:18.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.704256+0000) 2019-10-13T11:43:19.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:19.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:19.923248+0000) 2019-10-13T11:43:21.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:21.180+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.185829+0000) 2019-10-13T11:43:21.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:21.180+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.186237+0000) 2019-10-13T11:43:21.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:21.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.187198+0000) 2019-10-13T11:43:21.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:21.184+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.187908+0000) 2019-10-13T11:43:21.189 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:21.184+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.188766+0000) 2019-10-13T11:43:21.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:21.184+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.188877+0000) 2019-10-13T11:43:21.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:21.184+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.187559+0000) 2019-10-13T11:43:21.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:21.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:21.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:21.188+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.192859+0000) 2019-10-13T11:43:22.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:22.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.179578+0000) 2019-10-13T11:43:25.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:25.484+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:25.488833+0000) 2019-10-13T11:43:26.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:26.188+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.194725+0000) 2019-10-13T11:43:26.196 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:26.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.195770+0000) 2019-10-13T11:43:26.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:26.192+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.196301+0000) 2019-10-13T11:43:26.197 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:26.192+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.196685+0000) 2019-10-13T11:43:26.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:26.192+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.197405+0000) 2019-10-13T11:43:26.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:26.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.197784+0000) 2019-10-13T11:43:26.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:26.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.198111+0000) 2019-10-13T11:43:26.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:26.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:26.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:26.200+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.203273+0000) 2019-10-13T11:43:26.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:26.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:26.424070+0000) 2019-10-13T11:43:28.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:28.436+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.440339+0000) 2019-10-13T11:43:28.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:28.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.688338+0000) 2019-10-13T11:43:28.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:28.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.692847+0000) 2019-10-13T11:43:28.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:28.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.704458+0000) 2019-10-13T11:43:29.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:29.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:29.923446+0000) 2019-10-13T11:43:31.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:31.200+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.205189+0000) 2019-10-13T11:43:31.207 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:31.200+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.205782+0000) 2019-10-13T11:43:31.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:31.200+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.205767+0000) 2019-10-13T11:43:31.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:31.200+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.206492+0000) 2019-10-13T11:43:31.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:31.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.206790+0000) 2019-10-13T11:43:31.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:31.204+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.206965+0000) 2019-10-13T11:43:31.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:31.204+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.207642+0000) 2019-10-13T11:43:31.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:31.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:31.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:31.208+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.213859+0000) 2019-10-13T11:43:32.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:32.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.179749+0000) 2019-10-13T11:43:35.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:35.488+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:35.489088+0000) 2019-10-13T11:43:36.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:36.208+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.214043+0000) 2019-10-13T11:43:36.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:36.212+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.215469+0000) 2019-10-13T11:43:36.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:36.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.215849+0000) 2019-10-13T11:43:36.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:36.212+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.216620+0000) 2019-10-13T11:43:36.218 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:36.212+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.216372+0000) 2019-10-13T11:43:36.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:36.212+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.217056+0000) 2019-10-13T11:43:36.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:36.212+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.217618+0000) 2019-10-13T11:43:36.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:36.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:36.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:36.216+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.221209+0000) 2019-10-13T11:43:36.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:36.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:36.424246+0000) 2019-10-13T11:43:38.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:38.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.446878+0000) 2019-10-13T11:43:38.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:38.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.688537+0000) 2019-10-13T11:43:38.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:38.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.693040+0000) 2019-10-13T11:43:38.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:38.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.704678+0000) 2019-10-13T11:43:39.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:39.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:39.923695+0000) 2019-10-13T11:43:41.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:41.219+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.223441+0000) 2019-10-13T11:43:41.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:41.219+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.224074+0000) 2019-10-13T11:43:41.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:41.220+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.224392+0000) 2019-10-13T11:43:41.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:41.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.225602+0000) 2019-10-13T11:43:41.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:41.219+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.226305+0000) 2019-10-13T11:43:41.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:41.219+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.226708+0000) 2019-10-13T11:43:41.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:41.219+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.227222+0000) 2019-10-13T11:43:41.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:41.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:41.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:41.228+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.231866+0000) 2019-10-13T11:43:42.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:42.179+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.179926+0000) 2019-10-13T11:43:45.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:45.487+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:45.489284+0000) 2019-10-13T11:43:46.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:46.227+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.234289+0000) 2019-10-13T11:43:46.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:46.227+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.234724+0000) 2019-10-13T11:43:46.236 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:46.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.235110+0000) 2019-10-13T11:43:46.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:46.231+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.235732+0000) 2019-10-13T11:43:46.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:46.231+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.236139+0000) 2019-10-13T11:43:46.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:46.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.236514+0000) 2019-10-13T11:43:46.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:46.232+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.237491+0000) 2019-10-13T11:43:46.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:46.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:46.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:46.240+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.243456+0000) 2019-10-13T11:43:46.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:46.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:46.424450+0000) 2019-10-13T11:43:47.412 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:43:47.413 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:47.416 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:43:47.417 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:48.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:48.452+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.453358+0000) 2019-10-13T11:43:48.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:48.683+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.688748+0000) 2019-10-13T11:43:48.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:48.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.693240+0000) 2019-10-13T11:43:48.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:48.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.704906+0000) 2019-10-13T11:43:49.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:49.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:49.923922+0000) 2019-10-13T11:43:51.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:51.240+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.244027+0000) 2019-10-13T11:43:51.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:51.239+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.244755+0000) 2019-10-13T11:43:51.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:51.239+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.245161+0000) 2019-10-13T11:43:51.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:51.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.246297+0000) 2019-10-13T11:43:51.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:51.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.246807+0000) 2019-10-13T11:43:51.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:51.243+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.247369+0000) 2019-10-13T11:43:51.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:51.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.247452+0000) 2019-10-13T11:43:51.252 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:51.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:51.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:51.252+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.254468+0000) 2019-10-13T11:43:52.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:52.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.180123+0000) 2019-10-13T11:43:55.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:55.483+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:55.489507+0000) 2019-10-13T11:43:56.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:56.251+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.255082+0000) 2019-10-13T11:43:56.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:43:56.251+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.255690+0000) 2019-10-13T11:43:56.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:43:56.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.256253+0000) 2019-10-13T11:43:56.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:56.251+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.255957+0000) 2019-10-13T11:43:56.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:56.251+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.257027+0000) 2019-10-13T11:43:56.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:56.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.257542+0000) 2019-10-13T11:43:56.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:56.251+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.258536+0000) 2019-10-13T11:43:56.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:43:56.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:43:56.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:56.260+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.263642+0000) 2019-10-13T11:43:56.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:43:56.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:56.424662+0000) 2019-10-13T11:43:58.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:43:58.456+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.459471+0000) 2019-10-13T11:43:58.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:43:58.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.688947+0000) 2019-10-13T11:43:58.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:43:58.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.693437+0000) 2019-10-13T11:43:58.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:43:58.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.705135+0000) 2019-10-13T11:43:59.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:43:59.920+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:59.924142+0000) 2019-10-13T11:44:01.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:01.259+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.264553+0000) 2019-10-13T11:44:01.266 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:01.260+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.265218+0000) 2019-10-13T11:44:01.266 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:01.259+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.265640+0000) 2019-10-13T11:44:01.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:01.259+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.266234+0000) 2019-10-13T11:44:01.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:01.259+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.266856+0000) 2019-10-13T11:44:01.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:01.263+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.267915+0000) 2019-10-13T11:44:01.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:01.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.268523+0000) 2019-10-13T11:44:01.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:01.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:01.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:01.268+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.273828+0000) 2019-10-13T11:44:02.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:02.179+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.180307+0000) 2019-10-13T11:44:05.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:05.487+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:05.489740+0000) 2019-10-13T11:44:06.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:06.271+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.274744+0000) 2019-10-13T11:44:06.276 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:06.271+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.275175+0000) 2019-10-13T11:44:06.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:06.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.276413+0000) 2019-10-13T11:44:06.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:06.271+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.276532+0000) 2019-10-13T11:44:06.278 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:06.271+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.276979+0000) 2019-10-13T11:44:06.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:06.271+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.277962+0000) 2019-10-13T11:44:06.279 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:06.271+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.278417+0000) 2019-10-13T11:44:06.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:06.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:06.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:06.279+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.285057+0000) 2019-10-13T11:44:06.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:06.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:06.424840+0000) 2019-10-13T11:44:08.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:08.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.466064+0000) 2019-10-13T11:44:08.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:08.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.689141+0000) 2019-10-13T11:44:08.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:08.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.693664+0000) 2019-10-13T11:44:08.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:08.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.705354+0000) 2019-10-13T11:44:09.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:09.923+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:09.924370+0000) 2019-10-13T11:44:11.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:11.283+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.284501+0000) 2019-10-13T11:44:11.286 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:11.283+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.284802+0000) 2019-10-13T11:44:11.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:11.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.285240+0000) 2019-10-13T11:44:11.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:11.283+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.286511+0000) 2019-10-13T11:44:11.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:11.283+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.287059+0000) 2019-10-13T11:44:11.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:11.287+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.287859+0000) 2019-10-13T11:44:11.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:11.287+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.288411+0000) 2019-10-13T11:44:11.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:11.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:11.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:11.287+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.291859+0000) 2019-10-13T11:44:12.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:12.179+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.180489+0000) 2019-10-13T11:44:15.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:15.487+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:15.489939+0000) 2019-10-13T11:44:16.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:16.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.294383+0000) 2019-10-13T11:44:16.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:16.291+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.294820+0000) 2019-10-13T11:44:16.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:16.295+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.295591+0000) 2019-10-13T11:44:16.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:16.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.295737+0000) 2019-10-13T11:44:16.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:16.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.297522+0000) 2019-10-13T11:44:16.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:16.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.297863+0000) 2019-10-13T11:44:16.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:16.291+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.296743+0000) 2019-10-13T11:44:16.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:16.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:16.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:16.299+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.303325+0000) 2019-10-13T11:44:16.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:16.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:16.425055+0000) 2019-10-13T11:44:17.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:44:17.536 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:17.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:44:17.539 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:18.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:18.475+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.477797+0000) 2019-10-13T11:44:18.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:18.687+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.689368+0000) 2019-10-13T11:44:18.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:18.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.693881+0000) 2019-10-13T11:44:18.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:18.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.705506+0000) 2019-10-13T11:44:19.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:19.919+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:19.924595+0000) 2019-10-13T11:44:21.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:21.299+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.304548+0000) 2019-10-13T11:44:21.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:21.307+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.312546+0000) 2019-10-13T11:44:21.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:21.302+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.304372+0000) 2019-10-13T11:44:21.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:21.302+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.305211+0000) 2019-10-13T11:44:21.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:21.302+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.305687+0000) 2019-10-13T11:44:21.326 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:21.302+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.305990+0000) 2019-10-13T11:44:21.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:21.302+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.306733+0000) 2019-10-13T11:44:21.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:21.306+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.307856+0000) 2019-10-13T11:44:21.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:21.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:22.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:22.178+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.180670+0000) 2019-10-13T11:44:25.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:25.486+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:25.490138+0000) 2019-10-13T11:44:26.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:26.310+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.313605+0000) 2019-10-13T11:44:26.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:26.310+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.314586+0000) 2019-10-13T11:44:26.316 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:26.310+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.315030+0000) 2019-10-13T11:44:26.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:26.314+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.315470+0000) 2019-10-13T11:44:26.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:26.311+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.315948+0000) 2019-10-13T11:44:26.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:26.314+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.316648+0000) 2019-10-13T11:44:26.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:26.314+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.316416+0000) 2019-10-13T11:44:26.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:26.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 130995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:26.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:26.319+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.323102+0000) 2019-10-13T11:44:26.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:26.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:26.425243+0000) 2019-10-13T11:44:28.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:28.487+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.489537+0000) 2019-10-13T11:44:28.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:28.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.689575+0000) 2019-10-13T11:44:28.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:28.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.694107+0000) 2019-10-13T11:44:28.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:28.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.705722+0000) 2019-10-13T11:44:29.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:29.919+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:29.924792+0000) 2019-10-13T11:44:31.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:31.318+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.323183+0000) 2019-10-13T11:44:31.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:31.322+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.323752+0000) 2019-10-13T11:44:31.325 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:31.322+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.323969+0000) 2019-10-13T11:44:31.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:31.322+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.324361+0000) 2019-10-13T11:44:31.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:31.319+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.324614+0000) 2019-10-13T11:44:31.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:31.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.325305+0000) 2019-10-13T11:44:31.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:31.322+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.325779+0000) 2019-10-13T11:44:31.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:31.326+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:31.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:31.327+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.331465+0000) 2019-10-13T11:44:32.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:32.178+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.180882+0000) 2019-10-13T11:44:35.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:35.486+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:35.490372+0000) 2019-10-13T11:44:36.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:36.330+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.333217+0000) 2019-10-13T11:44:36.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:36.330+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.334547+0000) 2019-10-13T11:44:36.335 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:36.330+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.334091+0000) 2019-10-13T11:44:36.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:36.330+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.334857+0000) 2019-10-13T11:44:36.336 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:36.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.335584+0000) 2019-10-13T11:44:36.337 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:36.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.336141+0000) 2019-10-13T11:44:36.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:36.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.336815+0000) 2019-10-13T11:44:36.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:36.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:36.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:36.335+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.341931+0000) 2019-10-13T11:44:36.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:36.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:36.425452+0000) 2019-10-13T11:44:38.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:38.491+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.496188+0000) 2019-10-13T11:44:38.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:38.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.689774+0000) 2019-10-13T11:44:38.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:38.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.694306+0000) 2019-10-13T11:44:38.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:38.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.705953+0000) 2019-10-13T11:44:39.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:39.919+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:39.925013+0000) 2019-10-13T11:44:41.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:41.338+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.342636+0000) 2019-10-13T11:44:41.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:41.338+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.343159+0000) 2019-10-13T11:44:41.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:41.339+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.343616+0000) 2019-10-13T11:44:41.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:41.342+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.344296+0000) 2019-10-13T11:44:41.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:41.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.344431+0000) 2019-10-13T11:44:41.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:41.342+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.345379+0000) 2019-10-13T11:44:41.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:41.342+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.345724+0000) 2019-10-13T11:44:41.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:41.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:41.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:41.347+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.353005+0000) 2019-10-13T11:44:42.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:42.178+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.181058+0000) 2019-10-13T11:44:45.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:45.486+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:45.490576+0000) 2019-10-13T11:44:46.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:46.350+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.353123+0000) 2019-10-13T11:44:46.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:46.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.353864+0000) 2019-10-13T11:44:46.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:46.350+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.355157+0000) 2019-10-13T11:44:46.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:46.354+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.355550+0000) 2019-10-13T11:44:46.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:46.351+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.355647+0000) 2019-10-13T11:44:46.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:46.354+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.356042+0000) 2019-10-13T11:44:46.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:46.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.356438+0000) 2019-10-13T11:44:46.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:46.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:46.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:46.355+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.361360+0000) 2019-10-13T11:44:46.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:46.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:46.425626+0000) 2019-10-13T11:44:47.647 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:44:47.647 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:47.651 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:44:47.651 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:48.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:48.499+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.502782+0000) 2019-10-13T11:44:48.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:48.686+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.689980+0000) 2019-10-13T11:44:48.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:48.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.694518+0000) 2019-10-13T11:44:48.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:48.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.706154+0000) 2019-10-13T11:44:49.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:49.923+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:49.925236+0000) 2019-10-13T11:44:51.364 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:51.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.362954+0000) 2019-10-13T11:44:51.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:51.362+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.363587+0000) 2019-10-13T11:44:51.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:51.362+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.364325+0000) 2019-10-13T11:44:51.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:51.362+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.364839+0000) 2019-10-13T11:44:51.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:51.362+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.364853+0000) 2019-10-13T11:44:51.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:51.362+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.366256+0000) 2019-10-13T11:44:51.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:51.366+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.367448+0000) 2019-10-13T11:44:51.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:51.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:51.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:51.367+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.373109+0000) 2019-10-13T11:44:52.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:52.178+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.181271+0000) 2019-10-13T11:44:55.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:55.486+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:55.490809+0000) 2019-10-13T11:44:56.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:44:56.370+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.373123+0000) 2019-10-13T11:44:56.374 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:56.370+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.373762+0000) 2019-10-13T11:44:56.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:44:56.370+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.374508+0000) 2019-10-13T11:44:56.375 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:56.370+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.374728+0000) 2019-10-13T11:44:56.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:56.374+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.375426+0000) 2019-10-13T11:44:56.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:56.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.375901+0000) 2019-10-13T11:44:56.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:56.374+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.376381+0000) 2019-10-13T11:44:56.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:44:56.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:44:56.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:56.379+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.383118+0000) 2019-10-13T11:44:56.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:44:56.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:56.425802+0000) 2019-10-13T11:44:58.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:44:58.511+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.514462+0000) 2019-10-13T11:44:58.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:44:58.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.690181+0000) 2019-10-13T11:44:58.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:44:58.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.694716+0000) 2019-10-13T11:44:58.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:44:58.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.706381+0000) 2019-10-13T11:44:59.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:44:59.918+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:59.925440+0000) 2019-10-13T11:45:01.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:01.378+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.382413+0000) 2019-10-13T11:45:01.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:01.382+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.383726+0000) 2019-10-13T11:45:01.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:01.378+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.383926+0000) 2019-10-13T11:45:01.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:01.382+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.384588+0000) 2019-10-13T11:45:01.386 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:01.382+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.384956+0000) 2019-10-13T11:45:01.386 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:01.382+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.385543+0000) 2019-10-13T11:45:01.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:01.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.386237+0000) 2019-10-13T11:45:01.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:01.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131236 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:01.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:01.386+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.393470+0000) 2019-10-13T11:45:02.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:02.178+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.181453+0000) 2019-10-13T11:45:05.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:05.485+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:05.491009+0000) 2019-10-13T11:45:06.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:06.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.393450+0000) 2019-10-13T11:45:06.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:06.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.394433+0000) 2019-10-13T11:45:06.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:06.389+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.394913+0000) 2019-10-13T11:45:06.396 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:06.390+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.395209+0000) 2019-10-13T11:45:06.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:06.393+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.395538+0000) 2019-10-13T11:45:06.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:06.393+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.396175+0000) 2019-10-13T11:45:06.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:06.393+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.396424+0000) 2019-10-13T11:45:06.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:06.394+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.398775+0000) 2019-10-13T11:45:06.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:06.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:06.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:06.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:06.425992+0000) 2019-10-13T11:45:08.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:08.518+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.521093+0000) 2019-10-13T11:45:08.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:08.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.690409+0000) 2019-10-13T11:45:08.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:08.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.694908+0000) 2019-10-13T11:45:08.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:08.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.706626+0000) 2019-10-13T11:45:09.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:09.922+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:09.925626+0000) 2019-10-13T11:45:11.405 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:11.401+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.403651+0000) 2019-10-13T11:45:11.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:11.398+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.403915+0000) 2019-10-13T11:45:11.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:11.401+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.404602+0000) 2019-10-13T11:45:11.407 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:11.401+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.404856+0000) 2019-10-13T11:45:11.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:11.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.405481+0000) 2019-10-13T11:45:11.408 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:11.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.405967+0000) 2019-10-13T11:45:11.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:11.401+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.406225+0000) 2019-10-13T11:45:11.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:11.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:11.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:11.406+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.412800+0000) 2019-10-13T11:45:12.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:12.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.181664+0000) 2019-10-13T11:45:15.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:15.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:15.491208+0000) 2019-10-13T11:45:16.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:16.406+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.413215+0000) 2019-10-13T11:45:16.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:16.409+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.414097+0000) 2019-10-13T11:45:16.415 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:16.409+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.414424+0000) 2019-10-13T11:45:16.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:16.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.414828+0000) 2019-10-13T11:45:16.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:16.413+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.415634+0000) 2019-10-13T11:45:16.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:16.413+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.416715+0000) 2019-10-13T11:45:16.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:16.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.417375+0000) 2019-10-13T11:45:16.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:16.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:16.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:16.418+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.422225+0000) 2019-10-13T11:45:16.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:16.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:16.426108+0000) 2019-10-13T11:45:17.792 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:45:17.792 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:17.796 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:45:17.796 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:18.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:18.526+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.527671+0000) 2019-10-13T11:45:18.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:18.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.690627+0000) 2019-10-13T11:45:18.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:18.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.695100+0000) 2019-10-13T11:45:18.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:18.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.706839+0000) 2019-10-13T11:45:19.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:19.922+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:19.925881+0000) 2019-10-13T11:45:21.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:21.421+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.424356+0000) 2019-10-13T11:45:21.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:21.421+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.424595+0000) 2019-10-13T11:45:21.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:21.418+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.424755+0000) 2019-10-13T11:45:21.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:21.421+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.425296+0000) 2019-10-13T11:45:21.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:21.421+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.425737+0000) 2019-10-13T11:45:21.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:21.421+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.426467+0000) 2019-10-13T11:45:21.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:21.421+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.427002+0000) 2019-10-13T11:45:21.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:21.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:21.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:21.426+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.432909+0000) 2019-10-13T11:45:22.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:22.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.181839+0000) 2019-10-13T11:45:25.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:25.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:25.491408+0000) 2019-10-13T11:45:26.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:26.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.426263+0000) 2019-10-13T11:45:26.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:26.429+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.432992+0000) 2019-10-13T11:45:26.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:26.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.433456+0000) 2019-10-13T11:45:26.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:26.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.433974+0000) 2019-10-13T11:45:26.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:26.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.434213+0000) 2019-10-13T11:45:26.436 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:26.429+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.434825+0000) 2019-10-13T11:45:26.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:26.430+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.435500+0000) 2019-10-13T11:45:26.436 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:26.433+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.436075+0000) 2019-10-13T11:45:26.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:26.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:26.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:26.434+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:26.441553+0000) 2019-10-13T11:45:28.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:28.538+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.538871+0000) 2019-10-13T11:45:28.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:28.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.690825+0000) 2019-10-13T11:45:28.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:28.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.695296+0000) 2019-10-13T11:45:28.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:28.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.707058+0000) 2019-10-13T11:45:29.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:29.922+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:29.926124+0000) 2019-10-13T11:45:31.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:31.437+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.442779+0000) 2019-10-13T11:45:31.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:31.438+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.443081+0000) 2019-10-13T11:45:31.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:31.441+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.443592+0000) 2019-10-13T11:45:31.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:31.441+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.444203+0000) 2019-10-13T11:45:31.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:31.441+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.444413+0000) 2019-10-13T11:45:31.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:31.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.445016+0000) 2019-10-13T11:45:31.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:31.441+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.445020+0000) 2019-10-13T11:45:31.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:31.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:31.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:31.446+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.453651+0000) 2019-10-13T11:45:32.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:32.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.182018+0000) 2019-10-13T11:45:35.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:35.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:35.491577+0000) 2019-10-13T11:45:36.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:36.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.426460+0000) 2019-10-13T11:45:36.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:36.449+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.453276+0000) 2019-10-13T11:45:36.455 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:36.449+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.454139+0000) 2019-10-13T11:45:36.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:36.449+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.454120+0000) 2019-10-13T11:45:36.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:36.449+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.454757+0000) 2019-10-13T11:45:36.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:36.450+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.455506+0000) 2019-10-13T11:45:36.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:36.453+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.456284+0000) 2019-10-13T11:45:36.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:36.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.456624+0000) 2019-10-13T11:45:36.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:36.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:36.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:36.454+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:36.461791+0000) 2019-10-13T11:45:38.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:38.542+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.545342+0000) 2019-10-13T11:45:38.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:38.685+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.691053+0000) 2019-10-13T11:45:38.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:38.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.695537+0000) 2019-10-13T11:45:38.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:38.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.707256+0000) 2019-10-13T11:45:39.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:39.926+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:39.926386+0000) 2019-10-13T11:45:41.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:41.457+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.463058+0000) 2019-10-13T11:45:41.465 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:41.461+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.463748+0000) 2019-10-13T11:45:41.465 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:41.461+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.464087+0000) 2019-10-13T11:45:41.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:41.458+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.464335+0000) 2019-10-13T11:45:41.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:41.461+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.465220+0000) 2019-10-13T11:45:41.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:41.461+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.466237+0000) 2019-10-13T11:45:41.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:41.461+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.466896+0000) 2019-10-13T11:45:41.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:41.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:41.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:41.466+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.471358+0000) 2019-10-13T11:45:42.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:42.177+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.182178+0000) 2019-10-13T11:45:45.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:45.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:45.491800+0000) 2019-10-13T11:45:46.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:46.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.426675+0000) 2019-10-13T11:45:46.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:46.468+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.472429+0000) 2019-10-13T11:45:46.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:46.470+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.472616+0000) 2019-10-13T11:45:46.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:46.468+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.473658+0000) 2019-10-13T11:45:46.475 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:46.468+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.474533+0000) 2019-10-13T11:45:46.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:46.468+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.474982+0000) 2019-10-13T11:45:46.476 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:46.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.475719+0000) 2019-10-13T11:45:46.477 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:46.472+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.475779+0000) 2019-10-13T11:45:46.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:46.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131537 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:46.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:46.482+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:46.482840+0000) 2019-10-13T11:45:47.916 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:45:47.916 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:47.920 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:45:47.920 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:48.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:48.550+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.551889+0000) 2019-10-13T11:45:48.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:48.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.691337+0000) 2019-10-13T11:45:48.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:48.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.695732+0000) 2019-10-13T11:45:48.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:48.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.707490+0000) 2019-10-13T11:45:49.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:49.926+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:49.926601+0000) 2019-10-13T11:45:51.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:51.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.482901+0000) 2019-10-13T11:45:51.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:51.476+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.483205+0000) 2019-10-13T11:45:51.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:51.482+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.484144+0000) 2019-10-13T11:45:51.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:51.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.484915+0000) 2019-10-13T11:45:51.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:51.480+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.485210+0000) 2019-10-13T11:45:51.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:51.480+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.485672+0000) 2019-10-13T11:45:51.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:51.480+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.486054+0000) 2019-10-13T11:45:51.490 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:51.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:51.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:51.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.493246+0000) 2019-10-13T11:45:52.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:52.180+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.182363+0000) 2019-10-13T11:45:55.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:55.488+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:55.492006+0000) 2019-10-13T11:45:56.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:56.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.426901+0000) 2019-10-13T11:45:56.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:45:56.488+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.493578+0000) 2019-10-13T11:45:56.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:45:56.488+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.493880+0000) 2019-10-13T11:45:56.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:56.488+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.494387+0000) 2019-10-13T11:45:56.495 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:56.488+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.494847+0000) 2019-10-13T11:45:56.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:56.493+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.495271+0000) 2019-10-13T11:45:56.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:45:56.492+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.495745+0000) 2019-10-13T11:45:56.497 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:56.492+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.496000+0000) 2019-10-13T11:45:56.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:45:56.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:45:56.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:56.501+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:56.505403+0000) 2019-10-13T11:45:58.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:45:58.557+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.558385+0000) 2019-10-13T11:45:58.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:45:58.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.691537+0000) 2019-10-13T11:45:58.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:45:58.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.695923+0000) 2019-10-13T11:45:58.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:45:58.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.707705+0000) 2019-10-13T11:45:59.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:45:59.925+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:59.926794+0000) 2019-10-13T11:46:01.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:01.500+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.503927+0000) 2019-10-13T11:46:01.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:01.501+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.504259+0000) 2019-10-13T11:46:01.506 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:01.500+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.505045+0000) 2019-10-13T11:46:01.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:01.500+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.505609+0000) 2019-10-13T11:46:01.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:01.500+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.505912+0000) 2019-10-13T11:46:01.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:01.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.506569+0000) 2019-10-13T11:46:01.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:01.500+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.506990+0000) 2019-10-13T11:46:01.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:01.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:01.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:01.513+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.515874+0000) 2019-10-13T11:46:02.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:02.180+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.182534+0000) 2019-10-13T11:46:05.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:05.488+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:05.492204+0000) 2019-10-13T11:46:06.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:06.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.427088+0000) 2019-10-13T11:46:06.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:06.508+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.512942+0000) 2019-10-13T11:46:06.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:06.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.514341+0000) 2019-10-13T11:46:06.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:06.508+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.514429+0000) 2019-10-13T11:46:06.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:06.508+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.515260+0000) 2019-10-13T11:46:06.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:06.512+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.515783+0000) 2019-10-13T11:46:06.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:06.513+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.516451+0000) 2019-10-13T11:46:06.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:06.512+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.517078+0000) 2019-10-13T11:46:06.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:06.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:06.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:06.521+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:06.523869+0000) 2019-10-13T11:46:08.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:08.561+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.564896+0000) 2019-10-13T11:46:08.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:08.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.691779+0000) 2019-10-13T11:46:08.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:08.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.696133+0000) 2019-10-13T11:46:08.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:08.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.707918+0000) 2019-10-13T11:46:09.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:09.925+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:09.927006+0000) 2019-10-13T11:46:11.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:11.516+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.523030+0000) 2019-10-13T11:46:11.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:11.520+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.524413+0000) 2019-10-13T11:46:11.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:11.520+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.525039+0000) 2019-10-13T11:46:11.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:11.520+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.525195+0000) 2019-10-13T11:46:11.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:11.520+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.525389+0000) 2019-10-13T11:46:11.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:11.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.525940+0000) 2019-10-13T11:46:11.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:11.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.526372+0000) 2019-10-13T11:46:11.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:11.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:11.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:11.529+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.533204+0000) 2019-10-13T11:46:12.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:12.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.182733+0000) 2019-10-13T11:46:15.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:15.492+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:15.492413+0000) 2019-10-13T11:46:16.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:16.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.427281+0000) 2019-10-13T11:46:16.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:16.528+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.533897+0000) 2019-10-13T11:46:16.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:16.528+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.534412+0000) 2019-10-13T11:46:16.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:16.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.535175+0000) 2019-10-13T11:46:16.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:16.528+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.535258+0000) 2019-10-13T11:46:16.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:16.532+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.535593+0000) 2019-10-13T11:46:16.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:16.532+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.536134+0000) 2019-10-13T11:46:16.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:16.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.536912+0000) 2019-10-13T11:46:16.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:16.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:16.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:16.545+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:16.547142+0000) 2019-10-13T11:46:17.964 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:46:17.964 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:17.968 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:46:17.968 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:18.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:18.569+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.570781+0000) 2019-10-13T11:46:18.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:18.688+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.692009+0000) 2019-10-13T11:46:18.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:18.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.696326+0000) 2019-10-13T11:46:18.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:18.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.708125+0000) 2019-10-13T11:46:19.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:19.925+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:19.927208+0000) 2019-10-13T11:46:21.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:21.536+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.542672+0000) 2019-10-13T11:46:21.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:21.540+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.543751+0000) 2019-10-13T11:46:21.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:21.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.543111+0000) 2019-10-13T11:46:21.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:21.540+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.544322+0000) 2019-10-13T11:46:21.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:21.540+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.544941+0000) 2019-10-13T11:46:21.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:21.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.545237+0000) 2019-10-13T11:46:21.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:21.540+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.545583+0000) 2019-10-13T11:46:21.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:21.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:21.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:21.549+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.552131+0000) 2019-10-13T11:46:22.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:22.176+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.182929+0000) 2019-10-13T11:46:25.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:25.488+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:25.492645+0000) 2019-10-13T11:46:26.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:26.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.427500+0000) 2019-10-13T11:46:26.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:26.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.554252+0000) 2019-10-13T11:46:26.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:26.548+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.554539+0000) 2019-10-13T11:46:26.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:26.548+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.554897+0000) 2019-10-13T11:46:26.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:26.548+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.555254+0000) 2019-10-13T11:46:26.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:26.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.556502+0000) 2019-10-13T11:46:26.558 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:26.552+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.556997+0000) 2019-10-13T11:46:26.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:26.552+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.557163+0000) 2019-10-13T11:46:26.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:26.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:26.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:26.561+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:26.562470+0000) 2019-10-13T11:46:28.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:28.573+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.577266+0000) 2019-10-13T11:46:28.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:28.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.692209+0000) 2019-10-13T11:46:28.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:28.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.696531+0000) 2019-10-13T11:46:28.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:28.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.708319+0000) 2019-10-13T11:46:29.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:29.925+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:29.927410+0000) 2019-10-13T11:46:31.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:31.561+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.563700+0000) 2019-10-13T11:46:31.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:31.559+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.564553+0000) 2019-10-13T11:46:31.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:31.559+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.564918+0000) 2019-10-13T11:46:31.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:31.559+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.565966+0000) 2019-10-13T11:46:31.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:31.559+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.566415+0000) 2019-10-13T11:46:31.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:31.559+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.566736+0000) 2019-10-13T11:46:31.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:31.563+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.568209+0000) 2019-10-13T11:46:31.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:31.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:31.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:31.569+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.573538+0000) 2019-10-13T11:46:32.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:32.175+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.183136+0000) 2019-10-13T11:46:35.493 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:35.491+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:35.492844+0000) 2019-10-13T11:46:36.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:36.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.427697+0000) 2019-10-13T11:46:36.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:36.567+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.573437+0000) 2019-10-13T11:46:36.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:36.567+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.573823+0000) 2019-10-13T11:46:36.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:36.567+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.574632+0000) 2019-10-13T11:46:36.575 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:36.573+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.574934+0000) 2019-10-13T11:46:36.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:36.571+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.575485+0000) 2019-10-13T11:46:36.577 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:36.571+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.576785+0000) 2019-10-13T11:46:36.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:36.571+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.577064+0000) 2019-10-13T11:46:36.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:36.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:36.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:36.577+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:36.581730+0000) 2019-10-13T11:46:38.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:38.581+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.583222+0000) 2019-10-13T11:46:38.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:38.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.692422+0000) 2019-10-13T11:46:38.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:38.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.696732+0000) 2019-10-13T11:46:38.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:38.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.708552+0000) 2019-10-13T11:46:39.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:39.925+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:39.927618+0000) 2019-10-13T11:46:41.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:41.575+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.582653+0000) 2019-10-13T11:46:41.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:41.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.583369+0000) 2019-10-13T11:46:41.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:41.581+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.583639+0000) 2019-10-13T11:46:41.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:41.579+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.584251+0000) 2019-10-13T11:46:41.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:41.579+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.584736+0000) 2019-10-13T11:46:41.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:41.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.585269+0000) 2019-10-13T11:46:41.586 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:41.579+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.586036+0000) 2019-10-13T11:46:41.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:41.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:41.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:41.589+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.591994+0000) 2019-10-13T11:46:42.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:42.179+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.183342+0000) 2019-10-13T11:46:45.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:45.491+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:45.493050+0000) 2019-10-13T11:46:46.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:46.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.427845+0000) 2019-10-13T11:46:46.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:46.589+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.592380+0000) 2019-10-13T11:46:46.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:46.587+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.592899+0000) 2019-10-13T11:46:46.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:46.587+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.593741+0000) 2019-10-13T11:46:46.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:46.587+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.594369+0000) 2019-10-13T11:46:46.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:46.587+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.594740+0000) 2019-10-13T11:46:46.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:46.591+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.595471+0000) 2019-10-13T11:46:46.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:46.591+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.596021+0000) 2019-10-13T11:46:46.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:46.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:46.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:46.597+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:46.600402+0000) 2019-10-13T11:46:48.120 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:46:48.121 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:48.124 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:46:48.124 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:48.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:48.588+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.590254+0000) 2019-10-13T11:46:48.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:48.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.692624+0000) 2019-10-13T11:46:48.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:48.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.696963+0000) 2019-10-13T11:46:48.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:48.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.708745+0000) 2019-10-13T11:46:49.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:49.924+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:49.927818+0000) 2019-10-13T11:46:51.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:51.595+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.602902+0000) 2019-10-13T11:46:51.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:51.599+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.603357+0000) 2019-10-13T11:46:51.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:51.599+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.604018+0000) 2019-10-13T11:46:51.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:51.599+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.603855+0000) 2019-10-13T11:46:51.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:51.599+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.604590+0000) 2019-10-13T11:46:51.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:51.600+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.605218+0000) 2019-10-13T11:46:51.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:51.599+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.605631+0000) 2019-10-13T11:46:51.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:51.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 131973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:51.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:51.608+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.613188+0000) 2019-10-13T11:46:52.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:52.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.183543+0000) 2019-10-13T11:46:55.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:55.491+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:55.493283+0000) 2019-10-13T11:46:56.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:56.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.428046+0000) 2019-10-13T11:46:56.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:46:56.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.612008+0000) 2019-10-13T11:46:56.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:56.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.612663+0000) 2019-10-13T11:46:56.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:56.607+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.613186+0000) 2019-10-13T11:46:56.615 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:46:56.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.613803+0000) 2019-10-13T11:46:56.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:56.607+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.615275+0000) 2019-10-13T11:46:56.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:46:56.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.615595+0000) 2019-10-13T11:46:56.617 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:56.612+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.616029+0000) 2019-10-13T11:46:56.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:46:56.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:46:56.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:56.616+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:56.620921+0000) 2019-10-13T11:46:58.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:46:58.592+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.597219+0000) 2019-10-13T11:46:58.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:46:58.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.692821+0000) 2019-10-13T11:46:58.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:46:58.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.697195+0000) 2019-10-13T11:46:58.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:46:58.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.708942+0000) 2019-10-13T11:46:59.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:46:59.924+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:59.928020+0000) 2019-10-13T11:47:01.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:01.619+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.622501+0000) 2019-10-13T11:47:01.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:01.619+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.622837+0000) 2019-10-13T11:47:01.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:01.623+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.623633+0000) 2019-10-13T11:47:01.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:01.623+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.623882+0000) 2019-10-13T11:47:01.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:01.623+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.624138+0000) 2019-10-13T11:47:01.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:01.620+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.625016+0000) 2019-10-13T11:47:01.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:01.623+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.625757+0000) 2019-10-13T11:47:01.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:01.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:01.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:01.628+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.630840+0000) 2019-10-13T11:47:02.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:02.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.183737+0000) 2019-10-13T11:47:05.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:05.491+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:05.493494+0000) 2019-10-13T11:47:06.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:06.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.428265+0000) 2019-10-13T11:47:06.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:06.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.632368+0000) 2019-10-13T11:47:06.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:06.631+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.632480+0000) 2019-10-13T11:47:06.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:06.631+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.632827+0000) 2019-10-13T11:47:06.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:06.631+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.633121+0000) 2019-10-13T11:47:06.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:06.631+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.635197+0000) 2019-10-13T11:47:06.636 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:06.632+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.635788+0000) 2019-10-13T11:47:06.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:06.635+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.636078+0000) 2019-10-13T11:47:06.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:06.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:06.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:06.640+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:06.642500+0000) 2019-10-13T11:47:08.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:08.600+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.604215+0000) 2019-10-13T11:47:08.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:08.691+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.693013+0000) 2019-10-13T11:47:08.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:08.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.697421+0000) 2019-10-13T11:47:08.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:08.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.709174+0000) 2019-10-13T11:47:09.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:09.924+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:09.928253+0000) 2019-10-13T11:47:11.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:11.639+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.642057+0000) 2019-10-13T11:47:11.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:11.639+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.642212+0000) 2019-10-13T11:47:11.644 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:11.639+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.643022+0000) 2019-10-13T11:47:11.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:11.639+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.643057+0000) 2019-10-13T11:47:11.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:11.643+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.643923+0000) 2019-10-13T11:47:11.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:11.643+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.644897+0000) 2019-10-13T11:47:11.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:11.640+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.645067+0000) 2019-10-13T11:47:11.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:11.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:11.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:11.648+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.650756+0000) 2019-10-13T11:47:12.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:12.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.183975+0000) 2019-10-13T11:47:15.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:15.490+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:15.493715+0000) 2019-10-13T11:47:16.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:16.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.428456+0000) 2019-10-13T11:47:16.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:16.650+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.651448+0000) 2019-10-13T11:47:16.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:16.650+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.652106+0000) 2019-10-13T11:47:16.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:16.650+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.652248+0000) 2019-10-13T11:47:16.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:16.650+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.652665+0000) 2019-10-13T11:47:16.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:16.650+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.653273+0000) 2019-10-13T11:47:16.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:16.648+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.653401+0000) 2019-10-13T11:47:16.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:16.650+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.654186+0000) 2019-10-13T11:47:16.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:16.652+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:16.657438+0000) 2019-10-13T11:47:16.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:16.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132138 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:18.336 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:47:18.337 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:18.340 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:47:18.340 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:18.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:18.608+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.611928+0000) 2019-10-13T11:47:18.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:18.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.693213+0000) 2019-10-13T11:47:18.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:18.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.697597+0000) 2019-10-13T11:47:18.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:18.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.709385+0000) 2019-10-13T11:47:19.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:19.924+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:19.928409+0000) 2019-10-13T11:47:21.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:21.658+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.661623+0000) 2019-10-13T11:47:21.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:21.660+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.662395+0000) 2019-10-13T11:47:21.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:21.662+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.663433+0000) 2019-10-13T11:47:21.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:21.662+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.663824+0000) 2019-10-13T11:47:21.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:21.662+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.664142+0000) 2019-10-13T11:47:21.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:21.662+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.664494+0000) 2019-10-13T11:47:21.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:21.662+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.664872+0000) 2019-10-13T11:47:21.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:21.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:21.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:21.664+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.668433+0000) 2019-10-13T11:47:22.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:22.182+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.184157+0000) 2019-10-13T11:47:25.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:25.490+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:25.493869+0000) 2019-10-13T11:47:26.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:26.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.428656+0000) 2019-10-13T11:47:26.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:26.666+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.670711+0000) 2019-10-13T11:47:26.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:26.666+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.671102+0000) 2019-10-13T11:47:26.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:26.670+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.672310+0000) 2019-10-13T11:47:26.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:26.670+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.673044+0000) 2019-10-13T11:47:26.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:26.668+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.673440+0000) 2019-10-13T11:47:26.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:26.670+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.674576+0000) 2019-10-13T11:47:26.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:26.670+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.675149+0000) 2019-10-13T11:47:26.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:26.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:26.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:26.676+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:26.679418+0000) 2019-10-13T11:47:28.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:28.620+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.623234+0000) 2019-10-13T11:47:28.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:28.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.693405+0000) 2019-10-13T11:47:28.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:28.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.697785+0000) 2019-10-13T11:47:28.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:28.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.709614+0000) 2019-10-13T11:47:29.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:29.928+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:29.928604+0000) 2019-10-13T11:47:31.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:31.678+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.679645+0000) 2019-10-13T11:47:31.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:31.678+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.680102+0000) 2019-10-13T11:47:31.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:31.676+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.681996+0000) 2019-10-13T11:47:31.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:31.678+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.682761+0000) 2019-10-13T11:47:31.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:31.678+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.683032+0000) 2019-10-13T11:47:31.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:31.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.683659+0000) 2019-10-13T11:47:31.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:31.682+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.683919+0000) 2019-10-13T11:47:31.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:31.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:31.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:31.684+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.688688+0000) 2019-10-13T11:47:32.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:32.182+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.184363+0000) 2019-10-13T11:47:35.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:35.490+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:35.494040+0000) 2019-10-13T11:47:36.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:36.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.428869+0000) 2019-10-13T11:47:36.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:36.688+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.692084+0000) 2019-10-13T11:47:36.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:36.686+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.689255+0000) 2019-10-13T11:47:36.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:36.686+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.690101+0000) 2019-10-13T11:47:36.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:36.686+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.690855+0000) 2019-10-13T11:47:36.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:36.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.691495+0000) 2019-10-13T11:47:36.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:36.690+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.691891+0000) 2019-10-13T11:47:36.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:36.690+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.693225+0000) 2019-10-13T11:47:36.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:36.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:36.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:36.696+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:36.698685+0000) 2019-10-13T11:47:38.630 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:38.628+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.629626+0000) 2019-10-13T11:47:38.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:38.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.693607+0000) 2019-10-13T11:47:38.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:38.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.697991+0000) 2019-10-13T11:47:38.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:38.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.709840+0000) 2019-10-13T11:47:39.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:39.924+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:39.928796+0000) 2019-10-13T11:47:41.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:41.698+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.699394+0000) 2019-10-13T11:47:41.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:41.698+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.700805+0000) 2019-10-13T11:47:41.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:41.698+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.701307+0000) 2019-10-13T11:47:41.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:41.698+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.701637+0000) 2019-10-13T11:47:41.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:41.696+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.701728+0000) 2019-10-13T11:47:41.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:41.698+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.702718+0000) 2019-10-13T11:47:41.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:41.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.703378+0000) 2019-10-13T11:47:41.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:41.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:41.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:41.708+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.710267+0000) 2019-10-13T11:47:42.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:42.182+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.184546+0000) 2019-10-13T11:47:45.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:45.490+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:45.494291+0000) 2019-10-13T11:47:46.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:46.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.429076+0000) 2019-10-13T11:47:46.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:46.706+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.710211+0000) 2019-10-13T11:47:46.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:46.706+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.710655+0000) 2019-10-13T11:47:46.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:46.707+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.711088+0000) 2019-10-13T11:47:46.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:46.706+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.711156+0000) 2019-10-13T11:47:46.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:46.710+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.711918+0000) 2019-10-13T11:47:46.713 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:46.710+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.712315+0000) 2019-10-13T11:47:46.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:46.710+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.712498+0000) 2019-10-13T11:47:46.717 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:46.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:46.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:46.711+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:46.717708+0000) 2019-10-13T11:47:48.457 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:47:48.457 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:48.461 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:47:48.461 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:48.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:48.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.636273+0000) 2019-10-13T11:47:48.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:48.690+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.693833+0000) 2019-10-13T11:47:48.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:48.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.698190+0000) 2019-10-13T11:47:48.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:48.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.710040+0000) 2019-10-13T11:47:49.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:49.927+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:49.928988+0000) 2019-10-13T11:47:51.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:51.715+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.719370+0000) 2019-10-13T11:47:51.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:51.718+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.720089+0000) 2019-10-13T11:47:51.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:51.718+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.720441+0000) 2019-10-13T11:47:51.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:51.718+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.720783+0000) 2019-10-13T11:47:51.722 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:51.718+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.721195+0000) 2019-10-13T11:47:51.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:51.718+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.721787+0000) 2019-10-13T11:47:51.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:51.718+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.722563+0000) 2019-10-13T11:47:51.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:51.722+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:51.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:51.723+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.728792+0000) 2019-10-13T11:47:52.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:52.182+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.184725+0000) 2019-10-13T11:47:55.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:55.490+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:55.494484+0000) 2019-10-13T11:47:56.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:56.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.429300+0000) 2019-10-13T11:47:56.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:47:56.725+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.729745+0000) 2019-10-13T11:47:56.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:56.725+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.730696+0000) 2019-10-13T11:47:56.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:56.729+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.731937+0000) 2019-10-13T11:47:56.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:47:56.729+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.732515+0000) 2019-10-13T11:47:56.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:56.729+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.732997+0000) 2019-10-13T11:47:56.738 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:47:56.729+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.733613+0000) 2019-10-13T11:47:56.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:56.723+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.730072+0000) 2019-10-13T11:47:56.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:47:56.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:47:56.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:56.735+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:56.738571+0000) 2019-10-13T11:47:58.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:47:58.639+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.642860+0000) 2019-10-13T11:47:58.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:47:58.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.694092+0000) 2019-10-13T11:47:58.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:47:58.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.698373+0000) 2019-10-13T11:47:58.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:47:58.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.710242+0000) 2019-10-13T11:47:59.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:47:59.923+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:59.929175+0000) 2019-10-13T11:48:01.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:01.737+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.741084+0000) 2019-10-13T11:48:01.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:01.737+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.741818+0000) 2019-10-13T11:48:01.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:01.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.742385+0000) 2019-10-13T11:48:01.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:01.739+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.742915+0000) 2019-10-13T11:48:01.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:01.741+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.744006+0000) 2019-10-13T11:48:01.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:01.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.744513+0000) 2019-10-13T11:48:01.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:01.741+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.744976+0000) 2019-10-13T11:48:01.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:01.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132447 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:01.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:01.743+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.748166+0000) 2019-10-13T11:48:02.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:02.181+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.184906+0000) 2019-10-13T11:48:05.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:05.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:05.494679+0000) 2019-10-13T11:48:06.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:06.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.429498+0000) 2019-10-13T11:48:06.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:06.745+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.750619+0000) 2019-10-13T11:48:06.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:06.749+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.751793+0000) 2019-10-13T11:48:06.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:06.747+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.752482+0000) 2019-10-13T11:48:06.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:06.749+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.753305+0000) 2019-10-13T11:48:06.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:06.749+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.753550+0000) 2019-10-13T11:48:06.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:06.749+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.754550+0000) 2019-10-13T11:48:06.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:06.749+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.755105+0000) 2019-10-13T11:48:06.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:06.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:06.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:06.755+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:06.759379+0000) 2019-10-13T11:48:08.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:08.651+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.654482+0000) 2019-10-13T11:48:08.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:08.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.694299+0000) 2019-10-13T11:48:08.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:08.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.698524+0000) 2019-10-13T11:48:08.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:08.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.710465+0000) 2019-10-13T11:48:09.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:09.923+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:09.929322+0000) 2019-10-13T11:48:11.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:11.757+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.760379+0000) 2019-10-13T11:48:11.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:11.757+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.761505+0000) 2019-10-13T11:48:11.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:11.757+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.762610+0000) 2019-10-13T11:48:11.763 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:11.759+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.762652+0000) 2019-10-13T11:48:11.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:11.761+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.763482+0000) 2019-10-13T11:48:11.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:11.761+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.764112+0000) 2019-10-13T11:48:11.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:11.761+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.764840+0000) 2019-10-13T11:48:11.767 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:11.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:11.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:11.763+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.767411+0000) 2019-10-13T11:48:12.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:12.181+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.185084+0000) 2019-10-13T11:48:15.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:15.489+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:15.494882+0000) 2019-10-13T11:48:16.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:16.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.429691+0000) 2019-10-13T11:48:16.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:16.765+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.769247+0000) 2019-10-13T11:48:16.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:16.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.770350+0000) 2019-10-13T11:48:16.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:16.765+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.770883+0000) 2019-10-13T11:48:16.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:16.769+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.771730+0000) 2019-10-13T11:48:16.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:16.767+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.772018+0000) 2019-10-13T11:48:16.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:16.769+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.773202+0000) 2019-10-13T11:48:16.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:16.769+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.773569+0000) 2019-10-13T11:48:16.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:16.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:16.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:16.775+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:16.778795+0000) 2019-10-13T11:48:18.584 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:48:18.584 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:18.636 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:48:18.636 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:18.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:18.655+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.661268+0000) 2019-10-13T11:48:18.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:18.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.694495+0000) 2019-10-13T11:48:18.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:18.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.698755+0000) 2019-10-13T11:48:18.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:18.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.710657+0000) 2019-10-13T11:48:19.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:19.927+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:19.929549+0000) 2019-10-13T11:48:21.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:21.777+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.779668+0000) 2019-10-13T11:48:21.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:21.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.780048+0000) 2019-10-13T11:48:21.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:21.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.780476+0000) 2019-10-13T11:48:21.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:21.775+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.780811+0000) 2019-10-13T11:48:21.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:21.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.782075+0000) 2019-10-13T11:48:21.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:21.777+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.782546+0000) 2019-10-13T11:48:21.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:21.781+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.783448+0000) 2019-10-13T11:48:21.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:21.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:21.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:21.783+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.787754+0000) 2019-10-13T11:48:22.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:22.181+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.185262+0000) 2019-10-13T11:48:25.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:25.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:25.495052+0000) 2019-10-13T11:48:26.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:26.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.429881+0000) 2019-10-13T11:48:26.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:26.785+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.788321+0000) 2019-10-13T11:48:26.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:26.785+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.788613+0000) 2019-10-13T11:48:26.790 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:26.785+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.789699+0000) 2019-10-13T11:48:26.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:26.785+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.789922+0000) 2019-10-13T11:48:26.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:26.785+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.790555+0000) 2019-10-13T11:48:26.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:26.785+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.790914+0000) 2019-10-13T11:48:26.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:26.787+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.791444+0000) 2019-10-13T11:48:26.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:26.791+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:26.795113+0000) 2019-10-13T11:48:26.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:26.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:28.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:28.667+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.670296+0000) 2019-10-13T11:48:28.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:28.689+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.694740+0000) 2019-10-13T11:48:28.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:28.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.698954+0000) 2019-10-13T11:48:28.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:28.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.710853+0000) 2019-10-13T11:48:29.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:29.927+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:29.929765+0000) 2019-10-13T11:48:31.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:31.797+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.799474+0000) 2019-10-13T11:48:31.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:31.797+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.799741+0000) 2019-10-13T11:48:31.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:31.797+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.800225+0000) 2019-10-13T11:48:31.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:31.797+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.800901+0000) 2019-10-13T11:48:31.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:31.797+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.800454+0000) 2019-10-13T11:48:31.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:31.797+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.801413+0000) 2019-10-13T11:48:31.803 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:31.799+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.802424+0000) 2019-10-13T11:48:31.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:31.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:31.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:31.803+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.808402+0000) 2019-10-13T11:48:32.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:32.185+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.185438+0000) 2019-10-13T11:48:35.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:35.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:35.495346+0000) 2019-10-13T11:48:36.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:36.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.430107+0000) 2019-10-13T11:48:36.810 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:36.805+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.808982+0000) 2019-10-13T11:48:36.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:36.803+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.808876+0000) 2019-10-13T11:48:36.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:36.805+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.809921+0000) 2019-10-13T11:48:36.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:36.805+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.811171+0000) 2019-10-13T11:48:36.812 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:36.809+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.811507+0000) 2019-10-13T11:48:36.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:36.809+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.811825+0000) 2019-10-13T11:48:36.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:36.809+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.812169+0000) 2019-10-13T11:48:36.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:36.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132682 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:36.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:36.811+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:36.816434+0000) 2019-10-13T11:48:38.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:38.670+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.676325+0000) 2019-10-13T11:48:38.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:38.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.694945+0000) 2019-10-13T11:48:38.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:38.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.699188+0000) 2019-10-13T11:48:38.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:38.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.711045+0000) 2019-10-13T11:48:39.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:39.926+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:39.929955+0000) 2019-10-13T11:48:41.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:41.812+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.818792+0000) 2019-10-13T11:48:41.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:41.814+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.819594+0000) 2019-10-13T11:48:41.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:41.816+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.820508+0000) 2019-10-13T11:48:41.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:41.816+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.821045+0000) 2019-10-13T11:48:41.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:41.816+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.821550+0000) 2019-10-13T11:48:41.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:41.816+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.821624+0000) 2019-10-13T11:48:41.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:41.816+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.822030+0000) 2019-10-13T11:48:41.826 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:41.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:41.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:41.826+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.830447+0000) 2019-10-13T11:48:42.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:42.184+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.185619+0000) 2019-10-13T11:48:45.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:45.492+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:45.495546+0000) 2019-10-13T11:48:46.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:46.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.430317+0000) 2019-10-13T11:48:46.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:46.824+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.828803+0000) 2019-10-13T11:48:46.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:46.824+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.829142+0000) 2019-10-13T11:48:46.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:46.826+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.830479+0000) 2019-10-13T11:48:46.832 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:46.824+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.831017+0000) 2019-10-13T11:48:46.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:46.824+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.831188+0000) 2019-10-13T11:48:46.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:46.828+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.831785+0000) 2019-10-13T11:48:46.833 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:46.828+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.832084+0000) 2019-10-13T11:48:46.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:46.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:46.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:46.830+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:46.836530+0000) 2019-10-13T11:48:48.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:48.678+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.682648+0000) 2019-10-13T11:48:48.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:48.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.695170+0000) 2019-10-13T11:48:48.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:48.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.699376+0000) 2019-10-13T11:48:48.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:48.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.711242+0000) 2019-10-13T11:48:48.732 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:48:48.733 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:48.736 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:48:48.736 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:49.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:49.926+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:49.930178+0000) 2019-10-13T11:48:51.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:51.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.839846+0000) 2019-10-13T11:48:51.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:51.836+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.840239+0000) 2019-10-13T11:48:51.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:51.836+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.840580+0000) 2019-10-13T11:48:51.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:51.836+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.841020+0000) 2019-10-13T11:48:51.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:51.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.841492+0000) 2019-10-13T11:48:51.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:51.836+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.842096+0000) 2019-10-13T11:48:51.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:51.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.843163+0000) 2019-10-13T11:48:51.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:51.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:51.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:51.842+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.848649+0000) 2019-10-13T11:48:52.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:52.184+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.185804+0000) 2019-10-13T11:48:55.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:55.492+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:55.495741+0000) 2019-10-13T11:48:56.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:56.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.430516+0000) 2019-10-13T11:48:56.850 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:56.844+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.848849+0000) 2019-10-13T11:48:56.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:48:56.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.850125+0000) 2019-10-13T11:48:56.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:48:56.844+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.850376+0000) 2019-10-13T11:48:56.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:56.846+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.850903+0000) 2019-10-13T11:48:56.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:56.848+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.851686+0000) 2019-10-13T11:48:56.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:56.848+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.852017+0000) 2019-10-13T11:48:56.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:48:56.848+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.852732+0000) 2019-10-13T11:48:56.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:48:56.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:48:56.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:56.854+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:56.858658+0000) 2019-10-13T11:48:58.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:48:58.690+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.694440+0000) 2019-10-13T11:48:58.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:48:58.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.695358+0000) 2019-10-13T11:48:58.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:48:58.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.699568+0000) 2019-10-13T11:48:58.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:48:58.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.711466+0000) 2019-10-13T11:48:59.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:48:59.926+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:59.930372+0000) 2019-10-13T11:49:01.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:01.854+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.858755+0000) 2019-10-13T11:49:01.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:01.856+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.859594+0000) 2019-10-13T11:49:01.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:01.856+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.860199+0000) 2019-10-13T11:49:01.861 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:01.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.860654+0000) 2019-10-13T11:49:01.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:01.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.861612+0000) 2019-10-13T11:49:01.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:01.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.861800+0000) 2019-10-13T11:49:01.863 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:01.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.862683+0000) 2019-10-13T11:49:01.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:01.858+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.865244+0000) 2019-10-13T11:49:01.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:01.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:02.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:02.180+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.185985+0000) 2019-10-13T11:49:05.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:05.492+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:05.495969+0000) 2019-10-13T11:49:06.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:06.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.430666+0000) 2019-10-13T11:49:06.869 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:06.864+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.868840+0000) 2019-10-13T11:49:06.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:06.864+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.869627+0000) 2019-10-13T11:49:06.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:06.864+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.870470+0000) 2019-10-13T11:49:06.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:06.866+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.871246+0000) 2019-10-13T11:49:06.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:06.868+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.871528+0000) 2019-10-13T11:49:06.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:06.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.871862+0000) 2019-10-13T11:49:06.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:06.868+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.872221+0000) 2019-10-13T11:49:06.876 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:06.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:06.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:06.870+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:06.876036+0000) 2019-10-13T11:49:08.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:08.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.695561+0000) 2019-10-13T11:49:08.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:08.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.699792+0000) 2019-10-13T11:49:08.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:08.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.705660+0000) 2019-10-13T11:49:08.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:08.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.711659+0000) 2019-10-13T11:49:09.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:09.930+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.930571+0000) 2019-10-13T11:49:11.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:11.872+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.879060+0000) 2019-10-13T11:49:11.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:11.876+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.879733+0000) 2019-10-13T11:49:11.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:11.876+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.879948+0000) 2019-10-13T11:49:11.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:11.876+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.880436+0000) 2019-10-13T11:49:11.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:11.874+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.880629+0000) 2019-10-13T11:49:11.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:11.876+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.880992+0000) 2019-10-13T11:49:11.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:11.876+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.882261+0000) 2019-10-13T11:49:11.885 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:11.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:11.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:11.882+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.887179+0000) 2019-10-13T11:49:12.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:12.184+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.186164+0000) 2019-10-13T11:49:15.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:15.492+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:15.496216+0000) 2019-10-13T11:49:16.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:16.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.430859+0000) 2019-10-13T11:49:16.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:16.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.887362+0000) 2019-10-13T11:49:16.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:16.884+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.889058+0000) 2019-10-13T11:49:16.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:16.884+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.889301+0000) 2019-10-13T11:49:16.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:16.884+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.889705+0000) 2019-10-13T11:49:16.891 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:16.884+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.890235+0000) 2019-10-13T11:49:16.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:16.884+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.890095+0000) 2019-10-13T11:49:16.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:16.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.890837+0000) 2019-10-13T11:49:16.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:16.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:16.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:16.894+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:16.898993+0000) 2019-10-13T11:49:18.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:18.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.695766+0000) 2019-10-13T11:49:18.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:18.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.699985+0000) 2019-10-13T11:49:18.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:18.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.711891+0000) 2019-10-13T11:49:18.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:18.710+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.712227+0000) 2019-10-13T11:49:18.846 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:49:18.846 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:18.850 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:49:18.850 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:19.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:19.930+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.930768+0000) 2019-10-13T11:49:21.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:21.891+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.898023+0000) 2019-10-13T11:49:21.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:21.891+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.898710+0000) 2019-10-13T11:49:21.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:21.891+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.899184+0000) 2019-10-13T11:49:21.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:21.894+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.899555+0000) 2019-10-13T11:49:21.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:21.895+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.899836+0000) 2019-10-13T11:49:21.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:21.895+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.901099+0000) 2019-10-13T11:49:21.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:21.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.901388+0000) 2019-10-13T11:49:21.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:21.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 132989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:21.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:21.898+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.905122+0000) 2019-10-13T11:49:22.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:22.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.186360+0000) 2019-10-13T11:49:25.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:25.495+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:25.496411+0000) 2019-10-13T11:49:26.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:26.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.431081+0000) 2019-10-13T11:49:26.909 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:26.899+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.906999+0000) 2019-10-13T11:49:26.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:26.903+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.907382+0000) 2019-10-13T11:49:26.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:26.903+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.908640+0000) 2019-10-13T11:49:26.910 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:26.903+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.908823+0000) 2019-10-13T11:49:26.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:26.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.909005+0000) 2019-10-13T11:49:26.911 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:26.903+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.909438+0000) 2019-10-13T11:49:26.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:26.903+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.910561+0000) 2019-10-13T11:49:26.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:26.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:26.917 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:26.914+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:26.916322+0000) 2019-10-13T11:49:28.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:28.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.695957+0000) 2019-10-13T11:49:28.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:28.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.700212+0000) 2019-10-13T11:49:28.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:28.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.712110+0000) 2019-10-13T11:49:28.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:28.718+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.718776+0000) 2019-10-13T11:49:29.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:29.930+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.930996+0000) 2019-10-13T11:49:31.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:31.911+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.918459+0000) 2019-10-13T11:49:31.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:31.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.919177+0000) 2019-10-13T11:49:31.920 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:31.915+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.919282+0000) 2019-10-13T11:49:31.920 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:31.915+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.920135+0000) 2019-10-13T11:49:31.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:31.915+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.920546+0000) 2019-10-13T11:49:31.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:31.917+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.920873+0000) 2019-10-13T11:49:31.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:31.915+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.921773+0000) 2019-10-13T11:49:31.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:31.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:31.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:31.929+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.931451+0000) 2019-10-13T11:49:32.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:32.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.186564+0000) 2019-10-13T11:49:35.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:35.491+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:35.496610+0000) 2019-10-13T11:49:36.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:36.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.431355+0000) 2019-10-13T11:49:36.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:36.923+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.927922+0000) 2019-10-13T11:49:36.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:36.923+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.928188+0000) 2019-10-13T11:49:36.930 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:36.923+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.928581+0000) 2019-10-13T11:49:36.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:36.923+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.929283+0000) 2019-10-13T11:49:36.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:36.925+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.929494+0000) 2019-10-13T11:49:36.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:36.923+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.930343+0000) 2019-10-13T11:49:36.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:36.927+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.931336+0000) 2019-10-13T11:49:36.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:36.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133090 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:36.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:36.933+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:36.935221+0000) 2019-10-13T11:49:38.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:38.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.696117+0000) 2019-10-13T11:49:38.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:38.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.700407+0000) 2019-10-13T11:49:38.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:38.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.712304+0000) 2019-10-13T11:49:38.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:38.729+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.730510+0000) 2019-10-13T11:49:39.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:39.929+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.931193+0000) 2019-10-13T11:49:41.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:41.931+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.937706+0000) 2019-10-13T11:49:41.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:41.931+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.938078+0000) 2019-10-13T11:49:41.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:41.937+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.938821+0000) 2019-10-13T11:49:41.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:41.935+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.939382+0000) 2019-10-13T11:49:41.940 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:41.935+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.939766+0000) 2019-10-13T11:49:41.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:41.935+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.940986+0000) 2019-10-13T11:49:41.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:41.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.941649+0000) 2019-10-13T11:49:41.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:41.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133125 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:41.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:41.945+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.946555+0000) 2019-10-13T11:49:42.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:42.179+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.186743+0000) 2019-10-13T11:49:45.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:45.495+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:45.496833+0000) 2019-10-13T11:49:46.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:46.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.431576+0000) 2019-10-13T11:49:46.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:46.939+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.947250+0000) 2019-10-13T11:49:46.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:46.943+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.947775+0000) 2019-10-13T11:49:46.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:46.943+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.948816+0000) 2019-10-13T11:49:46.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:46.943+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.949092+0000) 2019-10-13T11:49:46.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:46.945+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.949298+0000) 2019-10-13T11:49:46.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:46.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.950192+0000) 2019-10-13T11:49:46.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:46.943+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.950849+0000) 2019-10-13T11:49:46.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:46.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:46.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:46.953+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:46.955139+0000) 2019-10-13T11:49:48.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:48.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.696314+0000) 2019-10-13T11:49:48.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:48.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.700607+0000) 2019-10-13T11:49:48.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:48.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.712507+0000) 2019-10-13T11:49:48.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:48.733+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.737118+0000) 2019-10-13T11:49:49.094 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:49:49.094 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:49.098 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:49:49.098 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:49.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:49.929+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.931426+0000) 2019-10-13T11:49:51.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:51.955+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.958065+0000) 2019-10-13T11:49:51.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:51.959+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.959299+0000) 2019-10-13T11:49:51.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:51.957+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.959500+0000) 2019-10-13T11:49:51.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:51.959+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.960257+0000) 2019-10-13T11:49:51.962 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:51.959+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.961087+0000) 2019-10-13T11:49:51.962 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:51.959+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.961111+0000) 2019-10-13T11:49:51.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:51.959+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.961823+0000) 2019-10-13T11:49:51.966 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:51.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:51.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:51.965+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.966758+0000) 2019-10-13T11:49:52.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:52.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:52.186945+0000) 2019-10-13T11:49:55.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:55.495+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:55.497032+0000) 2019-10-13T11:49:56.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:56.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.431767+0000) 2019-10-13T11:49:56.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:56.967+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.968313+0000) 2019-10-13T11:49:56.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:49:56.967+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.968691+0000) 2019-10-13T11:49:56.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:49:56.967+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.969000+0000) 2019-10-13T11:49:56.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:56.965+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.969549+0000) 2019-10-13T11:49:56.971 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:49:56.967+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.969487+0000) 2019-10-13T11:49:56.971 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:56.967+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.970345+0000) 2019-10-13T11:49:56.972 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:56.971+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.971499+0000) 2019-10-13T11:49:56.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:49:56.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:49:56.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:56.973+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:56.976988+0000) 2019-10-13T11:49:58.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:49:58.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.696514+0000) 2019-10-13T11:49:58.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:49:58.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.700831+0000) 2019-10-13T11:49:58.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:49:58.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.712702+0000) 2019-10-13T11:49:58.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:49:58.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.748773+0000) 2019-10-13T11:49:59.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:49:59.929+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.931630+0000) 2019-10-13T11:50:01.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:01.975+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.979199+0000) 2019-10-13T11:50:01.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:01.977+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.979497+0000) 2019-10-13T11:50:01.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:01.979+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.980596+0000) 2019-10-13T11:50:01.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:01.979+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.981766+0000) 2019-10-13T11:50:01.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:01.979+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.982419+0000) 2019-10-13T11:50:01.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:01.979+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.982783+0000) 2019-10-13T11:50:01.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:01.983+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.983631+0000) 2019-10-13T11:50:01.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:01.985+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.987532+0000) 2019-10-13T11:50:01.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:01.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133261 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:02.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:02.187+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:02.187124+0000) 2019-10-13T11:50:05.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:05.494+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:05.497242+0000) 2019-10-13T11:50:06.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:06.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.431963+0000) 2019-10-13T11:50:06.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:06.985+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.989683+0000) 2019-10-13T11:50:06.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:06.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.991393+0000) 2019-10-13T11:50:06.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:06.990+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.991738+0000) 2019-10-13T11:50:06.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:06.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.992254+0000) 2019-10-13T11:50:06.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:06.990+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.992791+0000) 2019-10-13T11:50:06.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:06.990+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.992698+0000) 2019-10-13T11:50:06.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:06.990+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.993210+0000) 2019-10-13T11:50:06.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:06.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:07.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:06.997+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:06.999720+0000) 2019-10-13T11:50:08.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:08.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.696712+0000) 2019-10-13T11:50:08.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:08.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.701021+0000) 2019-10-13T11:50:08.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:08.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.712927+0000) 2019-10-13T11:50:08.756 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:08.753+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.755314+0000) 2019-10-13T11:50:09.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:09.929+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.931823+0000) 2019-10-13T11:50:12.000 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:11.998+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:11.999456+0000) 2019-10-13T11:50:12.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:11.997+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:11.999877+0000) 2019-10-13T11:50:12.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:11.998+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.000493+0000) 2019-10-13T11:50:12.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:11.998+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.000827+0000) 2019-10-13T11:50:12.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:11.998+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.001506+0000) 2019-10-13T11:50:12.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:11.998+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.002448+0000) 2019-10-13T11:50:12.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:11.998+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.002929+0000) 2019-10-13T11:50:12.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:12.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:12.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:12.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.008252+0000) 2019-10-13T11:50:12.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:12.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:12.187336+0000) 2019-10-13T11:50:15.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:15.494+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:15.497459+0000) 2019-10-13T11:50:16.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:16.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:16.432174+0000) 2019-10-13T11:50:17.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:17.006+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.009932+0000) 2019-10-13T11:50:17.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:17.009+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.010707+0000) 2019-10-13T11:50:17.012 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:17.006+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.010582+0000) 2019-10-13T11:50:17.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:17.010+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.011339+0000) 2019-10-13T11:50:17.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:17.010+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.011977+0000) 2019-10-13T11:50:17.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:17.010+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.012895+0000) 2019-10-13T11:50:17.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:17.010+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.013458+0000) 2019-10-13T11:50:17.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:17.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:17.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:17.017+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.018781+0000) 2019-10-13T11:50:18.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:18.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.696905+0000) 2019-10-13T11:50:18.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:18.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.701225+0000) 2019-10-13T11:50:18.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:18.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.713141+0000) 2019-10-13T11:50:18.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:18.761+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.761824+0000) 2019-10-13T11:50:19.217 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:50:19.217 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:19.221 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:50:19.221 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:19.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:19.929+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.932050+0000) 2019-10-13T11:50:22.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:22.018+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.020852+0000) 2019-10-13T11:50:22.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:22.018+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.021263+0000) 2019-10-13T11:50:22.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:22.018+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.021810+0000) 2019-10-13T11:50:22.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:22.018+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.022024+0000) 2019-10-13T11:50:22.023 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:22.018+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.022320+0000) 2019-10-13T11:50:22.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:22.021+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.023460+0000) 2019-10-13T11:50:22.025 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:22.022+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.024525+0000) 2019-10-13T11:50:22.027 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:22.022+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133397 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:22.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:22.029+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.030666+0000) 2019-10-13T11:50:22.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:22.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:22.187513+0000) 2019-10-13T11:50:25.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:25.494+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:25.497672+0000) 2019-10-13T11:50:26.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:26.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:26.432366+0000) 2019-10-13T11:50:27.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:27.026+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.029755+0000) 2019-10-13T11:50:27.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:27.026+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.030490+0000) 2019-10-13T11:50:27.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:27.026+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.030909+0000) 2019-10-13T11:50:27.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:27.030+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.031368+0000) 2019-10-13T11:50:27.033 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:27.030+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.032463+0000) 2019-10-13T11:50:27.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:27.028+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.032804+0000) 2019-10-13T11:50:27.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:27.030+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.033601+0000) 2019-10-13T11:50:27.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:27.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:27.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:27.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.038775+0000) 2019-10-13T11:50:28.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:28.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.697116+0000) 2019-10-13T11:50:28.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:28.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.701449+0000) 2019-10-13T11:50:28.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:28.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.713368+0000) 2019-10-13T11:50:28.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:28.768+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.773392+0000) 2019-10-13T11:50:29.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:29.928+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.932301+0000) 2019-10-13T11:50:32.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:32.038+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.039524+0000) 2019-10-13T11:50:32.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:32.038+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.040471+0000) 2019-10-13T11:50:32.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:32.038+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.041752+0000) 2019-10-13T11:50:32.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:32.038+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.042476+0000) 2019-10-13T11:50:32.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:32.038+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.042217+0000) 2019-10-13T11:50:32.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:32.040+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.043156+0000) 2019-10-13T11:50:32.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:32.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.043178+0000) 2019-10-13T11:50:32.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:32.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:32.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:32.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.048731+0000) 2019-10-13T11:50:32.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:32.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:32.187702+0000) 2019-10-13T11:50:35.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:35.494+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:35.497903+0000) 2019-10-13T11:50:36.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:36.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:36.432555+0000) 2019-10-13T11:50:37.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:37.046+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.049779+0000) 2019-10-13T11:50:37.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:37.048+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.051221+0000) 2019-10-13T11:50:37.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:37.046+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.051051+0000) 2019-10-13T11:50:37.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:37.050+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.052229+0000) 2019-10-13T11:50:37.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:37.050+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.052656+0000) 2019-10-13T11:50:37.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:37.050+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.053232+0000) 2019-10-13T11:50:37.054 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:37.050+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.053767+0000) 2019-10-13T11:50:37.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:37.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:37.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:37.056+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.058994+0000) 2019-10-13T11:50:38.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:38.694+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.697317+0000) 2019-10-13T11:50:38.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:38.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.701650+0000) 2019-10-13T11:50:38.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:38.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.713569+0000) 2019-10-13T11:50:38.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:38.776+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.780024+0000) 2019-10-13T11:50:39.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:39.928+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.932508+0000) 2019-10-13T11:50:42.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:42.056+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.060410+0000) 2019-10-13T11:50:42.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:42.058+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.061118+0000) 2019-10-13T11:50:42.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:42.058+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.061866+0000) 2019-10-13T11:50:42.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:42.058+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.061913+0000) 2019-10-13T11:50:42.063 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:42.058+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.062442+0000) 2019-10-13T11:50:42.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:42.058+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.062851+0000) 2019-10-13T11:50:42.064 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:42.062+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.063992+0000) 2019-10-13T11:50:42.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:42.062+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133533 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:42.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:42.064+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.068740+0000) 2019-10-13T11:50:42.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:42.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:42.187888+0000) 2019-10-13T11:50:45.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:45.498+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:45.498138+0000) 2019-10-13T11:50:46.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:46.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:46.432765+0000) 2019-10-13T11:50:47.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:47.066+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.069726+0000) 2019-10-13T11:50:47.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:47.066+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.070144+0000) 2019-10-13T11:50:47.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:47.066+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.070847+0000) 2019-10-13T11:50:47.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:47.070+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.071461+0000) 2019-10-13T11:50:47.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:47.070+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.071885+0000) 2019-10-13T11:50:47.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:47.068+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.072071+0000) 2019-10-13T11:50:47.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:47.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.072891+0000) 2019-10-13T11:50:47.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:47.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:47.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:47.076+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.079515+0000) 2019-10-13T11:50:48.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:48.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.697523+0000) 2019-10-13T11:50:48.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:48.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.701842+0000) 2019-10-13T11:50:48.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:48.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.713823+0000) 2019-10-13T11:50:48.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:48.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.786370+0000) 2019-10-13T11:50:49.366 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:50:49.366 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:49.370 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:50:49.370 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:49.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:49.928+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.932731+0000) 2019-10-13T11:50:52.080 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:52.077+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.079552+0000) 2019-10-13T11:50:52.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:52.076+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.080240+0000) 2019-10-13T11:50:52.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:52.077+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.079852+0000) 2019-10-13T11:50:52.082 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:52.077+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.081094+0000) 2019-10-13T11:50:52.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:52.077+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.081457+0000) 2019-10-13T11:50:52.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:52.077+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.081855+0000) 2019-10-13T11:50:52.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:52.077+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.082431+0000) 2019-10-13T11:50:52.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:52.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:52.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:52.084+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.088274+0000) 2019-10-13T11:50:52.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:52.185+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:52.188080+0000) 2019-10-13T11:50:55.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:55.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:55.498356+0000) 2019-10-13T11:50:56.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:56.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:56.432965+0000) 2019-10-13T11:50:57.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:57.085+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.089122+0000) 2019-10-13T11:50:57.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:57.085+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.089549+0000) 2019-10-13T11:50:57.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:57.088+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.090427+0000) 2019-10-13T11:50:57.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:50:57.085+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.091134+0000) 2019-10-13T11:50:57.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:50:57.089+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.091393+0000) 2019-10-13T11:50:57.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:57.089+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.091689+0000) 2019-10-13T11:50:57.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:50:57.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.092830+0000) 2019-10-13T11:50:57.096 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:50:57.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:50:57.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:57.092+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.096542+0000) 2019-10-13T11:50:58.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:50:58.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.697726+0000) 2019-10-13T11:50:58.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:50:58.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.702080+0000) 2019-10-13T11:50:58.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:50:58.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.714020+0000) 2019-10-13T11:50:58.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:50:58.788+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.792780+0000) 2019-10-13T11:50:59.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:50:59.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.932961+0000) 2019-10-13T11:51:02.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:02.096+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.098533+0000) 2019-10-13T11:51:02.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:02.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.099091+0000) 2019-10-13T11:51:02.100 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:02.097+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.099777+0000) 2019-10-13T11:51:02.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:02.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.100214+0000) 2019-10-13T11:51:02.101 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:02.097+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.100690+0000) 2019-10-13T11:51:02.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:02.101+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.103392+0000) 2019-10-13T11:51:02.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:02.101+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.103493+0000) 2019-10-13T11:51:02.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:02.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:02.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:02.104+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.109395+0000) 2019-10-13T11:51:02.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:02.185+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:02.188298+0000) 2019-10-13T11:51:05.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:05.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:05.498555+0000) 2019-10-13T11:51:06.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:06.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:06.433171+0000) 2019-10-13T11:51:07.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:07.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.108733+0000) 2019-10-13T11:51:07.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:07.105+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.109508+0000) 2019-10-13T11:51:07.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:07.105+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.109950+0000) 2019-10-13T11:51:07.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:07.105+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.110391+0000) 2019-10-13T11:51:07.112 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:07.105+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.111109+0000) 2019-10-13T11:51:07.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:07.109+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.112407+0000) 2019-10-13T11:51:07.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:07.109+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.112590+0000) 2019-10-13T11:51:07.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:07.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:07.119 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:07.116+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.118434+0000) 2019-10-13T11:51:08.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:08.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.697931+0000) 2019-10-13T11:51:08.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:08.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.702311+0000) 2019-10-13T11:51:08.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:08.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.714214+0000) 2019-10-13T11:51:08.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:08.796+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.799354+0000) 2019-10-13T11:51:09.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:09.928+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.933137+0000) 2019-10-13T11:51:12.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:12.117+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.119494+0000) 2019-10-13T11:51:12.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:12.117+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.119926+0000) 2019-10-13T11:51:12.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:12.117+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.120354+0000) 2019-10-13T11:51:12.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:12.116+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.120817+0000) 2019-10-13T11:51:12.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:12.117+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.121175+0000) 2019-10-13T11:51:12.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:12.117+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.121770+0000) 2019-10-13T11:51:12.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:12.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.122956+0000) 2019-10-13T11:51:12.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:12.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:12.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:12.124+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.127225+0000) 2019-10-13T11:51:12.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:12.185+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:12.188530+0000) 2019-10-13T11:51:15.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:15.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:15.498783+0000) 2019-10-13T11:51:16.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:16.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:16.433385+0000) 2019-10-13T11:51:17.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:17.125+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.128694+0000) 2019-10-13T11:51:17.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:17.125+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.129249+0000) 2019-10-13T11:51:17.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:17.125+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.129662+0000) 2019-10-13T11:51:17.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:17.125+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.131192+0000) 2019-10-13T11:51:17.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:17.129+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.131673+0000) 2019-10-13T11:51:17.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:17.128+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.132303+0000) 2019-10-13T11:51:17.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:17.129+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.132913+0000) 2019-10-13T11:51:17.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:17.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:17.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:17.136+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.138465+0000) 2019-10-13T11:51:18.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:18.693+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.698135+0000) 2019-10-13T11:51:18.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:18.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.702533+0000) 2019-10-13T11:51:18.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:18.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.714438+0000) 2019-10-13T11:51:18.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:18.808+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.811075+0000) 2019-10-13T11:51:19.501 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:51:19.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:19.505 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:51:19.505 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:19.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:19.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.933331+0000) 2019-10-13T11:51:22.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:22.137+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.140207+0000) 2019-10-13T11:51:22.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:22.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.140506+0000) 2019-10-13T11:51:22.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:22.137+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.140522+0000) 2019-10-13T11:51:22.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:22.137+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.141011+0000) 2019-10-13T11:51:22.143 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:22.137+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.141282+0000) 2019-10-13T11:51:22.143 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:22.137+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.141719+0000) 2019-10-13T11:51:22.143 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:22.137+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.142078+0000) 2019-10-13T11:51:22.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:22.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:22.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:22.147+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.150156+0000) 2019-10-13T11:51:22.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:22.185+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:22.188702+0000) 2019-10-13T11:51:25.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:25.493+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:25.499003+0000) 2019-10-13T11:51:26.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:26.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:26.433574+0000) 2019-10-13T11:51:27.151 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:27.145+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.150428+0000) 2019-10-13T11:51:27.152 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:27.145+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.150950+0000) 2019-10-13T11:51:27.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:27.149+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.151627+0000) 2019-10-13T11:51:27.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:27.147+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.151812+0000) 2019-10-13T11:51:27.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:27.149+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.152513+0000) 2019-10-13T11:51:27.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:27.149+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.153063+0000) 2019-10-13T11:51:27.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:27.149+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.153480+0000) 2019-10-13T11:51:27.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:27.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:27.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:27.155+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.160388+0000) 2019-10-13T11:51:28.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:28.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.698329+0000) 2019-10-13T11:51:28.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:28.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.702757+0000) 2019-10-13T11:51:28.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:28.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.714639+0000) 2019-10-13T11:51:28.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:28.811+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.817737+0000) 2019-10-13T11:51:29.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:29.931+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.933533+0000) 2019-10-13T11:51:32.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:32.155+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.161050+0000) 2019-10-13T11:51:32.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:32.156+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.161809+0000) 2019-10-13T11:51:32.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:32.156+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.162156+0000) 2019-10-13T11:51:32.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:32.156+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.162970+0000) 2019-10-13T11:51:32.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:32.156+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.163146+0000) 2019-10-13T11:51:32.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:32.160+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.163667+0000) 2019-10-13T11:51:32.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:32.160+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.164790+0000) 2019-10-13T11:51:32.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:32.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:32.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:32.167+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.172453+0000) 2019-10-13T11:51:32.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:32.184+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:32.188859+0000) 2019-10-13T11:51:35.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:35.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:35.499218+0000) 2019-10-13T11:51:36.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:36.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:36.433806+0000) 2019-10-13T11:51:37.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:37.167+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.172642+0000) 2019-10-13T11:51:37.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:37.168+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.173106+0000) 2019-10-13T11:51:37.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:37.168+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.173747+0000) 2019-10-13T11:51:37.175 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:37.168+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.174341+0000) 2019-10-13T11:51:37.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:37.168+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.174556+0000) 2019-10-13T11:51:37.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:37.168+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.174902+0000) 2019-10-13T11:51:37.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:37.172+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.175421+0000) 2019-10-13T11:51:37.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:37.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133905 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:37.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:37.179+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.182444+0000) 2019-10-13T11:51:38.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:38.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.698530+0000) 2019-10-13T11:51:38.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:38.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.702950+0000) 2019-10-13T11:51:38.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:38.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.714834+0000) 2019-10-13T11:51:38.825 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:38.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.824293+0000) 2019-10-13T11:51:39.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:39.927+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.933758+0000) 2019-10-13T11:51:42.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:42.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.182409+0000) 2019-10-13T11:51:42.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:42.176+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.183067+0000) 2019-10-13T11:51:42.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:42.180+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.183647+0000) 2019-10-13T11:51:42.184 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:42.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.183757+0000) 2019-10-13T11:51:42.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:42.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.184263+0000) 2019-10-13T11:51:42.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:42.180+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.184533+0000) 2019-10-13T11:51:42.186 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:42.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.184717+0000) 2019-10-13T11:51:42.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:42.184+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.189028+0000) 2019-10-13T11:51:42.190 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:42.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:42.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:42.187+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:42.191266+0000) 2019-10-13T11:51:45.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:45.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:45.499406+0000) 2019-10-13T11:51:46.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:46.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:46.434003+0000) 2019-10-13T11:51:47.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:47.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.192683+0000) 2019-10-13T11:51:47.194 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:47.188+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.193479+0000) 2019-10-13T11:51:47.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:47.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.194396+0000) 2019-10-13T11:51:47.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:47.188+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.194756+0000) 2019-10-13T11:51:47.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:47.188+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.195056+0000) 2019-10-13T11:51:47.196 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:47.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.195603+0000) 2019-10-13T11:51:47.197 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:47.191+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.195972+0000) 2019-10-13T11:51:47.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:47.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 133972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:47.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:47.195+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.201871+0000) 2019-10-13T11:51:48.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:48.692+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.698763+0000) 2019-10-13T11:51:48.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:48.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.703165+0000) 2019-10-13T11:51:48.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:48.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.715029+0000) 2019-10-13T11:51:48.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:48.827+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.830877+0000) 2019-10-13T11:51:49.620 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:51:49.621 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:49.624 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:51:49.624 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:49.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:49.931+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.933955+0000) 2019-10-13T11:51:52.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:52.188+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.189238+0000) 2019-10-13T11:51:52.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:52.200+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.203561+0000) 2019-10-13T11:51:52.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:52.200+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.204035+0000) 2019-10-13T11:51:52.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:52.200+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.204838+0000) 2019-10-13T11:51:52.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:52.200+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.204953+0000) 2019-10-13T11:51:52.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:52.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.205437+0000) 2019-10-13T11:51:52.206 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:52.200+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.205620+0000) 2019-10-13T11:51:52.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:52.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.207321+0000) 2019-10-13T11:51:52.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:52.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:52.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:52.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:52.215921+0000) 2019-10-13T11:51:55.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:55.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:55.499636+0000) 2019-10-13T11:51:56.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:56.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:56.434201+0000) 2019-10-13T11:51:57.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:57.208+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.213024+0000) 2019-10-13T11:51:57.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:51:57.208+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.214544+0000) 2019-10-13T11:51:57.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:57.208+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.214909+0000) 2019-10-13T11:51:57.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:57.211+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.215015+0000) 2019-10-13T11:51:57.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:51:57.212+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.215485+0000) 2019-10-13T11:51:57.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:51:57.212+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.216044+0000) 2019-10-13T11:51:57.218 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:57.212+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.216821+0000) 2019-10-13T11:51:57.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:57.215+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.219211+0000) 2019-10-13T11:51:57.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:51:57.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:51:58.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:51:58.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.698964+0000) 2019-10-13T11:51:58.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:51:58.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.703368+0000) 2019-10-13T11:51:58.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:51:58.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.715252+0000) 2019-10-13T11:51:58.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:51:58.835+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.837494+0000) 2019-10-13T11:51:59.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:51:59.931+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.934181+0000) 2019-10-13T11:52:02.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:02.188+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.189504+0000) 2019-10-13T11:52:02.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:02.219+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.223807+0000) 2019-10-13T11:52:02.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:02.220+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.224264+0000) 2019-10-13T11:52:02.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:02.220+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.224788+0000) 2019-10-13T11:52:02.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:02.220+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.225123+0000) 2019-10-13T11:52:02.227 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:02.220+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.225528+0000) 2019-10-13T11:52:02.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:02.220+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.226228+0000) 2019-10-13T11:52:02.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:02.220+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.226481+0000) 2019-10-13T11:52:02.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:02.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:02.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:02.231+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:02.235652+0000) 2019-10-13T11:52:05.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:05.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:05.499867+0000) 2019-10-13T11:52:06.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:06.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:06.434418+0000) 2019-10-13T11:52:07.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:07.228+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.232908+0000) 2019-10-13T11:52:07.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:07.228+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.233922+0000) 2019-10-13T11:52:07.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:07.228+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.235045+0000) 2019-10-13T11:52:07.236 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:07.231+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.235260+0000) 2019-10-13T11:52:07.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:07.232+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.235918+0000) 2019-10-13T11:52:07.237 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:07.232+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.236585+0000) 2019-10-13T11:52:07.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:07.232+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.236901+0000) 2019-10-13T11:52:07.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:07.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134108 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:07.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:07.239+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.242497+0000) 2019-10-13T11:52:08.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:08.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.699180+0000) 2019-10-13T11:52:08.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:08.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.703567+0000) 2019-10-13T11:52:08.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:08.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.715492+0000) 2019-10-13T11:52:08.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:08.843+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.844114+0000) 2019-10-13T11:52:09.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:09.931+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.934393+0000) 2019-10-13T11:52:12.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:12.188+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.189738+0000) 2019-10-13T11:52:12.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:12.240+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.244197+0000) 2019-10-13T11:52:12.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:12.240+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.244415+0000) 2019-10-13T11:52:12.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:12.239+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.245176+0000) 2019-10-13T11:52:12.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:12.240+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.245051+0000) 2019-10-13T11:52:12.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:12.240+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.245717+0000) 2019-10-13T11:52:12.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:12.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.246722+0000) 2019-10-13T11:52:12.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:12.244+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.247865+0000) 2019-10-13T11:52:12.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:12.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:12.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:12.247+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:12.253502+0000) 2019-10-13T11:52:15.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:15.499+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:15.500088+0000) 2019-10-13T11:52:16.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:16.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:16.434627+0000) 2019-10-13T11:52:17.254 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:17.246+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.253703+0000) 2019-10-13T11:52:17.255 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:17.247+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.254485+0000) 2019-10-13T11:52:17.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:17.247+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.255274+0000) 2019-10-13T11:52:17.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:17.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.256047+0000) 2019-10-13T11:52:17.257 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:17.251+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.256369+0000) 2019-10-13T11:52:17.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:17.251+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.257021+0000) 2019-10-13T11:52:17.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:17.251+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.257544+0000) 2019-10-13T11:52:17.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:17.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.260654+0000) 2019-10-13T11:52:17.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:17.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:18.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:18.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.699378+0000) 2019-10-13T11:52:18.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:18.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.703793+0000) 2019-10-13T11:52:18.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:18.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.715724+0000) 2019-10-13T11:52:18.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:18.850+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.855243+0000) 2019-10-13T11:52:19.891 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:52:19.891 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:19.894 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:52:19.895 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:19.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:19.930+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.934640+0000) 2019-10-13T11:52:22.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:22.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.189966+0000) 2019-10-13T11:52:22.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:22.259+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.264327+0000) 2019-10-13T11:52:22.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:22.259+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.264805+0000) 2019-10-13T11:52:22.266 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:22.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.265489+0000) 2019-10-13T11:52:22.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:22.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.266841+0000) 2019-10-13T11:52:22.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:22.263+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.267597+0000) 2019-10-13T11:52:22.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:22.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.268046+0000) 2019-10-13T11:52:22.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:22.263+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.268410+0000) 2019-10-13T11:52:22.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:22.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:22.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:22.266+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:22.273477+0000) 2019-10-13T11:52:25.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:25.499+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:25.500303+0000) 2019-10-13T11:52:26.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:26.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:26.434820+0000) 2019-10-13T11:52:27.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:27.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.276554+0000) 2019-10-13T11:52:27.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:27.271+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.274436+0000) 2019-10-13T11:52:27.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:27.271+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.275060+0000) 2019-10-13T11:52:27.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:27.271+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.276303+0000) 2019-10-13T11:52:27.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:27.271+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.277305+0000) 2019-10-13T11:52:27.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:27.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.278029+0000) 2019-10-13T11:52:27.283 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:27.271+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.278372+0000) 2019-10-13T11:52:27.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:27.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:27.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:27.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.283222+0000) 2019-10-13T11:52:28.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:28.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.699608+0000) 2019-10-13T11:52:28.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:28.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.704022+0000) 2019-10-13T11:52:28.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:28.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.715921+0000) 2019-10-13T11:52:28.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:28.858+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.861828+0000) 2019-10-13T11:52:29.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:29.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.934846+0000) 2019-10-13T11:52:32.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:32.183+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.190183+0000) 2019-10-13T11:52:32.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:32.279+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.285409+0000) 2019-10-13T11:52:32.287 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:32.279+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.285947+0000) 2019-10-13T11:52:32.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:32.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.286124+0000) 2019-10-13T11:52:32.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:32.283+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.287761+0000) 2019-10-13T11:52:32.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:32.283+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.288151+0000) 2019-10-13T11:52:32.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:32.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.288567+0000) 2019-10-13T11:52:32.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:32.283+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.289089+0000) 2019-10-13T11:52:32.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:32.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:32.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:32.286+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:32.293059+0000) 2019-10-13T11:52:35.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:35.499+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:35.500518+0000) 2019-10-13T11:52:36.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:36.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:36.435012+0000) 2019-10-13T11:52:37.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:37.291+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.294748+0000) 2019-10-13T11:52:37.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:37.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.295594+0000) 2019-10-13T11:52:37.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:37.291+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.296756+0000) 2019-10-13T11:52:37.298 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:37.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.297142+0000) 2019-10-13T11:52:37.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:37.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.297800+0000) 2019-10-13T11:52:37.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:37.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.298313+0000) 2019-10-13T11:52:37.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:37.291+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.299125+0000) 2019-10-13T11:52:37.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:37.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:37.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:37.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.302644+0000) 2019-10-13T11:52:38.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:38.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.699807+0000) 2019-10-13T11:52:38.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:38.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.704171+0000) 2019-10-13T11:52:38.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:38.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.716136+0000) 2019-10-13T11:52:38.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:38.866+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.868216+0000) 2019-10-13T11:52:39.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:39.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.935041+0000) 2019-10-13T11:52:42.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:42.187+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.190402+0000) 2019-10-13T11:52:42.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:42.302+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.305055+0000) 2019-10-13T11:52:42.306 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:42.299+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.305894+0000) 2019-10-13T11:52:42.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:42.299+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.306352+0000) 2019-10-13T11:52:42.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:42.299+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.306958+0000) 2019-10-13T11:52:42.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:42.303+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.308087+0000) 2019-10-13T11:52:42.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:42.303+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.308387+0000) 2019-10-13T11:52:42.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:42.303+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.308789+0000) 2019-10-13T11:52:42.313 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:42.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:42.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:42.306+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:42.313792+0000) 2019-10-13T11:52:45.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:45.499+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:45.500717+0000) 2019-10-13T11:52:46.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:46.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:46.435216+0000) 2019-10-13T11:52:47.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:47.310+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.315281+0000) 2019-10-13T11:52:47.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:47.315+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.315850+0000) 2019-10-13T11:52:47.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:47.315+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.317069+0000) 2019-10-13T11:52:47.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:47.315+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.317651+0000) 2019-10-13T11:52:47.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:47.315+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.317798+0000) 2019-10-13T11:52:47.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:47.315+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.318533+0000) 2019-10-13T11:52:47.320 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:47.319+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.319414+0000) 2019-10-13T11:52:47.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:47.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:47.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:47.318+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.324414+0000) 2019-10-13T11:52:48.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:48.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.700005+0000) 2019-10-13T11:52:48.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:48.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.704372+0000) 2019-10-13T11:52:48.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:48.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.716345+0000) 2019-10-13T11:52:48.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:48.874+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.874636+0000) 2019-10-13T11:52:49.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:49.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.935230+0000) 2019-10-13T11:52:50.010 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:52:50.010 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:50.013 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:52:50.014 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:52.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:52.187+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.190637+0000) 2019-10-13T11:52:52.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:52.323+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.325679+0000) 2019-10-13T11:52:52.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:52.323+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.326018+0000) 2019-10-13T11:52:52.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:52.322+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.326305+0000) 2019-10-13T11:52:52.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:52.323+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.326809+0000) 2019-10-13T11:52:52.329 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:52.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.327472+0000) 2019-10-13T11:52:52.330 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:52.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.328280+0000) 2019-10-13T11:52:52.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:52.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.328955+0000) 2019-10-13T11:52:52.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:52.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134415 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:52.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:52.330+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:52.335889+0000) 2019-10-13T11:52:55.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:55.499+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:55.500946+0000) 2019-10-13T11:52:56.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:56.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:56.435410+0000) 2019-10-13T11:52:57.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:57.334+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.335591+0000) 2019-10-13T11:52:57.337 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:57.334+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.336437+0000) 2019-10-13T11:52:57.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:52:57.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.337072+0000) 2019-10-13T11:52:57.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:57.330+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.337416+0000) 2019-10-13T11:52:57.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:52:57.334+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.338022+0000) 2019-10-13T11:52:57.339 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:57.334+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.338153+0000) 2019-10-13T11:52:57.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:52:57.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.338875+0000) 2019-10-13T11:52:57.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:52:57.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:52:57.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:57.338+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.345406+0000) 2019-10-13T11:52:58.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:52:58.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.700246+0000) 2019-10-13T11:52:58.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:52:58.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.704584+0000) 2019-10-13T11:52:58.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:52:58.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.716557+0000) 2019-10-13T11:52:58.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:52:58.878+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.880999+0000) 2019-10-13T11:52:59.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:52:59.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.935447+0000) 2019-10-13T11:53:02.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:02.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.190853+0000) 2019-10-13T11:53:02.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:02.342+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.345883+0000) 2019-10-13T11:53:02.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:02.342+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.347386+0000) 2019-10-13T11:53:02.349 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:02.346+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.347888+0000) 2019-10-13T11:53:02.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:02.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.348624+0000) 2019-10-13T11:53:02.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:02.346+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.348342+0000) 2019-10-13T11:53:02.350 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:02.346+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.349273+0000) 2019-10-13T11:53:02.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:02.346+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.349808+0000) 2019-10-13T11:53:02.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:02.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:02.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:02.346+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:02.353525+0000) 2019-10-13T11:53:05.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:05.498+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:05.501144+0000) 2019-10-13T11:53:06.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:06.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:06.435604+0000) 2019-10-13T11:53:07.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:07.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.356331+0000) 2019-10-13T11:53:07.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:07.354+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.357113+0000) 2019-10-13T11:53:07.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:07.354+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.357871+0000) 2019-10-13T11:53:07.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:07.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.357812+0000) 2019-10-13T11:53:07.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:07.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.358201+0000) 2019-10-13T11:53:07.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:07.358+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.358542+0000) 2019-10-13T11:53:07.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:07.358+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.359581+0000) 2019-10-13T11:53:07.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:07.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:07.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:07.366+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.366509+0000) 2019-10-13T11:53:08.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:08.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.700444+0000) 2019-10-13T11:53:08.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:08.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.704776+0000) 2019-10-13T11:53:08.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:08.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.716784+0000) 2019-10-13T11:53:08.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:08.886+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.887577+0000) 2019-10-13T11:53:09.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:09.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.935657+0000) 2019-10-13T11:53:12.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:12.186+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.191046+0000) 2019-10-13T11:53:12.368 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:12.362+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.366966+0000) 2019-10-13T11:53:12.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:12.365+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.367726+0000) 2019-10-13T11:53:12.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:12.366+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.367774+0000) 2019-10-13T11:53:12.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:12.366+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.368711+0000) 2019-10-13T11:53:12.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:12.366+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.368846+0000) 2019-10-13T11:53:12.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:12.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.369189+0000) 2019-10-13T11:53:12.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:12.366+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.369471+0000) 2019-10-13T11:53:12.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:12.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:12.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:12.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:12.376611+0000) 2019-10-13T11:53:15.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:15.498+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:15.501363+0000) 2019-10-13T11:53:16.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:16.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:16.435820+0000) 2019-10-13T11:53:17.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:17.374+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.376075+0000) 2019-10-13T11:53:17.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:17.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.376888+0000) 2019-10-13T11:53:17.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:17.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.377105+0000) 2019-10-13T11:53:17.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:17.374+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.377464+0000) 2019-10-13T11:53:17.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:17.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.377711+0000) 2019-10-13T11:53:17.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:17.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.378325+0000) 2019-10-13T11:53:17.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:17.377+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.379167+0000) 2019-10-13T11:53:17.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:17.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:17.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:17.381+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.385752+0000) 2019-10-13T11:53:18.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:18.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.700646+0000) 2019-10-13T11:53:18.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:18.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.704974+0000) 2019-10-13T11:53:18.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:18.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.717038+0000) 2019-10-13T11:53:18.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:18.893+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.894025+0000) 2019-10-13T11:53:19.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:19.933+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.935867+0000) 2019-10-13T11:53:20.147 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:53:20.147 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:20.151 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:53:20.151 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:22.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:22.190+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.191253+0000) 2019-10-13T11:53:22.387 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:22.382+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.385880+0000) 2019-10-13T11:53:22.387 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:22.382+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.386476+0000) 2019-10-13T11:53:22.388 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:22.385+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.386698+0000) 2019-10-13T11:53:22.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:22.386+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.387848+0000) 2019-10-13T11:53:22.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:22.386+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.387936+0000) 2019-10-13T11:53:22.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:22.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.388200+0000) 2019-10-13T11:53:22.389 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:22.386+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.388916+0000) 2019-10-13T11:53:22.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:22.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:22.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:22.393+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:22.397322+0000) 2019-10-13T11:53:25.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:25.498+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:25.501564+0000) 2019-10-13T11:53:26.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:26.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:26.436044+0000) 2019-10-13T11:53:27.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:27.390+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.395107+0000) 2019-10-13T11:53:27.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:27.393+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.396044+0000) 2019-10-13T11:53:27.397 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:27.394+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.396030+0000) 2019-10-13T11:53:27.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:27.394+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.396604+0000) 2019-10-13T11:53:27.398 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:27.394+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.396908+0000) 2019-10-13T11:53:27.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:27.394+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.397909+0000) 2019-10-13T11:53:27.399 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:27.394+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.398421+0000) 2019-10-13T11:53:27.402 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:27.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:27.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:27.401+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.406083+0000) 2019-10-13T11:53:28.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:28.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.700859+0000) 2019-10-13T11:53:28.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:28.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.705173+0000) 2019-10-13T11:53:28.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:28.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.717256+0000) 2019-10-13T11:53:28.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:28.897+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.900566+0000) 2019-10-13T11:53:29.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:29.933+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.936095+0000) 2019-10-13T11:53:32.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:32.190+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.191492+0000) 2019-10-13T11:53:32.406 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:32.402+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.405340+0000) 2019-10-13T11:53:32.406 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:32.402+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.405400+0000) 2019-10-13T11:53:32.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:32.402+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.406014+0000) 2019-10-13T11:53:32.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:32.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.406674+0000) 2019-10-13T11:53:32.408 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:32.402+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.407092+0000) 2019-10-13T11:53:32.408 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:32.406+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.408010+0000) 2019-10-13T11:53:32.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:32.405+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.408131+0000) 2019-10-13T11:53:32.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:32.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:32.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:32.409+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:32.412494+0000) 2019-10-13T11:53:35.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:35.498+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:35.501761+0000) 2019-10-13T11:53:36.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:36.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:36.436240+0000) 2019-10-13T11:53:37.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:37.410+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.414403+0000) 2019-10-13T11:53:37.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:37.410+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.415201+0000) 2019-10-13T11:53:37.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:37.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.416340+0000) 2019-10-13T11:53:37.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:37.414+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.416511+0000) 2019-10-13T11:53:37.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:37.414+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.416916+0000) 2019-10-13T11:53:37.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:37.414+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.417327+0000) 2019-10-13T11:53:37.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:37.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.417569+0000) 2019-10-13T11:53:37.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:37.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:37.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:37.421+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.422304+0000) 2019-10-13T11:53:38.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:38.698+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.701065+0000) 2019-10-13T11:53:38.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:38.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.705369+0000) 2019-10-13T11:53:38.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:38.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.717493+0000) 2019-10-13T11:53:38.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:38.905+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.907801+0000) 2019-10-13T11:53:39.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:39.933+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.936323+0000) 2019-10-13T11:53:42.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:42.189+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.191679+0000) 2019-10-13T11:53:42.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:42.421+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.425158+0000) 2019-10-13T11:53:42.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:42.421+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.425542+0000) 2019-10-13T11:53:42.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:42.421+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.425878+0000) 2019-10-13T11:53:42.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:42.421+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.426179+0000) 2019-10-13T11:53:42.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:42.425+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.426416+0000) 2019-10-13T11:53:42.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:42.421+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.426870+0000) 2019-10-13T11:53:42.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:42.425+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.427625+0000) 2019-10-13T11:53:42.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:42.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:42.435 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:42.433+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:42.434181+0000) 2019-10-13T11:53:45.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:45.497+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:45.501957+0000) 2019-10-13T11:53:46.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:46.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:46.436461+0000) 2019-10-13T11:53:47.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:47.429+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.434824+0000) 2019-10-13T11:53:47.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:47.433+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.435429+0000) 2019-10-13T11:53:47.437 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:47.433+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.435770+0000) 2019-10-13T11:53:47.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:47.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.436445+0000) 2019-10-13T11:53:47.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:47.433+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.436958+0000) 2019-10-13T11:53:47.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:47.433+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.438105+0000) 2019-10-13T11:53:47.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:47.433+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.438426+0000) 2019-10-13T11:53:47.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:47.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:47.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:47.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.440793+0000) 2019-10-13T11:53:48.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:48.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.701265+0000) 2019-10-13T11:53:48.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:48.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.705564+0000) 2019-10-13T11:53:48.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:48.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.717725+0000) 2019-10-13T11:53:48.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:48.913+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.916728+0000) 2019-10-13T11:53:49.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:49.933+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.936516+0000) 2019-10-13T11:53:50.271 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:53:50.272 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:50.275 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:53:50.276 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:52.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:52.189+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.191855+0000) 2019-10-13T11:53:52.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:52.441+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.444217+0000) 2019-10-13T11:53:52.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:52.441+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.444877+0000) 2019-10-13T11:53:52.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:52.441+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.445200+0000) 2019-10-13T11:53:52.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:52.441+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.446233+0000) 2019-10-13T11:53:52.448 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:52.445+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.447906+0000) 2019-10-13T11:53:52.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:52.445+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.448426+0000) 2019-10-13T11:53:52.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:52.445+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.449149+0000) 2019-10-13T11:53:52.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:52.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:52.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:52.449+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:52.452742+0000) 2019-10-13T11:53:55.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:55.497+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:55.502159+0000) 2019-10-13T11:53:56.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:56.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:56.436648+0000) 2019-10-13T11:53:57.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:53:57.449+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.454174+0000) 2019-10-13T11:53:57.455 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:57.449+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.454677+0000) 2019-10-13T11:53:57.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:53:57.453+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.455642+0000) 2019-10-13T11:53:57.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:57.453+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.455967+0000) 2019-10-13T11:53:57.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:57.453+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.456403+0000) 2019-10-13T11:53:57.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:57.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.456978+0000) 2019-10-13T11:53:57.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:53:57.453+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.458588+0000) 2019-10-13T11:53:57.461 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:53:57.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:53:57.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:57.461+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.464291+0000) 2019-10-13T11:53:58.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:53:58.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.701480+0000) 2019-10-13T11:53:58.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:53:58.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.705756+0000) 2019-10-13T11:53:58.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:53:58.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.717943+0000) 2019-10-13T11:53:58.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:53:58.921+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.923438+0000) 2019-10-13T11:53:59.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:53:59.933+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.936712+0000) 2019-10-13T11:54:02.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:02.189+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.192055+0000) 2019-10-13T11:54:02.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:02.461+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.464392+0000) 2019-10-13T11:54:02.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:02.461+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.464640+0000) 2019-10-13T11:54:02.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:02.461+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.465201+0000) 2019-10-13T11:54:02.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:02.461+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.465947+0000) 2019-10-13T11:54:02.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:02.461+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.466656+0000) 2019-10-13T11:54:02.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:02.461+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.467096+0000) 2019-10-13T11:54:02.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:02.465+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.467580+0000) 2019-10-13T11:54:02.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:02.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:02.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:02.469+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:02.472552+0000) 2019-10-13T11:54:05.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:05.497+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:05.502356+0000) 2019-10-13T11:54:06.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:06.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:06.436841+0000) 2019-10-13T11:54:07.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:07.469+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.473725+0000) 2019-10-13T11:54:07.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:07.472+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.475246+0000) 2019-10-13T11:54:07.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:07.473+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.475923+0000) 2019-10-13T11:54:07.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:07.473+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.476308+0000) 2019-10-13T11:54:07.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:07.473+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.476746+0000) 2019-10-13T11:54:07.477 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:07.473+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.477045+0000) 2019-10-13T11:54:07.478 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:07.473+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.477905+0000) 2019-10-13T11:54:07.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:07.476+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.480788+0000) 2019-10-13T11:54:07.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:07.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:08.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:08.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.701714+0000) 2019-10-13T11:54:08.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:08.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.705959+0000) 2019-10-13T11:54:08.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:08.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.718155+0000) 2019-10-13T11:54:08.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:08.928+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.932352+0000) 2019-10-13T11:54:09.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:09.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.936945+0000) 2019-10-13T11:54:12.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:12.189+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.192298+0000) 2019-10-13T11:54:12.486 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:12.481+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.485105+0000) 2019-10-13T11:54:12.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:12.480+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.485749+0000) 2019-10-13T11:54:12.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:12.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.486895+0000) 2019-10-13T11:54:12.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:12.481+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.486914+0000) 2019-10-13T11:54:12.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:12.485+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.487388+0000) 2019-10-13T11:54:12.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:12.485+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.487918+0000) 2019-10-13T11:54:12.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:12.485+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.488272+0000) 2019-10-13T11:54:12.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:12.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:12.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:12.488+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:12.492877+0000) 2019-10-13T11:54:15.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:15.497+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:15.502552+0000) 2019-10-13T11:54:16.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:16.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:16.437055+0000) 2019-10-13T11:54:17.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:17.489+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.494321+0000) 2019-10-13T11:54:17.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:17.489+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.494852+0000) 2019-10-13T11:54:17.496 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:17.493+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.495452+0000) 2019-10-13T11:54:17.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:17.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.495480+0000) 2019-10-13T11:54:17.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:17.492+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.496779+0000) 2019-10-13T11:54:17.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:17.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.497315+0000) 2019-10-13T11:54:17.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:17.493+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.497755+0000) 2019-10-13T11:54:17.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:17.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 134996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:17.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:17.500+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.503521+0000) 2019-10-13T11:54:18.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:18.697+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.701945+0000) 2019-10-13T11:54:18.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:18.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.706137+0000) 2019-10-13T11:54:18.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:18.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.718375+0000) 2019-10-13T11:54:18.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:18.936+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.938516+0000) 2019-10-13T11:54:19.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:19.936+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.937149+0000) 2019-10-13T11:54:20.452 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:54:20.452 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:20.456 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:54:20.456 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:22.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:22.189+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.192500+0000) 2019-10-13T11:54:22.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:22.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.503817+0000) 2019-10-13T11:54:22.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:22.501+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.504117+0000) 2019-10-13T11:54:22.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:22.501+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.505246+0000) 2019-10-13T11:54:22.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:22.501+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.505561+0000) 2019-10-13T11:54:22.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:22.501+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.505976+0000) 2019-10-13T11:54:22.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:22.501+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.506508+0000) 2019-10-13T11:54:22.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:22.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.506637+0000) 2019-10-13T11:54:22.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:22.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:22.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:22.508+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:22.510919+0000) 2019-10-13T11:54:25.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:25.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:25.502726+0000) 2019-10-13T11:54:26.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:26.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:26.437201+0000) 2019-10-13T11:54:27.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:27.508+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.513489+0000) 2019-10-13T11:54:27.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:27.508+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.514418+0000) 2019-10-13T11:54:27.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:27.508+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.515079+0000) 2019-10-13T11:54:27.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:27.512+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.515734+0000) 2019-10-13T11:54:27.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:27.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.515480+0000) 2019-10-13T11:54:27.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:27.512+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.516052+0000) 2019-10-13T11:54:27.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:27.512+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.516908+0000) 2019-10-13T11:54:27.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:27.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:27.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:27.516+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.522114+0000) 2019-10-13T11:54:28.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:28.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.702154+0000) 2019-10-13T11:54:28.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:28.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.706341+0000) 2019-10-13T11:54:28.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:28.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.718579+0000) 2019-10-13T11:54:28.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:28.944+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.944890+0000) 2019-10-13T11:54:29.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:29.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.937337+0000) 2019-10-13T11:54:32.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:32.188+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.192725+0000) 2019-10-13T11:54:32.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:32.520+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.523106+0000) 2019-10-13T11:54:32.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:32.516+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.523261+0000) 2019-10-13T11:54:32.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:32.520+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.523587+0000) 2019-10-13T11:54:32.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:32.520+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.524087+0000) 2019-10-13T11:54:32.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:32.520+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.524897+0000) 2019-10-13T11:54:32.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:32.520+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.525577+0000) 2019-10-13T11:54:32.528 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:32.520+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.525276+0000) 2019-10-13T11:54:32.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:32.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:32.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:32.528+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:32.531018+0000) 2019-10-13T11:54:35.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:35.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:35.502945+0000) 2019-10-13T11:54:36.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:36.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:36.437386+0000) 2019-10-13T11:54:37.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:37.528+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.532147+0000) 2019-10-13T11:54:37.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:37.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.533181+0000) 2019-10-13T11:54:37.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:37.528+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.533431+0000) 2019-10-13T11:54:37.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:37.528+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.533889+0000) 2019-10-13T11:54:37.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:37.528+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.534876+0000) 2019-10-13T11:54:37.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:37.532+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.535671+0000) 2019-10-13T11:54:37.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:37.532+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.535092+0000) 2019-10-13T11:54:37.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:37.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:37.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:37.540+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.543588+0000) 2019-10-13T11:54:38.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:38.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.702367+0000) 2019-10-13T11:54:38.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:38.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.706539+0000) 2019-10-13T11:54:38.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:38.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.718758+0000) 2019-10-13T11:54:38.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:38.952+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.956014+0000) 2019-10-13T11:54:39.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:39.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.937529+0000) 2019-10-13T11:54:42.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:42.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.192925+0000) 2019-10-13T11:54:42.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:42.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.541629+0000) 2019-10-13T11:54:42.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:42.536+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.542223+0000) 2019-10-13T11:54:42.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:42.536+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.543073+0000) 2019-10-13T11:54:42.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:42.540+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.543374+0000) 2019-10-13T11:54:42.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:42.540+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.544159+0000) 2019-10-13T11:54:42.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:42.540+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.544871+0000) 2019-10-13T11:54:42.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:42.540+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.545325+0000) 2019-10-13T11:54:42.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:42.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:42.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:42.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:42.554000+0000) 2019-10-13T11:54:45.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:45.496+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:45.503174+0000) 2019-10-13T11:54:46.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:46.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:46.437580+0000) 2019-10-13T11:54:47.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:47.548+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.551559+0000) 2019-10-13T11:54:47.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:47.548+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.552045+0000) 2019-10-13T11:54:47.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:47.548+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.552473+0000) 2019-10-13T11:54:47.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:47.548+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.553547+0000) 2019-10-13T11:54:47.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:47.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.553919+0000) 2019-10-13T11:54:47.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:47.548+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.554340+0000) 2019-10-13T11:54:47.555 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:47.548+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.554817+0000) 2019-10-13T11:54:47.559 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:47.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:47.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:47.556+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.558797+0000) 2019-10-13T11:54:48.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:48.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.702572+0000) 2019-10-13T11:54:48.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:48.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.706759+0000) 2019-10-13T11:54:48.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:48.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.718993+0000) 2019-10-13T11:54:48.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:48.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.962566+0000) 2019-10-13T11:54:49.938 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:49.932+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.937754+0000) 2019-10-13T11:54:50.568 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:54:50.568 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:50.571 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:54:50.572 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:52.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:52.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.193169+0000) 2019-10-13T11:54:52.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:52.556+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.561408+0000) 2019-10-13T11:54:52.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:52.556+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.561845+0000) 2019-10-13T11:54:52.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:52.556+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.562511+0000) 2019-10-13T11:54:52.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:52.556+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.563266+0000) 2019-10-13T11:54:52.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:52.560+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.563731+0000) 2019-10-13T11:54:52.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:52.560+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.564321+0000) 2019-10-13T11:54:52.565 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:52.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.564698+0000) 2019-10-13T11:54:52.568 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:52.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:52.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:52.568+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:52.570956+0000) 2019-10-13T11:54:55.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:55.500+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:55.503378+0000) 2019-10-13T11:54:56.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:56.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:56.437793+0000) 2019-10-13T11:54:57.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:57.564+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.571013+0000) 2019-10-13T11:54:57.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:54:57.568+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.571612+0000) 2019-10-13T11:54:57.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:54:57.568+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.571779+0000) 2019-10-13T11:54:57.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:57.568+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.572125+0000) 2019-10-13T11:54:57.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:54:57.568+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.573334+0000) 2019-10-13T11:54:57.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:57.568+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.573634+0000) 2019-10-13T11:54:57.575 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:57.572+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.574557+0000) 2019-10-13T11:54:57.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:54:57.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:54:57.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:57.576+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.580918+0000) 2019-10-13T11:54:58.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:54:58.696+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.702774+0000) 2019-10-13T11:54:58.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:54:58.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.706971+0000) 2019-10-13T11:54:58.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:54:58.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.719207+0000) 2019-10-13T11:54:58.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:54:58.964+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.968902+0000) 2019-10-13T11:54:59.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:54:59.931+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.937945+0000) 2019-10-13T11:55:02.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:02.188+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.193371+0000) 2019-10-13T11:55:02.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:02.576+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.580987+0000) 2019-10-13T11:55:02.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:02.576+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.581240+0000) 2019-10-13T11:55:02.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:02.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.581812+0000) 2019-10-13T11:55:02.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:02.576+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.582200+0000) 2019-10-13T11:55:02.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:02.579+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.582360+0000) 2019-10-13T11:55:02.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:02.576+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.582903+0000) 2019-10-13T11:55:02.584 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:02.580+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.583706+0000) 2019-10-13T11:55:02.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:02.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:02.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:02.587+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:02.592163+0000) 2019-10-13T11:55:05.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:05.500+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:05.503610+0000) 2019-10-13T11:55:06.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:06.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:06.437982+0000) 2019-10-13T11:55:07.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:07.583+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.590326+0000) 2019-10-13T11:55:07.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:07.587+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.591358+0000) 2019-10-13T11:55:07.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:07.587+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.591473+0000) 2019-10-13T11:55:07.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:07.587+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.592108+0000) 2019-10-13T11:55:07.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:07.587+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.592422+0000) 2019-10-13T11:55:07.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:07.587+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.592949+0000) 2019-10-13T11:55:07.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:07.587+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.593158+0000) 2019-10-13T11:55:07.598 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:07.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:07.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:07.595+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.598419+0000) 2019-10-13T11:55:08.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:08.695+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.702967+0000) 2019-10-13T11:55:08.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:08.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.707182+0000) 2019-10-13T11:55:08.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:08.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.719494+0000) 2019-10-13T11:55:08.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:08.971+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.975293+0000) 2019-10-13T11:55:09.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:09.935+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.938166+0000) 2019-10-13T11:55:12.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:12.191+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.193598+0000) 2019-10-13T11:55:12.601 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:12.595+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.600617+0000) 2019-10-13T11:55:12.602 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:12.595+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.600550+0000) 2019-10-13T11:55:12.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:12.595+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.601928+0000) 2019-10-13T11:55:12.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:12.595+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.603141+0000) 2019-10-13T11:55:12.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:12.595+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.602445+0000) 2019-10-13T11:55:12.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:12.599+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.603906+0000) 2019-10-13T11:55:12.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:12.599+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.604342+0000) 2019-10-13T11:55:12.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:12.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:12.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:12.603+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:12.607887+0000) 2019-10-13T11:55:15.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:15.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:15.503831+0000) 2019-10-13T11:55:16.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:16.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:16.438226+0000) 2019-10-13T11:55:17.611 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:17.607+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.610254+0000) 2019-10-13T11:55:17.612 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:17.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.611522+0000) 2019-10-13T11:55:17.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:17.607+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.611684+0000) 2019-10-13T11:55:17.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:17.607+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.611987+0000) 2019-10-13T11:55:17.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:17.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.613056+0000) 2019-10-13T11:55:17.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:17.607+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.613520+0000) 2019-10-13T11:55:17.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:17.607+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.613804+0000) 2019-10-13T11:55:17.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:17.611+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.616018+0000) 2019-10-13T11:55:17.618 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:17.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:18.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:18.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.703163+0000) 2019-10-13T11:55:18.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:18.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.707384+0000) 2019-10-13T11:55:18.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:18.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.719718+0000) 2019-10-13T11:55:18.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:18.979+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.981737+0000) 2019-10-13T11:55:19.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:19.935+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.938358+0000) 2019-10-13T11:55:20.716 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:55:20.716 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:20.720 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:55:20.720 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:22.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:22.191+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.193800+0000) 2019-10-13T11:55:22.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:22.615+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.620559+0000) 2019-10-13T11:55:22.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:22.615+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.621245+0000) 2019-10-13T11:55:22.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:22.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.621881+0000) 2019-10-13T11:55:22.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:22.615+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.623031+0000) 2019-10-13T11:55:22.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:22.619+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.623333+0000) 2019-10-13T11:55:22.624 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:22.619+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.623537+0000) 2019-10-13T11:55:22.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:22.619+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.624153+0000) 2019-10-13T11:55:22.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:22.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:22.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:22.623+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:22.628825+0000) 2019-10-13T11:55:25.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:25.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:25.504032+0000) 2019-10-13T11:55:26.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:26.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:26.438416+0000) 2019-10-13T11:55:27.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:27.627+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.630701+0000) 2019-10-13T11:55:27.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:27.627+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.631283+0000) 2019-10-13T11:55:27.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:27.627+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.631785+0000) 2019-10-13T11:55:27.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:27.627+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.632066+0000) 2019-10-13T11:55:27.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:27.627+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.632516+0000) 2019-10-13T11:55:27.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:27.627+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.634032+0000) 2019-10-13T11:55:27.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:27.627+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.634636+0000) 2019-10-13T11:55:27.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:27.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:27.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:27.635+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.641492+0000) 2019-10-13T11:55:28.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:28.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.703340+0000) 2019-10-13T11:55:28.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:28.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.707556+0000) 2019-10-13T11:55:28.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:28.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.719932+0000) 2019-10-13T11:55:28.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:28.987+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.988107+0000) 2019-10-13T11:55:29.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:29.935+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.938556+0000) 2019-10-13T11:55:32.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:32.191+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.194031+0000) 2019-10-13T11:55:32.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:32.635+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.641420+0000) 2019-10-13T11:55:32.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:32.635+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.642033+0000) 2019-10-13T11:55:32.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:32.635+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.642926+0000) 2019-10-13T11:55:32.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:32.639+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.643337+0000) 2019-10-13T11:55:32.645 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:32.639+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.644182+0000) 2019-10-13T11:55:32.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:32.639+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.644419+0000) 2019-10-13T11:55:32.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:32.639+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.645360+0000) 2019-10-13T11:55:32.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:32.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:32.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:32.643+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:32.649618+0000) 2019-10-13T11:55:35.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:35.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:35.504248+0000) 2019-10-13T11:55:36.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:36.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:36.438664+0000) 2019-10-13T11:55:37.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:37.651+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.652899+0000) 2019-10-13T11:55:37.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:37.651+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.653313+0000) 2019-10-13T11:55:37.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:37.651+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.653711+0000) 2019-10-13T11:55:37.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:37.651+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.654187+0000) 2019-10-13T11:55:37.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:37.651+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.654668+0000) 2019-10-13T11:55:37.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:37.651+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.655209+0000) 2019-10-13T11:55:37.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:37.655+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.655746+0000) 2019-10-13T11:55:37.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:37.659+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135545 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:37.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:37.655+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.661693+0000) 2019-10-13T11:55:38.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:38.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.703556+0000) 2019-10-13T11:55:38.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:38.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.707784+0000) 2019-10-13T11:55:38.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:38.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.720124+0000) 2019-10-13T11:55:38.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:38.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.994580+0000) 2019-10-13T11:55:39.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:39.935+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.938761+0000) 2019-10-13T11:55:42.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:42.191+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.194249+0000) 2019-10-13T11:55:42.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:42.663+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.663598+0000) 2019-10-13T11:55:42.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:42.663+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.664498+0000) 2019-10-13T11:55:42.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:42.663+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.665043+0000) 2019-10-13T11:55:42.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:42.659+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.665205+0000) 2019-10-13T11:55:42.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:42.663+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.665828+0000) 2019-10-13T11:55:42.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:42.663+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.666402+0000) 2019-10-13T11:55:42.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:42.663+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.666762+0000) 2019-10-13T11:55:42.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:42.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:42.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:42.667+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:42.672530+0000) 2019-10-13T11:55:45.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:45.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:45.504449+0000) 2019-10-13T11:55:46.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:46.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:46.438857+0000) 2019-10-13T11:55:47.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:47.671+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.673337+0000) 2019-10-13T11:55:47.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:47.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.673703+0000) 2019-10-13T11:55:47.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:47.671+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.674217+0000) 2019-10-13T11:55:47.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:47.675+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.675764+0000) 2019-10-13T11:55:47.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:47.675+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.675987+0000) 2019-10-13T11:55:47.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:47.675+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.676496+0000) 2019-10-13T11:55:47.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:47.675+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.677122+0000) 2019-10-13T11:55:47.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:47.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:47.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:47.675+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.681441+0000) 2019-10-13T11:55:48.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:48.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.703788+0000) 2019-10-13T11:55:48.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:48.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.707983+0000) 2019-10-13T11:55:48.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:48.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.720326+0000) 2019-10-13T11:55:49.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:48.999+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.001059+0000) 2019-10-13T11:55:49.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:49.935+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.938975+0000) 2019-10-13T11:55:50.827 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:55:50.827 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:50.831 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:55:50.831 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:52.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:52.190+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.194454+0000) 2019-10-13T11:55:52.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:52.682+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.684755+0000) 2019-10-13T11:55:52.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:52.682+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.684902+0000) 2019-10-13T11:55:52.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:52.682+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.685475+0000) 2019-10-13T11:55:52.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:52.682+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.685956+0000) 2019-10-13T11:55:52.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:52.682+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.686712+0000) 2019-10-13T11:55:52.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:52.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.686953+0000) 2019-10-13T11:55:52.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:52.686+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.687723+0000) 2019-10-13T11:55:52.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:52.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:52.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:52.687+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:52.691752+0000) 2019-10-13T11:55:55.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:55.502+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:55.504645+0000) 2019-10-13T11:55:56.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:56.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:56.439049+0000) 2019-10-13T11:55:57.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:57.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.693286+0000) 2019-10-13T11:55:57.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:55:57.690+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.694554+0000) 2019-10-13T11:55:57.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:57.690+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.694939+0000) 2019-10-13T11:55:57.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:55:57.690+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.695161+0000) 2019-10-13T11:55:57.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:57.694+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.695516+0000) 2019-10-13T11:55:57.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:55:57.694+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.696109+0000) 2019-10-13T11:55:57.697 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:57.690+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.696227+0000) 2019-10-13T11:55:57.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:55:57.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:55:57.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:57.694+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.702059+0000) 2019-10-13T11:55:58.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:55:58.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.703988+0000) 2019-10-13T11:55:58.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:55:58.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.708182+0000) 2019-10-13T11:55:58.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:55:58.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.720519+0000) 2019-10-13T11:55:59.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:55:59.002+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.007737+0000) 2019-10-13T11:55:59.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:55:59.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.939174+0000) 2019-10-13T11:56:02.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:02.190+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.194663+0000) 2019-10-13T11:56:02.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:02.698+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.702647+0000) 2019-10-13T11:56:02.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:02.702+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.703590+0000) 2019-10-13T11:56:02.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:02.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.704322+0000) 2019-10-13T11:56:02.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:02.698+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.704511+0000) 2019-10-13T11:56:02.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:02.702+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.705167+0000) 2019-10-13T11:56:02.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:02.702+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.706026+0000) 2019-10-13T11:56:02.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:02.702+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.706462+0000) 2019-10-13T11:56:02.710 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:02.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:02.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:02.706+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:02.712199+0000) 2019-10-13T11:56:05.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:05.502+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:05.504885+0000) 2019-10-13T11:56:06.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:06.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:06.439219+0000) 2019-10-13T11:56:07.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:07.710+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.713505+0000) 2019-10-13T11:56:07.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:07.706+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.713545+0000) 2019-10-13T11:56:07.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:07.710+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.714122+0000) 2019-10-13T11:56:07.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:07.710+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.714563+0000) 2019-10-13T11:56:07.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:07.710+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.715239+0000) 2019-10-13T11:56:07.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:07.714+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.715968+0000) 2019-10-13T11:56:07.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:07.714+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.716733+0000) 2019-10-13T11:56:07.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:07.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:07.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:07.714+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.721301+0000) 2019-10-13T11:56:08.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:08.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.704200+0000) 2019-10-13T11:56:08.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:08.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.708389+0000) 2019-10-13T11:56:08.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:08.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.720719+0000) 2019-10-13T11:56:09.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:09.010+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.014209+0000) 2019-10-13T11:56:09.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:09.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.939402+0000) 2019-10-13T11:56:12.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:12.190+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.194860+0000) 2019-10-13T11:56:12.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:12.718+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.722433+0000) 2019-10-13T11:56:12.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:12.722+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.723453+0000) 2019-10-13T11:56:12.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:12.722+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.723802+0000) 2019-10-13T11:56:12.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:12.722+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.724779+0000) 2019-10-13T11:56:12.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:12.718+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.725089+0000) 2019-10-13T11:56:12.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:12.722+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.725594+0000) 2019-10-13T11:56:12.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:12.722+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.726012+0000) 2019-10-13T11:56:12.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:12.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:12.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:12.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:12.731215+0000) 2019-10-13T11:56:15.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:15.502+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:15.505086+0000) 2019-10-13T11:56:16.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:16.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:16.439412+0000) 2019-10-13T11:56:17.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:17.730+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.732810+0000) 2019-10-13T11:56:17.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:17.730+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.733352+0000) 2019-10-13T11:56:17.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:17.730+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.734787+0000) 2019-10-13T11:56:17.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:17.730+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.735214+0000) 2019-10-13T11:56:17.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:17.730+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.735316+0000) 2019-10-13T11:56:17.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:17.734+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.735907+0000) 2019-10-13T11:56:17.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:17.734+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.736669+0000) 2019-10-13T11:56:17.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:17.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:17.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:17.738+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.742674+0000) 2019-10-13T11:56:18.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:18.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.704380+0000) 2019-10-13T11:56:18.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:18.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.708565+0000) 2019-10-13T11:56:18.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:18.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.720915+0000) 2019-10-13T11:56:19.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:19.018+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.020700+0000) 2019-10-13T11:56:19.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:19.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.939612+0000) 2019-10-13T11:56:20.929 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:56:20.930 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:20.933 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:56:20.934 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:22.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:22.194+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.195087+0000) 2019-10-13T11:56:22.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:22.738+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.742570+0000) 2019-10-13T11:56:22.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:22.742+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.743089+0000) 2019-10-13T11:56:22.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:22.742+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.744656+0000) 2019-10-13T11:56:22.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:22.742+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.744928+0000) 2019-10-13T11:56:22.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:22.742+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.745623+0000) 2019-10-13T11:56:22.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:22.738+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.745910+0000) 2019-10-13T11:56:22.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:22.742+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.746709+0000) 2019-10-13T11:56:22.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:22.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:22.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:22.746+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:22.750714+0000) 2019-10-13T11:56:25.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:25.502+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:25.505238+0000) 2019-10-13T11:56:26.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:26.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:26.439599+0000) 2019-10-13T11:56:27.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:27.746+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.752085+0000) 2019-10-13T11:56:27.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:27.750+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.752704+0000) 2019-10-13T11:56:27.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:27.750+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.753670+0000) 2019-10-13T11:56:27.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:27.750+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.753871+0000) 2019-10-13T11:56:27.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:27.750+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.754237+0000) 2019-10-13T11:56:27.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:27.750+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.754720+0000) 2019-10-13T11:56:27.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:27.750+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.754994+0000) 2019-10-13T11:56:27.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:27.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:27.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:27.754+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.759476+0000) 2019-10-13T11:56:28.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:28.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.704555+0000) 2019-10-13T11:56:28.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:28.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.708736+0000) 2019-10-13T11:56:28.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:28.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.721105+0000) 2019-10-13T11:56:29.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:29.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.027419+0000) 2019-10-13T11:56:29.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:29.934+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.939838+0000) 2019-10-13T11:56:32.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:32.193+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.195292+0000) 2019-10-13T11:56:32.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:32.757+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.762083+0000) 2019-10-13T11:56:32.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:32.757+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.762623+0000) 2019-10-13T11:56:32.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:32.761+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.763760+0000) 2019-10-13T11:56:32.765 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:32.761+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.764034+0000) 2019-10-13T11:56:32.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:32.761+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.764358+0000) 2019-10-13T11:56:32.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:32.758+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.764877+0000) 2019-10-13T11:56:32.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:32.761+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.766032+0000) 2019-10-13T11:56:32.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:32.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:32.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:32.766+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:32.771737+0000) 2019-10-13T11:56:35.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:35.501+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:35.505479+0000) 2019-10-13T11:56:36.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:36.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:36.439805+0000) 2019-10-13T11:56:37.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:37.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.771983+0000) 2019-10-13T11:56:37.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:37.769+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.772709+0000) 2019-10-13T11:56:37.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:37.769+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.773304+0000) 2019-10-13T11:56:37.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:37.769+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.773032+0000) 2019-10-13T11:56:37.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:37.769+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.774457+0000) 2019-10-13T11:56:37.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:37.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.775189+0000) 2019-10-13T11:56:37.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:37.773+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.775545+0000) 2019-10-13T11:56:37.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:37.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:37.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:37.774+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.779189+0000) 2019-10-13T11:56:38.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:38.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.704759+0000) 2019-10-13T11:56:38.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:38.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.708921+0000) 2019-10-13T11:56:38.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:38.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.721281+0000) 2019-10-13T11:56:39.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:39.030+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.033913+0000) 2019-10-13T11:56:39.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:39.938+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.940055+0000) 2019-10-13T11:56:42.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:42.193+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.195568+0000) 2019-10-13T11:56:42.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:42.777+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.782008+0000) 2019-10-13T11:56:42.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:42.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.782356+0000) 2019-10-13T11:56:42.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:42.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.782757+0000) 2019-10-13T11:56:42.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:42.781+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.783431+0000) 2019-10-13T11:56:42.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:42.781+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.784704+0000) 2019-10-13T11:56:42.785 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:42.782+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.784788+0000) 2019-10-13T11:56:42.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:42.781+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.785405+0000) 2019-10-13T11:56:42.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:42.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 135996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:42.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:42.790+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:42.792127+0000) 2019-10-13T11:56:45.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:45.501+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:45.505706+0000) 2019-10-13T11:56:46.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:46.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:46.439979+0000) 2019-10-13T11:56:47.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:47.789+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.792265+0000) 2019-10-13T11:56:47.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:47.789+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.792688+0000) 2019-10-13T11:56:47.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:47.789+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.792974+0000) 2019-10-13T11:56:47.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:47.789+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.793563+0000) 2019-10-13T11:56:47.795 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:47.790+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.793688+0000) 2019-10-13T11:56:47.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:47.789+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.794669+0000) 2019-10-13T11:56:47.795 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:47.789+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.794445+0000) 2019-10-13T11:56:47.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:47.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:47.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:47.798+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.798576+0000) 2019-10-13T11:56:48.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:48.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.704954+0000) 2019-10-13T11:56:48.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:48.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.709123+0000) 2019-10-13T11:56:48.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:48.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.721420+0000) 2019-10-13T11:56:49.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:49.037+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.040499+0000) 2019-10-13T11:56:49.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:49.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.940256+0000) 2019-10-13T11:56:51.041 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:56:51.041 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:51.045 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:56:51.045 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:52.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:52.193+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.195778+0000) 2019-10-13T11:56:52.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:52.797+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.801208+0000) 2019-10-13T11:56:52.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:52.797+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.802640+0000) 2019-10-13T11:56:52.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:52.801+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.803376+0000) 2019-10-13T11:56:52.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:52.801+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.803819+0000) 2019-10-13T11:56:52.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:52.801+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.804100+0000) 2019-10-13T11:56:52.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:52.801+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.804470+0000) 2019-10-13T11:56:52.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:52.801+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.805159+0000) 2019-10-13T11:56:52.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:52.805+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:52.808360+0000) 2019-10-13T11:56:52.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:52.805+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:55.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:55.501+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:55.505966+0000) 2019-10-13T11:56:56.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:56.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:56.440148+0000) 2019-10-13T11:56:57.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:57.809+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.812037+0000) 2019-10-13T11:56:57.813 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:56:57.809+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.812702+0000) 2019-10-13T11:56:57.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:57.809+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.813027+0000) 2019-10-13T11:56:57.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:57.809+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.814003+0000) 2019-10-13T11:56:57.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:56:57.809+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.814038+0000) 2019-10-13T11:56:57.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:57.809+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.814336+0000) 2019-10-13T11:56:57.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:56:57.809+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.814943+0000) 2019-10-13T11:56:57.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:56:57.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:56:57.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:57.817+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.821958+0000) 2019-10-13T11:56:58.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:56:58.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.705137+0000) 2019-10-13T11:56:58.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:56:58.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.709326+0000) 2019-10-13T11:56:58.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:56:58.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.721599+0000) 2019-10-13T11:56:59.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:56:59.045+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.046873+0000) 2019-10-13T11:56:59.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:56:59.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.940477+0000) 2019-10-13T11:57:02.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:02.193+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.196018+0000) 2019-10-13T11:57:02.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:02.817+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.822689+0000) 2019-10-13T11:57:02.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:02.817+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.823082+0000) 2019-10-13T11:57:02.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:02.821+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.823803+0000) 2019-10-13T11:57:02.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:02.821+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.823917+0000) 2019-10-13T11:57:02.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:02.821+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.824323+0000) 2019-10-13T11:57:02.826 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:02.821+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.825160+0000) 2019-10-13T11:57:02.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:02.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.825763+0000) 2019-10-13T11:57:02.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:02.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:02.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:02.829+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:02.830945+0000) 2019-10-13T11:57:05.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:05.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.506172+0000) 2019-10-13T11:57:06.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:06.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:06.440340+0000) 2019-10-13T11:57:07.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:07.825+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.831038+0000) 2019-10-13T11:57:07.832 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:07.829+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.831648+0000) 2019-10-13T11:57:07.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:07.829+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.832747+0000) 2019-10-13T11:57:07.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:07.829+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.833549+0000) 2019-10-13T11:57:07.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:07.829+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.833493+0000) 2019-10-13T11:57:07.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:07.829+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.833976+0000) 2019-10-13T11:57:07.835 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:07.829+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.834488+0000) 2019-10-13T11:57:07.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:07.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:07.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:07.837+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.838744+0000) 2019-10-13T11:57:08.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:08.701+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.705335+0000) 2019-10-13T11:57:08.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:08.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.709533+0000) 2019-10-13T11:57:08.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:08.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.721781+0000) 2019-10-13T11:57:09.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:09.049+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:09.053351+0000) 2019-10-13T11:57:09.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:09.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:09.940673+0000) 2019-10-13T11:57:12.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:12.193+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.196234+0000) 2019-10-13T11:57:12.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:12.837+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.840974+0000) 2019-10-13T11:57:12.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:12.837+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.842189+0000) 2019-10-13T11:57:12.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:12.837+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.842863+0000) 2019-10-13T11:57:12.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:12.841+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.843734+0000) 2019-10-13T11:57:12.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:12.841+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.844227+0000) 2019-10-13T11:57:12.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:12.841+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.844415+0000) 2019-10-13T11:57:12.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:12.841+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.845141+0000) 2019-10-13T11:57:12.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:12.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:12.852 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:12.849+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:12.851532+0000) 2019-10-13T11:57:15.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:15.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.506399+0000) 2019-10-13T11:57:16.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:16.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:16.440533+0000) 2019-10-13T11:57:17.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:17.844+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.851266+0000) 2019-10-13T11:57:17.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:17.848+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.852236+0000) 2019-10-13T11:57:17.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:17.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.851972+0000) 2019-10-13T11:57:17.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:17.849+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.852933+0000) 2019-10-13T11:57:17.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:17.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.854027+0000) 2019-10-13T11:57:17.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:17.848+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.854477+0000) 2019-10-13T11:57:17.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:17.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.855154+0000) 2019-10-13T11:57:17.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:17.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:17.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:17.857+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.858714+0000) 2019-10-13T11:57:18.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:18.700+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.705544+0000) 2019-10-13T11:57:18.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:18.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.709707+0000) 2019-10-13T11:57:18.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:18.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.721959+0000) 2019-10-13T11:57:19.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:19.057+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:19.060164+0000) 2019-10-13T11:57:19.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:19.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:19.940900+0000) 2019-10-13T11:57:21.321 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:57:21.321 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:21.325 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:57:21.325 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:22.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:22.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.196436+0000) 2019-10-13T11:57:22.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:22.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.861134+0000) 2019-10-13T11:57:22.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:22.857+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.861246+0000) 2019-10-13T11:57:22.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:22.856+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.862233+0000) 2019-10-13T11:57:22.863 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:22.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.862571+0000) 2019-10-13T11:57:22.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:22.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.862816+0000) 2019-10-13T11:57:22.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:22.860+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.863529+0000) 2019-10-13T11:57:22.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:22.860+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.864410+0000) 2019-10-13T11:57:22.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:22.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:22.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:22.869+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:22.870347+0000) 2019-10-13T11:57:25.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:25.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.506626+0000) 2019-10-13T11:57:26.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:26.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:26.440718+0000) 2019-10-13T11:57:27.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:27.864+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.870890+0000) 2019-10-13T11:57:27.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:27.868+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.871873+0000) 2019-10-13T11:57:27.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:27.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.872194+0000) 2019-10-13T11:57:27.873 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:27.868+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.872662+0000) 2019-10-13T11:57:27.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:27.868+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.873318+0000) 2019-10-13T11:57:27.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:27.873+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.874230+0000) 2019-10-13T11:57:27.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:27.868+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.875075+0000) 2019-10-13T11:57:27.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:27.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:27.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:27.877+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.881120+0000) 2019-10-13T11:57:28.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:28.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.705718+0000) 2019-10-13T11:57:28.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:28.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.709908+0000) 2019-10-13T11:57:28.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:28.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.722096+0000) 2019-10-13T11:57:29.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:29.065+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.066178+0000) 2019-10-13T11:57:29.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:29.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.941095+0000) 2019-10-13T11:57:32.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:32.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.196604+0000) 2019-10-13T11:57:32.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:32.876+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.881377+0000) 2019-10-13T11:57:32.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:32.876+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.881825+0000) 2019-10-13T11:57:32.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:32.876+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.882768+0000) 2019-10-13T11:57:32.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:32.881+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.883203+0000) 2019-10-13T11:57:32.884 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:32.880+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.883645+0000) 2019-10-13T11:57:32.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:32.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.884120+0000) 2019-10-13T11:57:32.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:32.880+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.884674+0000) 2019-10-13T11:57:32.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:32.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:32.889 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:32.885+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:32.888895+0000) 2019-10-13T11:57:35.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:35.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.506824+0000) 2019-10-13T11:57:36.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:36.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:36.440907+0000) 2019-10-13T11:57:37.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:37.884+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.891105+0000) 2019-10-13T11:57:37.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:37.888+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.891388+0000) 2019-10-13T11:57:37.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:37.888+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.891825+0000) 2019-10-13T11:57:37.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:37.888+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.892094+0000) 2019-10-13T11:57:37.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:37.889+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.892913+0000) 2019-10-13T11:57:37.894 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:37.888+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.894119+0000) 2019-10-13T11:57:37.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:37.892+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.895394+0000) 2019-10-13T11:57:37.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:37.892+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:37.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:37.897+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.902051+0000) 2019-10-13T11:57:38.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:38.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.705852+0000) 2019-10-13T11:57:38.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:38.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.710102+0000) 2019-10-13T11:57:38.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:38.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.722275+0000) 2019-10-13T11:57:39.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:39.069+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.072686+0000) 2019-10-13T11:57:39.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:39.937+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.941281+0000) 2019-10-13T11:57:42.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:42.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.196918+0000) 2019-10-13T11:57:42.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:42.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.901897+0000) 2019-10-13T11:57:42.903 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:42.901+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.902715+0000) 2019-10-13T11:57:42.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:42.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.903576+0000) 2019-10-13T11:57:42.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:42.900+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.904174+0000) 2019-10-13T11:57:42.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:42.900+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.904500+0000) 2019-10-13T11:57:42.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:42.900+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.905464+0000) 2019-10-13T11:57:42.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:42.900+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.905737+0000) 2019-10-13T11:57:42.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:42.905+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:42.907997+0000) 2019-10-13T11:57:42.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:42.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:45.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:45.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.507013+0000) 2019-10-13T11:57:46.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:46.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:46.441104+0000) 2019-10-13T11:57:47.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:47.908+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.911255+0000) 2019-10-13T11:57:47.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:47.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.912162+0000) 2019-10-13T11:57:47.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:47.908+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.912229+0000) 2019-10-13T11:57:47.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:47.908+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.913231+0000) 2019-10-13T11:57:47.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:47.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.913564+0000) 2019-10-13T11:57:47.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:47.908+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.913828+0000) 2019-10-13T11:57:47.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:47.908+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.914421+0000) 2019-10-13T11:57:47.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:47.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:47.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:47.916+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.920883+0000) 2019-10-13T11:57:48.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:48.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.706041+0000) 2019-10-13T11:57:48.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:48.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.710278+0000) 2019-10-13T11:57:48.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:48.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.722463+0000) 2019-10-13T11:57:49.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:49.076+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.079274+0000) 2019-10-13T11:57:49.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:49.936+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.941499+0000) 2019-10-13T11:57:51.426 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:57:51.426 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:51.430 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:57:51.430 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:52.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:52.192+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.197144+0000) 2019-10-13T11:57:52.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:52.916+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.921714+0000) 2019-10-13T11:57:52.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:52.916+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.922506+0000) 2019-10-13T11:57:52.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:52.916+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.923072+0000) 2019-10-13T11:57:52.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:52.920+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.923349+0000) 2019-10-13T11:57:52.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:52.920+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.923502+0000) 2019-10-13T11:57:52.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:52.920+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.924120+0000) 2019-10-13T11:57:52.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:52.920+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.925127+0000) 2019-10-13T11:57:52.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:52.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:52.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:52.928+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:52.931501+0000) 2019-10-13T11:57:55.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:55.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.507216+0000) 2019-10-13T11:57:56.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:56.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:56.441299+0000) 2019-10-13T11:57:57.932 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:57.928+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.931627+0000) 2019-10-13T11:57:57.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:57.928+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.931717+0000) 2019-10-13T11:57:57.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:57.928+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.932181+0000) 2019-10-13T11:57:57.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:57:57.928+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.932619+0000) 2019-10-13T11:57:57.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:57.928+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.933533+0000) 2019-10-13T11:57:57.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:57:57.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.934067+0000) 2019-10-13T11:57:57.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:57:57.932+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.935661+0000) 2019-10-13T11:57:57.939 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:57:57.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:57:57.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:57.940+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.943764+0000) 2019-10-13T11:57:58.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:57:58.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.706218+0000) 2019-10-13T11:57:58.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:57:58.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.710396+0000) 2019-10-13T11:57:58.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:57:58.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.722638+0000) 2019-10-13T11:57:59.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:57:59.084+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.085948+0000) 2019-10-13T11:57:59.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:57:59.936+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.941688+0000) 2019-10-13T11:58:02.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:02.195+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.197347+0000) 2019-10-13T11:58:02.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:02.935+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.941990+0000) 2019-10-13T11:58:02.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:02.935+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.942826+0000) 2019-10-13T11:58:02.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:02.939+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.943356+0000) 2019-10-13T11:58:02.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:02.939+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.944424+0000) 2019-10-13T11:58:02.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:02.940+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.944678+0000) 2019-10-13T11:58:02.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:02.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.945194+0000) 2019-10-13T11:58:02.946 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:02.939+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.946094+0000) 2019-10-13T11:58:02.949 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:02.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:02.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:02.948+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:02.950687+0000) 2019-10-13T11:58:05.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:05.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.507492+0000) 2019-10-13T11:58:06.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:06.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:06.441495+0000) 2019-10-13T11:58:07.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:07.947+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.952365+0000) 2019-10-13T11:58:07.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:07.948+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.952595+0000) 2019-10-13T11:58:07.954 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:07.947+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.952972+0000) 2019-10-13T11:58:07.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:07.947+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.953443+0000) 2019-10-13T11:58:07.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:07.947+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.954289+0000) 2019-10-13T11:58:07.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:07.947+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.954619+0000) 2019-10-13T11:58:07.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:07.947+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.955010+0000) 2019-10-13T11:58:07.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:07.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:07.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:07.956+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.959995+0000) 2019-10-13T11:58:08.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:08.699+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.706409+0000) 2019-10-13T11:58:08.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:08.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.710576+0000) 2019-10-13T11:58:08.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:08.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.722809+0000) 2019-10-13T11:58:09.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:09.092+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.097725+0000) 2019-10-13T11:58:09.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:09.936+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.941854+0000) 2019-10-13T11:58:12.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:12.195+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.197553+0000) 2019-10-13T11:58:12.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:12.960+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.962569+0000) 2019-10-13T11:58:12.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:12.959+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.963630+0000) 2019-10-13T11:58:12.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:12.959+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.964159+0000) 2019-10-13T11:58:12.965 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:12.959+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.964536+0000) 2019-10-13T11:58:12.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:12.959+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.965363+0000) 2019-10-13T11:58:12.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:12.959+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.966270+0000) 2019-10-13T11:58:12.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:12.959+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.966525+0000) 2019-10-13T11:58:12.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:12.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:12.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:12.968+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:12.973079+0000) 2019-10-13T11:58:15.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:15.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.507714+0000) 2019-10-13T11:58:16.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:16.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:16.441689+0000) 2019-10-13T11:58:17.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:17.967+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.973054+0000) 2019-10-13T11:58:17.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:17.967+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.973659+0000) 2019-10-13T11:58:17.975 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:17.967+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.973752+0000) 2019-10-13T11:58:17.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:17.967+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.974534+0000) 2019-10-13T11:58:17.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:17.972+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.974633+0000) 2019-10-13T11:58:17.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:17.971+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.975490+0000) 2019-10-13T11:58:17.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:17.971+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.976250+0000) 2019-10-13T11:58:17.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:17.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136647 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:17.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:17.980+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.983681+0000) 2019-10-13T11:58:18.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:18.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.706618+0000) 2019-10-13T11:58:18.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:18.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.710766+0000) 2019-10-13T11:58:18.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:18.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.722989+0000) 2019-10-13T11:58:19.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:19.100+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.104202+0000) 2019-10-13T11:58:19.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:19.936+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.942057+0000) 2019-10-13T11:58:21.543 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:58:21.543 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:21.547 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:58:21.547 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:22.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:22.195+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.197755+0000) 2019-10-13T11:58:22.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:22.975+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.982191+0000) 2019-10-13T11:58:22.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:22.975+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.982456+0000) 2019-10-13T11:58:22.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:22.975+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.982909+0000) 2019-10-13T11:58:22.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:22.979+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.983388+0000) 2019-10-13T11:58:22.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:22.980+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.983866+0000) 2019-10-13T11:58:22.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:22.979+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.984405+0000) 2019-10-13T11:58:22.986 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:22.979+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.985316+0000) 2019-10-13T11:58:22.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:22.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:22.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:22.984+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:22.990077+0000) 2019-10-13T11:58:25.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:25.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.507918+0000) 2019-10-13T11:58:26.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:26.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:26.441900+0000) 2019-10-13T11:58:27.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:27.987+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.991136+0000) 2019-10-13T11:58:27.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:27.987+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.992252+0000) 2019-10-13T11:58:27.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:27.987+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.992792+0000) 2019-10-13T11:58:27.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:27.987+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.993518+0000) 2019-10-13T11:58:27.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:27.987+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.993833+0000) 2019-10-13T11:58:27.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:27.988+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.994091+0000) 2019-10-13T11:58:27.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:27.987+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.994599+0000) 2019-10-13T11:58:27.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:27.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:28.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:27.996+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.999984+0000) 2019-10-13T11:58:28.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:28.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.706792+0000) 2019-10-13T11:58:28.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:28.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.710938+0000) 2019-10-13T11:58:28.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:28.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.723200+0000) 2019-10-13T11:58:29.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:29.108+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.110871+0000) 2019-10-13T11:58:29.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:29.940+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.942302+0000) 2019-10-13T11:58:32.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:32.195+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:32.197958+0000) 2019-10-13T11:58:33.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:32.999+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.001700+0000) 2019-10-13T11:58:33.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:32.999+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.002019+0000) 2019-10-13T11:58:33.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:32.999+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.002310+0000) 2019-10-13T11:58:33.004 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:32.999+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.003111+0000) 2019-10-13T11:58:33.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:32.999+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.003256+0000) 2019-10-13T11:58:33.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:33.000+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.003555+0000) 2019-10-13T11:58:33.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:33.003+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.004488+0000) 2019-10-13T11:58:33.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:33.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:33.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:33.008+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.011201+0000) 2019-10-13T11:58:35.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:35.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.508159+0000) 2019-10-13T11:58:36.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:36.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:36.442094+0000) 2019-10-13T11:58:38.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:38.011+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.011409+0000) 2019-10-13T11:58:38.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:38.011+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.012333+0000) 2019-10-13T11:58:38.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:38.011+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.013651+0000) 2019-10-13T11:58:38.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:38.011+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.014615+0000) 2019-10-13T11:58:38.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:38.011+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.014924+0000) 2019-10-13T11:58:38.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:38.015+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.015332+0000) 2019-10-13T11:58:38.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:38.011+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.016096+0000) 2019-10-13T11:58:38.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:38.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:38.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:38.015+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.019400+0000) 2019-10-13T11:58:38.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:38.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.706970+0000) 2019-10-13T11:58:38.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:38.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.711110+0000) 2019-10-13T11:58:38.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:38.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.723430+0000) 2019-10-13T11:58:39.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:39.115+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.117446+0000) 2019-10-13T11:58:39.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:39.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.942492+0000) 2019-10-13T11:58:42.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:42.194+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:42.198176+0000) 2019-10-13T11:58:43.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:43.019+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.022354+0000) 2019-10-13T11:58:43.024 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:43.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.023492+0000) 2019-10-13T11:58:43.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:43.022+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.023884+0000) 2019-10-13T11:58:43.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:43.022+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.024281+0000) 2019-10-13T11:58:43.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:43.022+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.024555+0000) 2019-10-13T11:58:43.026 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:43.022+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.024913+0000) 2019-10-13T11:58:43.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:43.022+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.025503+0000) 2019-10-13T11:58:43.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:43.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:43.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:43.027+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.031630+0000) 2019-10-13T11:58:45.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:45.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.508375+0000) 2019-10-13T11:58:46.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:46.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:46.442288+0000) 2019-10-13T11:58:48.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:48.030+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.033907+0000) 2019-10-13T11:58:48.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:48.030+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.034255+0000) 2019-10-13T11:58:48.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:48.031+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.035416+0000) 2019-10-13T11:58:48.037 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:48.034+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.035908+0000) 2019-10-13T11:58:48.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:48.034+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.036323+0000) 2019-10-13T11:58:48.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:48.034+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.036754+0000) 2019-10-13T11:58:48.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:48.034+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.037086+0000) 2019-10-13T11:58:48.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:48.038+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:48.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:48.039+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.043685+0000) 2019-10-13T11:58:48.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:48.702+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.707145+0000) 2019-10-13T11:58:48.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:48.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.711289+0000) 2019-10-13T11:58:48.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:48.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.723603+0000) 2019-10-13T11:58:49.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:49.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.123926+0000) 2019-10-13T11:58:49.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:49.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.942686+0000) 2019-10-13T11:58:51.791 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:58:51.791 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:51.795 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:58:51.795 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:52.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:52.194+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:52.198381+0000) 2019-10-13T11:58:53.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:53.039+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.044801+0000) 2019-10-13T11:58:53.047 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:53.042+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.045448+0000) 2019-10-13T11:58:53.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:53.042+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.045616+0000) 2019-10-13T11:58:53.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:53.042+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.046402+0000) 2019-10-13T11:58:53.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:53.042+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.046643+0000) 2019-10-13T11:58:53.048 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:53.042+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.047087+0000) 2019-10-13T11:58:53.049 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:53.046+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.047447+0000) 2019-10-13T11:58:53.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:53.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:53.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:53.051+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.054397+0000) 2019-10-13T11:58:55.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:55.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.508577+0000) 2019-10-13T11:58:56.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:56.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:56.442476+0000) 2019-10-13T11:58:58.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:58.054+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.055701+0000) 2019-10-13T11:58:58.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:58.051+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.055885+0000) 2019-10-13T11:58:58.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:58:58.054+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.056847+0000) 2019-10-13T11:58:58.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:58:58.054+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.057363+0000) 2019-10-13T11:58:58.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:58:58.054+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.057778+0000) 2019-10-13T11:58:58.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:58.054+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.058692+0000) 2019-10-13T11:58:58.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:58.054+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.059172+0000) 2019-10-13T11:58:58.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:58:58.058+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:58:58.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:58.059+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.063822+0000) 2019-10-13T11:58:58.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:58:58.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.707348+0000) 2019-10-13T11:58:58.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:58:58.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.711503+0000) 2019-10-13T11:58:58.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:58:58.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.723812+0000) 2019-10-13T11:58:59.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:58:59.127+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.130512+0000) 2019-10-13T11:58:59.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:58:59.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.942879+0000) 2019-10-13T11:59:02.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:02.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:02.198588+0000) 2019-10-13T11:59:03.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:03.062+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.066002+0000) 2019-10-13T11:59:03.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:03.062+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.066717+0000) 2019-10-13T11:59:03.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:03.066+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.067468+0000) 2019-10-13T11:59:03.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:03.063+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.067728+0000) 2019-10-13T11:59:03.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:03.066+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.068325+0000) 2019-10-13T11:59:03.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:03.066+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.068771+0000) 2019-10-13T11:59:03.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:03.066+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.069208+0000) 2019-10-13T11:59:03.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:03.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:03.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:03.067+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.072741+0000) 2019-10-13T11:59:05.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:05.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.508806+0000) 2019-10-13T11:59:06.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:06.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:06.442687+0000) 2019-10-13T11:59:08.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:08.071+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.075233+0000) 2019-10-13T11:59:08.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:08.074+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.075867+0000) 2019-10-13T11:59:08.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:08.074+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.076166+0000) 2019-10-13T11:59:08.078 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:08.074+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.076558+0000) 2019-10-13T11:59:08.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:08.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.076927+0000) 2019-10-13T11:59:08.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:08.074+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.078327+0000) 2019-10-13T11:59:08.079 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:08.074+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.078435+0000) 2019-10-13T11:59:08.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:08.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 136975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:08.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:08.079+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.085320+0000) 2019-10-13T11:59:08.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:08.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.707518+0000) 2019-10-13T11:59:08.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:08.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.711634+0000) 2019-10-13T11:59:08.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:08.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.723986+0000) 2019-10-13T11:59:09.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:09.135+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.136985+0000) 2019-10-13T11:59:09.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:09.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.943092+0000) 2019-10-13T11:59:12.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:12.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:12.198785+0000) 2019-10-13T11:59:13.087 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:13.082+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.086177+0000) 2019-10-13T11:59:13.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:13.082+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.086943+0000) 2019-10-13T11:59:13.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:13.083+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.087037+0000) 2019-10-13T11:59:13.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:13.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.087600+0000) 2019-10-13T11:59:13.089 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:13.086+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.088225+0000) 2019-10-13T11:59:13.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:13.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.088301+0000) 2019-10-13T11:59:13.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:13.086+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.088871+0000) 2019-10-13T11:59:13.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:13.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:13.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:13.091+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.097405+0000) 2019-10-13T11:59:15.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:15.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.509004+0000) 2019-10-13T11:59:16.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:16.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:16.442916+0000) 2019-10-13T11:59:18.096 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:18.094+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.095659+0000) 2019-10-13T11:59:18.097 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:18.094+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.096452+0000) 2019-10-13T11:59:18.097 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:18.094+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.097195+0000) 2019-10-13T11:59:18.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:18.091+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.097420+0000) 2019-10-13T11:59:18.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:18.094+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.097775+0000) 2019-10-13T11:59:18.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:18.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.098412+0000) 2019-10-13T11:59:18.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:18.098+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.099442+0000) 2019-10-13T11:59:18.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:18.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:18.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:18.099+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.103123+0000) 2019-10-13T11:59:18.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:18.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.707701+0000) 2019-10-13T11:59:18.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:18.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.711824+0000) 2019-10-13T11:59:18.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:18.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.724176+0000) 2019-10-13T11:59:19.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:19.143+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.143533+0000) 2019-10-13T11:59:19.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:19.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.943243+0000) 2019-10-13T11:59:21.847 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:59:21.847 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:21.851 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:59:21.851 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:22.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:22.194+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:22.198988+0000) 2019-10-13T11:59:23.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:23.103+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.106280+0000) 2019-10-13T11:59:23.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:23.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.107258+0000) 2019-10-13T11:59:23.108 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:23.106+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.107624+0000) 2019-10-13T11:59:23.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:23.106+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.108499+0000) 2019-10-13T11:59:23.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:23.106+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.108490+0000) 2019-10-13T11:59:23.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:23.106+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.107987+0000) 2019-10-13T11:59:23.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:23.106+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.109362+0000) 2019-10-13T11:59:23.112 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:23.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:23.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:23.107+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.112943+0000) 2019-10-13T11:59:25.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:25.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.509225+0000) 2019-10-13T11:59:26.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:26.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:26.443170+0000) 2019-10-13T11:59:28.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:28.109+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.115229+0000) 2019-10-13T11:59:28.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:28.113+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.115578+0000) 2019-10-13T11:59:28.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:28.113+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.116025+0000) 2019-10-13T11:59:28.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:28.111+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.117097+0000) 2019-10-13T11:59:28.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:28.113+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.117610+0000) 2019-10-13T11:59:28.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:28.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.118373+0000) 2019-10-13T11:59:28.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:28.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.118873+0000) 2019-10-13T11:59:28.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:28.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:28.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:28.119+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.123985+0000) 2019-10-13T11:59:28.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:28.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.707893+0000) 2019-10-13T11:59:28.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:28.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.711987+0000) 2019-10-13T11:59:28.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:28.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.724383+0000) 2019-10-13T11:59:29.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:29.147+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.150348+0000) 2019-10-13T11:59:29.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:29.939+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.943472+0000) 2019-10-13T11:59:32.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:32.197+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:32.199217+0000) 2019-10-13T11:59:33.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:33.121+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.125976+0000) 2019-10-13T11:59:33.127 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:33.121+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.126483+0000) 2019-10-13T11:59:33.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:33.121+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.127062+0000) 2019-10-13T11:59:33.128 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:33.125+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.127669+0000) 2019-10-13T11:59:33.129 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:33.125+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.128753+0000) 2019-10-13T11:59:33.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:33.122+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.128773+0000) 2019-10-13T11:59:33.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:33.125+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.129233+0000) 2019-10-13T11:59:33.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:33.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:33.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:33.130+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.135238+0000) 2019-10-13T11:59:35.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:35.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.509460+0000) 2019-10-13T11:59:36.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:36.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:36.443400+0000) 2019-10-13T11:59:38.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:38.133+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.136584+0000) 2019-10-13T11:59:38.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:38.133+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.136982+0000) 2019-10-13T11:59:38.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:38.133+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.137572+0000) 2019-10-13T11:59:38.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:38.130+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.137662+0000) 2019-10-13T11:59:38.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:38.133+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.138577+0000) 2019-10-13T11:59:38.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:38.137+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.139366+0000) 2019-10-13T11:59:38.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:38.137+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.139634+0000) 2019-10-13T11:59:38.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:38.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:38.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:38.138+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.145132+0000) 2019-10-13T11:59:38.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:38.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.708069+0000) 2019-10-13T11:59:38.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:38.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.712164+0000) 2019-10-13T11:59:38.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:38.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.724558+0000) 2019-10-13T11:59:39.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:39.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.161972+0000) 2019-10-13T11:59:39.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:39.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.943680+0000) 2019-10-13T11:59:42.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:42.197+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:42.199415+0000) 2019-10-13T11:59:43.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:43.141+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.146501+0000) 2019-10-13T11:59:43.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:43.145+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.147522+0000) 2019-10-13T11:59:43.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:43.145+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.148059+0000) 2019-10-13T11:59:43.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:43.142+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.148297+0000) 2019-10-13T11:59:43.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:43.145+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.148840+0000) 2019-10-13T11:59:43.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:43.145+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.149174+0000) 2019-10-13T11:59:43.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:43.145+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.149733+0000) 2019-10-13T11:59:43.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:43.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:43.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:43.146+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.154086+0000) 2019-10-13T11:59:45.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:45.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.509660+0000) 2019-10-13T11:59:46.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:46.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:46.443588+0000) 2019-10-13T11:59:48.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:48.153+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.156075+0000) 2019-10-13T11:59:48.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:48.153+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.156784+0000) 2019-10-13T11:59:48.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:48.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.157413+0000) 2019-10-13T11:59:48.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:48.150+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.157897+0000) 2019-10-13T11:59:48.159 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:48.153+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.158493+0000) 2019-10-13T11:59:48.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:48.153+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.159102+0000) 2019-10-13T11:59:48.160 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:48.157+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.159399+0000) 2019-10-13T11:59:48.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:48.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:48.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:48.158+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.164367+0000) 2019-10-13T11:59:48.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:48.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.708242+0000) 2019-10-13T11:59:48.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:48.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.712336+0000) 2019-10-13T11:59:48.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:48.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.724783+0000) 2019-10-13T11:59:49.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:49.166+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.168396+0000) 2019-10-13T11:59:49.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:49.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.943906+0000) 2019-10-13T11:59:51.989 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T11:59:51.989 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:51.993 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T11:59:51.993 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:52.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:52.197+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:52.199643+0000) 2019-10-13T11:59:53.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:53.161+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.166916+0000) 2019-10-13T11:59:53.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:53.161+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.167129+0000) 2019-10-13T11:59:53.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:53.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.167433+0000) 2019-10-13T11:59:53.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:53.165+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.167954+0000) 2019-10-13T11:59:53.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:53.165+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.169276+0000) 2019-10-13T11:59:53.170 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:53.162+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.169495+0000) 2019-10-13T11:59:53.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:53.165+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.170074+0000) 2019-10-13T11:59:53.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:53.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:53.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:53.170+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.174221+0000) 2019-10-13T11:59:55.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:55.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.509865+0000) 2019-10-13T11:59:56.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:56.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:56.443789+0000) 2019-10-13T11:59:58.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T11:59:58.173+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.177101+0000) 2019-10-13T11:59:58.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:58.173+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.177872+0000) 2019-10-13T11:59:58.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:58.173+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.178216+0000) 2019-10-13T11:59:58.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:58.174+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.178287+0000) 2019-10-13T11:59:58.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T11:59:58.173+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.179092+0000) 2019-10-13T11:59:58.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T11:59:58.177+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.180058+0000) 2019-10-13T11:59:58.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:58.177+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.180511+0000) 2019-10-13T11:59:58.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T11:59:58.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T11:59:58.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:58.178+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.183645+0000) 2019-10-13T11:59:58.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T11:59:58.705+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.708432+0000) 2019-10-13T11:59:58.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T11:59:58.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.712512+0000) 2019-10-13T11:59:58.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T11:59:58.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.724973+0000) 2019-10-13T11:59:59.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T11:59:59.174+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.175060+0000) 2019-10-13T11:59:59.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T11:59:59.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.944103+0000) 2019-10-13T12:00:02.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:02.197+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:02.199844+0000) 2019-10-13T12:00:03.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:03.181+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.187146+0000) 2019-10-13T12:00:03.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:03.182+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.188166+0000) 2019-10-13T12:00:03.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:03.185+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.187793+0000) 2019-10-13T12:00:03.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:03.185+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.188611+0000) 2019-10-13T12:00:03.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:03.185+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.189202+0000) 2019-10-13T12:00:03.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:03.185+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.188898+0000) 2019-10-13T12:00:03.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:03.185+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.189634+0000) 2019-10-13T12:00:03.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:03.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:03.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:03.190+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.195431+0000) 2019-10-13T12:00:05.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:05.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.510066+0000) 2019-10-13T12:00:06.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:06.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:06.444006+0000) 2019-10-13T12:00:08.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:08.192+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.197596+0000) 2019-10-13T12:00:08.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:08.192+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.197961+0000) 2019-10-13T12:00:08.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:08.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.198387+0000) 2019-10-13T12:00:08.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:08.196+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.199493+0000) 2019-10-13T12:00:08.200 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:08.196+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.200116+0000) 2019-10-13T12:00:08.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:08.196+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.201020+0000) 2019-10-13T12:00:08.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:08.194+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.201371+0000) 2019-10-13T12:00:08.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:08.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:08.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:08.202+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.207024+0000) 2019-10-13T12:00:08.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:08.704+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.708617+0000) 2019-10-13T12:00:08.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:08.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.712685+0000) 2019-10-13T12:00:08.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:08.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.725178+0000) 2019-10-13T12:00:09.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:09.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.181304+0000) 2019-10-13T12:00:09.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:09.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.944295+0000) 2019-10-13T12:00:12.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:12.196+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:12.200043+0000) 2019-10-13T12:00:13.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:13.200+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.206411+0000) 2019-10-13T12:00:13.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:13.204+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.207371+0000) 2019-10-13T12:00:13.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:13.202+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.207553+0000) 2019-10-13T12:00:13.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:13.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.208190+0000) 2019-10-13T12:00:13.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:13.204+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.208507+0000) 2019-10-13T12:00:13.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:13.204+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.209054+0000) 2019-10-13T12:00:13.210 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:13.204+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.210028+0000) 2019-10-13T12:00:13.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:13.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:13.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:13.210+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.216813+0000) 2019-10-13T12:00:15.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:15.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.510263+0000) 2019-10-13T12:00:16.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:16.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:16.444217+0000) 2019-10-13T12:00:18.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:18.212+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.217307+0000) 2019-10-13T12:00:18.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:18.212+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.218075+0000) 2019-10-13T12:00:18.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:18.212+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.218600+0000) 2019-10-13T12:00:18.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:18.212+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.218871+0000) 2019-10-13T12:00:18.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:18.216+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.219661+0000) 2019-10-13T12:00:18.220 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:18.216+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.220137+0000) 2019-10-13T12:00:18.221 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:18.214+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.220494+0000) 2019-10-13T12:00:18.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:18.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:18.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:18.218+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.223949+0000) 2019-10-13T12:00:18.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:18.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.708830+0000) 2019-10-13T12:00:18.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:18.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.712866+0000) 2019-10-13T12:00:18.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:18.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.725350+0000) 2019-10-13T12:00:19.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:19.186+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.187826+0000) 2019-10-13T12:00:19.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:19.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.944498+0000) 2019-10-13T12:00:22.103 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:00:22.103 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:22.107 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:00:22.107 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:22.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:22.196+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:22.200270+0000) 2019-10-13T12:00:23.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:23.224+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.227667+0000) 2019-10-13T12:00:23.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:23.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.228178+0000) 2019-10-13T12:00:23.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:23.224+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.228652+0000) 2019-10-13T12:00:23.230 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:23.226+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.228851+0000) 2019-10-13T12:00:23.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:23.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.229418+0000) 2019-10-13T12:00:23.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:23.224+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.230918+0000) 2019-10-13T12:00:23.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:23.228+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.231403+0000) 2019-10-13T12:00:23.234 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:23.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137472 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:23.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:23.238+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.238691+0000) 2019-10-13T12:00:25.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:25.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.510462+0000) 2019-10-13T12:00:26.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:26.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:26.444406+0000) 2019-10-13T12:00:28.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:28.233+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.237068+0000) 2019-10-13T12:00:28.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:28.232+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.238231+0000) 2019-10-13T12:00:28.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:28.232+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.239090+0000) 2019-10-13T12:00:28.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:28.236+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.239378+0000) 2019-10-13T12:00:28.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:28.236+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.239672+0000) 2019-10-13T12:00:28.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:28.236+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.239981+0000) 2019-10-13T12:00:28.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:28.236+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.241126+0000) 2019-10-13T12:00:28.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:28.241+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.243833+0000) 2019-10-13T12:00:28.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:28.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:28.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:28.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.709009+0000) 2019-10-13T12:00:28.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:28.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.713034+0000) 2019-10-13T12:00:28.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:28.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.725519+0000) 2019-10-13T12:00:29.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:29.193+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.194181+0000) 2019-10-13T12:00:29.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:29.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.944689+0000) 2019-10-13T12:00:32.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:32.200+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:32.200481+0000) 2019-10-13T12:00:33.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:33.244+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.247627+0000) 2019-10-13T12:00:33.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:33.244+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.249021+0000) 2019-10-13T12:00:33.250 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:33.244+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.249624+0000) 2019-10-13T12:00:33.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:33.249+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.250287+0000) 2019-10-13T12:00:33.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:33.244+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.250787+0000) 2019-10-13T12:00:33.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:33.244+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.251162+0000) 2019-10-13T12:00:33.253 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:33.248+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.251773+0000) 2019-10-13T12:00:33.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:33.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:33.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:33.253+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.255397+0000) 2019-10-13T12:00:35.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:35.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.510663+0000) 2019-10-13T12:00:36.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:36.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:36.444594+0000) 2019-10-13T12:00:38.259 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:38.252+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.258050+0000) 2019-10-13T12:00:38.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:38.257+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.258279+0000) 2019-10-13T12:00:38.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:38.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.258799+0000) 2019-10-13T12:00:38.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:38.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.259539+0000) 2019-10-13T12:00:38.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:38.256+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.259759+0000) 2019-10-13T12:00:38.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:38.256+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.260203+0000) 2019-10-13T12:00:38.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:38.256+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.260973+0000) 2019-10-13T12:00:38.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:38.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:38.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:38.261+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.265218+0000) 2019-10-13T12:00:38.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:38.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.709216+0000) 2019-10-13T12:00:38.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:38.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.713220+0000) 2019-10-13T12:00:38.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:38.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.725695+0000) 2019-10-13T12:00:39.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:39.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.200546+0000) 2019-10-13T12:00:39.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:39.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.944884+0000) 2019-10-13T12:00:42.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:42.200+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:42.200711+0000) 2019-10-13T12:00:43.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:43.265+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.268185+0000) 2019-10-13T12:00:43.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:43.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.268743+0000) 2019-10-13T12:00:43.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:43.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.269312+0000) 2019-10-13T12:00:43.270 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:43.264+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.269530+0000) 2019-10-13T12:00:43.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:43.264+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.269827+0000) 2019-10-13T12:00:43.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:43.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.271097+0000) 2019-10-13T12:00:43.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:43.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.271367+0000) 2019-10-13T12:00:43.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:43.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:43.278 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:43.273+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.277053+0000) 2019-10-13T12:00:45.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:45.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.510860+0000) 2019-10-13T12:00:46.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:46.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:46.444858+0000) 2019-10-13T12:00:48.278 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:48.272+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.277793+0000) 2019-10-13T12:00:48.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:48.276+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.279377+0000) 2019-10-13T12:00:48.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:48.277+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.279811+0000) 2019-10-13T12:00:48.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:48.276+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.280310+0000) 2019-10-13T12:00:48.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:48.276+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.281111+0000) 2019-10-13T12:00:48.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:48.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.281065+0000) 2019-10-13T12:00:48.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:48.276+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.281492+0000) 2019-10-13T12:00:48.285 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:48.280+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:48.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:48.285+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.287297+0000) 2019-10-13T12:00:48.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:48.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.709395+0000) 2019-10-13T12:00:48.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:48.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.713398+0000) 2019-10-13T12:00:48.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:48.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.725873+0000) 2019-10-13T12:00:49.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:49.205+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.206984+0000) 2019-10-13T12:00:49.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:49.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.945086+0000) 2019-10-13T12:00:52.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:52.199+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:52.200922+0000) 2019-10-13T12:00:52.214 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:00:52.214 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:52.218 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:00:52.218 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:53.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:53.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.288494+0000) 2019-10-13T12:00:53.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:53.283+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.288709+0000) 2019-10-13T12:00:53.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:53.285+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.289660+0000) 2019-10-13T12:00:53.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:53.283+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.290595+0000) 2019-10-13T12:00:53.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:53.283+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.291064+0000) 2019-10-13T12:00:53.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:53.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.291141+0000) 2019-10-13T12:00:53.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:53.287+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.291758+0000) 2019-10-13T12:00:53.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:53.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:53.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:53.293+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.296484+0000) 2019-10-13T12:00:55.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:55.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.511067+0000) 2019-10-13T12:00:56.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:56.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:56.445057+0000) 2019-10-13T12:00:58.299 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:58.291+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.298045+0000) 2019-10-13T12:00:58.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:58.297+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.299565+0000) 2019-10-13T12:00:58.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:00:58.295+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.300260+0000) 2019-10-13T12:00:58.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:58.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.300896+0000) 2019-10-13T12:00:58.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:00:58.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.300670+0000) 2019-10-13T12:00:58.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:00:58.295+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.301433+0000) 2019-10-13T12:00:58.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:58.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.301920+0000) 2019-10-13T12:00:58.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:00:58.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:00:58.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:58.309+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.311130+0000) 2019-10-13T12:00:58.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:00:58.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.709541+0000) 2019-10-13T12:00:58.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:00:58.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.713603+0000) 2019-10-13T12:00:58.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:00:58.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.726041+0000) 2019-10-13T12:00:59.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:00:59.209+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.213433+0000) 2019-10-13T12:00:59.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:00:59.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.945316+0000) 2019-10-13T12:01:02.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:02.199+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:02.201154+0000) 2019-10-13T12:01:03.310 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:03.303+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.309348+0000) 2019-10-13T12:01:03.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:03.305+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.309677+0000) 2019-10-13T12:01:03.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:03.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.310156+0000) 2019-10-13T12:01:03.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:03.303+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.310497+0000) 2019-10-13T12:01:03.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:03.303+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.311234+0000) 2019-10-13T12:01:03.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:03.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.311519+0000) 2019-10-13T12:01:03.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:03.307+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.311689+0000) 2019-10-13T12:01:03.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:03.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:03.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:03.317+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.318325+0000) 2019-10-13T12:01:05.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:05.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.511253+0000) 2019-10-13T12:01:06.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:06.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:06.445257+0000) 2019-10-13T12:01:08.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:08.317+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.319927+0000) 2019-10-13T12:01:08.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:08.315+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.319790+0000) 2019-10-13T12:01:08.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:08.315+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.320531+0000) 2019-10-13T12:01:08.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:08.315+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.320868+0000) 2019-10-13T12:01:08.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:08.315+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.321369+0000) 2019-10-13T12:01:08.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:08.315+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.321658+0000) 2019-10-13T12:01:08.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:08.315+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.322328+0000) 2019-10-13T12:01:08.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:08.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:08.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:08.325+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.329222+0000) 2019-10-13T12:01:08.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:08.703+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.709730+0000) 2019-10-13T12:01:08.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:08.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.713772+0000) 2019-10-13T12:01:08.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:08.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.726218+0000) 2019-10-13T12:01:09.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:09.217+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.219858+0000) 2019-10-13T12:01:09.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:09.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.945506+0000) 2019-10-13T12:01:12.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:12.199+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:12.201416+0000) 2019-10-13T12:01:13.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:13.323+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.330185+0000) 2019-10-13T12:01:13.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:13.323+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.330051+0000) 2019-10-13T12:01:13.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:13.329+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.330461+0000) 2019-10-13T12:01:13.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:13.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.331227+0000) 2019-10-13T12:01:13.332 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:13.327+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.331835+0000) 2019-10-13T12:01:13.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:13.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.332396+0000) 2019-10-13T12:01:13.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:13.327+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.333317+0000) 2019-10-13T12:01:13.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:13.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:13.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:13.337+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.339250+0000) 2019-10-13T12:01:15.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:15.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.511519+0000) 2019-10-13T12:01:16.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:16.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:16.445460+0000) 2019-10-13T12:01:18.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:18.335+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.339490+0000) 2019-10-13T12:01:18.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:18.335+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.339868+0000) 2019-10-13T12:01:18.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:18.337+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.340006+0000) 2019-10-13T12:01:18.342 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:18.335+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.340686+0000) 2019-10-13T12:01:18.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:18.335+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.341567+0000) 2019-10-13T12:01:18.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:18.335+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.342126+0000) 2019-10-13T12:01:18.343 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:18.335+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.342592+0000) 2019-10-13T12:01:18.345 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:18.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:18.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:18.341+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.345533+0000) 2019-10-13T12:01:18.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:18.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.709940+0000) 2019-10-13T12:01:18.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:18.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.713945+0000) 2019-10-13T12:01:18.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:18.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.726393+0000) 2019-10-13T12:01:19.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:19.225+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.226246+0000) 2019-10-13T12:01:19.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:19.941+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.945730+0000) 2019-10-13T12:01:22.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:22.199+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:22.201641+0000) 2019-10-13T12:01:22.347 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:01:22.348 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:22.351 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:01:22.351 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:23.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:23.347+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.348758+0000) 2019-10-13T12:01:23.350 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:23.347+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.349093+0000) 2019-10-13T12:01:23.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:23.347+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.349875+0000) 2019-10-13T12:01:23.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:23.348+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.350218+0000) 2019-10-13T12:01:23.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:23.347+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.351182+0000) 2019-10-13T12:01:23.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:23.351+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.351511+0000) 2019-10-13T12:01:23.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:23.351+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.352141+0000) 2019-10-13T12:01:23.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:23.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137867 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:23.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:23.356+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.358418+0000) 2019-10-13T12:01:25.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:25.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.511728+0000) 2019-10-13T12:01:26.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:26.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:26.445654+0000) 2019-10-13T12:01:28.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:28.355+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.358380+0000) 2019-10-13T12:01:28.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:28.359+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.359443+0000) 2019-10-13T12:01:28.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:28.359+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.359747+0000) 2019-10-13T12:01:28.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:28.359+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.360203+0000) 2019-10-13T12:01:28.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:28.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.360647+0000) 2019-10-13T12:01:28.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:28.359+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.361364+0000) 2019-10-13T12:01:28.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:28.359+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.362440+0000) 2019-10-13T12:01:28.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:28.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:28.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:28.368+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.371589+0000) 2019-10-13T12:01:28.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:28.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.710118+0000) 2019-10-13T12:01:28.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:28.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.714125+0000) 2019-10-13T12:01:28.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:28.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.726570+0000) 2019-10-13T12:01:29.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:29.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.232914+0000) 2019-10-13T12:01:29.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:29.940+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.945937+0000) 2019-10-13T12:01:32.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:32.199+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:32.201834+0000) 2019-10-13T12:01:33.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:33.367+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.368712+0000) 2019-10-13T12:01:33.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:33.367+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.369512+0000) 2019-10-13T12:01:33.371 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:33.367+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.370925+0000) 2019-10-13T12:01:33.372 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:33.368+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.370968+0000) 2019-10-13T12:01:33.372 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:33.371+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.371470+0000) 2019-10-13T12:01:33.373 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:33.371+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.371583+0000) 2019-10-13T12:01:33.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:33.371+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.372148+0000) 2019-10-13T12:01:33.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:33.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:33.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:33.372+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.375865+0000) 2019-10-13T12:01:35.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:35.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.511957+0000) 2019-10-13T12:01:36.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:36.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:36.445873+0000) 2019-10-13T12:01:38.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:38.374+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.379003+0000) 2019-10-13T12:01:38.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:38.378+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.379735+0000) 2019-10-13T12:01:38.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:38.378+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.380310+0000) 2019-10-13T12:01:38.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:38.376+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.380494+0000) 2019-10-13T12:01:38.383 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:38.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.380859+0000) 2019-10-13T12:01:38.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:38.378+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.381366+0000) 2019-10-13T12:01:38.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:38.378+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.382118+0000) 2019-10-13T12:01:38.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:38.384+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.387093+0000) 2019-10-13T12:01:38.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:38.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:38.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:38.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.710316+0000) 2019-10-13T12:01:38.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:38.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.714347+0000) 2019-10-13T12:01:38.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:38.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.726777+0000) 2019-10-13T12:01:39.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:39.236+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.239400+0000) 2019-10-13T12:01:39.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:39.944+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.946168+0000) 2019-10-13T12:01:42.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:42.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:42.202066+0000) 2019-10-13T12:01:43.389 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:43.384+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.388201+0000) 2019-10-13T12:01:43.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:43.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.388841+0000) 2019-10-13T12:01:43.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:43.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.389419+0000) 2019-10-13T12:01:43.390 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:43.386+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.389774+0000) 2019-10-13T12:01:43.391 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:43.386+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.390136+0000) 2019-10-13T12:01:43.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:43.386+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.390413+0000) 2019-10-13T12:01:43.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:43.390+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.391399+0000) 2019-10-13T12:01:43.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:43.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 137999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:43.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:43.396+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.398249+0000) 2019-10-13T12:01:45.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:45.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.512186+0000) 2019-10-13T12:01:46.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:46.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:46.446100+0000) 2019-10-13T12:01:48.407 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:48.394+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.398624+0000) 2019-10-13T12:01:48.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:48.398+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.399367+0000) 2019-10-13T12:01:48.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:48.398+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.400371+0000) 2019-10-13T12:01:48.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:48.396+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.400532+0000) 2019-10-13T12:01:48.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:48.398+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.401038+0000) 2019-10-13T12:01:48.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:48.398+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.401403+0000) 2019-10-13T12:01:48.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:48.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.401684+0000) 2019-10-13T12:01:48.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:48.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:48.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:48.404+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.406291+0000) 2019-10-13T12:01:48.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:48.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.710488+0000) 2019-10-13T12:01:48.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:48.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.714483+0000) 2019-10-13T12:01:48.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:48.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.726973+0000) 2019-10-13T12:01:49.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:49.240+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.245737+0000) 2019-10-13T12:01:49.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:49.944+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.946362+0000) 2019-10-13T12:01:52.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:52.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:52.202292+0000) 2019-10-13T12:01:52.454 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:01:52.454 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:52.458 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:01:52.458 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:53.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:53.406+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.409197+0000) 2019-10-13T12:01:53.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:53.406+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.409763+0000) 2019-10-13T12:01:53.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:53.406+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.410165+0000) 2019-10-13T12:01:53.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:53.408+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.410992+0000) 2019-10-13T12:01:53.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:53.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.411912+0000) 2019-10-13T12:01:53.413 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:53.410+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.412123+0000) 2019-10-13T12:01:53.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:53.410+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.412642+0000) 2019-10-13T12:01:53.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:53.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:53.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:53.412+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.417058+0000) 2019-10-13T12:01:55.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:55.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.512390+0000) 2019-10-13T12:01:56.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:56.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:56.446316+0000) 2019-10-13T12:01:58.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:58.418+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.419442+0000) 2019-10-13T12:01:58.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:58.418+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.419945+0000) 2019-10-13T12:01:58.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:58.416+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.421172+0000) 2019-10-13T12:01:58.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:01:58.418+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.421822+0000) 2019-10-13T12:01:58.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:58.418+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.422695+0000) 2019-10-13T12:01:58.424 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:01:58.418+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.423187+0000) 2019-10-13T12:01:58.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:01:58.422+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.423460+0000) 2019-10-13T12:01:58.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:01:58.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:01:58.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:58.424+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.428274+0000) 2019-10-13T12:01:58.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:01:58.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.710665+0000) 2019-10-13T12:01:58.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:01:58.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.714689+0000) 2019-10-13T12:01:58.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:01:58.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.727148+0000) 2019-10-13T12:01:59.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:01:59.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.252247+0000) 2019-10-13T12:01:59.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:01:59.944+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.946574+0000) 2019-10-13T12:02:02.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:02.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:02.202494+0000) 2019-10-13T12:02:03.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:03.426+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.430607+0000) 2019-10-13T12:02:03.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:03.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.430687+0000) 2019-10-13T12:02:03.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:03.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.431696+0000) 2019-10-13T12:02:03.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:03.430+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.433350+0000) 2019-10-13T12:02:03.434 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:03.428+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.433575+0000) 2019-10-13T12:02:03.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:03.430+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.434023+0000) 2019-10-13T12:02:03.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:03.430+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.434437+0000) 2019-10-13T12:02:03.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:03.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:03.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:03.436+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.438236+0000) 2019-10-13T12:02:05.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:05.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.512607+0000) 2019-10-13T12:02:06.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:06.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:06.446508+0000) 2019-10-13T12:02:08.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:08.436+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.441629+0000) 2019-10-13T12:02:08.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:08.438+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.442592+0000) 2019-10-13T12:02:08.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:08.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.443074+0000) 2019-10-13T12:02:08.444 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:08.442+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.443362+0000) 2019-10-13T12:02:08.444 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:08.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.443699+0000) 2019-10-13T12:02:08.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:08.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.443993+0000) 2019-10-13T12:02:08.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:08.442+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.444462+0000) 2019-10-13T12:02:08.449 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:08.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:08.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:08.444+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.448596+0000) 2019-10-13T12:02:08.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:08.706+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.710845+0000) 2019-10-13T12:02:08.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:08.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.714863+0000) 2019-10-13T12:02:08.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:08.726+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.727345+0000) 2019-10-13T12:02:09.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:09.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.258798+0000) 2019-10-13T12:02:09.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:09.944+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.946733+0000) 2019-10-13T12:02:12.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:12.198+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:12.202696+0000) 2019-10-13T12:02:13.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:13.448+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.451370+0000) 2019-10-13T12:02:13.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:13.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.452185+0000) 2019-10-13T12:02:13.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:13.450+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.453524+0000) 2019-10-13T12:02:13.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:13.450+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.453963+0000) 2019-10-13T12:02:13.455 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:13.450+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.454244+0000) 2019-10-13T12:02:13.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:13.450+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.454849+0000) 2019-10-13T12:02:13.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:13.454+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.455673+0000) 2019-10-13T12:02:13.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:13.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138195 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:13.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:13.456+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.458576+0000) 2019-10-13T12:02:15.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:15.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.512811+0000) 2019-10-13T12:02:16.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:16.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:16.446704+0000) 2019-10-13T12:02:18.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:18.457+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.462111+0000) 2019-10-13T12:02:18.464 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:18.457+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.462439+0000) 2019-10-13T12:02:18.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:18.457+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.462864+0000) 2019-10-13T12:02:18.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:18.459+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.464128+0000) 2019-10-13T12:02:18.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:18.461+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.464733+0000) 2019-10-13T12:02:18.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:18.461+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.464902+0000) 2019-10-13T12:02:18.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:18.461+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.465218+0000) 2019-10-13T12:02:18.469 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:18.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:18.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:18.463+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.470092+0000) 2019-10-13T12:02:18.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:18.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.711022+0000) 2019-10-13T12:02:18.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:18.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.715054+0000) 2019-10-13T12:02:18.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:18.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.727523+0000) 2019-10-13T12:02:19.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:19.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.265170+0000) 2019-10-13T12:02:19.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:19.943+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.946939+0000) 2019-10-13T12:02:22.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:22.201+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:22.202904+0000) 2019-10-13T12:02:22.657 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:02:22.657 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:22.661 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:02:22.661 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:23.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:23.469+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.471813+0000) 2019-10-13T12:02:23.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:23.469+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.472105+0000) 2019-10-13T12:02:23.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:23.469+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.472708+0000) 2019-10-13T12:02:23.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:23.469+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.472847+0000) 2019-10-13T12:02:23.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:23.469+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.473547+0000) 2019-10-13T12:02:23.475 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:23.469+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.474043+0000) 2019-10-13T12:02:23.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:23.471+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.474596+0000) 2019-10-13T12:02:23.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:23.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:23.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:23.475+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.480193+0000) 2019-10-13T12:02:25.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:25.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.513036+0000) 2019-10-13T12:02:26.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:26.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:26.446893+0000) 2019-10-13T12:02:28.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:28.477+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.482373+0000) 2019-10-13T12:02:28.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:28.477+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.482610+0000) 2019-10-13T12:02:28.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:28.477+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.482969+0000) 2019-10-13T12:02:28.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:28.481+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.484221+0000) 2019-10-13T12:02:28.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:28.481+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.484616+0000) 2019-10-13T12:02:28.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:28.479+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.485257+0000) 2019-10-13T12:02:28.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:28.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.486276+0000) 2019-10-13T12:02:28.490 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:28.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:28.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:28.483+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.489446+0000) 2019-10-13T12:02:28.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:28.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.711235+0000) 2019-10-13T12:02:28.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:28.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.715224+0000) 2019-10-13T12:02:28.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:28.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.727732+0000) 2019-10-13T12:02:29.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:29.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.271715+0000) 2019-10-13T12:02:29.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:29.943+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.947159+0000) 2019-10-13T12:02:32.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:32.197+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:32.203101+0000) 2019-10-13T12:02:33.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:33.489+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.492462+0000) 2019-10-13T12:02:33.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:33.487+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.493426+0000) 2019-10-13T12:02:33.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:33.489+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.494044+0000) 2019-10-13T12:02:33.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:33.489+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.494851+0000) 2019-10-13T12:02:33.496 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:33.489+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.495058+0000) 2019-10-13T12:02:33.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:33.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.495586+0000) 2019-10-13T12:02:33.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:33.493+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.496770+0000) 2019-10-13T12:02:33.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:33.495+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.499075+0000) 2019-10-13T12:02:33.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:33.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:35.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:35.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.513261+0000) 2019-10-13T12:02:36.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:36.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:36.447128+0000) 2019-10-13T12:02:38.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:38.497+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.502780+0000) 2019-10-13T12:02:38.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:38.497+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.503116+0000) 2019-10-13T12:02:38.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:38.501+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.503704+0000) 2019-10-13T12:02:38.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:38.501+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.504126+0000) 2019-10-13T12:02:38.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:38.501+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.504647+0000) 2019-10-13T12:02:38.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:38.499+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.505200+0000) 2019-10-13T12:02:38.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:38.501+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.506510+0000) 2019-10-13T12:02:38.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:38.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:38.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:38.507+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.512351+0000) 2019-10-13T12:02:38.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:38.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.711435+0000) 2019-10-13T12:02:38.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:38.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.715397+0000) 2019-10-13T12:02:38.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:38.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.727930+0000) 2019-10-13T12:02:39.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:39.275+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.278123+0000) 2019-10-13T12:02:39.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:39.943+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.947353+0000) 2019-10-13T12:02:42.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:42.201+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:42.203331+0000) 2019-10-13T12:02:43.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:43.509+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.512995+0000) 2019-10-13T12:02:43.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:43.509+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.513544+0000) 2019-10-13T12:02:43.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:43.507+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.513570+0000) 2019-10-13T12:02:43.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:43.509+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.514603+0000) 2019-10-13T12:02:43.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:43.509+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.514972+0000) 2019-10-13T12:02:43.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:43.509+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.515151+0000) 2019-10-13T12:02:43.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:43.513+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.515536+0000) 2019-10-13T12:02:43.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:43.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:43.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:43.519+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.525125+0000) 2019-10-13T12:02:45.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:45.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.513469+0000) 2019-10-13T12:02:46.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:46.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:46.447341+0000) 2019-10-13T12:02:48.524 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:48.517+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.523241+0000) 2019-10-13T12:02:48.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:48.521+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.523828+0000) 2019-10-13T12:02:48.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:48.521+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.524635+0000) 2019-10-13T12:02:48.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:48.521+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.525230+0000) 2019-10-13T12:02:48.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:48.519+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.525400+0000) 2019-10-13T12:02:48.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:48.521+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.526087+0000) 2019-10-13T12:02:48.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:48.521+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.526269+0000) 2019-10-13T12:02:48.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:48.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:48.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:48.527+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.531255+0000) 2019-10-13T12:02:48.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:48.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.711614+0000) 2019-10-13T12:02:48.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:48.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.715580+0000) 2019-10-13T12:02:48.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:48.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.728103+0000) 2019-10-13T12:02:49.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:49.283+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.284493+0000) 2019-10-13T12:02:49.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:49.947+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.947581+0000) 2019-10-13T12:02:52.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:52.201+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:52.203570+0000) 2019-10-13T12:02:52.784 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:02:52.784 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:52.788 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:02:52.788 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:53.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:53.529+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.533617+0000) 2019-10-13T12:02:53.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:53.529+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.534094+0000) 2019-10-13T12:02:53.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:53.529+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.534884+0000) 2019-10-13T12:02:53.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:53.531+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.535238+0000) 2019-10-13T12:02:53.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:53.533+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.535751+0000) 2019-10-13T12:02:53.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:53.533+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.536028+0000) 2019-10-13T12:02:53.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:53.533+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.536673+0000) 2019-10-13T12:02:53.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:53.537+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:53.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:53.539+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.542437+0000) 2019-10-13T12:02:55.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:55.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.513633+0000) 2019-10-13T12:02:56.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:56.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:56.447531+0000) 2019-10-13T12:02:58.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:58.541+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.543599+0000) 2019-10-13T12:02:58.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:58.539+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.544104+0000) 2019-10-13T12:02:58.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:02:58.541+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.544558+0000) 2019-10-13T12:02:58.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:58.541+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.544948+0000) 2019-10-13T12:02:58.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:58.541+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.545653+0000) 2019-10-13T12:02:58.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:02:58.541+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.546160+0000) 2019-10-13T12:02:58.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:02:58.541+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.546455+0000) 2019-10-13T12:02:58.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:58.543+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.549428+0000) 2019-10-13T12:02:58.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:02:58.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:02:58.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:02:58.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.711796+0000) 2019-10-13T12:02:58.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:02:58.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.715756+0000) 2019-10-13T12:02:58.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:02:58.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.728304+0000) 2019-10-13T12:02:59.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:02:59.291+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.293519+0000) 2019-10-13T12:02:59.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:02:59.943+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.947775+0000) 2019-10-13T12:03:02.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:02.200+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:02.203770+0000) 2019-10-13T12:03:03.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:03.548+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.554575+0000) 2019-10-13T12:03:03.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:03.551+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.554829+0000) 2019-10-13T12:03:03.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:03.548+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.555275+0000) 2019-10-13T12:03:03.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:03.552+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.555513+0000) 2019-10-13T12:03:03.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:03.552+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.555937+0000) 2019-10-13T12:03:03.557 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:03.552+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.556551+0000) 2019-10-13T12:03:03.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:03.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.557277+0000) 2019-10-13T12:03:03.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:03.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:03.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:03.559+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.565121+0000) 2019-10-13T12:03:05.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:05.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.513853+0000) 2019-10-13T12:03:06.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:06.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:06.447725+0000) 2019-10-13T12:03:08.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:08.560+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.564294+0000) 2019-10-13T12:03:08.566 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:08.559+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.564405+0000) 2019-10-13T12:03:08.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:08.560+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.565055+0000) 2019-10-13T12:03:08.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:08.560+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.565790+0000) 2019-10-13T12:03:08.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:08.560+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.566400+0000) 2019-10-13T12:03:08.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:08.560+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.566909+0000) 2019-10-13T12:03:08.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:08.564+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.568299+0000) 2019-10-13T12:03:08.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:08.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:08.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:08.567+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.572218+0000) 2019-10-13T12:03:08.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:08.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.711980+0000) 2019-10-13T12:03:08.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:08.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.715935+0000) 2019-10-13T12:03:08.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:08.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.728512+0000) 2019-10-13T12:03:09.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:09.295+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.299992+0000) 2019-10-13T12:03:09.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:09.947+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.947968+0000) 2019-10-13T12:03:12.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:12.200+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:12.203976+0000) 2019-10-13T12:03:13.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:13.568+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.574144+0000) 2019-10-13T12:03:13.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:13.568+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.574558+0000) 2019-10-13T12:03:13.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:13.568+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.575194+0000) 2019-10-13T12:03:13.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:13.570+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.576196+0000) 2019-10-13T12:03:13.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:13.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.576625+0000) 2019-10-13T12:03:13.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:13.572+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.577079+0000) 2019-10-13T12:03:13.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:13.572+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.577795+0000) 2019-10-13T12:03:13.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:13.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:13.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:13.574+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.581746+0000) 2019-10-13T12:03:15.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:15.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.514065+0000) 2019-10-13T12:03:16.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:16.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:16.447936+0000) 2019-10-13T12:03:18.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:18.580+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.584445+0000) 2019-10-13T12:03:18.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:18.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.584745+0000) 2019-10-13T12:03:18.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:18.580+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.586010+0000) 2019-10-13T12:03:18.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:18.580+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.586375+0000) 2019-10-13T12:03:18.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:18.580+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.586873+0000) 2019-10-13T12:03:18.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:18.584+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.587755+0000) 2019-10-13T12:03:18.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:18.584+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.588410+0000) 2019-10-13T12:03:18.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:18.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138616 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:18.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:18.586+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.590849+0000) 2019-10-13T12:03:18.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:18.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.712180+0000) 2019-10-13T12:03:18.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:18.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.716143+0000) 2019-10-13T12:03:18.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:18.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.728691+0000) 2019-10-13T12:03:19.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:19.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.310855+0000) 2019-10-13T12:03:19.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:19.946+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.948189+0000) 2019-10-13T12:03:22.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:22.200+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:22.204171+0000) 2019-10-13T12:03:22.913 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:03:22.913 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:22.916 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:03:22.917 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:23.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:23.592+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.595572+0000) 2019-10-13T12:03:23.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:23.592+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.595892+0000) 2019-10-13T12:03:23.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:23.592+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.596206+0000) 2019-10-13T12:03:23.598 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:23.590+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.596769+0000) 2019-10-13T12:03:23.598 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:23.592+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.597192+0000) 2019-10-13T12:03:23.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:23.592+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.597527+0000) 2019-10-13T12:03:23.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:23.592+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.598658+0000) 2019-10-13T12:03:23.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:23.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:23.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:23.598+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.602581+0000) 2019-10-13T12:03:25.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:25.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.514334+0000) 2019-10-13T12:03:26.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:26.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:26.448144+0000) 2019-10-13T12:03:28.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:28.600+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.605154+0000) 2019-10-13T12:03:28.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:28.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.606103+0000) 2019-10-13T12:03:28.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:28.600+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.606718+0000) 2019-10-13T12:03:28.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:28.600+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.607227+0000) 2019-10-13T12:03:28.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:28.604+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.607678+0000) 2019-10-13T12:03:28.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:28.602+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.608318+0000) 2019-10-13T12:03:28.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:28.604+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.608844+0000) 2019-10-13T12:03:28.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:28.606+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.611170+0000) 2019-10-13T12:03:28.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:28.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:28.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:28.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.712340+0000) 2019-10-13T12:03:28.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:28.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.716308+0000) 2019-10-13T12:03:28.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:28.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.728891+0000) 2019-10-13T12:03:29.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:29.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.317363+0000) 2019-10-13T12:03:29.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:29.942+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.948388+0000) 2019-10-13T12:03:32.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:32.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:32.204368+0000) 2019-10-13T12:03:33.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:33.608+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.615275+0000) 2019-10-13T12:03:33.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:33.612+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.615727+0000) 2019-10-13T12:03:33.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:33.612+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.616082+0000) 2019-10-13T12:03:33.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:33.612+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.616567+0000) 2019-10-13T12:03:33.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:33.612+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.617072+0000) 2019-10-13T12:03:33.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:33.610+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.617348+0000) 2019-10-13T12:03:33.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:33.612+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.617980+0000) 2019-10-13T12:03:33.622 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:33.616+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:33.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:33.618+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.624783+0000) 2019-10-13T12:03:35.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:35.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.514566+0000) 2019-10-13T12:03:36.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:36.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:36.448339+0000) 2019-10-13T12:03:38.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:38.620+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.625013+0000) 2019-10-13T12:03:38.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:38.622+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.627027+0000) 2019-10-13T12:03:38.628 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:38.624+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.627729+0000) 2019-10-13T12:03:38.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:38.624+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.627939+0000) 2019-10-13T12:03:38.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:38.624+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.628221+0000) 2019-10-13T12:03:38.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:38.624+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.628757+0000) 2019-10-13T12:03:38.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:38.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.629590+0000) 2019-10-13T12:03:38.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:38.628+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:38.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:38.626+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.632326+0000) 2019-10-13T12:03:38.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:38.708+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.712551+0000) 2019-10-13T12:03:38.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:38.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.716477+0000) 2019-10-13T12:03:38.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:38.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.729112+0000) 2019-10-13T12:03:39.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:39.322+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.323169+0000) 2019-10-13T12:03:39.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:39.946+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.948579+0000) 2019-10-13T12:03:42.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:42.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:42.204569+0000) 2019-10-13T12:03:43.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:43.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.635564+0000) 2019-10-13T12:03:43.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:43.631+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.636385+0000) 2019-10-13T12:03:43.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:43.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.636942+0000) 2019-10-13T12:03:43.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:43.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.637549+0000) 2019-10-13T12:03:43.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:43.631+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.637734+0000) 2019-10-13T12:03:43.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:43.631+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.638041+0000) 2019-10-13T12:03:43.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:43.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.638854+0000) 2019-10-13T12:03:43.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:43.638+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.642182+0000) 2019-10-13T12:03:43.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:43.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:45.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:45.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.514788+0000) 2019-10-13T12:03:46.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:46.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:46.448497+0000) 2019-10-13T12:03:48.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:48.639+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.646094+0000) 2019-10-13T12:03:48.648 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:48.639+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.646958+0000) 2019-10-13T12:03:48.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:48.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.647641+0000) 2019-10-13T12:03:48.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:48.642+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.647963+0000) 2019-10-13T12:03:48.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:48.643+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.648709+0000) 2019-10-13T12:03:48.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:48.643+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.648973+0000) 2019-10-13T12:03:48.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:48.643+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.649219+0000) 2019-10-13T12:03:48.654 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:48.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:48.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:48.650+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.654665+0000) 2019-10-13T12:03:48.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:48.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.712734+0000) 2019-10-13T12:03:48.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:48.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.716670+0000) 2019-10-13T12:03:48.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:48.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.729284+0000) 2019-10-13T12:03:49.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:49.326+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.329611+0000) 2019-10-13T12:03:49.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:49.946+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.948776+0000) 2019-10-13T12:03:52.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:52.203+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:52.204714+0000) 2019-10-13T12:03:53.030 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:03:53.030 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:53.034 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:03:53.034 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:53.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:53.651+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.656449+0000) 2019-10-13T12:03:53.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:53.651+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.657107+0000) 2019-10-13T12:03:53.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:53.651+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.657852+0000) 2019-10-13T12:03:53.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:53.651+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.658475+0000) 2019-10-13T12:03:53.660 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:53.651+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.658978+0000) 2019-10-13T12:03:53.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:53.655+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.659410+0000) 2019-10-13T12:03:53.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:53.654+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.659617+0000) 2019-10-13T12:03:53.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:53.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:53.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:53.662+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.666438+0000) 2019-10-13T12:03:55.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:55.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.514990+0000) 2019-10-13T12:03:56.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:56.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:56.448684+0000) 2019-10-13T12:03:58.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:03:58.659+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.666066+0000) 2019-10-13T12:03:58.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:58.659+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.666760+0000) 2019-10-13T12:03:58.668 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:58.659+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.667168+0000) 2019-10-13T12:03:58.668 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:03:58.663+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.668074+0000) 2019-10-13T12:03:58.669 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:03:58.663+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.668175+0000) 2019-10-13T12:03:58.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:58.663+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.668689+0000) 2019-10-13T12:03:58.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:58.666+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.669613+0000) 2019-10-13T12:03:58.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:03:58.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:03:58.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:58.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.678676+0000) 2019-10-13T12:03:58.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:03:58.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.712911+0000) 2019-10-13T12:03:58.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:03:58.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.716882+0000) 2019-10-13T12:03:58.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:03:58.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.729469+0000) 2019-10-13T12:03:59.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:03:59.334+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.336414+0000) 2019-10-13T12:03:59.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:03:59.946+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.948967+0000) 2019-10-13T12:04:02.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:02.203+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:02.204920+0000) 2019-10-13T12:04:03.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:03.671+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.677376+0000) 2019-10-13T12:04:03.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:03.671+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.678310+0000) 2019-10-13T12:04:03.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:03.671+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.678689+0000) 2019-10-13T12:04:03.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:03.675+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.679518+0000) 2019-10-13T12:04:03.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:03.678+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.679525+0000) 2019-10-13T12:04:03.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:03.675+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.680214+0000) 2019-10-13T12:04:03.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:03.675+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.680983+0000) 2019-10-13T12:04:03.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:03.678+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.682013+0000) 2019-10-13T12:04:03.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:03.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:05.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:05.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.515189+0000) 2019-10-13T12:04:06.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:06.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:06.448908+0000) 2019-10-13T12:04:08.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:08.685+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.687979+0000) 2019-10-13T12:04:08.690 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:08.683+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.688687+0000) 2019-10-13T12:04:08.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:08.683+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.689272+0000) 2019-10-13T12:04:08.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:08.683+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.689768+0000) 2019-10-13T12:04:08.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:08.683+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.690660+0000) 2019-10-13T12:04:08.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:08.683+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.691040+0000) 2019-10-13T12:04:08.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:08.687+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.692388+0000) 2019-10-13T12:04:08.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:08.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:08.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:08.697+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.699117+0000) 2019-10-13T12:04:08.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:08.707+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.713126+0000) 2019-10-13T12:04:08.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:08.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.717052+0000) 2019-10-13T12:04:08.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:08.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.729648+0000) 2019-10-13T12:04:09.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:09.341+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.342843+0000) 2019-10-13T12:04:09.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:09.945+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.949159+0000) 2019-10-13T12:04:12.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:12.203+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:12.205183+0000) 2019-10-13T12:04:13.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:13.695+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.699185+0000) 2019-10-13T12:04:13.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:13.697+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.699593+0000) 2019-10-13T12:04:13.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:13.699+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.700458+0000) 2019-10-13T12:04:13.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:13.699+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.701415+0000) 2019-10-13T12:04:13.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:13.699+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.702035+0000) 2019-10-13T12:04:13.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:13.699+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.702305+0000) 2019-10-13T12:04:13.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:13.699+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.702675+0000) 2019-10-13T12:04:13.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:13.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 138975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:13.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:13.705+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.706583+0000) 2019-10-13T12:04:15.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:15.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.515387+0000) 2019-10-13T12:04:16.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:16.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:16.449104+0000) 2019-10-13T12:04:18.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:18.707+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.708926+0000) 2019-10-13T12:04:18.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:18.707+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.709270+0000) 2019-10-13T12:04:18.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:18.707+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.709556+0000) 2019-10-13T12:04:18.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:18.707+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.710391+0000) 2019-10-13T12:04:18.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:18.707+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.711111+0000) 2019-10-13T12:04:18.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:18.709+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.712074+0000) 2019-10-13T12:04:18.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:18.711+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.712403+0000) 2019-10-13T12:04:18.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:18.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.713262+0000) 2019-10-13T12:04:18.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:18.715+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:18.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:18.713+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.716895+0000) 2019-10-13T12:04:18.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:18.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.717180+0000) 2019-10-13T12:04:18.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:18.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.729804+0000) 2019-10-13T12:04:19.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:19.345+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.349531+0000) 2019-10-13T12:04:19.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:19.945+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.949375+0000) 2019-10-13T12:04:22.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:22.203+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:22.205389+0000) 2019-10-13T12:04:23.101 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:04:23.102 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:23.105 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:04:23.105 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:23.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:23.717+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.719249+0000) 2019-10-13T12:04:23.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:23.719+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.721071+0000) 2019-10-13T12:04:23.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:23.719+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.721282+0000) 2019-10-13T12:04:23.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:23.719+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.721588+0000) 2019-10-13T12:04:23.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:23.719+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.721889+0000) 2019-10-13T12:04:23.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:23.719+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.722337+0000) 2019-10-13T12:04:23.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:23.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.723365+0000) 2019-10-13T12:04:23.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:23.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:23.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:23.725+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.729805+0000) 2019-10-13T12:04:25.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:25.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.515581+0000) 2019-10-13T12:04:26.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:26.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:26.449330+0000) 2019-10-13T12:04:28.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:28.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.713449+0000) 2019-10-13T12:04:28.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:28.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.717479+0000) 2019-10-13T12:04:28.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:28.726+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.730891+0000) 2019-10-13T12:04:28.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:28.726+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.730973+0000) 2019-10-13T12:04:28.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:28.729+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.731700+0000) 2019-10-13T12:04:28.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:28.730+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.732365+0000) 2019-10-13T12:04:28.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:28.730+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.732772+0000) 2019-10-13T12:04:28.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:28.730+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.733373+0000) 2019-10-13T12:04:28.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:28.730+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.734494+0000) 2019-10-13T12:04:28.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:28.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:28.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:28.737+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.740208+0000) 2019-10-13T12:04:29.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:29.357+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.360714+0000) 2019-10-13T12:04:29.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:29.945+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.949568+0000) 2019-10-13T12:04:32.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:32.202+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:32.205590+0000) 2019-10-13T12:04:33.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:33.738+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.740131+0000) 2019-10-13T12:04:33.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:33.738+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.741315+0000) 2019-10-13T12:04:33.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:33.738+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.742399+0000) 2019-10-13T12:04:33.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:33.741+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.742770+0000) 2019-10-13T12:04:33.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:33.738+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.743170+0000) 2019-10-13T12:04:33.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:33.742+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.743590+0000) 2019-10-13T12:04:33.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:33.742+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.743778+0000) 2019-10-13T12:04:33.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:33.741+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.745244+0000) 2019-10-13T12:04:33.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:33.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:35.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:35.478+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.480548+0000) 2019-10-13T12:04:35.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:35.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.515784+0000) 2019-10-13T12:04:36.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:36.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:36.449551+0000) 2019-10-13T12:04:38.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:38.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.713657+0000) 2019-10-13T12:04:38.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:38.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.717673+0000) 2019-10-13T12:04:38.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:38.746+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.751109+0000) 2019-10-13T12:04:38.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:38.750+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.751849+0000) 2019-10-13T12:04:38.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:38.750+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.752449+0000) 2019-10-13T12:04:38.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:38.749+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.752757+0000) 2019-10-13T12:04:38.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:38.750+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.753515+0000) 2019-10-13T12:04:38.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:38.750+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.753863+0000) 2019-10-13T12:04:38.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:38.750+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.754460+0000) 2019-10-13T12:04:38.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:38.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:38.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:38.761+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.763801+0000) 2019-10-13T12:04:39.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:39.369+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.371269+0000) 2019-10-13T12:04:39.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:39.949+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.949761+0000) 2019-10-13T12:04:42.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:42.202+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:42.205807+0000) 2019-10-13T12:04:43.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:43.758+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.761771+0000) 2019-10-13T12:04:43.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:43.758+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.762638+0000) 2019-10-13T12:04:43.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:43.758+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.763013+0000) 2019-10-13T12:04:43.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:43.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.763131+0000) 2019-10-13T12:04:43.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:43.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.764150+0000) 2019-10-13T12:04:43.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:43.762+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.765160+0000) 2019-10-13T12:04:43.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:43.762+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.765470+0000) 2019-10-13T12:04:43.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:43.765+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.768581+0000) 2019-10-13T12:04:43.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:43.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:45.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:45.478+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:45.480746+0000) 2019-10-13T12:04:45.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:45.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:45.515983+0000) 2019-10-13T12:04:46.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:46.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.449768+0000) 2019-10-13T12:04:48.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:48.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.713891+0000) 2019-10-13T12:04:48.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:48.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.717885+0000) 2019-10-13T12:04:48.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:48.770+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.773000+0000) 2019-10-13T12:04:48.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:48.770+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.773435+0000) 2019-10-13T12:04:48.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:48.769+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.773667+0000) 2019-10-13T12:04:48.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:48.770+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.774370+0000) 2019-10-13T12:04:48.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:48.770+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.775009+0000) 2019-10-13T12:04:48.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:48.774+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.775537+0000) 2019-10-13T12:04:48.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:48.774+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.776142+0000) 2019-10-13T12:04:48.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:48.778+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:48.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:48.777+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.780790+0000) 2019-10-13T12:04:49.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:49.373+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.377741+0000) 2019-10-13T12:04:49.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:49.949+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.949951+0000) 2019-10-13T12:04:52.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:52.202+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:52.206026+0000) 2019-10-13T12:04:53.252 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:04:53.252 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:53.256 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:04:53.256 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:53.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:53.778+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.782798+0000) 2019-10-13T12:04:53.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:53.778+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.783223+0000) 2019-10-13T12:04:53.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:53.782+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.783661+0000) 2019-10-13T12:04:53.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:53.782+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.784805+0000) 2019-10-13T12:04:53.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:53.781+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.784769+0000) 2019-10-13T12:04:53.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:53.782+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.785225+0000) 2019-10-13T12:04:53.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:53.782+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.785925+0000) 2019-10-13T12:04:53.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:53.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:53.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:53.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.792946+0000) 2019-10-13T12:04:55.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:55.478+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:55.480942+0000) 2019-10-13T12:04:55.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:55.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:55.516189+0000) 2019-10-13T12:04:56.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:56.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.449964+0000) 2019-10-13T12:04:58.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:58.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.714088+0000) 2019-10-13T12:04:58.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:58.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.718094+0000) 2019-10-13T12:04:58.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:04:58.790+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.792909+0000) 2019-10-13T12:04:58.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:04:58.790+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.793690+0000) 2019-10-13T12:04:58.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:04:58.790+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.794206+0000) 2019-10-13T12:04:58.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:04:58.790+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.794378+0000) 2019-10-13T12:04:58.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:04:58.790+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.794759+0000) 2019-10-13T12:04:58.796 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:58.793+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.795422+0000) 2019-10-13T12:04:58.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:04:58.794+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.796934+0000) 2019-10-13T12:04:58.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:04:58.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:04:58.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:58.801+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.802480+0000) 2019-10-13T12:04:59.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:04:59.381+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.384305+0000) 2019-10-13T12:04:59.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:04:59.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.950174+0000) 2019-10-13T12:05:02.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:02.206+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:02.206231+0000) 2019-10-13T12:05:03.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:03.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.803140+0000) 2019-10-13T12:05:03.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:03.802+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.804440+0000) 2019-10-13T12:05:03.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:03.802+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.804514+0000) 2019-10-13T12:05:03.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:03.802+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.804967+0000) 2019-10-13T12:05:03.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:03.802+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.805448+0000) 2019-10-13T12:05:03.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:03.802+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.805906+0000) 2019-10-13T12:05:03.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:03.804+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.806330+0000) 2019-10-13T12:05:03.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:03.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:03.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:03.808+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.812966+0000) 2019-10-13T12:05:05.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:05.478+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.481173+0000) 2019-10-13T12:05:05.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:05.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.516431+0000) 2019-10-13T12:05:06.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:06.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.450147+0000) 2019-10-13T12:05:08.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:08.710+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.714287+0000) 2019-10-13T12:05:08.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:08.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.718295+0000) 2019-10-13T12:05:08.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:08.810+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.812696+0000) 2019-10-13T12:05:08.814 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:08.810+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.813212+0000) 2019-10-13T12:05:08.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:08.810+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.813484+0000) 2019-10-13T12:05:08.815 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:08.810+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.814707+0000) 2019-10-13T12:05:08.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:08.814+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.815523+0000) 2019-10-13T12:05:08.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:08.814+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.816042+0000) 2019-10-13T12:05:08.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:08.812+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.816417+0000) 2019-10-13T12:05:08.821 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:08.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:08.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:08.816+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.821583+0000) 2019-10-13T12:05:09.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:09.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:09.390691+0000) 2019-10-13T12:05:09.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:09.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:09.950371+0000) 2019-10-13T12:05:12.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:12.205+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:12.206434+0000) 2019-10-13T12:05:13.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:13.820+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.823195+0000) 2019-10-13T12:05:13.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:13.821+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.823780+0000) 2019-10-13T12:05:13.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:13.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.824036+0000) 2019-10-13T12:05:13.825 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:13.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.824704+0000) 2019-10-13T12:05:13.826 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:13.821+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.825269+0000) 2019-10-13T12:05:13.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:13.821+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.826347+0000) 2019-10-13T12:05:13.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:13.821+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.826740+0000) 2019-10-13T12:05:13.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:13.828+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.830819+0000) 2019-10-13T12:05:13.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:13.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:15.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:15.477+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.481375+0000) 2019-10-13T12:05:15.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:15.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.516738+0000) 2019-10-13T12:05:16.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:16.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.450349+0000) 2019-10-13T12:05:18.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:18.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.714502+0000) 2019-10-13T12:05:18.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:18.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.718515+0000) 2019-10-13T12:05:18.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:18.829+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.834467+0000) 2019-10-13T12:05:18.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:18.829+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.834821+0000) 2019-10-13T12:05:18.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:18.833+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.835625+0000) 2019-10-13T12:05:18.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:18.833+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.835889+0000) 2019-10-13T12:05:18.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:18.833+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.836482+0000) 2019-10-13T12:05:18.838 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:18.832+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.837301+0000) 2019-10-13T12:05:18.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:18.833+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.837678+0000) 2019-10-13T12:05:18.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:18.837+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:18.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:18.840+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.843740+0000) 2019-10-13T12:05:19.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:19.396+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:19.397315+0000) 2019-10-13T12:05:19.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:19.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:19.950567+0000) 2019-10-13T12:05:22.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:22.201+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:22.206636+0000) 2019-10-13T12:05:23.358 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:05:23.358 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:23.361 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:05:23.362 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:23.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:23.841+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.844675+0000) 2019-10-13T12:05:23.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:23.841+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.845442+0000) 2019-10-13T12:05:23.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:23.840+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.845736+0000) 2019-10-13T12:05:23.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:23.841+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.846610+0000) 2019-10-13T12:05:23.847 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:23.841+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.846959+0000) 2019-10-13T12:05:23.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:23.845+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.847770+0000) 2019-10-13T12:05:23.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:23.845+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.848257+0000) 2019-10-13T12:05:23.852 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:23.849+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:23.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:23.848+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.852973+0000) 2019-10-13T12:05:25.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:25.477+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.481617+0000) 2019-10-13T12:05:25.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:25.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.516943+0000) 2019-10-13T12:05:26.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:26.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.450567+0000) 2019-10-13T12:05:28.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:28.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.714705+0000) 2019-10-13T12:05:28.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:28.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.718711+0000) 2019-10-13T12:05:28.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:28.853+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.855510+0000) 2019-10-13T12:05:28.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:28.853+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.855870+0000) 2019-10-13T12:05:28.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:28.853+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.856194+0000) 2019-10-13T12:05:28.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:28.852+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.856366+0000) 2019-10-13T12:05:28.858 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:28.853+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.857160+0000) 2019-10-13T12:05:28.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:28.853+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.856866+0000) 2019-10-13T12:05:28.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:28.853+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.858385+0000) 2019-10-13T12:05:28.862 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:28.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:28.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:28.860+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.863699+0000) 2019-10-13T12:05:29.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:29.404+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:29.409168+0000) 2019-10-13T12:05:29.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:29.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:29.950764+0000) 2019-10-13T12:05:32.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:32.201+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:32.206836+0000) 2019-10-13T12:05:33.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:33.861+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.865356+0000) 2019-10-13T12:05:33.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:33.860+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.865615+0000) 2019-10-13T12:05:33.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:33.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.866417+0000) 2019-10-13T12:05:33.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:33.861+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.866502+0000) 2019-10-13T12:05:33.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:33.861+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.866927+0000) 2019-10-13T12:05:33.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:33.861+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.867242+0000) 2019-10-13T12:05:33.869 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:33.865+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.868473+0000) 2019-10-13T12:05:33.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:33.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:33.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:33.872+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.875494+0000) 2019-10-13T12:05:35.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:35.481+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.481851+0000) 2019-10-13T12:05:35.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:35.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.517156+0000) 2019-10-13T12:05:36.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:36.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.450798+0000) 2019-10-13T12:05:38.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:38.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.714924+0000) 2019-10-13T12:05:38.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:38.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.718938+0000) 2019-10-13T12:05:38.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:38.869+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.874615+0000) 2019-10-13T12:05:38.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:38.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.875410+0000) 2019-10-13T12:05:38.876 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:38.873+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.875687+0000) 2019-10-13T12:05:38.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:38.872+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.875993+0000) 2019-10-13T12:05:38.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:38.873+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.876551+0000) 2019-10-13T12:05:38.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:38.873+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.877303+0000) 2019-10-13T12:05:38.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:38.873+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.877115+0000) 2019-10-13T12:05:38.882 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:38.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:38.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:38.880+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.883245+0000) 2019-10-13T12:05:39.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:39.412+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:39.415653+0000) 2019-10-13T12:05:39.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:39.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:39.950974+0000) 2019-10-13T12:05:42.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:42.205+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:42.207040+0000) 2019-10-13T12:05:43.885 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:43.881+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.884327+0000) 2019-10-13T12:05:43.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:43.881+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.885166+0000) 2019-10-13T12:05:43.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:43.881+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.885403+0000) 2019-10-13T12:05:43.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:43.884+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.886746+0000) 2019-10-13T12:05:43.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:43.881+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.886147+0000) 2019-10-13T12:05:43.888 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:43.885+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.887286+0000) 2019-10-13T12:05:43.888 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:43.885+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.887785+0000) 2019-10-13T12:05:43.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:43.885+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:43.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:43.888+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.891750+0000) 2019-10-13T12:05:45.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:45.477+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.482049+0000) 2019-10-13T12:05:45.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:45.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.517360+0000) 2019-10-13T12:05:46.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:46.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.451007+0000) 2019-10-13T12:05:48.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:48.709+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.715126+0000) 2019-10-13T12:05:48.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:48.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.719134+0000) 2019-10-13T12:05:48.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:48.889+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.894270+0000) 2019-10-13T12:05:48.896 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:48.889+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.894530+0000) 2019-10-13T12:05:48.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:48.892+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.895495+0000) 2019-10-13T12:05:48.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:48.893+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.896466+0000) 2019-10-13T12:05:48.897 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:48.893+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.896613+0000) 2019-10-13T12:05:48.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:48.893+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.897172+0000) 2019-10-13T12:05:48.898 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:48.893+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.897566+0000) 2019-10-13T12:05:48.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:48.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:48.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:48.900+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.902840+0000) 2019-10-13T12:05:49.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:49.420+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:49.422197+0000) 2019-10-13T12:05:49.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:49.948+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:49.951229+0000) 2019-10-13T12:05:52.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:52.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:52.207267+0000) 2019-10-13T12:05:53.422 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:05:53.422 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:53.426 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:05:53.426 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:53.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:53.900+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.904831+0000) 2019-10-13T12:05:53.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:53.900+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.905884+0000) 2019-10-13T12:05:53.907 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:53.900+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.906823+0000) 2019-10-13T12:05:53.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:53.904+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.907584+0000) 2019-10-13T12:05:53.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:53.904+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.908067+0000) 2019-10-13T12:05:53.912 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:53.904+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.907351+0000) 2019-10-13T12:05:53.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:53.904+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.908868+0000) 2019-10-13T12:05:53.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:53.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:53.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:53.912+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.914566+0000) 2019-10-13T12:05:55.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:55.476+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.482282+0000) 2019-10-13T12:05:55.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:55.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.517584+0000) 2019-10-13T12:05:56.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:56.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.451200+0000) 2019-10-13T12:05:58.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:58.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.715363+0000) 2019-10-13T12:05:58.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:58.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.719362+0000) 2019-10-13T12:05:58.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:58.911+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.915864+0000) 2019-10-13T12:05:58.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:05:58.912+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.916411+0000) 2019-10-13T12:05:58.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:05:58.912+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.916802+0000) 2019-10-13T12:05:58.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:05:58.912+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.917307+0000) 2019-10-13T12:05:58.918 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:05:58.912+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.917572+0000) 2019-10-13T12:05:58.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:05:58.912+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.918844+0000) 2019-10-13T12:05:58.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:05:58.912+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.918913+0000) 2019-10-13T12:05:58.923 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:05:58.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:05:58.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:58.919+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.925136+0000) 2019-10-13T12:05:59.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:05:59.427+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:59.428567+0000) 2019-10-13T12:05:59.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:05:59.947+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:59.951454+0000) 2019-10-13T12:06:02.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:02.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:02.207503+0000) 2019-10-13T12:06:03.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:03.920+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.925996+0000) 2019-10-13T12:06:03.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:03.923+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.926516+0000) 2019-10-13T12:06:03.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:03.920+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.926317+0000) 2019-10-13T12:06:03.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:03.920+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.927194+0000) 2019-10-13T12:06:03.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:03.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.927827+0000) 2019-10-13T12:06:03.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:03.924+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.928344+0000) 2019-10-13T12:06:03.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:03.924+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.929294+0000) 2019-10-13T12:06:03.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:03.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:03.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:03.927+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.933710+0000) 2019-10-13T12:06:05.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:05.476+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.482535+0000) 2019-10-13T12:06:05.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:05.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.517787+0000) 2019-10-13T12:06:06.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:06.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.451388+0000) 2019-10-13T12:06:08.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:08.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.715605+0000) 2019-10-13T12:06:08.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:08.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.719546+0000) 2019-10-13T12:06:08.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:08.932+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.935620+0000) 2019-10-13T12:06:08.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:08.931+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.936434+0000) 2019-10-13T12:06:08.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:08.932+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.936796+0000) 2019-10-13T12:06:08.938 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:08.932+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.937131+0000) 2019-10-13T12:06:08.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:08.932+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.938108+0000) 2019-10-13T12:06:08.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:08.932+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.938889+0000) 2019-10-13T12:06:08.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:08.932+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.939106+0000) 2019-10-13T12:06:08.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:08.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:08.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:08.939+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.945830+0000) 2019-10-13T12:06:09.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:09.435+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:09.440169+0000) 2019-10-13T12:06:09.952 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:09.947+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:09.951607+0000) 2019-10-13T12:06:12.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:12.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:12.207735+0000) 2019-10-13T12:06:13.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:13.943+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.946360+0000) 2019-10-13T12:06:13.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:13.944+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.948115+0000) 2019-10-13T12:06:13.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:13.944+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.948329+0000) 2019-10-13T12:06:13.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:13.944+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.948951+0000) 2019-10-13T12:06:13.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:13.944+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.949144+0000) 2019-10-13T12:06:13.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:13.944+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.949347+0000) 2019-10-13T12:06:13.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:13.944+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.950134+0000) 2019-10-13T12:06:13.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:13.947+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.953585+0000) 2019-10-13T12:06:13.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:13.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:15.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:15.480+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.482736+0000) 2019-10-13T12:06:15.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:15.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.517994+0000) 2019-10-13T12:06:16.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:16.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.451586+0000) 2019-10-13T12:06:18.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:18.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.715815+0000) 2019-10-13T12:06:18.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:18.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.719760+0000) 2019-10-13T12:06:18.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:18.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.957704+0000) 2019-10-13T12:06:18.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:18.951+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.957802+0000) 2019-10-13T12:06:18.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:18.952+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.959186+0000) 2019-10-13T12:06:18.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:18.952+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.958870+0000) 2019-10-13T12:06:18.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:18.956+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.959643+0000) 2019-10-13T12:06:18.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:18.956+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.960084+0000) 2019-10-13T12:06:18.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:18.956+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.960526+0000) 2019-10-13T12:06:18.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:18.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:18.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:18.963+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.967009+0000) 2019-10-13T12:06:19.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:19.443+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:19.446803+0000) 2019-10-13T12:06:19.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:19.951+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:19.951760+0000) 2019-10-13T12:06:22.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:22.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:22.207973+0000) 2019-10-13T12:06:23.546 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:06:23.547 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:23.550 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:06:23.550 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:23.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:23.964+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.968474+0000) 2019-10-13T12:06:23.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:23.964+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.969009+0000) 2019-10-13T12:06:23.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:23.963+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.969111+0000) 2019-10-13T12:06:23.971 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:23.964+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.970275+0000) 2019-10-13T12:06:23.971 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:23.964+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.970537+0000) 2019-10-13T12:06:23.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:23.964+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.970939+0000) 2019-10-13T12:06:23.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:23.964+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.971204+0000) 2019-10-13T12:06:23.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:23.971+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.975458+0000) 2019-10-13T12:06:23.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:23.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:25.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:25.480+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.482969+0000) 2019-10-13T12:06:25.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:25.516+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.518196+0000) 2019-10-13T12:06:26.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:26.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.451818+0000) 2019-10-13T12:06:28.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:28.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.716031+0000) 2019-10-13T12:06:28.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:28.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.719976+0000) 2019-10-13T12:06:28.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:28.972+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.979275+0000) 2019-10-13T12:06:28.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:28.976+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.979976+0000) 2019-10-13T12:06:28.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:28.976+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.980513+0000) 2019-10-13T12:06:28.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:28.975+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.980666+0000) 2019-10-13T12:06:28.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:28.976+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.981014+0000) 2019-10-13T12:06:28.982 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:28.976+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.981566+0000) 2019-10-13T12:06:28.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:28.976+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.981510+0000) 2019-10-13T12:06:28.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:28.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:28.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:28.983+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.986401+0000) 2019-10-13T12:06:29.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:29.451+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:29.453246+0000) 2019-10-13T12:06:29.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:29.951+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:29.951937+0000) 2019-10-13T12:06:32.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:32.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:32.208174+0000) 2019-10-13T12:06:33.990 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:33.984+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.989667+0000) 2019-10-13T12:06:33.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:33.984+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.990236+0000) 2019-10-13T12:06:33.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:33.984+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.991206+0000) 2019-10-13T12:06:33.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:33.987+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.991131+0000) 2019-10-13T12:06:33.992 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:33.988+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.991717+0000) 2019-10-13T12:06:33.993 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:33.988+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.992387+0000) 2019-10-13T12:06:33.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:33.988+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.992651+0000) 2019-10-13T12:06:33.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:33.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:33.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:33.995+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.998707+0000) 2019-10-13T12:06:35.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:35.479+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.483171+0000) 2019-10-13T12:06:35.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:35.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.518411+0000) 2019-10-13T12:06:36.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:36.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.452036+0000) 2019-10-13T12:06:38.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:38.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.716273+0000) 2019-10-13T12:06:38.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:38.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.720172+0000) 2019-10-13T12:06:39.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:38.995+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.001503+0000) 2019-10-13T12:06:39.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:38.995+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.001499+0000) 2019-10-13T12:06:39.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:38.995+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.002042+0000) 2019-10-13T12:06:39.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:38.995+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.002494+0000) 2019-10-13T12:06:39.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:38.995+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.002749+0000) 2019-10-13T12:06:39.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:38.995+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.003135+0000) 2019-10-13T12:06:39.004 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:38.999+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.004102+0000) 2019-10-13T12:06:39.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:39.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139920 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:39.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:39.003+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.008519+0000) 2019-10-13T12:06:39.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:39.459+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.459701+0000) 2019-10-13T12:06:39.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:39.951+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:39.952107+0000) 2019-10-13T12:06:42.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:42.207+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:42.208371+0000) 2019-10-13T12:06:44.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:44.007+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.011166+0000) 2019-10-13T12:06:44.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:44.007+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.012197+0000) 2019-10-13T12:06:44.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:44.007+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.012625+0000) 2019-10-13T12:06:44.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:44.007+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.013090+0000) 2019-10-13T12:06:44.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:44.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.013661+0000) 2019-10-13T12:06:44.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:44.007+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.014267+0000) 2019-10-13T12:06:44.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:44.007+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.014267+0000) 2019-10-13T12:06:44.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:44.011+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.017472+0000) 2019-10-13T12:06:44.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:44.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:45.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:45.479+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.483356+0000) 2019-10-13T12:06:45.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:45.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.518594+0000) 2019-10-13T12:06:46.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:46.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.452232+0000) 2019-10-13T12:06:48.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:48.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.716472+0000) 2019-10-13T12:06:48.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:48.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.720371+0000) 2019-10-13T12:06:49.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:49.015+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.022197+0000) 2019-10-13T12:06:49.023 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:49.015+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.022706+0000) 2019-10-13T12:06:49.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:49.019+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.022966+0000) 2019-10-13T12:06:49.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:49.019+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.023701+0000) 2019-10-13T12:06:49.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:49.019+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.023963+0000) 2019-10-13T12:06:49.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:49.019+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.024302+0000) 2019-10-13T12:06:49.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:49.019+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.025157+0000) 2019-10-13T12:06:49.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:49.023+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.026617+0000) 2019-10-13T12:06:49.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:49.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 139988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:49.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:49.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.466167+0000) 2019-10-13T12:06:49.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:49.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:49.952313+0000) 2019-10-13T12:06:52.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:52.207+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:52.208619+0000) 2019-10-13T12:06:53.657 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:06:53.658 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:53.661 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:06:53.661 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:54.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:54.026+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.032664+0000) 2019-10-13T12:06:54.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:54.027+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.033605+0000) 2019-10-13T12:06:54.034 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:54.027+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.034080+0000) 2019-10-13T12:06:54.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:54.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.034364+0000) 2019-10-13T12:06:54.035 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:54.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.034942+0000) 2019-10-13T12:06:54.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:54.027+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.035170+0000) 2019-10-13T12:06:54.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:54.031+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.035744+0000) 2019-10-13T12:06:54.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:54.034+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.039026+0000) 2019-10-13T12:06:54.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:54.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:55.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:55.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.483615+0000) 2019-10-13T12:06:55.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:55.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.518804+0000) 2019-10-13T12:06:56.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:56.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.452449+0000) 2019-10-13T12:06:58.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:58.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.716666+0000) 2019-10-13T12:06:58.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:58.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.720568+0000) 2019-10-13T12:06:59.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:06:59.039+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.043559+0000) 2019-10-13T12:06:59.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:06:59.039+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.044317+0000) 2019-10-13T12:06:59.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:06:59.039+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.044716+0000) 2019-10-13T12:06:59.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:06:59.039+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.045124+0000) 2019-10-13T12:06:59.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:06:59.039+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.045709+0000) 2019-10-13T12:06:59.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:59.038+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.045864+0000) 2019-10-13T12:06:59.047 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:06:59.039+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.046731+0000) 2019-10-13T12:06:59.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:06:59.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:06:59.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:59.046+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.050274+0000) 2019-10-13T12:06:59.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:06:59.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.472485+0000) 2019-10-13T12:06:59.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:06:59.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:59.952516+0000) 2019-10-13T12:07:02.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:02.207+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:02.208852+0000) 2019-10-13T12:07:04.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:04.047+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.053522+0000) 2019-10-13T12:07:04.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:04.047+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.054039+0000) 2019-10-13T12:07:04.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:04.047+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.054923+0000) 2019-10-13T12:07:04.056 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:04.051+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.055831+0000) 2019-10-13T12:07:04.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:04.051+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.056578+0000) 2019-10-13T12:07:04.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:04.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.056658+0000) 2019-10-13T12:07:04.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:04.051+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.057466+0000) 2019-10-13T12:07:04.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:04.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:04.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:04.054+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.061603+0000) 2019-10-13T12:07:05.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:05.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.483843+0000) 2019-10-13T12:07:05.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:05.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.519020+0000) 2019-10-13T12:07:06.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:06.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.452640+0000) 2019-10-13T12:07:08.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:08.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.716822+0000) 2019-10-13T12:07:08.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:08.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.720770+0000) 2019-10-13T12:07:09.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:09.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.063770+0000) 2019-10-13T12:07:09.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:09.063+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.064268+0000) 2019-10-13T12:07:09.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:09.063+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.065421+0000) 2019-10-13T12:07:09.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:09.063+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.065510+0000) 2019-10-13T12:07:09.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:09.058+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.065488+0000) 2019-10-13T12:07:09.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:09.063+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.066202+0000) 2019-10-13T12:07:09.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:09.067+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.067530+0000) 2019-10-13T12:07:09.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:09.062+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.069822+0000) 2019-10-13T12:07:09.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:09.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:09.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:09.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.483791+0000) 2019-10-13T12:07:09.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:09.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:09.952687+0000) 2019-10-13T12:07:12.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:12.207+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:12.209058+0000) 2019-10-13T12:07:14.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:14.071+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.075214+0000) 2019-10-13T12:07:14.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:14.075+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.075460+0000) 2019-10-13T12:07:14.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:14.070+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.077071+0000) 2019-10-13T12:07:14.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:14.075+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.076508+0000) 2019-10-13T12:07:14.078 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:14.075+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.077423+0000) 2019-10-13T12:07:14.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:14.075+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.077785+0000) 2019-10-13T12:07:14.079 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:14.075+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.078104+0000) 2019-10-13T12:07:14.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:14.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:14.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:14.078+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.083572+0000) 2019-10-13T12:07:15.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:15.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.484055+0000) 2019-10-13T12:07:15.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:15.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.519204+0000) 2019-10-13T12:07:16.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:16.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.452861+0000) 2019-10-13T12:07:18.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:18.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.717048+0000) 2019-10-13T12:07:18.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:18.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.720966+0000) 2019-10-13T12:07:19.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:19.082+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.085556+0000) 2019-10-13T12:07:19.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:19.082+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.086113+0000) 2019-10-13T12:07:19.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:19.082+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.086295+0000) 2019-10-13T12:07:19.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:19.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.087134+0000) 2019-10-13T12:07:19.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:19.086+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.087703+0000) 2019-10-13T12:07:19.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:19.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.088264+0000) 2019-10-13T12:07:19.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:19.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.088540+0000) 2019-10-13T12:07:19.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:19.086+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.091216+0000) 2019-10-13T12:07:19.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:19.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:19.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:19.486+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.490266+0000) 2019-10-13T12:07:19.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:19.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:19.952858+0000) 2019-10-13T12:07:22.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:22.206+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:22.209304+0000) 2019-10-13T12:07:23.799 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:07:23.800 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:23.803 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:07:23.803 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:24.096 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:24.094+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.095713+0000) 2019-10-13T12:07:24.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:24.094+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.096243+0000) 2019-10-13T12:07:24.098 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:24.094+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.097053+0000) 2019-10-13T12:07:24.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:24.094+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.097550+0000) 2019-10-13T12:07:24.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:24.090+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.097869+0000) 2019-10-13T12:07:24.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:24.094+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.098912+0000) 2019-10-13T12:07:24.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:24.098+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.099541+0000) 2019-10-13T12:07:24.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:24.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140224 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:24.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:24.102+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.107027+0000) 2019-10-13T12:07:25.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:25.482+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.484298+0000) 2019-10-13T12:07:25.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:25.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.519439+0000) 2019-10-13T12:07:26.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:26.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.453075+0000) 2019-10-13T12:07:28.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:28.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.717253+0000) 2019-10-13T12:07:28.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:28.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.721177+0000) 2019-10-13T12:07:29.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:29.102+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.105214+0000) 2019-10-13T12:07:29.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:29.102+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.105634+0000) 2019-10-13T12:07:29.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:29.102+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.105838+0000) 2019-10-13T12:07:29.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:29.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.107225+0000) 2019-10-13T12:07:29.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:29.106+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.108180+0000) 2019-10-13T12:07:29.109 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:29.106+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.108339+0000) 2019-10-13T12:07:29.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:29.102+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.108631+0000) 2019-10-13T12:07:29.112 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:29.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:29.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:29.110+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.115228+0000) 2019-10-13T12:07:29.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:29.494+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.496840+0000) 2019-10-13T12:07:29.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:29.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:29.953102+0000) 2019-10-13T12:07:32.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:32.206+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:32.209538+0000) 2019-10-13T12:07:34.116 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:34.110+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.114464+0000) 2019-10-13T12:07:34.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:34.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.115834+0000) 2019-10-13T12:07:34.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:34.114+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.116100+0000) 2019-10-13T12:07:34.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:34.110+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.116298+0000) 2019-10-13T12:07:34.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:34.114+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.117139+0000) 2019-10-13T12:07:34.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:34.114+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.118480+0000) 2019-10-13T12:07:34.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:34.114+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.118988+0000) 2019-10-13T12:07:34.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:34.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:34.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:34.118+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.122797+0000) 2019-10-13T12:07:35.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:35.482+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.484507+0000) 2019-10-13T12:07:35.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:35.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.519671+0000) 2019-10-13T12:07:36.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:36.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.453292+0000) 2019-10-13T12:07:38.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:38.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.717476+0000) 2019-10-13T12:07:38.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:38.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.721416+0000) 2019-10-13T12:07:39.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:39.122+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.125936+0000) 2019-10-13T12:07:39.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:39.122+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.126234+0000) 2019-10-13T12:07:39.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:39.126+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.127378+0000) 2019-10-13T12:07:39.128 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:39.126+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.127572+0000) 2019-10-13T12:07:39.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:39.126+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.127844+0000) 2019-10-13T12:07:39.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:39.126+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.128435+0000) 2019-10-13T12:07:39.129 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:39.126+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.128827+0000) 2019-10-13T12:07:39.133 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:39.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:39.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:39.134+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.134777+0000) 2019-10-13T12:07:39.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:39.502+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.503078+0000) 2019-10-13T12:07:39.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:39.950+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:39.953234+0000) 2019-10-13T12:07:42.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:42.206+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:42.209744+0000) 2019-10-13T12:07:44.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:44.134+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.135703+0000) 2019-10-13T12:07:44.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:44.134+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.136559+0000) 2019-10-13T12:07:44.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:44.134+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.137034+0000) 2019-10-13T12:07:44.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:44.134+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.137278+0000) 2019-10-13T12:07:44.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:44.134+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.138786+0000) 2019-10-13T12:07:44.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:44.134+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.138929+0000) 2019-10-13T12:07:44.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:44.137+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.139826+0000) 2019-10-13T12:07:44.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:44.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140360 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:44.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:44.141+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.143960+0000) 2019-10-13T12:07:45.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:45.482+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.484709+0000) 2019-10-13T12:07:45.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:45.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.519879+0000) 2019-10-13T12:07:46.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:46.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.453486+0000) 2019-10-13T12:07:48.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:48.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.717705+0000) 2019-10-13T12:07:48.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:48.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.721575+0000) 2019-10-13T12:07:49.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:49.146+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.147543+0000) 2019-10-13T12:07:49.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:49.145+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.148251+0000) 2019-10-13T12:07:49.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:49.146+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.148641+0000) 2019-10-13T12:07:49.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:49.146+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.149349+0000) 2019-10-13T12:07:49.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:49.146+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.149748+0000) 2019-10-13T12:07:49.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:49.146+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.150099+0000) 2019-10-13T12:07:49.151 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:49.146+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.150666+0000) 2019-10-13T12:07:49.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:49.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:49.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:49.153+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.154505+0000) 2019-10-13T12:07:49.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:49.509+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.512682+0000) 2019-10-13T12:07:49.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:49.949+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:49.953424+0000) 2019-10-13T12:07:52.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:52.206+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:52.209937+0000) 2019-10-13T12:07:53.931 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:07:53.931 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:53.935 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:07:53.935 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:54.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:54.153+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.156567+0000) 2019-10-13T12:07:54.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:54.154+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.158373+0000) 2019-10-13T12:07:54.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:54.154+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.158733+0000) 2019-10-13T12:07:54.160 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:54.154+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.159104+0000) 2019-10-13T12:07:54.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:54.158+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.159556+0000) 2019-10-13T12:07:54.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:54.158+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.160234+0000) 2019-10-13T12:07:54.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:54.158+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.160897+0000) 2019-10-13T12:07:54.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:54.161+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.163609+0000) 2019-10-13T12:07:54.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:54.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:55.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:55.482+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.484930+0000) 2019-10-13T12:07:55.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:55.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.520104+0000) 2019-10-13T12:07:56.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:56.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.453709+0000) 2019-10-13T12:07:58.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:58.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.717918+0000) 2019-10-13T12:07:58.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:58.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.721785+0000) 2019-10-13T12:07:59.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:07:59.162+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.167024+0000) 2019-10-13T12:07:59.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:07:59.166+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.167452+0000) 2019-10-13T12:07:59.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:07:59.166+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.167892+0000) 2019-10-13T12:07:59.169 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:59.165+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.168105+0000) 2019-10-13T12:07:59.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:07:59.166+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.168774+0000) 2019-10-13T12:07:59.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:07:59.166+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.169544+0000) 2019-10-13T12:07:59.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:07:59.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.169970+0000) 2019-10-13T12:07:59.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:07:59.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:07:59.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:59.173+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.176841+0000) 2019-10-13T12:07:59.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:07:59.517+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.519324+0000) 2019-10-13T12:07:59.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:07:59.949+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:59.953599+0000) 2019-10-13T12:08:02.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:02.205+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:02.210149+0000) 2019-10-13T12:08:04.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:04.173+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.177006+0000) 2019-10-13T12:08:04.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:04.173+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.177953+0000) 2019-10-13T12:08:04.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:04.173+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.178529+0000) 2019-10-13T12:08:04.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:04.173+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.178739+0000) 2019-10-13T12:08:04.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:04.177+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.179992+0000) 2019-10-13T12:08:04.181 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:04.177+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.180247+0000) 2019-10-13T12:08:04.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:04.177+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.180867+0000) 2019-10-13T12:08:04.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:04.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140496 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:04.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:04.181+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.184839+0000) 2019-10-13T12:08:05.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:05.481+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.485084+0000) 2019-10-13T12:08:05.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:05.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.520307+0000) 2019-10-13T12:08:06.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:06.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.453929+0000) 2019-10-13T12:08:08.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:08.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.718118+0000) 2019-10-13T12:08:08.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:08.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.721998+0000) 2019-10-13T12:08:09.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:09.185+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.187376+0000) 2019-10-13T12:08:09.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:09.185+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.187851+0000) 2019-10-13T12:08:09.189 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:09.185+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.188545+0000) 2019-10-13T12:08:09.190 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:09.185+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.189199+0000) 2019-10-13T12:08:09.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:09.185+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.189778+0000) 2019-10-13T12:08:09.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:09.185+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.190575+0000) 2019-10-13T12:08:09.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:09.185+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.190804+0000) 2019-10-13T12:08:09.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:09.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:09.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:09.193+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.195298+0000) 2019-10-13T12:08:09.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:09.525+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.525786+0000) 2019-10-13T12:08:09.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:09.953+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:09.954360+0000) 2019-10-13T12:08:12.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:12.205+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:12.210344+0000) 2019-10-13T12:08:14.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:14.193+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.197474+0000) 2019-10-13T12:08:14.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:14.197+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.198189+0000) 2019-10-13T12:08:14.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:14.193+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.198873+0000) 2019-10-13T12:08:14.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:14.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.199382+0000) 2019-10-13T12:08:14.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:14.197+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.200377+0000) 2019-10-13T12:08:14.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:14.197+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.200810+0000) 2019-10-13T12:08:14.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:14.197+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.200117+0000) 2019-10-13T12:08:14.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:14.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:14.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:14.205+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.207116+0000) 2019-10-13T12:08:15.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:15.481+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.485282+0000) 2019-10-13T12:08:15.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:15.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.520508+0000) 2019-10-13T12:08:16.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:16.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.454151+0000) 2019-10-13T12:08:18.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:18.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.718351+0000) 2019-10-13T12:08:18.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:18.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.722193+0000) 2019-10-13T12:08:19.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:19.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.208198+0000) 2019-10-13T12:08:19.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:19.205+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.208696+0000) 2019-10-13T12:08:19.210 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:19.205+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.208898+0000) 2019-10-13T12:08:19.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:19.205+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.209213+0000) 2019-10-13T12:08:19.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:19.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.209409+0000) 2019-10-13T12:08:19.211 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:19.205+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.210200+0000) 2019-10-13T12:08:19.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:19.209+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.211390+0000) 2019-10-13T12:08:19.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:19.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:19.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:19.213+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.214896+0000) 2019-10-13T12:08:19.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:19.529+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.532361+0000) 2019-10-13T12:08:19.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:19.953+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:19.954564+0000) 2019-10-13T12:08:22.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:22.209+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:22.210548+0000) 2019-10-13T12:08:24.084 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:08:24.084 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:24.088 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:08:24.088 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:24.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:24.213+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.217567+0000) 2019-10-13T12:08:24.219 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:24.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.218163+0000) 2019-10-13T12:08:24.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:24.217+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.218304+0000) 2019-10-13T12:08:24.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:24.213+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.219078+0000) 2019-10-13T12:08:24.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:24.217+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.219819+0000) 2019-10-13T12:08:24.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:24.217+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.220085+0000) 2019-10-13T12:08:24.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:24.217+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.221147+0000) 2019-10-13T12:08:24.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:24.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:24.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:24.225+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.226176+0000) 2019-10-13T12:08:25.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:25.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.485484+0000) 2019-10-13T12:08:25.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:25.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.520741+0000) 2019-10-13T12:08:26.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:26.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.454366+0000) 2019-10-13T12:08:28.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:28.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.718544+0000) 2019-10-13T12:08:28.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:28.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.722403+0000) 2019-10-13T12:08:29.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:29.225+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.227657+0000) 2019-10-13T12:08:29.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:29.225+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.228213+0000) 2019-10-13T12:08:29.229 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:29.225+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.228638+0000) 2019-10-13T12:08:29.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:29.225+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.229390+0000) 2019-10-13T12:08:29.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:29.225+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.230143+0000) 2019-10-13T12:08:29.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:29.225+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.230707+0000) 2019-10-13T12:08:29.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:29.225+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.231259+0000) 2019-10-13T12:08:29.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:29.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:29.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:29.233+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.235995+0000) 2019-10-13T12:08:29.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:29.537+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.538859+0000) 2019-10-13T12:08:29.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:29.953+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:29.954743+0000) 2019-10-13T12:08:32.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:32.209+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:32.210751+0000) 2019-10-13T12:08:34.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:34.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.239402+0000) 2019-10-13T12:08:34.241 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:34.237+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.240006+0000) 2019-10-13T12:08:34.241 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:34.237+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.240134+0000) 2019-10-13T12:08:34.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:34.237+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.240506+0000) 2019-10-13T12:08:34.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:34.237+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.240809+0000) 2019-10-13T12:08:34.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:34.237+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.241532+0000) 2019-10-13T12:08:34.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:34.237+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.242105+0000) 2019-10-13T12:08:34.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:34.241+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:34.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:34.249+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.250237+0000) 2019-10-13T12:08:35.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:35.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.485685+0000) 2019-10-13T12:08:35.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:35.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.520943+0000) 2019-10-13T12:08:36.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:36.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.454559+0000) 2019-10-13T12:08:38.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:38.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.718769+0000) 2019-10-13T12:08:38.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:38.717+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.722599+0000) 2019-10-13T12:08:39.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:39.245+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.249031+0000) 2019-10-13T12:08:39.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:39.245+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.249723+0000) 2019-10-13T12:08:39.251 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:39.245+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.249934+0000) 2019-10-13T12:08:39.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:39.245+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.250280+0000) 2019-10-13T12:08:39.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:39.248+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.250848+0000) 2019-10-13T12:08:39.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:39.249+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.251354+0000) 2019-10-13T12:08:39.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:39.249+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.252084+0000) 2019-10-13T12:08:39.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:39.253+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:39.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:39.256+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.258763+0000) 2019-10-13T12:08:39.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:39.544+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.545563+0000) 2019-10-13T12:08:39.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:39.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:39.954884+0000) 2019-10-13T12:08:42.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:42.209+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:42.210952+0000) 2019-10-13T12:08:44.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:44.257+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.260162+0000) 2019-10-13T12:08:44.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:44.257+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.260947+0000) 2019-10-13T12:08:44.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:44.257+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.261502+0000) 2019-10-13T12:08:44.262 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:44.257+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.261945+0000) 2019-10-13T12:08:44.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:44.257+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.262493+0000) 2019-10-13T12:08:44.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:44.257+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.263047+0000) 2019-10-13T12:08:44.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:44.260+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.263167+0000) 2019-10-13T12:08:44.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:44.260+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.265385+0000) 2019-10-13T12:08:44.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:44.265+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:45.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:45.480+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.485914+0000) 2019-10-13T12:08:45.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:45.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.521177+0000) 2019-10-13T12:08:46.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:46.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.454748+0000) 2019-10-13T12:08:48.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:48.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.718997+0000) 2019-10-13T12:08:48.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:48.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.722836+0000) 2019-10-13T12:08:49.271 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:49.264+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.270417+0000) 2019-10-13T12:08:49.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:49.268+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.271887+0000) 2019-10-13T12:08:49.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:49.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.272345+0000) 2019-10-13T12:08:49.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:49.268+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.272523+0000) 2019-10-13T12:08:49.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:49.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.272800+0000) 2019-10-13T12:08:49.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:49.268+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.273312+0000) 2019-10-13T12:08:49.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:49.268+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.274420+0000) 2019-10-13T12:08:49.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:49.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:49.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:49.276+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.281328+0000) 2019-10-13T12:08:49.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:49.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.551926+0000) 2019-10-13T12:08:49.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:49.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:49.955083+0000) 2019-10-13T12:08:52.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:52.204+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:52.211152+0000) 2019-10-13T12:08:54.161 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:08:54.162 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:54.166 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:08:54.166 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:54.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:54.276+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.282452+0000) 2019-10-13T12:08:54.284 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:54.276+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.282845+0000) 2019-10-13T12:08:54.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:54.280+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.283480+0000) 2019-10-13T12:08:54.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:54.280+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.283640+0000) 2019-10-13T12:08:54.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:54.280+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.284336+0000) 2019-10-13T12:08:54.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:54.280+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.284798+0000) 2019-10-13T12:08:54.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:54.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.285318+0000) 2019-10-13T12:08:54.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:54.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:54.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:54.288+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.290547+0000) 2019-10-13T12:08:55.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:55.484+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.486145+0000) 2019-10-13T12:08:55.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:55.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.521380+0000) 2019-10-13T12:08:56.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:56.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.454959+0000) 2019-10-13T12:08:58.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:58.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.719199+0000) 2019-10-13T12:08:58.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:58.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.723028+0000) 2019-10-13T12:08:59.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:08:59.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.291902+0000) 2019-10-13T12:08:59.293 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:08:59.288+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.292558+0000) 2019-10-13T12:08:59.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:59.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.293147+0000) 2019-10-13T12:08:59.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:08:59.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.293222+0000) 2019-10-13T12:08:59.294 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:08:59.288+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.293553+0000) 2019-10-13T12:08:59.295 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:08:59.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.293976+0000) 2019-10-13T12:08:59.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:08:59.288+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.294290+0000) 2019-10-13T12:08:59.300 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:08:59.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:08:59.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:59.296+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.301185+0000) 2019-10-13T12:08:59.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:08:59.556+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.558449+0000) 2019-10-13T12:08:59.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:08:59.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:59.955234+0000) 2019-10-13T12:09:02.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:02.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:02.211346+0000) 2019-10-13T12:09:04.304 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:04.300+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.303437+0000) 2019-10-13T12:09:04.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:04.300+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.303510+0000) 2019-10-13T12:09:04.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:04.300+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.304881+0000) 2019-10-13T12:09:04.306 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:04.300+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.305145+0000) 2019-10-13T12:09:04.306 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:04.300+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.305490+0000) 2019-10-13T12:09:04.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:04.300+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.306115+0000) 2019-10-13T12:09:04.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:04.300+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.306662+0000) 2019-10-13T12:09:04.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:04.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:04.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:04.308+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.311951+0000) 2019-10-13T12:09:05.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:05.480+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.486344+0000) 2019-10-13T12:09:05.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:05.516+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.521593+0000) 2019-10-13T12:09:06.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:06.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.455155+0000) 2019-10-13T12:09:08.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:08.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.719395+0000) 2019-10-13T12:09:08.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:08.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.723226+0000) 2019-10-13T12:09:09.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:09.308+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.314033+0000) 2019-10-13T12:09:09.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:09.308+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.314698+0000) 2019-10-13T12:09:09.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:09.308+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.315051+0000) 2019-10-13T12:09:09.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:09.312+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.315394+0000) 2019-10-13T12:09:09.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:09.312+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.315787+0000) 2019-10-13T12:09:09.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:09.312+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.316440+0000) 2019-10-13T12:09:09.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:09.312+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.316927+0000) 2019-10-13T12:09:09.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:09.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:09.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:09.320+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.322898+0000) 2019-10-13T12:09:09.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:09.560+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.565026+0000) 2019-10-13T12:09:09.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:09.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:09.955406+0000) 2019-10-13T12:09:12.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:12.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:12.211554+0000) 2019-10-13T12:09:14.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:14.320+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.324612+0000) 2019-10-13T12:09:14.326 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:14.320+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.324913+0000) 2019-10-13T12:09:14.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:14.320+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.325436+0000) 2019-10-13T12:09:14.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:14.320+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.326888+0000) 2019-10-13T12:09:14.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:14.324+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.326975+0000) 2019-10-13T12:09:14.328 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:14.324+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.327463+0000) 2019-10-13T12:09:14.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:14.324+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.327725+0000) 2019-10-13T12:09:14.332 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:14.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 140967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:14.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:14.328+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.332089+0000) 2019-10-13T12:09:15.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:15.484+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.486547+0000) 2019-10-13T12:09:15.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:15.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.521771+0000) 2019-10-13T12:09:16.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:16.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.455350+0000) 2019-10-13T12:09:18.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:18.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.719591+0000) 2019-10-13T12:09:18.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:18.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.723396+0000) 2019-10-13T12:09:19.337 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:19.332+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.335992+0000) 2019-10-13T12:09:19.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:19.332+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.336389+0000) 2019-10-13T12:09:19.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:19.332+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.336645+0000) 2019-10-13T12:09:19.338 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:19.332+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.337349+0000) 2019-10-13T12:09:19.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:19.332+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.337081+0000) 2019-10-13T12:09:19.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:19.332+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.338389+0000) 2019-10-13T12:09:19.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:19.336+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.338394+0000) 2019-10-13T12:09:19.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:19.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:19.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:19.340+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.345020+0000) 2019-10-13T12:09:19.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:19.576+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.576754+0000) 2019-10-13T12:09:19.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:19.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:19.955574+0000) 2019-10-13T12:09:22.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:22.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:22.211774+0000) 2019-10-13T12:09:24.264 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:09:24.264 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:24.268 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:09:24.268 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:24.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:24.340+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.345876+0000) 2019-10-13T12:09:24.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:24.340+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.346540+0000) 2019-10-13T12:09:24.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:24.344+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.347614+0000) 2019-10-13T12:09:24.349 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:24.344+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.348095+0000) 2019-10-13T12:09:24.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:24.344+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.348559+0000) 2019-10-13T12:09:24.350 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:24.344+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.349309+0000) 2019-10-13T12:09:24.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:24.344+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.349386+0000) 2019-10-13T12:09:24.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:24.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:24.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:24.352+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.355259+0000) 2019-10-13T12:09:25.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:25.484+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.486736+0000) 2019-10-13T12:09:25.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:25.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.521972+0000) 2019-10-13T12:09:26.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:26.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.455544+0000) 2019-10-13T12:09:28.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:28.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.719783+0000) 2019-10-13T12:09:28.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:28.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.723628+0000) 2019-10-13T12:09:29.357 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:29.351+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.356108+0000) 2019-10-13T12:09:29.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:29.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.356586+0000) 2019-10-13T12:09:29.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:29.351+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.358111+0000) 2019-10-13T12:09:29.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:29.351+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.358473+0000) 2019-10-13T12:09:29.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:29.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.358958+0000) 2019-10-13T12:09:29.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:29.355+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.359527+0000) 2019-10-13T12:09:29.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:29.355+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.360238+0000) 2019-10-13T12:09:29.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:29.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:29.367 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:29.360+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.366083+0000) 2019-10-13T12:09:29.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:29.580+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.583143+0000) 2019-10-13T12:09:29.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:29.952+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:29.955748+0000) 2019-10-13T12:09:32.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:32.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:32.211972+0000) 2019-10-13T12:09:34.369 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:34.363+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.367660+0000) 2019-10-13T12:09:34.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:34.363+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.368144+0000) 2019-10-13T12:09:34.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:34.363+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.369049+0000) 2019-10-13T12:09:34.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:34.363+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.369349+0000) 2019-10-13T12:09:34.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:34.363+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.370038+0000) 2019-10-13T12:09:34.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:34.363+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.370796+0000) 2019-10-13T12:09:34.372 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:34.367+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.371030+0000) 2019-10-13T12:09:34.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:34.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:34.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:34.371+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.376901+0000) 2019-10-13T12:09:35.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:35.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.486969+0000) 2019-10-13T12:09:35.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:35.519+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.522164+0000) 2019-10-13T12:09:36.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:36.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.455768+0000) 2019-10-13T12:09:38.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:38.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.719987+0000) 2019-10-13T12:09:38.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:38.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.723780+0000) 2019-10-13T12:09:39.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:39.371+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.377389+0000) 2019-10-13T12:09:39.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:39.371+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.377850+0000) 2019-10-13T12:09:39.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:39.371+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.378487+0000) 2019-10-13T12:09:39.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:39.371+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.379064+0000) 2019-10-13T12:09:39.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:39.371+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.379041+0000) 2019-10-13T12:09:39.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:39.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.380080+0000) 2019-10-13T12:09:39.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:39.375+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.380839+0000) 2019-10-13T12:09:39.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:39.379+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.382516+0000) 2019-10-13T12:09:39.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:39.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:39.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:39.583+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.589572+0000) 2019-10-13T12:09:39.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:39.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:39.955940+0000) 2019-10-13T12:09:42.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:42.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:42.212172+0000) 2019-10-13T12:09:44.389 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:44.383+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.387735+0000) 2019-10-13T12:09:44.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:44.383+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.388256+0000) 2019-10-13T12:09:44.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:44.383+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.388935+0000) 2019-10-13T12:09:44.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:44.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.389707+0000) 2019-10-13T12:09:44.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:44.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.389798+0000) 2019-10-13T12:09:44.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:44.383+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.390650+0000) 2019-10-13T12:09:44.391 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:44.387+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.390800+0000) 2019-10-13T12:09:44.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:44.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:44.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:44.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.398904+0000) 2019-10-13T12:09:45.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:45.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.487145+0000) 2019-10-13T12:09:45.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:45.519+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.522373+0000) 2019-10-13T12:09:46.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:46.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.455964+0000) 2019-10-13T12:09:48.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:48.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.720185+0000) 2019-10-13T12:09:48.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:48.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.723994+0000) 2019-10-13T12:09:49.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:49.395+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.399627+0000) 2019-10-13T12:09:49.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:49.395+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.399996+0000) 2019-10-13T12:09:49.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:49.395+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.400248+0000) 2019-10-13T12:09:49.401 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:49.395+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.400980+0000) 2019-10-13T12:09:49.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:49.395+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.401362+0000) 2019-10-13T12:09:49.402 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:49.395+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.401765+0000) 2019-10-13T12:09:49.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:49.395+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.402194+0000) 2019-10-13T12:09:49.406 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:49.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:49.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:49.403+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.409209+0000) 2019-10-13T12:09:49.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:49.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.595972+0000) 2019-10-13T12:09:49.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:49.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:49.956135+0000) 2019-10-13T12:09:52.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:52.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:52.212374+0000) 2019-10-13T12:09:54.374 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:09:54.375 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:54.378 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:09:54.378 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:54.409 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:54.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.409011+0000) 2019-10-13T12:09:54.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:54.403+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.408983+0000) 2019-10-13T12:09:54.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:54.403+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.409577+0000) 2019-10-13T12:09:54.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:54.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.410230+0000) 2019-10-13T12:09:54.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:54.403+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.410352+0000) 2019-10-13T12:09:54.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:54.403+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.410840+0000) 2019-10-13T12:09:54.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:54.407+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.411876+0000) 2019-10-13T12:09:54.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:54.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:54.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:54.411+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.417153+0000) 2019-10-13T12:09:55.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:55.483+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.487362+0000) 2019-10-13T12:09:55.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:55.519+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.522586+0000) 2019-10-13T12:09:56.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:56.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.456175+0000) 2019-10-13T12:09:58.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:58.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.720375+0000) 2019-10-13T12:09:58.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:58.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.724236+0000) 2019-10-13T12:09:59.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:09:59.419+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.418741+0000) 2019-10-13T12:09:59.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:09:59.419+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.419447+0000) 2019-10-13T12:09:59.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:09:59.419+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.420073+0000) 2019-10-13T12:09:59.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:09:59.419+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.420640+0000) 2019-10-13T12:09:59.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:09:59.419+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.421909+0000) 2019-10-13T12:09:59.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:59.415+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.421484+0000) 2019-10-13T12:09:59.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:09:59.419+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.422855+0000) 2019-10-13T12:09:59.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:09:59.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:09:59.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:59.423+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.426812+0000) 2019-10-13T12:09:59.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:09:59.599+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.602555+0000) 2019-10-13T12:09:59.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:09:59.951+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:59.956307+0000) 2019-10-13T12:10:02.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:02.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:02.212601+0000) 2019-10-13T12:10:04.430 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:04.427+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.429724+0000) 2019-10-13T12:10:04.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:04.427+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.430089+0000) 2019-10-13T12:10:04.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:04.427+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.430643+0000) 2019-10-13T12:10:04.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:04.431+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.431355+0000) 2019-10-13T12:10:04.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:04.431+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.431779+0000) 2019-10-13T12:10:04.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:04.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.432131+0000) 2019-10-13T12:10:04.433 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:04.427+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.432583+0000) 2019-10-13T12:10:04.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:04.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:04.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:04.435+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.440928+0000) 2019-10-13T12:10:05.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:05.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.487546+0000) 2019-10-13T12:10:05.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:05.519+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.522779+0000) 2019-10-13T12:10:06.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:06.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.456361+0000) 2019-10-13T12:10:08.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:08.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.720571+0000) 2019-10-13T12:10:08.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:08.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.724446+0000) 2019-10-13T12:10:09.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:09.439+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.439768+0000) 2019-10-13T12:10:09.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:09.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.439891+0000) 2019-10-13T12:10:09.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:09.439+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.440614+0000) 2019-10-13T12:10:09.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:09.439+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.440837+0000) 2019-10-13T12:10:09.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:09.439+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.441577+0000) 2019-10-13T12:10:09.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:09.439+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.442407+0000) 2019-10-13T12:10:09.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:09.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.442504+0000) 2019-10-13T12:10:09.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:09.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141336 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:09.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:09.443+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.449226+0000) 2019-10-13T12:10:09.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:09.607+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.608974+0000) 2019-10-13T12:10:09.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:09.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:09.956475+0000) 2019-10-13T12:10:12.213 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:12.210+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:12.212824+0000) 2019-10-13T12:10:14.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:14.446+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.450372+0000) 2019-10-13T12:10:14.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:14.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.450227+0000) 2019-10-13T12:10:14.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:14.446+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.450934+0000) 2019-10-13T12:10:14.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:14.446+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.450854+0000) 2019-10-13T12:10:14.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:14.450+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.451384+0000) 2019-10-13T12:10:14.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:14.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.452303+0000) 2019-10-13T12:10:14.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:14.450+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.452158+0000) 2019-10-13T12:10:14.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:14.451+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.457778+0000) 2019-10-13T12:10:14.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:14.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:15.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:15.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.487721+0000) 2019-10-13T12:10:15.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:15.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.522992+0000) 2019-10-13T12:10:16.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:16.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.456583+0000) 2019-10-13T12:10:18.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:18.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.720795+0000) 2019-10-13T12:10:18.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:18.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.724679+0000) 2019-10-13T12:10:19.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:19.455+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.459939+0000) 2019-10-13T12:10:19.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:19.458+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.460348+0000) 2019-10-13T12:10:19.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:19.458+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.460258+0000) 2019-10-13T12:10:19.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:19.458+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.461751+0000) 2019-10-13T12:10:19.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:19.458+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.462035+0000) 2019-10-13T12:10:19.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:19.458+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.462004+0000) 2019-10-13T12:10:19.463 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:19.458+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.462917+0000) 2019-10-13T12:10:19.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:19.466+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:19.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:19.463+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.469637+0000) 2019-10-13T12:10:19.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:19.615+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.620776+0000) 2019-10-13T12:10:19.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:19.951+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:19.956650+0000) 2019-10-13T12:10:22.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:22.210+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:22.213023+0000) 2019-10-13T12:10:24.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:24.466+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.471178+0000) 2019-10-13T12:10:24.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:24.470+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.471678+0000) 2019-10-13T12:10:24.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:24.470+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.472120+0000) 2019-10-13T12:10:24.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:24.467+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.472304+0000) 2019-10-13T12:10:24.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:24.470+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.472336+0000) 2019-10-13T12:10:24.474 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:24.470+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.473567+0000) 2019-10-13T12:10:24.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:24.470+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.474250+0000) 2019-10-13T12:10:24.478 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:10:24.478 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:24.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:24.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:24.482 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:10:24.482 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:24.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:24.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.483781+0000) 2019-10-13T12:10:25.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:25.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.487892+0000) 2019-10-13T12:10:25.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:25.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.523198+0000) 2019-10-13T12:10:26.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:26.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.456779+0000) 2019-10-13T12:10:28.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:28.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.720985+0000) 2019-10-13T12:10:28.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:28.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.724840+0000) 2019-10-13T12:10:29.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:29.478+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.481725+0000) 2019-10-13T12:10:29.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:29.478+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.482427+0000) 2019-10-13T12:10:29.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:29.478+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.482857+0000) 2019-10-13T12:10:29.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:29.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.483336+0000) 2019-10-13T12:10:29.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:29.482+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.483962+0000) 2019-10-13T12:10:29.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:29.482+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.484365+0000) 2019-10-13T12:10:29.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:29.478+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.485086+0000) 2019-10-13T12:10:29.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:29.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:29.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:29.486+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.491320+0000) 2019-10-13T12:10:29.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:29.622+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.626538+0000) 2019-10-13T12:10:29.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:29.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:29.956831+0000) 2019-10-13T12:10:32.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:32.210+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:32.213221+0000) 2019-10-13T12:10:34.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:34.490+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.491399+0000) 2019-10-13T12:10:34.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:34.490+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.492229+0000) 2019-10-13T12:10:34.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:34.490+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.493118+0000) 2019-10-13T12:10:34.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:34.490+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.493648+0000) 2019-10-13T12:10:34.494 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:34.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.493975+0000) 2019-10-13T12:10:34.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:34.490+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.494366+0000) 2019-10-13T12:10:34.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:34.490+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.495162+0000) 2019-10-13T12:10:34.499 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:34.494+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:34.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:34.494+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.500836+0000) 2019-10-13T12:10:35.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:35.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.488115+0000) 2019-10-13T12:10:35.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:35.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.523389+0000) 2019-10-13T12:10:36.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:36.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.456937+0000) 2019-10-13T12:10:38.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:38.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.721193+0000) 2019-10-13T12:10:38.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:38.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.725031+0000) 2019-10-13T12:10:39.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:39.498+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.502365+0000) 2019-10-13T12:10:39.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:39.498+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.502812+0000) 2019-10-13T12:10:39.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:39.502+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.503510+0000) 2019-10-13T12:10:39.505 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:39.502+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.504117+0000) 2019-10-13T12:10:39.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:39.502+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.504933+0000) 2019-10-13T12:10:39.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:39.502+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.505333+0000) 2019-10-13T12:10:39.506 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:39.498+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.505555+0000) 2019-10-13T12:10:39.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:39.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:39.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:39.506+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.510949+0000) 2019-10-13T12:10:39.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:39.630+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.634111+0000) 2019-10-13T12:10:39.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:39.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:39.957005+0000) 2019-10-13T12:10:42.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:42.210+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:42.213427+0000) 2019-10-13T12:10:44.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:44.510+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.512091+0000) 2019-10-13T12:10:44.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:44.510+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.513108+0000) 2019-10-13T12:10:44.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:44.510+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.513328+0000) 2019-10-13T12:10:44.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:44.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.513734+0000) 2019-10-13T12:10:44.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:44.510+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.514268+0000) 2019-10-13T12:10:44.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:44.510+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.515112+0000) 2019-10-13T12:10:44.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:44.514+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.515542+0000) 2019-10-13T12:10:44.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:44.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:44.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:44.518+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.522731+0000) 2019-10-13T12:10:45.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:45.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.488301+0000) 2019-10-13T12:10:45.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:45.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.523606+0000) 2019-10-13T12:10:46.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:46.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.457119+0000) 2019-10-13T12:10:48.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:48.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.721396+0000) 2019-10-13T12:10:48.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:48.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.725258+0000) 2019-10-13T12:10:49.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:49.518+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.523086+0000) 2019-10-13T12:10:49.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:49.522+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.523650+0000) 2019-10-13T12:10:49.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:49.522+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.524051+0000) 2019-10-13T12:10:49.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:49.518+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.524336+0000) 2019-10-13T12:10:49.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:49.522+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.525221+0000) 2019-10-13T12:10:49.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:49.522+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.525579+0000) 2019-10-13T12:10:49.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:49.522+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.526619+0000) 2019-10-13T12:10:49.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:49.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:49.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:49.526+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.531934+0000) 2019-10-13T12:10:49.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:49.638+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.640869+0000) 2019-10-13T12:10:49.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:49.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:49.957185+0000) 2019-10-13T12:10:52.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:52.210+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:52.213634+0000) 2019-10-13T12:10:54.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:54.529+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.533217+0000) 2019-10-13T12:10:54.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:54.529+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.533931+0000) 2019-10-13T12:10:54.535 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:54.529+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.534646+0000) 2019-10-13T12:10:54.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:54.529+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.534766+0000) 2019-10-13T12:10:54.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:54.530+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.534802+0000) 2019-10-13T12:10:54.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:54.533+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.535871+0000) 2019-10-13T12:10:54.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:54.533+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.536506+0000) 2019-10-13T12:10:54.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:54.537+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:54.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:54.538+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.543229+0000) 2019-10-13T12:10:54.581 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:10:54.582 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:54.586 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:10:54.586 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:55.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:55.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.488498+0000) 2019-10-13T12:10:55.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:55.521+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.523825+0000) 2019-10-13T12:10:56.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:56.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.457315+0000) 2019-10-13T12:10:58.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:58.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.721613+0000) 2019-10-13T12:10:58.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:58.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.725456+0000) 2019-10-13T12:10:59.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:59.538+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.544532+0000) 2019-10-13T12:10:59.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:10:59.541+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.545014+0000) 2019-10-13T12:10:59.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:10:59.541+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.545350+0000) 2019-10-13T12:10:59.547 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:10:59.541+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.545825+0000) 2019-10-13T12:10:59.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:10:59.541+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.546691+0000) 2019-10-13T12:10:59.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:10:59.541+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.546391+0000) 2019-10-13T12:10:59.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:10:59.541+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.547241+0000) 2019-10-13T12:10:59.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:10:59.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:10:59.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:59.550+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.554672+0000) 2019-10-13T12:10:59.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:10:59.650+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.652446+0000) 2019-10-13T12:10:59.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:10:59.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:59.957355+0000) 2019-10-13T12:11:02.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:02.209+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:02.213830+0000) 2019-10-13T12:11:04.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:04.553+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.555858+0000) 2019-10-13T12:11:04.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:04.550+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.555998+0000) 2019-10-13T12:11:04.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:04.553+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.556506+0000) 2019-10-13T12:11:04.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:04.553+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.556723+0000) 2019-10-13T12:11:04.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:04.553+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.557029+0000) 2019-10-13T12:11:04.559 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:04.553+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.557476+0000) 2019-10-13T12:11:04.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:04.553+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.558056+0000) 2019-10-13T12:11:04.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:04.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:04.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:04.562+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.567136+0000) 2019-10-13T12:11:05.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:05.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.488694+0000) 2019-10-13T12:11:05.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:05.521+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.524023+0000) 2019-10-13T12:11:06.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:06.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.457529+0000) 2019-10-13T12:11:08.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:08.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.721810+0000) 2019-10-13T12:11:08.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:08.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.725675+0000) 2019-10-13T12:11:09.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:09.561+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.565750+0000) 2019-10-13T12:11:09.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:09.561+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.566202+0000) 2019-10-13T12:11:09.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:09.561+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.567175+0000) 2019-10-13T12:11:09.568 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:09.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.567626+0000) 2019-10-13T12:11:09.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:09.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.569017+0000) 2019-10-13T12:11:09.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:09.565+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.568539+0000) 2019-10-13T12:11:09.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:09.565+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.569054+0000) 2019-10-13T12:11:09.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:09.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:09.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:09.570+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.574350+0000) 2019-10-13T12:11:09.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:09.658+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.658959+0000) 2019-10-13T12:11:09.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:09.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:09.957528+0000) 2019-10-13T12:11:12.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:12.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:12.214004+0000) 2019-10-13T12:11:14.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:14.573+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.576755+0000) 2019-10-13T12:11:14.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:14.573+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.577131+0000) 2019-10-13T12:11:14.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:14.573+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.577747+0000) 2019-10-13T12:11:14.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:14.573+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.577956+0000) 2019-10-13T12:11:14.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:14.574+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.579116+0000) 2019-10-13T12:11:14.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:14.573+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.578594+0000) 2019-10-13T12:11:14.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:14.577+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.580342+0000) 2019-10-13T12:11:14.584 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:14.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:14.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:14.582+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.586264+0000) 2019-10-13T12:11:15.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:15.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.488870+0000) 2019-10-13T12:11:15.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:15.521+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.524199+0000) 2019-10-13T12:11:16.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:16.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.457723+0000) 2019-10-13T12:11:18.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:18.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.722000+0000) 2019-10-13T12:11:18.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:18.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.725926+0000) 2019-10-13T12:11:19.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:19.581+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.586257+0000) 2019-10-13T12:11:19.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:19.581+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.586914+0000) 2019-10-13T12:11:19.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:19.586+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.587461+0000) 2019-10-13T12:11:19.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:19.585+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.588004+0000) 2019-10-13T12:11:19.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:19.585+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.588775+0000) 2019-10-13T12:11:19.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:19.585+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.589143+0000) 2019-10-13T12:11:19.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:19.585+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.589385+0000) 2019-10-13T12:11:19.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:19.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:19.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:19.594+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.594246+0000) 2019-10-13T12:11:19.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:19.662+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.665307+0000) 2019-10-13T12:11:19.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:19.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:19.957699+0000) 2019-10-13T12:11:22.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:22.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:22.214212+0000) 2019-10-13T12:11:24.598 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:24.593+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.597146+0000) 2019-10-13T12:11:24.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:24.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.597654+0000) 2019-10-13T12:11:24.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:24.593+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.597944+0000) 2019-10-13T12:11:24.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:24.593+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.598284+0000) 2019-10-13T12:11:24.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:24.593+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.598491+0000) 2019-10-13T12:11:24.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:24.597+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.599485+0000) 2019-10-13T12:11:24.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:24.597+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.600663+0000) 2019-10-13T12:11:24.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:24.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:24.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:24.605+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.606549+0000) 2019-10-13T12:11:24.626 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:11:24.626 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:24.629 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:11:24.629 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:25.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:25.485+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.489047+0000) 2019-10-13T12:11:25.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:25.521+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.524406+0000) 2019-10-13T12:11:26.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:26.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.457915+0000) 2019-10-13T12:11:28.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:28.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.722195+0000) 2019-10-13T12:11:28.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:28.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.726118+0000) 2019-10-13T12:11:29.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:29.601+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.607003+0000) 2019-10-13T12:11:29.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:29.605+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.608079+0000) 2019-10-13T12:11:29.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:29.605+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.608562+0000) 2019-10-13T12:11:29.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:29.605+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.609152+0000) 2019-10-13T12:11:29.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:29.605+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.609562+0000) 2019-10-13T12:11:29.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:29.605+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.609990+0000) 2019-10-13T12:11:29.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:29.605+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.610685+0000) 2019-10-13T12:11:29.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:29.613+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.614986+0000) 2019-10-13T12:11:29.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:29.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:29.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:29.669+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.671581+0000) 2019-10-13T12:11:29.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:29.953+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:29.957874+0000) 2019-10-13T12:11:32.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:32.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:32.214414+0000) 2019-10-13T12:11:34.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:34.613+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.619204+0000) 2019-10-13T12:11:34.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:34.617+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.619701+0000) 2019-10-13T12:11:34.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:34.617+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.619824+0000) 2019-10-13T12:11:34.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:34.617+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.620621+0000) 2019-10-13T12:11:34.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:34.617+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.621276+0000) 2019-10-13T12:11:34.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:34.617+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.621314+0000) 2019-10-13T12:11:34.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:34.617+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.621757+0000) 2019-10-13T12:11:34.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:34.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:34.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:34.625+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.626520+0000) 2019-10-13T12:11:35.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:35.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.489227+0000) 2019-10-13T12:11:35.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:35.521+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.524597+0000) 2019-10-13T12:11:36.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:36.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.458108+0000) 2019-10-13T12:11:38.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:38.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.722392+0000) 2019-10-13T12:11:38.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:38.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.726345+0000) 2019-10-13T12:11:39.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:39.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.629375+0000) 2019-10-13T12:11:39.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:39.624+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.629857+0000) 2019-10-13T12:11:39.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:39.624+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.629848+0000) 2019-10-13T12:11:39.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:39.624+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.630631+0000) 2019-10-13T12:11:39.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:39.629+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.630865+0000) 2019-10-13T12:11:39.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:39.624+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.631264+0000) 2019-10-13T12:11:39.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:39.628+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.632127+0000) 2019-10-13T12:11:39.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:39.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:39.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:39.633+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.637837+0000) 2019-10-13T12:11:39.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:39.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.678255+0000) 2019-10-13T12:11:39.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:39.953+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:39.958046+0000) 2019-10-13T12:11:42.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:42.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:42.214590+0000) 2019-10-13T12:11:44.640 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:44.637+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.639152+0000) 2019-10-13T12:11:44.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:44.636+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.639578+0000) 2019-10-13T12:11:44.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:44.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.640280+0000) 2019-10-13T12:11:44.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:44.636+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.639994+0000) 2019-10-13T12:11:44.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:44.636+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.640697+0000) 2019-10-13T12:11:44.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:44.636+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.641245+0000) 2019-10-13T12:11:44.642 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:44.636+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.642005+0000) 2019-10-13T12:11:44.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:44.641+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.644833+0000) 2019-10-13T12:11:44.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:44.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 141970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:45.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:45.488+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.489403+0000) 2019-10-13T12:11:45.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:45.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.524793+0000) 2019-10-13T12:11:46.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:46.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.458300+0000) 2019-10-13T12:11:48.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:48.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.722595+0000) 2019-10-13T12:11:48.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:48.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.726539+0000) 2019-10-13T12:11:49.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:49.644+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.649223+0000) 2019-10-13T12:11:49.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:49.644+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.649483+0000) 2019-10-13T12:11:49.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:49.644+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.650057+0000) 2019-10-13T12:11:49.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:49.644+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.650462+0000) 2019-10-13T12:11:49.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:49.649+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.650915+0000) 2019-10-13T12:11:49.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:49.648+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.651426+0000) 2019-10-13T12:11:49.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:49.648+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.652063+0000) 2019-10-13T12:11:49.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:49.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:49.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:49.653+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.656982+0000) 2019-10-13T12:11:49.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:49.685+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.687867+0000) 2019-10-13T12:11:49.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:49.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:49.958218+0000) 2019-10-13T12:11:52.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:52.208+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:52.214771+0000) 2019-10-13T12:11:54.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:54.652+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.658511+0000) 2019-10-13T12:11:54.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:54.657+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.660177+0000) 2019-10-13T12:11:54.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:54.656+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.660811+0000) 2019-10-13T12:11:54.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:54.656+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.661210+0000) 2019-10-13T12:11:54.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:54.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.661520+0000) 2019-10-13T12:11:54.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:54.656+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.661904+0000) 2019-10-13T12:11:54.663 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:54.656+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.662811+0000) 2019-10-13T12:11:54.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:54.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:54.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:54.665+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.668581+0000) 2019-10-13T12:11:54.682 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:11:54.682 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:54.686 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:11:54.686 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:55.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:55.488+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.489550+0000) 2019-10-13T12:11:55.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:55.524+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.524948+0000) 2019-10-13T12:11:56.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:56.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.458498+0000) 2019-10-13T12:11:58.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:58.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.722825+0000) 2019-10-13T12:11:58.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:58.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.726757+0000) 2019-10-13T12:11:59.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:11:59.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.669683+0000) 2019-10-13T12:11:59.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:11:59.664+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.670317+0000) 2019-10-13T12:11:59.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:11:59.664+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.671103+0000) 2019-10-13T12:11:59.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:59.669+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.671223+0000) 2019-10-13T12:11:59.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:11:59.668+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.671765+0000) 2019-10-13T12:11:59.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:11:59.668+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.672135+0000) 2019-10-13T12:11:59.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:11:59.668+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.672499+0000) 2019-10-13T12:11:59.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:11:59.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142073 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:11:59.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:59.677+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.679669+0000) 2019-10-13T12:11:59.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:11:59.689+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.691895+0000) 2019-10-13T12:11:59.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:11:59.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:59.958409+0000) 2019-10-13T12:12:02.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:02.212+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:02.214954+0000) 2019-10-13T12:12:04.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:04.676+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.680032+0000) 2019-10-13T12:12:04.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:04.676+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.681714+0000) 2019-10-13T12:12:04.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:04.676+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.682140+0000) 2019-10-13T12:12:04.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:04.676+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.682474+0000) 2019-10-13T12:12:04.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:04.676+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.682816+0000) 2019-10-13T12:12:04.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:04.681+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.683217+0000) 2019-10-13T12:12:04.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:04.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.683767+0000) 2019-10-13T12:12:04.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:04.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:04.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:04.685+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.687760+0000) 2019-10-13T12:12:05.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:05.484+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.489751+0000) 2019-10-13T12:12:05.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:05.520+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.525166+0000) 2019-10-13T12:12:06.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:06.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.458695+0000) 2019-10-13T12:12:08.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:08.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.723029+0000) 2019-10-13T12:12:08.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:08.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.726919+0000) 2019-10-13T12:12:09.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:09.684+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.689356+0000) 2019-10-13T12:12:09.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:09.685+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.689546+0000) 2019-10-13T12:12:09.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:09.684+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.690760+0000) 2019-10-13T12:12:09.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:09.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.691151+0000) 2019-10-13T12:12:09.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:09.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.692110+0000) 2019-10-13T12:12:09.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:09.688+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.692796+0000) 2019-10-13T12:12:09.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:09.688+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.693277+0000) 2019-10-13T12:12:09.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:09.693+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.695522+0000) 2019-10-13T12:12:09.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:09.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:09.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:09.697+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.698802+0000) 2019-10-13T12:12:09.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:09.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:09.958578+0000) 2019-10-13T12:12:12.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:12.212+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:12.215138+0000) 2019-10-13T12:12:14.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:14.696+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.699684+0000) 2019-10-13T12:12:14.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:14.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.700285+0000) 2019-10-13T12:12:14.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:14.696+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.701185+0000) 2019-10-13T12:12:14.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:14.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.701779+0000) 2019-10-13T12:12:14.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:14.696+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.702161+0000) 2019-10-13T12:12:14.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:14.701+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.702366+0000) 2019-10-13T12:12:14.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:14.700+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.704137+0000) 2019-10-13T12:12:14.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:14.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:14.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:14.705+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.709106+0000) 2019-10-13T12:12:15.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:15.484+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.489961+0000) 2019-10-13T12:12:15.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:15.524+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.525344+0000) 2019-10-13T12:12:16.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:16.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.458884+0000) 2019-10-13T12:12:18.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:18.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.723229+0000) 2019-10-13T12:12:18.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:18.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.727107+0000) 2019-10-13T12:12:19.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:19.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.706172+0000) 2019-10-13T12:12:19.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:19.704+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.710411+0000) 2019-10-13T12:12:19.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:19.704+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.710828+0000) 2019-10-13T12:12:19.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:19.708+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.711830+0000) 2019-10-13T12:12:19.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:19.708+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.712362+0000) 2019-10-13T12:12:19.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:19.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.712922+0000) 2019-10-13T12:12:19.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:19.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.712922+0000) 2019-10-13T12:12:19.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:19.708+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.713623+0000) 2019-10-13T12:12:19.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:19.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:19.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:19.716+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.721040+0000) 2019-10-13T12:12:19.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:19.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:19.958780+0000) 2019-10-13T12:12:22.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:22.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:22.215335+0000) 2019-10-13T12:12:24.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:24.716+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.720952+0000) 2019-10-13T12:12:24.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:24.715+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.721793+0000) 2019-10-13T12:12:24.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:24.715+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.722348+0000) 2019-10-13T12:12:24.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:24.715+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.722194+0000) 2019-10-13T12:12:24.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:24.715+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.723173+0000) 2019-10-13T12:12:24.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:24.719+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.723778+0000) 2019-10-13T12:12:24.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:24.719+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.723938+0000) 2019-10-13T12:12:24.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:24.724+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.727899+0000) 2019-10-13T12:12:24.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:24.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:24.739 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:12:24.740 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:24.743 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:12:24.743 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:25.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:25.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.490120+0000) 2019-10-13T12:12:25.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:25.523+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.525537+0000) 2019-10-13T12:12:26.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:26.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:26.459071+0000) 2019-10-13T12:12:28.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:28.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.723484+0000) 2019-10-13T12:12:28.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:28.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.727335+0000) 2019-10-13T12:12:29.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:29.712+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.713006+0000) 2019-10-13T12:12:29.732 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:29.727+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.731822+0000) 2019-10-13T12:12:29.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:29.727+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.732452+0000) 2019-10-13T12:12:29.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:29.727+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.732850+0000) 2019-10-13T12:12:29.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:29.727+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.734238+0000) 2019-10-13T12:12:29.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:29.727+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.734435+0000) 2019-10-13T12:12:29.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:29.732+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.734648+0000) 2019-10-13T12:12:29.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:29.731+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.735834+0000) 2019-10-13T12:12:29.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:29.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:29.742 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:29.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.741630+0000) 2019-10-13T12:12:29.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:29.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:29.958965+0000) 2019-10-13T12:12:32.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:32.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.215514+0000) 2019-10-13T12:12:34.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:34.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.742699+0000) 2019-10-13T12:12:34.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:34.739+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.743473+0000) 2019-10-13T12:12:34.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:34.739+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.743735+0000) 2019-10-13T12:12:34.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:34.740+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.743936+0000) 2019-10-13T12:12:34.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:34.739+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.744491+0000) 2019-10-13T12:12:34.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:34.739+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.745487+0000) 2019-10-13T12:12:34.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:34.739+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.746175+0000) 2019-10-13T12:12:34.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:34.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:34.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:34.748+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.750573+0000) 2019-10-13T12:12:35.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:35.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:35.490328+0000) 2019-10-13T12:12:35.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:35.519+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:35.525717+0000) 2019-10-13T12:12:36.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:36.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.459247+0000) 2019-10-13T12:12:38.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:38.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.723697+0000) 2019-10-13T12:12:38.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:38.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.727583+0000) 2019-10-13T12:12:39.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:39.716+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.719563+0000) 2019-10-13T12:12:39.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:39.747+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.752052+0000) 2019-10-13T12:12:39.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:39.747+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.753118+0000) 2019-10-13T12:12:39.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:39.747+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.753425+0000) 2019-10-13T12:12:39.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:39.747+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.753819+0000) 2019-10-13T12:12:39.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:39.748+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.753954+0000) 2019-10-13T12:12:39.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:39.747+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.754658+0000) 2019-10-13T12:12:39.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:39.751+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.755465+0000) 2019-10-13T12:12:39.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:39.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142339 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:39.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:39.760+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.762474+0000) 2019-10-13T12:12:39.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:39.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:39.959140+0000) 2019-10-13T12:12:42.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:42.215+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.215708+0000) 2019-10-13T12:12:44.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:44.755+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.763181+0000) 2019-10-13T12:12:44.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:44.759+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.763451+0000) 2019-10-13T12:12:44.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:44.759+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.764459+0000) 2019-10-13T12:12:44.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:44.760+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.764818+0000) 2019-10-13T12:12:44.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:44.759+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.765239+0000) 2019-10-13T12:12:44.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:44.759+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.765760+0000) 2019-10-13T12:12:44.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:44.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.766640+0000) 2019-10-13T12:12:44.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:44.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:44.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:44.768+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.773695+0000) 2019-10-13T12:12:45.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:45.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:45.490508+0000) 2019-10-13T12:12:45.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:45.523+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:45.525896+0000) 2019-10-13T12:12:46.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:46.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.459493+0000) 2019-10-13T12:12:48.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:48.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.723894+0000) 2019-10-13T12:12:48.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:48.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.727829+0000) 2019-10-13T12:12:49.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:49.724+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.726133+0000) 2019-10-13T12:12:49.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:49.767+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.772732+0000) 2019-10-13T12:12:49.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:49.767+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.773360+0000) 2019-10-13T12:12:49.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:49.767+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.773636+0000) 2019-10-13T12:12:49.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:49.767+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.774257+0000) 2019-10-13T12:12:49.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:49.767+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.774838+0000) 2019-10-13T12:12:49.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:49.771+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.775518+0000) 2019-10-13T12:12:49.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:49.772+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.776443+0000) 2019-10-13T12:12:49.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:49.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:49.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:49.780+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.783281+0000) 2019-10-13T12:12:49.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:49.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:49.959307+0000) 2019-10-13T12:12:52.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:52.215+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.215909+0000) 2019-10-13T12:12:54.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:54.783+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.783625+0000) 2019-10-13T12:12:54.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:54.783+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.784385+0000) 2019-10-13T12:12:54.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:54.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.784685+0000) 2019-10-13T12:12:54.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:54.783+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.785483+0000) 2019-10-13T12:12:54.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:54.783+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.786276+0000) 2019-10-13T12:12:54.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:54.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.786042+0000) 2019-10-13T12:12:54.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:54.784+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.787448+0000) 2019-10-13T12:12:54.792 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:54.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:54.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:54.788+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.791409+0000) 2019-10-13T12:12:54.859 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:12:54.859 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:54.863 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:12:54.863 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:55.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:55.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:55.490685+0000) 2019-10-13T12:12:55.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:55.523+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:55.526076+0000) 2019-10-13T12:12:56.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:56.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.459683+0000) 2019-10-13T12:12:58.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:58.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.724112+0000) 2019-10-13T12:12:58.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:58.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.728029+0000) 2019-10-13T12:12:59.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:59.732+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.732891+0000) 2019-10-13T12:12:59.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:12:59.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.794705+0000) 2019-10-13T12:12:59.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:12:59.795+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.795752+0000) 2019-10-13T12:12:59.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:12:59.795+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.796177+0000) 2019-10-13T12:12:59.797 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:12:59.795+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.796785+0000) 2019-10-13T12:12:59.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:12:59.795+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.797161+0000) 2019-10-13T12:12:59.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:12:59.799+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.799451+0000) 2019-10-13T12:12:59.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:59.796+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.799370+0000) 2019-10-13T12:12:59.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:12:59.799+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142470 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:12:59.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:12:59.800+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.804796+0000) 2019-10-13T12:12:59.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:12:59.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:59.959479+0000) 2019-10-13T12:13:02.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:02.215+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.216102+0000) 2019-10-13T12:13:04.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:04.800+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.805103+0000) 2019-10-13T12:13:04.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:04.802+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.806547+0000) 2019-10-13T12:13:04.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:04.802+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.807275+0000) 2019-10-13T12:13:04.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:04.806+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.807914+0000) 2019-10-13T12:13:04.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:04.806+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.808132+0000) 2019-10-13T12:13:04.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:04.806+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.808514+0000) 2019-10-13T12:13:04.809 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:04.806+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.809073+0000) 2019-10-13T12:13:04.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:04.810+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:04.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:04.808+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.812192+0000) 2019-10-13T12:13:05.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:05.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:05.490868+0000) 2019-10-13T12:13:05.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:05.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:05.526262+0000) 2019-10-13T12:13:06.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:06.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.459881+0000) 2019-10-13T12:13:08.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:08.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.724325+0000) 2019-10-13T12:13:08.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:08.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.728253+0000) 2019-10-13T12:13:09.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:09.736+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.739351+0000) 2019-10-13T12:13:09.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:09.814+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.815872+0000) 2019-10-13T12:13:09.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:09.814+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.816334+0000) 2019-10-13T12:13:09.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:09.812+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.816582+0000) 2019-10-13T12:13:09.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:09.814+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.817057+0000) 2019-10-13T12:13:09.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:09.814+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.817314+0000) 2019-10-13T12:13:09.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:09.814+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.817640+0000) 2019-10-13T12:13:09.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:09.814+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.818237+0000) 2019-10-13T12:13:09.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:09.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:09.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:09.820+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.825324+0000) 2019-10-13T12:13:09.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:09.956+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:09.959650+0000) 2019-10-13T12:13:12.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:12.214+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.216309+0000) 2019-10-13T12:13:14.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:14.826+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.828736+0000) 2019-10-13T12:13:14.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:14.826+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.828959+0000) 2019-10-13T12:13:14.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:14.826+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.828414+0000) 2019-10-13T12:13:14.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:14.826+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.829467+0000) 2019-10-13T12:13:14.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:14.826+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.830144+0000) 2019-10-13T12:13:14.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:14.823+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.829778+0000) 2019-10-13T12:13:14.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:14.826+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.831173+0000) 2019-10-13T12:13:14.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:14.827+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.833634+0000) 2019-10-13T12:13:14.835 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:14.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:15.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:15.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:15.491067+0000) 2019-10-13T12:13:15.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:15.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:15.526462+0000) 2019-10-13T12:13:16.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:16.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.460086+0000) 2019-10-13T12:13:18.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:18.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.724557+0000) 2019-10-13T12:13:18.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:18.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.728455+0000) 2019-10-13T12:13:19.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:19.739+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.745808+0000) 2019-10-13T12:13:19.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:19.835+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.838285+0000) 2019-10-13T12:13:19.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:19.834+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.838874+0000) 2019-10-13T12:13:19.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:19.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.838889+0000) 2019-10-13T12:13:19.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:19.838+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.839383+0000) 2019-10-13T12:13:19.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:19.838+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.840251+0000) 2019-10-13T12:13:19.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:19.838+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.841276+0000) 2019-10-13T12:13:19.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:19.838+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.841770+0000) 2019-10-13T12:13:19.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:19.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:19.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:19.843+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.847065+0000) 2019-10-13T12:13:19.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:19.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:19.959829+0000) 2019-10-13T12:13:22.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:22.214+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.216538+0000) 2019-10-13T12:13:24.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:24.846+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.847409+0000) 2019-10-13T12:13:24.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:24.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.849151+0000) 2019-10-13T12:13:24.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:24.846+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.849349+0000) 2019-10-13T12:13:24.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:24.846+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.849902+0000) 2019-10-13T12:13:24.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:24.846+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.850869+0000) 2019-10-13T12:13:24.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:24.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.851118+0000) 2019-10-13T12:13:24.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:24.850+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.851916+0000) 2019-10-13T12:13:24.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:24.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:24.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:24.851+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.855129+0000) 2019-10-13T12:13:24.953 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:13:24.954 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:24.957 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:13:24.957 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:25.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:25.486+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:25.491274+0000) 2019-10-13T12:13:25.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:25.526+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:25.526649+0000) 2019-10-13T12:13:26.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:26.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.460323+0000) 2019-10-13T12:13:28.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:28.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.724757+0000) 2019-10-13T12:13:28.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:28.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.728678+0000) 2019-10-13T12:13:29.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:29.751+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.752336+0000) 2019-10-13T12:13:29.859 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:29.854+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.857918+0000) 2019-10-13T12:13:29.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:29.854+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.859024+0000) 2019-10-13T12:13:29.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:29.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.859634+0000) 2019-10-13T12:13:29.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:29.858+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.860449+0000) 2019-10-13T12:13:29.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:29.858+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.861051+0000) 2019-10-13T12:13:29.862 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:29.858+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.861583+0000) 2019-10-13T12:13:29.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:29.858+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.862208+0000) 2019-10-13T12:13:29.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:29.859+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.864640+0000) 2019-10-13T12:13:29.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:29.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:29.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:29.959+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:29.960021+0000) 2019-10-13T12:13:32.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:32.214+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.216769+0000) 2019-10-13T12:13:34.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:34.866+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.869254+0000) 2019-10-13T12:13:34.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:34.866+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.870385+0000) 2019-10-13T12:13:34.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:34.866+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.870602+0000) 2019-10-13T12:13:34.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:34.867+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.871010+0000) 2019-10-13T12:13:34.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:34.870+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.871418+0000) 2019-10-13T12:13:34.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:34.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.871901+0000) 2019-10-13T12:13:34.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:34.870+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.872931+0000) 2019-10-13T12:13:34.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:34.871+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.874938+0000) 2019-10-13T12:13:34.876 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:34.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:35.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:35.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:35.491483+0000) 2019-10-13T12:13:35.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:35.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:35.526850+0000) 2019-10-13T12:13:36.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:36.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.460586+0000) 2019-10-13T12:13:38.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:38.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.724946+0000) 2019-10-13T12:13:38.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:38.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.728883+0000) 2019-10-13T12:13:39.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:39.755+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.759001+0000) 2019-10-13T12:13:39.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:39.875+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.878441+0000) 2019-10-13T12:13:39.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:39.878+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.879368+0000) 2019-10-13T12:13:39.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:39.878+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.879787+0000) 2019-10-13T12:13:39.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:39.878+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.881157+0000) 2019-10-13T12:13:39.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:39.878+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.881382+0000) 2019-10-13T12:13:39.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:39.878+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.881720+0000) 2019-10-13T12:13:39.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:39.878+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.882659+0000) 2019-10-13T12:13:39.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:39.883+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.886950+0000) 2019-10-13T12:13:39.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:39.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:39.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:39.959+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:39.960219+0000) 2019-10-13T12:13:42.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:42.214+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.217033+0000) 2019-10-13T12:13:44.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:44.886+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.890259+0000) 2019-10-13T12:13:44.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:44.886+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.890829+0000) 2019-10-13T12:13:44.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:44.890+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.891993+0000) 2019-10-13T12:13:44.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:44.887+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.892063+0000) 2019-10-13T12:13:44.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:44.890+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.892603+0000) 2019-10-13T12:13:44.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:44.890+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.892998+0000) 2019-10-13T12:13:44.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:44.890+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.893874+0000) 2019-10-13T12:13:44.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:44.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142762 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:44.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:44.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.898823+0000) 2019-10-13T12:13:45.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:45.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:45.491662+0000) 2019-10-13T12:13:45.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:45.522+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:45.527035+0000) 2019-10-13T12:13:46.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:46.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.460789+0000) 2019-10-13T12:13:48.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:48.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.725143+0000) 2019-10-13T12:13:48.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:48.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.729111+0000) 2019-10-13T12:13:49.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:49.763+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.766861+0000) 2019-10-13T12:13:49.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:49.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.901655+0000) 2019-10-13T12:13:49.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:49.897+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.902408+0000) 2019-10-13T12:13:49.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:49.897+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.903249+0000) 2019-10-13T12:13:49.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:49.901+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.903635+0000) 2019-10-13T12:13:49.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:49.901+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.904242+0000) 2019-10-13T12:13:49.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:49.901+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.904600+0000) 2019-10-13T12:13:49.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:49.899+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.904954+0000) 2019-10-13T12:13:49.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:49.903+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.908923+0000) 2019-10-13T12:13:49.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:49.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:49.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:49.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:49.960386+0000) 2019-10-13T12:13:52.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:52.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.217259+0000) 2019-10-13T12:13:54.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:54.909+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.912433+0000) 2019-10-13T12:13:54.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:54.909+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.912610+0000) 2019-10-13T12:13:54.914 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:54.909+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.913259+0000) 2019-10-13T12:13:54.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:54.909+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.913569+0000) 2019-10-13T12:13:54.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:54.907+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.913669+0000) 2019-10-13T12:13:54.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:54.909+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.914447+0000) 2019-10-13T12:13:54.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:54.909+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.914983+0000) 2019-10-13T12:13:54.919 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:54.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:54.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:54.915+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.921677+0000) 2019-10-13T12:13:55.057 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:13:55.057 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:55.060 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:13:55.061 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:55.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:55.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:55.491835+0000) 2019-10-13T12:13:55.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:55.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:55.527261+0000) 2019-10-13T12:13:56.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:56.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.460981+0000) 2019-10-13T12:13:58.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:58.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.725347+0000) 2019-10-13T12:13:58.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:58.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.729310+0000) 2019-10-13T12:13:59.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:59.767+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.773276+0000) 2019-10-13T12:13:59.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:13:59.917+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.921972+0000) 2019-10-13T12:13:59.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:59.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.922302+0000) 2019-10-13T12:13:59.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:13:59.917+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.923201+0000) 2019-10-13T12:13:59.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:13:59.921+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.923689+0000) 2019-10-13T12:13:59.925 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:13:59.921+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.924234+0000) 2019-10-13T12:13:59.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:13:59.921+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.923383+0000) 2019-10-13T12:13:59.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:13:59.921+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.925010+0000) 2019-10-13T12:13:59.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:13:59.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:13:59.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:13:59.927+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.931560+0000) 2019-10-13T12:13:59.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:13:59.955+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:59.960541+0000) 2019-10-13T12:14:02.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:02.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.217477+0000) 2019-10-13T12:14:04.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:04.927+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.932890+0000) 2019-10-13T12:14:04.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:04.929+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.933935+0000) 2019-10-13T12:14:04.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:04.929+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.933674+0000) 2019-10-13T12:14:04.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:04.929+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.934648+0000) 2019-10-13T12:14:04.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:04.933+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.935542+0000) 2019-10-13T12:14:04.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:04.933+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.936433+0000) 2019-10-13T12:14:04.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:04.933+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.936078+0000) 2019-10-13T12:14:04.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:04.937+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:04.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:04.939+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.942237+0000) 2019-10-13T12:14:05.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:05.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:05.492083+0000) 2019-10-13T12:14:05.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:05.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:05.527472+0000) 2019-10-13T12:14:06.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:06.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.461195+0000) 2019-10-13T12:14:08.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:08.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.725550+0000) 2019-10-13T12:14:08.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:08.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.729531+0000) 2019-10-13T12:14:09.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:09.778+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.779766+0000) 2019-10-13T12:14:09.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:09.941+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.944524+0000) 2019-10-13T12:14:09.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:09.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.944745+0000) 2019-10-13T12:14:09.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:09.941+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.945280+0000) 2019-10-13T12:14:09.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:09.941+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.945750+0000) 2019-10-13T12:14:09.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:09.941+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.946501+0000) 2019-10-13T12:14:09.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:09.941+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.947107+0000) 2019-10-13T12:14:09.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:09.945+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.948254+0000) 2019-10-13T12:14:09.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:09.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:09.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:09.946+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.952333+0000) 2019-10-13T12:14:09.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:09.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:09.960679+0000) 2019-10-13T12:14:12.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:12.217+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.217692+0000) 2019-10-13T12:14:14.956 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:14.949+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.955173+0000) 2019-10-13T12:14:14.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:14.953+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.955588+0000) 2019-10-13T12:14:14.957 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:14.950+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.955980+0000) 2019-10-13T12:14:14.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:14.953+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.957116+0000) 2019-10-13T12:14:14.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:14.953+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.957593+0000) 2019-10-13T12:14:14.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:14.953+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.958173+0000) 2019-10-13T12:14:14.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:14.953+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.958824+0000) 2019-10-13T12:14:14.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:14.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:14.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:14.958+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.963186+0000) 2019-10-13T12:14:15.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:15.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:15.492218+0000) 2019-10-13T12:14:15.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:15.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:15.527679+0000) 2019-10-13T12:14:16.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:16.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.461410+0000) 2019-10-13T12:14:18.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:18.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.725767+0000) 2019-10-13T12:14:18.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:18.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.729745+0000) 2019-10-13T12:14:19.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:19.790+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.790838+0000) 2019-10-13T12:14:19.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:19.954+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.960831+0000) 2019-10-13T12:14:19.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:19.961+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.965526+0000) 2019-10-13T12:14:19.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:19.961+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.965973+0000) 2019-10-13T12:14:19.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:19.962+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.966315+0000) 2019-10-13T12:14:19.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:19.961+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.966921+0000) 2019-10-13T12:14:19.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:19.965+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.968489+0000) 2019-10-13T12:14:19.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:19.965+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.968859+0000) 2019-10-13T12:14:19.970 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:19.965+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.969281+0000) 2019-10-13T12:14:19.973 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:19.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 142990 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:19.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:19.970+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:19.975927+0000) 2019-10-13T12:14:22.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:22.213+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.217840+0000) 2019-10-13T12:14:24.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:24.973+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.976581+0000) 2019-10-13T12:14:24.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:24.970+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.977068+0000) 2019-10-13T12:14:24.978 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:24.973+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.977197+0000) 2019-10-13T12:14:24.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:24.973+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.978187+0000) 2019-10-13T12:14:24.979 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:24.973+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.978663+0000) 2019-10-13T12:14:24.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:24.973+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.978896+0000) 2019-10-13T12:14:24.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:24.977+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.979440+0000) 2019-10-13T12:14:24.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:24.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:24.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:24.978+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.983929+0000) 2019-10-13T12:14:25.180 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:14:25.180 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:25.184 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:14:25.184 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:25.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:25.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:25.492406+0000) 2019-10-13T12:14:25.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:25.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:25.527870+0000) 2019-10-13T12:14:26.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:26.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.461607+0000) 2019-10-13T12:14:28.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:28.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.725965+0000) 2019-10-13T12:14:28.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:28.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.729939+0000) 2019-10-13T12:14:29.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:29.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.797489+0000) 2019-10-13T12:14:29.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:29.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.961024+0000) 2019-10-13T12:14:29.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:29.980+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.986964+0000) 2019-10-13T12:14:29.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:29.984+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.988307+0000) 2019-10-13T12:14:29.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:29.984+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.988801+0000) 2019-10-13T12:14:29.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:29.984+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.989162+0000) 2019-10-13T12:14:29.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:29.982+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.990011+0000) 2019-10-13T12:14:29.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:29.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.990416+0000) 2019-10-13T12:14:29.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:29.988+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.991430+0000) 2019-10-13T12:14:29.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:29.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143057 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:29.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:29.990+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:29.997129+0000) 2019-10-13T12:14:32.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:32.212+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.218035+0000) 2019-10-13T12:14:34.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:34.992+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.997639+0000) 2019-10-13T12:14:34.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:34.992+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.998606+0000) 2019-10-13T12:14:34.999 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:34.992+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.998921+0000) 2019-10-13T12:14:35.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:34.994+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.999195+0000) 2019-10-13T12:14:35.000 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:34.996+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.999661+0000) 2019-10-13T12:14:35.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:34.996+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.000941+0000) 2019-10-13T12:14:35.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:34.996+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.001383+0000) 2019-10-13T12:14:35.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:35.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:35.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:34.998+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.006101+0000) 2019-10-13T12:14:35.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:35.488+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.492585+0000) 2019-10-13T12:14:35.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:35.524+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.528075+0000) 2019-10-13T12:14:36.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:36.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.461756+0000) 2019-10-13T12:14:38.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:38.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.726174+0000) 2019-10-13T12:14:38.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:38.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.730131+0000) 2019-10-13T12:14:39.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:39.802+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:39.803853+0000) 2019-10-13T12:14:39.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:39.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:39.961218+0000) 2019-10-13T12:14:40.009 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:40.004+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.008751+0000) 2019-10-13T12:14:40.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:40.002+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.009407+0000) 2019-10-13T12:14:40.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:40.004+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.010306+0000) 2019-10-13T12:14:40.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:40.004+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.010681+0000) 2019-10-13T12:14:40.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:40.008+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.011776+0000) 2019-10-13T12:14:40.013 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:40.008+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.012049+0000) 2019-10-13T12:14:40.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:40.008+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.012518+0000) 2019-10-13T12:14:40.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:40.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:40.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:40.010+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:40.015956+0000) 2019-10-13T12:14:42.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:42.212+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.218240+0000) 2019-10-13T12:14:45.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:45.012+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.018899+0000) 2019-10-13T12:14:45.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:45.016+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.019957+0000) 2019-10-13T12:14:45.021 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:45.016+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.020734+0000) 2019-10-13T12:14:45.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:45.016+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.021475+0000) 2019-10-13T12:14:45.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:45.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.021214+0000) 2019-10-13T12:14:45.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:45.016+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.021671+0000) 2019-10-13T12:14:45.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:45.016+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.022416+0000) 2019-10-13T12:14:45.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:45.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:45.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:45.022+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.027359+0000) 2019-10-13T12:14:45.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:45.492+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.492765+0000) 2019-10-13T12:14:45.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:45.524+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.528264+0000) 2019-10-13T12:14:46.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:46.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.461954+0000) 2019-10-13T12:14:48.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:48.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.726378+0000) 2019-10-13T12:14:48.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:48.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.730345+0000) 2019-10-13T12:14:49.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:49.814+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:49.815135+0000) 2019-10-13T12:14:49.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:49.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:49.961418+0000) 2019-10-13T12:14:50.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:50.024+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.028187+0000) 2019-10-13T12:14:50.030 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:50.024+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.028970+0000) 2019-10-13T12:14:50.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:50.024+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.030883+0000) 2019-10-13T12:14:50.032 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:50.024+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.030792+0000) 2019-10-13T12:14:50.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:50.028+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.031768+0000) 2019-10-13T12:14:50.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:50.028+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.032142+0000) 2019-10-13T12:14:50.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:50.026+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.032675+0000) 2019-10-13T12:14:50.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:50.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:50.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:50.034+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:50.038533+0000) 2019-10-13T12:14:52.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:52.216+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.218442+0000) 2019-10-13T12:14:55.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:55.036+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.039722+0000) 2019-10-13T12:14:55.041 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:55.036+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.039905+0000) 2019-10-13T12:14:55.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:55.036+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.040433+0000) 2019-10-13T12:14:55.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:14:55.036+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.041180+0000) 2019-10-13T12:14:55.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:55.036+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.041567+0000) 2019-10-13T12:14:55.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:55.036+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.042417+0000) 2019-10-13T12:14:55.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:55.042+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.042527+0000) 2019-10-13T12:14:55.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:14:55.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:14:55.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:55.046+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.047597+0000) 2019-10-13T12:14:55.303 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:14:55.303 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:55.307 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:14:55.307 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:55.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:14:55.492+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.492970+0000) 2019-10-13T12:14:55.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:14:55.524+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.528464+0000) 2019-10-13T12:14:56.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:14:56.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.462151+0000) 2019-10-13T12:14:58.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:14:58.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.726635+0000) 2019-10-13T12:14:58.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:14:58.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.730542+0000) 2019-10-13T12:14:59.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:14:59.818+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:59.821642+0000) 2019-10-13T12:14:59.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:14:59.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:59.961630+0000) 2019-10-13T12:15:00.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:00.044+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.049710+0000) 2019-10-13T12:15:00.051 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:00.044+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.050280+0000) 2019-10-13T12:15:00.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:00.044+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.050395+0000) 2019-10-13T12:15:00.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:00.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.050915+0000) 2019-10-13T12:15:00.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:00.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.051690+0000) 2019-10-13T12:15:00.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:00.048+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.052696+0000) 2019-10-13T12:15:00.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:00.048+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.053018+0000) 2019-10-13T12:15:00.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:00.054+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:00.056858+0000) 2019-10-13T12:15:00.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:00.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143254 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:02.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:02.216+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.218674+0000) 2019-10-13T12:15:05.062 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:05.056+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.060976+0000) 2019-10-13T12:15:05.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:05.056+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.061447+0000) 2019-10-13T12:15:05.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:05.056+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.061752+0000) 2019-10-13T12:15:05.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:05.061+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.062228+0000) 2019-10-13T12:15:05.063 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:05.056+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.062613+0000) 2019-10-13T12:15:05.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:05.056+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.063202+0000) 2019-10-13T12:15:05.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:05.060+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.063646+0000) 2019-10-13T12:15:05.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:05.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:05.072 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:05.069+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.071189+0000) 2019-10-13T12:15:05.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:05.488+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.493150+0000) 2019-10-13T12:15:05.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:05.528+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.528672+0000) 2019-10-13T12:15:06.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:06.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.462377+0000) 2019-10-13T12:15:08.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:08.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.726831+0000) 2019-10-13T12:15:08.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:08.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.730758+0000) 2019-10-13T12:15:09.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:09.829+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:09.832930+0000) 2019-10-13T12:15:09.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:09.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:09.961774+0000) 2019-10-13T12:15:10.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:10.068+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.071284+0000) 2019-10-13T12:15:10.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:10.068+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.071906+0000) 2019-10-13T12:15:10.073 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:10.068+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.072550+0000) 2019-10-13T12:15:10.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:10.068+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.073255+0000) 2019-10-13T12:15:10.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:10.068+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.073712+0000) 2019-10-13T12:15:10.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:10.068+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.074184+0000) 2019-10-13T12:15:10.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:10.073+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.075264+0000) 2019-10-13T12:15:10.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:10.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:10.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:10.077+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:10.081477+0000) 2019-10-13T12:15:12.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:12.212+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.218865+0000) 2019-10-13T12:15:15.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:15.075+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.081756+0000) 2019-10-13T12:15:15.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:15.075+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.082166+0000) 2019-10-13T12:15:15.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:15.081+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.082705+0000) 2019-10-13T12:15:15.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:15.079+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.083469+0000) 2019-10-13T12:15:15.084 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:15.079+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.084141+0000) 2019-10-13T12:15:15.085 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:15.079+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.084892+0000) 2019-10-13T12:15:15.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:15.079+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.085553+0000) 2019-10-13T12:15:15.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:15.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:15.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:15.089+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.090300+0000) 2019-10-13T12:15:15.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:15.487+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.493327+0000) 2019-10-13T12:15:15.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:15.527+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.528853+0000) 2019-10-13T12:15:16.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:16.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.462575+0000) 2019-10-13T12:15:18.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:18.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.727042+0000) 2019-10-13T12:15:18.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:18.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.730976+0000) 2019-10-13T12:15:19.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:19.837+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:19.839513+0000) 2019-10-13T12:15:19.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:19.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:19.961912+0000) 2019-10-13T12:15:20.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:20.087+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.092642+0000) 2019-10-13T12:15:20.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:20.087+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.092753+0000) 2019-10-13T12:15:20.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:20.093+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.094461+0000) 2019-10-13T12:15:20.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:20.087+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.095013+0000) 2019-10-13T12:15:20.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:20.091+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.095263+0000) 2019-10-13T12:15:20.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:20.091+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.095942+0000) 2019-10-13T12:15:20.097 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:20.091+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.096365+0000) 2019-10-13T12:15:20.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:20.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:20.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:20.101+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:20.103784+0000) 2019-10-13T12:15:22.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:22.211+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.219093+0000) 2019-10-13T12:15:25.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:25.095+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.103111+0000) 2019-10-13T12:15:25.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:25.099+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.103415+0000) 2019-10-13T12:15:25.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:25.099+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.104047+0000) 2019-10-13T12:15:25.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:25.099+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.104246+0000) 2019-10-13T12:15:25.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:25.099+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.104752+0000) 2019-10-13T12:15:25.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:25.101+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.105115+0000) 2019-10-13T12:15:25.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:25.099+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.106205+0000) 2019-10-13T12:15:25.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:25.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:25.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:25.109+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.111136+0000) 2019-10-13T12:15:25.410 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:15:25.410 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:25.414 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:15:25.414 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:25.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:25.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.493506+0000) 2019-10-13T12:15:25.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:25.527+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.529013+0000) 2019-10-13T12:15:26.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:26.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.462794+0000) 2019-10-13T12:15:28.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:28.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.727240+0000) 2019-10-13T12:15:28.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:28.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.731167+0000) 2019-10-13T12:15:29.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:29.841+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:29.846025+0000) 2019-10-13T12:15:29.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:29.957+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:29.962097+0000) 2019-10-13T12:15:30.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:30.107+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.114636+0000) 2019-10-13T12:15:30.116 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:30.107+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.114962+0000) 2019-10-13T12:15:30.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:30.113+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.115184+0000) 2019-10-13T12:15:30.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:30.111+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.116244+0000) 2019-10-13T12:15:30.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:30.111+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.116890+0000) 2019-10-13T12:15:30.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:30.111+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.117193+0000) 2019-10-13T12:15:30.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:30.111+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.117789+0000) 2019-10-13T12:15:30.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:30.117+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:30.120612+0000) 2019-10-13T12:15:30.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:30.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143453 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:32.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:32.215+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.219337+0000) 2019-10-13T12:15:35.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:35.119+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.124217+0000) 2019-10-13T12:15:35.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:35.119+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.124975+0000) 2019-10-13T12:15:35.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:35.119+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.125045+0000) 2019-10-13T12:15:35.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:35.119+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.125616+0000) 2019-10-13T12:15:35.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:35.121+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.125673+0000) 2019-10-13T12:15:35.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:35.119+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.126735+0000) 2019-10-13T12:15:35.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:35.119+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.127238+0000) 2019-10-13T12:15:35.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:35.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:35.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:35.129+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.133571+0000) 2019-10-13T12:15:35.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:35.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.493690+0000) 2019-10-13T12:15:35.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:35.527+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.529227+0000) 2019-10-13T12:15:36.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:36.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.462990+0000) 2019-10-13T12:15:38.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:38.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.727471+0000) 2019-10-13T12:15:38.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:38.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.731377+0000) 2019-10-13T12:15:39.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:39.849+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:39.852356+0000) 2019-10-13T12:15:39.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:39.961+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:39.962329+0000) 2019-10-13T12:15:40.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:40.127+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.135183+0000) 2019-10-13T12:15:40.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:40.133+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.135256+0000) 2019-10-13T12:15:40.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:40.131+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.135935+0000) 2019-10-13T12:15:40.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:40.131+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.136220+0000) 2019-10-13T12:15:40.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:40.131+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.136542+0000) 2019-10-13T12:15:40.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:40.131+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.136847+0000) 2019-10-13T12:15:40.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:40.131+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.137525+0000) 2019-10-13T12:15:40.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:40.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143518 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:40.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:40.145+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:40.147186+0000) 2019-10-13T12:15:42.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:42.219+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.219565+0000) 2019-10-13T12:15:45.146 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:45.143+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.144979+0000) 2019-10-13T12:15:45.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:45.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.145596+0000) 2019-10-13T12:15:45.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:45.143+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.145591+0000) 2019-10-13T12:15:45.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:45.141+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.145840+0000) 2019-10-13T12:15:45.148 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:45.143+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.146941+0000) 2019-10-13T12:15:45.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:45.143+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.146307+0000) 2019-10-13T12:15:45.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:45.147+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.147457+0000) 2019-10-13T12:15:45.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:45.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:45.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:45.153+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.156583+0000) 2019-10-13T12:15:45.494 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:45.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.493873+0000) 2019-10-13T12:15:45.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:45.527+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.529369+0000) 2019-10-13T12:15:46.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:46.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.463186+0000) 2019-10-13T12:15:48.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:48.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.727687+0000) 2019-10-13T12:15:48.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:48.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.731571+0000) 2019-10-13T12:15:49.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:49.857+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:49.858918+0000) 2019-10-13T12:15:49.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:49.961+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:49.962540+0000) 2019-10-13T12:15:50.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:50.155+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.155430+0000) 2019-10-13T12:15:50.157 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:50.155+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.156001+0000) 2019-10-13T12:15:50.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:50.155+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.156253+0000) 2019-10-13T12:15:50.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:50.155+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.156824+0000) 2019-10-13T12:15:50.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:50.155+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.157386+0000) 2019-10-13T12:15:50.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:50.153+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.157867+0000) 2019-10-13T12:15:50.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:50.155+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.158598+0000) 2019-10-13T12:15:50.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:50.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:50.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:50.161+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:50.164755+0000) 2019-10-13T12:15:52.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:52.219+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.219805+0000) 2019-10-13T12:15:55.167 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:55.163+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.165549+0000) 2019-10-13T12:15:55.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:15:55.163+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.166163+0000) 2019-10-13T12:15:55.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:55.167+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.167520+0000) 2019-10-13T12:15:55.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:55.167+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.168125+0000) 2019-10-13T12:15:55.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:55.167+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.168635+0000) 2019-10-13T12:15:55.170 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:55.167+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.169190+0000) 2019-10-13T12:15:55.170 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:55.165+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.169495+0000) 2019-10-13T12:15:55.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:15:55.171+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:15:55.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:55.169+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.173609+0000) 2019-10-13T12:15:55.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:15:55.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.494054+0000) 2019-10-13T12:15:55.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:15:55.527+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.529575+0000) 2019-10-13T12:15:55.558 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:15:55.558 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:55.561 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:15:55.562 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:56.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:15:56.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.463398+0000) 2019-10-13T12:15:58.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:15:58.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.727885+0000) 2019-10-13T12:15:58.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:15:58.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.731763+0000) 2019-10-13T12:15:59.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:15:59.864+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:59.865432+0000) 2019-10-13T12:15:59.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:15:59.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:59.962729+0000) 2019-10-13T12:16:00.177 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:00.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.176474+0000) 2019-10-13T12:16:00.178 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:00.174+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.176902+0000) 2019-10-13T12:16:00.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:00.174+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.177187+0000) 2019-10-13T12:16:00.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:00.174+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.177587+0000) 2019-10-13T12:16:00.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:00.174+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.177858+0000) 2019-10-13T12:16:00.179 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:00.174+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.178640+0000) 2019-10-13T12:16:00.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:00.176+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.178711+0000) 2019-10-13T12:16:00.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:00.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:00.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:00.180+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:00.184439+0000) 2019-10-13T12:16:02.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:02.218+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.220003+0000) 2019-10-13T12:16:05.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:05.182+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.186506+0000) 2019-10-13T12:16:05.188 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:05.182+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.186924+0000) 2019-10-13T12:16:05.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:05.184+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.187871+0000) 2019-10-13T12:16:05.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:05.186+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.188361+0000) 2019-10-13T12:16:05.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:05.186+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.188734+0000) 2019-10-13T12:16:05.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:05.186+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.189066+0000) 2019-10-13T12:16:05.191 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:05.186+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.190508+0000) 2019-10-13T12:16:05.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:05.188+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.193018+0000) 2019-10-13T12:16:05.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:05.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:05.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:05.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.494222+0000) 2019-10-13T12:16:05.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:05.526+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.529759+0000) 2019-10-13T12:16:06.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:06.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.463622+0000) 2019-10-13T12:16:08.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:08.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.728077+0000) 2019-10-13T12:16:08.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:08.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.731953+0000) 2019-10-13T12:16:09.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:09.868+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:09.871931+0000) 2019-10-13T12:16:09.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:09.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:09.962920+0000) 2019-10-13T12:16:10.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:10.194+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.197347+0000) 2019-10-13T12:16:10.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:10.194+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.197860+0000) 2019-10-13T12:16:10.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:10.194+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.198192+0000) 2019-10-13T12:16:10.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:10.198+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.199457+0000) 2019-10-13T12:16:10.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:10.196+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.199529+0000) 2019-10-13T12:16:10.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:10.194+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.198865+0000) 2019-10-13T12:16:10.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:10.198+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.200132+0000) 2019-10-13T12:16:10.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:10.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:10.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:10.204+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:10.206796+0000) 2019-10-13T12:16:12.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:12.218+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.220225+0000) 2019-10-13T12:16:15.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:15.206+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.207969+0000) 2019-10-13T12:16:15.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:15.206+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.208688+0000) 2019-10-13T12:16:15.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:15.206+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.209363+0000) 2019-10-13T12:16:15.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:15.206+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.209881+0000) 2019-10-13T12:16:15.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:15.206+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.210201+0000) 2019-10-13T12:16:15.211 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:15.206+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.210798+0000) 2019-10-13T12:16:15.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:15.208+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.211163+0000) 2019-10-13T12:16:15.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:15.210+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:15.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:15.216+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.219791+0000) 2019-10-13T12:16:15.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:15.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.494412+0000) 2019-10-13T12:16:15.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:15.526+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.529942+0000) 2019-10-13T12:16:16.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:16.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.463818+0000) 2019-10-13T12:16:18.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:18.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.728282+0000) 2019-10-13T12:16:18.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:18.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.732162+0000) 2019-10-13T12:16:19.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:19.876+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:19.878530+0000) 2019-10-13T12:16:19.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:19.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:19.963109+0000) 2019-10-13T12:16:20.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:20.214+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.218098+0000) 2019-10-13T12:16:20.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:20.214+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.218445+0000) 2019-10-13T12:16:20.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:20.216+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.219214+0000) 2019-10-13T12:16:20.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:20.218+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.221148+0000) 2019-10-13T12:16:20.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:20.218+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.221288+0000) 2019-10-13T12:16:20.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:20.218+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.221737+0000) 2019-10-13T12:16:20.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:20.218+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.222270+0000) 2019-10-13T12:16:20.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:20.220+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:20.225466+0000) 2019-10-13T12:16:20.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:20.222+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143782 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:22.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:22.218+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.220420+0000) 2019-10-13T12:16:25.230 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:25.224+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.229644+0000) 2019-10-13T12:16:25.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:25.226+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.230532+0000) 2019-10-13T12:16:25.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:25.226+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.230906+0000) 2019-10-13T12:16:25.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:25.230+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.231330+0000) 2019-10-13T12:16:25.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:25.230+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.232061+0000) 2019-10-13T12:16:25.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:25.230+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.232281+0000) 2019-10-13T12:16:25.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:25.230+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.232705+0000) 2019-10-13T12:16:25.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:25.232+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.235630+0000) 2019-10-13T12:16:25.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:25.234+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:25.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:25.494+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.494611+0000) 2019-10-13T12:16:25.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:25.526+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.530153+0000) 2019-10-13T12:16:25.687 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:16:25.687 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:25.691 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:16:25.691 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:26.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:26.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.464042+0000) 2019-10-13T12:16:28.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:28.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.728476+0000) 2019-10-13T12:16:28.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:28.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.732361+0000) 2019-10-13T12:16:29.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:29.880+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:29.885181+0000) 2019-10-13T12:16:29.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:29.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:29.963300+0000) 2019-10-13T12:16:30.241 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:30.236+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.239990+0000) 2019-10-13T12:16:30.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:30.238+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.240441+0000) 2019-10-13T12:16:30.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:30.238+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.240790+0000) 2019-10-13T12:16:30.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:30.238+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.241157+0000) 2019-10-13T12:16:30.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:30.238+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.241807+0000) 2019-10-13T12:16:30.243 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:30.238+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.242007+0000) 2019-10-13T12:16:30.243 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:30.238+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.242361+0000) 2019-10-13T12:16:30.248 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:30.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:30.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:30.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:30.248829+0000) 2019-10-13T12:16:32.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:32.218+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.220653+0000) 2019-10-13T12:16:35.251 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:35.248+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.250465+0000) 2019-10-13T12:16:35.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:35.246+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.250402+0000) 2019-10-13T12:16:35.252 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:35.250+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.251707+0000) 2019-10-13T12:16:35.253 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:35.250+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.252539+0000) 2019-10-13T12:16:35.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:35.250+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.252917+0000) 2019-10-13T12:16:35.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:35.250+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.252904+0000) 2019-10-13T12:16:35.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:35.250+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.254168+0000) 2019-10-13T12:16:35.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:35.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:35.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:35.256+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.258789+0000) 2019-10-13T12:16:35.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:35.490+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.494773+0000) 2019-10-13T12:16:35.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:35.526+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.530353+0000) 2019-10-13T12:16:36.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:36.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.464238+0000) 2019-10-13T12:16:38.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:38.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.728673+0000) 2019-10-13T12:16:38.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:38.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.732587+0000) 2019-10-13T12:16:39.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:39.888+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:39.891663+0000) 2019-10-13T12:16:39.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:39.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:39.963488+0000) 2019-10-13T12:16:40.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:40.257+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.261751+0000) 2019-10-13T12:16:40.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:40.260+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.262133+0000) 2019-10-13T12:16:40.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:40.257+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.262570+0000) 2019-10-13T12:16:40.264 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:40.257+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.262958+0000) 2019-10-13T12:16:40.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:40.261+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.263494+0000) 2019-10-13T12:16:40.265 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:40.261+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.264866+0000) 2019-10-13T12:16:40.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:40.261+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.265751+0000) 2019-10-13T12:16:40.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:40.265+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143916 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:40.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:40.264+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:40.269920+0000) 2019-10-13T12:16:42.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:42.217+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.220829+0000) 2019-10-13T12:16:45.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:45.269+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.273893+0000) 2019-10-13T12:16:45.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:45.269+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.274707+0000) 2019-10-13T12:16:45.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:45.269+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.274935+0000) 2019-10-13T12:16:45.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:45.273+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.275471+0000) 2019-10-13T12:16:45.276 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:45.272+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.275760+0000) 2019-10-13T12:16:45.277 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:45.273+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.276221+0000) 2019-10-13T12:16:45.277 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:45.273+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.276587+0000) 2019-10-13T12:16:45.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:45.276+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.279622+0000) 2019-10-13T12:16:45.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:45.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143948 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:45.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:45.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.494953+0000) 2019-10-13T12:16:45.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:45.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.530516+0000) 2019-10-13T12:16:46.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:46.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.464431+0000) 2019-10-13T12:16:48.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:48.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.728872+0000) 2019-10-13T12:16:48.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:48.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.732799+0000) 2019-10-13T12:16:49.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:49.896+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:49.898181+0000) 2019-10-13T12:16:49.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:49.960+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:49.963679+0000) 2019-10-13T12:16:50.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:50.281+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.283866+0000) 2019-10-13T12:16:50.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:50.281+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.284550+0000) 2019-10-13T12:16:50.286 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:50.279+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.285461+0000) 2019-10-13T12:16:50.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:50.281+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.286445+0000) 2019-10-13T12:16:50.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:50.281+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.287128+0000) 2019-10-13T12:16:50.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:50.285+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.287561+0000) 2019-10-13T12:16:50.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:50.285+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.287829+0000) 2019-10-13T12:16:50.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:50.287+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:50.292095+0000) 2019-10-13T12:16:50.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:50.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 143984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:52.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:52.217+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.221025+0000) 2019-10-13T12:16:55.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:55.293+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.295879+0000) 2019-10-13T12:16:55.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:55.291+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.296473+0000) 2019-10-13T12:16:55.298 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:55.293+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.297525+0000) 2019-10-13T12:16:55.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:55.293+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.298058+0000) 2019-10-13T12:16:55.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:55.293+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.299007+0000) 2019-10-13T12:16:55.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:55.293+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.298609+0000) 2019-10-13T12:16:55.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:16:55.297+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.299417+0000) 2019-10-13T12:16:55.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:16:55.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144014 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:16:55.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:55.299+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.303384+0000) 2019-10-13T12:16:55.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:16:55.489+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.495162+0000) 2019-10-13T12:16:55.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:16:55.529+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.530734+0000) 2019-10-13T12:16:55.816 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:16:55.816 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:55.820 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:16:55.820 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:56.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:16:56.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.464664+0000) 2019-10-13T12:16:58.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:16:58.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.729074+0000) 2019-10-13T12:16:58.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:16:58.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.732997+0000) 2019-10-13T12:16:59.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:16:59.899+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:59.904639+0000) 2019-10-13T12:16:59.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:16:59.963+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:59.963871+0000) 2019-10-13T12:17:00.307 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:00.301+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.306410+0000) 2019-10-13T12:17:00.308 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:00.301+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.306738+0000) 2019-10-13T12:17:00.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:00.303+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.307722+0000) 2019-10-13T12:17:00.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:00.305+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.308168+0000) 2019-10-13T12:17:00.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:00.305+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.308792+0000) 2019-10-13T12:17:00.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:00.305+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.308978+0000) 2019-10-13T12:17:00.310 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:00.305+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.309649+0000) 2019-10-13T12:17:00.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:00.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144049 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:00.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:00.315+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:00.318593+0000) 2019-10-13T12:17:02.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:02.217+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.221226+0000) 2019-10-13T12:17:05.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:05.313+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.316373+0000) 2019-10-13T12:17:05.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:05.311+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.316840+0000) 2019-10-13T12:17:05.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:05.313+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.318124+0000) 2019-10-13T12:17:05.319 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:05.313+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.318590+0000) 2019-10-13T12:17:05.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:05.313+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.319135+0000) 2019-10-13T12:17:05.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:05.317+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.320122+0000) 2019-10-13T12:17:05.321 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:05.317+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.320466+0000) 2019-10-13T12:17:05.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:05.319+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.323699+0000) 2019-10-13T12:17:05.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:05.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:05.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:05.493+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.495357+0000) 2019-10-13T12:17:05.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:05.525+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.530975+0000) 2019-10-13T12:17:06.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:06.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.464877+0000) 2019-10-13T12:17:08.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:08.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.729272+0000) 2019-10-13T12:17:08.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:08.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.733191+0000) 2019-10-13T12:17:09.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:09.907+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:09.911204+0000) 2019-10-13T12:17:09.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:09.963+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:09.964064+0000) 2019-10-13T12:17:10.329 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:10.325+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.328568+0000) 2019-10-13T12:17:10.330 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:10.323+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.328703+0000) 2019-10-13T12:17:10.330 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:10.325+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.329258+0000) 2019-10-13T12:17:10.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:10.325+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.329649+0000) 2019-10-13T12:17:10.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:10.325+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.330243+0000) 2019-10-13T12:17:10.332 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:10.325+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.330558+0000) 2019-10-13T12:17:10.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:10.329+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.331716+0000) 2019-10-13T12:17:10.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:10.327+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:10.333958+0000) 2019-10-13T12:17:10.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:10.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144117 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:12.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:12.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.221433+0000) 2019-10-13T12:17:15.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:15.335+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.338641+0000) 2019-10-13T12:17:15.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:15.337+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.339415+0000) 2019-10-13T12:17:15.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:15.337+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.340451+0000) 2019-10-13T12:17:15.342 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:15.337+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.341349+0000) 2019-10-13T12:17:15.342 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:15.337+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.341647+0000) 2019-10-13T12:17:15.343 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:15.337+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.342683+0000) 2019-10-13T12:17:15.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:15.337+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.343097+0000) 2019-10-13T12:17:15.347 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:15.341+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:15.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:15.343+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.348530+0000) 2019-10-13T12:17:15.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:15.493+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.495542+0000) 2019-10-13T12:17:15.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:15.529+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.531160+0000) 2019-10-13T12:17:16.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:16.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.465056+0000) 2019-10-13T12:17:18.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:18.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.729497+0000) 2019-10-13T12:17:18.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:18.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.733346+0000) 2019-10-13T12:17:19.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:19.919+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:19.922845+0000) 2019-10-13T12:17:19.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:19.959+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:19.964259+0000) 2019-10-13T12:17:20.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:20.345+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.350673+0000) 2019-10-13T12:17:20.352 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:20.345+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.351277+0000) 2019-10-13T12:17:20.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:20.347+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.351548+0000) 2019-10-13T12:17:20.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:20.349+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.351911+0000) 2019-10-13T12:17:20.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:20.349+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.352568+0000) 2019-10-13T12:17:20.354 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:20.349+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.353428+0000) 2019-10-13T12:17:20.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:20.349+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.353871+0000) 2019-10-13T12:17:20.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:20.353+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:20.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:20.355+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:20.359063+0000) 2019-10-13T12:17:22.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:22.216+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.221584+0000) 2019-10-13T12:17:25.361 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:25.356+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.360510+0000) 2019-10-13T12:17:25.362 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:25.356+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.361074+0000) 2019-10-13T12:17:25.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:25.356+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.362016+0000) 2019-10-13T12:17:25.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:25.356+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.362730+0000) 2019-10-13T12:17:25.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:25.356+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.363047+0000) 2019-10-13T12:17:25.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:25.360+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.363439+0000) 2019-10-13T12:17:25.365 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:25.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.363783+0000) 2019-10-13T12:17:25.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:25.363+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.367503+0000) 2019-10-13T12:17:25.369 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:25.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:25.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:25.492+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.495721+0000) 2019-10-13T12:17:25.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:25.528+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.531370+0000) 2019-10-13T12:17:25.885 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:17:25.885 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:25.889 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:17:25.889 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:26.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:26.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.465306+0000) 2019-10-13T12:17:28.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:28.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.729728+0000) 2019-10-13T12:17:28.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:28.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.733560+0000) 2019-10-13T12:17:29.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:29.927+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:29.929460+0000) 2019-10-13T12:17:29.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:29.963+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:29.964448+0000) 2019-10-13T12:17:30.372 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:30.368+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.371430+0000) 2019-10-13T12:17:30.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:30.368+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.371948+0000) 2019-10-13T12:17:30.373 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:30.368+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.372618+0000) 2019-10-13T12:17:30.373 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:30.368+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.372983+0000) 2019-10-13T12:17:30.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:30.368+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.373961+0000) 2019-10-13T12:17:30.375 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:30.368+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.374781+0000) 2019-10-13T12:17:30.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:30.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.375126+0000) 2019-10-13T12:17:30.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:30.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144251 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:30.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:30.375+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:30.379746+0000) 2019-10-13T12:17:32.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:32.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.221786+0000) 2019-10-13T12:17:35.382 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:35.376+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.381603+0000) 2019-10-13T12:17:35.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:35.379+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.382066+0000) 2019-10-13T12:17:35.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:35.376+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.382720+0000) 2019-10-13T12:17:35.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:35.380+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.383598+0000) 2019-10-13T12:17:35.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:35.380+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.384132+0000) 2019-10-13T12:17:35.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:35.380+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.383727+0000) 2019-10-13T12:17:35.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:35.380+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.385258+0000) 2019-10-13T12:17:35.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:35.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:35.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:35.387+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.390639+0000) 2019-10-13T12:17:35.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:35.492+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.495906+0000) 2019-10-13T12:17:35.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:35.528+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.531571+0000) 2019-10-13T12:17:36.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:36.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.465500+0000) 2019-10-13T12:17:38.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:38.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.729932+0000) 2019-10-13T12:17:38.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:38.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.733791+0000) 2019-10-13T12:17:39.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:39.935+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:39.938438+0000) 2019-10-13T12:17:39.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:39.959+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:39.964656+0000) 2019-10-13T12:17:40.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:40.388+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.391791+0000) 2019-10-13T12:17:40.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:40.388+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.392597+0000) 2019-10-13T12:17:40.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:40.388+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.392810+0000) 2019-10-13T12:17:40.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:40.387+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.393591+0000) 2019-10-13T12:17:40.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:40.388+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.394252+0000) 2019-10-13T12:17:40.395 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:40.388+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.395023+0000) 2019-10-13T12:17:40.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:40.392+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.395366+0000) 2019-10-13T12:17:40.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:40.396+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:40.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:40.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:40.400875+0000) 2019-10-13T12:17:42.222 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:42.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.221978+0000) 2019-10-13T12:17:45.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:45.400+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.403651+0000) 2019-10-13T12:17:45.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:45.400+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.404483+0000) 2019-10-13T12:17:45.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:45.398+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.404382+0000) 2019-10-13T12:17:45.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:45.400+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.405246+0000) 2019-10-13T12:17:45.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:45.400+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.405536+0000) 2019-10-13T12:17:45.407 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:45.400+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.406569+0000) 2019-10-13T12:17:45.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:45.400+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.406834+0000) 2019-10-13T12:17:45.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:45.404+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:45.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:45.406+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.412200+0000) 2019-10-13T12:17:45.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:45.492+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.496104+0000) 2019-10-13T12:17:45.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:45.528+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.531777+0000) 2019-10-13T12:17:46.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:46.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.465689+0000) 2019-10-13T12:17:48.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:48.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.730193+0000) 2019-10-13T12:17:48.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:48.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.733991+0000) 2019-10-13T12:17:49.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:49.946+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:49.949452+0000) 2019-10-13T12:17:49.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:49.962+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:49.964853+0000) 2019-10-13T12:17:50.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:50.406+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.414001+0000) 2019-10-13T12:17:50.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:50.408+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.415182+0000) 2019-10-13T12:17:50.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:50.412+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.415590+0000) 2019-10-13T12:17:50.417 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:50.412+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.415951+0000) 2019-10-13T12:17:50.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:50.412+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.416512+0000) 2019-10-13T12:17:50.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:50.412+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.416682+0000) 2019-10-13T12:17:50.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:50.412+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.417399+0000) 2019-10-13T12:17:50.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:50.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:50.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:50.418+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:50.423932+0000) 2019-10-13T12:17:52.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:52.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.222194+0000) 2019-10-13T12:17:55.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:55.418+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.425002+0000) 2019-10-13T12:17:55.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:55.420+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.426264+0000) 2019-10-13T12:17:55.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:55.420+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.426091+0000) 2019-10-13T12:17:55.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:55.420+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.426987+0000) 2019-10-13T12:17:55.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:55.424+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.427819+0000) 2019-10-13T12:17:55.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:17:55.424+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.428126+0000) 2019-10-13T12:17:55.429 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:55.424+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.428739+0000) 2019-10-13T12:17:55.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:17:55.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:17:55.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:55.426+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.433632+0000) 2019-10-13T12:17:55.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:17:55.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.496288+0000) 2019-10-13T12:17:55.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:17:55.528+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.531963+0000) 2019-10-13T12:17:55.997 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:17:55.997 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:56.001 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:17:56.001 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:56.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:17:56.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.465885+0000) 2019-10-13T12:17:58.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:17:58.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.730386+0000) 2019-10-13T12:17:58.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:17:58.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.734215+0000) 2019-10-13T12:17:59.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:17:59.958+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:59.959919+0000) 2019-10-13T12:17:59.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:17:59.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:59.965052+0000) 2019-10-13T12:18:00.437 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:00.432+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.436046+0000) 2019-10-13T12:18:00.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:00.430+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.436411+0000) 2019-10-13T12:18:00.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:00.432+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.437189+0000) 2019-10-13T12:18:00.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:00.432+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.437588+0000) 2019-10-13T12:18:00.439 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:00.432+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.438655+0000) 2019-10-13T12:18:00.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:00.432+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.439178+0000) 2019-10-13T12:18:00.440 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:00.436+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.439918+0000) 2019-10-13T12:18:00.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:00.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:00.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:00.442+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:00.446296+0000) 2019-10-13T12:18:02.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:02.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.222408+0000) 2019-10-13T12:18:05.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:05.443+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.447091+0000) 2019-10-13T12:18:05.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:05.443+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.447710+0000) 2019-10-13T12:18:05.449 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:05.443+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.448252+0000) 2019-10-13T12:18:05.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:05.443+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.449103+0000) 2019-10-13T12:18:05.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:05.442+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.449265+0000) 2019-10-13T12:18:05.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:05.443+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.449756+0000) 2019-10-13T12:18:05.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:05.443+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.450490+0000) 2019-10-13T12:18:05.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:05.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:05.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:05.450+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.457715+0000) 2019-10-13T12:18:05.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:05.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.496464+0000) 2019-10-13T12:18:05.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:05.531+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.532136+0000) 2019-10-13T12:18:06.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:06.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.466060+0000) 2019-10-13T12:18:08.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:08.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.730594+0000) 2019-10-13T12:18:08.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:08.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.734417+0000) 2019-10-13T12:18:09.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:09.962+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:09.965284+0000) 2019-10-13T12:18:09.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:09.970+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:09.970986+0000) 2019-10-13T12:18:10.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:10.451+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.457814+0000) 2019-10-13T12:18:10.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:10.451+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.458103+0000) 2019-10-13T12:18:10.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:10.454+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.459186+0000) 2019-10-13T12:18:10.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:10.455+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.460023+0000) 2019-10-13T12:18:10.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:10.455+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.460265+0000) 2019-10-13T12:18:10.463 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:10.455+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.460716+0000) 2019-10-13T12:18:10.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:10.455+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.461188+0000) 2019-10-13T12:18:10.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:10.459+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:10.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:10.462+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:10.466507+0000) 2019-10-13T12:18:12.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:12.219+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.222614+0000) 2019-10-13T12:18:15.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:15.463+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.468569+0000) 2019-10-13T12:18:15.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:15.463+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.469424+0000) 2019-10-13T12:18:15.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:15.463+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.470008+0000) 2019-10-13T12:18:15.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:15.466+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.470564+0000) 2019-10-13T12:18:15.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:15.463+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.470265+0000) 2019-10-13T12:18:15.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:15.467+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.471421+0000) 2019-10-13T12:18:15.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:15.467+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.472012+0000) 2019-10-13T12:18:15.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:15.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144549 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:15.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:15.470+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.477895+0000) 2019-10-13T12:18:15.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:15.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.496640+0000) 2019-10-13T12:18:15.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:15.531+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.532347+0000) 2019-10-13T12:18:16.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:16.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.466281+0000) 2019-10-13T12:18:18.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:18.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.730854+0000) 2019-10-13T12:18:18.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:18.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.734647+0000) 2019-10-13T12:18:19.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:19.958+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:19.965504+0000) 2019-10-13T12:18:19.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:19.974+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:19.982067+0000) 2019-10-13T12:18:20.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:20.475+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.478611+0000) 2019-10-13T12:18:20.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:20.475+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.479963+0000) 2019-10-13T12:18:20.481 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:20.475+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.480638+0000) 2019-10-13T12:18:20.482 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:20.475+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.481310+0000) 2019-10-13T12:18:20.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:20.474+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.481433+0000) 2019-10-13T12:18:20.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:20.475+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.482315+0000) 2019-10-13T12:18:20.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:20.475+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.482916+0000) 2019-10-13T12:18:20.487 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:20.479+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:20.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:20.482+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:20.487987+0000) 2019-10-13T12:18:22.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:22.219+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.222816+0000) 2019-10-13T12:18:25.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:25.483+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.490553+0000) 2019-10-13T12:18:25.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:25.483+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.490703+0000) 2019-10-13T12:18:25.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:25.486+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.491011+0000) 2019-10-13T12:18:25.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:25.487+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.491866+0000) 2019-10-13T12:18:25.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:25.487+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.492809+0000) 2019-10-13T12:18:25.494 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:25.487+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.493062+0000) 2019-10-13T12:18:25.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:25.487+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.493684+0000) 2019-10-13T12:18:25.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:25.491+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.496814+0000) 2019-10-13T12:18:25.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:25.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:25.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:25.494+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.498668+0000) 2019-10-13T12:18:25.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:25.531+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.532551+0000) 2019-10-13T12:18:26.092 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:18:26.092 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:26.095 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:18:26.096 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:26.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:26.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.466455+0000) 2019-10-13T12:18:28.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:28.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.731054+0000) 2019-10-13T12:18:28.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:28.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.734871+0000) 2019-10-13T12:18:29.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:29.962+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:29.965697+0000) 2019-10-13T12:18:29.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:29.986+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:29.988974+0000) 2019-10-13T12:18:30.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:30.495+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.501793+0000) 2019-10-13T12:18:30.503 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:30.495+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.502142+0000) 2019-10-13T12:18:30.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:30.495+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.502765+0000) 2019-10-13T12:18:30.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:30.495+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.502907+0000) 2019-10-13T12:18:30.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:30.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.503548+0000) 2019-10-13T12:18:30.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:30.498+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.504172+0000) 2019-10-13T12:18:30.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:30.499+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.504505+0000) 2019-10-13T12:18:30.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:30.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:30.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:30.506+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:30.511825+0000) 2019-10-13T12:18:32.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:32.219+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.223016+0000) 2019-10-13T12:18:35.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:35.495+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.496993+0000) 2019-10-13T12:18:35.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:35.507+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.510953+0000) 2019-10-13T12:18:35.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:35.511+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.511794+0000) 2019-10-13T12:18:35.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:35.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.512064+0000) 2019-10-13T12:18:35.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:35.511+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.512356+0000) 2019-10-13T12:18:35.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:35.511+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.513093+0000) 2019-10-13T12:18:35.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:35.510+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.513225+0000) 2019-10-13T12:18:35.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:35.511+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.513924+0000) 2019-10-13T12:18:35.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:35.515+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:35.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:35.518+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.520886+0000) 2019-10-13T12:18:35.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:35.531+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.532718+0000) 2019-10-13T12:18:36.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:36.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.466581+0000) 2019-10-13T12:18:38.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:38.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.731250+0000) 2019-10-13T12:18:38.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:38.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.735065+0000) 2019-10-13T12:18:39.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:39.961+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:39.965905+0000) 2019-10-13T12:18:39.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:39.993+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:39.995608+0000) 2019-10-13T12:18:40.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:40.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.522064+0000) 2019-10-13T12:18:40.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:40.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.522726+0000) 2019-10-13T12:18:40.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:40.519+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.522999+0000) 2019-10-13T12:18:40.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:40.523+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.523526+0000) 2019-10-13T12:18:40.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:40.521+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.524336+0000) 2019-10-13T12:18:40.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:40.523+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.524772+0000) 2019-10-13T12:18:40.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:40.523+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.524850+0000) 2019-10-13T12:18:40.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:40.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:40.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:40.529+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:40.531849+0000) 2019-10-13T12:18:42.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:42.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.223213+0000) 2019-10-13T12:18:45.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:45.495+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.497190+0000) 2019-10-13T12:18:45.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:45.531+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.532097+0000) 2019-10-13T12:18:45.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:45.531+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.532852+0000) 2019-10-13T12:18:45.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:45.531+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.533142+0000) 2019-10-13T12:18:45.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:45.531+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.534215+0000) 2019-10-13T12:18:45.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:45.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.534808+0000) 2019-10-13T12:18:45.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:45.531+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.535277+0000) 2019-10-13T12:18:45.537 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:45.535+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.536345+0000) 2019-10-13T12:18:45.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:45.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:45.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:45.541+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.542262+0000) 2019-10-13T12:18:46.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:46.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.466759+0000) 2019-10-13T12:18:48.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:48.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.731541+0000) 2019-10-13T12:18:48.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:48.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.735297+0000) 2019-10-13T12:18:49.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:49.965+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:49.966095+0000) 2019-10-13T12:18:50.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:50.005+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.006950+0000) 2019-10-13T12:18:50.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:50.538+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.543158+0000) 2019-10-13T12:18:50.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:50.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.543956+0000) 2019-10-13T12:18:50.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:50.542+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.544391+0000) 2019-10-13T12:18:50.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:50.542+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.545261+0000) 2019-10-13T12:18:50.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:50.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.546007+0000) 2019-10-13T12:18:50.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:50.542+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.546282+0000) 2019-10-13T12:18:50.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:50.546+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.547430+0000) 2019-10-13T12:18:50.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:50.546+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:50.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:50.549+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:50.552100+0000) 2019-10-13T12:18:52.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:52.222+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.223414+0000) 2019-10-13T12:18:52.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:52.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.283387+0000) 2019-10-13T12:18:55.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:55.494+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.497403+0000) 2019-10-13T12:18:55.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:18:55.550+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.554356+0000) 2019-10-13T12:18:55.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:55.550+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.555233+0000) 2019-10-13T12:18:55.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:18:55.554+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.555673+0000) 2019-10-13T12:18:55.557 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:55.554+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.556143+0000) 2019-10-13T12:18:55.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:55.554+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.556882+0000) 2019-10-13T12:18:55.558 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:55.553+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.557148+0000) 2019-10-13T12:18:55.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:18:55.554+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.557571+0000) 2019-10-13T12:18:55.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:18:55.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:18:55.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:18:55.557+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.561158+0000) 2019-10-13T12:18:56.202 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:18:56.202 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:56.206 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:18:56.206 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:56.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:18:56.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.466959+0000) 2019-10-13T12:18:58.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:18:58.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.731694+0000) 2019-10-13T12:18:58.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:18:58.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.735516+0000) 2019-10-13T12:18:59.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:18:59.965+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:59.966328+0000) 2019-10-13T12:19:00.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:00.009+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.013788+0000) 2019-10-13T12:19:00.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:00.562+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.565339+0000) 2019-10-13T12:19:00.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:00.562+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.566255+0000) 2019-10-13T12:19:00.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:00.561+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.566034+0000) 2019-10-13T12:19:00.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:00.562+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.566961+0000) 2019-10-13T12:19:00.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:00.566+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.567829+0000) 2019-10-13T12:19:00.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:00.566+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.568467+0000) 2019-10-13T12:19:00.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:00.566+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.568881+0000) 2019-10-13T12:19:00.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:00.569+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:00.571233+0000) 2019-10-13T12:19:00.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:00.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:02.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:02.222+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.223642+0000) 2019-10-13T12:19:02.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:02.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.283579+0000) 2019-10-13T12:19:05.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:05.494+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.497604+0000) 2019-10-13T12:19:05.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:05.574+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.575681+0000) 2019-10-13T12:19:05.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:05.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.576015+0000) 2019-10-13T12:19:05.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:05.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.576614+0000) 2019-10-13T12:19:05.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:05.574+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.577190+0000) 2019-10-13T12:19:05.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:05.573+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.577445+0000) 2019-10-13T12:19:05.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:05.574+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.578220+0000) 2019-10-13T12:19:05.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:05.574+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.578828+0000) 2019-10-13T12:19:05.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:05.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:05.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:05.585+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.587247+0000) 2019-10-13T12:19:06.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:06.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.467151+0000) 2019-10-13T12:19:08.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:08.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.731931+0000) 2019-10-13T12:19:08.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:08.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.735746+0000) 2019-10-13T12:19:09.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:09.965+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:09.966543+0000) 2019-10-13T12:19:10.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:10.017+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.020249+0000) 2019-10-13T12:19:10.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:10.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.585168+0000) 2019-10-13T12:19:10.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:10.582+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.585643+0000) 2019-10-13T12:19:10.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:10.582+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.586141+0000) 2019-10-13T12:19:10.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:10.586+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.587287+0000) 2019-10-13T12:19:10.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:10.585+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.587656+0000) 2019-10-13T12:19:10.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:10.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.587953+0000) 2019-10-13T12:19:10.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:10.586+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.588272+0000) 2019-10-13T12:19:10.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:10.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:10.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:10.593+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:10.597177+0000) 2019-10-13T12:19:12.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:12.222+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.223832+0000) 2019-10-13T12:19:12.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:12.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.283816+0000) 2019-10-13T12:19:15.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:15.494+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.497788+0000) 2019-10-13T12:19:15.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:15.594+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.595857+0000) 2019-10-13T12:19:15.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:15.594+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.596739+0000) 2019-10-13T12:19:15.598 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:15.593+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.596799+0000) 2019-10-13T12:19:15.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:15.594+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.597843+0000) 2019-10-13T12:19:15.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:15.594+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.598166+0000) 2019-10-13T12:19:15.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:15.594+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.598759+0000) 2019-10-13T12:19:15.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:15.594+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.598956+0000) 2019-10-13T12:19:15.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:15.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:15.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:15.605+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.606574+0000) 2019-10-13T12:19:16.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:16.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.467344+0000) 2019-10-13T12:19:18.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:18.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.732128+0000) 2019-10-13T12:19:18.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:18.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.735951+0000) 2019-10-13T12:19:19.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:19.965+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:19.966766+0000) 2019-10-13T12:19:20.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:20.025+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.026832+0000) 2019-10-13T12:19:20.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:20.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.607248+0000) 2019-10-13T12:19:20.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:20.605+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.607384+0000) 2019-10-13T12:19:20.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:20.606+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.608057+0000) 2019-10-13T12:19:20.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:20.606+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.608846+0000) 2019-10-13T12:19:20.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:20.606+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.609100+0000) 2019-10-13T12:19:20.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:20.606+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.609545+0000) 2019-10-13T12:19:20.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:20.606+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.609845+0000) 2019-10-13T12:19:20.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:20.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 144988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:20.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:20.613+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:20.614456+0000) 2019-10-13T12:19:22.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:22.222+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.224047+0000) 2019-10-13T12:19:22.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:22.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.284048+0000) 2019-10-13T12:19:25.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:25.494+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.497987+0000) 2019-10-13T12:19:25.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:25.614+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.616517+0000) 2019-10-13T12:19:25.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:25.613+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.617047+0000) 2019-10-13T12:19:25.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:25.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.618506+0000) 2019-10-13T12:19:25.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:25.614+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.619171+0000) 2019-10-13T12:19:25.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:25.618+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.619699+0000) 2019-10-13T12:19:25.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:25.618+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.620026+0000) 2019-10-13T12:19:25.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:25.618+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.619823+0000) 2019-10-13T12:19:25.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:25.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:25.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:25.621+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.624767+0000) 2019-10-13T12:19:26.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:19:26.263 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:26.267 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:19:26.267 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:26.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:26.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.467523+0000) 2019-10-13T12:19:28.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:28.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.732285+0000) 2019-10-13T12:19:28.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:28.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.736160+0000) 2019-10-13T12:19:29.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:29.965+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:29.966988+0000) 2019-10-13T12:19:30.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:30.033+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.037096+0000) 2019-10-13T12:19:30.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:30.622+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.626749+0000) 2019-10-13T12:19:30.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:30.625+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.627174+0000) 2019-10-13T12:19:30.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:30.626+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.628256+0000) 2019-10-13T12:19:30.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:30.626+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.629293+0000) 2019-10-13T12:19:30.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:30.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.629018+0000) 2019-10-13T12:19:30.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:30.626+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.629665+0000) 2019-10-13T12:19:30.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:30.626+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.630500+0000) 2019-10-13T12:19:30.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:30.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:30.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:30.633+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:30.634882+0000) 2019-10-13T12:19:32.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:32.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.224248+0000) 2019-10-13T12:19:32.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:32.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.284252+0000) 2019-10-13T12:19:35.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:35.493+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.498184+0000) 2019-10-13T12:19:35.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:35.633+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.637980+0000) 2019-10-13T12:19:35.639 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:35.636+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.638635+0000) 2019-10-13T12:19:35.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:35.633+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.639160+0000) 2019-10-13T12:19:35.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:35.637+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.639593+0000) 2019-10-13T12:19:35.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:35.637+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.639898+0000) 2019-10-13T12:19:35.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:35.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.640437+0000) 2019-10-13T12:19:35.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:35.637+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.641007+0000) 2019-10-13T12:19:35.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:35.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145084 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:35.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:35.644+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.649204+0000) 2019-10-13T12:19:36.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:36.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.467698+0000) 2019-10-13T12:19:38.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:38.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.732519+0000) 2019-10-13T12:19:38.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:38.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.736370+0000) 2019-10-13T12:19:39.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:39.964+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:39.967221+0000) 2019-10-13T12:19:40.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:40.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.047100+0000) 2019-10-13T12:19:40.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:40.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.648127+0000) 2019-10-13T12:19:40.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:40.645+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.648828+0000) 2019-10-13T12:19:40.650 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:40.645+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.648976+0000) 2019-10-13T12:19:40.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:40.645+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.649306+0000) 2019-10-13T12:19:40.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:40.645+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.650181+0000) 2019-10-13T12:19:40.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:40.648+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.650545+0000) 2019-10-13T12:19:40.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:40.645+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.651000+0000) 2019-10-13T12:19:40.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:40.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:40.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:40.656+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:40.659508+0000) 2019-10-13T12:19:42.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:42.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.224484+0000) 2019-10-13T12:19:42.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:42.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.284488+0000) 2019-10-13T12:19:45.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:45.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.498574+0000) 2019-10-13T12:19:45.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:45.653+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.658156+0000) 2019-10-13T12:19:45.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:45.657+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.659281+0000) 2019-10-13T12:19:45.660 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:45.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.659766+0000) 2019-10-13T12:19:45.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:45.657+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.660064+0000) 2019-10-13T12:19:45.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:45.657+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.660656+0000) 2019-10-13T12:19:45.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:45.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.660803+0000) 2019-10-13T12:19:45.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:45.656+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.661333+0000) 2019-10-13T12:19:45.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:45.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:45.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:45.660+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.665455+0000) 2019-10-13T12:19:46.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:46.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.467890+0000) 2019-10-13T12:19:48.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:48.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.732750+0000) 2019-10-13T12:19:48.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:48.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.736566+0000) 2019-10-13T12:19:49.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:49.964+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:49.967443+0000) 2019-10-13T12:19:50.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:50.048+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.053712+0000) 2019-10-13T12:19:50.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:50.665+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.668194+0000) 2019-10-13T12:19:50.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:50.665+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.668707+0000) 2019-10-13T12:19:50.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:50.665+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.669079+0000) 2019-10-13T12:19:50.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:50.665+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.669744+0000) 2019-10-13T12:19:50.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:50.665+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.670281+0000) 2019-10-13T12:19:50.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:50.668+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.670430+0000) 2019-10-13T12:19:50.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:50.665+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.670923+0000) 2019-10-13T12:19:50.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:50.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:50.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:50.676+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:50.679733+0000) 2019-10-13T12:19:52.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:52.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.224712+0000) 2019-10-13T12:19:52.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:52.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.284701+0000) 2019-10-13T12:19:55.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:55.493+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.498776+0000) 2019-10-13T12:19:55.681 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:19:55.677+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.679693+0000) 2019-10-13T12:19:55.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:19:55.677+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.680443+0000) 2019-10-13T12:19:55.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:55.676+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.680739+0000) 2019-10-13T12:19:55.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:55.677+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.681124+0000) 2019-10-13T12:19:55.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:55.677+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.681852+0000) 2019-10-13T12:19:55.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:55.677+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.682647+0000) 2019-10-13T12:19:55.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:19:55.677+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.683164+0000) 2019-10-13T12:19:55.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:19:55.681+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:19:55.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:19:55.684+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.687536+0000) 2019-10-13T12:19:56.397 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:19:56.397 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:56.401 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:19:56.401 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:56.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:19:56.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.468072+0000) 2019-10-13T12:19:58.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:19:58.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.732949+0000) 2019-10-13T12:19:58.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:19:58.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.736792+0000) 2019-10-13T12:19:59.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:19:59.964+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:59.967664+0000) 2019-10-13T12:20:00.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:00.060+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.060642+0000) 2019-10-13T12:20:00.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:00.685+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.690414+0000) 2019-10-13T12:20:00.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:00.688+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.690740+0000) 2019-10-13T12:20:00.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:00.685+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.691250+0000) 2019-10-13T12:20:00.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:00.689+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.691539+0000) 2019-10-13T12:20:00.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:00.689+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.692077+0000) 2019-10-13T12:20:00.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:00.689+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.692508+0000) 2019-10-13T12:20:00.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:00.689+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.693301+0000) 2019-10-13T12:20:00.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:00.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:00.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:00.696+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:00.698936+0000) 2019-10-13T12:20:02.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:02.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.224951+0000) 2019-10-13T12:20:02.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:02.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.284907+0000) 2019-10-13T12:20:05.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:05.493+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.498975+0000) 2019-10-13T12:20:05.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:05.697+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.700799+0000) 2019-10-13T12:20:05.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:05.697+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.701339+0000) 2019-10-13T12:20:05.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:05.697+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.701900+0000) 2019-10-13T12:20:05.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:05.697+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.701755+0000) 2019-10-13T12:20:05.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:05.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.702592+0000) 2019-10-13T12:20:05.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:05.697+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.703192+0000) 2019-10-13T12:20:05.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:05.701+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.704257+0000) 2019-10-13T12:20:05.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:05.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:05.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:05.708+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.710636+0000) 2019-10-13T12:20:06.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:06.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:06.468248+0000) 2019-10-13T12:20:08.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:08.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.733148+0000) 2019-10-13T12:20:08.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:08.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.737025+0000) 2019-10-13T12:20:09.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:09.964+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:09.967856+0000) 2019-10-13T12:20:10.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:10.064+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.067160+0000) 2019-10-13T12:20:10.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:10.705+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.710474+0000) 2019-10-13T12:20:10.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:10.709+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.711379+0000) 2019-10-13T12:20:10.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:10.709+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.711751+0000) 2019-10-13T12:20:10.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:10.709+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.712845+0000) 2019-10-13T12:20:10.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:10.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.713048+0000) 2019-10-13T12:20:10.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:10.709+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.713548+0000) 2019-10-13T12:20:10.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:10.709+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.715014+0000) 2019-10-13T12:20:10.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:10.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:10.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:10.716+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:10.719842+0000) 2019-10-13T12:20:12.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:12.221+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.225173+0000) 2019-10-13T12:20:12.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:12.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.285142+0000) 2019-10-13T12:20:15.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:15.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.499174+0000) 2019-10-13T12:20:15.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:15.716+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.722161+0000) 2019-10-13T12:20:15.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:15.716+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.722693+0000) 2019-10-13T12:20:15.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:15.716+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.722999+0000) 2019-10-13T12:20:15.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:15.720+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.723244+0000) 2019-10-13T12:20:15.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:15.720+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.723978+0000) 2019-10-13T12:20:15.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:15.720+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.724710+0000) 2019-10-13T12:20:15.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:15.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.725290+0000) 2019-10-13T12:20:15.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:15.724+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.728185+0000) 2019-10-13T12:20:15.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:15.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:16.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:16.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:16.468424+0000) 2019-10-13T12:20:18.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:18.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.733374+0000) 2019-10-13T12:20:18.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:18.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.737221+0000) 2019-10-13T12:20:19.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:19.964+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:19.968090+0000) 2019-10-13T12:20:20.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:20.072+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.075131+0000) 2019-10-13T12:20:20.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:20.728+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.732780+0000) 2019-10-13T12:20:20.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:20.728+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.733115+0000) 2019-10-13T12:20:20.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:20.728+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.733880+0000) 2019-10-13T12:20:20.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:20.728+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.734229+0000) 2019-10-13T12:20:20.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:20.728+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.734641+0000) 2019-10-13T12:20:20.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:20.732+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.734709+0000) 2019-10-13T12:20:20.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:20.732+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.735878+0000) 2019-10-13T12:20:20.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:20.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145388 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:20.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:20.740+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:20.742567+0000) 2019-10-13T12:20:22.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:22.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.225393+0000) 2019-10-13T12:20:22.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:22.280+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.285340+0000) 2019-10-13T12:20:25.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:25.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.499335+0000) 2019-10-13T12:20:25.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:25.740+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.743481+0000) 2019-10-13T12:20:25.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:25.740+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.744413+0000) 2019-10-13T12:20:25.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:25.740+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.744792+0000) 2019-10-13T12:20:25.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:25.740+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.745183+0000) 2019-10-13T12:20:25.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:25.740+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.745998+0000) 2019-10-13T12:20:25.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:25.740+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.747020+0000) 2019-10-13T12:20:25.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:25.740+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.745418+0000) 2019-10-13T12:20:25.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:25.748+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.750598+0000) 2019-10-13T12:20:25.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:25.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:26.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:26.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:26.468596+0000) 2019-10-13T12:20:26.520 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:20:26.520 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:26.524 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:20:26.524 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:28.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:28.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.733593+0000) 2019-10-13T12:20:28.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:28.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.737431+0000) 2019-10-13T12:20:29.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:29.963+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:29.968284+0000) 2019-10-13T12:20:30.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:30.079+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.081564+0000) 2019-10-13T12:20:30.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:30.747+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.753898+0000) 2019-10-13T12:20:30.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:30.748+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.753547+0000) 2019-10-13T12:20:30.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:30.748+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.754384+0000) 2019-10-13T12:20:30.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:30.748+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.755273+0000) 2019-10-13T12:20:30.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:30.752+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.755728+0000) 2019-10-13T12:20:30.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:30.752+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.756212+0000) 2019-10-13T12:20:30.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:30.752+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.757118+0000) 2019-10-13T12:20:30.762 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:30.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:30.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:30.759+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:30.764322+0000) 2019-10-13T12:20:32.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:32.224+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.225642+0000) 2019-10-13T12:20:32.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:32.280+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.285568+0000) 2019-10-13T12:20:35.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:35.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.499539+0000) 2019-10-13T12:20:35.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:35.759+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.764346+0000) 2019-10-13T12:20:35.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:35.760+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.765622+0000) 2019-10-13T12:20:35.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:35.760+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.766238+0000) 2019-10-13T12:20:35.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:35.760+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.766414+0000) 2019-10-13T12:20:35.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:35.760+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.766872+0000) 2019-10-13T12:20:35.768 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:35.764+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.767916+0000) 2019-10-13T12:20:35.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:35.764+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.768754+0000) 2019-10-13T12:20:35.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:35.767+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.771272+0000) 2019-10-13T12:20:35.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:35.768+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:36.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:36.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:36.468775+0000) 2019-10-13T12:20:38.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:38.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.733794+0000) 2019-10-13T12:20:38.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:38.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.737629+0000) 2019-10-13T12:20:39.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:39.967+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:39.968490+0000) 2019-10-13T12:20:40.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:40.083+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.088155+0000) 2019-10-13T12:20:40.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:40.772+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.776757+0000) 2019-10-13T12:20:40.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:40.772+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.777213+0000) 2019-10-13T12:20:40.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:40.772+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.778424+0000) 2019-10-13T12:20:40.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:40.775+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.778787+0000) 2019-10-13T12:20:40.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:40.776+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.779817+0000) 2019-10-13T12:20:40.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:40.776+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.780309+0000) 2019-10-13T12:20:40.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:40.776+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.780899+0000) 2019-10-13T12:20:40.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:40.780+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:40.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:40.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:40.784420+0000) 2019-10-13T12:20:42.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:42.220+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.225846+0000) 2019-10-13T12:20:42.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:42.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.285767+0000) 2019-10-13T12:20:45.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:45.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.499742+0000) 2019-10-13T12:20:45.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:45.780+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.786636+0000) 2019-10-13T12:20:45.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:45.780+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.786724+0000) 2019-10-13T12:20:45.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:45.784+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.788392+0000) 2019-10-13T12:20:45.789 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:45.784+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.788796+0000) 2019-10-13T12:20:45.790 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:45.783+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.789003+0000) 2019-10-13T12:20:45.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:45.784+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.789420+0000) 2019-10-13T12:20:45.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:45.784+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.790701+0000) 2019-10-13T12:20:45.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:45.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145553 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:45.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:45.791+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.795120+0000) 2019-10-13T12:20:46.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:46.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:46.468947+0000) 2019-10-13T12:20:48.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:48.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.733991+0000) 2019-10-13T12:20:48.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:48.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.737859+0000) 2019-10-13T12:20:49.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:49.967+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:49.968702+0000) 2019-10-13T12:20:50.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:50.091+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.094638+0000) 2019-10-13T12:20:50.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:50.792+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.797180+0000) 2019-10-13T12:20:50.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:50.792+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.797992+0000) 2019-10-13T12:20:50.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:50.792+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.798641+0000) 2019-10-13T12:20:50.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:50.792+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.799143+0000) 2019-10-13T12:20:50.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:50.796+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.800008+0000) 2019-10-13T12:20:50.801 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:50.795+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.800877+0000) 2019-10-13T12:20:50.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:50.796+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.801356+0000) 2019-10-13T12:20:50.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:50.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:50.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:50.803+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:50.806734+0000) 2019-10-13T12:20:52.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:52.224+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.226042+0000) 2019-10-13T12:20:52.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:52.280+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.285968+0000) 2019-10-13T12:20:55.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:55.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.499945+0000) 2019-10-13T12:20:55.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:20:55.804+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.808657+0000) 2019-10-13T12:20:55.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:55.804+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.809974+0000) 2019-10-13T12:20:55.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:55.804+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.810335+0000) 2019-10-13T12:20:55.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:55.807+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.810485+0000) 2019-10-13T12:20:55.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:20:55.808+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.811471+0000) 2019-10-13T12:20:55.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:55.808+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.811744+0000) 2019-10-13T12:20:55.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:20:55.808+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.812165+0000) 2019-10-13T12:20:55.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:20:55.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:20:55.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:20:55.811+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.816051+0000) 2019-10-13T12:20:56.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:20:56.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:56.469120+0000) 2019-10-13T12:20:56.643 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:20:56.643 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:56.665 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:20:56.666 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:58.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:20:58.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.734187+0000) 2019-10-13T12:20:58.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:20:58.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.738101+0000) 2019-10-13T12:20:59.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:20:59.963+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:59.968897+0000) 2019-10-13T12:21:00.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:00.099+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.100732+0000) 2019-10-13T12:21:00.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:00.811+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.818669+0000) 2019-10-13T12:21:00.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:00.815+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.819502+0000) 2019-10-13T12:21:00.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:00.815+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.820104+0000) 2019-10-13T12:21:00.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:00.815+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.821110+0000) 2019-10-13T12:21:00.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:00.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.822015+0000) 2019-10-13T12:21:00.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:00.815+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.822615+0000) 2019-10-13T12:21:00.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:00.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.822844+0000) 2019-10-13T12:21:00.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:00.819+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:00.825287+0000) 2019-10-13T12:21:00.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:00.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:02.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:02.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.226242+0000) 2019-10-13T12:21:02.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:02.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.286189+0000) 2019-10-13T12:21:05.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:05.499+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.500134+0000) 2019-10-13T12:21:05.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:05.823+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.830287+0000) 2019-10-13T12:21:05.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:05.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.831650+0000) 2019-10-13T12:21:05.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:05.827+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.832123+0000) 2019-10-13T12:21:05.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:05.827+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.832661+0000) 2019-10-13T12:21:05.833 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:05.827+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.832981+0000) 2019-10-13T12:21:05.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:05.827+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.833488+0000) 2019-10-13T12:21:05.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:05.827+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.834461+0000) 2019-10-13T12:21:05.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:05.831+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.835566+0000) 2019-10-13T12:21:05.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:05.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:06.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:06.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:06.469295+0000) 2019-10-13T12:21:08.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:08.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.734379+0000) 2019-10-13T12:21:08.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:08.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.738282+0000) 2019-10-13T12:21:09.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:09.967+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:09.969125+0000) 2019-10-13T12:21:10.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:10.103+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.107133+0000) 2019-10-13T12:21:10.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:10.835+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.840080+0000) 2019-10-13T12:21:10.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:10.835+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.840426+0000) 2019-10-13T12:21:10.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:10.835+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.841254+0000) 2019-10-13T12:21:10.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:10.835+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.841749+0000) 2019-10-13T12:21:10.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:10.835+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.842724+0000) 2019-10-13T12:21:10.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:10.839+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.843481+0000) 2019-10-13T12:21:10.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:10.839+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.843848+0000) 2019-10-13T12:21:10.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:10.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:10.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:10.843+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:10.848825+0000) 2019-10-13T12:21:12.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:12.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.226441+0000) 2019-10-13T12:21:12.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:12.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.286434+0000) 2019-10-13T12:21:15.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:15.499+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.500331+0000) 2019-10-13T12:21:15.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:15.843+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.850710+0000) 2019-10-13T12:21:15.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:15.843+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.851127+0000) 2019-10-13T12:21:15.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:15.847+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.852744+0000) 2019-10-13T12:21:15.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:15.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.852978+0000) 2019-10-13T12:21:15.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:15.847+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.853532+0000) 2019-10-13T12:21:15.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:15.847+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.854015+0000) 2019-10-13T12:21:15.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:15.847+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.855200+0000) 2019-10-13T12:21:15.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:15.851+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.856673+0000) 2019-10-13T12:21:15.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:15.851+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:16.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:16.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:16.469521+0000) 2019-10-13T12:21:18.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:18.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.734601+0000) 2019-10-13T12:21:18.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:18.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.738484+0000) 2019-10-13T12:21:19.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:19.967+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:19.969319+0000) 2019-10-13T12:21:20.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:20.111+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.116113+0000) 2019-10-13T12:21:20.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:20.855+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.861252+0000) 2019-10-13T12:21:20.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:20.855+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.861841+0000) 2019-10-13T12:21:20.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:20.855+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.862190+0000) 2019-10-13T12:21:20.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:20.855+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.862653+0000) 2019-10-13T12:21:20.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:20.859+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.863563+0000) 2019-10-13T12:21:20.865 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:20.859+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.863882+0000) 2019-10-13T12:21:20.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:20.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.864537+0000) 2019-10-13T12:21:20.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:20.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:20.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:20.863+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:20.869877+0000) 2019-10-13T12:21:22.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:22.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.226640+0000) 2019-10-13T12:21:22.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:22.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.286645+0000) 2019-10-13T12:21:25.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:25.499+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.500551+0000) 2019-10-13T12:21:25.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:25.867+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.870906+0000) 2019-10-13T12:21:25.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:25.867+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.871870+0000) 2019-10-13T12:21:25.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:25.867+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.872472+0000) 2019-10-13T12:21:25.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:25.867+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.872854+0000) 2019-10-13T12:21:25.874 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:25.867+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.873669+0000) 2019-10-13T12:21:25.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:25.866+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.873810+0000) 2019-10-13T12:21:25.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:25.867+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.874400+0000) 2019-10-13T12:21:25.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:25.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:25.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:25.874+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.879267+0000) 2019-10-13T12:21:26.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:26.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:26.469703+0000) 2019-10-13T12:21:26.805 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:21:26.805 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:26.809 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:21:26.809 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:28.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:28.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.734792+0000) 2019-10-13T12:21:28.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:28.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.738682+0000) 2019-10-13T12:21:29.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:29.966+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:29.969512+0000) 2019-10-13T12:21:30.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:30.118+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.122735+0000) 2019-10-13T12:21:30.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:30.879+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.881113+0000) 2019-10-13T12:21:30.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:30.874+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.881772+0000) 2019-10-13T12:21:30.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:30.879+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.881786+0000) 2019-10-13T12:21:30.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:30.879+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.883169+0000) 2019-10-13T12:21:30.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:30.883+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.883347+0000) 2019-10-13T12:21:30.885 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:30.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.884347+0000) 2019-10-13T12:21:30.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:30.883+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.884600+0000) 2019-10-13T12:21:30.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:30.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:30.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:30.886+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:30.891521+0000) 2019-10-13T12:21:32.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:32.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.226847+0000) 2019-10-13T12:21:32.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:32.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.286856+0000) 2019-10-13T12:21:35.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:35.499+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.500777+0000) 2019-10-13T12:21:35.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:35.891+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.892001+0000) 2019-10-13T12:21:35.894 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:35.886+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.893030+0000) 2019-10-13T12:21:35.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:35.891+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.892854+0000) 2019-10-13T12:21:35.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:35.891+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.893839+0000) 2019-10-13T12:21:35.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:35.891+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.894455+0000) 2019-10-13T12:21:35.897 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:35.891+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.895271+0000) 2019-10-13T12:21:35.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:35.895+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.895688+0000) 2019-10-13T12:21:35.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:35.899+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:35.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:35.894+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.900554+0000) 2019-10-13T12:21:36.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:36.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:36.469893+0000) 2019-10-13T12:21:38.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:38.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.735018+0000) 2019-10-13T12:21:38.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:38.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.738872+0000) 2019-10-13T12:21:39.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:39.966+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:39.969709+0000) 2019-10-13T12:21:40.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:40.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.129523+0000) 2019-10-13T12:21:40.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:40.898+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.902883+0000) 2019-10-13T12:21:40.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:40.902+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.903512+0000) 2019-10-13T12:21:40.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:40.902+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.904089+0000) 2019-10-13T12:21:40.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:40.902+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.904785+0000) 2019-10-13T12:21:40.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:40.902+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.905318+0000) 2019-10-13T12:21:40.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:40.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.906628+0000) 2019-10-13T12:21:40.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:40.906+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.908082+0000) 2019-10-13T12:21:40.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:40.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:40.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:40.906+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:40.912306+0000) 2019-10-13T12:21:42.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:42.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.227050+0000) 2019-10-13T12:21:42.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:42.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.287106+0000) 2019-10-13T12:21:45.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:45.498+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.500998+0000) 2019-10-13T12:21:45.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:45.910+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.914331+0000) 2019-10-13T12:21:45.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:45.910+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.914933+0000) 2019-10-13T12:21:45.916 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:45.910+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.915195+0000) 2019-10-13T12:21:45.916 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:45.914+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.915804+0000) 2019-10-13T12:21:45.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:45.910+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.916393+0000) 2019-10-13T12:21:45.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:45.914+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.917009+0000) 2019-10-13T12:21:45.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:45.914+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.917476+0000) 2019-10-13T12:21:45.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:45.914+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.919537+0000) 2019-10-13T12:21:45.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:45.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:46.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:46.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:46.470083+0000) 2019-10-13T12:21:48.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:48.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.735224+0000) 2019-10-13T12:21:48.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:48.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.739096+0000) 2019-10-13T12:21:49.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:49.966+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:49.969904+0000) 2019-10-13T12:21:50.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:50.130+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.136173+0000) 2019-10-13T12:21:50.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:50.922+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.924972+0000) 2019-10-13T12:21:50.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:50.922+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.924921+0000) 2019-10-13T12:21:50.926 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:50.922+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.925529+0000) 2019-10-13T12:21:50.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:50.922+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.926260+0000) 2019-10-13T12:21:50.927 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:50.922+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.926716+0000) 2019-10-13T12:21:50.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:50.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.926983+0000) 2019-10-13T12:21:50.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:50.926+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.927456+0000) 2019-10-13T12:21:50.932 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:50.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 145995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:50.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:50.926+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:50.932233+0000) 2019-10-13T12:21:52.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:52.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.227246+0000) 2019-10-13T12:21:52.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:52.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.287343+0000) 2019-10-13T12:21:55.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:55.498+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.501193+0000) 2019-10-13T12:21:55.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:55.930+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.934627+0000) 2019-10-13T12:21:55.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:21:55.930+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.935199+0000) 2019-10-13T12:21:55.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:55.934+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.935669+0000) 2019-10-13T12:21:55.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:55.930+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.935812+0000) 2019-10-13T12:21:55.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:21:55.934+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.936721+0000) 2019-10-13T12:21:55.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:55.934+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.937203+0000) 2019-10-13T12:21:55.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:21:55.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.937508+0000) 2019-10-13T12:21:55.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:21:55.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:21:55.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:21:55.938+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.944826+0000) 2019-10-13T12:21:56.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:21:56.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:56.470277+0000) 2019-10-13T12:21:56.904 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:21:56.904 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:56.908 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:21:56.908 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:58.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:21:58.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.735399+0000) 2019-10-13T12:21:58.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:21:58.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.739374+0000) 2019-10-13T12:21:59.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:21:59.966+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:59.970097+0000) 2019-10-13T12:22:00.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:00.138+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.142586+0000) 2019-10-13T12:22:00.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:00.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.945031+0000) 2019-10-13T12:22:00.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:00.942+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.945927+0000) 2019-10-13T12:22:00.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:00.942+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.946346+0000) 2019-10-13T12:22:00.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:00.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.947398+0000) 2019-10-13T12:22:00.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:00.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.948310+0000) 2019-10-13T12:22:00.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:00.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.948804+0000) 2019-10-13T12:22:00.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:00.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.949129+0000) 2019-10-13T12:22:00.953 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:00.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146063 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:00.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:00.946+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:00.953260+0000) 2019-10-13T12:22:02.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:02.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.227481+0000) 2019-10-13T12:22:02.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:02.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.287573+0000) 2019-10-13T12:22:05.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:05.498+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.501348+0000) 2019-10-13T12:22:05.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:05.954+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.955885+0000) 2019-10-13T12:22:05.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:05.954+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.957593+0000) 2019-10-13T12:22:05.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:05.954+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.958237+0000) 2019-10-13T12:22:05.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:05.954+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.958527+0000) 2019-10-13T12:22:05.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:05.954+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.958614+0000) 2019-10-13T12:22:05.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:05.954+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.958893+0000) 2019-10-13T12:22:05.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:05.958+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.959740+0000) 2019-10-13T12:22:05.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:05.962+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:05.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:05.958+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.964331+0000) 2019-10-13T12:22:06.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:06.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:06.470470+0000) 2019-10-13T12:22:08.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:08.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.735599+0000) 2019-10-13T12:22:08.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:08.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.739627+0000) 2019-10-13T12:22:09.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:09.970+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:09.970331+0000) 2019-10-13T12:22:10.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:10.146+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.149271+0000) 2019-10-13T12:22:10.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:10.962+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.966872+0000) 2019-10-13T12:22:10.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:10.966+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.967523+0000) 2019-10-13T12:22:10.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:10.966+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.968576+0000) 2019-10-13T12:22:10.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:10.966+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.969021+0000) 2019-10-13T12:22:10.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:10.966+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.969095+0000) 2019-10-13T12:22:10.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:10.966+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.969527+0000) 2019-10-13T12:22:10.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:10.966+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.970627+0000) 2019-10-13T12:22:10.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:10.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:10.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:10.974+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:10.976056+0000) 2019-10-13T12:22:12.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:12.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.227684+0000) 2019-10-13T12:22:12.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:12.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.287789+0000) 2019-10-13T12:22:15.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:15.498+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.501585+0000) 2019-10-13T12:22:15.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:15.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.977081+0000) 2019-10-13T12:22:15.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:15.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.977637+0000) 2019-10-13T12:22:15.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:15.974+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.979161+0000) 2019-10-13T12:22:15.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:15.978+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.979592+0000) 2019-10-13T12:22:15.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:15.978+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.980304+0000) 2019-10-13T12:22:15.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:15.978+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.980632+0000) 2019-10-13T12:22:15.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:15.978+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.981297+0000) 2019-10-13T12:22:15.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:15.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:15.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:15.986+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.988195+0000) 2019-10-13T12:22:16.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:16.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:16.470619+0000) 2019-10-13T12:22:18.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:18.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.735805+0000) 2019-10-13T12:22:18.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:18.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.739806+0000) 2019-10-13T12:22:19.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:19.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:19.970525+0000) 2019-10-13T12:22:20.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:20.153+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.156050+0000) 2019-10-13T12:22:20.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:20.986+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.988889+0000) 2019-10-13T12:22:20.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:20.986+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.989659+0000) 2019-10-13T12:22:20.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:20.986+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.989916+0000) 2019-10-13T12:22:20.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:20.989+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.991101+0000) 2019-10-13T12:22:20.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:20.990+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.991451+0000) 2019-10-13T12:22:20.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:20.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.991851+0000) 2019-10-13T12:22:20.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:20.990+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.992234+0000) 2019-10-13T12:22:20.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:20.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:20.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:20.993+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:20.996483+0000) 2019-10-13T12:22:22.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:22.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.227879+0000) 2019-10-13T12:22:22.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:22.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.287990+0000) 2019-10-13T12:22:25.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:25.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.501777+0000) 2019-10-13T12:22:25.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:25.993+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.998696+0000) 2019-10-13T12:22:26.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:25.993+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.999136+0000) 2019-10-13T12:22:26.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:25.997+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.999840+0000) 2019-10-13T12:22:26.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:25.997+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.000042+0000) 2019-10-13T12:22:26.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:25.997+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.001158+0000) 2019-10-13T12:22:26.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:25.997+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.001683+0000) 2019-10-13T12:22:26.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:25.997+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.002125+0000) 2019-10-13T12:22:26.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:26.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:26.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:26.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.008903+0000) 2019-10-13T12:22:26.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:26.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.470810+0000) 2019-10-13T12:22:26.956 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:22:26.956 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:26.960 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:22:26.960 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:28.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:28.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.736020+0000) 2019-10-13T12:22:28.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:28.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.740023+0000) 2019-10-13T12:22:29.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:29.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:29.970718+0000) 2019-10-13T12:22:30.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:30.161+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:30.163273+0000) 2019-10-13T12:22:31.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:31.005+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.008549+0000) 2019-10-13T12:22:31.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:31.005+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.008709+0000) 2019-10-13T12:22:31.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:31.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.009358+0000) 2019-10-13T12:22:31.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:31.005+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.010020+0000) 2019-10-13T12:22:31.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:31.005+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.011063+0000) 2019-10-13T12:22:31.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:31.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.011500+0000) 2019-10-13T12:22:31.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:31.009+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.012176+0000) 2019-10-13T12:22:31.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:31.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:31.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:31.017+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.019202+0000) 2019-10-13T12:22:32.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:32.225+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.228119+0000) 2019-10-13T12:22:32.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:32.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.288194+0000) 2019-10-13T12:22:35.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:35.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:35.501978+0000) 2019-10-13T12:22:36.020 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:36.013+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.018900+0000) 2019-10-13T12:22:36.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:36.013+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.018978+0000) 2019-10-13T12:22:36.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:36.013+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.019239+0000) 2019-10-13T12:22:36.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:36.017+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.020205+0000) 2019-10-13T12:22:36.021 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:36.017+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.021007+0000) 2019-10-13T12:22:36.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:36.017+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.021347+0000) 2019-10-13T12:22:36.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:36.017+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.021631+0000) 2019-10-13T12:22:36.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:36.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:36.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:36.025+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.028462+0000) 2019-10-13T12:22:36.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:36.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.470990+0000) 2019-10-13T12:22:38.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:38.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.736212+0000) 2019-10-13T12:22:38.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:38.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.740217+0000) 2019-10-13T12:22:39.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:39.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:39.970953+0000) 2019-10-13T12:22:40.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:40.169+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:40.170343+0000) 2019-10-13T12:22:41.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:41.025+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.029242+0000) 2019-10-13T12:22:41.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:41.025+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.029502+0000) 2019-10-13T12:22:41.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:41.025+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.029920+0000) 2019-10-13T12:22:41.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:41.025+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.030518+0000) 2019-10-13T12:22:41.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:41.025+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.031238+0000) 2019-10-13T12:22:41.032 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:41.029+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.031412+0000) 2019-10-13T12:22:41.033 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:41.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.032179+0000) 2019-10-13T12:22:41.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:41.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:41.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:41.037+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.038261+0000) 2019-10-13T12:22:42.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:42.225+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.228352+0000) 2019-10-13T12:22:42.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:42.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.288387+0000) 2019-10-13T12:22:45.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:45.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:45.502209+0000) 2019-10-13T12:22:46.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:46.037+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.039381+0000) 2019-10-13T12:22:46.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:46.037+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.040516+0000) 2019-10-13T12:22:46.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:46.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.041022+0000) 2019-10-13T12:22:46.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:46.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.041208+0000) 2019-10-13T12:22:46.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:46.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.041976+0000) 2019-10-13T12:22:46.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:46.037+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.042501+0000) 2019-10-13T12:22:46.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:46.037+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.042777+0000) 2019-10-13T12:22:46.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:46.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:46.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:46.045+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.049030+0000) 2019-10-13T12:22:46.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:46.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.471123+0000) 2019-10-13T12:22:48.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:48.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.736459+0000) 2019-10-13T12:22:48.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:48.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.740467+0000) 2019-10-13T12:22:49.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:49.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:49.971180+0000) 2019-10-13T12:22:50.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:50.177+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:50.181831+0000) 2019-10-13T12:22:51.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:51.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.050386+0000) 2019-10-13T12:22:51.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:51.049+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.050774+0000) 2019-10-13T12:22:51.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:51.049+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.051560+0000) 2019-10-13T12:22:51.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:51.049+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.051854+0000) 2019-10-13T12:22:51.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:51.049+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.052717+0000) 2019-10-13T12:22:51.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:51.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.052891+0000) 2019-10-13T12:22:51.055 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:51.049+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.054107+0000) 2019-10-13T12:22:51.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:51.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146403 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:51.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:51.057+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.058988+0000) 2019-10-13T12:22:52.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:52.225+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.228568+0000) 2019-10-13T12:22:52.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:52.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.288585+0000) 2019-10-13T12:22:55.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:55.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:55.502423+0000) 2019-10-13T12:22:56.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:22:56.057+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.060929+0000) 2019-10-13T12:22:56.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:22:56.057+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.061689+0000) 2019-10-13T12:22:56.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:56.061+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.062190+0000) 2019-10-13T12:22:56.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:56.057+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.062869+0000) 2019-10-13T12:22:56.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:56.057+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.063178+0000) 2019-10-13T12:22:56.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:56.061+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.064137+0000) 2019-10-13T12:22:56.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:22:56.061+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.064568+0000) 2019-10-13T12:22:56.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:22:56.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:22:56.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:22:56.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.070223+0000) 2019-10-13T12:22:56.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:22:56.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.471334+0000) 2019-10-13T12:22:57.067 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:22:57.068 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:57.072 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:22:57.072 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:58.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:22:58.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.736681+0000) 2019-10-13T12:22:58.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:22:58.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.740659+0000) 2019-10-13T12:22:59.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:22:59.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:59.971400+0000) 2019-10-13T12:23:00.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:00.185+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:00.188234+0000) 2019-10-13T12:23:01.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:01.069+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.071529+0000) 2019-10-13T12:23:01.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:01.069+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.071963+0000) 2019-10-13T12:23:01.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:01.069+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.072512+0000) 2019-10-13T12:23:01.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:01.069+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.073539+0000) 2019-10-13T12:23:01.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:01.069+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.073749+0000) 2019-10-13T12:23:01.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:01.069+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.074507+0000) 2019-10-13T12:23:01.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:01.069+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.074845+0000) 2019-10-13T12:23:01.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:01.077+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.079379+0000) 2019-10-13T12:23:01.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:01.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:02.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:02.225+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.228804+0000) 2019-10-13T12:23:02.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:02.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.288787+0000) 2019-10-13T12:23:05.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:05.497+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:05.502618+0000) 2019-10-13T12:23:06.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:06.076+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.082311+0000) 2019-10-13T12:23:06.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:06.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.082948+0000) 2019-10-13T12:23:06.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:06.080+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.083918+0000) 2019-10-13T12:23:06.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:06.081+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.084275+0000) 2019-10-13T12:23:06.085 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:06.080+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.084572+0000) 2019-10-13T12:23:06.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:06.080+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.085255+0000) 2019-10-13T12:23:06.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:06.080+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.086367+0000) 2019-10-13T12:23:06.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:06.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:06.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:06.089+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.091495+0000) 2019-10-13T12:23:06.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:06.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.471512+0000) 2019-10-13T12:23:08.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:08.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.736916+0000) 2019-10-13T12:23:08.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:08.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.740853+0000) 2019-10-13T12:23:09.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:09.969+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:09.971599+0000) 2019-10-13T12:23:10.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:10.193+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:10.194629+0000) 2019-10-13T12:23:11.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:11.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.094028+0000) 2019-10-13T12:23:11.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:11.088+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.094582+0000) 2019-10-13T12:23:11.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:11.092+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.095398+0000) 2019-10-13T12:23:11.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:11.093+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.095529+0000) 2019-10-13T12:23:11.098 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:11.092+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.096409+0000) 2019-10-13T12:23:11.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:11.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.096706+0000) 2019-10-13T12:23:11.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:11.092+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.097363+0000) 2019-10-13T12:23:11.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:11.096+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:11.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:11.097+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.101415+0000) 2019-10-13T12:23:12.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:12.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.229002+0000) 2019-10-13T12:23:12.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:12.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.288984+0000) 2019-10-13T12:23:15.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:15.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:15.502815+0000) 2019-10-13T12:23:16.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:16.100+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.104644+0000) 2019-10-13T12:23:16.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:16.100+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.104977+0000) 2019-10-13T12:23:16.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:16.100+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.105943+0000) 2019-10-13T12:23:16.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:16.100+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.106713+0000) 2019-10-13T12:23:16.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:16.100+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.107134+0000) 2019-10-13T12:23:16.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:16.100+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.107235+0000) 2019-10-13T12:23:16.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:16.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.107646+0000) 2019-10-13T12:23:16.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:16.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:16.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:16.108+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.112325+0000) 2019-10-13T12:23:16.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:16.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.471693+0000) 2019-10-13T12:23:18.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:18.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.737148+0000) 2019-10-13T12:23:18.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:18.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.741045+0000) 2019-10-13T12:23:19.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:19.968+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:19.971811+0000) 2019-10-13T12:23:20.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:20.204+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:20.206333+0000) 2019-10-13T12:23:21.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:21.112+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.115806+0000) 2019-10-13T12:23:21.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:21.112+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.116496+0000) 2019-10-13T12:23:21.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:21.112+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.116234+0000) 2019-10-13T12:23:21.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:21.112+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.116970+0000) 2019-10-13T12:23:21.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:21.112+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.117770+0000) 2019-10-13T12:23:21.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:21.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.119611+0000) 2019-10-13T12:23:21.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:21.116+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.120247+0000) 2019-10-13T12:23:21.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:21.120+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.123442+0000) 2019-10-13T12:23:21.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:21.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:22.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:22.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.229199+0000) 2019-10-13T12:23:22.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:22.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.289197+0000) 2019-10-13T12:23:25.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:25.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:25.503013+0000) 2019-10-13T12:23:26.127 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:26.120+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.126572+0000) 2019-10-13T12:23:26.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:26.124+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.127572+0000) 2019-10-13T12:23:26.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:26.124+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.127905+0000) 2019-10-13T12:23:26.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:26.124+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.128258+0000) 2019-10-13T12:23:26.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:26.124+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.128886+0000) 2019-10-13T12:23:26.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:26.124+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.129307+0000) 2019-10-13T12:23:26.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:26.128+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.130464+0000) 2019-10-13T12:23:26.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:26.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:26.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:26.132+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.136106+0000) 2019-10-13T12:23:26.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:26.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.471871+0000) 2019-10-13T12:23:27.195 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:23:27.195 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:27.199 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:23:27.199 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:28.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:28.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.737343+0000) 2019-10-13T12:23:28.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:28.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.741185+0000) 2019-10-13T12:23:29.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:29.968+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:29.972002+0000) 2019-10-13T12:23:30.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:30.208+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:30.212805+0000) 2019-10-13T12:23:31.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:31.132+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.137845+0000) 2019-10-13T12:23:31.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:31.132+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.138174+0000) 2019-10-13T12:23:31.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:31.132+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.138918+0000) 2019-10-13T12:23:31.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:31.136+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.140117+0000) 2019-10-13T12:23:31.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:31.136+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.139997+0000) 2019-10-13T12:23:31.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:31.136+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.140620+0000) 2019-10-13T12:23:31.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:31.136+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.141359+0000) 2019-10-13T12:23:31.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:31.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:31.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:31.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.146789+0000) 2019-10-13T12:23:32.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:32.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.229416+0000) 2019-10-13T12:23:32.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:32.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.289434+0000) 2019-10-13T12:23:35.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:35.496+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:35.503241+0000) 2019-10-13T12:23:36.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:36.144+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.148669+0000) 2019-10-13T12:23:36.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:36.144+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.149312+0000) 2019-10-13T12:23:36.151 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:36.144+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.149775+0000) 2019-10-13T12:23:36.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:36.144+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.150570+0000) 2019-10-13T12:23:36.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:36.144+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.151188+0000) 2019-10-13T12:23:36.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:36.144+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.151156+0000) 2019-10-13T12:23:36.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:36.148+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.151499+0000) 2019-10-13T12:23:36.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:36.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:36.159 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:36.152+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.157812+0000) 2019-10-13T12:23:36.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:36.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.472038+0000) 2019-10-13T12:23:38.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:38.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.737545+0000) 2019-10-13T12:23:38.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:38.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.741385+0000) 2019-10-13T12:23:39.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:39.968+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:39.972227+0000) 2019-10-13T12:23:40.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:40.216+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:40.218869+0000) 2019-10-13T12:23:41.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:41.156+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.159384+0000) 2019-10-13T12:23:41.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:41.156+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.159864+0000) 2019-10-13T12:23:41.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:41.156+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.161081+0000) 2019-10-13T12:23:41.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:41.156+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.162202+0000) 2019-10-13T12:23:41.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:41.156+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.162556+0000) 2019-10-13T12:23:41.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:41.160+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.163415+0000) 2019-10-13T12:23:41.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:41.160+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.163710+0000) 2019-10-13T12:23:41.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:41.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:41.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:41.164+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.167788+0000) 2019-10-13T12:23:42.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:42.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.229652+0000) 2019-10-13T12:23:42.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:42.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.289630+0000) 2019-10-13T12:23:45.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:45.500+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:45.503492+0000) 2019-10-13T12:23:46.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:46.164+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.170210+0000) 2019-10-13T12:23:46.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:46.164+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.170970+0000) 2019-10-13T12:23:46.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:46.168+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.171188+0000) 2019-10-13T12:23:46.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:46.168+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.173386+0000) 2019-10-13T12:23:46.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:46.168+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.174104+0000) 2019-10-13T12:23:46.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:46.168+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.173397+0000) 2019-10-13T12:23:46.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:46.168+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.173850+0000) 2019-10-13T12:23:46.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:46.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146776 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:46.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:46.172+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.177021+0000) 2019-10-13T12:23:46.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:46.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.472217+0000) 2019-10-13T12:23:48.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:48.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.737762+0000) 2019-10-13T12:23:48.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:48.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.741607+0000) 2019-10-13T12:23:49.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:49.972+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:49.972466+0000) 2019-10-13T12:23:50.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:50.228+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:50.230597+0000) 2019-10-13T12:23:51.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:51.175+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.181000+0000) 2019-10-13T12:23:51.182 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:51.175+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.181171+0000) 2019-10-13T12:23:51.183 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:51.175+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.182145+0000) 2019-10-13T12:23:51.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:51.175+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.182613+0000) 2019-10-13T12:23:51.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:51.175+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.183017+0000) 2019-10-13T12:23:51.184 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:51.179+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.183424+0000) 2019-10-13T12:23:51.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:51.180+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.183645+0000) 2019-10-13T12:23:51.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:51.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:51.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:51.184+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.187867+0000) 2019-10-13T12:23:52.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:52.227+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.229850+0000) 2019-10-13T12:23:52.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:52.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.289830+0000) 2019-10-13T12:23:55.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:55.499+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:55.503699+0000) 2019-10-13T12:23:56.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:56.183+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.190796+0000) 2019-10-13T12:23:56.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:56.183+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.191218+0000) 2019-10-13T12:23:56.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:23:56.187+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.191917+0000) 2019-10-13T12:23:56.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:23:56.187+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.192324+0000) 2019-10-13T12:23:56.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:23:56.187+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.193212+0000) 2019-10-13T12:23:56.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:56.187+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.193551+0000) 2019-10-13T12:23:56.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:56.192+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.194130+0000) 2019-10-13T12:23:56.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:23:56.191+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:23:56.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:23:56.196+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.200018+0000) 2019-10-13T12:23:56.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:23:56.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.472390+0000) 2019-10-13T12:23:57.274 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:23:57.274 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:57.277 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:23:57.278 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:58.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:23:58.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.737991+0000) 2019-10-13T12:23:58.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:23:58.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.741801+0000) 2019-10-13T12:23:59.973 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:23:59.968+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:59.972663+0000) 2019-10-13T12:24:00.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:00.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:00.236953+0000) 2019-10-13T12:24:01.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:01.195+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.201165+0000) 2019-10-13T12:24:01.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:01.195+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.201486+0000) 2019-10-13T12:24:01.203 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:01.200+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.202629+0000) 2019-10-13T12:24:01.204 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:01.195+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.202559+0000) 2019-10-13T12:24:01.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:01.195+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.203058+0000) 2019-10-13T12:24:01.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:01.199+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.203452+0000) 2019-10-13T12:24:01.205 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:01.199+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.204211+0000) 2019-10-13T12:24:01.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:01.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:01.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:01.208+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.210669+0000) 2019-10-13T12:24:02.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:02.227+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.230053+0000) 2019-10-13T12:24:02.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:02.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.290026+0000) 2019-10-13T12:24:05.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:05.503+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:05.503910+0000) 2019-10-13T12:24:06.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:06.207+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.211463+0000) 2019-10-13T12:24:06.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:06.208+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.212059+0000) 2019-10-13T12:24:06.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:06.207+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.212313+0000) 2019-10-13T12:24:06.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:06.207+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.212597+0000) 2019-10-13T12:24:06.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:06.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.213570+0000) 2019-10-13T12:24:06.215 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:06.207+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.214241+0000) 2019-10-13T12:24:06.216 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:06.211+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.215359+0000) 2019-10-13T12:24:06.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:06.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:06.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:06.220+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.223433+0000) 2019-10-13T12:24:06.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:06.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.472565+0000) 2019-10-13T12:24:08.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:08.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.738193+0000) 2019-10-13T12:24:08.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:08.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.741988+0000) 2019-10-13T12:24:09.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:09.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:09.972907+0000) 2019-10-13T12:24:10.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:10.239+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:10.243419+0000) 2019-10-13T12:24:11.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:11.219+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.223351+0000) 2019-10-13T12:24:11.225 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:11.219+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.223798+0000) 2019-10-13T12:24:11.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:11.219+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.224551+0000) 2019-10-13T12:24:11.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:11.219+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.224897+0000) 2019-10-13T12:24:11.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:11.219+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.225487+0000) 2019-10-13T12:24:11.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:11.219+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.225677+0000) 2019-10-13T12:24:11.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:11.219+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.226987+0000) 2019-10-13T12:24:11.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:11.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:11.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:11.231+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.234658+0000) 2019-10-13T12:24:12.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:12.223+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.230274+0000) 2019-10-13T12:24:12.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:12.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.290228+0000) 2019-10-13T12:24:15.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:15.503+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:15.504138+0000) 2019-10-13T12:24:16.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:16.227+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.233460+0000) 2019-10-13T12:24:16.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:16.227+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.233827+0000) 2019-10-13T12:24:16.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:16.231+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.234332+0000) 2019-10-13T12:24:16.236 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:16.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.234622+0000) 2019-10-13T12:24:16.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:16.227+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.234943+0000) 2019-10-13T12:24:16.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:16.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.235289+0000) 2019-10-13T12:24:16.237 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:16.231+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.236071+0000) 2019-10-13T12:24:16.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:16.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 146979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:16.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:16.239+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.242664+0000) 2019-10-13T12:24:16.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:16.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.472745+0000) 2019-10-13T12:24:18.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:18.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.738416+0000) 2019-10-13T12:24:18.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:18.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.742191+0000) 2019-10-13T12:24:19.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:19.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:19.973144+0000) 2019-10-13T12:24:20.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:20.247+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:20.249988+0000) 2019-10-13T12:24:21.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:21.243+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.244701+0000) 2019-10-13T12:24:21.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:21.243+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.245437+0000) 2019-10-13T12:24:21.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:21.243+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.245877+0000) 2019-10-13T12:24:21.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:21.243+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.246087+0000) 2019-10-13T12:24:21.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:21.243+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.246537+0000) 2019-10-13T12:24:21.248 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:21.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.247037+0000) 2019-10-13T12:24:21.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:21.243+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.246943+0000) 2019-10-13T12:24:21.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:21.247+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.251385+0000) 2019-10-13T12:24:21.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:21.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:22.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:22.227+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.230478+0000) 2019-10-13T12:24:22.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:22.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.290427+0000) 2019-10-13T12:24:25.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:25.503+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:25.504395+0000) 2019-10-13T12:24:26.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:26.255+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.255296+0000) 2019-10-13T12:24:26.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:26.255+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.256132+0000) 2019-10-13T12:24:26.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:26.255+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.256222+0000) 2019-10-13T12:24:26.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:26.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.256613+0000) 2019-10-13T12:24:26.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:26.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.257366+0000) 2019-10-13T12:24:26.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:26.255+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.258014+0000) 2019-10-13T12:24:26.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:26.255+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.259106+0000) 2019-10-13T12:24:26.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:26.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:26.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:26.259+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.265456+0000) 2019-10-13T12:24:26.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:26.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.472945+0000) 2019-10-13T12:24:27.406 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:24:27.406 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:27.410 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:24:27.410 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:28.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:28.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.738630+0000) 2019-10-13T12:24:28.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:28.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.742389+0000) 2019-10-13T12:24:29.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:29.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:29.973339+0000) 2019-10-13T12:24:30.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:30.255+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:30.256518+0000) 2019-10-13T12:24:31.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:31.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.266013+0000) 2019-10-13T12:24:31.267 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:31.263+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.266687+0000) 2019-10-13T12:24:31.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:31.263+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.266442+0000) 2019-10-13T12:24:31.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:31.267+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.267445+0000) 2019-10-13T12:24:31.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:31.267+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.268186+0000) 2019-10-13T12:24:31.269 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:31.267+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.268635+0000) 2019-10-13T12:24:31.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:31.267+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.269629+0000) 2019-10-13T12:24:31.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:31.267+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.273187+0000) 2019-10-13T12:24:31.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:31.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:32.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:32.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.230684+0000) 2019-10-13T12:24:32.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:32.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.290654+0000) 2019-10-13T12:24:35.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:35.502+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:35.504622+0000) 2019-10-13T12:24:36.279 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:36.274+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.278629+0000) 2019-10-13T12:24:36.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:36.274+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.279155+0000) 2019-10-13T12:24:36.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:36.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.279605+0000) 2019-10-13T12:24:36.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:36.278+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.279915+0000) 2019-10-13T12:24:36.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:36.278+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.281668+0000) 2019-10-13T12:24:36.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:36.275+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.282048+0000) 2019-10-13T12:24:36.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:36.278+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.283048+0000) 2019-10-13T12:24:36.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:36.282+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:36.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:36.283+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.286998+0000) 2019-10-13T12:24:36.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:36.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.473121+0000) 2019-10-13T12:24:38.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:38.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.738857+0000) 2019-10-13T12:24:38.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:38.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.742581+0000) 2019-10-13T12:24:39.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:39.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:39.973531+0000) 2019-10-13T12:24:40.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:40.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:40.262956+0000) 2019-10-13T12:24:41.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:41.286+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.289593+0000) 2019-10-13T12:24:41.291 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:41.286+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.289920+0000) 2019-10-13T12:24:41.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:41.286+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.290897+0000) 2019-10-13T12:24:41.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:41.286+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.291147+0000) 2019-10-13T12:24:41.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:41.290+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.292030+0000) 2019-10-13T12:24:41.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:41.287+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.292313+0000) 2019-10-13T12:24:41.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:41.290+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.292820+0000) 2019-10-13T12:24:41.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:41.291+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.297251+0000) 2019-10-13T12:24:41.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:41.294+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:42.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:42.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.230873+0000) 2019-10-13T12:24:42.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:42.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.290879+0000) 2019-10-13T12:24:45.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:45.502+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:45.504818+0000) 2019-10-13T12:24:46.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:46.298+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.301407+0000) 2019-10-13T12:24:46.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:46.298+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.301578+0000) 2019-10-13T12:24:46.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:46.298+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.301942+0000) 2019-10-13T12:24:46.303 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:46.298+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.302897+0000) 2019-10-13T12:24:46.304 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:46.302+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.303560+0000) 2019-10-13T12:24:46.304 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:46.302+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.303852+0000) 2019-10-13T12:24:46.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:46.299+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.304077+0000) 2019-10-13T12:24:46.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:46.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:46.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:46.303+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.309533+0000) 2019-10-13T12:24:46.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:46.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.473295+0000) 2019-10-13T12:24:48.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:48.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.739057+0000) 2019-10-13T12:24:48.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:48.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.742800+0000) 2019-10-13T12:24:49.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:49.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:49.973719+0000) 2019-10-13T12:24:50.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:50.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:50.269283+0000) 2019-10-13T12:24:51.313 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:51.310+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.311527+0000) 2019-10-13T12:24:51.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:51.310+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.312987+0000) 2019-10-13T12:24:51.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:51.310+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.313356+0000) 2019-10-13T12:24:51.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:51.310+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.313696+0000) 2019-10-13T12:24:51.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:51.310+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.314024+0000) 2019-10-13T12:24:51.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:51.310+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.314609+0000) 2019-10-13T12:24:51.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:51.311+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.315345+0000) 2019-10-13T12:24:51.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:51.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:51.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:51.315+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.320447+0000) 2019-10-13T12:24:52.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:52.226+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.231071+0000) 2019-10-13T12:24:52.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:52.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.291062+0000) 2019-10-13T12:24:55.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:55.502+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:55.505011+0000) 2019-10-13T12:24:56.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:24:56.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.323459+0000) 2019-10-13T12:24:56.325 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:56.322+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.323916+0000) 2019-10-13T12:24:56.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:24:56.322+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.324635+0000) 2019-10-13T12:24:56.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:56.322+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.325004+0000) 2019-10-13T12:24:56.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:56.322+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.326034+0000) 2019-10-13T12:24:56.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:56.323+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.326254+0000) 2019-10-13T12:24:56.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:24:56.322+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.326796+0000) 2019-10-13T12:24:56.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:24:56.327+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.330819+0000) 2019-10-13T12:24:56.332 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:24:56.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147251 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:24:56.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:24:56.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.473486+0000) 2019-10-13T12:24:57.529 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:24:57.529 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:57.533 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:24:57.533 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:58.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:24:58.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.739283+0000) 2019-10-13T12:24:58.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:24:58.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.742992+0000) 2019-10-13T12:24:59.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:24:59.971+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:59.973925+0000) 2019-10-13T12:25:00.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:00.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:00.280463+0000) 2019-10-13T12:25:01.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:01.330+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.335124+0000) 2019-10-13T12:25:01.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:01.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.335646+0000) 2019-10-13T12:25:01.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:01.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.336217+0000) 2019-10-13T12:25:01.337 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:01.334+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.336874+0000) 2019-10-13T12:25:01.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:01.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.337400+0000) 2019-10-13T12:25:01.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:01.334+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.337781+0000) 2019-10-13T12:25:01.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:01.334+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.338106+0000) 2019-10-13T12:25:01.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:01.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:01.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:01.343+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.347874+0000) 2019-10-13T12:25:02.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:02.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.231286+0000) 2019-10-13T12:25:02.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:02.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.291277+0000) 2019-10-13T12:25:05.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:05.502+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:05.505209+0000) 2019-10-13T12:25:06.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:06.342+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.345762+0000) 2019-10-13T12:25:06.347 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:06.342+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.346928+0000) 2019-10-13T12:25:06.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:06.342+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.347163+0000) 2019-10-13T12:25:06.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:06.346+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.347847+0000) 2019-10-13T12:25:06.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:06.346+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.348458+0000) 2019-10-13T12:25:06.350 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:06.342+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.348690+0000) 2019-10-13T12:25:06.350 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:06.346+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.348927+0000) 2019-10-13T12:25:06.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:06.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:06.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:06.346+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.353776+0000) 2019-10-13T12:25:06.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:06.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.473667+0000) 2019-10-13T12:25:08.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:08.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.739523+0000) 2019-10-13T12:25:08.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:08.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.743199+0000) 2019-10-13T12:25:09.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:09.970+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:09.974174+0000) 2019-10-13T12:25:10.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:10.286+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:10.286932+0000) 2019-10-13T12:25:11.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:11.354+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.357404+0000) 2019-10-13T12:25:11.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:11.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.357483+0000) 2019-10-13T12:25:11.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:11.354+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.357943+0000) 2019-10-13T12:25:11.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:11.354+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.358274+0000) 2019-10-13T12:25:11.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:11.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.358878+0000) 2019-10-13T12:25:11.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:11.354+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.359336+0000) 2019-10-13T12:25:11.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:11.358+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.359584+0000) 2019-10-13T12:25:11.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:11.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:11.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:11.358+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.365670+0000) 2019-10-13T12:25:12.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:12.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.231500+0000) 2019-10-13T12:25:12.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:12.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.291528+0000) 2019-10-13T12:25:15.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:15.501+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:15.505452+0000) 2019-10-13T12:25:16.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:16.361+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.366957+0000) 2019-10-13T12:25:16.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:16.365+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.367731+0000) 2019-10-13T12:25:16.369 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:16.365+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.367988+0000) 2019-10-13T12:25:16.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:16.362+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.368279+0000) 2019-10-13T12:25:16.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:16.365+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.368933+0000) 2019-10-13T12:25:16.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:16.365+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.369233+0000) 2019-10-13T12:25:16.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:16.365+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.369676+0000) 2019-10-13T12:25:16.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:16.366+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.373630+0000) 2019-10-13T12:25:16.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:16.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:16.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:16.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.473848+0000) 2019-10-13T12:25:18.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:18.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.739733+0000) 2019-10-13T12:25:18.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:18.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.743395+0000) 2019-10-13T12:25:19.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:19.970+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:19.974370+0000) 2019-10-13T12:25:20.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:20.290+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:20.293375+0000) 2019-10-13T12:25:21.378 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:21.373+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.377434+0000) 2019-10-13T12:25:21.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:21.373+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.378432+0000) 2019-10-13T12:25:21.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:21.373+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.378818+0000) 2019-10-13T12:25:21.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:21.373+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.378947+0000) 2019-10-13T12:25:21.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:21.374+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.379505+0000) 2019-10-13T12:25:21.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:21.377+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.380176+0000) 2019-10-13T12:25:21.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:21.377+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.380584+0000) 2019-10-13T12:25:21.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:21.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:21.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:21.378+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.385990+0000) 2019-10-13T12:25:22.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:22.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.231676+0000) 2019-10-13T12:25:22.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:22.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.291674+0000) 2019-10-13T12:25:25.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:25.501+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:25.505650+0000) 2019-10-13T12:25:26.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:26.385+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.388477+0000) 2019-10-13T12:25:26.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:26.385+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.389063+0000) 2019-10-13T12:25:26.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:26.385+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.389082+0000) 2019-10-13T12:25:26.391 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:26.382+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.389357+0000) 2019-10-13T12:25:26.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:26.385+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.390086+0000) 2019-10-13T12:25:26.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:26.385+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.390945+0000) 2019-10-13T12:25:26.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:26.385+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.390749+0000) 2019-10-13T12:25:26.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:26.389+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:26.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:26.394+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.399399+0000) 2019-10-13T12:25:26.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:26.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.474033+0000) 2019-10-13T12:25:27.656 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:25:27.656 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:27.660 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:25:27.660 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:28.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:28.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.739931+0000) 2019-10-13T12:25:28.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:28.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.743586+0000) 2019-10-13T12:25:29.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:29.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:29.974590+0000) 2019-10-13T12:25:30.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:30.298+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:30.299858+0000) 2019-10-13T12:25:31.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:31.393+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.398658+0000) 2019-10-13T12:25:31.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:31.397+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.399718+0000) 2019-10-13T12:25:31.401 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:31.397+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.400128+0000) 2019-10-13T12:25:31.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:31.397+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.400778+0000) 2019-10-13T12:25:31.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:31.394+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.401069+0000) 2019-10-13T12:25:31.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:31.397+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.401484+0000) 2019-10-13T12:25:31.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:31.397+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.400469+0000) 2019-10-13T12:25:31.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:31.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:31.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:31.402+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.408474+0000) 2019-10-13T12:25:32.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:32.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.231856+0000) 2019-10-13T12:25:32.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:32.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.291859+0000) 2019-10-13T12:25:35.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:35.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:35.505849+0000) 2019-10-13T12:25:36.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:36.405+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.410316+0000) 2019-10-13T12:25:36.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:36.405+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.410710+0000) 2019-10-13T12:25:36.425 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:36.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.412191+0000) 2019-10-13T12:25:36.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:36.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.412638+0000) 2019-10-13T12:25:36.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:36.409+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.413079+0000) 2019-10-13T12:25:36.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:36.409+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.414090+0000) 2019-10-13T12:25:36.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:36.410+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.414674+0000) 2019-10-13T12:25:36.432 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:36.410+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.417260+0000) 2019-10-13T12:25:36.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:36.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:36.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:36.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.474216+0000) 2019-10-13T12:25:38.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:38.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.740163+0000) 2019-10-13T12:25:38.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:38.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.743778+0000) 2019-10-13T12:25:39.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:39.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:39.974787+0000) 2019-10-13T12:25:40.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:40.302+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:40.306259+0000) 2019-10-13T12:25:41.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:41.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.420693+0000) 2019-10-13T12:25:41.422 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:41.417+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.421222+0000) 2019-10-13T12:25:41.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:41.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.421929+0000) 2019-10-13T12:25:41.423 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:41.417+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.422671+0000) 2019-10-13T12:25:41.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:41.418+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.422767+0000) 2019-10-13T12:25:41.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:41.421+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.423581+0000) 2019-10-13T12:25:41.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:41.421+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.423851+0000) 2019-10-13T12:25:41.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:41.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:41.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:41.422+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.429627+0000) 2019-10-13T12:25:42.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:42.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.232056+0000) 2019-10-13T12:25:42.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:42.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.292068+0000) 2019-10-13T12:25:45.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:45.501+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:45.506046+0000) 2019-10-13T12:25:46.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:46.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.432429+0000) 2019-10-13T12:25:46.434 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:46.429+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.433174+0000) 2019-10-13T12:25:46.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:46.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.433463+0000) 2019-10-13T12:25:46.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:46.429+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.433795+0000) 2019-10-13T12:25:46.436 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:46.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.435165+0000) 2019-10-13T12:25:46.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:46.430+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.435618+0000) 2019-10-13T12:25:46.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:46.433+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.436042+0000) 2019-10-13T12:25:46.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:46.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:46.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:46.434+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.441662+0000) 2019-10-13T12:25:46.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:46.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.474415+0000) 2019-10-13T12:25:48.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:48.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.740365+0000) 2019-10-13T12:25:48.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:48.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.743968+0000) 2019-10-13T12:25:49.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:49.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:49.974977+0000) 2019-10-13T12:25:50.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:50.310+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:50.312137+0000) 2019-10-13T12:25:51.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:51.441+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.444085+0000) 2019-10-13T12:25:51.446 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:51.441+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.444364+0000) 2019-10-13T12:25:51.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:51.442+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.444981+0000) 2019-10-13T12:25:51.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:51.441+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.444913+0000) 2019-10-13T12:25:51.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:51.441+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.445195+0000) 2019-10-13T12:25:51.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:51.441+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.446643+0000) 2019-10-13T12:25:51.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:51.441+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.446963+0000) 2019-10-13T12:25:51.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:51.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:51.454 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:51.450+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.452901+0000) 2019-10-13T12:25:52.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:52.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.232261+0000) 2019-10-13T12:25:52.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:52.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.292256+0000) 2019-10-13T12:25:55.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:55.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:55.506234+0000) 2019-10-13T12:25:56.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:56.453+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.455538+0000) 2019-10-13T12:25:56.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:56.454+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.455945+0000) 2019-10-13T12:25:56.458 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:25:56.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.456628+0000) 2019-10-13T12:25:56.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:56.453+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.457051+0000) 2019-10-13T12:25:56.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:56.453+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.457427+0000) 2019-10-13T12:25:56.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:25:56.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.457813+0000) 2019-10-13T12:25:56.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:25:56.453+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.458394+0000) 2019-10-13T12:25:56.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:25:56.458+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.461530+0000) 2019-10-13T12:25:56.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:25:56.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:25:56.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:25:56.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.474590+0000) 2019-10-13T12:25:57.935 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:25:57.935 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:57.939 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:25:57.940 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:58.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:25:58.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.740593+0000) 2019-10-13T12:25:58.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:25:58.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.744195+0000) 2019-10-13T12:25:59.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:25:59.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:59.975171+0000) 2019-10-13T12:26:00.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:00.317+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:00.318696+0000) 2019-10-13T12:26:01.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:01.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.466529+0000) 2019-10-13T12:26:01.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:01.460+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.467138+0000) 2019-10-13T12:26:01.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:01.464+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.468114+0000) 2019-10-13T12:26:01.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:01.464+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.468723+0000) 2019-10-13T12:26:01.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:01.464+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.468720+0000) 2019-10-13T12:26:01.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:01.464+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.469371+0000) 2019-10-13T12:26:01.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:01.464+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.469882+0000) 2019-10-13T12:26:01.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:01.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:01.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:01.473+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.475521+0000) 2019-10-13T12:26:02.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:02.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.232443+0000) 2019-10-13T12:26:02.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:02.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.292438+0000) 2019-10-13T12:26:05.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:05.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:05.506433+0000) 2019-10-13T12:26:06.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:06.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.474777+0000) 2019-10-13T12:26:06.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:06.472+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.478507+0000) 2019-10-13T12:26:06.480 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:06.476+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.479292+0000) 2019-10-13T12:26:06.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:06.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.479526+0000) 2019-10-13T12:26:06.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:06.476+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.479974+0000) 2019-10-13T12:26:06.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:06.476+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.480645+0000) 2019-10-13T12:26:06.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:06.476+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.481433+0000) 2019-10-13T12:26:06.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:06.476+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.481721+0000) 2019-10-13T12:26:06.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:06.481+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.485206+0000) 2019-10-13T12:26:06.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:06.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:08.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:08.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.740825+0000) 2019-10-13T12:26:08.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:08.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.744399+0000) 2019-10-13T12:26:09.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:09.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:09.975422+0000) 2019-10-13T12:26:10.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:10.321+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:10.325150+0000) 2019-10-13T12:26:11.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:11.484+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.488221+0000) 2019-10-13T12:26:11.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:11.484+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.489587+0000) 2019-10-13T12:26:11.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:11.489+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.490181+0000) 2019-10-13T12:26:11.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:11.484+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.490718+0000) 2019-10-13T12:26:11.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:11.484+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.491091+0000) 2019-10-13T12:26:11.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:11.488+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.492142+0000) 2019-10-13T12:26:11.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:11.488+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.492467+0000) 2019-10-13T12:26:11.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:11.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:11.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:11.497+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.500226+0000) 2019-10-13T12:26:12.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:12.228+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.232612+0000) 2019-10-13T12:26:12.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:12.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.292620+0000) 2019-10-13T12:26:15.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:15.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:15.506636+0000) 2019-10-13T12:26:16.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:16.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.474997+0000) 2019-10-13T12:26:16.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:16.492+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.499141+0000) 2019-10-13T12:26:16.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:16.496+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.500187+0000) 2019-10-13T12:26:16.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:16.497+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.500524+0000) 2019-10-13T12:26:16.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:16.496+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.501613+0000) 2019-10-13T12:26:16.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:16.496+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.501473+0000) 2019-10-13T12:26:16.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:16.496+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.500944+0000) 2019-10-13T12:26:16.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:16.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.502351+0000) 2019-10-13T12:26:16.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:16.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:16.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:16.505+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.507395+0000) 2019-10-13T12:26:18.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:18.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.741054+0000) 2019-10-13T12:26:18.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:18.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.744628+0000) 2019-10-13T12:26:19.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:19.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:19.975626+0000) 2019-10-13T12:26:20.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:20.329+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:20.331728+0000) 2019-10-13T12:26:21.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:21.504+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.509987+0000) 2019-10-13T12:26:21.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:21.504+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.511119+0000) 2019-10-13T12:26:21.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:21.508+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.511492+0000) 2019-10-13T12:26:21.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:21.508+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.511881+0000) 2019-10-13T12:26:21.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:21.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.512890+0000) 2019-10-13T12:26:21.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:21.508+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.513218+0000) 2019-10-13T12:26:21.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:21.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.513623+0000) 2019-10-13T12:26:21.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:21.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:21.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:21.517+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.518298+0000) 2019-10-13T12:26:22.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:22.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.232787+0000) 2019-10-13T12:26:22.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:22.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.292806+0000) 2019-10-13T12:26:25.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:25.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:25.506840+0000) 2019-10-13T12:26:26.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:26.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.475194+0000) 2019-10-13T12:26:26.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:26.516+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.521123+0000) 2019-10-13T12:26:26.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:26.516+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.521921+0000) 2019-10-13T12:26:26.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:26.521+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.522178+0000) 2019-10-13T12:26:26.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:26.516+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.522715+0000) 2019-10-13T12:26:26.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:26.520+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.523294+0000) 2019-10-13T12:26:26.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:26.516+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.523034+0000) 2019-10-13T12:26:26.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:26.520+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.524311+0000) 2019-10-13T12:26:26.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:26.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:26.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:26.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.535232+0000) 2019-10-13T12:26:28.005 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:26:28.005 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:28.009 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:26:28.009 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:28.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:28.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.741255+0000) 2019-10-13T12:26:28.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:28.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.744824+0000) 2019-10-13T12:26:29.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:29.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:29.975850+0000) 2019-10-13T12:26:30.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:30.337+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:30.338180+0000) 2019-10-13T12:26:31.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:31.528+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.531853+0000) 2019-10-13T12:26:31.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:31.528+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.532094+0000) 2019-10-13T12:26:31.534 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:31.528+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.532546+0000) 2019-10-13T12:26:31.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:31.528+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.533600+0000) 2019-10-13T12:26:31.535 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:31.528+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.533998+0000) 2019-10-13T12:26:31.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:31.533+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.534406+0000) 2019-10-13T12:26:31.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:31.528+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.535198+0000) 2019-10-13T12:26:31.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:31.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:31.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:31.541+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.543853+0000) 2019-10-13T12:26:32.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:32.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.232974+0000) 2019-10-13T12:26:32.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:32.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.293048+0000) 2019-10-13T12:26:35.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:35.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:35.507044+0000) 2019-10-13T12:26:36.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:36.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.475408+0000) 2019-10-13T12:26:36.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:36.536+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.541512+0000) 2019-10-13T12:26:36.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:36.536+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.542108+0000) 2019-10-13T12:26:36.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:36.536+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.542831+0000) 2019-10-13T12:26:36.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:36.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.542996+0000) 2019-10-13T12:26:36.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:36.540+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.543623+0000) 2019-10-13T12:26:36.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:36.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.544387+0000) 2019-10-13T12:26:36.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:36.540+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.544644+0000) 2019-10-13T12:26:36.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:36.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:36.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:36.549+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.551635+0000) 2019-10-13T12:26:38.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:38.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.741455+0000) 2019-10-13T12:26:38.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:38.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.745065+0000) 2019-10-13T12:26:39.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:39.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:39.976037+0000) 2019-10-13T12:26:40.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:40.345+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:40.347568+0000) 2019-10-13T12:26:41.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:41.547+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.552254+0000) 2019-10-13T12:26:41.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:41.547+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.552855+0000) 2019-10-13T12:26:41.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:41.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.553782+0000) 2019-10-13T12:26:41.555 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:41.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.554571+0000) 2019-10-13T12:26:41.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:41.553+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.555504+0000) 2019-10-13T12:26:41.557 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:41.551+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.556038+0000) 2019-10-13T12:26:41.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:41.551+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.556440+0000) 2019-10-13T12:26:41.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:41.557+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.561004+0000) 2019-10-13T12:26:41.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:41.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:42.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:42.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.233198+0000) 2019-10-13T12:26:42.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:42.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.293234+0000) 2019-10-13T12:26:45.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:45.503+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:45.507246+0000) 2019-10-13T12:26:46.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:46.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.475612+0000) 2019-10-13T12:26:46.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:46.559+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.565018+0000) 2019-10-13T12:26:46.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:46.559+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.565324+0000) 2019-10-13T12:26:46.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:46.559+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.565813+0000) 2019-10-13T12:26:46.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:46.559+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.566312+0000) 2019-10-13T12:26:46.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:46.559+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.566736+0000) 2019-10-13T12:26:46.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:46.559+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.567056+0000) 2019-10-13T12:26:46.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:46.565+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.568244+0000) 2019-10-13T12:26:46.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:46.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 147995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:46.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:46.573+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.574238+0000) 2019-10-13T12:26:48.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:48.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.741669+0000) 2019-10-13T12:26:48.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:48.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.745281+0000) 2019-10-13T12:26:49.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:49.973+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:49.976236+0000) 2019-10-13T12:26:50.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:50.353+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:50.354335+0000) 2019-10-13T12:26:51.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:51.571+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.575546+0000) 2019-10-13T12:26:51.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:51.571+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.576247+0000) 2019-10-13T12:26:51.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:51.571+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.576695+0000) 2019-10-13T12:26:51.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:51.571+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.576999+0000) 2019-10-13T12:26:51.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:51.571+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.577375+0000) 2019-10-13T12:26:51.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:51.571+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.578220+0000) 2019-10-13T12:26:51.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:51.577+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.578781+0000) 2019-10-13T12:26:51.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:51.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:51.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:51.581+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.583833+0000) 2019-10-13T12:26:52.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:52.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.233432+0000) 2019-10-13T12:26:52.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:52.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.293412+0000) 2019-10-13T12:26:55.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:55.503+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:55.507521+0000) 2019-10-13T12:26:56.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:56.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.475808+0000) 2019-10-13T12:26:56.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:26:56.579+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.586437+0000) 2019-10-13T12:26:56.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:26:56.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.586708+0000) 2019-10-13T12:26:56.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:26:56.583+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.587858+0000) 2019-10-13T12:26:56.589 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:56.584+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.588316+0000) 2019-10-13T12:26:56.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:56.583+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.589088+0000) 2019-10-13T12:26:56.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:26:56.583+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.589435+0000) 2019-10-13T12:26:56.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:56.583+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.590089+0000) 2019-10-13T12:26:56.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:26:56.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:26:56.596 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:26:56.592+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.595166+0000) 2019-10-13T12:26:58.115 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:26:58.116 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:58.120 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:26:58.120 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:58.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:26:58.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.741868+0000) 2019-10-13T12:26:58.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:26:58.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.745475+0000) 2019-10-13T12:26:59.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:26:59.972+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:59.976462+0000) 2019-10-13T12:27:00.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:00.360+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:00.360953+0000) 2019-10-13T12:27:01.598 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:01.591+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.597498+0000) 2019-10-13T12:27:01.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:01.591+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.597680+0000) 2019-10-13T12:27:01.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:01.592+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.598010+0000) 2019-10-13T12:27:01.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:01.591+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.598353+0000) 2019-10-13T12:27:01.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:01.591+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.598667+0000) 2019-10-13T12:27:01.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:01.591+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.599132+0000) 2019-10-13T12:27:01.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:01.595+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.599916+0000) 2019-10-13T12:27:01.606 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:01.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:01.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:01.604+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.606751+0000) 2019-10-13T12:27:02.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:02.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.233612+0000) 2019-10-13T12:27:02.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:02.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.293594+0000) 2019-10-13T12:27:05.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:05.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:05.507723+0000) 2019-10-13T12:27:06.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:06.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.475999+0000) 2019-10-13T12:27:06.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:06.603+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.608638+0000) 2019-10-13T12:27:06.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:06.603+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.608878+0000) 2019-10-13T12:27:06.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:06.604+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.609081+0000) 2019-10-13T12:27:06.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:06.603+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.609927+0000) 2019-10-13T12:27:06.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:06.603+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.610783+0000) 2019-10-13T12:27:06.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:06.603+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.611131+0000) 2019-10-13T12:27:06.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:06.607+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.611569+0000) 2019-10-13T12:27:06.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:06.612+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.615805+0000) 2019-10-13T12:27:06.617 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:06.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:08.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:08.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.742084+0000) 2019-10-13T12:27:08.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:08.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.745670+0000) 2019-10-13T12:27:09.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:09.972+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:09.976691+0000) 2019-10-13T12:27:10.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:10.364+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:10.367522+0000) 2019-10-13T12:27:11.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:11.615+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.618787+0000) 2019-10-13T12:27:11.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:11.619+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.619721+0000) 2019-10-13T12:27:11.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:11.616+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.620126+0000) 2019-10-13T12:27:11.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:11.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.620935+0000) 2019-10-13T12:27:11.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:11.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.621511+0000) 2019-10-13T12:27:11.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:11.619+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.621867+0000) 2019-10-13T12:27:11.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:11.619+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.622852+0000) 2019-10-13T12:27:11.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:11.624+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.626692+0000) 2019-10-13T12:27:11.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:11.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:12.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:12.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.233792+0000) 2019-10-13T12:27:12.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:12.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.293774+0000) 2019-10-13T12:27:15.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:15.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:15.507920+0000) 2019-10-13T12:27:16.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:16.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.476202+0000) 2019-10-13T12:27:16.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:16.627+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.630677+0000) 2019-10-13T12:27:16.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:16.631+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.631699+0000) 2019-10-13T12:27:16.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:16.631+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.632381+0000) 2019-10-13T12:27:16.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:16.631+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.632770+0000) 2019-10-13T12:27:16.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:16.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.633304+0000) 2019-10-13T12:27:16.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:16.628+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.633852+0000) 2019-10-13T12:27:16.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:16.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.634564+0000) 2019-10-13T12:27:16.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:16.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:16.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:16.636+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.638171+0000) 2019-10-13T12:27:18.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:18.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.742308+0000) 2019-10-13T12:27:18.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:18.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.745865+0000) 2019-10-13T12:27:19.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:19.976+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:19.976918+0000) 2019-10-13T12:27:20.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:20.368+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:20.374021+0000) 2019-10-13T12:27:21.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:21.639+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.641855+0000) 2019-10-13T12:27:21.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:21.636+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.642025+0000) 2019-10-13T12:27:21.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:21.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.643525+0000) 2019-10-13T12:27:21.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:21.639+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.642836+0000) 2019-10-13T12:27:21.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:21.643+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.643742+0000) 2019-10-13T12:27:21.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:21.643+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.644221+0000) 2019-10-13T12:27:21.646 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:21.643+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.644968+0000) 2019-10-13T12:27:21.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:21.647+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:21.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:21.644+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.648656+0000) 2019-10-13T12:27:22.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:22.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.233968+0000) 2019-10-13T12:27:22.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:22.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.293951+0000) 2019-10-13T12:27:25.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:25.506+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:25.508128+0000) 2019-10-13T12:27:26.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:26.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.476397+0000) 2019-10-13T12:27:26.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:26.648+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.652487+0000) 2019-10-13T12:27:26.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:26.650+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.652920+0000) 2019-10-13T12:27:26.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:26.650+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.653282+0000) 2019-10-13T12:27:26.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:26.650+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.653431+0000) 2019-10-13T12:27:26.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:26.650+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.653826+0000) 2019-10-13T12:27:26.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:26.650+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.654387+0000) 2019-10-13T12:27:26.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:26.650+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.655257+0000) 2019-10-13T12:27:26.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:26.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:26.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:26.656+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.661434+0000) 2019-10-13T12:27:28.238 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:27:28.238 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:28.241 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:27:28.242 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:28.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:28.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.742460+0000) 2019-10-13T12:27:28.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:28.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.746099+0000) 2019-10-13T12:27:29.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:29.976+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:29.977086+0000) 2019-10-13T12:27:30.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:30.376+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:30.380557+0000) 2019-10-13T12:27:31.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:31.658+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.662669+0000) 2019-10-13T12:27:31.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:31.658+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.662403+0000) 2019-10-13T12:27:31.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:31.662+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.663674+0000) 2019-10-13T12:27:31.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:31.662+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.664298+0000) 2019-10-13T12:27:31.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:31.660+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.664719+0000) 2019-10-13T12:27:31.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:31.662+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.665205+0000) 2019-10-13T12:27:31.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:31.662+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.665471+0000) 2019-10-13T12:27:31.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:31.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:31.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:31.668+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.670505+0000) 2019-10-13T12:27:32.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:32.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:32.234146+0000) 2019-10-13T12:27:32.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:32.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:32.294166+0000) 2019-10-13T12:27:35.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:35.506+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:35.508327+0000) 2019-10-13T12:27:36.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:36.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.476625+0000) 2019-10-13T12:27:36.674 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:36.670+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.672653+0000) 2019-10-13T12:27:36.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:36.670+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.673151+0000) 2019-10-13T12:27:36.675 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:36.670+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.674036+0000) 2019-10-13T12:27:36.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:36.670+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.674503+0000) 2019-10-13T12:27:36.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:36.672+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.674877+0000) 2019-10-13T12:27:36.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:36.674+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.675683+0000) 2019-10-13T12:27:36.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:36.674+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.676476+0000) 2019-10-13T12:27:36.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:36.676+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.680202+0000) 2019-10-13T12:27:36.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:36.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:38.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:38.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.742722+0000) 2019-10-13T12:27:38.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:38.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.746298+0000) 2019-10-13T12:27:39.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:39.976+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.977311+0000) 2019-10-13T12:27:40.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:40.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:40.392134+0000) 2019-10-13T12:27:41.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:41.682+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.684114+0000) 2019-10-13T12:27:41.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:41.682+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.684493+0000) 2019-10-13T12:27:41.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:41.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.685165+0000) 2019-10-13T12:27:41.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:41.682+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.685723+0000) 2019-10-13T12:27:41.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:41.682+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.686089+0000) 2019-10-13T12:27:41.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:41.682+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.686620+0000) 2019-10-13T12:27:41.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:41.684+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.687563+0000) 2019-10-13T12:27:41.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:41.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148360 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:41.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:41.688+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.693577+0000) 2019-10-13T12:27:42.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:42.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:42.234317+0000) 2019-10-13T12:27:42.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:42.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:42.294352+0000) 2019-10-13T12:27:45.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:45.506+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:45.508551+0000) 2019-10-13T12:27:46.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:46.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.476883+0000) 2019-10-13T12:27:46.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:46.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.694536+0000) 2019-10-13T12:27:46.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:46.694+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.695586+0000) 2019-10-13T12:27:46.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:46.694+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.696017+0000) 2019-10-13T12:27:46.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:46.694+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.696456+0000) 2019-10-13T12:27:46.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:46.694+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.697328+0000) 2019-10-13T12:27:46.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:46.691+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.697555+0000) 2019-10-13T12:27:46.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:46.694+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.698258+0000) 2019-10-13T12:27:46.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:46.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:46.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:46.699+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.703456+0000) 2019-10-13T12:27:48.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:48.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.742926+0000) 2019-10-13T12:27:48.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:48.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.746440+0000) 2019-10-13T12:27:49.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:49.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.977520+0000) 2019-10-13T12:27:50.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:50.395+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:50.398792+0000) 2019-10-13T12:27:51.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:51.702+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.705558+0000) 2019-10-13T12:27:51.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:51.702+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.706015+0000) 2019-10-13T12:27:51.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:51.702+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.706455+0000) 2019-10-13T12:27:51.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:51.702+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.707283+0000) 2019-10-13T12:27:51.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:51.703+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.707439+0000) 2019-10-13T12:27:51.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:51.706+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.708055+0000) 2019-10-13T12:27:51.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:51.706+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.708658+0000) 2019-10-13T12:27:51.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:51.707+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.712994+0000) 2019-10-13T12:27:51.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:51.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:52.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:52.234+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:52.234502+0000) 2019-10-13T12:27:52.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:52.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:52.294556+0000) 2019-10-13T12:27:55.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:55.506+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:55.508753+0000) 2019-10-13T12:27:56.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:56.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.477099+0000) 2019-10-13T12:27:56.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:27:56.714+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.716450+0000) 2019-10-13T12:27:56.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:27:56.714+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.717228+0000) 2019-10-13T12:27:56.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:56.714+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.717504+0000) 2019-10-13T12:27:56.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:56.711+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.717859+0000) 2019-10-13T12:27:56.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:27:56.714+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.718243+0000) 2019-10-13T12:27:56.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:56.714+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.719124+0000) 2019-10-13T12:27:56.721 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:27:56.718+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.719933+0000) 2019-10-13T12:27:56.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:27:56.722+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:27:56.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:27:56.723+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.726862+0000) 2019-10-13T12:27:58.361 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:27:58.362 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:58.365 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:27:58.366 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:58.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:27:58.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.743125+0000) 2019-10-13T12:27:58.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:27:58.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.746588+0000) 2019-10-13T12:27:59.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:27:59.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.977716+0000) 2019-10-13T12:28:00.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:00.399+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:00.405472+0000) 2019-10-13T12:28:01.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:01.726+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.728450+0000) 2019-10-13T12:28:01.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:01.723+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.728822+0000) 2019-10-13T12:28:01.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:01.726+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.729252+0000) 2019-10-13T12:28:01.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:01.726+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.730051+0000) 2019-10-13T12:28:01.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:01.726+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.730593+0000) 2019-10-13T12:28:01.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:01.730+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.732131+0000) 2019-10-13T12:28:01.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:01.730+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.733102+0000) 2019-10-13T12:28:01.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:01.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148492 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:01.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:01.731+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.737660+0000) 2019-10-13T12:28:02.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:02.230+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:02.234673+0000) 2019-10-13T12:28:02.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:02.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:02.294746+0000) 2019-10-13T12:28:05.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:05.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:05.508956+0000) 2019-10-13T12:28:06.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:06.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.477333+0000) 2019-10-13T12:28:06.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:06.733+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.739132+0000) 2019-10-13T12:28:06.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:06.737+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.739758+0000) 2019-10-13T12:28:06.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:06.737+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.740334+0000) 2019-10-13T12:28:06.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:06.737+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.740947+0000) 2019-10-13T12:28:06.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:06.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.741310+0000) 2019-10-13T12:28:06.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:06.737+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.741659+0000) 2019-10-13T12:28:06.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:06.735+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.742056+0000) 2019-10-13T12:28:06.753 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:06.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:06.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:06.743+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.749810+0000) 2019-10-13T12:28:08.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:08.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.743374+0000) 2019-10-13T12:28:08.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:08.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.746747+0000) 2019-10-13T12:28:09.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:09.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.977938+0000) 2019-10-13T12:28:10.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:10.407+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:10.412126+0000) 2019-10-13T12:28:11.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:11.745+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.750935+0000) 2019-10-13T12:28:11.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:11.749+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.751936+0000) 2019-10-13T12:28:11.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:11.749+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.751989+0000) 2019-10-13T12:28:11.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:11.749+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.752590+0000) 2019-10-13T12:28:11.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:11.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.752926+0000) 2019-10-13T12:28:11.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:11.749+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.753527+0000) 2019-10-13T12:28:11.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:11.751+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.754484+0000) 2019-10-13T12:28:11.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:11.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:11.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:11.755+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.759117+0000) 2019-10-13T12:28:12.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:12.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:12.234864+0000) 2019-10-13T12:28:12.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:12.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:12.294930+0000) 2019-10-13T12:28:15.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:15.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:15.509171+0000) 2019-10-13T12:28:16.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:16.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.477562+0000) 2019-10-13T12:28:16.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:16.757+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.761106+0000) 2019-10-13T12:28:16.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:16.757+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.761630+0000) 2019-10-13T12:28:16.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:16.759+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.762370+0000) 2019-10-13T12:28:16.765 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:16.757+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.762990+0000) 2019-10-13T12:28:16.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:16.761+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.763592+0000) 2019-10-13T12:28:16.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:16.761+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.764586+0000) 2019-10-13T12:28:16.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:16.761+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.765079+0000) 2019-10-13T12:28:16.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:16.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:16.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:16.767+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.770428+0000) 2019-10-13T12:28:18.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:18.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.743592+0000) 2019-10-13T12:28:18.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:18.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.746979+0000) 2019-10-13T12:28:19.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:19.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.978127+0000) 2019-10-13T12:28:20.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:20.415+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:20.418679+0000) 2019-10-13T12:28:21.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:21.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.770940+0000) 2019-10-13T12:28:21.772 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:21.765+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.771113+0000) 2019-10-13T12:28:21.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:21.769+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.772137+0000) 2019-10-13T12:28:21.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:21.769+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.772528+0000) 2019-10-13T12:28:21.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:21.769+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.772865+0000) 2019-10-13T12:28:21.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:21.769+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.773304+0000) 2019-10-13T12:28:21.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:21.767+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.773526+0000) 2019-10-13T12:28:21.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:21.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148625 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:21.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:21.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.782474+0000) 2019-10-13T12:28:22.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:22.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:22.235077+0000) 2019-10-13T12:28:22.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:22.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:22.295127+0000) 2019-10-13T12:28:25.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:25.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:25.509377+0000) 2019-10-13T12:28:26.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:26.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.477819+0000) 2019-10-13T12:28:26.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:26.775+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.781930+0000) 2019-10-13T12:28:26.784 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:26.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.782423+0000) 2019-10-13T12:28:26.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:26.781+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.783780+0000) 2019-10-13T12:28:26.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:26.781+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.784663+0000) 2019-10-13T12:28:26.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:26.781+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.785453+0000) 2019-10-13T12:28:26.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:26.781+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.785938+0000) 2019-10-13T12:28:26.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:26.781+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.786257+0000) 2019-10-13T12:28:26.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:26.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:26.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:26.787+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.790977+0000) 2019-10-13T12:28:28.491 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:28:28.492 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:28.495 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:28:28.495 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:28.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:28.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.743797+0000) 2019-10-13T12:28:28.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:28.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.747175+0000) 2019-10-13T12:28:29.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:29.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.978359+0000) 2019-10-13T12:28:30.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:30.419+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:30.425106+0000) 2019-10-13T12:28:31.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:31.789+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.794028+0000) 2019-10-13T12:28:31.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:31.789+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.794960+0000) 2019-10-13T12:28:31.796 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:31.791+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.795178+0000) 2019-10-13T12:28:31.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:31.793+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.795521+0000) 2019-10-13T12:28:31.797 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:31.793+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.796475+0000) 2019-10-13T12:28:31.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:31.793+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.796943+0000) 2019-10-13T12:28:31.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:31.793+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.797447+0000) 2019-10-13T12:28:31.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:31.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:31.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:31.799+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.805098+0000) 2019-10-13T12:28:32.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:32.229+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:32.235256+0000) 2019-10-13T12:28:32.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:32.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:32.295335+0000) 2019-10-13T12:28:35.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:35.505+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:35.509572+0000) 2019-10-13T12:28:36.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:36.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.478012+0000) 2019-10-13T12:28:36.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:36.801+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.804220+0000) 2019-10-13T12:28:36.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:36.799+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.806089+0000) 2019-10-13T12:28:36.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:36.801+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.806668+0000) 2019-10-13T12:28:36.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:36.801+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.806799+0000) 2019-10-13T12:28:36.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:36.801+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.807257+0000) 2019-10-13T12:28:36.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:36.805+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.807613+0000) 2019-10-13T12:28:36.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:36.805+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.807958+0000) 2019-10-13T12:28:36.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:36.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:36.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:36.811+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.814940+0000) 2019-10-13T12:28:38.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:38.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.744013+0000) 2019-10-13T12:28:38.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:38.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.747439+0000) 2019-10-13T12:28:39.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:39.975+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.978518+0000) 2019-10-13T12:28:40.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:40.435+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:40.436385+0000) 2019-10-13T12:28:41.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:41.813+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.815284+0000) 2019-10-13T12:28:41.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:41.813+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.815700+0000) 2019-10-13T12:28:41.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:41.810+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.816952+0000) 2019-10-13T12:28:41.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:41.813+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.817652+0000) 2019-10-13T12:28:41.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:41.813+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.818251+0000) 2019-10-13T12:28:41.819 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:41.813+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.818654+0000) 2019-10-13T12:28:41.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:41.813+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.819171+0000) 2019-10-13T12:28:41.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:41.818+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.822442+0000) 2019-10-13T12:28:41.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:41.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148759 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:42.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:42.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:42.235468+0000) 2019-10-13T12:28:42.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:42.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:42.295547+0000) 2019-10-13T12:28:45.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:45.509+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:45.509772+0000) 2019-10-13T12:28:46.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:46.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.478207+0000) 2019-10-13T12:28:46.827 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:46.821+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.826177+0000) 2019-10-13T12:28:46.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:46.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.827105+0000) 2019-10-13T12:28:46.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:46.825+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.827560+0000) 2019-10-13T12:28:46.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:46.825+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.828085+0000) 2019-10-13T12:28:46.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:46.825+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.828421+0000) 2019-10-13T12:28:46.830 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:46.822+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.828890+0000) 2019-10-13T12:28:46.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:46.825+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.829939+0000) 2019-10-13T12:28:46.835 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:46.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:46.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:46.830+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.835116+0000) 2019-10-13T12:28:48.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:48.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.744205+0000) 2019-10-13T12:28:48.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:48.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.747647+0000) 2019-10-13T12:28:49.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:49.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.978732+0000) 2019-10-13T12:28:50.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:50.446+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:50.447992+0000) 2019-10-13T12:28:51.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:51.832+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.837784+0000) 2019-10-13T12:28:51.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:51.832+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.838413+0000) 2019-10-13T12:28:51.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:51.832+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.838672+0000) 2019-10-13T12:28:51.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:51.832+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.839151+0000) 2019-10-13T12:28:51.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:51.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.839500+0000) 2019-10-13T12:28:51.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:51.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.840676+0000) 2019-10-13T12:28:51.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:51.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.841808+0000) 2019-10-13T12:28:51.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:51.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:51.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:51.842+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.847123+0000) 2019-10-13T12:28:52.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:52.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:52.235599+0000) 2019-10-13T12:28:52.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:52.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:52.295756+0000) 2019-10-13T12:28:55.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:55.508+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:55.509980+0000) 2019-10-13T12:28:56.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:56.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.478425+0000) 2019-10-13T12:28:56.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:28:56.844+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.849377+0000) 2019-10-13T12:28:56.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:28:56.844+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.850066+0000) 2019-10-13T12:28:56.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:56.844+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.850580+0000) 2019-10-13T12:28:56.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:28:56.848+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.851384+0000) 2019-10-13T12:28:56.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:28:56.848+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.852245+0000) 2019-10-13T12:28:56.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:56.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.852824+0000) 2019-10-13T12:28:56.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:56.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.853788+0000) 2019-10-13T12:28:56.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:28:56.850+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.855998+0000) 2019-10-13T12:28:56.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:28:56.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:28:58.602 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:28:58.602 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:58.634 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:28:58.634 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:58.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:28:58.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.744392+0000) 2019-10-13T12:28:58.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:28:58.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.747871+0000) 2019-10-13T12:28:59.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:28:59.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.978966+0000) 2019-10-13T12:29:00.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:00.450+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:00.454394+0000) 2019-10-13T12:29:01.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:01.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.860119+0000) 2019-10-13T12:29:01.862 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:01.856+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.860882+0000) 2019-10-13T12:29:01.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:01.856+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.861258+0000) 2019-10-13T12:29:01.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:01.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.862200+0000) 2019-10-13T12:29:01.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:01.856+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.862536+0000) 2019-10-13T12:29:01.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:01.858+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.862826+0000) 2019-10-13T12:29:01.865 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:01.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.863168+0000) 2019-10-13T12:29:01.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:01.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:01.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:01.866+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.870792+0000) 2019-10-13T12:29:02.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:02.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:02.235843+0000) 2019-10-13T12:29:02.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:02.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:02.295946+0000) 2019-10-13T12:29:05.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:05.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:05.510262+0000) 2019-10-13T12:29:06.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:06.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.478620+0000) 2019-10-13T12:29:06.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:06.868+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.871645+0000) 2019-10-13T12:29:06.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:06.868+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.872193+0000) 2019-10-13T12:29:06.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:06.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.872329+0000) 2019-10-13T12:29:06.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:06.868+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.873090+0000) 2019-10-13T12:29:06.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:06.866+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.873418+0000) 2019-10-13T12:29:06.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:06.868+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.873691+0000) 2019-10-13T12:29:06.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:06.868+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.874000+0000) 2019-10-13T12:29:06.880 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:06.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:06.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:06.874+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.880459+0000) 2019-10-13T12:29:08.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:08.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.744597+0000) 2019-10-13T12:29:08.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:08.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.748094+0000) 2019-10-13T12:29:09.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:09.978+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.979166+0000) 2019-10-13T12:29:10.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:10.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:10.466133+0000) 2019-10-13T12:29:11.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:11.876+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.882803+0000) 2019-10-13T12:29:11.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:11.876+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.883057+0000) 2019-10-13T12:29:11.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:11.878+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.883456+0000) 2019-10-13T12:29:11.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:11.880+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.884867+0000) 2019-10-13T12:29:11.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:11.880+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.885133+0000) 2019-10-13T12:29:11.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:11.880+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.886136+0000) 2019-10-13T12:29:11.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:11.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.886226+0000) 2019-10-13T12:29:11.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:11.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:11.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:11.886+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.891053+0000) 2019-10-13T12:29:12.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:12.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:12.236036+0000) 2019-10-13T12:29:12.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:12.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:12.296143+0000) 2019-10-13T12:29:15.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:15.504+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:15.510458+0000) 2019-10-13T12:29:16.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:16.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.478815+0000) 2019-10-13T12:29:16.895 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:16.888+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.893676+0000) 2019-10-13T12:29:16.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:16.888+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.894477+0000) 2019-10-13T12:29:16.896 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:16.888+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.894931+0000) 2019-10-13T12:29:16.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:16.892+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.895562+0000) 2019-10-13T12:29:16.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:16.892+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.896253+0000) 2019-10-13T12:29:16.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:16.890+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.896478+0000) 2019-10-13T12:29:16.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:16.892+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.897493+0000) 2019-10-13T12:29:16.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:16.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 148991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:16.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:16.902+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.907079+0000) 2019-10-13T12:29:18.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:18.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.744858+0000) 2019-10-13T12:29:18.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:18.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.748306+0000) 2019-10-13T12:29:19.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:19.974+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.979354+0000) 2019-10-13T12:29:20.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:20.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:20.472174+0000) 2019-10-13T12:29:21.906 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:21.900+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.905129+0000) 2019-10-13T12:29:21.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:21.902+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.905395+0000) 2019-10-13T12:29:21.907 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:21.900+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.906060+0000) 2019-10-13T12:29:21.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:21.900+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.906824+0000) 2019-10-13T12:29:21.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:21.904+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.907431+0000) 2019-10-13T12:29:21.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:21.904+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.908043+0000) 2019-10-13T12:29:21.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:21.904+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.908696+0000) 2019-10-13T12:29:21.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:21.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:21.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:21.910+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.915231+0000) 2019-10-13T12:29:22.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:22.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:22.236229+0000) 2019-10-13T12:29:22.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:22.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:22.296343+0000) 2019-10-13T12:29:25.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:25.508+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:25.510656+0000) 2019-10-13T12:29:26.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:26.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.479041+0000) 2019-10-13T12:29:26.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:26.912+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.916293+0000) 2019-10-13T12:29:26.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:26.910+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.916803+0000) 2019-10-13T12:29:26.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:26.912+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.917237+0000) 2019-10-13T12:29:26.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:26.912+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.917814+0000) 2019-10-13T12:29:26.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:26.916+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.919625+0000) 2019-10-13T12:29:26.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:26.916+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.919899+0000) 2019-10-13T12:29:26.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:26.916+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.920479+0000) 2019-10-13T12:29:26.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:26.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:26.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:26.926+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.927960+0000) 2019-10-13T12:29:28.706 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:29:28.706 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:28.710 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:29:28.710 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:28.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:28.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.745028+0000) 2019-10-13T12:29:28.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:28.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.748498+0000) 2019-10-13T12:29:29.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:29.978+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.979586+0000) 2019-10-13T12:29:30.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:30.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:30.482967+0000) 2019-10-13T12:29:31.930 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:31.923+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.928557+0000) 2019-10-13T12:29:31.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:31.923+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.929670+0000) 2019-10-13T12:29:31.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:31.923+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.930300+0000) 2019-10-13T12:29:31.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:31.923+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.930309+0000) 2019-10-13T12:29:31.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:31.930+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.930578+0000) 2019-10-13T12:29:31.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:31.923+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.930993+0000) 2019-10-13T12:29:31.933 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:31.927+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.932214+0000) 2019-10-13T12:29:31.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:31.934+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.935749+0000) 2019-10-13T12:29:31.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:31.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:32.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:32.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:32.236407+0000) 2019-10-13T12:29:32.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:32.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:32.296540+0000) 2019-10-13T12:29:35.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:35.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:35.510877+0000) 2019-10-13T12:29:36.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:36.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.479251+0000) 2019-10-13T12:29:36.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:36.935+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.939396+0000) 2019-10-13T12:29:36.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:36.935+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.940753+0000) 2019-10-13T12:29:36.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:36.937+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.941207+0000) 2019-10-13T12:29:36.943 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:36.935+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.941547+0000) 2019-10-13T12:29:36.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:36.935+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.941845+0000) 2019-10-13T12:29:36.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:36.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.942361+0000) 2019-10-13T12:29:36.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:36.935+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.942720+0000) 2019-10-13T12:29:36.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:36.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149125 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:36.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:36.945+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.949623+0000) 2019-10-13T12:29:38.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:38.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.745241+0000) 2019-10-13T12:29:38.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:38.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.748700+0000) 2019-10-13T12:29:39.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:39.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.979814+0000) 2019-10-13T12:29:40.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:40.485+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:40.489464+0000) 2019-10-13T12:29:41.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:41.949+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.950292+0000) 2019-10-13T12:29:41.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:41.943+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.950791+0000) 2019-10-13T12:29:41.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:41.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.951162+0000) 2019-10-13T12:29:41.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:41.947+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.951617+0000) 2019-10-13T12:29:41.953 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:41.947+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.952355+0000) 2019-10-13T12:29:41.954 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:41.947+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.952062+0000) 2019-10-13T12:29:41.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:41.947+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.953855+0000) 2019-10-13T12:29:41.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:41.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:41.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:41.957+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.958975+0000) 2019-10-13T12:29:42.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:42.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:42.236587+0000) 2019-10-13T12:29:42.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:42.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:42.296722+0000) 2019-10-13T12:29:45.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:45.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:45.511082+0000) 2019-10-13T12:29:46.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:46.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.479517+0000) 2019-10-13T12:29:46.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:46.955+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.961053+0000) 2019-10-13T12:29:46.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:46.955+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.961914+0000) 2019-10-13T12:29:46.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:46.955+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.962841+0000) 2019-10-13T12:29:46.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:46.961+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.962997+0000) 2019-10-13T12:29:46.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:46.959+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.963493+0000) 2019-10-13T12:29:46.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:46.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.964268+0000) 2019-10-13T12:29:46.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:46.959+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.965094+0000) 2019-10-13T12:29:46.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:46.965+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.967717+0000) 2019-10-13T12:29:46.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:46.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:48.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:48.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.745442+0000) 2019-10-13T12:29:48.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:48.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.748916+0000) 2019-10-13T12:29:49.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:49.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.980020+0000) 2019-10-13T12:29:50.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:50.493+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:50.496081+0000) 2019-10-13T12:29:51.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:51.967+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.972367+0000) 2019-10-13T12:29:51.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:51.967+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.972807+0000) 2019-10-13T12:29:51.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:51.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.973537+0000) 2019-10-13T12:29:51.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:51.967+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.974060+0000) 2019-10-13T12:29:51.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:51.967+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.974367+0000) 2019-10-13T12:29:51.976 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:51.971+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.975592+0000) 2019-10-13T12:29:51.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:51.971+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.976595+0000) 2019-10-13T12:29:51.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:51.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:51.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:51.981+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.984279+0000) 2019-10-13T12:29:52.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:52.231+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:52.236822+0000) 2019-10-13T12:29:52.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:52.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:52.296915+0000) 2019-10-13T12:29:55.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:55.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:55.511282+0000) 2019-10-13T12:29:56.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:56.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.479711+0000) 2019-10-13T12:29:56.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:56.979+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.983499+0000) 2019-10-13T12:29:56.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:56.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.984423+0000) 2019-10-13T12:29:56.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:29:56.979+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.984757+0000) 2019-10-13T12:29:56.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:29:56.979+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.984946+0000) 2019-10-13T12:29:56.987 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:29:56.979+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.986210+0000) 2019-10-13T12:29:56.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:56.979+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.986711+0000) 2019-10-13T12:29:56.988 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:29:56.983+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.987569+0000) 2019-10-13T12:29:56.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:29:56.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:29:56.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:29:56.989+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.994074+0000) 2019-10-13T12:29:58.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:29:58.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.745639+0000) 2019-10-13T12:29:58.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:29:58.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.749128+0000) 2019-10-13T12:29:58.838 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:29:58.838 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:58.842 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:29:58.842 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:59.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:29:59.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.980217+0000) 2019-10-13T12:30:00.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:00.501+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:00.502434+0000) 2019-10-13T12:30:01.996 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:01.991+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.994816+0000) 2019-10-13T12:30:01.996 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:01.995+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.995424+0000) 2019-10-13T12:30:01.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:01.995+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.996288+0000) 2019-10-13T12:30:01.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:01.993+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.996651+0000) 2019-10-13T12:30:01.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:01.995+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.997307+0000) 2019-10-13T12:30:01.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:01.995+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.998093+0000) 2019-10-13T12:30:01.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:01.995+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.998331+0000) 2019-10-13T12:30:02.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:01.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:02.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:02.001+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:02.005052+0000) 2019-10-13T12:30:02.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:02.235+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:02.237042+0000) 2019-10-13T12:30:02.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:02.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:02.297126+0000) 2019-10-13T12:30:05.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:05.511+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:05.511514+0000) 2019-10-13T12:30:06.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:06.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:06.479934+0000) 2019-10-13T12:30:07.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:07.003+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.005917+0000) 2019-10-13T12:30:07.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:07.003+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.006920+0000) 2019-10-13T12:30:07.008 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:07.007+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.007469+0000) 2019-10-13T12:30:07.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:07.007+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.007757+0000) 2019-10-13T12:30:07.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:07.005+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.008057+0000) 2019-10-13T12:30:07.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:07.007+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.008361+0000) 2019-10-13T12:30:07.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:07.007+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.008696+0000) 2019-10-13T12:30:07.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:07.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:07.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:07.013+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.014940+0000) 2019-10-13T12:30:08.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:08.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.745868+0000) 2019-10-13T12:30:08.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:08.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.749327+0000) 2019-10-13T12:30:09.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:09.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.980457+0000) 2019-10-13T12:30:10.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:10.505+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:10.508833+0000) 2019-10-13T12:30:12.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:12.015+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.016011+0000) 2019-10-13T12:30:12.018 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:12.015+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.017137+0000) 2019-10-13T12:30:12.019 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:12.015+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.017877+0000) 2019-10-13T12:30:12.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:12.015+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.018174+0000) 2019-10-13T12:30:12.019 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:12.015+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.018507+0000) 2019-10-13T12:30:12.020 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:12.019+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.019441+0000) 2019-10-13T12:30:12.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:12.017+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.019988+0000) 2019-10-13T12:30:12.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:12.021+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.023793+0000) 2019-10-13T12:30:12.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:12.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:12.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:12.235+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.237227+0000) 2019-10-13T12:30:12.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:12.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:12.297312+0000) 2019-10-13T12:30:15.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:15.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:15.511726+0000) 2019-10-13T12:30:16.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:16.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:16.480132+0000) 2019-10-13T12:30:17.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:17.022+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.026885+0000) 2019-10-13T12:30:17.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:17.026+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.027658+0000) 2019-10-13T12:30:17.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:17.026+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.028238+0000) 2019-10-13T12:30:17.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:17.025+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.028722+0000) 2019-10-13T12:30:17.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:17.026+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.029630+0000) 2019-10-13T12:30:17.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:17.026+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.029866+0000) 2019-10-13T12:30:17.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:17.026+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.030226+0000) 2019-10-13T12:30:17.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:17.029+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.033953+0000) 2019-10-13T12:30:17.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:17.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:18.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:18.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.746057+0000) 2019-10-13T12:30:18.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:18.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.749518+0000) 2019-10-13T12:30:19.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:19.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.980647+0000) 2019-10-13T12:30:20.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:20.517+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:20.519480+0000) 2019-10-13T12:30:22.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:22.034+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.038413+0000) 2019-10-13T12:30:22.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:22.034+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.038911+0000) 2019-10-13T12:30:22.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:22.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.039720+0000) 2019-10-13T12:30:22.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:22.038+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.039938+0000) 2019-10-13T12:30:22.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:22.038+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.040407+0000) 2019-10-13T12:30:22.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:22.038+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.040678+0000) 2019-10-13T12:30:22.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:22.037+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.041226+0000) 2019-10-13T12:30:22.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:22.041+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.044210+0000) 2019-10-13T12:30:22.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:22.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:22.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:22.234+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.237407+0000) 2019-10-13T12:30:22.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:22.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:22.297528+0000) 2019-10-13T12:30:25.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:25.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:25.511952+0000) 2019-10-13T12:30:26.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:26.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:26.480356+0000) 2019-10-13T12:30:27.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:27.046+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.050646+0000) 2019-10-13T12:30:27.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:27.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.051029+0000) 2019-10-13T12:30:27.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:27.049+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.051967+0000) 2019-10-13T12:30:27.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:27.050+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.051516+0000) 2019-10-13T12:30:27.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:27.050+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.052165+0000) 2019-10-13T12:30:27.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:27.050+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.052644+0000) 2019-10-13T12:30:27.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:27.050+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.053367+0000) 2019-10-13T12:30:27.058 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:27.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:27.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:27.057+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.060200+0000) 2019-10-13T12:30:28.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:28.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.746289+0000) 2019-10-13T12:30:28.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:28.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.749728+0000) 2019-10-13T12:30:28.935 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:30:28.936 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:28.939 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:30:28.940 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:29.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:29.977+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.980838+0000) 2019-10-13T12:30:30.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:30.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:30.525978+0000) 2019-10-13T12:30:32.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:32.056+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.060046+0000) 2019-10-13T12:30:32.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:32.058+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.060372+0000) 2019-10-13T12:30:32.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:32.058+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.061014+0000) 2019-10-13T12:30:32.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:32.058+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.061773+0000) 2019-10-13T12:30:32.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:32.058+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.062084+0000) 2019-10-13T12:30:32.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:32.058+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.062874+0000) 2019-10-13T12:30:32.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:32.058+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.063227+0000) 2019-10-13T12:30:32.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:32.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:32.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:32.068+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.070364+0000) 2019-10-13T12:30:32.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:32.234+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.237618+0000) 2019-10-13T12:30:32.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:32.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:32.297729+0000) 2019-10-13T12:30:35.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:35.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:35.512176+0000) 2019-10-13T12:30:36.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:36.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:36.480553+0000) 2019-10-13T12:30:37.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:37.066+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.070492+0000) 2019-10-13T12:30:37.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:37.070+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.071929+0000) 2019-10-13T12:30:37.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:37.070+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.072852+0000) 2019-10-13T12:30:37.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:37.070+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.073553+0000) 2019-10-13T12:30:37.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:37.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.074113+0000) 2019-10-13T12:30:37.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:37.072+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.074451+0000) 2019-10-13T12:30:37.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:37.070+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.074715+0000) 2019-10-13T12:30:37.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:37.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:37.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:37.076+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.079636+0000) 2019-10-13T12:30:38.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:38.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.746516+0000) 2019-10-13T12:30:38.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:38.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.749943+0000) 2019-10-13T12:30:39.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:39.980+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.981069+0000) 2019-10-13T12:30:40.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:40.536+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:40.536994+0000) 2019-10-13T12:30:42.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:42.078+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.082229+0000) 2019-10-13T12:30:42.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:42.078+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.082863+0000) 2019-10-13T12:30:42.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:42.082+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.083593+0000) 2019-10-13T12:30:42.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:42.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.083898+0000) 2019-10-13T12:30:42.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:42.080+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.084688+0000) 2019-10-13T12:30:42.086 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:42.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.084577+0000) 2019-10-13T12:30:42.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:42.082+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.084994+0000) 2019-10-13T12:30:42.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:42.084+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.089219+0000) 2019-10-13T12:30:42.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:42.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149561 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:42.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:42.234+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.237749+0000) 2019-10-13T12:30:42.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:42.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:42.297905+0000) 2019-10-13T12:30:45.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:45.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:45.512375+0000) 2019-10-13T12:30:46.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:46.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:46.480787+0000) 2019-10-13T12:30:47.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:47.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.093413+0000) 2019-10-13T12:30:47.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:47.090+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.093828+0000) 2019-10-13T12:30:47.096 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:47.090+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.094460+0000) 2019-10-13T12:30:47.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:47.090+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.094815+0000) 2019-10-13T12:30:47.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:47.094+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.095554+0000) 2019-10-13T12:30:47.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:47.092+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.095595+0000) 2019-10-13T12:30:47.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:47.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.097103+0000) 2019-10-13T12:30:47.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:47.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149594 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:47.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:47.100+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.102480+0000) 2019-10-13T12:30:48.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:48.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.746745+0000) 2019-10-13T12:30:48.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:48.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.750143+0000) 2019-10-13T12:30:49.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:49.976+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.981287+0000) 2019-10-13T12:30:50.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:50.540+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:50.543395+0000) 2019-10-13T12:30:52.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:52.102+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.104683+0000) 2019-10-13T12:30:52.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:52.102+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.104884+0000) 2019-10-13T12:30:52.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:52.102+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.105477+0000) 2019-10-13T12:30:52.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:52.102+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.106205+0000) 2019-10-13T12:30:52.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:52.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.106581+0000) 2019-10-13T12:30:52.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:52.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.107023+0000) 2019-10-13T12:30:52.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:52.104+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.108037+0000) 2019-10-13T12:30:52.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:52.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:52.115 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:52.108+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.114005+0000) 2019-10-13T12:30:52.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:52.234+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.237941+0000) 2019-10-13T12:30:52.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:52.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:52.298089+0000) 2019-10-13T12:30:55.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:55.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:55.512573+0000) 2019-10-13T12:30:56.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:56.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:56.480987+0000) 2019-10-13T12:30:57.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:30:57.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.115388+0000) 2019-10-13T12:30:57.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:30:57.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.116336+0000) 2019-10-13T12:30:57.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:57.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.116740+0000) 2019-10-13T12:30:57.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:57.113+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.116913+0000) 2019-10-13T12:30:57.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:30:57.113+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.117327+0000) 2019-10-13T12:30:57.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:30:57.113+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.117536+0000) 2019-10-13T12:30:57.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:57.112+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.117936+0000) 2019-10-13T12:30:57.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:30:57.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:30:57.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:30:57.120+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.124820+0000) 2019-10-13T12:30:58.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:30:58.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.746941+0000) 2019-10-13T12:30:58.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:30:58.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.750350+0000) 2019-10-13T12:30:59.055 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:30:59.056 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:59.059 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:30:59.060 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:59.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:30:59.980+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:59.981512+0000) 2019-10-13T12:31:00.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:00.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:00.550690+0000) 2019-10-13T12:31:02.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:02.121+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.125035+0000) 2019-10-13T12:31:02.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:02.124+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.127672+0000) 2019-10-13T12:31:02.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:02.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.134385+0000) 2019-10-13T12:31:02.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:02.121+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.126149+0000) 2019-10-13T12:31:02.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:02.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.298275+0000) 2019-10-13T12:31:02.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:02.121+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.126720+0000) 2019-10-13T12:31:02.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:02.125+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.128490+0000) 2019-10-13T12:31:02.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:02.125+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.128733+0000) 2019-10-13T12:31:02.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:02.125+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.129217+0000) 2019-10-13T12:31:02.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:02.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:02.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:02.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:02.238149+0000) 2019-10-13T12:31:05.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:05.509+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:05.512773+0000) 2019-10-13T12:31:06.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:06.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:06.481180+0000) 2019-10-13T12:31:07.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:07.133+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.137583+0000) 2019-10-13T12:31:07.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:07.133+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.138580+0000) 2019-10-13T12:31:07.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:07.133+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.138785+0000) 2019-10-13T12:31:07.140 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:07.137+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.139342+0000) 2019-10-13T12:31:07.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:07.137+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.139449+0000) 2019-10-13T12:31:07.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:07.136+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.139777+0000) 2019-10-13T12:31:07.142 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:07.137+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.140131+0000) 2019-10-13T12:31:07.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:07.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:07.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:07.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.146818+0000) 2019-10-13T12:31:08.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:08.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.747172+0000) 2019-10-13T12:31:08.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:08.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.750542+0000) 2019-10-13T12:31:09.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:09.976+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:09.981729+0000) 2019-10-13T12:31:10.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:10.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:10.557158+0000) 2019-10-13T12:31:12.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:12.145+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.148913+0000) 2019-10-13T12:31:12.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:12.145+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.149647+0000) 2019-10-13T12:31:12.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:12.145+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.149915+0000) 2019-10-13T12:31:12.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:12.145+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.150265+0000) 2019-10-13T12:31:12.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:12.145+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.150616+0000) 2019-10-13T12:31:12.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:12.149+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.151361+0000) 2019-10-13T12:31:12.154 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:12.148+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.151938+0000) 2019-10-13T12:31:12.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:12.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:12.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:12.156+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.162039+0000) 2019-10-13T12:31:12.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:12.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.238368+0000) 2019-10-13T12:31:12.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:12.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:12.298467+0000) 2019-10-13T12:31:15.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:15.509+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:15.512974+0000) 2019-10-13T12:31:16.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:16.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:16.481376+0000) 2019-10-13T12:31:17.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:17.157+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.160478+0000) 2019-10-13T12:31:17.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:17.157+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.160934+0000) 2019-10-13T12:31:17.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:17.157+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.161427+0000) 2019-10-13T12:31:17.163 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:17.157+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.162123+0000) 2019-10-13T12:31:17.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:17.157+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.162702+0000) 2019-10-13T12:31:17.164 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:17.161+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.163612+0000) 2019-10-13T12:31:17.165 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:17.160+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.163723+0000) 2019-10-13T12:31:17.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:17.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149794 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:17.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:17.168+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.172743+0000) 2019-10-13T12:31:18.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:18.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.747398+0000) 2019-10-13T12:31:18.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:18.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.750740+0000) 2019-10-13T12:31:19.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:19.980+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:19.981985+0000) 2019-10-13T12:31:20.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:20.560+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:20.563614+0000) 2019-10-13T12:31:22.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:22.169+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.172412+0000) 2019-10-13T12:31:22.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:22.169+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.172829+0000) 2019-10-13T12:31:22.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:22.169+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.173279+0000) 2019-10-13T12:31:22.175 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:22.169+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.174008+0000) 2019-10-13T12:31:22.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:22.169+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.175116+0000) 2019-10-13T12:31:22.176 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:22.173+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.175707+0000) 2019-10-13T12:31:22.177 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:22.172+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.176160+0000) 2019-10-13T12:31:22.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:22.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:22.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:22.176+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.181592+0000) 2019-10-13T12:31:22.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:22.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.238551+0000) 2019-10-13T12:31:22.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:22.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:22.298686+0000) 2019-10-13T12:31:25.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:25.509+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:25.513175+0000) 2019-10-13T12:31:26.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:26.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:26.481545+0000) 2019-10-13T12:31:27.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:27.177+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.182646+0000) 2019-10-13T12:31:27.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:27.181+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.183748+0000) 2019-10-13T12:31:27.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:27.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.184187+0000) 2019-10-13T12:31:27.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:27.181+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.184532+0000) 2019-10-13T12:31:27.186 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:27.181+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.184969+0000) 2019-10-13T12:31:27.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:27.181+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.185092+0000) 2019-10-13T12:31:27.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:27.181+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.186235+0000) 2019-10-13T12:31:27.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:27.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:27.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:27.191+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.194805+0000) 2019-10-13T12:31:28.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:28.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.747592+0000) 2019-10-13T12:31:28.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:28.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.750931+0000) 2019-10-13T12:31:29.166 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:31:29.166 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:29.170 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:31:29.170 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:29.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:29.979+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:29.982201+0000) 2019-10-13T12:31:30.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:30.567+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:30.570208+0000) 2019-10-13T12:31:32.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:32.191+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.194186+0000) 2019-10-13T12:31:32.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:32.189+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.195112+0000) 2019-10-13T12:31:32.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:32.193+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.195430+0000) 2019-10-13T12:31:32.197 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:32.193+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.195905+0000) 2019-10-13T12:31:32.197 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:32.193+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.196118+0000) 2019-10-13T12:31:32.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:32.193+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.196407+0000) 2019-10-13T12:31:32.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:32.193+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.196875+0000) 2019-10-13T12:31:32.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:32.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:32.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:32.199+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.202940+0000) 2019-10-13T12:31:32.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:32.233+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.238677+0000) 2019-10-13T12:31:32.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:32.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:32.298884+0000) 2019-10-13T12:31:35.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:35.513+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:35.513425+0000) 2019-10-13T12:31:36.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:36.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:36.481709+0000) 2019-10-13T12:31:37.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:37.201+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.205292+0000) 2019-10-13T12:31:37.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:37.201+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.205658+0000) 2019-10-13T12:31:37.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:37.201+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.206483+0000) 2019-10-13T12:31:37.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:37.203+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.207089+0000) 2019-10-13T12:31:37.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:37.205+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.207408+0000) 2019-10-13T12:31:37.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:37.205+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.207764+0000) 2019-10-13T12:31:37.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:37.205+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.208191+0000) 2019-10-13T12:31:37.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:37.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:37.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:37.211+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.215706+0000) 2019-10-13T12:31:38.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:38.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.747750+0000) 2019-10-13T12:31:38.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:38.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.751144+0000) 2019-10-13T12:31:39.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:39.979+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:39.982429+0000) 2019-10-13T12:31:40.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:40.579+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:40.581667+0000) 2019-10-13T12:31:42.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:42.212+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.215984+0000) 2019-10-13T12:31:42.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:42.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.216337+0000) 2019-10-13T12:31:42.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:42.212+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.216605+0000) 2019-10-13T12:31:42.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:42.212+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.217239+0000) 2019-10-13T12:31:42.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:42.212+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.218576+0000) 2019-10-13T12:31:42.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:42.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.219205+0000) 2019-10-13T12:31:42.220 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:42.216+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.219493+0000) 2019-10-13T12:31:42.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:42.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149963 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:42.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:42.223+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.229351+0000) 2019-10-13T12:31:42.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:42.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.238822+0000) 2019-10-13T12:31:42.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:42.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:42.299087+0000) 2019-10-13T12:31:45.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:45.512+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:45.513632+0000) 2019-10-13T12:31:46.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:46.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:46.481855+0000) 2019-10-13T12:31:47.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:47.220+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.226332+0000) 2019-10-13T12:31:47.228 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:47.220+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.227044+0000) 2019-10-13T12:31:47.229 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:47.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.227423+0000) 2019-10-13T12:31:47.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:47.224+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.228214+0000) 2019-10-13T12:31:47.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:47.224+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.228792+0000) 2019-10-13T12:31:47.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:47.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.230068+0000) 2019-10-13T12:31:47.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:47.227+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.230613+0000) 2019-10-13T12:31:47.234 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:47.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 149995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:47.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:47.231+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.236560+0000) 2019-10-13T12:31:48.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:48.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.747968+0000) 2019-10-13T12:31:48.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:48.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.751340+0000) 2019-10-13T12:31:49.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:49.979+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:49.982647+0000) 2019-10-13T12:31:50.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:50.587+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:50.588108+0000) 2019-10-13T12:31:52.238 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:52.232+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.237340+0000) 2019-10-13T12:31:52.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:52.232+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.238335+0000) 2019-10-13T12:31:52.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:52.232+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.238906+0000) 2019-10-13T12:31:52.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:52.232+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.238939+0000) 2019-10-13T12:31:52.240 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:52.236+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.239432+0000) 2019-10-13T12:31:52.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:52.236+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.240014+0000) 2019-10-13T12:31:52.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:52.235+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.241684+0000) 2019-10-13T12:31:52.243 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:52.236+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.241841+0000) 2019-10-13T12:31:52.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:52.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:52.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:52.239+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.245985+0000) 2019-10-13T12:31:52.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:52.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:52.299326+0000) 2019-10-13T12:31:55.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:55.508+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:55.513827+0000) 2019-10-13T12:31:56.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:56.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:56.482105+0000) 2019-10-13T12:31:57.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:31:57.240+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.246667+0000) 2019-10-13T12:31:57.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:57.240+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.247255+0000) 2019-10-13T12:31:57.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:57.243+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.247756+0000) 2019-10-13T12:31:57.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:31:57.244+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.247969+0000) 2019-10-13T12:31:57.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:31:57.244+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.248589+0000) 2019-10-13T12:31:57.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:31:57.244+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.249567+0000) 2019-10-13T12:31:57.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:57.244+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.250111+0000) 2019-10-13T12:31:57.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:31:57.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:31:57.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:31:57.251+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.256992+0000) 2019-10-13T12:31:58.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:31:58.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.748149+0000) 2019-10-13T12:31:58.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:31:58.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.751569+0000) 2019-10-13T12:31:59.223 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:31:59.223 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:59.227 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:31:59.227 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:59.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:31:59.979+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:59.982857+0000) 2019-10-13T12:32:00.596 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:00.591+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:00.594780+0000) 2019-10-13T12:32:02.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:02.236+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.239092+0000) 2019-10-13T12:32:02.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:02.252+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.257876+0000) 2019-10-13T12:32:02.260 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:02.252+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.258082+0000) 2019-10-13T12:32:02.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:02.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.258452+0000) 2019-10-13T12:32:02.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:02.255+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.258885+0000) 2019-10-13T12:32:02.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:02.256+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.259933+0000) 2019-10-13T12:32:02.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:02.256+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.260391+0000) 2019-10-13T12:32:02.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:02.256+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.260681+0000) 2019-10-13T12:32:02.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:02.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:02.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:02.263+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.267472+0000) 2019-10-13T12:32:02.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:02.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:02.299506+0000) 2019-10-13T12:32:05.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:05.512+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:05.514041+0000) 2019-10-13T12:32:06.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:06.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:06.482299+0000) 2019-10-13T12:32:07.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:07.264+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.267490+0000) 2019-10-13T12:32:07.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:07.263+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.268261+0000) 2019-10-13T12:32:07.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:07.264+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.268792+0000) 2019-10-13T12:32:07.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:07.264+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.269231+0000) 2019-10-13T12:32:07.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:07.264+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.269693+0000) 2019-10-13T12:32:07.271 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:07.264+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.269911+0000) 2019-10-13T12:32:07.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:07.264+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.271117+0000) 2019-10-13T12:32:07.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:07.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150128 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:07.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:07.275+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.278395+0000) 2019-10-13T12:32:08.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:08.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.748382+0000) 2019-10-13T12:32:08.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:08.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.751780+0000) 2019-10-13T12:32:09.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:09.979+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:09.983006+0000) 2019-10-13T12:32:10.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:10.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:10.601210+0000) 2019-10-13T12:32:12.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:12.236+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.239243+0000) 2019-10-13T12:32:12.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:12.272+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.277958+0000) 2019-10-13T12:32:12.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:12.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.278993+0000) 2019-10-13T12:32:12.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:12.276+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.279375+0000) 2019-10-13T12:32:12.281 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:12.276+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.279642+0000) 2019-10-13T12:32:12.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:12.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.280559+0000) 2019-10-13T12:32:12.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:12.275+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.280735+0000) 2019-10-13T12:32:12.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:12.276+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.281272+0000) 2019-10-13T12:32:12.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:12.280+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150163 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:12.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:12.283+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.286437+0000) 2019-10-13T12:32:12.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:12.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:12.299663+0000) 2019-10-13T12:32:15.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:15.512+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:15.514278+0000) 2019-10-13T12:32:16.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:16.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:16.482502+0000) 2019-10-13T12:32:17.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:17.284+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.289290+0000) 2019-10-13T12:32:17.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:17.284+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.289814+0000) 2019-10-13T12:32:17.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:17.284+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.290696+0000) 2019-10-13T12:32:17.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:17.284+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.291262+0000) 2019-10-13T12:32:17.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:17.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.291904+0000) 2019-10-13T12:32:17.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:17.288+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.292596+0000) 2019-10-13T12:32:17.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:17.287+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.293219+0000) 2019-10-13T12:32:17.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:17.291+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.296478+0000) 2019-10-13T12:32:17.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:17.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150196 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:18.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:18.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.748573+0000) 2019-10-13T12:32:18.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:18.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.751970+0000) 2019-10-13T12:32:19.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:19.982+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:19.983199+0000) 2019-10-13T12:32:20.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:20.606+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:20.607642+0000) 2019-10-13T12:32:22.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:22.236+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.239528+0000) 2019-10-13T12:32:22.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:22.296+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.299755+0000) 2019-10-13T12:32:22.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:22.294+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.301185+0000) 2019-10-13T12:32:22.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:22.296+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.301854+0000) 2019-10-13T12:32:22.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:22.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.302272+0000) 2019-10-13T12:32:22.304 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:22.296+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.302876+0000) 2019-10-13T12:32:22.304 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:22.296+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.302620+0000) 2019-10-13T12:32:22.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:22.300+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.304116+0000) 2019-10-13T12:32:22.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:22.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:22.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:22.302+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:22.307604+0000) 2019-10-13T12:32:25.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:25.511+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:25.514464+0000) 2019-10-13T12:32:26.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:26.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:26.482692+0000) 2019-10-13T12:32:27.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:27.303+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.310705+0000) 2019-10-13T12:32:27.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:27.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.312206+0000) 2019-10-13T12:32:27.313 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:27.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.312415+0000) 2019-10-13T12:32:27.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:27.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.312887+0000) 2019-10-13T12:32:27.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:27.307+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.314067+0000) 2019-10-13T12:32:27.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:27.307+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.314691+0000) 2019-10-13T12:32:27.316 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:27.307+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.314902+0000) 2019-10-13T12:32:27.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:27.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:27.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:27.314+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.320177+0000) 2019-10-13T12:32:28.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:28.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.748767+0000) 2019-10-13T12:32:28.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:28.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.752160+0000) 2019-10-13T12:32:29.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:29.047+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:29.050337+0000) 2019-10-13T12:32:29.280 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:32:29.281 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:29.284 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:32:29.284 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:29.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:29.978+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:29.983407+0000) 2019-10-13T12:32:30.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:30.606+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:30.613957+0000) 2019-10-13T12:32:32.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:32.235+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.239729+0000) 2019-10-13T12:32:32.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:32.318+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.321738+0000) 2019-10-13T12:32:32.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:32.315+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.322661+0000) 2019-10-13T12:32:32.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:32.315+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.322944+0000) 2019-10-13T12:32:32.324 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:32.319+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.323580+0000) 2019-10-13T12:32:32.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:32.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.323893+0000) 2019-10-13T12:32:32.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:32.319+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.324245+0000) 2019-10-13T12:32:32.326 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:32.319+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.324426+0000) 2019-10-13T12:32:32.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:32.326+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:32.330474+0000) 2019-10-13T12:32:32.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:32.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150297 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:35.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:35.507+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:35.514656+0000) 2019-10-13T12:32:36.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:36.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:36.482890+0000) 2019-10-13T12:32:37.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:37.327+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.333800+0000) 2019-10-13T12:32:37.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:37.327+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.334272+0000) 2019-10-13T12:32:37.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:37.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.334788+0000) 2019-10-13T12:32:37.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:37.331+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.335416+0000) 2019-10-13T12:32:37.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:37.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.335223+0000) 2019-10-13T12:32:37.337 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:37.331+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.335787+0000) 2019-10-13T12:32:37.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:37.331+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.337169+0000) 2019-10-13T12:32:37.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:37.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:37.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:37.338+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.343169+0000) 2019-10-13T12:32:38.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:38.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.748961+0000) 2019-10-13T12:32:38.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:38.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.752389+0000) 2019-10-13T12:32:39.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:39.043+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:39.050574+0000) 2019-10-13T12:32:39.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:39.982+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:39.983609+0000) 2019-10-13T12:32:40.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:40.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:40.620464+0000) 2019-10-13T12:32:42.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:42.235+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.239976+0000) 2019-10-13T12:32:42.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:42.339+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.344973+0000) 2019-10-13T12:32:42.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:42.339+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.345366+0000) 2019-10-13T12:32:42.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:42.339+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.345705+0000) 2019-10-13T12:32:42.347 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:42.339+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.346439+0000) 2019-10-13T12:32:42.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:42.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.347404+0000) 2019-10-13T12:32:42.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:42.343+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.348048+0000) 2019-10-13T12:32:42.349 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:42.342+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.348212+0000) 2019-10-13T12:32:42.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:42.347+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:42.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:42.346+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:42.352645+0000) 2019-10-13T12:32:45.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:45.511+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:45.514853+0000) 2019-10-13T12:32:46.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:46.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:46.483144+0000) 2019-10-13T12:32:47.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:47.351+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.355564+0000) 2019-10-13T12:32:47.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:47.351+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.356374+0000) 2019-10-13T12:32:47.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:47.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.356808+0000) 2019-10-13T12:32:47.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:47.351+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.357352+0000) 2019-10-13T12:32:47.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:47.350+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.357551+0000) 2019-10-13T12:32:47.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:47.351+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.357870+0000) 2019-10-13T12:32:47.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:47.351+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.358672+0000) 2019-10-13T12:32:47.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:47.358+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.362340+0000) 2019-10-13T12:32:47.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:47.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:48.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:48.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.749166+0000) 2019-10-13T12:32:48.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:48.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.752582+0000) 2019-10-13T12:32:49.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:49.047+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:49.050802+0000) 2019-10-13T12:32:49.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:49.982+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:49.983802+0000) 2019-10-13T12:32:50.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:50.622+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:50.626947+0000) 2019-10-13T12:32:52.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:52.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.240175+0000) 2019-10-13T12:32:52.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:52.363+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.366452+0000) 2019-10-13T12:32:52.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:52.362+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.367217+0000) 2019-10-13T12:32:52.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:52.367+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.367531+0000) 2019-10-13T12:32:52.369 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:52.367+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.367838+0000) 2019-10-13T12:32:52.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:52.367+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.368768+0000) 2019-10-13T12:32:52.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:52.367+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.369100+0000) 2019-10-13T12:32:52.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:52.367+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.369441+0000) 2019-10-13T12:32:52.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:52.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150431 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:52.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:52.370+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:52.375555+0000) 2019-10-13T12:32:55.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:55.511+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:55.515055+0000) 2019-10-13T12:32:56.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:56.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:56.483367+0000) 2019-10-13T12:32:57.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:57.375+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.377570+0000) 2019-10-13T12:32:57.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:32:57.375+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.378464+0000) 2019-10-13T12:32:57.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:57.379+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.379390+0000) 2019-10-13T12:32:57.381 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:57.374+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.379517+0000) 2019-10-13T12:32:57.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:32:57.379+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.379986+0000) 2019-10-13T12:32:57.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:32:57.379+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.380425+0000) 2019-10-13T12:32:57.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:57.379+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.381043+0000) 2019-10-13T12:32:57.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:32:57.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:32:57.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:32:57.382+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.386342+0000) 2019-10-13T12:32:58.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:32:58.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.749390+0000) 2019-10-13T12:32:58.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:32:58.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.752811+0000) 2019-10-13T12:32:59.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:32:59.047+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:59.051014+0000) 2019-10-13T12:32:59.417 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:32:59.417 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:59.421 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:32:59.421 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:59.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:32:59.978+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:59.984036+0000) 2019-10-13T12:33:00.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:00.630+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:00.633398+0000) 2019-10-13T12:33:02.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:02.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.240372+0000) 2019-10-13T12:33:02.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:02.386+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.388882+0000) 2019-10-13T12:33:02.391 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:02.387+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.389757+0000) 2019-10-13T12:33:02.391 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:02.387+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.390198+0000) 2019-10-13T12:33:02.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:02.387+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.390275+0000) 2019-10-13T12:33:02.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:02.391+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.391617+0000) 2019-10-13T12:33:02.393 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:02.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.392059+0000) 2019-10-13T12:33:02.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:02.391+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.392747+0000) 2019-10-13T12:33:02.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:02.390+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:02.397210+0000) 2019-10-13T12:33:02.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:02.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:05.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:05.510+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:05.515284+0000) 2019-10-13T12:33:06.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:06.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:06.483584+0000) 2019-10-13T12:33:07.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:07.398+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.400290+0000) 2019-10-13T12:33:07.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:07.398+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.401698+0000) 2019-10-13T12:33:07.403 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:07.398+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.401972+0000) 2019-10-13T12:33:07.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:07.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.403121+0000) 2019-10-13T12:33:07.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:07.402+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.403648+0000) 2019-10-13T12:33:07.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:07.402+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.404351+0000) 2019-10-13T12:33:07.406 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:07.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.405000+0000) 2019-10-13T12:33:07.408 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:07.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:07.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:07.406+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.408700+0000) 2019-10-13T12:33:08.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:08.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.749590+0000) 2019-10-13T12:33:08.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:08.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.753011+0000) 2019-10-13T12:33:09.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:09.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:09.051256+0000) 2019-10-13T12:33:09.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:09.982+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:09.984235+0000) 2019-10-13T12:33:10.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:10.638+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:10.639790+0000) 2019-10-13T12:33:12.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:12.238+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.240606+0000) 2019-10-13T12:33:12.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:12.406+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.410674+0000) 2019-10-13T12:33:12.413 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:12.410+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.411084+0000) 2019-10-13T12:33:12.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:12.410+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.411291+0000) 2019-10-13T12:33:12.414 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:12.410+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.412417+0000) 2019-10-13T12:33:12.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:12.410+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.412793+0000) 2019-10-13T12:33:12.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:12.410+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.413428+0000) 2019-10-13T12:33:12.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:12.410+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.414606+0000) 2019-10-13T12:33:12.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:12.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:12.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:12.418+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:12.419342+0000) 2019-10-13T12:33:15.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:15.514+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:15.515558+0000) 2019-10-13T12:33:16.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:16.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:16.483788+0000) 2019-10-13T12:33:17.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:17.418+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.421828+0000) 2019-10-13T12:33:17.423 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:17.418+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.422594+0000) 2019-10-13T12:33:17.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:17.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.422995+0000) 2019-10-13T12:33:17.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:17.422+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.423876+0000) 2019-10-13T12:33:17.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:17.421+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.424607+0000) 2019-10-13T12:33:17.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:17.422+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.425108+0000) 2019-10-13T12:33:17.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:17.422+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.425604+0000) 2019-10-13T12:33:17.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:17.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:17.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:17.429+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.430491+0000) 2019-10-13T12:33:18.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:18.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.749784+0000) 2019-10-13T12:33:18.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:18.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.753236+0000) 2019-10-13T12:33:19.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:19.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:19.051534+0000) 2019-10-13T12:33:19.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:19.981+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:19.984436+0000) 2019-10-13T12:33:20.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:20.645+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:20.646356+0000) 2019-10-13T12:33:22.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:22.238+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.240823+0000) 2019-10-13T12:33:22.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:22.430+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.432898+0000) 2019-10-13T12:33:22.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:22.430+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.433700+0000) 2019-10-13T12:33:22.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:22.430+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.433963+0000) 2019-10-13T12:33:22.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:22.430+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.434269+0000) 2019-10-13T12:33:22.436 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:22.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.435001+0000) 2019-10-13T12:33:22.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:22.433+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.436335+0000) 2019-10-13T12:33:22.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:22.434+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.436652+0000) 2019-10-13T12:33:22.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:22.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:22.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:22.441+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:22.442361+0000) 2019-10-13T12:33:25.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:25.514+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:25.515767+0000) 2019-10-13T12:33:26.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:26.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:26.484014+0000) 2019-10-13T12:33:27.444 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:27.438+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.443087+0000) 2019-10-13T12:33:27.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:27.442+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.443508+0000) 2019-10-13T12:33:27.446 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:27.442+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.444849+0000) 2019-10-13T12:33:27.446 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:27.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.445093+0000) 2019-10-13T12:33:27.447 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:27.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.446058+0000) 2019-10-13T12:33:27.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:27.442+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.446273+0000) 2019-10-13T12:33:27.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:27.445+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.446607+0000) 2019-10-13T12:33:27.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:27.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150664 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:27.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:27.449+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.452558+0000) 2019-10-13T12:33:28.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:28.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.749975+0000) 2019-10-13T12:33:28.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:28.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.753429+0000) 2019-10-13T12:33:29.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:29.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:29.051770+0000) 2019-10-13T12:33:29.555 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:33:29.555 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:29.558 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:33:29.559 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:29.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:29.981+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:29.984632+0000) 2019-10-13T12:33:30.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:30.657+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:30.658186+0000) 2019-10-13T12:33:32.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:32.238+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.241017+0000) 2019-10-13T12:33:32.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:32.450+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.454838+0000) 2019-10-13T12:33:32.456 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:32.454+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.455514+0000) 2019-10-13T12:33:32.457 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:32.454+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.455537+0000) 2019-10-13T12:33:32.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:32.454+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.456109+0000) 2019-10-13T12:33:32.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:32.454+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.457283+0000) 2019-10-13T12:33:32.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:32.454+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.457743+0000) 2019-10-13T12:33:32.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:32.457+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.458417+0000) 2019-10-13T12:33:32.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:32.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:32.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:32.461+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:32.463398+0000) 2019-10-13T12:33:35.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:35.514+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:35.515969+0000) 2019-10-13T12:33:36.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:36.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:36.484211+0000) 2019-10-13T12:33:37.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:37.462+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.465818+0000) 2019-10-13T12:33:37.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:37.462+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.466229+0000) 2019-10-13T12:33:37.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:37.466+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.467391+0000) 2019-10-13T12:33:37.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:37.466+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.467468+0000) 2019-10-13T12:33:37.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:37.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.467813+0000) 2019-10-13T12:33:37.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:37.466+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.468787+0000) 2019-10-13T12:33:37.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:37.466+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.469033+0000) 2019-10-13T12:33:37.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:37.470+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:37.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:37.473+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.474636+0000) 2019-10-13T12:33:38.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:38.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.750200+0000) 2019-10-13T12:33:38.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:38.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.753656+0000) 2019-10-13T12:33:39.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:39.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:39.051972+0000) 2019-10-13T12:33:39.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:39.981+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:39.984828+0000) 2019-10-13T12:33:40.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:40.661+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:40.664608+0000) 2019-10-13T12:33:42.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:42.238+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.241229+0000) 2019-10-13T12:33:42.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:42.474+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.476944+0000) 2019-10-13T12:33:42.478 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:42.474+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.477334+0000) 2019-10-13T12:33:42.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:42.473+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.477761+0000) 2019-10-13T12:33:42.480 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:42.474+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.478115+0000) 2019-10-13T12:33:42.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:42.474+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.478589+0000) 2019-10-13T12:33:42.480 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:42.474+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.478947+0000) 2019-10-13T12:33:42.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:42.478+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.479895+0000) 2019-10-13T12:33:42.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:42.482+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:42.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:42.485+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:42.486672+0000) 2019-10-13T12:33:45.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:45.514+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:45.516195+0000) 2019-10-13T12:33:46.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:46.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:46.484403+0000) 2019-10-13T12:33:47.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:47.486+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.488007+0000) 2019-10-13T12:33:47.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:47.486+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.488735+0000) 2019-10-13T12:33:47.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:47.486+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.489276+0000) 2019-10-13T12:33:47.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:47.486+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.489762+0000) 2019-10-13T12:33:47.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:47.486+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.490235+0000) 2019-10-13T12:33:47.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:47.486+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.490910+0000) 2019-10-13T12:33:47.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:47.489+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.491594+0000) 2019-10-13T12:33:47.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:47.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150798 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:47.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:47.497+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.499225+0000) 2019-10-13T12:33:48.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:48.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.750399+0000) 2019-10-13T12:33:48.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:48.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.753850+0000) 2019-10-13T12:33:49.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:49.049+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:49.052200+0000) 2019-10-13T12:33:49.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:49.981+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:49.984980+0000) 2019-10-13T12:33:50.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:50.673+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:50.675300+0000) 2019-10-13T12:33:52.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:52.237+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.241457+0000) 2019-10-13T12:33:52.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:52.493+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.498270+0000) 2019-10-13T12:33:52.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:52.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.498717+0000) 2019-10-13T12:33:52.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:52.497+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.499485+0000) 2019-10-13T12:33:52.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:52.497+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.499771+0000) 2019-10-13T12:33:52.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:52.497+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.500965+0000) 2019-10-13T12:33:52.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:52.497+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.501751+0000) 2019-10-13T12:33:52.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:52.497+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.501984+0000) 2019-10-13T12:33:52.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:52.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150832 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:52.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:52.505+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:52.507237+0000) 2019-10-13T12:33:55.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:55.513+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:55.516404+0000) 2019-10-13T12:33:56.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:56.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:56.484597+0000) 2019-10-13T12:33:57.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:57.505+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.508901+0000) 2019-10-13T12:33:57.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:33:57.505+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.509222+0000) 2019-10-13T12:33:57.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:57.505+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.509707+0000) 2019-10-13T12:33:57.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:57.509+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.510599+0000) 2019-10-13T12:33:57.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:33:57.509+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.511361+0000) 2019-10-13T12:33:57.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:57.509+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.511829+0000) 2019-10-13T12:33:57.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:33:57.509+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.511951+0000) 2019-10-13T12:33:57.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:33:57.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:33:57.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:33:57.513+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.516978+0000) 2019-10-13T12:33:58.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:33:58.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.750629+0000) 2019-10-13T12:33:58.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:33:58.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.754048+0000) 2019-10-13T12:33:59.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:33:59.049+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:59.052413+0000) 2019-10-13T12:33:59.681 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:33:59.681 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:59.685 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:33:59.685 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:59.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:33:59.981+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:59.985188+0000) 2019-10-13T12:34:00.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:00.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:00.681719+0000) 2019-10-13T12:34:02.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:02.237+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.241672+0000) 2019-10-13T12:34:02.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:02.517+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.519661+0000) 2019-10-13T12:34:02.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:02.517+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.520520+0000) 2019-10-13T12:34:02.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:02.517+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.520784+0000) 2019-10-13T12:34:02.522 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:02.517+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.521050+0000) 2019-10-13T12:34:02.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:02.517+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.521443+0000) 2019-10-13T12:34:02.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:02.517+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.521869+0000) 2019-10-13T12:34:02.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:02.517+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.522220+0000) 2019-10-13T12:34:02.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:02.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:02.529 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:02.525+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:02.528654+0000) 2019-10-13T12:34:05.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:05.513+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:05.516631+0000) 2019-10-13T12:34:06.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:06.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:06.484799+0000) 2019-10-13T12:34:07.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:07.525+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.529075+0000) 2019-10-13T12:34:07.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:07.525+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.529462+0000) 2019-10-13T12:34:07.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:07.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.529949+0000) 2019-10-13T12:34:07.532 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:07.525+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.530078+0000) 2019-10-13T12:34:07.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:07.525+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.531269+0000) 2019-10-13T12:34:07.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:07.529+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.531666+0000) 2019-10-13T12:34:07.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:07.529+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.532075+0000) 2019-10-13T12:34:07.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:07.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150933 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:07.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:07.537+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.538580+0000) 2019-10-13T12:34:08.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:08.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.750854+0000) 2019-10-13T12:34:08.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:08.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.754241+0000) 2019-10-13T12:34:09.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:09.049+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:09.052620+0000) 2019-10-13T12:34:09.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:09.984+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:09.985384+0000) 2019-10-13T12:34:10.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:10.688+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:10.691270+0000) 2019-10-13T12:34:12.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:12.241+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.241873+0000) 2019-10-13T12:34:12.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:12.533+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.538829+0000) 2019-10-13T12:34:12.541 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:12.537+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.540672+0000) 2019-10-13T12:34:12.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:12.537+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.541242+0000) 2019-10-13T12:34:12.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:12.536+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.541727+0000) 2019-10-13T12:34:12.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:12.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.542267+0000) 2019-10-13T12:34:12.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:12.537+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.542632+0000) 2019-10-13T12:34:12.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:12.537+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.543075+0000) 2019-10-13T12:34:12.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:12.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:12.547243+0000) 2019-10-13T12:34:12.548 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:12.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 150968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:15.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:15.513+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:15.516836+0000) 2019-10-13T12:34:16.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:16.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:16.484991+0000) 2019-10-13T12:34:17.552 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:17.545+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.550998+0000) 2019-10-13T12:34:17.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:17.548+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.551082+0000) 2019-10-13T12:34:17.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:17.549+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.552025+0000) 2019-10-13T12:34:17.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:17.549+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.552423+0000) 2019-10-13T12:34:17.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:17.549+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.553030+0000) 2019-10-13T12:34:17.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:17.549+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.553486+0000) 2019-10-13T12:34:17.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:17.549+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.554199+0000) 2019-10-13T12:34:17.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:17.552+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.557836+0000) 2019-10-13T12:34:17.559 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:17.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:18.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:18.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.751069+0000) 2019-10-13T12:34:18.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:18.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.754434+0000) 2019-10-13T12:34:19.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:19.049+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:19.052774+0000) 2019-10-13T12:34:19.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:19.980+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:19.985607+0000) 2019-10-13T12:34:20.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:20.696+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:20.697648+0000) 2019-10-13T12:34:22.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:22.237+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.242094+0000) 2019-10-13T12:34:22.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:22.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.562962+0000) 2019-10-13T12:34:22.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:22.557+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.563066+0000) 2019-10-13T12:34:22.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:22.561+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.563503+0000) 2019-10-13T12:34:22.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:22.561+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.563937+0000) 2019-10-13T12:34:22.566 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:22.561+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.564483+0000) 2019-10-13T12:34:22.566 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:22.561+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.564636+0000) 2019-10-13T12:34:22.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:22.561+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.565622+0000) 2019-10-13T12:34:22.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:22.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:22.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:22.568+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:22.570963+0000) 2019-10-13T12:34:25.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:25.513+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:25.517031+0000) 2019-10-13T12:34:26.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:26.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:26.485192+0000) 2019-10-13T12:34:27.574 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:27.569+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.572542+0000) 2019-10-13T12:34:27.574 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:27.568+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.573090+0000) 2019-10-13T12:34:27.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:27.569+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.573589+0000) 2019-10-13T12:34:27.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:27.569+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.574474+0000) 2019-10-13T12:34:27.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:27.569+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.574888+0000) 2019-10-13T12:34:27.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:27.569+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.575245+0000) 2019-10-13T12:34:27.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:27.573+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.575639+0000) 2019-10-13T12:34:27.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:27.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:27.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:27.580+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.582988+0000) 2019-10-13T12:34:28.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:28.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.751265+0000) 2019-10-13T12:34:28.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:28.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.754654+0000) 2019-10-13T12:34:29.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:29.049+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:29.052982+0000) 2019-10-13T12:34:29.792 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:34:29.792 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:29.796 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:34:29.796 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:29.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:29.980+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:29.985825+0000) 2019-10-13T12:34:30.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:30.700+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:30.704156+0000) 2019-10-13T12:34:32.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:32.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.242294+0000) 2019-10-13T12:34:32.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:32.580+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.583726+0000) 2019-10-13T12:34:32.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:32.580+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.584318+0000) 2019-10-13T12:34:32.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:32.580+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.584415+0000) 2019-10-13T12:34:32.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:32.580+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.585693+0000) 2019-10-13T12:34:32.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:32.580+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.586039+0000) 2019-10-13T12:34:32.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:32.580+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.586320+0000) 2019-10-13T12:34:32.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:32.584+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.587475+0000) 2019-10-13T12:34:32.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:32.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:32.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:32.588+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:32.594071+0000) 2019-10-13T12:34:35.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:35.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:35.517235+0000) 2019-10-13T12:34:36.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:36.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:36.485377+0000) 2019-10-13T12:34:37.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:37.588+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.595094+0000) 2019-10-13T12:34:37.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:37.592+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.595801+0000) 2019-10-13T12:34:37.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:37.592+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.597439+0000) 2019-10-13T12:34:37.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:37.592+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.597894+0000) 2019-10-13T12:34:37.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:37.592+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.598416+0000) 2019-10-13T12:34:37.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:37.592+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.598715+0000) 2019-10-13T12:34:37.600 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:37.592+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.599146+0000) 2019-10-13T12:34:37.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:37.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:37.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:37.600+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.603285+0000) 2019-10-13T12:34:38.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:38.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.751495+0000) 2019-10-13T12:34:38.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:38.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.754848+0000) 2019-10-13T12:34:39.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:39.048+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.053207+0000) 2019-10-13T12:34:39.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:39.984+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.986024+0000) 2019-10-13T12:34:40.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:40.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:40.710489+0000) 2019-10-13T12:34:42.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:42.236+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.242489+0000) 2019-10-13T12:34:42.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:42.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.607130+0000) 2019-10-13T12:34:42.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:42.604+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.608215+0000) 2019-10-13T12:34:42.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:42.604+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.608445+0000) 2019-10-13T12:34:42.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:42.604+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.608862+0000) 2019-10-13T12:34:42.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:42.604+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.609329+0000) 2019-10-13T12:34:42.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:42.604+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.609625+0000) 2019-10-13T12:34:42.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:42.604+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.610504+0000) 2019-10-13T12:34:42.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:42.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:42.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:42.612+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:42.614410+0000) 2019-10-13T12:34:45.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:45.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:45.517466+0000) 2019-10-13T12:34:46.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:46.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:46.485581+0000) 2019-10-13T12:34:47.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:47.612+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.616940+0000) 2019-10-13T12:34:47.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:47.612+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.617435+0000) 2019-10-13T12:34:47.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:47.612+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.617934+0000) 2019-10-13T12:34:47.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:47.612+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.618471+0000) 2019-10-13T12:34:47.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:47.616+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.619514+0000) 2019-10-13T12:34:47.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:47.616+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.619933+0000) 2019-10-13T12:34:47.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:47.616+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.619491+0000) 2019-10-13T12:34:47.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:47.620+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.623789+0000) 2019-10-13T12:34:47.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:47.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:48.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:48.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:48.751720+0000) 2019-10-13T12:34:48.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:48.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:48.755034+0000) 2019-10-13T12:34:49.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:49.052+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.053443+0000) 2019-10-13T12:34:49.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:49.984+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.986234+0000) 2019-10-13T12:34:50.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:50.720+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:50.721945+0000) 2019-10-13T12:34:52.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:52.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.242690+0000) 2019-10-13T12:34:52.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:52.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.628555+0000) 2019-10-13T12:34:52.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:52.624+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.628897+0000) 2019-10-13T12:34:52.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:52.624+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.629184+0000) 2019-10-13T12:34:52.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:52.628+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.630513+0000) 2019-10-13T12:34:52.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:52.624+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.630889+0000) 2019-10-13T12:34:52.632 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:52.628+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.631541+0000) 2019-10-13T12:34:52.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:52.628+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.631658+0000) 2019-10-13T12:34:52.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:52.632+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:52.634557+0000) 2019-10-13T12:34:52.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:52.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:55.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:55.512+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:55.517657+0000) 2019-10-13T12:34:56.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:56.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:56.485782+0000) 2019-10-13T12:34:57.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:34:57.632+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.638925+0000) 2019-10-13T12:34:57.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:57.636+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.639415+0000) 2019-10-13T12:34:57.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:57.636+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.639953+0000) 2019-10-13T12:34:57.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:34:57.636+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.640513+0000) 2019-10-13T12:34:57.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:57.640+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.642621+0000) 2019-10-13T12:34:57.644 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:57.636+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.643148+0000) 2019-10-13T12:34:57.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:34:57.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.643241+0000) 2019-10-13T12:34:57.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:34:57.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:34:57.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:34:57.644+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.647912+0000) 2019-10-13T12:34:58.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:34:58.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:58.751918+0000) 2019-10-13T12:34:58.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:34:58.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:58.755243+0000) 2019-10-13T12:34:59.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:34:59.052+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.053648+0000) 2019-10-13T12:34:59.886 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:34:59.886 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:59.889 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:34:59.890 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:59.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:34:59.984+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.986485+0000) 2019-10-13T12:35:00.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:00.728+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:00.728354+0000) 2019-10-13T12:35:02.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:02.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.242906+0000) 2019-10-13T12:35:02.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:02.644+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.649621+0000) 2019-10-13T12:35:02.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:02.647+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.650667+0000) 2019-10-13T12:35:02.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:02.644+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.650970+0000) 2019-10-13T12:35:02.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:02.648+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.651614+0000) 2019-10-13T12:35:02.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:02.648+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.652268+0000) 2019-10-13T12:35:02.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:02.648+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.652378+0000) 2019-10-13T12:35:02.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:02.648+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.652865+0000) 2019-10-13T12:35:02.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:02.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:02.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:02.655+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:02.661815+0000) 2019-10-13T12:35:05.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:05.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:05.517871+0000) 2019-10-13T12:35:06.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:06.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:06.485999+0000) 2019-10-13T12:35:07.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:07.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.660153+0000) 2019-10-13T12:35:07.662 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:07.656+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.660735+0000) 2019-10-13T12:35:07.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:07.655+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.661944+0000) 2019-10-13T12:35:07.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:07.656+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.662164+0000) 2019-10-13T12:35:07.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:07.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.662756+0000) 2019-10-13T12:35:07.664 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:07.660+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.663347+0000) 2019-10-13T12:35:07.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:07.660+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.663751+0000) 2019-10-13T12:35:07.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:07.663+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.667960+0000) 2019-10-13T12:35:07.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:07.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:08.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:08.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:08.752131+0000) 2019-10-13T12:35:08.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:08.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:08.755462+0000) 2019-10-13T12:35:09.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:09.048+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.053864+0000) 2019-10-13T12:35:09.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:09.983+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.986683+0000) 2019-10-13T12:35:10.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:10.735+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:10.739799+0000) 2019-10-13T12:35:12.244 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:12.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.243110+0000) 2019-10-13T12:35:12.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:12.668+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.671646+0000) 2019-10-13T12:35:12.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:12.668+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.673044+0000) 2019-10-13T12:35:12.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:12.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.673631+0000) 2019-10-13T12:35:12.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:12.668+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.673386+0000) 2019-10-13T12:35:12.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:12.668+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.673858+0000) 2019-10-13T12:35:12.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:12.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.674252+0000) 2019-10-13T12:35:12.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:12.668+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.674836+0000) 2019-10-13T12:35:12.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:12.675+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:12.678300+0000) 2019-10-13T12:35:12.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:12.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:15.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:15.511+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:15.518093+0000) 2019-10-13T12:35:16.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:16.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:16.486238+0000) 2019-10-13T12:35:17.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:17.675+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.682709+0000) 2019-10-13T12:35:17.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:17.679+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.683543+0000) 2019-10-13T12:35:17.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:17.679+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.683939+0000) 2019-10-13T12:35:17.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:17.679+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.684641+0000) 2019-10-13T12:35:17.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:17.679+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.685253+0000) 2019-10-13T12:35:17.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:17.679+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.685640+0000) 2019-10-13T12:35:17.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:17.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.686104+0000) 2019-10-13T12:35:17.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:17.683+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151415 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:17.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:17.687+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.691782+0000) 2019-10-13T12:35:18.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:18.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:18.752332+0000) 2019-10-13T12:35:18.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:18.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:18.755666+0000) 2019-10-13T12:35:19.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:19.051+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.054099+0000) 2019-10-13T12:35:19.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:19.983+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.986878+0000) 2019-10-13T12:35:20.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:20.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:20.746172+0000) 2019-10-13T12:35:22.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:22.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.243340+0000) 2019-10-13T12:35:22.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:22.687+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.692724+0000) 2019-10-13T12:35:22.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:22.687+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.693250+0000) 2019-10-13T12:35:22.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:22.687+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.693635+0000) 2019-10-13T12:35:22.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:22.687+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.693980+0000) 2019-10-13T12:35:22.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:22.687+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.694867+0000) 2019-10-13T12:35:22.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:22.691+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.696785+0000) 2019-10-13T12:35:22.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:22.691+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.696330+0000) 2019-10-13T12:35:22.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:22.695+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:22.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:22.695+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:22.701549+0000) 2019-10-13T12:35:25.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:25.515+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:25.518291+0000) 2019-10-13T12:35:26.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:26.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:26.486485+0000) 2019-10-13T12:35:27.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:27.699+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.703015+0000) 2019-10-13T12:35:27.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:27.699+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.704411+0000) 2019-10-13T12:35:27.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:27.699+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.704972+0000) 2019-10-13T12:35:27.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:27.699+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.705336+0000) 2019-10-13T12:35:27.707 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:27.699+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.706616+0000) 2019-10-13T12:35:27.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:27.699+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.706923+0000) 2019-10-13T12:35:27.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:27.703+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.707608+0000) 2019-10-13T12:35:27.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:27.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:27.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:27.707+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.712027+0000) 2019-10-13T12:35:28.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:28.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:28.752530+0000) 2019-10-13T12:35:28.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:28.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:28.755866+0000) 2019-10-13T12:35:29.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:29.051+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.054250+0000) 2019-10-13T12:35:29.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:29.983+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.987072+0000) 2019-10-13T12:35:29.997 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:35:29.997 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:30.006 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:35:30.006 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:30.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:30.751+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:30.752561+0000) 2019-10-13T12:35:32.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:32.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.243542+0000) 2019-10-13T12:35:32.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:32.707+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.713984+0000) 2019-10-13T12:35:32.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:32.707+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.714427+0000) 2019-10-13T12:35:32.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:32.707+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.715049+0000) 2019-10-13T12:35:32.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:32.711+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.715709+0000) 2019-10-13T12:35:32.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:32.711+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.716280+0000) 2019-10-13T12:35:32.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:32.711+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.717218+0000) 2019-10-13T12:35:32.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:32.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.717806+0000) 2019-10-13T12:35:32.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:32.719+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:32.722354+0000) 2019-10-13T12:35:32.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:32.715+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:35.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:35.515+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:35.518492+0000) 2019-10-13T12:35:36.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:36.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:36.486677+0000) 2019-10-13T12:35:37.727 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:37.719+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.725642+0000) 2019-10-13T12:35:37.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:37.719+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.725972+0000) 2019-10-13T12:35:37.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:37.719+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.726304+0000) 2019-10-13T12:35:37.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:37.719+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.726750+0000) 2019-10-13T12:35:37.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:37.723+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.728652+0000) 2019-10-13T12:35:37.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:37.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.728843+0000) 2019-10-13T12:35:37.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:37.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.729648+0000) 2019-10-13T12:35:37.735 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:37.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:37.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:37.731+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.734170+0000) 2019-10-13T12:35:38.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:38.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:38.752754+0000) 2019-10-13T12:35:38.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:38.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:38.756063+0000) 2019-10-13T12:35:39.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:39.047+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.054458+0000) 2019-10-13T12:35:39.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:39.983+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.987261+0000) 2019-10-13T12:35:40.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:40.763+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:40.764398+0000) 2019-10-13T12:35:42.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:42.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.243771+0000) 2019-10-13T12:35:42.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:42.731+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.737519+0000) 2019-10-13T12:35:42.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:42.731+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.738404+0000) 2019-10-13T12:35:42.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:42.735+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.739489+0000) 2019-10-13T12:35:42.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:42.735+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.739605+0000) 2019-10-13T12:35:42.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:42.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.740041+0000) 2019-10-13T12:35:42.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:42.735+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.740633+0000) 2019-10-13T12:35:42.742 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:42.735+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.740844+0000) 2019-10-13T12:35:42.745 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:42.739+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:42.743890+0000) 2019-10-13T12:35:42.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:42.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:45.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:45.515+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:45.518737+0000) 2019-10-13T12:35:46.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:46.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:46.486869+0000) 2019-10-13T12:35:47.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:47.747+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.748255+0000) 2019-10-13T12:35:47.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:47.743+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.748823+0000) 2019-10-13T12:35:47.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:47.747+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.749222+0000) 2019-10-13T12:35:47.751 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:47.747+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.749911+0000) 2019-10-13T12:35:47.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:47.747+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.750162+0000) 2019-10-13T12:35:47.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:47.747+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.750748+0000) 2019-10-13T12:35:47.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:47.747+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.751017+0000) 2019-10-13T12:35:47.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:47.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:47.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:47.755+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.759818+0000) 2019-10-13T12:35:48.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:48.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:48.752982+0000) 2019-10-13T12:35:48.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:48.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:48.756288+0000) 2019-10-13T12:35:49.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:49.051+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.054690+0000) 2019-10-13T12:35:49.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:49.987+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.987455+0000) 2019-10-13T12:35:50.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:50.767+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:50.770794+0000) 2019-10-13T12:35:52.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:52.243+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.243971+0000) 2019-10-13T12:35:52.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:52.755+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.758996+0000) 2019-10-13T12:35:52.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:52.759+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.759380+0000) 2019-10-13T12:35:52.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:52.759+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.759789+0000) 2019-10-13T12:35:52.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:52.759+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.760566+0000) 2019-10-13T12:35:52.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:52.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.761371+0000) 2019-10-13T12:35:52.763 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:52.755+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.761852+0000) 2019-10-13T12:35:52.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:52.759+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.762974+0000) 2019-10-13T12:35:52.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:52.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:52.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:52.763+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:52.767240+0000) 2019-10-13T12:35:55.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:55.515+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:55.518996+0000) 2019-10-13T12:35:56.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:56.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:56.487098+0000) 2019-10-13T12:35:57.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:35:57.766+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.769682+0000) 2019-10-13T12:35:57.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:35:57.766+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.770170+0000) 2019-10-13T12:35:57.772 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:35:57.766+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.770760+0000) 2019-10-13T12:35:57.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:57.766+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.771092+0000) 2019-10-13T12:35:57.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:57.770+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.771354+0000) 2019-10-13T12:35:57.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:57.770+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.771960+0000) 2019-10-13T12:35:57.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:57.770+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.773110+0000) 2019-10-13T12:35:57.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:35:57.774+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:35:57.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:35:57.778+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.782815+0000) 2019-10-13T12:35:58.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:35:58.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:58.753184+0000) 2019-10-13T12:35:58.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:35:58.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:58.756485+0000) 2019-10-13T12:35:59.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:35:59.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.054935+0000) 2019-10-13T12:35:59.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:35:59.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.987709+0000) 2019-10-13T12:36:00.267 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:36:00.267 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:00.271 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:36:00.271 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:00.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:00.774+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:00.777339+0000) 2019-10-13T12:36:02.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:02.242+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.244167+0000) 2019-10-13T12:36:02.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:02.778+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.780303+0000) 2019-10-13T12:36:02.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:02.774+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.780629+0000) 2019-10-13T12:36:02.783 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:02.778+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.781172+0000) 2019-10-13T12:36:02.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:02.778+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.781487+0000) 2019-10-13T12:36:02.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:02.778+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.781727+0000) 2019-10-13T12:36:02.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:02.778+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.782669+0000) 2019-10-13T12:36:02.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:02.778+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.783132+0000) 2019-10-13T12:36:02.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:02.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:02.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:02.782+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:02.788847+0000) 2019-10-13T12:36:05.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:05.514+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:05.519218+0000) 2019-10-13T12:36:06.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:06.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:06.487324+0000) 2019-10-13T12:36:07.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:07.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.790364+0000) 2019-10-13T12:36:07.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:07.790+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.791642+0000) 2019-10-13T12:36:07.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:07.790+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.792511+0000) 2019-10-13T12:36:07.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:07.790+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.792749+0000) 2019-10-13T12:36:07.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:07.786+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.793381+0000) 2019-10-13T12:36:07.795 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:07.790+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.793779+0000) 2019-10-13T12:36:07.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:07.790+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.794142+0000) 2019-10-13T12:36:07.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:07.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:07.801 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:07.794+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.799920+0000) 2019-10-13T12:36:08.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:08.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:08.753361+0000) 2019-10-13T12:36:08.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:08.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:08.756680+0000) 2019-10-13T12:36:09.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:09.050+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.055147+0000) 2019-10-13T12:36:09.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:09.982+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.987917+0000) 2019-10-13T12:36:10.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:10.782+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:10.783764+0000) 2019-10-13T12:36:12.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:12.242+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.244382+0000) 2019-10-13T12:36:12.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:12.798+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.801732+0000) 2019-10-13T12:36:12.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:12.798+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.802986+0000) 2019-10-13T12:36:12.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:12.798+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.803249+0000) 2019-10-13T12:36:12.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:12.802+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.804314+0000) 2019-10-13T12:36:12.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:12.802+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.804799+0000) 2019-10-13T12:36:12.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:12.798+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.805086+0000) 2019-10-13T12:36:12.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:12.802+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.805415+0000) 2019-10-13T12:36:12.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:12.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:12.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:12.806+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:12.811948+0000) 2019-10-13T12:36:15.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:15.518+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:15.519432+0000) 2019-10-13T12:36:16.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:16.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:16.487556+0000) 2019-10-13T12:36:17.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:17.810+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.813372+0000) 2019-10-13T12:36:17.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:17.810+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.814066+0000) 2019-10-13T12:36:17.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:17.810+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.814321+0000) 2019-10-13T12:36:17.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:17.810+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.815199+0000) 2019-10-13T12:36:17.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:17.814+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.816149+0000) 2019-10-13T12:36:17.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:17.810+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.816301+0000) 2019-10-13T12:36:17.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:17.814+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.816508+0000) 2019-10-13T12:36:17.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:17.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151827 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:17.825 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:17.818+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.824176+0000) 2019-10-13T12:36:18.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:18.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:18.753554+0000) 2019-10-13T12:36:18.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:18.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:18.756887+0000) 2019-10-13T12:36:19.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:19.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.055365+0000) 2019-10-13T12:36:19.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:19.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.988107+0000) 2019-10-13T12:36:20.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:20.786+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:20.790185+0000) 2019-10-13T12:36:22.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:22.242+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.244602+0000) 2019-10-13T12:36:22.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:22.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.825826+0000) 2019-10-13T12:36:22.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:22.826+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.827298+0000) 2019-10-13T12:36:22.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:22.826+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.827782+0000) 2019-10-13T12:36:22.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:22.826+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.828143+0000) 2019-10-13T12:36:22.830 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:22.822+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.828379+0000) 2019-10-13T12:36:22.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:22.826+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.828871+0000) 2019-10-13T12:36:22.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:22.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.829114+0000) 2019-10-13T12:36:22.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:22.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:22.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:22.826+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:22.833759+0000) 2019-10-13T12:36:25.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:25.518+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:25.519629+0000) 2019-10-13T12:36:26.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:26.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:26.487749+0000) 2019-10-13T12:36:27.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:27.834+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.837368+0000) 2019-10-13T12:36:27.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:27.834+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.837048+0000) 2019-10-13T12:36:27.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:27.834+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.837703+0000) 2019-10-13T12:36:27.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:27.834+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.838384+0000) 2019-10-13T12:36:27.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:27.834+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.838345+0000) 2019-10-13T12:36:27.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:27.834+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.838890+0000) 2019-10-13T12:36:27.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:27.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.839831+0000) 2019-10-13T12:36:27.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:27.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:27.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:27.838+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.845891+0000) 2019-10-13T12:36:28.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:28.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:28.753734+0000) 2019-10-13T12:36:28.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:28.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:28.757066+0000) 2019-10-13T12:36:29.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:29.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.055582+0000) 2019-10-13T12:36:30.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:29.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.988302+0000) 2019-10-13T12:36:30.344 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:36:30.344 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:30.373 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:36:30.374 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:30.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:30.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:30.796840+0000) 2019-10-13T12:36:32.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:32.242+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.244798+0000) 2019-10-13T12:36:32.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:32.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.848790+0000) 2019-10-13T12:36:32.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:32.846+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.848877+0000) 2019-10-13T12:36:32.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:32.842+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.849564+0000) 2019-10-13T12:36:32.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:32.846+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.849870+0000) 2019-10-13T12:36:32.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:32.846+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.850195+0000) 2019-10-13T12:36:32.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:32.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.850622+0000) 2019-10-13T12:36:32.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:32.846+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.851185+0000) 2019-10-13T12:36:32.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:32.850+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:32.854197+0000) 2019-10-13T12:36:32.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:32.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:35.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:35.518+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:35.519832+0000) 2019-10-13T12:36:36.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:36.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:36.487898+0000) 2019-10-13T12:36:37.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:37.854+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.858901+0000) 2019-10-13T12:36:37.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:37.858+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.859794+0000) 2019-10-13T12:36:37.861 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:37.858+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.860133+0000) 2019-10-13T12:36:37.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:37.858+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.860316+0000) 2019-10-13T12:36:37.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:37.854+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.861181+0000) 2019-10-13T12:36:37.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:37.858+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.861462+0000) 2019-10-13T12:36:37.864 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:37.858+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.862611+0000) 2019-10-13T12:36:37.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:37.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 151965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:37.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:37.862+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.868577+0000) 2019-10-13T12:36:38.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:38.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:38.753865+0000) 2019-10-13T12:36:38.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:38.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:38.757234+0000) 2019-10-13T12:36:39.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:39.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.055801+0000) 2019-10-13T12:36:39.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:39.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.988494+0000) 2019-10-13T12:36:40.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:40.802+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:40.803625+0000) 2019-10-13T12:36:42.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:42.241+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.245004+0000) 2019-10-13T12:36:42.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:42.865+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.870268+0000) 2019-10-13T12:36:42.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:42.865+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.870551+0000) 2019-10-13T12:36:42.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:42.865+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.870988+0000) 2019-10-13T12:36:42.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:42.869+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.871387+0000) 2019-10-13T12:36:42.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:42.869+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.872939+0000) 2019-10-13T12:36:42.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:42.866+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.873476+0000) 2019-10-13T12:36:42.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:42.869+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.873718+0000) 2019-10-13T12:36:42.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:42.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:42.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:42.874+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:42.878249+0000) 2019-10-13T12:36:45.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:45.517+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:45.520085+0000) 2019-10-13T12:36:46.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:46.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:46.488093+0000) 2019-10-13T12:36:47.882 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:47.877+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.880590+0000) 2019-10-13T12:36:47.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:47.877+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.880981+0000) 2019-10-13T12:36:47.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:47.877+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.881641+0000) 2019-10-13T12:36:47.883 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:47.877+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.881851+0000) 2019-10-13T12:36:47.884 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:47.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.882670+0000) 2019-10-13T12:36:47.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:47.877+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.882946+0000) 2019-10-13T12:36:47.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:47.882+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.884090+0000) 2019-10-13T12:36:47.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:47.885+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:47.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:47.886+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.888665+0000) 2019-10-13T12:36:48.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:48.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:48.754071+0000) 2019-10-13T12:36:48.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:48.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:48.757424+0000) 2019-10-13T12:36:49.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:49.053+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.056017+0000) 2019-10-13T12:36:49.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:49.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.988699+0000) 2019-10-13T12:36:50.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:50.806+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:50.810033+0000) 2019-10-13T12:36:52.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:52.241+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.245194+0000) 2019-10-13T12:36:52.893 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:52.889+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.891544+0000) 2019-10-13T12:36:52.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:52.889+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.892011+0000) 2019-10-13T12:36:52.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:52.889+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.893211+0000) 2019-10-13T12:36:52.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:52.889+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.893499+0000) 2019-10-13T12:36:52.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:52.889+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.894446+0000) 2019-10-13T12:36:52.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:52.893+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.895428+0000) 2019-10-13T12:36:52.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:52.889+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.894574+0000) 2019-10-13T12:36:52.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:52.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:52.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:52.897+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:52.900434+0000) 2019-10-13T12:36:55.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:55.517+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:55.520287+0000) 2019-10-13T12:36:56.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:56.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:56.488286+0000) 2019-10-13T12:36:57.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:36:57.897+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.902509+0000) 2019-10-13T12:36:57.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:57.897+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.902877+0000) 2019-10-13T12:36:57.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:36:57.897+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.903151+0000) 2019-10-13T12:36:57.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:57.901+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.904108+0000) 2019-10-13T12:36:57.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:57.901+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.904834+0000) 2019-10-13T12:36:57.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:57.901+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.905373+0000) 2019-10-13T12:36:57.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:36:57.901+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.905560+0000) 2019-10-13T12:36:57.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:36:57.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:36:57.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:36:57.909+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.910572+0000) 2019-10-13T12:36:58.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:36:58.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:58.754278+0000) 2019-10-13T12:36:58.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:36:58.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:58.757616+0000) 2019-10-13T12:36:59.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:36:59.053+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.056230+0000) 2019-10-13T12:36:59.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:36:59.985+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.988990+0000) 2019-10-13T12:37:00.574 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:37:00.574 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:00.578 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:37:00.578 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:00.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:00.813+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:00.816449+0000) 2019-10-13T12:37:02.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:02.241+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.245457+0000) 2019-10-13T12:37:02.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:02.909+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.913389+0000) 2019-10-13T12:37:02.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:02.909+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.913814+0000) 2019-10-13T12:37:02.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:02.909+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.914273+0000) 2019-10-13T12:37:02.916 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:02.909+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.914662+0000) 2019-10-13T12:37:02.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:02.909+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.915007+0000) 2019-10-13T12:37:02.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:02.913+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.915642+0000) 2019-10-13T12:37:02.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:02.913+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.916135+0000) 2019-10-13T12:37:02.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:02.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:02.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:02.921+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:02.922443+0000) 2019-10-13T12:37:05.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:05.517+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:05.520483+0000) 2019-10-13T12:37:06.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:06.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:06.488516+0000) 2019-10-13T12:37:07.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:07.921+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.923722+0000) 2019-10-13T12:37:07.926 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:07.921+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.924502+0000) 2019-10-13T12:37:07.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:07.921+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.925456+0000) 2019-10-13T12:37:07.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:07.921+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.925961+0000) 2019-10-13T12:37:07.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:07.925+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.926206+0000) 2019-10-13T12:37:07.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:07.921+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.926640+0000) 2019-10-13T12:37:07.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:07.921+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.926882+0000) 2019-10-13T12:37:07.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:07.929+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.931580+0000) 2019-10-13T12:37:07.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:07.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:08.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:08.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:08.754458+0000) 2019-10-13T12:37:08.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:08.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:08.757816+0000) 2019-10-13T12:37:09.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:09.053+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.056437+0000) 2019-10-13T12:37:09.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:09.985+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.989181+0000) 2019-10-13T12:37:10.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:10.821+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:10.822996+0000) 2019-10-13T12:37:12.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:12.241+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.245661+0000) 2019-10-13T12:37:12.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:12.929+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.934195+0000) 2019-10-13T12:37:12.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:12.933+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.935588+0000) 2019-10-13T12:37:12.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:12.933+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.936174+0000) 2019-10-13T12:37:12.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:12.933+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.936504+0000) 2019-10-13T12:37:12.938 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:12.933+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.937060+0000) 2019-10-13T12:37:12.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:12.933+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.938024+0000) 2019-10-13T12:37:12.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:12.933+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.937729+0000) 2019-10-13T12:37:12.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:12.937+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:12.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:12.941+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:12.944584+0000) 2019-10-13T12:37:15.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:15.517+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:15.520689+0000) 2019-10-13T12:37:16.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:16.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:16.488703+0000) 2019-10-13T12:37:17.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:17.941+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.945981+0000) 2019-10-13T12:37:17.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:17.941+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.946215+0000) 2019-10-13T12:37:17.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:17.941+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.946603+0000) 2019-10-13T12:37:17.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:17.941+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.947213+0000) 2019-10-13T12:37:17.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:17.945+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.948000+0000) 2019-10-13T12:37:17.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:17.945+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.948489+0000) 2019-10-13T12:37:17.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:17.945+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.948949+0000) 2019-10-13T12:37:17.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:17.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152241 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:17.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:17.953+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.956065+0000) 2019-10-13T12:37:18.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:18.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:18.754638+0000) 2019-10-13T12:37:18.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:18.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:18.758020+0000) 2019-10-13T12:37:19.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:19.053+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.056639+0000) 2019-10-13T12:37:19.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:19.985+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.989393+0000) 2019-10-13T12:37:20.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:20.825+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:20.829416+0000) 2019-10-13T12:37:22.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:22.245+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.245871+0000) 2019-10-13T12:37:22.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:22.953+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.956715+0000) 2019-10-13T12:37:22.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:22.952+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.957337+0000) 2019-10-13T12:37:22.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:22.952+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.958055+0000) 2019-10-13T12:37:22.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:22.952+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.958764+0000) 2019-10-13T12:37:22.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:22.952+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.958344+0000) 2019-10-13T12:37:22.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:22.956+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.959499+0000) 2019-10-13T12:37:22.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:22.956+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.960226+0000) 2019-10-13T12:37:22.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:22.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:22.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:22.965+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:22.967095+0000) 2019-10-13T12:37:25.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:25.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:25.520901+0000) 2019-10-13T12:37:26.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:26.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:26.488915+0000) 2019-10-13T12:37:27.969 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:27.964+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.967768+0000) 2019-10-13T12:37:27.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:27.964+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.967876+0000) 2019-10-13T12:37:27.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:27.965+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.968421+0000) 2019-10-13T12:37:27.971 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:27.964+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.969298+0000) 2019-10-13T12:37:27.971 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:27.964+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.969633+0000) 2019-10-13T12:37:27.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:27.964+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.970093+0000) 2019-10-13T12:37:27.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:27.964+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.970995+0000) 2019-10-13T12:37:27.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:27.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:27.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:27.977+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.978666+0000) 2019-10-13T12:37:28.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:28.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:28.754813+0000) 2019-10-13T12:37:28.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:28.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:28.758193+0000) 2019-10-13T12:37:29.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:29.052+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.056845+0000) 2019-10-13T12:37:29.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:29.985+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.989610+0000) 2019-10-13T12:37:30.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:30.833+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:30.835873+0000) 2019-10-13T12:37:30.922 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:37:30.922 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:30.926 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:37:30.926 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:32.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:32.244+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.246125+0000) 2019-10-13T12:37:32.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:32.972+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.977964+0000) 2019-10-13T12:37:32.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:32.972+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.978564+0000) 2019-10-13T12:37:32.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:32.972+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.978990+0000) 2019-10-13T12:37:32.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:32.976+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.979616+0000) 2019-10-13T12:37:32.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:32.976+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.980226+0000) 2019-10-13T12:37:32.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:32.976+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.980683+0000) 2019-10-13T12:37:32.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:32.977+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.981207+0000) 2019-10-13T12:37:32.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:32.980+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:32.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:32.981+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:32.985979+0000) 2019-10-13T12:37:35.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:35.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:35.521128+0000) 2019-10-13T12:37:36.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:36.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:36.489110+0000) 2019-10-13T12:37:37.989 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:37.984+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.987726+0000) 2019-10-13T12:37:37.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:37.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.988532+0000) 2019-10-13T12:37:37.990 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:37.984+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.988853+0000) 2019-10-13T12:37:37.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:37.985+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.990104+0000) 2019-10-13T12:37:37.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:37.984+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.990770+0000) 2019-10-13T12:37:37.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:37.988+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.991527+0000) 2019-10-13T12:37:37.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:37.988+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.992255+0000) 2019-10-13T12:37:37.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:37.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152379 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:37.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:37.993+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.996125+0000) 2019-10-13T12:37:38.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:38.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:38.755006+0000) 2019-10-13T12:37:38.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:38.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:38.758390+0000) 2019-10-13T12:37:39.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:39.052+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.057079+0000) 2019-10-13T12:37:39.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:39.989+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.989830+0000) 2019-10-13T12:37:40.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:40.841+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:40.842693+0000) 2019-10-13T12:37:42.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:42.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:42.246336+0000) 2019-10-13T12:37:43.000 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:42.992+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:42.997982+0000) 2019-10-13T12:37:43.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:42.992+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:42.998426+0000) 2019-10-13T12:37:43.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:42.992+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:42.998908+0000) 2019-10-13T12:37:43.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:42.996+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:42.999646+0000) 2019-10-13T12:37:43.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:42.997+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:43.000181+0000) 2019-10-13T12:37:43.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:42.996+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:43.000899+0000) 2019-10-13T12:37:43.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:42.996+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:43.001746+0000) 2019-10-13T12:37:43.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:43.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:43.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:43.005+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:43.006884+0000) 2019-10-13T12:37:45.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:45.520+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:45.521365+0000) 2019-10-13T12:37:46.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:46.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:46.489304+0000) 2019-10-13T12:37:48.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:48.004+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.009125+0000) 2019-10-13T12:37:48.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:48.004+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.009677+0000) 2019-10-13T12:37:48.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:48.004+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.010192+0000) 2019-10-13T12:37:48.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:48.004+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.010638+0000) 2019-10-13T12:37:48.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:48.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.011122+0000) 2019-10-13T12:37:48.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:48.008+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.011749+0000) 2019-10-13T12:37:48.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:48.008+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.012961+0000) 2019-10-13T12:37:48.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:48.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152447 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:48.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:48.016+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.018585+0000) 2019-10-13T12:37:48.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:48.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.755183+0000) 2019-10-13T12:37:48.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:48.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.758620+0000) 2019-10-13T12:37:49.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:49.056+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.057300+0000) 2019-10-13T12:37:49.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:49.984+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.990061+0000) 2019-10-13T12:37:50.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:50.844+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:50.849133+0000) 2019-10-13T12:37:52.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:52.240+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:52.246535+0000) 2019-10-13T12:37:53.022 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:53.016+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.020500+0000) 2019-10-13T12:37:53.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:53.016+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.021596+0000) 2019-10-13T12:37:53.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:53.016+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.022236+0000) 2019-10-13T12:37:53.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:53.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.022988+0000) 2019-10-13T12:37:53.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:53.016+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.023247+0000) 2019-10-13T12:37:53.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:53.020+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.023822+0000) 2019-10-13T12:37:53.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:53.020+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.024597+0000) 2019-10-13T12:37:53.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:53.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:53.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:53.024+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:53.028283+0000) 2019-10-13T12:37:55.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:55.516+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:55.521557+0000) 2019-10-13T12:37:56.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:56.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:56.489513+0000) 2019-10-13T12:37:58.033 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:37:58.024+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.031143+0000) 2019-10-13T12:37:58.034 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:58.028+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.032859+0000) 2019-10-13T12:37:58.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:58.028+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.033379+0000) 2019-10-13T12:37:58.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:37:58.028+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.033945+0000) 2019-10-13T12:37:58.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:58.028+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.033849+0000) 2019-10-13T12:37:58.036 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:58.028+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.034343+0000) 2019-10-13T12:37:58.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:37:58.028+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.034754+0000) 2019-10-13T12:37:58.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:37:58.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.038664+0000) 2019-10-13T12:37:58.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:37:58.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:37:58.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:37:58.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.755392+0000) 2019-10-13T12:37:58.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:37:58.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.758790+0000) 2019-10-13T12:37:59.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:37:59.052+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.057533+0000) 2019-10-13T12:37:59.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:37:59.988+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.990319+0000) 2019-10-13T12:38:00.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:00.852+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:00.855781+0000) 2019-10-13T12:38:01.045 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:38:01.045 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:01.049 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:38:01.049 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:02.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:02.244+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:02.246733+0000) 2019-10-13T12:38:03.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:03.036+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.041983+0000) 2019-10-13T12:38:03.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:03.036+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.042825+0000) 2019-10-13T12:38:03.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:03.040+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.043652+0000) 2019-10-13T12:38:03.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:03.040+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.043860+0000) 2019-10-13T12:38:03.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:03.040+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.044634+0000) 2019-10-13T12:38:03.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:03.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.045073+0000) 2019-10-13T12:38:03.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:03.040+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.046418+0000) 2019-10-13T12:38:03.051 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:03.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:03.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:03.048+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:03.052226+0000) 2019-10-13T12:38:05.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:05.515+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:05.521746+0000) 2019-10-13T12:38:06.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:06.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:06.489709+0000) 2019-10-13T12:38:08.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:08.047+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.053810+0000) 2019-10-13T12:38:08.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:08.047+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.054250+0000) 2019-10-13T12:38:08.056 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:08.047+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.054935+0000) 2019-10-13T12:38:08.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:08.047+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.054601+0000) 2019-10-13T12:38:08.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:08.051+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.055682+0000) 2019-10-13T12:38:08.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:08.051+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.056022+0000) 2019-10-13T12:38:08.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:08.052+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.056267+0000) 2019-10-13T12:38:08.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:08.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:08.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:08.060+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.063470+0000) 2019-10-13T12:38:08.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:08.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.755575+0000) 2019-10-13T12:38:08.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:08.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.758967+0000) 2019-10-13T12:38:09.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:09.055+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.057771+0000) 2019-10-13T12:38:09.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:09.988+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.990560+0000) 2019-10-13T12:38:10.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:10.860+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:10.862343+0000) 2019-10-13T12:38:12.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:12.243+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:12.246951+0000) 2019-10-13T12:38:13.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:13.059+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.064755+0000) 2019-10-13T12:38:13.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:13.059+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.065251+0000) 2019-10-13T12:38:13.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:13.059+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.066681+0000) 2019-10-13T12:38:13.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:13.059+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.067000+0000) 2019-10-13T12:38:13.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:13.063+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.067797+0000) 2019-10-13T12:38:13.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:13.063+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.068270+0000) 2019-10-13T12:38:13.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:13.064+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.068869+0000) 2019-10-13T12:38:13.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:13.068+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:13.072127+0000) 2019-10-13T12:38:13.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:13.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:15.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:15.519+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:15.521958+0000) 2019-10-13T12:38:16.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:16.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:16.489921+0000) 2019-10-13T12:38:18.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:18.072+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.076417+0000) 2019-10-13T12:38:18.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:18.071+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.077055+0000) 2019-10-13T12:38:18.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:18.071+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.077236+0000) 2019-10-13T12:38:18.079 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:18.071+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.077569+0000) 2019-10-13T12:38:18.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:18.071+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.077738+0000) 2019-10-13T12:38:18.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:18.071+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.078819+0000) 2019-10-13T12:38:18.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:18.071+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.079151+0000) 2019-10-13T12:38:18.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:18.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152651 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:18.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:18.080+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.085010+0000) 2019-10-13T12:38:18.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:18.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.755777+0000) 2019-10-13T12:38:18.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:18.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.759135+0000) 2019-10-13T12:38:19.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:19.055+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.057969+0000) 2019-10-13T12:38:19.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:19.988+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.990768+0000) 2019-10-13T12:38:20.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:20.868+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:20.868776+0000) 2019-10-13T12:38:22.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:22.239+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:22.247142+0000) 2019-10-13T12:38:23.088 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:23.079+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.086933+0000) 2019-10-13T12:38:23.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:23.083+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.087374+0000) 2019-10-13T12:38:23.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:23.083+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.087985+0000) 2019-10-13T12:38:23.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:23.084+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.088710+0000) 2019-10-13T12:38:23.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:23.083+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.088996+0000) 2019-10-13T12:38:23.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:23.083+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.089765+0000) 2019-10-13T12:38:23.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:23.083+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.090529+0000) 2019-10-13T12:38:23.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:23.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:23.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:23.092+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:23.095784+0000) 2019-10-13T12:38:25.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:25.519+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:25.522163+0000) 2019-10-13T12:38:26.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:26.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:26.490117+0000) 2019-10-13T12:38:28.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:28.091+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.098068+0000) 2019-10-13T12:38:28.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:28.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.098315+0000) 2019-10-13T12:38:28.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:28.091+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.099164+0000) 2019-10-13T12:38:28.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:28.095+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.099514+0000) 2019-10-13T12:38:28.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:28.095+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.100002+0000) 2019-10-13T12:38:28.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:28.095+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.100383+0000) 2019-10-13T12:38:28.102 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:28.095+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.100708+0000) 2019-10-13T12:38:28.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:28.100+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.105287+0000) 2019-10-13T12:38:28.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:28.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:28.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:28.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.755972+0000) 2019-10-13T12:38:28.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:28.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.759314+0000) 2019-10-13T12:38:29.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:29.055+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.058182+0000) 2019-10-13T12:38:29.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:29.988+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.990953+0000) 2019-10-13T12:38:30.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:30.872+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:30.875344+0000) 2019-10-13T12:38:31.156 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:38:31.156 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:31.160 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:38:31.160 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:32.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:32.243+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:32.247341+0000) 2019-10-13T12:38:33.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:33.103+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.108136+0000) 2019-10-13T12:38:33.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:33.103+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.109446+0000) 2019-10-13T12:38:33.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:33.104+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.109714+0000) 2019-10-13T12:38:33.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:33.103+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.110463+0000) 2019-10-13T12:38:33.112 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:33.103+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.110912+0000) 2019-10-13T12:38:33.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:33.107+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.111401+0000) 2019-10-13T12:38:33.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:33.107+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.111690+0000) 2019-10-13T12:38:33.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:33.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:33.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:33.112+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:33.116536+0000) 2019-10-13T12:38:35.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:35.519+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:35.522392+0000) 2019-10-13T12:38:36.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:36.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:36.490316+0000) 2019-10-13T12:38:38.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:38.115+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.119244+0000) 2019-10-13T12:38:38.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:38.119+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.120317+0000) 2019-10-13T12:38:38.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:38.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.120703+0000) 2019-10-13T12:38:38.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:38.119+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.120989+0000) 2019-10-13T12:38:38.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:38.119+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.121665+0000) 2019-10-13T12:38:38.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:38.119+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.121745+0000) 2019-10-13T12:38:38.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:38.119+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.122292+0000) 2019-10-13T12:38:38.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:38.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:38.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:38.124+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.128868+0000) 2019-10-13T12:38:38.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:38.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.756173+0000) 2019-10-13T12:38:38.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:38.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.759444+0000) 2019-10-13T12:38:39.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:39.055+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.058364+0000) 2019-10-13T12:38:39.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:39.987+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.991147+0000) 2019-10-13T12:38:40.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:40.879+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:40.881858+0000) 2019-10-13T12:38:42.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:42.247+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:42.247568+0000) 2019-10-13T12:38:43.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:43.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.129885+0000) 2019-10-13T12:38:43.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:43.127+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.130614+0000) 2019-10-13T12:38:43.132 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:43.127+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.130903+0000) 2019-10-13T12:38:43.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:43.131+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.131815+0000) 2019-10-13T12:38:43.133 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:43.131+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.131998+0000) 2019-10-13T12:38:43.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:43.131+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.132423+0000) 2019-10-13T12:38:43.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:43.131+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.133235+0000) 2019-10-13T12:38:43.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:43.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:43.141 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:43.135+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:43.139843+0000) 2019-10-13T12:38:45.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:45.519+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:45.522590+0000) 2019-10-13T12:38:46.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:46.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:46.490510+0000) 2019-10-13T12:38:48.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:48.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.141142+0000) 2019-10-13T12:38:48.143 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:48.139+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.141500+0000) 2019-10-13T12:38:48.143 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:48.139+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.142217+0000) 2019-10-13T12:38:48.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:48.139+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.142986+0000) 2019-10-13T12:38:48.144 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:48.143+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.143566+0000) 2019-10-13T12:38:48.145 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:48.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.144273+0000) 2019-10-13T12:38:48.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:48.143+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.145196+0000) 2019-10-13T12:38:48.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:48.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:48.153 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:48.147+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.151762+0000) 2019-10-13T12:38:48.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:48.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.756351+0000) 2019-10-13T12:38:48.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:48.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.759654+0000) 2019-10-13T12:38:49.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:49.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.058546+0000) 2019-10-13T12:38:49.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:49.987+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.991374+0000) 2019-10-13T12:38:50.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:50.887+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:50.888358+0000) 2019-10-13T12:38:52.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:52.246+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:52.247831+0000) 2019-10-13T12:38:53.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:53.150+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.151783+0000) 2019-10-13T12:38:53.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:53.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.152176+0000) 2019-10-13T12:38:53.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:53.150+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.152864+0000) 2019-10-13T12:38:53.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:53.150+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.153288+0000) 2019-10-13T12:38:53.155 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:53.150+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.153948+0000) 2019-10-13T12:38:53.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:53.150+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.154386+0000) 2019-10-13T12:38:53.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:53.151+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.154643+0000) 2019-10-13T12:38:53.161 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:53.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:53.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:53.155+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:53.161029+0000) 2019-10-13T12:38:55.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:55.518+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:55.522792+0000) 2019-10-13T12:38:56.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:56.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:56.490703+0000) 2019-10-13T12:38:58.164 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:58.158+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.162494+0000) 2019-10-13T12:38:58.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:38:58.158+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.163118+0000) 2019-10-13T12:38:58.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:38:58.162+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.163614+0000) 2019-10-13T12:38:58.165 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:38:58.162+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.164523+0000) 2019-10-13T12:38:58.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:58.159+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.164811+0000) 2019-10-13T12:38:58.166 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:58.162+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.165297+0000) 2019-10-13T12:38:58.167 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:58.162+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.165350+0000) 2019-10-13T12:38:58.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:38:58.163+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.168579+0000) 2019-10-13T12:38:58.172 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:38:58.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:38:58.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:38:58.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.756528+0000) 2019-10-13T12:38:58.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:38:58.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.759832+0000) 2019-10-13T12:38:59.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:38:59.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.058728+0000) 2019-10-13T12:38:59.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:38:59.987+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.991604+0000) 2019-10-13T12:39:00.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:00.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:00.897389+0000) 2019-10-13T12:39:01.283 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:39:01.283 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:01.287 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:39:01.287 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:02.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:02.246+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:02.248049+0000) 2019-10-13T12:39:03.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:03.170+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.174087+0000) 2019-10-13T12:39:03.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:03.170+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.175153+0000) 2019-10-13T12:39:03.177 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:03.174+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.175756+0000) 2019-10-13T12:39:03.177 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:03.171+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.176548+0000) 2019-10-13T12:39:03.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:03.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.177083+0000) 2019-10-13T12:39:03.178 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:03.174+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.177630+0000) 2019-10-13T12:39:03.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:03.174+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.178014+0000) 2019-10-13T12:39:03.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:03.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152953 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:03.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:03.179+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:03.183853+0000) 2019-10-13T12:39:05.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:05.518+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:05.522949+0000) 2019-10-13T12:39:06.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:06.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:06.490901+0000) 2019-10-13T12:39:08.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:08.182+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.185336+0000) 2019-10-13T12:39:08.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:08.182+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.185630+0000) 2019-10-13T12:39:08.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:08.182+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.185997+0000) 2019-10-13T12:39:08.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:08.182+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.186975+0000) 2019-10-13T12:39:08.189 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:08.186+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.187439+0000) 2019-10-13T12:39:08.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:08.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.188057+0000) 2019-10-13T12:39:08.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:08.186+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.188301+0000) 2019-10-13T12:39:08.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:08.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 152985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:08.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:08.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.199071+0000) 2019-10-13T12:39:08.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:08.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.756738+0000) 2019-10-13T12:39:08.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:08.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.760009+0000) 2019-10-13T12:39:09.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:09.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.058938+0000) 2019-10-13T12:39:09.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:09.991+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.991827+0000) 2019-10-13T12:39:10.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:10.899+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:10.903833+0000) 2019-10-13T12:39:12.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:12.246+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:12.248255+0000) 2019-10-13T12:39:13.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:13.194+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.196510+0000) 2019-10-13T12:39:13.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:13.194+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.196934+0000) 2019-10-13T12:39:13.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:13.194+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.197522+0000) 2019-10-13T12:39:13.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:13.194+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.198481+0000) 2019-10-13T12:39:13.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:13.195+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.199034+0000) 2019-10-13T12:39:13.200 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:13.194+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.199167+0000) 2019-10-13T12:39:13.201 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:13.198+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.199574+0000) 2019-10-13T12:39:13.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:13.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153020 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:13.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:13.203+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:13.206951+0000) 2019-10-13T12:39:15.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:15.522+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:15.523144+0000) 2019-10-13T12:39:16.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:16.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:16.491095+0000) 2019-10-13T12:39:18.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:18.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.207777+0000) 2019-10-13T12:39:18.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:18.206+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.208098+0000) 2019-10-13T12:39:18.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:18.206+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.208460+0000) 2019-10-13T12:39:18.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:18.206+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.208876+0000) 2019-10-13T12:39:18.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:18.206+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.209447+0000) 2019-10-13T12:39:18.211 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:18.206+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.210613+0000) 2019-10-13T12:39:18.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:18.206+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.210924+0000) 2019-10-13T12:39:18.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:18.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:18.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:18.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.217312+0000) 2019-10-13T12:39:18.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:18.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.756870+0000) 2019-10-13T12:39:18.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:18.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.760188+0000) 2019-10-13T12:39:19.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:19.054+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.059127+0000) 2019-10-13T12:39:19.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:19.987+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.992019+0000) 2019-10-13T12:39:20.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:20.907+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:20.910371+0000) 2019-10-13T12:39:22.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:22.246+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:22.248450+0000) 2019-10-13T12:39:23.220 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:23.214+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.218925+0000) 2019-10-13T12:39:23.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:23.218+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.219337+0000) 2019-10-13T12:39:23.221 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:23.218+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.219721+0000) 2019-10-13T12:39:23.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:23.218+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.220423+0000) 2019-10-13T12:39:23.222 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:23.218+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.221235+0000) 2019-10-13T12:39:23.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:23.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.221681+0000) 2019-10-13T12:39:23.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:23.218+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.222496+0000) 2019-10-13T12:39:23.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:23.222+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:23.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:23.223+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:23.226512+0000) 2019-10-13T12:39:25.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:25.522+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:25.523396+0000) 2019-10-13T12:39:26.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:26.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:26.491289+0000) 2019-10-13T12:39:28.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:28.226+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.229305+0000) 2019-10-13T12:39:28.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:28.226+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.229747+0000) 2019-10-13T12:39:28.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:28.226+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.230389+0000) 2019-10-13T12:39:28.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:28.226+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.230927+0000) 2019-10-13T12:39:28.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:28.230+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.231736+0000) 2019-10-13T12:39:28.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:28.230+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.232246+0000) 2019-10-13T12:39:28.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:28.227+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.233491+0000) 2019-10-13T12:39:28.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:28.234+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:28.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:28.231+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.237653+0000) 2019-10-13T12:39:28.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:28.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.757066+0000) 2019-10-13T12:39:28.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:28.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.760405+0000) 2019-10-13T12:39:29.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:29.058+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.059340+0000) 2019-10-13T12:39:29.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:29.991+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.992214+0000) 2019-10-13T12:39:30.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:30.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:30.916741+0000) 2019-10-13T12:39:31.404 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:39:31.404 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:31.408 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:39:31.408 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:32.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:32.245+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:32.248660+0000) 2019-10-13T12:39:33.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:33.237+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.240885+0000) 2019-10-13T12:39:33.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:33.237+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.242080+0000) 2019-10-13T12:39:33.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:33.237+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.242632+0000) 2019-10-13T12:39:33.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:33.239+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.242997+0000) 2019-10-13T12:39:33.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:33.237+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.243141+0000) 2019-10-13T12:39:33.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:33.241+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.243520+0000) 2019-10-13T12:39:33.246 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:33.241+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.244248+0000) 2019-10-13T12:39:33.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:33.243+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:33.247710+0000) 2019-10-13T12:39:33.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:33.245+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:35.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:35.521+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:35.523595+0000) 2019-10-13T12:39:36.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:36.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:36.491511+0000) 2019-10-13T12:39:38.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:38.246+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.252205+0000) 2019-10-13T12:39:38.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:38.249+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.252707+0000) 2019-10-13T12:39:38.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:38.249+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.253528+0000) 2019-10-13T12:39:38.255 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:38.249+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.254121+0000) 2019-10-13T12:39:38.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:38.249+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.253826+0000) 2019-10-13T12:39:38.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:38.249+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.254603+0000) 2019-10-13T12:39:38.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:38.249+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.255045+0000) 2019-10-13T12:39:38.261 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:38.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:38.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:38.258+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.262700+0000) 2019-10-13T12:39:38.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:38.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.757243+0000) 2019-10-13T12:39:38.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:38.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.760600+0000) 2019-10-13T12:39:39.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:39.057+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.059536+0000) 2019-10-13T12:39:39.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:39.990+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.992407+0000) 2019-10-13T12:39:40.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:40.922+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:40.923148+0000) 2019-10-13T12:39:42.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:42.245+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:42.248939+0000) 2019-10-13T12:39:43.265 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:43.261+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.263564+0000) 2019-10-13T12:39:43.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:43.261+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.265349+0000) 2019-10-13T12:39:43.267 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:43.261+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.265783+0000) 2019-10-13T12:39:43.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:43.261+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.266236+0000) 2019-10-13T12:39:43.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:43.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.266667+0000) 2019-10-13T12:39:43.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:43.265+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.267350+0000) 2019-10-13T12:39:43.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:43.265+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.268448+0000) 2019-10-13T12:39:43.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:43.269+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153221 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:43.276 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:43.270+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:43.275452+0000) 2019-10-13T12:39:45.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:45.521+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:45.523839+0000) 2019-10-13T12:39:46.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:46.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:46.491704+0000) 2019-10-13T12:39:48.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:48.269+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.275283+0000) 2019-10-13T12:39:48.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:48.273+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.275763+0000) 2019-10-13T12:39:48.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:48.273+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.277121+0000) 2019-10-13T12:39:48.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:48.273+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.277331+0000) 2019-10-13T12:39:48.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:48.273+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.277651+0000) 2019-10-13T12:39:48.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:48.273+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.278363+0000) 2019-10-13T12:39:48.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:48.274+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.278789+0000) 2019-10-13T12:39:48.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:48.277+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:48.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:48.282+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.287296+0000) 2019-10-13T12:39:48.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:48.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.757422+0000) 2019-10-13T12:39:48.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:48.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.760769+0000) 2019-10-13T12:39:49.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:49.057+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.059725+0000) 2019-10-13T12:39:49.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:49.986+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.992599+0000) 2019-10-13T12:39:50.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:50.926+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:50.929407+0000) 2019-10-13T12:39:52.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:52.245+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:52.249140+0000) 2019-10-13T12:39:53.287 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:53.281+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.285862+0000) 2019-10-13T12:39:53.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:53.281+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.286324+0000) 2019-10-13T12:39:53.288 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:53.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.286960+0000) 2019-10-13T12:39:53.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:53.285+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.287848+0000) 2019-10-13T12:39:53.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:53.285+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.287876+0000) 2019-10-13T12:39:53.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:53.285+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.288320+0000) 2019-10-13T12:39:53.291 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:53.285+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.289713+0000) 2019-10-13T12:39:53.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:53.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:53.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:53.290+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:53.294782+0000) 2019-10-13T12:39:55.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:55.521+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:55.524043+0000) 2019-10-13T12:39:56.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:56.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:56.491941+0000) 2019-10-13T12:39:58.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:58.293+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.297701+0000) 2019-10-13T12:39:58.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:58.293+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.297937+0000) 2019-10-13T12:39:58.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:58.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.298908+0000) 2019-10-13T12:39:58.300 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:58.293+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.299138+0000) 2019-10-13T12:39:58.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:39:58.297+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.299477+0000) 2019-10-13T12:39:58.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:39:58.297+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.299856+0000) 2019-10-13T12:39:58.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:39:58.297+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.300618+0000) 2019-10-13T12:39:58.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:39:58.301+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:39:58.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:39:58.302+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.306397+0000) 2019-10-13T12:39:58.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:39:58.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.757596+0000) 2019-10-13T12:39:58.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:39:58.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.760942+0000) 2019-10-13T12:39:59.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:39:59.057+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.059924+0000) 2019-10-13T12:39:59.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:39:59.990+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.992791+0000) 2019-10-13T12:40:00.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:00.934+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:00.935889+0000) 2019-10-13T12:40:01.514 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:40:01.515 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:01.518 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:40:01.518 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:02.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:02.249+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:02.249373+0000) 2019-10-13T12:40:03.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:03.305+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.308406+0000) 2019-10-13T12:40:03.310 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:03.305+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.308949+0000) 2019-10-13T12:40:03.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:03.305+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.309242+0000) 2019-10-13T12:40:03.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:03.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.310345+0000) 2019-10-13T12:40:03.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:03.305+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.310615+0000) 2019-10-13T12:40:03.312 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:03.309+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.311449+0000) 2019-10-13T12:40:03.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:03.309+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.311879+0000) 2019-10-13T12:40:03.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:03.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:03.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:03.310+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:03.318031+0000) 2019-10-13T12:40:05.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:05.521+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:05.524241+0000) 2019-10-13T12:40:06.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:06.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:06.492194+0000) 2019-10-13T12:40:08.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:08.317+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.320400+0000) 2019-10-13T12:40:08.322 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:08.317+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.320837+0000) 2019-10-13T12:40:08.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:08.317+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.321284+0000) 2019-10-13T12:40:08.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:08.317+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.321947+0000) 2019-10-13T12:40:08.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:08.317+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.322900+0000) 2019-10-13T12:40:08.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:08.317+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.323065+0000) 2019-10-13T12:40:08.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:08.318+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.323530+0000) 2019-10-13T12:40:08.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:08.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:08.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:08.322+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.328939+0000) 2019-10-13T12:40:08.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:08.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.757769+0000) 2019-10-13T12:40:08.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:08.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.761118+0000) 2019-10-13T12:40:09.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:09.057+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.060113+0000) 2019-10-13T12:40:09.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:09.990+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.992991+0000) 2019-10-13T12:40:10.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:10.942+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:10.942441+0000) 2019-10-13T12:40:12.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:12.245+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:12.249592+0000) 2019-10-13T12:40:13.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:13.329+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.331345+0000) 2019-10-13T12:40:13.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:13.329+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.331675+0000) 2019-10-13T12:40:13.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:13.329+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.332283+0000) 2019-10-13T12:40:13.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:13.329+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.333010+0000) 2019-10-13T12:40:13.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:13.329+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.334009+0000) 2019-10-13T12:40:13.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:13.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.334410+0000) 2019-10-13T12:40:13.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:13.329+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.334741+0000) 2019-10-13T12:40:13.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:13.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:13.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:13.334+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:13.339336+0000) 2019-10-13T12:40:15.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:15.520+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:15.524472+0000) 2019-10-13T12:40:16.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:16.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:16.492407+0000) 2019-10-13T12:40:18.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:18.336+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.342868+0000) 2019-10-13T12:40:18.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:18.336+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.342811+0000) 2019-10-13T12:40:18.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:18.340+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.343298+0000) 2019-10-13T12:40:18.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:18.340+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.343877+0000) 2019-10-13T12:40:18.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:18.340+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.344879+0000) 2019-10-13T12:40:18.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:18.340+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.345169+0000) 2019-10-13T12:40:18.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:18.338+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.345822+0000) 2019-10-13T12:40:18.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:18.346+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.350964+0000) 2019-10-13T12:40:18.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:18.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:18.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:18.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.757982+0000) 2019-10-13T12:40:18.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:18.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.761291+0000) 2019-10-13T12:40:19.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:19.056+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.060322+0000) 2019-10-13T12:40:19.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:19.990+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.993199+0000) 2019-10-13T12:40:20.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:20.946+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:20.948906+0000) 2019-10-13T12:40:22.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:22.248+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:22.249816+0000) 2019-10-13T12:40:23.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:23.348+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.353606+0000) 2019-10-13T12:40:23.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:23.348+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.354747+0000) 2019-10-13T12:40:23.356 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:23.348+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.355065+0000) 2019-10-13T12:40:23.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:23.352+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.355894+0000) 2019-10-13T12:40:23.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:23.352+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.356027+0000) 2019-10-13T12:40:23.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:23.352+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.356795+0000) 2019-10-13T12:40:23.358 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:23.354+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.357488+0000) 2019-10-13T12:40:23.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:23.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:23.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:23.362+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:23.363340+0000) 2019-10-13T12:40:25.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:25.520+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:25.524696+0000) 2019-10-13T12:40:26.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:26.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:26.492631+0000) 2019-10-13T12:40:28.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:28.360+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.365866+0000) 2019-10-13T12:40:28.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:28.365+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.366831+0000) 2019-10-13T12:40:28.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:28.364+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.367563+0000) 2019-10-13T12:40:28.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:28.364+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.368266+0000) 2019-10-13T12:40:28.370 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:28.364+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.368586+0000) 2019-10-13T12:40:28.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:28.364+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.369028+0000) 2019-10-13T12:40:28.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:28.364+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.369528+0000) 2019-10-13T12:40:28.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:28.369+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.373407+0000) 2019-10-13T12:40:28.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:28.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153519 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:28.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:28.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.758171+0000) 2019-10-13T12:40:28.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:28.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.761459+0000) 2019-10-13T12:40:29.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:29.056+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.060512+0000) 2019-10-13T12:40:29.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:29.989+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.993395+0000) 2019-10-13T12:40:30.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:30.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:30.955502+0000) 2019-10-13T12:40:31.634 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:40:31.634 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:31.638 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:40:31.638 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:32.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:32.244+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:32.250048+0000) 2019-10-13T12:40:33.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:33.372+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.377857+0000) 2019-10-13T12:40:33.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:33.377+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.378385+0000) 2019-10-13T12:40:33.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:33.376+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.379463+0000) 2019-10-13T12:40:33.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:33.376+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.379853+0000) 2019-10-13T12:40:33.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:33.376+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.380494+0000) 2019-10-13T12:40:33.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:33.376+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.380623+0000) 2019-10-13T12:40:33.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:33.376+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.381519+0000) 2019-10-13T12:40:33.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:33.381+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:33.384824+0000) 2019-10-13T12:40:33.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:33.380+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:35.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:35.524+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:35.524905+0000) 2019-10-13T12:40:36.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:36.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:36.492839+0000) 2019-10-13T12:40:38.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:38.384+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.388315+0000) 2019-10-13T12:40:38.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:38.384+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.389040+0000) 2019-10-13T12:40:38.391 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:38.384+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.389478+0000) 2019-10-13T12:40:38.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:38.384+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.389752+0000) 2019-10-13T12:40:38.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:38.384+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.390274+0000) 2019-10-13T12:40:38.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:38.384+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.391119+0000) 2019-10-13T12:40:38.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:38.389+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.391916+0000) 2019-10-13T12:40:38.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:38.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:38.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:38.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.398573+0000) 2019-10-13T12:40:38.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:38.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.758348+0000) 2019-10-13T12:40:38.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:38.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.761635+0000) 2019-10-13T12:40:39.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:39.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.060706+0000) 2019-10-13T12:40:39.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:39.989+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.993590+0000) 2019-10-13T12:40:40.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:40.961+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:40.962121+0000) 2019-10-13T12:40:42.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:42.244+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:42.250277+0000) 2019-10-13T12:40:43.401 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:43.392+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.399099+0000) 2019-10-13T12:40:43.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:43.396+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.399509+0000) 2019-10-13T12:40:43.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:43.396+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.399935+0000) 2019-10-13T12:40:43.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:43.397+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.401053+0000) 2019-10-13T12:40:43.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:43.396+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.401268+0000) 2019-10-13T12:40:43.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:43.396+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.402185+0000) 2019-10-13T12:40:43.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:43.396+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.403018+0000) 2019-10-13T12:40:43.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:43.405+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:43.406891+0000) 2019-10-13T12:40:43.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:43.404+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:45.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:45.520+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:45.525131+0000) 2019-10-13T12:40:46.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:46.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:46.493065+0000) 2019-10-13T12:40:48.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:48.404+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.411271+0000) 2019-10-13T12:40:48.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:48.408+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.411426+0000) 2019-10-13T12:40:48.413 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:48.409+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.411703+0000) 2019-10-13T12:40:48.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:48.408+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.412960+0000) 2019-10-13T12:40:48.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:48.408+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.413298+0000) 2019-10-13T12:40:48.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:48.408+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.413466+0000) 2019-10-13T12:40:48.415 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:48.408+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.413724+0000) 2019-10-13T12:40:48.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:48.412+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:48.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:48.417+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.420791+0000) 2019-10-13T12:40:48.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:48.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.758541+0000) 2019-10-13T12:40:48.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:48.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.761804+0000) 2019-10-13T12:40:49.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:49.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.060900+0000) 2019-10-13T12:40:49.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:49.989+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.993785+0000) 2019-10-13T12:40:50.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:50.965+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:50.968544+0000) 2019-10-13T12:40:52.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:52.248+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:52.250479+0000) 2019-10-13T12:40:53.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:53.416+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.422385+0000) 2019-10-13T12:40:53.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:53.416+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.422810+0000) 2019-10-13T12:40:53.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:53.421+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.423523+0000) 2019-10-13T12:40:53.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:53.420+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.423830+0000) 2019-10-13T12:40:53.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:53.420+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.424515+0000) 2019-10-13T12:40:53.426 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:53.420+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.424791+0000) 2019-10-13T12:40:53.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:53.420+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.425336+0000) 2019-10-13T12:40:53.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:53.425+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:53.428609+0000) 2019-10-13T12:40:53.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:53.424+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:55.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:55.520+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:55.525329+0000) 2019-10-13T12:40:56.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:56.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:56.493287+0000) 2019-10-13T12:40:58.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:58.427+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.433748+0000) 2019-10-13T12:40:58.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:40:58.427+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.434253+0000) 2019-10-13T12:40:58.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:58.427+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.434979+0000) 2019-10-13T12:40:58.446 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:40:58.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.435553+0000) 2019-10-13T12:40:58.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:58.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.435947+0000) 2019-10-13T12:40:58.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:58.431+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.436062+0000) 2019-10-13T12:40:58.447 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:40:58.431+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.436471+0000) 2019-10-13T12:40:58.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:40:58.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:40:58.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:40:58.445+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.446203+0000) 2019-10-13T12:40:58.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:40:58.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.758725+0000) 2019-10-13T12:40:58.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:40:58.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.761980+0000) 2019-10-13T12:40:59.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:40:59.059+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.061082+0000) 2019-10-13T12:40:59.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:40:59.993+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.993981+0000) 2019-10-13T12:41:00.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:00.973+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:00.975096+0000) 2019-10-13T12:41:01.676 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:41:01.676 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:01.680 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:41:01.680 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:02.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:02.243+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:02.250742+0000) 2019-10-13T12:41:03.446 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:03.441+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.444748+0000) 2019-10-13T12:41:03.446 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:03.439+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.445388+0000) 2019-10-13T12:41:03.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:03.439+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.445991+0000) 2019-10-13T12:41:03.448 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:03.439+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.446672+0000) 2019-10-13T12:41:03.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:03.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.446919+0000) 2019-10-13T12:41:03.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:03.443+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.447447+0000) 2019-10-13T12:41:03.450 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:03.443+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.449380+0000) 2019-10-13T12:41:03.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:03.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:03.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:03.453+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:03.454697+0000) 2019-10-13T12:41:05.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:05.523+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:05.525519+0000) 2019-10-13T12:41:06.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:06.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:06.493472+0000) 2019-10-13T12:41:08.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:08.451+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.456452+0000) 2019-10-13T12:41:08.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:08.451+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.456951+0000) 2019-10-13T12:41:08.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:08.451+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.457155+0000) 2019-10-13T12:41:08.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:08.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.457844+0000) 2019-10-13T12:41:08.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:08.451+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.458546+0000) 2019-10-13T12:41:08.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:08.451+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.458662+0000) 2019-10-13T12:41:08.460 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:08.451+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.459054+0000) 2019-10-13T12:41:08.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:08.459+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153782 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:08.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:08.465+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.468233+0000) 2019-10-13T12:41:08.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:08.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.758903+0000) 2019-10-13T12:41:08.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:08.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.762181+0000) 2019-10-13T12:41:09.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:09.059+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.061270+0000) 2019-10-13T12:41:09.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:09.993+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.994173+0000) 2019-10-13T12:41:10.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:10.985+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:10.986625+0000) 2019-10-13T12:41:12.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:12.247+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:12.250945+0000) 2019-10-13T12:41:13.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:13.463+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.467387+0000) 2019-10-13T12:41:13.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:13.463+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.467772+0000) 2019-10-13T12:41:13.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:13.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.468508+0000) 2019-10-13T12:41:13.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:13.463+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.468817+0000) 2019-10-13T12:41:13.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:13.463+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.469081+0000) 2019-10-13T12:41:13.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:13.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.469352+0000) 2019-10-13T12:41:13.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:13.463+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.469872+0000) 2019-10-13T12:41:13.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:13.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:13.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:13.473+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:13.476604+0000) 2019-10-13T12:41:15.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:15.523+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:15.525732+0000) 2019-10-13T12:41:16.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:16.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:16.493679+0000) 2019-10-13T12:41:18.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:18.475+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.479376+0000) 2019-10-13T12:41:18.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:18.475+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.480034+0000) 2019-10-13T12:41:18.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:18.475+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.480430+0000) 2019-10-13T12:41:18.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:18.477+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.481625+0000) 2019-10-13T12:41:18.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:18.475+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.481470+0000) 2019-10-13T12:41:18.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:18.475+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.482115+0000) 2019-10-13T12:41:18.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:18.475+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.482426+0000) 2019-10-13T12:41:18.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:18.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:18.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:18.489+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.490920+0000) 2019-10-13T12:41:18.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:18.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.759087+0000) 2019-10-13T12:41:18.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:18.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.762360+0000) 2019-10-13T12:41:19.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:19.059+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.061457+0000) 2019-10-13T12:41:19.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:19.993+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.994371+0000) 2019-10-13T12:41:20.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:20.997+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:20.997855+0000) 2019-10-13T12:41:22.252 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:22.247+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:22.251153+0000) 2019-10-13T12:41:23.492 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:23.487+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.490680+0000) 2019-10-13T12:41:23.492 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:23.487+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.490949+0000) 2019-10-13T12:41:23.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:23.487+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.491602+0000) 2019-10-13T12:41:23.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:23.487+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.492924+0000) 2019-10-13T12:41:23.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:23.487+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.493159+0000) 2019-10-13T12:41:23.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:23.488+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.493465+0000) 2019-10-13T12:41:23.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:23.487+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.493743+0000) 2019-10-13T12:41:23.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:23.496+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:23.498421+0000) 2019-10-13T12:41:23.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:23.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:25.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:25.519+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:25.525923+0000) 2019-10-13T12:41:26.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:26.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:26.493893+0000) 2019-10-13T12:41:28.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:28.499+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.502351+0000) 2019-10-13T12:41:28.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:28.503+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.503519+0000) 2019-10-13T12:41:28.505 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:28.500+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.504335+0000) 2019-10-13T12:41:28.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:28.503+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.504901+0000) 2019-10-13T12:41:28.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:28.503+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.505261+0000) 2019-10-13T12:41:28.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:28.503+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.505858+0000) 2019-10-13T12:41:28.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:28.503+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.505965+0000) 2019-10-13T12:41:28.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:28.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153914 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:28.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:28.508+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.513608+0000) 2019-10-13T12:41:28.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:28.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.759261+0000) 2019-10-13T12:41:28.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:28.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.762538+0000) 2019-10-13T12:41:29.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:29.059+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.061659+0000) 2019-10-13T12:41:29.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:29.992+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.994554+0000) 2019-10-13T12:41:31.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:31.008+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:31.009344+0000) 2019-10-13T12:41:31.870 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:41:31.870 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:31.874 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:41:31.874 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:32.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:32.251+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:32.251371+0000) 2019-10-13T12:41:33.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:33.508+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.513350+0000) 2019-10-13T12:41:33.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:33.511+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.513594+0000) 2019-10-13T12:41:33.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:33.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.513944+0000) 2019-10-13T12:41:33.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:33.515+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.515703+0000) 2019-10-13T12:41:33.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:33.515+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.516251+0000) 2019-10-13T12:41:33.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:33.515+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.516633+0000) 2019-10-13T12:41:33.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:33.515+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.517242+0000) 2019-10-13T12:41:33.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:33.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:33.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:33.516+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:33.521813+0000) 2019-10-13T12:41:35.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:35.523+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:35.526123+0000) 2019-10-13T12:41:36.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:36.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:36.494087+0000) 2019-10-13T12:41:38.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:38.523+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.524128+0000) 2019-10-13T12:41:38.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:38.523+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.525261+0000) 2019-10-13T12:41:38.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:38.523+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.525465+0000) 2019-10-13T12:41:38.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:38.523+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.526177+0000) 2019-10-13T12:41:38.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:38.523+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.526740+0000) 2019-10-13T12:41:38.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:38.524+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.527231+0000) 2019-10-13T12:41:38.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:38.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.528317+0000) 2019-10-13T12:41:38.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:38.528+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.531184+0000) 2019-10-13T12:41:38.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:38.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 153981 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:38.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:38.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.759430+0000) 2019-10-13T12:41:38.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:38.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.762750+0000) 2019-10-13T12:41:39.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:39.059+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.061837+0000) 2019-10-13T12:41:39.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:39.992+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.994746+0000) 2019-10-13T12:41:41.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:41.016+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:41.020958+0000) 2019-10-13T12:41:42.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:42.250+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:42.251609+0000) 2019-10-13T12:41:43.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:43.534+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.535705+0000) 2019-10-13T12:41:43.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:43.534+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.536341+0000) 2019-10-13T12:41:43.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:43.534+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.536724+0000) 2019-10-13T12:41:43.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:43.532+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.537773+0000) 2019-10-13T12:41:43.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:43.534+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.538193+0000) 2019-10-13T12:41:43.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:43.534+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.538912+0000) 2019-10-13T12:41:43.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:43.534+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.539002+0000) 2019-10-13T12:41:43.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:43.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:43.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:43.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:43.546686+0000) 2019-10-13T12:41:45.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:45.522+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:45.526325+0000) 2019-10-13T12:41:46.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:46.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:46.494283+0000) 2019-10-13T12:41:48.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:48.542+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.546872+0000) 2019-10-13T12:41:48.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:48.546+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.547621+0000) 2019-10-13T12:41:48.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:48.546+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.547981+0000) 2019-10-13T12:41:48.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:48.546+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.548382+0000) 2019-10-13T12:41:48.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:48.546+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.548686+0000) 2019-10-13T12:41:48.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:48.544+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.549780+0000) 2019-10-13T12:41:48.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:48.546+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.550126+0000) 2019-10-13T12:41:48.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:48.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:48.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:48.552+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.555836+0000) 2019-10-13T12:41:48.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:48.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.759606+0000) 2019-10-13T12:41:48.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:48.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.762923+0000) 2019-10-13T12:41:49.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:49.058+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:49.062016+0000) 2019-10-13T12:41:49.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:49.992+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:49.994954+0000) 2019-10-13T12:41:51.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:51.024+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:51.027496+0000) 2019-10-13T12:41:52.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:52.250+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:52.251827+0000) 2019-10-13T12:41:53.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:53.554+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.557572+0000) 2019-10-13T12:41:53.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:53.552+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.557876+0000) 2019-10-13T12:41:53.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:53.554+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.558657+0000) 2019-10-13T12:41:53.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:53.558+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.560207+0000) 2019-10-13T12:41:53.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:53.558+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.560487+0000) 2019-10-13T12:41:53.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:53.558+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.560796+0000) 2019-10-13T12:41:53.563 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:53.558+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.561631+0000) 2019-10-13T12:41:53.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:53.562+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:53.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:53.564+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:53.567872+0000) 2019-10-13T12:41:55.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:55.522+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.526524+0000) 2019-10-13T12:41:56.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:56.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:56.494478+0000) 2019-10-13T12:41:58.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:41:58.566+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.569343+0000) 2019-10-13T12:41:58.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:58.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.569668+0000) 2019-10-13T12:41:58.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:41:58.566+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.570439+0000) 2019-10-13T12:41:58.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:41:58.566+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.570183+0000) 2019-10-13T12:41:58.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:58.566+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.571027+0000) 2019-10-13T12:41:58.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:58.570+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.571551+0000) 2019-10-13T12:41:58.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:58.570+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.572371+0000) 2019-10-13T12:41:58.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:41:58.574+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:41:58.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:41:58.572+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.577722+0000) 2019-10-13T12:41:58.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:41:58.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.759877+0000) 2019-10-13T12:41:58.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:41:58.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.763150+0000) 2019-10-13T12:41:59.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:41:59.058+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:59.062202+0000) 2019-10-13T12:41:59.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:41:59.992+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:59.995146+0000) 2019-10-13T12:42:01.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:01.032+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:01.033918+0000) 2019-10-13T12:42:01.984 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:42:01.984 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:01.988 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:42:01.988 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:02.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:02.250+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:02.252061+0000) 2019-10-13T12:42:03.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:03.578+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.579968+0000) 2019-10-13T12:42:03.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:03.578+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.580304+0000) 2019-10-13T12:42:03.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:03.578+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.581695+0000) 2019-10-13T12:42:03.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:03.580+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.582540+0000) 2019-10-13T12:42:03.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:03.578+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.583085+0000) 2019-10-13T12:42:03.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:03.582+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.583655+0000) 2019-10-13T12:42:03.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:03.582+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.584218+0000) 2019-10-13T12:42:03.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:03.584+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:03.588272+0000) 2019-10-13T12:42:03.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:03.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:05.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:05.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.526758+0000) 2019-10-13T12:42:06.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:06.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:06.494673+0000) 2019-10-13T12:42:08.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:08.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.592000+0000) 2019-10-13T12:42:08.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:08.590+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.592499+0000) 2019-10-13T12:42:08.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:08.590+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.593433+0000) 2019-10-13T12:42:08.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:08.590+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.594259+0000) 2019-10-13T12:42:08.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:08.592+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.594741+0000) 2019-10-13T12:42:08.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:08.590+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.594992+0000) 2019-10-13T12:42:08.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:08.594+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.595522+0000) 2019-10-13T12:42:08.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:08.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:08.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:08.600+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.602522+0000) 2019-10-13T12:42:08.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:08.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.760069+0000) 2019-10-13T12:42:08.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:08.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.763352+0000) 2019-10-13T12:42:09.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:09.062+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:09.062387+0000) 2019-10-13T12:42:09.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:09.992+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:09.995335+0000) 2019-10-13T12:42:11.047 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:11.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:11.045496+0000) 2019-10-13T12:42:12.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:12.250+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:12.252286+0000) 2019-10-13T12:42:13.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:13.598+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.603226+0000) 2019-10-13T12:42:13.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:13.602+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.604585+0000) 2019-10-13T12:42:13.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:13.600+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.605298+0000) 2019-10-13T12:42:13.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:13.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.604845+0000) 2019-10-13T12:42:13.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:13.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.606210+0000) 2019-10-13T12:42:13.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:13.602+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.607083+0000) 2019-10-13T12:42:13.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:13.606+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.607451+0000) 2019-10-13T12:42:13.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:13.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:13.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:13.612+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:13.614757+0000) 2019-10-13T12:42:15.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:15.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.526964+0000) 2019-10-13T12:42:16.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:16.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:16.494867+0000) 2019-10-13T12:42:18.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:18.614+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.615569+0000) 2019-10-13T12:42:18.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:18.614+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.616623+0000) 2019-10-13T12:42:18.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:18.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.616863+0000) 2019-10-13T12:42:18.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:18.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.617450+0000) 2019-10-13T12:42:18.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:18.611+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.617824+0000) 2019-10-13T12:42:18.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:18.614+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.618330+0000) 2019-10-13T12:42:18.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:18.614+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.618764+0000) 2019-10-13T12:42:18.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:18.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:18.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:18.623+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.627176+0000) 2019-10-13T12:42:18.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:18.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.760266+0000) 2019-10-13T12:42:18.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:18.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.763487+0000) 2019-10-13T12:42:19.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:19.058+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:19.062598+0000) 2019-10-13T12:42:19.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:19.991+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:19.995534+0000) 2019-10-13T12:42:21.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:21.055+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:21.057216+0000) 2019-10-13T12:42:22.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:22.250+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:22.252531+0000) 2019-10-13T12:42:23.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:23.621+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.627105+0000) 2019-10-13T12:42:23.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:23.625+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.627659+0000) 2019-10-13T12:42:23.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:23.625+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.628240+0000) 2019-10-13T12:42:23.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:23.625+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.629331+0000) 2019-10-13T12:42:23.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:23.625+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.629648+0000) 2019-10-13T12:42:23.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:23.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.630207+0000) 2019-10-13T12:42:23.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:23.627+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.631028+0000) 2019-10-13T12:42:23.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:23.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:23.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:23.631+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:23.635602+0000) 2019-10-13T12:42:25.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:25.521+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.527164+0000) 2019-10-13T12:42:26.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:26.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:26.495058+0000) 2019-10-13T12:42:28.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:28.633+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.638381+0000) 2019-10-13T12:42:28.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:28.633+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.639120+0000) 2019-10-13T12:42:28.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:28.637+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.639763+0000) 2019-10-13T12:42:28.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:28.637+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.640453+0000) 2019-10-13T12:42:28.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:28.637+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.640980+0000) 2019-10-13T12:42:28.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:28.635+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.641429+0000) 2019-10-13T12:42:28.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:28.637+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.642579+0000) 2019-10-13T12:42:28.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:28.643+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.646284+0000) 2019-10-13T12:42:28.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:28.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:28.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:28.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.760464+0000) 2019-10-13T12:42:28.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:28.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.763678+0000) 2019-10-13T12:42:29.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:29.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:29.062780+0000) 2019-10-13T12:42:29.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:29.991+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:29.995727+0000) 2019-10-13T12:42:31.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:31.059+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:31.063637+0000) 2019-10-13T12:42:32.128 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:42:32.129 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:32.134 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:42:32.134 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:32.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:32.249+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:32.252732+0000) 2019-10-13T12:42:33.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:33.645+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.649910+0000) 2019-10-13T12:42:33.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:33.645+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.650484+0000) 2019-10-13T12:42:33.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:33.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.651480+0000) 2019-10-13T12:42:33.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:33.649+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.652842+0000) 2019-10-13T12:42:33.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:33.649+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.653033+0000) 2019-10-13T12:42:33.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:33.647+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.653747+0000) 2019-10-13T12:42:33.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:33.649+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.653306+0000) 2019-10-13T12:42:33.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:33.651+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:33.657890+0000) 2019-10-13T12:42:33.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:33.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:35.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:35.525+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.527359+0000) 2019-10-13T12:42:36.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:36.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:36.495250+0000) 2019-10-13T12:42:38.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:38.657+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.661742+0000) 2019-10-13T12:42:38.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:38.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.662010+0000) 2019-10-13T12:42:38.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:38.661+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.663390+0000) 2019-10-13T12:42:38.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:38.661+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.663627+0000) 2019-10-13T12:42:38.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:38.661+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.664004+0000) 2019-10-13T12:42:38.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:38.661+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.664457+0000) 2019-10-13T12:42:38.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:38.659+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.664942+0000) 2019-10-13T12:42:38.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:38.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:38.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:38.667+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.671961+0000) 2019-10-13T12:42:38.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:38.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.760637+0000) 2019-10-13T12:42:38.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:38.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.763876+0000) 2019-10-13T12:42:39.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:39.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:39.062999+0000) 2019-10-13T12:42:39.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:39.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:39.995975+0000) 2019-10-13T12:42:41.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:41.067+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:41.070183+0000) 2019-10-13T12:42:42.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:42.249+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:42.252948+0000) 2019-10-13T12:42:43.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:43.669+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.671882+0000) 2019-10-13T12:42:43.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:43.669+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.673377+0000) 2019-10-13T12:42:43.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:43.669+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.673747+0000) 2019-10-13T12:42:43.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:43.671+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.674247+0000) 2019-10-13T12:42:43.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:43.669+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.674737+0000) 2019-10-13T12:42:43.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:43.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.675266+0000) 2019-10-13T12:42:43.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:43.673+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.675798+0000) 2019-10-13T12:42:43.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:43.675+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:43.679721+0000) 2019-10-13T12:42:43.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:43.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:45.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:45.525+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.527556+0000) 2019-10-13T12:42:46.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:46.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:46.495494+0000) 2019-10-13T12:42:48.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:48.681+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.683559+0000) 2019-10-13T12:42:48.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:48.681+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.683761+0000) 2019-10-13T12:42:48.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:48.681+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.684200+0000) 2019-10-13T12:42:48.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:48.681+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.684606+0000) 2019-10-13T12:42:48.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:48.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.685245+0000) 2019-10-13T12:42:48.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:48.681+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.685880+0000) 2019-10-13T12:42:48.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:48.681+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.686160+0000) 2019-10-13T12:42:48.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:48.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:48.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:48.687+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.692785+0000) 2019-10-13T12:42:48.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:48.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.760817+0000) 2019-10-13T12:42:48.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:48.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.764077+0000) 2019-10-13T12:42:49.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:49.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:49.063220+0000) 2019-10-13T12:42:49.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:49.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:49.996199+0000) 2019-10-13T12:42:51.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:51.079+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:51.080230+0000) 2019-10-13T12:42:52.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:52.249+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:52.253144+0000) 2019-10-13T12:42:53.696 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:53.689+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.694391+0000) 2019-10-13T12:42:53.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:53.689+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.694765+0000) 2019-10-13T12:42:53.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:53.693+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.695622+0000) 2019-10-13T12:42:53.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:53.693+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.695848+0000) 2019-10-13T12:42:53.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:53.691+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.697555+0000) 2019-10-13T12:42:53.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:53.693+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.697698+0000) 2019-10-13T12:42:53.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:53.693+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.698137+0000) 2019-10-13T12:42:53.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:53.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:53.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:53.699+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:53.702713+0000) 2019-10-13T12:42:55.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:55.525+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.527760+0000) 2019-10-13T12:42:56.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:56.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:56.495727+0000) 2019-10-13T12:42:58.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:58.699+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.704977+0000) 2019-10-13T12:42:58.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:58.701+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.705606+0000) 2019-10-13T12:42:58.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:42:58.701+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.706495+0000) 2019-10-13T12:42:58.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:58.701+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.706720+0000) 2019-10-13T12:42:58.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:58.705+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.707388+0000) 2019-10-13T12:42:58.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:42:58.705+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.707849+0000) 2019-10-13T12:42:58.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:42:58.705+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.708299+0000) 2019-10-13T12:42:58.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:42:58.707+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.712491+0000) 2019-10-13T12:42:58.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:42:58.709+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:42:58.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:42:58.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.760992+0000) 2019-10-13T12:42:58.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:42:58.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.764250+0000) 2019-10-13T12:42:59.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:42:59.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:59.063411+0000) 2019-10-13T12:42:59.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:42:59.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:59.996412+0000) 2019-10-13T12:43:01.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:01.083+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:01.086785+0000) 2019-10-13T12:43:02.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:02.249+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:02.253360+0000) 2019-10-13T12:43:02.265 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:43:02.266 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:02.269 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:43:02.269 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:03.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:03.713+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.716235+0000) 2019-10-13T12:43:03.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:03.711+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.716753+0000) 2019-10-13T12:43:03.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:03.713+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.717682+0000) 2019-10-13T12:43:03.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:03.713+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.716978+0000) 2019-10-13T12:43:03.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:03.713+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.717883+0000) 2019-10-13T12:43:03.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:03.713+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.718337+0000) 2019-10-13T12:43:03.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:03.717+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.719362+0000) 2019-10-13T12:43:03.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:03.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:03.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:03.719+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:03.725072+0000) 2019-10-13T12:43:05.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:05.524+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.527967+0000) 2019-10-13T12:43:06.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:06.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:06.495924+0000) 2019-10-13T12:43:08.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:08.720+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.726884+0000) 2019-10-13T12:43:08.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:08.724+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.727777+0000) 2019-10-13T12:43:08.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:08.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.728390+0000) 2019-10-13T12:43:08.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:08.724+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.728511+0000) 2019-10-13T12:43:08.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:08.723+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.729245+0000) 2019-10-13T12:43:08.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:08.724+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.729561+0000) 2019-10-13T12:43:08.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:08.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.729711+0000) 2019-10-13T12:43:08.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:08.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:08.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:08.731+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.736276+0000) 2019-10-13T12:43:08.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:08.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.761170+0000) 2019-10-13T12:43:08.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:08.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.764383+0000) 2019-10-13T12:43:09.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:09.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:09.063600+0000) 2019-10-13T12:43:09.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:09.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:09.996614+0000) 2019-10-13T12:43:11.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:11.091+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:11.093310+0000) 2019-10-13T12:43:12.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:12.252+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:12.253556+0000) 2019-10-13T12:43:13.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:13.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.738216+0000) 2019-10-13T12:43:13.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:13.736+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.740004+0000) 2019-10-13T12:43:13.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:13.736+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.740437+0000) 2019-10-13T12:43:13.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:13.736+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.740560+0000) 2019-10-13T12:43:13.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:13.736+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.740901+0000) 2019-10-13T12:43:13.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:13.734+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.741394+0000) 2019-10-13T12:43:13.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:13.736+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.741605+0000) 2019-10-13T12:43:13.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:13.738+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:13.744609+0000) 2019-10-13T12:43:13.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:13.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:15.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:15.524+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.528172+0000) 2019-10-13T12:43:16.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:16.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:16.496122+0000) 2019-10-13T12:43:18.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:18.744+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.751027+0000) 2019-10-13T12:43:18.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:18.748+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.751704+0000) 2019-10-13T12:43:18.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:18.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.751797+0000) 2019-10-13T12:43:18.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:18.746+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.752318+0000) 2019-10-13T12:43:18.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:18.748+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.753139+0000) 2019-10-13T12:43:18.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:18.748+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.753535+0000) 2019-10-13T12:43:18.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:18.748+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.753817+0000) 2019-10-13T12:43:18.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:18.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:18.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:18.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.761297+0000) 2019-10-13T12:43:18.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:18.754+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.761719+0000) 2019-10-13T12:43:18.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:18.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.764500+0000) 2019-10-13T12:43:19.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:19.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:19.063816+0000) 2019-10-13T12:43:19.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:19.994+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:19.996828+0000) 2019-10-13T12:43:21.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:21.102+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:21.104054+0000) 2019-10-13T12:43:22.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:22.252+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:22.253761+0000) 2019-10-13T12:43:23.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:23.756+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.762810+0000) 2019-10-13T12:43:23.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:23.760+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.763556+0000) 2019-10-13T12:43:23.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:23.760+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.764078+0000) 2019-10-13T12:43:23.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:23.760+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.764915+0000) 2019-10-13T12:43:23.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:23.760+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.765728+0000) 2019-10-13T12:43:23.768 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:23.762+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.766595+0000) 2019-10-13T12:43:23.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:23.760+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.766475+0000) 2019-10-13T12:43:23.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:23.764+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:23.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:23.766+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:23.771138+0000) 2019-10-13T12:43:25.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:25.524+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.528353+0000) 2019-10-13T12:43:26.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:26.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:26.496321+0000) 2019-10-13T12:43:28.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:28.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.761465+0000) 2019-10-13T12:43:28.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:28.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.764676+0000) 2019-10-13T12:43:28.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:28.768+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.773637+0000) 2019-10-13T12:43:28.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:28.768+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.774240+0000) 2019-10-13T12:43:28.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:28.768+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.774926+0000) 2019-10-13T12:43:28.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:28.770+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.775403+0000) 2019-10-13T12:43:28.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:28.772+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.776367+0000) 2019-10-13T12:43:28.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:28.772+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.776985+0000) 2019-10-13T12:43:28.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:28.772+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.777382+0000) 2019-10-13T12:43:28.782 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:28.774+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.781121+0000) 2019-10-13T12:43:28.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:28.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154706 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:29.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:29.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:29.064015+0000) 2019-10-13T12:43:29.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:29.990+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:29.997040+0000) 2019-10-13T12:43:31.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:31.114+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:31.115117+0000) 2019-10-13T12:43:32.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:32.248+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:32.253954+0000) 2019-10-13T12:43:32.393 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:43:32.393 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:32.397 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:43:32.397 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:33.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:33.780+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.786206+0000) 2019-10-13T12:43:33.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:33.780+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.786892+0000) 2019-10-13T12:43:33.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:33.784+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.787293+0000) 2019-10-13T12:43:33.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:33.782+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.787868+0000) 2019-10-13T12:43:33.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:33.784+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.787910+0000) 2019-10-13T12:43:33.790 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:33.784+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.788717+0000) 2019-10-13T12:43:33.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:33.784+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.789141+0000) 2019-10-13T12:43:33.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:33.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:33.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:33.786+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:33.791813+0000) 2019-10-13T12:43:35.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:35.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.528542+0000) 2019-10-13T12:43:36.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:36.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:36.496547+0000) 2019-10-13T12:43:38.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:38.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.761661+0000) 2019-10-13T12:43:38.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:38.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.764868+0000) 2019-10-13T12:43:38.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:38.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.797325+0000) 2019-10-13T12:43:38.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:38.792+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.797674+0000) 2019-10-13T12:43:38.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:38.794+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.799177+0000) 2019-10-13T12:43:38.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:38.792+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.799124+0000) 2019-10-13T12:43:38.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:38.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.799469+0000) 2019-10-13T12:43:38.802 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:38.796+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.801381+0000) 2019-10-13T12:43:38.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:38.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.802045+0000) 2019-10-13T12:43:38.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:38.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:38.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:38.802+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.807336+0000) 2019-10-13T12:43:39.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:39.060+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:39.064210+0000) 2019-10-13T12:43:39.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:39.994+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:39.997255+0000) 2019-10-13T12:43:41.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:41.118+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:41.121593+0000) 2019-10-13T12:43:42.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:42.252+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:42.254161+0000) 2019-10-13T12:43:43.810 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:43.804+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.808975+0000) 2019-10-13T12:43:43.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:43.802+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.809485+0000) 2019-10-13T12:43:43.811 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:43.804+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.810029+0000) 2019-10-13T12:43:43.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:43.804+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.810156+0000) 2019-10-13T12:43:43.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:43.804+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.810843+0000) 2019-10-13T12:43:43.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:43.808+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.812234+0000) 2019-10-13T12:43:43.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:43.808+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.812455+0000) 2019-10-13T12:43:43.818 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:43.812+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:43.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:43.814+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:43.818261+0000) 2019-10-13T12:43:45.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:45.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.528744+0000) 2019-10-13T12:43:46.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:46.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:46.496744+0000) 2019-10-13T12:43:48.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:48.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.761863+0000) 2019-10-13T12:43:48.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:48.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.765060+0000) 2019-10-13T12:43:48.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:48.815+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.820198+0000) 2019-10-13T12:43:48.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:48.815+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.820973+0000) 2019-10-13T12:43:48.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:48.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.821725+0000) 2019-10-13T12:43:48.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:48.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.821798+0000) 2019-10-13T12:43:48.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:48.818+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.822292+0000) 2019-10-13T12:43:48.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:48.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.822477+0000) 2019-10-13T12:43:48.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:48.815+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.822753+0000) 2019-10-13T12:43:48.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:48.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:48.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:48.826+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.830597+0000) 2019-10-13T12:43:49.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:49.063+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:49.064392+0000) 2019-10-13T12:43:49.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:49.994+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:49.997467+0000) 2019-10-13T12:43:51.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:51.130+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:51.131725+0000) 2019-10-13T12:43:52.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:52.251+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:52.254321+0000) 2019-10-13T12:43:53.832 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:53.823+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.830683+0000) 2019-10-13T12:43:53.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:53.823+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.831200+0000) 2019-10-13T12:43:53.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:53.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.832239+0000) 2019-10-13T12:43:53.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:53.827+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.833077+0000) 2019-10-13T12:43:53.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:53.827+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.833519+0000) 2019-10-13T12:43:53.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:53.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.834204+0000) 2019-10-13T12:43:53.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:53.830+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.834441+0000) 2019-10-13T12:43:53.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:53.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:53.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:53.834+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:53.841398+0000) 2019-10-13T12:43:55.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:55.527+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.528935+0000) 2019-10-13T12:43:56.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:56.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:56.496944+0000) 2019-10-13T12:43:58.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:58.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.762104+0000) 2019-10-13T12:43:58.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:58.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.765285+0000) 2019-10-13T12:43:58.843 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:43:58.835+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.841393+0000) 2019-10-13T12:43:58.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:43:58.835+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.841687+0000) 2019-10-13T12:43:58.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:43:58.835+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.842509+0000) 2019-10-13T12:43:58.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:58.835+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.843268+0000) 2019-10-13T12:43:58.845 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:43:58.839+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.843734+0000) 2019-10-13T12:43:58.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:43:58.839+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.844731+0000) 2019-10-13T12:43:58.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:58.842+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.845172+0000) 2019-10-13T12:43:58.851 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:43:58.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154904 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:43:58.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:43:58.846+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.849255+0000) 2019-10-13T12:43:59.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:43:59.063+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:59.064576+0000) 2019-10-13T12:43:59.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:43:59.994+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:59.997684+0000) 2019-10-13T12:44:01.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:01.134+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:01.138059+0000) 2019-10-13T12:44:02.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:02.247+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:02.254525+0000) 2019-10-13T12:44:02.488 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:44:02.488 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:02.492 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:44:02.492 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:03.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:03.850+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.853160+0000) 2019-10-13T12:44:03.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:03.847+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.853386+0000) 2019-10-13T12:44:03.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:03.847+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.854277+0000) 2019-10-13T12:44:03.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:03.847+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.854551+0000) 2019-10-13T12:44:03.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:03.851+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.855391+0000) 2019-10-13T12:44:03.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:03.851+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.856199+0000) 2019-10-13T12:44:03.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:03.851+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.856309+0000) 2019-10-13T12:44:03.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:03.858+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:03.860063+0000) 2019-10-13T12:44:03.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:03.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:05.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:05.523+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.529139+0000) 2019-10-13T12:44:06.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:06.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:06.497141+0000) 2019-10-13T12:44:08.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:08.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.762332+0000) 2019-10-13T12:44:08.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:08.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.765483+0000) 2019-10-13T12:44:08.865 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:08.859+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.862952+0000) 2019-10-13T12:44:08.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:08.861+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.864136+0000) 2019-10-13T12:44:08.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:08.859+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.864717+0000) 2019-10-13T12:44:08.867 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:08.859+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.865205+0000) 2019-10-13T12:44:08.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:08.859+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.865530+0000) 2019-10-13T12:44:08.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:08.859+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.865733+0000) 2019-10-13T12:44:08.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:08.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.866226+0000) 2019-10-13T12:44:08.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:08.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 154969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:08.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:08.873+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.875456+0000) 2019-10-13T12:44:09.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:09.063+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:09.064765+0000) 2019-10-13T12:44:09.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:09.993+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:09.997904+0000) 2019-10-13T12:44:11.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:11.141+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:11.144489+0000) 2019-10-13T12:44:12.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:12.251+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:12.254735+0000) 2019-10-13T12:44:13.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:13.873+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.875186+0000) 2019-10-13T12:44:13.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:13.871+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.875864+0000) 2019-10-13T12:44:13.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:13.871+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.876501+0000) 2019-10-13T12:44:13.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:13.871+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.876654+0000) 2019-10-13T12:44:13.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:13.871+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.876981+0000) 2019-10-13T12:44:13.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:13.871+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.877729+0000) 2019-10-13T12:44:13.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:13.871+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.878163+0000) 2019-10-13T12:44:13.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:13.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:13.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:13.881+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:13.885585+0000) 2019-10-13T12:44:15.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:15.527+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.529340+0000) 2019-10-13T12:44:16.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:16.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:16.497340+0000) 2019-10-13T12:44:18.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:18.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.762527+0000) 2019-10-13T12:44:18.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:18.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.765703+0000) 2019-10-13T12:44:18.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:18.883+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.886935+0000) 2019-10-13T12:44:18.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:18.883+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.887204+0000) 2019-10-13T12:44:18.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:18.885+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.887597+0000) 2019-10-13T12:44:18.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:18.887+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.888154+0000) 2019-10-13T12:44:18.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:18.887+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.888498+0000) 2019-10-13T12:44:18.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:18.887+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.889131+0000) 2019-10-13T12:44:18.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:18.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.889228+0000) 2019-10-13T12:44:18.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:18.893+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.894284+0000) 2019-10-13T12:44:18.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:18.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155035 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:19.066 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:19.063+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:19.064948+0000) 2019-10-13T12:44:19.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:19.993+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:19.998101+0000) 2019-10-13T12:44:21.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:21.149+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:21.151049+0000) 2019-10-13T12:44:22.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:22.251+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:22.254953+0000) 2019-10-13T12:44:23.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:23.895+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.898218+0000) 2019-10-13T12:44:23.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:23.895+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.898787+0000) 2019-10-13T12:44:23.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:23.895+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.899129+0000) 2019-10-13T12:44:23.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:23.899+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.899474+0000) 2019-10-13T12:44:23.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:23.897+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.899901+0000) 2019-10-13T12:44:23.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:23.899+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.900466+0000) 2019-10-13T12:44:23.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:23.899+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.901310+0000) 2019-10-13T12:44:23.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:23.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:23.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:23.905+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:23.906523+0000) 2019-10-13T12:44:25.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:25.527+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.529564+0000) 2019-10-13T12:44:26.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:26.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:26.497580+0000) 2019-10-13T12:44:28.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:28.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.762729+0000) 2019-10-13T12:44:28.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:28.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.765913+0000) 2019-10-13T12:44:28.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:28.907+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.908955+0000) 2019-10-13T12:44:28.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:28.909+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.910469+0000) 2019-10-13T12:44:28.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:28.907+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.910127+0000) 2019-10-13T12:44:28.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:28.907+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.910796+0000) 2019-10-13T12:44:28.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:28.911+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.911470+0000) 2019-10-13T12:44:28.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:28.911+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.911519+0000) 2019-10-13T12:44:28.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:28.911+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.912661+0000) 2019-10-13T12:44:28.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:28.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155101 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:28.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:28.917+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.918163+0000) 2019-10-13T12:44:29.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:29.063+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:29.065132+0000) 2019-10-13T12:44:30.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:29.997+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:29.998297+0000) 2019-10-13T12:44:31.159 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:31.153+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:31.157707+0000) 2019-10-13T12:44:32.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:32.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:32.255152+0000) 2019-10-13T12:44:32.606 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:44:32.607 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:32.610 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:44:32.610 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:33.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:33.918+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.919929+0000) 2019-10-13T12:44:33.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:33.917+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.920996+0000) 2019-10-13T12:44:33.922 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:33.918+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.920516+0000) 2019-10-13T12:44:33.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:33.918+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.922073+0000) 2019-10-13T12:44:33.924 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:33.918+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.922300+0000) 2019-10-13T12:44:33.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:33.922+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.923396+0000) 2019-10-13T12:44:33.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:33.922+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.923920+0000) 2019-10-13T12:44:33.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:33.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:33.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:33.929+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:33.933594+0000) 2019-10-13T12:44:35.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:35.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.529762+0000) 2019-10-13T12:44:36.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:36.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:36.497804+0000) 2019-10-13T12:44:38.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:38.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.762928+0000) 2019-10-13T12:44:38.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:38.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.766127+0000) 2019-10-13T12:44:38.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:38.930+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.932387+0000) 2019-10-13T12:44:38.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:38.930+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.933028+0000) 2019-10-13T12:44:38.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:38.930+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.933641+0000) 2019-10-13T12:44:38.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:38.930+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.934347+0000) 2019-10-13T12:44:38.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:38.930+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.934993+0000) 2019-10-13T12:44:38.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:38.930+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.935014+0000) 2019-10-13T12:44:38.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:38.933+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.935434+0000) 2019-10-13T12:44:38.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:38.937+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.940053+0000) 2019-10-13T12:44:38.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:38.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:39.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:39.062+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:39.065332+0000) 2019-10-13T12:44:40.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:39.997+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:39.998484+0000) 2019-10-13T12:44:41.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:41.161+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:41.164127+0000) 2019-10-13T12:44:42.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:42.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:42.255388+0000) 2019-10-13T12:44:43.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:43.942+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.943598+0000) 2019-10-13T12:44:43.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:43.942+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.943984+0000) 2019-10-13T12:44:43.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:43.942+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.944825+0000) 2019-10-13T12:44:43.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:43.942+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.945430+0000) 2019-10-13T12:44:43.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:43.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.946472+0000) 2019-10-13T12:44:43.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:43.942+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.947248+0000) 2019-10-13T12:44:43.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:43.945+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.947986+0000) 2019-10-13T12:44:43.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:43.949+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:43.950859+0000) 2019-10-13T12:44:43.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:43.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:45.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:45.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.529991+0000) 2019-10-13T12:44:46.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:46.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:46.497975+0000) 2019-10-13T12:44:48.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:48.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.763150+0000) 2019-10-13T12:44:48.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:48.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.766350+0000) 2019-10-13T12:44:48.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:48.950+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.955087+0000) 2019-10-13T12:44:48.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:48.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.955975+0000) 2019-10-13T12:44:48.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:48.954+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.955557+0000) 2019-10-13T12:44:48.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:48.954+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.956358+0000) 2019-10-13T12:44:48.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:48.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.956546+0000) 2019-10-13T12:44:48.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:48.954+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.957189+0000) 2019-10-13T12:44:48.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:48.954+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.957939+0000) 2019-10-13T12:44:48.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:48.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:48.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:48.961+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.963467+0000) 2019-10-13T12:44:49.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:49.062+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:49.065532+0000) 2019-10-13T12:44:50.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:49.997+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:49.998715+0000) 2019-10-13T12:44:51.172 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:51.169+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:51.170735+0000) 2019-10-13T12:44:52.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:52.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:52.255593+0000) 2019-10-13T12:44:53.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:53.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.966338+0000) 2019-10-13T12:44:53.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:53.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.967111+0000) 2019-10-13T12:44:53.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:53.965+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.967448+0000) 2019-10-13T12:44:53.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:53.966+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.967805+0000) 2019-10-13T12:44:53.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:53.966+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.968033+0000) 2019-10-13T12:44:53.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:53.966+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.968266+0000) 2019-10-13T12:44:53.971 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:53.966+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.969776+0000) 2019-10-13T12:44:53.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:53.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:53.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:53.973+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:53.974992+0000) 2019-10-13T12:44:55.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:55.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.530190+0000) 2019-10-13T12:44:56.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:56.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:56.498198+0000) 2019-10-13T12:44:58.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:58.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.763362+0000) 2019-10-13T12:44:58.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:58.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.766577+0000) 2019-10-13T12:44:58.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:44:58.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.976498+0000) 2019-10-13T12:44:58.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:58.973+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.976928+0000) 2019-10-13T12:44:58.979 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:44:58.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.977631+0000) 2019-10-13T12:44:58.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:44:58.974+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.978595+0000) 2019-10-13T12:44:58.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:44:58.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.979091+0000) 2019-10-13T12:44:58.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:44:58.978+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.979549+0000) 2019-10-13T12:44:58.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:58.978+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.980365+0000) 2019-10-13T12:44:58.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:44:58.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:44:58.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:44:58.985+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.986417+0000) 2019-10-13T12:44:59.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:44:59.062+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:59.065750+0000) 2019-10-13T12:45:00.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:44:59.997+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:59.998969+0000) 2019-10-13T12:45:01.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:01.176+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:01.177146+0000) 2019-10-13T12:45:02.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:02.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:02.255798+0000) 2019-10-13T12:45:02.663 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:45:02.663 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:02.666 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:45:02.667 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:03.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:03.986+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.988217+0000) 2019-10-13T12:45:03.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:03.986+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.988763+0000) 2019-10-13T12:45:03.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:03.986+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.989445+0000) 2019-10-13T12:45:03.992 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:03.986+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.990845+0000) 2019-10-13T12:45:03.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:03.986+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.991182+0000) 2019-10-13T12:45:03.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:03.988+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.991748+0000) 2019-10-13T12:45:03.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:03.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.992322+0000) 2019-10-13T12:45:03.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:03.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155333 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:03.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:03.992+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:03.996603+0000) 2019-10-13T12:45:05.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:05.526+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.530449+0000) 2019-10-13T12:45:06.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:06.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:06.498397+0000) 2019-10-13T12:45:08.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:08.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:08.763563+0000) 2019-10-13T12:45:08.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:08.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:08.766778+0000) 2019-10-13T12:45:09.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:08.998+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.000058+0000) 2019-10-13T12:45:09.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:08.998+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.000596+0000) 2019-10-13T12:45:09.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:08.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.000992+0000) 2019-10-13T12:45:09.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:08.998+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.001367+0000) 2019-10-13T12:45:09.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:08.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.002369+0000) 2019-10-13T12:45:09.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:09.000+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.002959+0000) 2019-10-13T12:45:09.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:09.002+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.003501+0000) 2019-10-13T12:45:09.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:09.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:09.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:09.008+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.011224+0000) 2019-10-13T12:45:09.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:09.062+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.065890+0000) 2019-10-13T12:45:10.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:09.996+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:09.999166+0000) 2019-10-13T12:45:11.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:11.180+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:11.183591+0000) 2019-10-13T12:45:12.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:12.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:12.256003+0000) 2019-10-13T12:45:14.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:14.005+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.010849+0000) 2019-10-13T12:45:14.013 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:14.009+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.011335+0000) 2019-10-13T12:45:14.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:14.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.012034+0000) 2019-10-13T12:45:14.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:14.009+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.012275+0000) 2019-10-13T12:45:14.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:14.009+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.013005+0000) 2019-10-13T12:45:14.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:14.009+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.013003+0000) 2019-10-13T12:45:14.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:14.009+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.013454+0000) 2019-10-13T12:45:14.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:14.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.017896+0000) 2019-10-13T12:45:14.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:14.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:15.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:15.525+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.530646+0000) 2019-10-13T12:45:16.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:16.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:16.498598+0000) 2019-10-13T12:45:18.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:18.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:18.763757+0000) 2019-10-13T12:45:18.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:18.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:18.766970+0000) 2019-10-13T12:45:19.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:19.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.022408+0000) 2019-10-13T12:45:19.025 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:19.017+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.021912+0000) 2019-10-13T12:45:19.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:19.017+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.023166+0000) 2019-10-13T12:45:19.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:19.021+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.023293+0000) 2019-10-13T12:45:19.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:19.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.023704+0000) 2019-10-13T12:45:19.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:19.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.024433+0000) 2019-10-13T12:45:19.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:19.021+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.024798+0000) 2019-10-13T12:45:19.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:19.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:19.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:19.028+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.030699+0000) 2019-10-13T12:45:19.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:19.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.066089+0000) 2019-10-13T12:45:20.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:19.996+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:19.999380+0000) 2019-10-13T12:45:21.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:21.184+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:21.189919+0000) 2019-10-13T12:45:22.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:22.253+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:22.256225+0000) 2019-10-13T12:45:24.034 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:24.029+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.032831+0000) 2019-10-13T12:45:24.035 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:24.029+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.033484+0000) 2019-10-13T12:45:24.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:24.029+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.034080+0000) 2019-10-13T12:45:24.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:24.032+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.034341+0000) 2019-10-13T12:45:24.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:24.033+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.035636+0000) 2019-10-13T12:45:24.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:24.033+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.036042+0000) 2019-10-13T12:45:24.038 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:24.033+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.036639+0000) 2019-10-13T12:45:24.042 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:24.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:24.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:24.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.041692+0000) 2019-10-13T12:45:25.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:25.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.530880+0000) 2019-10-13T12:45:26.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:26.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:26.498799+0000) 2019-10-13T12:45:28.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:28.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:28.763947+0000) 2019-10-13T12:45:28.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:28.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:28.767181+0000) 2019-10-13T12:45:29.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:29.041+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.043991+0000) 2019-10-13T12:45:29.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:29.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.044428+0000) 2019-10-13T12:45:29.047 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:29.041+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.045610+0000) 2019-10-13T12:45:29.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:29.044+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.046184+0000) 2019-10-13T12:45:29.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:29.041+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.046302+0000) 2019-10-13T12:45:29.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:29.045+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.047473+0000) 2019-10-13T12:45:29.049 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:29.045+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.047406+0000) 2019-10-13T12:45:29.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:29.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155498 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:29.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:29.048+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.053323+0000) 2019-10-13T12:45:29.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:29.061+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.066214+0000) 2019-10-13T12:45:30.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:29.996+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:29.999607+0000) 2019-10-13T12:45:31.198 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:31.192+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:31.196601+0000) 2019-10-13T12:45:32.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:32.253+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:32.256437+0000) 2019-10-13T12:45:32.771 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:45:32.771 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:32.775 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:45:32.775 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:34.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:34.053+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.055745+0000) 2019-10-13T12:45:34.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:34.053+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.056224+0000) 2019-10-13T12:45:34.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:34.053+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.056812+0000) 2019-10-13T12:45:34.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:34.053+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.057178+0000) 2019-10-13T12:45:34.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:34.052+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.057913+0000) 2019-10-13T12:45:34.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:34.053+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.058084+0000) 2019-10-13T12:45:34.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:34.057+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.059399+0000) 2019-10-13T12:45:34.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:34.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155534 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:34.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:34.060+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.064490+0000) 2019-10-13T12:45:35.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:35.525+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.531106+0000) 2019-10-13T12:45:36.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:36.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:36.499026+0000) 2019-10-13T12:45:38.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:38.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:38.764167+0000) 2019-10-13T12:45:38.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:38.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:38.767374+0000) 2019-10-13T12:45:39.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:39.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.065980+0000) 2019-10-13T12:45:39.068 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:39.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.067248+0000) 2019-10-13T12:45:39.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:39.064+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.067721+0000) 2019-10-13T12:45:39.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:39.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.068007+0000) 2019-10-13T12:45:39.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:39.065+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.069109+0000) 2019-10-13T12:45:39.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:39.065+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.069435+0000) 2019-10-13T12:45:39.072 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:39.065+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.071076+0000) 2019-10-13T12:45:39.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:39.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:39.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:39.072+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.076918+0000) 2019-10-13T12:45:40.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:39.996+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:39.999781+0000) 2019-10-13T12:45:41.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:41.208+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:41.208306+0000) 2019-10-13T12:45:42.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:42.253+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:42.256637+0000) 2019-10-13T12:45:44.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:44.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.078560+0000) 2019-10-13T12:45:44.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:44.073+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.078933+0000) 2019-10-13T12:45:44.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:44.077+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.079357+0000) 2019-10-13T12:45:44.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:44.077+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.080278+0000) 2019-10-13T12:45:44.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:44.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.080910+0000) 2019-10-13T12:45:44.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:44.077+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.081029+0000) 2019-10-13T12:45:44.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:44.077+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.081566+0000) 2019-10-13T12:45:44.087 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:44.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155601 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:44.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:44.084+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.086297+0000) 2019-10-13T12:45:45.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:45.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.531331+0000) 2019-10-13T12:45:45.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:45.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.816834+0000) 2019-10-13T12:45:46.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:46.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:46.499228+0000) 2019-10-13T12:45:48.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:48.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:48.764522+0000) 2019-10-13T12:45:48.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:48.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:48.767559+0000) 2019-10-13T12:45:49.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:49.085+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.089054+0000) 2019-10-13T12:45:49.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:49.085+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.090225+0000) 2019-10-13T12:45:49.092 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:49.085+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.090691+0000) 2019-10-13T12:45:49.092 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:49.085+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.091162+0000) 2019-10-13T12:45:49.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:49.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.091537+0000) 2019-10-13T12:45:49.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:49.089+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.092091+0000) 2019-10-13T12:45:49.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:49.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.092541+0000) 2019-10-13T12:45:49.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:49.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:49.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:49.096+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:49.098853+0000) 2019-10-13T12:45:50.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:49.996+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.000012+0000) 2019-10-13T12:45:51.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:51.212+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:51.214772+0000) 2019-10-13T12:45:52.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:52.253+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:52.256878+0000) 2019-10-13T12:45:54.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:54.097+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.101372+0000) 2019-10-13T12:45:54.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:54.097+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.101608+0000) 2019-10-13T12:45:54.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:54.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.102042+0000) 2019-10-13T12:45:54.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:54.097+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.102624+0000) 2019-10-13T12:45:54.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:54.097+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.103029+0000) 2019-10-13T12:45:54.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:54.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.103606+0000) 2019-10-13T12:45:54.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:54.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.103715+0000) 2019-10-13T12:45:54.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:54.105+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:54.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:54.108+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.110801+0000) 2019-10-13T12:45:55.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:55.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.531542+0000) 2019-10-13T12:45:55.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:55.813+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.817035+0000) 2019-10-13T12:45:56.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:56.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:56.499447+0000) 2019-10-13T12:45:58.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:58.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:58.764736+0000) 2019-10-13T12:45:58.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:58.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:58.767777+0000) 2019-10-13T12:45:59.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:59.107+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.111585+0000) 2019-10-13T12:45:59.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:45:59.108+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.111891+0000) 2019-10-13T12:45:59.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:45:59.108+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.112781+0000) 2019-10-13T12:45:59.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:45:59.108+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.113831+0000) 2019-10-13T12:45:59.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:45:59.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.114052+0000) 2019-10-13T12:45:59.116 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:45:59.108+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.114551+0000) 2019-10-13T12:45:59.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:45:59.108+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.114904+0000) 2019-10-13T12:45:59.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:45:59.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:45:59.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:45:59.119+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:59.122348+0000) 2019-10-13T12:46:00.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:45:59.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.000153+0000) 2019-10-13T12:46:01.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:01.223+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:01.226578+0000) 2019-10-13T12:46:02.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:02.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:02.257051+0000) 2019-10-13T12:46:02.883 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:46:02.883 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:02.887 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:46:02.887 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:04.125 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:04.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.124026+0000) 2019-10-13T12:46:04.126 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:04.120+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.124325+0000) 2019-10-13T12:46:04.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:04.120+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.124800+0000) 2019-10-13T12:46:04.127 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:04.120+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.124949+0000) 2019-10-13T12:46:04.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:04.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.125350+0000) 2019-10-13T12:46:04.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:04.119+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.125979+0000) 2019-10-13T12:46:04.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:04.120+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.126530+0000) 2019-10-13T12:46:04.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:04.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:04.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:04.127+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.133503+0000) 2019-10-13T12:46:05.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:05.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.531693+0000) 2019-10-13T12:46:05.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:05.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.817266+0000) 2019-10-13T12:46:06.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:06.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:06.499680+0000) 2019-10-13T12:46:08.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:08.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:08.764929+0000) 2019-10-13T12:46:08.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:08.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:08.767972+0000) 2019-10-13T12:46:09.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:09.128+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.135008+0000) 2019-10-13T12:46:09.137 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:09.132+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.135498+0000) 2019-10-13T12:46:09.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:09.132+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.135829+0000) 2019-10-13T12:46:09.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:09.132+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.136406+0000) 2019-10-13T12:46:09.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:09.132+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.136679+0000) 2019-10-13T12:46:09.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:09.132+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.137824+0000) 2019-10-13T12:46:09.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:09.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.138186+0000) 2019-10-13T12:46:09.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:09.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:09.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:09.139+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:09.143347+0000) 2019-10-13T12:46:10.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:09.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.000318+0000) 2019-10-13T12:46:11.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:11.235+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:11.238073+0000) 2019-10-13T12:46:12.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:12.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:12.257265+0000) 2019-10-13T12:46:14.147 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:14.140+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.145400+0000) 2019-10-13T12:46:14.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:14.140+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.145708+0000) 2019-10-13T12:46:14.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:14.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.146211+0000) 2019-10-13T12:46:14.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:14.140+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.147091+0000) 2019-10-13T12:46:14.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:14.144+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.147743+0000) 2019-10-13T12:46:14.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:14.144+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.148115+0000) 2019-10-13T12:46:14.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:14.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.148941+0000) 2019-10-13T12:46:14.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:14.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:14.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:14.151+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.156088+0000) 2019-10-13T12:46:15.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:15.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.531894+0000) 2019-10-13T12:46:15.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:15.812+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.817471+0000) 2019-10-13T12:46:16.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:16.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:16.499908+0000) 2019-10-13T12:46:18.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:18.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:18.765127+0000) 2019-10-13T12:46:18.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:18.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:18.768164+0000) 2019-10-13T12:46:19.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:19.152+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.156985+0000) 2019-10-13T12:46:19.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:19.152+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.157362+0000) 2019-10-13T12:46:19.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:19.152+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.157850+0000) 2019-10-13T12:46:19.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:19.152+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.158313+0000) 2019-10-13T12:46:19.160 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:19.152+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.158773+0000) 2019-10-13T12:46:19.161 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:19.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.159273+0000) 2019-10-13T12:46:19.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:19.156+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.160024+0000) 2019-10-13T12:46:19.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:19.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:19.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:19.159+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:19.166102+0000) 2019-10-13T12:46:20.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:19.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.000515+0000) 2019-10-13T12:46:21.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:21.239+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:21.244481+0000) 2019-10-13T12:46:22.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:22.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:22.257503+0000) 2019-10-13T12:46:24.168 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:24.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.167397+0000) 2019-10-13T12:46:24.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:24.164+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.167636+0000) 2019-10-13T12:46:24.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:24.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.168370+0000) 2019-10-13T12:46:24.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:24.164+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.169318+0000) 2019-10-13T12:46:24.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:24.164+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.169682+0000) 2019-10-13T12:46:24.172 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:24.164+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.170171+0000) 2019-10-13T12:46:24.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:24.164+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.170467+0000) 2019-10-13T12:46:24.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:24.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:24.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:24.171+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.175392+0000) 2019-10-13T12:46:25.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:25.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.532093+0000) 2019-10-13T12:46:25.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:25.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.817674+0000) 2019-10-13T12:46:26.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:26.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:26.500122+0000) 2019-10-13T12:46:28.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:28.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:28.765348+0000) 2019-10-13T12:46:28.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:28.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:28.768360+0000) 2019-10-13T12:46:29.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:29.172+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.178115+0000) 2019-10-13T12:46:29.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:29.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.178545+0000) 2019-10-13T12:46:29.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:29.172+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.178902+0000) 2019-10-13T12:46:29.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:29.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.179600+0000) 2019-10-13T12:46:29.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:29.176+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.179924+0000) 2019-10-13T12:46:29.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:29.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.181261+0000) 2019-10-13T12:46:29.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:29.176+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.182001+0000) 2019-10-13T12:46:29.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:29.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:29.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:29.183+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:29.187018+0000) 2019-10-13T12:46:30.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:29.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.000702+0000) 2019-10-13T12:46:31.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:31.247+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:31.253452+0000) 2019-10-13T12:46:32.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:32.252+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:32.257697+0000) 2019-10-13T12:46:33.023 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:46:33.024 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:33.027 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:46:33.027 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:34.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:34.187+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.190460+0000) 2019-10-13T12:46:34.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:34.184+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.190076+0000) 2019-10-13T12:46:34.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:34.184+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.190830+0000) 2019-10-13T12:46:34.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:34.184+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.191211+0000) 2019-10-13T12:46:34.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:34.188+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.191567+0000) 2019-10-13T12:46:34.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:34.188+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.192172+0000) 2019-10-13T12:46:34.194 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:34.188+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.192654+0000) 2019-10-13T12:46:34.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:34.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:34.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:34.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.202332+0000) 2019-10-13T12:46:35.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:35.528+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.532317+0000) 2019-10-13T12:46:35.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:35.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.817883+0000) 2019-10-13T12:46:36.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:36.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:36.500323+0000) 2019-10-13T12:46:38.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:38.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:38.765536+0000) 2019-10-13T12:46:38.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:38.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:38.768555+0000) 2019-10-13T12:46:39.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:39.196+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.201190+0000) 2019-10-13T12:46:39.203 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:39.195+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.201945+0000) 2019-10-13T12:46:39.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:39.196+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.202124+0000) 2019-10-13T12:46:39.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:39.196+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.202648+0000) 2019-10-13T12:46:39.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:39.200+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.203431+0000) 2019-10-13T12:46:39.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:39.200+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.203998+0000) 2019-10-13T12:46:39.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:39.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.204695+0000) 2019-10-13T12:46:39.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:39.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 155975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:39.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:39.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:39.210340+0000) 2019-10-13T12:46:40.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:39.995+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.000898+0000) 2019-10-13T12:46:41.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:41.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:41.259745+0000) 2019-10-13T12:46:42.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:42.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:42.257871+0000) 2019-10-13T12:46:44.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:44.207+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.213363+0000) 2019-10-13T12:46:44.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:44.207+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.213640+0000) 2019-10-13T12:46:44.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:44.207+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.214025+0000) 2019-10-13T12:46:44.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:44.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.214389+0000) 2019-10-13T12:46:44.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:44.207+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.214813+0000) 2019-10-13T12:46:44.217 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:44.207+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.215106+0000) 2019-10-13T12:46:44.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:44.211+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.215577+0000) 2019-10-13T12:46:44.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:44.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:44.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:44.219+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.222588+0000) 2019-10-13T12:46:45.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:45.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.532531+0000) 2019-10-13T12:46:45.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:45.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.818099+0000) 2019-10-13T12:46:46.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:46.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:46.500534+0000) 2019-10-13T12:46:48.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:48.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:48.765693+0000) 2019-10-13T12:46:48.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:48.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:48.768779+0000) 2019-10-13T12:46:49.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:49.219+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.223239+0000) 2019-10-13T12:46:49.225 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:49.219+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.223839+0000) 2019-10-13T12:46:49.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:49.219+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.224341+0000) 2019-10-13T12:46:49.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:49.219+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.224818+0000) 2019-10-13T12:46:49.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:49.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.225082+0000) 2019-10-13T12:46:49.227 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:49.219+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.225831+0000) 2019-10-13T12:46:49.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:49.222+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.226847+0000) 2019-10-13T12:46:49.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:49.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:49.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:49.226+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:49.231777+0000) 2019-10-13T12:46:50.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:49.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.001084+0000) 2019-10-13T12:46:51.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:51.262+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:51.266181+0000) 2019-10-13T12:46:52.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:52.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:52.258084+0000) 2019-10-13T12:46:54.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:54.227+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.234454+0000) 2019-10-13T12:46:54.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:54.227+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.234834+0000) 2019-10-13T12:46:54.237 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:54.231+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.235522+0000) 2019-10-13T12:46:54.238 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:54.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.236396+0000) 2019-10-13T12:46:54.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:54.231+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.236933+0000) 2019-10-13T12:46:54.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:54.231+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.237347+0000) 2019-10-13T12:46:54.239 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:54.230+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.237855+0000) 2019-10-13T12:46:54.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:54.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:54.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:54.238+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.242666+0000) 2019-10-13T12:46:55.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:55.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.532753+0000) 2019-10-13T12:46:55.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:55.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.818294+0000) 2019-10-13T12:46:56.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:56.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:56.500756+0000) 2019-10-13T12:46:58.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:58.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:58.765916+0000) 2019-10-13T12:46:58.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:58.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:58.768992+0000) 2019-10-13T12:46:59.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:46:59.239+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.245624+0000) 2019-10-13T12:46:59.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:46:59.239+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.246484+0000) 2019-10-13T12:46:59.248 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:46:59.239+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.246949+0000) 2019-10-13T12:46:59.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:46:59.239+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.247111+0000) 2019-10-13T12:46:59.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:46:59.243+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.247696+0000) 2019-10-13T12:46:59.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:46:59.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.248510+0000) 2019-10-13T12:46:59.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:59.242+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.249133+0000) 2019-10-13T12:46:59.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:46:59.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:46:59.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:46:59.250+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:59.254893+0000) 2019-10-13T12:47:00.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:46:59.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.001253+0000) 2019-10-13T12:47:01.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:01.270+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:01.272873+0000) 2019-10-13T12:47:02.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:02.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:02.258258+0000) 2019-10-13T12:47:03.381 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:47:03.381 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:03.385 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:47:03.385 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:04.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:04.251+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.256023+0000) 2019-10-13T12:47:04.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:04.250+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.256489+0000) 2019-10-13T12:47:04.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:04.251+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.256634+0000) 2019-10-13T12:47:04.259 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:04.251+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.257369+0000) 2019-10-13T12:47:04.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:04.251+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.257979+0000) 2019-10-13T12:47:04.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:04.255+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.259486+0000) 2019-10-13T12:47:04.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:04.255+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.259532+0000) 2019-10-13T12:47:04.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:04.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:04.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:04.258+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.265647+0000) 2019-10-13T12:47:05.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:05.527+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.532951+0000) 2019-10-13T12:47:05.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:05.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.818495+0000) 2019-10-13T12:47:06.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:06.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:06.500991+0000) 2019-10-13T12:47:08.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:08.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:08.766133+0000) 2019-10-13T12:47:08.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:08.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:08.769213+0000) 2019-10-13T12:47:09.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:09.263+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.266224+0000) 2019-10-13T12:47:09.268 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:09.263+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.267059+0000) 2019-10-13T12:47:09.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:09.263+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.268214+0000) 2019-10-13T12:47:09.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:09.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.268952+0000) 2019-10-13T12:47:09.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:09.263+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.269156+0000) 2019-10-13T12:47:09.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:09.263+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.269751+0000) 2019-10-13T12:47:09.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:09.263+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.270531+0000) 2019-10-13T12:47:09.276 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:09.270+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:09.274562+0000) 2019-10-13T12:47:09.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:09.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:10.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:09.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.001423+0000) 2019-10-13T12:47:11.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:11.282+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:11.284691+0000) 2019-10-13T12:47:12.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:12.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:12.258460+0000) 2019-10-13T12:47:14.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:14.275+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.277996+0000) 2019-10-13T12:47:14.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:14.279+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.279694+0000) 2019-10-13T12:47:14.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:14.279+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.280067+0000) 2019-10-13T12:47:14.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:14.279+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.280435+0000) 2019-10-13T12:47:14.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:14.279+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.280892+0000) 2019-10-13T12:47:14.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:14.274+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.281735+0000) 2019-10-13T12:47:14.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:14.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.282240+0000) 2019-10-13T12:47:14.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:14.278+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.285708+0000) 2019-10-13T12:47:14.287 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:14.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156211 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:15.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:15.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.533153+0000) 2019-10-13T12:47:15.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:15.815+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.818704+0000) 2019-10-13T12:47:16.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:16.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:16.501181+0000) 2019-10-13T12:47:18.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:18.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:18.766362+0000) 2019-10-13T12:47:18.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:18.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:18.769442+0000) 2019-10-13T12:47:19.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:19.287+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.289894+0000) 2019-10-13T12:47:19.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:19.287+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.290408+0000) 2019-10-13T12:47:19.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:19.287+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.291101+0000) 2019-10-13T12:47:19.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:19.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.291679+0000) 2019-10-13T12:47:19.293 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:19.291+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.291779+0000) 2019-10-13T12:47:19.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:19.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.292657+0000) 2019-10-13T12:47:19.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:19.291+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.293621+0000) 2019-10-13T12:47:19.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:19.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156246 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:19.299 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:19.290+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:19.297944+0000) 2019-10-13T12:47:20.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:19.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.001594+0000) 2019-10-13T12:47:21.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:21.290+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:21.291232+0000) 2019-10-13T12:47:22.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:22.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:22.258643+0000) 2019-10-13T12:47:24.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:24.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.299935+0000) 2019-10-13T12:47:24.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:24.298+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.300193+0000) 2019-10-13T12:47:24.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:24.298+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.300788+0000) 2019-10-13T12:47:24.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:24.298+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.301383+0000) 2019-10-13T12:47:24.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:24.298+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.301665+0000) 2019-10-13T12:47:24.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:24.298+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.302257+0000) 2019-10-13T12:47:24.304 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:24.298+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.302588+0000) 2019-10-13T12:47:24.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:24.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:24.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:24.306+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.311980+0000) 2019-10-13T12:47:25.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:25.530+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.533352+0000) 2019-10-13T12:47:25.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:25.814+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.818940+0000) 2019-10-13T12:47:26.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:26.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:26.501385+0000) 2019-10-13T12:47:28.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:28.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:28.766559+0000) 2019-10-13T12:47:28.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:28.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:28.769669+0000) 2019-10-13T12:47:29.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:29.306+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.311135+0000) 2019-10-13T12:47:29.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:29.310+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.311367+0000) 2019-10-13T12:47:29.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:29.310+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.312057+0000) 2019-10-13T12:47:29.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:29.306+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.313867+0000) 2019-10-13T12:47:29.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:29.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.313962+0000) 2019-10-13T12:47:29.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:29.310+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.314349+0000) 2019-10-13T12:47:29.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:29.310+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.314693+0000) 2019-10-13T12:47:29.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:29.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:29.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:29.314+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:29.319802+0000) 2019-10-13T12:47:30.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:29.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.001770+0000) 2019-10-13T12:47:31.299 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:31.294+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:31.297626+0000) 2019-10-13T12:47:32.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:32.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:32.258816+0000) 2019-10-13T12:47:33.512 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:47:33.512 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:33.516 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:47:33.516 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:34.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:34.318+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.322477+0000) 2019-10-13T12:47:34.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:34.318+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.323166+0000) 2019-10-13T12:47:34.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:34.322+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.323672+0000) 2019-10-13T12:47:34.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:34.322+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.324221+0000) 2019-10-13T12:47:34.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:34.318+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.324916+0000) 2019-10-13T12:47:34.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:34.322+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.325503+0000) 2019-10-13T12:47:34.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:34.322+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.325193+0000) 2019-10-13T12:47:34.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:34.326+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:34.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:34.326+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.330674+0000) 2019-10-13T12:47:35.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:35.530+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.533571+0000) 2019-10-13T12:47:35.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:35.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.819149+0000) 2019-10-13T12:47:36.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:36.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:36.501592+0000) 2019-10-13T12:47:38.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:38.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:38.766787+0000) 2019-10-13T12:47:38.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:38.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:38.769888+0000) 2019-10-13T12:47:39.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:39.330+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.332977+0000) 2019-10-13T12:47:39.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:39.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.333359+0000) 2019-10-13T12:47:39.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:39.330+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.333620+0000) 2019-10-13T12:47:39.336 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:39.330+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.334219+0000) 2019-10-13T12:47:39.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:39.330+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.334715+0000) 2019-10-13T12:47:39.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:39.330+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.335256+0000) 2019-10-13T12:47:39.337 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:39.330+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.334970+0000) 2019-10-13T12:47:39.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:39.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:39.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:39.342+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:39.342672+0000) 2019-10-13T12:47:40.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:39.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.001939+0000) 2019-10-13T12:47:41.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:41.302+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:41.304355+0000) 2019-10-13T12:47:42.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:42.254+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:42.259006+0000) 2019-10-13T12:47:44.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:44.342+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.343553+0000) 2019-10-13T12:47:44.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:44.342+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.343914+0000) 2019-10-13T12:47:44.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:44.342+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.344319+0000) 2019-10-13T12:47:44.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:44.342+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.344997+0000) 2019-10-13T12:47:44.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:44.341+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.345831+0000) 2019-10-13T12:47:44.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:44.342+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.346249+0000) 2019-10-13T12:47:44.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:44.342+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.346753+0000) 2019-10-13T12:47:44.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:44.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:44.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:44.349+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.354126+0000) 2019-10-13T12:47:45.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:45.530+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.533797+0000) 2019-10-13T12:47:45.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:45.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.819391+0000) 2019-10-13T12:47:46.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:46.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:46.501781+0000) 2019-10-13T12:47:48.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:48.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:48.766983+0000) 2019-10-13T12:47:48.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:48.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:48.770117+0000) 2019-10-13T12:47:49.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:49.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.354515+0000) 2019-10-13T12:47:49.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:49.350+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.354954+0000) 2019-10-13T12:47:49.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:49.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.355824+0000) 2019-10-13T12:47:49.357 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:49.354+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.356037+0000) 2019-10-13T12:47:49.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:49.354+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.356678+0000) 2019-10-13T12:47:49.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:49.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.357132+0000) 2019-10-13T12:47:49.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:49.354+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.358064+0000) 2019-10-13T12:47:49.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:49.357+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:49.361936+0000) 2019-10-13T12:47:49.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:49.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:50.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:49.997+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.002106+0000) 2019-10-13T12:47:51.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:51.309+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:51.310959+0000) 2019-10-13T12:47:52.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:52.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:52.259194+0000) 2019-10-13T12:47:54.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:54.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.365579+0000) 2019-10-13T12:47:54.368 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:54.362+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.366727+0000) 2019-10-13T12:47:54.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:54.366+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.367478+0000) 2019-10-13T12:47:54.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:54.365+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.368234+0000) 2019-10-13T12:47:54.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:54.366+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.368710+0000) 2019-10-13T12:47:54.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:54.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.368907+0000) 2019-10-13T12:47:54.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:54.366+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.369649+0000) 2019-10-13T12:47:54.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:54.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:54.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:54.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.374481+0000) 2019-10-13T12:47:55.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:55.530+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.533993+0000) 2019-10-13T12:47:55.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:55.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.819597+0000) 2019-10-13T12:47:56.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:56.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:56.502001+0000) 2019-10-13T12:47:58.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:58.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:58.767190+0000) 2019-10-13T12:47:58.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:58.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:58.770310+0000) 2019-10-13T12:47:59.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:47:59.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.376895+0000) 2019-10-13T12:47:59.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:47:59.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.378221+0000) 2019-10-13T12:47:59.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:47:59.374+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.378793+0000) 2019-10-13T12:47:59.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:47:59.377+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.379221+0000) 2019-10-13T12:47:59.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:47:59.378+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.379624+0000) 2019-10-13T12:47:59.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:47:59.378+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.379953+0000) 2019-10-13T12:47:59.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:47:59.378+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.380389+0000) 2019-10-13T12:47:59.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:47:59.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:47:59.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:47:59.385+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:59.389367+0000) 2019-10-13T12:48:00.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:00.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.002273+0000) 2019-10-13T12:48:01.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:01.313+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:01.317418+0000) 2019-10-13T12:48:02.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:02.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:02.259354+0000) 2019-10-13T12:48:03.636 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:48:03.636 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:03.640 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:48:03.640 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:04.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:04.386+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.389023+0000) 2019-10-13T12:48:04.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:04.385+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.389396+0000) 2019-10-13T12:48:04.392 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:04.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.389645+0000) 2019-10-13T12:48:04.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:04.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.390156+0000) 2019-10-13T12:48:04.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:04.386+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.390423+0000) 2019-10-13T12:48:04.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:04.390+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.391651+0000) 2019-10-13T12:48:04.394 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:04.390+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.392175+0000) 2019-10-13T12:48:04.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:04.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:04.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:04.397+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.398194+0000) 2019-10-13T12:48:05.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:05.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.534181+0000) 2019-10-13T12:48:05.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:05.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.819802+0000) 2019-10-13T12:48:06.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:06.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:06.502194+0000) 2019-10-13T12:48:08.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:08.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:08.767386+0000) 2019-10-13T12:48:08.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:08.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:08.770502+0000) 2019-10-13T12:48:09.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:09.397+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.399525+0000) 2019-10-13T12:48:09.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:09.397+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.399840+0000) 2019-10-13T12:48:09.402 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:09.397+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.400888+0000) 2019-10-13T12:48:09.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:09.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.401306+0000) 2019-10-13T12:48:09.403 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:09.397+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.401897+0000) 2019-10-13T12:48:09.404 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:09.397+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.402128+0000) 2019-10-13T12:48:09.404 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:09.397+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.402947+0000) 2019-10-13T12:48:09.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:09.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156585 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:09.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:09.405+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:09.407836+0000) 2019-10-13T12:48:10.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:10.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.002440+0000) 2019-10-13T12:48:11.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:11.321+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:11.323879+0000) 2019-10-13T12:48:12.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:12.257+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:12.259525+0000) 2019-10-13T12:48:14.412 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:14.405+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.410981+0000) 2019-10-13T12:48:14.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:14.409+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.411621+0000) 2019-10-13T12:48:14.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:14.409+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.411912+0000) 2019-10-13T12:48:14.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:14.409+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.413199+0000) 2019-10-13T12:48:14.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:14.409+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.413518+0000) 2019-10-13T12:48:14.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:14.409+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.414281+0000) 2019-10-13T12:48:14.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:14.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.414692+0000) 2019-10-13T12:48:14.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:14.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156616 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:14.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:14.421+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.425054+0000) 2019-10-13T12:48:15.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:15.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.534378+0000) 2019-10-13T12:48:15.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:15.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.820005+0000) 2019-10-13T12:48:16.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:16.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:16.502387+0000) 2019-10-13T12:48:18.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:18.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:18.767624+0000) 2019-10-13T12:48:18.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:18.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:18.770728+0000) 2019-10-13T12:48:19.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:19.417+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.422380+0000) 2019-10-13T12:48:19.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:19.417+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.422732+0000) 2019-10-13T12:48:19.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:19.421+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.423341+0000) 2019-10-13T12:48:19.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:19.421+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.423637+0000) 2019-10-13T12:48:19.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:19.421+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.424347+0000) 2019-10-13T12:48:19.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:19.421+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.424865+0000) 2019-10-13T12:48:19.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:19.421+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.424985+0000) 2019-10-13T12:48:19.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:19.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:19.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:19.433+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:19.435685+0000) 2019-10-13T12:48:20.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:20.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.002609+0000) 2019-10-13T12:48:21.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:21.329+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:21.330382+0000) 2019-10-13T12:48:22.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:22.257+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:22.259707+0000) 2019-10-13T12:48:24.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:24.429+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.433557+0000) 2019-10-13T12:48:24.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:24.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.434542+0000) 2019-10-13T12:48:24.436 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:24.429+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.434871+0000) 2019-10-13T12:48:24.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:24.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.435381+0000) 2019-10-13T12:48:24.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:24.433+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.435798+0000) 2019-10-13T12:48:24.437 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:24.433+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.435977+0000) 2019-10-13T12:48:24.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:24.433+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.437023+0000) 2019-10-13T12:48:24.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:24.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156683 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:24.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:24.441+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.443079+0000) 2019-10-13T12:48:25.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:25.533+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.534609+0000) 2019-10-13T12:48:25.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:25.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.820221+0000) 2019-10-13T12:48:26.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:26.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:26.502584+0000) 2019-10-13T12:48:28.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:28.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:28.767842+0000) 2019-10-13T12:48:28.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:28.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:28.770946+0000) 2019-10-13T12:48:29.447 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:29.441+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.445341+0000) 2019-10-13T12:48:29.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:29.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.446183+0000) 2019-10-13T12:48:29.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:29.445+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.446440+0000) 2019-10-13T12:48:29.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:29.441+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.446624+0000) 2019-10-13T12:48:29.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:29.441+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.446979+0000) 2019-10-13T12:48:29.449 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:29.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.447283+0000) 2019-10-13T12:48:29.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:29.445+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.448513+0000) 2019-10-13T12:48:29.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:29.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:29.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:29.453+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:29.454798+0000) 2019-10-13T12:48:30.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:30.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.002776+0000) 2019-10-13T12:48:31.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:31.333+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:31.336848+0000) 2019-10-13T12:48:32.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:32.257+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:32.259862+0000) 2019-10-13T12:48:33.722 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:48:33.722 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:33.726 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:48:33.726 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:34.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:34.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.456360+0000) 2019-10-13T12:48:34.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:34.453+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.457200+0000) 2019-10-13T12:48:34.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:34.453+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.457522+0000) 2019-10-13T12:48:34.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:34.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.458035+0000) 2019-10-13T12:48:34.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:34.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.458403+0000) 2019-10-13T12:48:34.460 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:34.453+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.458840+0000) 2019-10-13T12:48:34.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:34.457+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.459696+0000) 2019-10-13T12:48:34.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:34.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:34.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:34.461+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.466112+0000) 2019-10-13T12:48:35.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:35.529+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.534809+0000) 2019-10-13T12:48:35.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:35.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.820456+0000) 2019-10-13T12:48:36.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:36.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:36.502825+0000) 2019-10-13T12:48:38.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:38.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:38.768033+0000) 2019-10-13T12:48:38.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:38.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:38.771147+0000) 2019-10-13T12:48:39.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:39.465+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.468451+0000) 2019-10-13T12:48:39.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:39.464+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.468980+0000) 2019-10-13T12:48:39.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:39.465+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.469404+0000) 2019-10-13T12:48:39.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:39.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.469599+0000) 2019-10-13T12:48:39.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:39.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.470023+0000) 2019-10-13T12:48:39.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:39.465+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.470332+0000) 2019-10-13T12:48:39.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:39.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.470704+0000) 2019-10-13T12:48:39.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:39.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:39.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:39.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:39.478283+0000) 2019-10-13T12:48:40.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:40.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.002930+0000) 2019-10-13T12:48:41.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:41.340+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:41.343242+0000) 2019-10-13T12:48:42.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:42.257+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:42.260087+0000) 2019-10-13T12:48:44.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:44.477+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.479423+0000) 2019-10-13T12:48:44.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:44.477+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.481028+0000) 2019-10-13T12:48:44.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:44.477+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.481384+0000) 2019-10-13T12:48:44.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:44.476+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.481963+0000) 2019-10-13T12:48:44.483 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:44.477+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.482299+0000) 2019-10-13T12:48:44.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:44.477+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.482843+0000) 2019-10-13T12:48:44.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:44.481+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.483556+0000) 2019-10-13T12:48:44.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:44.481+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:44.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:44.488+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.491025+0000) 2019-10-13T12:48:45.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:45.533+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.535053+0000) 2019-10-13T12:48:45.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:45.817+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.820653+0000) 2019-10-13T12:48:46.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:46.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:46.503016+0000) 2019-10-13T12:48:48.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:48.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:48.768269+0000) 2019-10-13T12:48:48.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:48.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:48.771377+0000) 2019-10-13T12:48:49.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:49.484+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.490615+0000) 2019-10-13T12:48:49.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:49.484+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.491118+0000) 2019-10-13T12:48:49.493 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:49.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.492002+0000) 2019-10-13T12:48:49.494 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:49.488+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.492364+0000) 2019-10-13T12:48:49.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:49.488+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.493230+0000) 2019-10-13T12:48:49.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:49.488+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.493525+0000) 2019-10-13T12:48:49.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:49.488+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.493739+0000) 2019-10-13T12:48:49.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:49.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:49.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:49.496+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:49.500290+0000) 2019-10-13T12:48:50.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:50.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:50.003068+0000) 2019-10-13T12:48:51.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:51.348+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.349651+0000) 2019-10-13T12:48:52.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:52.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:52.260313+0000) 2019-10-13T12:48:54.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:54.496+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.503002+0000) 2019-10-13T12:48:54.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:54.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.504084+0000) 2019-10-13T12:48:54.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:54.500+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.504512+0000) 2019-10-13T12:48:54.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:54.500+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.504749+0000) 2019-10-13T12:48:54.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:54.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.505710+0000) 2019-10-13T12:48:54.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:54.500+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.505967+0000) 2019-10-13T12:48:54.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:54.500+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.505419+0000) 2019-10-13T12:48:54.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:54.504+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.509778+0000) 2019-10-13T12:48:54.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:54.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:55.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:55.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:55.535253+0000) 2019-10-13T12:48:55.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:55.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:55.820857+0000) 2019-10-13T12:48:56.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:56.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.503209+0000) 2019-10-13T12:48:58.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:58.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:58.768464+0000) 2019-10-13T12:48:58.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:58.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:58.771544+0000) 2019-10-13T12:48:59.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:48:59.508+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.513330+0000) 2019-10-13T12:48:59.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:48:59.508+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.513774+0000) 2019-10-13T12:48:59.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:48:59.512+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.514350+0000) 2019-10-13T12:48:59.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:48:59.512+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.515428+0000) 2019-10-13T12:48:59.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:48:59.512+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.516014+0000) 2019-10-13T12:48:59.518 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:48:59.512+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.516048+0000) 2019-10-13T12:48:59.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:48:59.512+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.516792+0000) 2019-10-13T12:48:59.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:48:59.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:48:59.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:48:59.520+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:59.522699+0000) 2019-10-13T12:49:00.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:00.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:00.003235+0000) 2019-10-13T12:49:01.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:01.352+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.355801+0000) 2019-10-13T12:49:02.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:02.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:02.260507+0000) 2019-10-13T12:49:03.816 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:49:03.816 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:03.820 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:49:03.820 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:04.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:04.520+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.523665+0000) 2019-10-13T12:49:04.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:04.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.524084+0000) 2019-10-13T12:49:04.526 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:04.520+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.524566+0000) 2019-10-13T12:49:04.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:04.520+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.525151+0000) 2019-10-13T12:49:04.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:04.520+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.525783+0000) 2019-10-13T12:49:04.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:04.520+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.526012+0000) 2019-10-13T12:49:04.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:04.520+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.526993+0000) 2019-10-13T12:49:04.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:04.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:04.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:04.528+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.533227+0000) 2019-10-13T12:49:05.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:05.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:05.535491+0000) 2019-10-13T12:49:05.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:05.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:05.821070+0000) 2019-10-13T12:49:06.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:06.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.503417+0000) 2019-10-13T12:49:08.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:08.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:08.768703+0000) 2019-10-13T12:49:08.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:08.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:08.771729+0000) 2019-10-13T12:49:09.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:09.528+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.534137+0000) 2019-10-13T12:49:09.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:09.528+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.534560+0000) 2019-10-13T12:49:09.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:09.532+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.536004+0000) 2019-10-13T12:49:09.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:09.532+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.536219+0000) 2019-10-13T12:49:09.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:09.532+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.536766+0000) 2019-10-13T12:49:09.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:09.532+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.537587+0000) 2019-10-13T12:49:09.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:09.532+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.537251+0000) 2019-10-13T12:49:09.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:09.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 156992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:09.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:09.540+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:09.544134+0000) 2019-10-13T12:49:10.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:10.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:10.003372+0000) 2019-10-13T12:49:11.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:11.360+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.362266+0000) 2019-10-13T12:49:12.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:12.256+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:12.260720+0000) 2019-10-13T12:49:14.548 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:14.540+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.546219+0000) 2019-10-13T12:49:14.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:14.540+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.547000+0000) 2019-10-13T12:49:14.549 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:14.544+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.547488+0000) 2019-10-13T12:49:14.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:14.544+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.548223+0000) 2019-10-13T12:49:14.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:14.544+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.548453+0000) 2019-10-13T12:49:14.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:14.544+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.548742+0000) 2019-10-13T12:49:14.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:14.544+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.549124+0000) 2019-10-13T12:49:14.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:14.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.553007+0000) 2019-10-13T12:49:14.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:14.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:15.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:15.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:15.535694+0000) 2019-10-13T12:49:15.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:15.816+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:15.821272+0000) 2019-10-13T12:49:16.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:16.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.503640+0000) 2019-10-13T12:49:18.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:18.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:18.768919+0000) 2019-10-13T12:49:18.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:18.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:18.771922+0000) 2019-10-13T12:49:19.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:19.552+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.557285+0000) 2019-10-13T12:49:19.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:19.552+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.558353+0000) 2019-10-13T12:49:19.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:19.552+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.558862+0000) 2019-10-13T12:49:19.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:19.556+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.559404+0000) 2019-10-13T12:49:19.561 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:19.556+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.559835+0000) 2019-10-13T12:49:19.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:19.556+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.560275+0000) 2019-10-13T12:49:19.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:19.556+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.560556+0000) 2019-10-13T12:49:19.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:19.560+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:19.564373+0000) 2019-10-13T12:49:19.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:19.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:20.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:20.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:20.003542+0000) 2019-10-13T12:49:21.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:21.368+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.372069+0000) 2019-10-13T12:49:22.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:22.260+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:22.260924+0000) 2019-10-13T12:49:24.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:24.564+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.567278+0000) 2019-10-13T12:49:24.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:24.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.569165+0000) 2019-10-13T12:49:24.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:24.564+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.569493+0000) 2019-10-13T12:49:24.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:24.564+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.569790+0000) 2019-10-13T12:49:24.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:24.564+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.570287+0000) 2019-10-13T12:49:24.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:24.564+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.570702+0000) 2019-10-13T12:49:24.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:24.568+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.571287+0000) 2019-10-13T12:49:24.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:24.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157091 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:24.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:24.572+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.575076+0000) 2019-10-13T12:49:25.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:25.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:25.535871+0000) 2019-10-13T12:49:25.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:25.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:25.821485+0000) 2019-10-13T12:49:26.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:26.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.503842+0000) 2019-10-13T12:49:28.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:28.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:28.769114+0000) 2019-10-13T12:49:28.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:28.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:28.772122+0000) 2019-10-13T12:49:29.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:29.576+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.579839+0000) 2019-10-13T12:49:29.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:29.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.580331+0000) 2019-10-13T12:49:29.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:29.576+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.580907+0000) 2019-10-13T12:49:29.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:29.576+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.581285+0000) 2019-10-13T12:49:29.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:29.580+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.582397+0000) 2019-10-13T12:49:29.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:29.576+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.582841+0000) 2019-10-13T12:49:29.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:29.580+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.583422+0000) 2019-10-13T12:49:29.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:29.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:29.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:29.588+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:29.591882+0000) 2019-10-13T12:49:30.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:30.000+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:30.003714+0000) 2019-10-13T12:49:31.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:31.376+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.378497+0000) 2019-10-13T12:49:32.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:32.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:32.261128+0000) 2019-10-13T12:49:33.935 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:49:33.935 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:33.942 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:49:33.942 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:34.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:34.587+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.591038+0000) 2019-10-13T12:49:34.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:34.587+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.591962+0000) 2019-10-13T12:49:34.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:34.587+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.592709+0000) 2019-10-13T12:49:34.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:34.587+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.593615+0000) 2019-10-13T12:49:34.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:34.587+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.594062+0000) 2019-10-13T12:49:34.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:34.591+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.594463+0000) 2019-10-13T12:49:34.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:34.587+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.594669+0000) 2019-10-13T12:49:34.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:34.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:34.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:34.595+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.600032+0000) 2019-10-13T12:49:35.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:35.535+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:35.536045+0000) 2019-10-13T12:49:35.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:35.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:35.821703+0000) 2019-10-13T12:49:36.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:36.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.504034+0000) 2019-10-13T12:49:38.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:38.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:38.769372+0000) 2019-10-13T12:49:38.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:38.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:38.772319+0000) 2019-10-13T12:49:39.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:39.595+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.602721+0000) 2019-10-13T12:49:39.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:39.595+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.602985+0000) 2019-10-13T12:49:39.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:39.599+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.603473+0000) 2019-10-13T12:49:39.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:39.599+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.603826+0000) 2019-10-13T12:49:39.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:39.599+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.605264+0000) 2019-10-13T12:49:39.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:39.599+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.605786+0000) 2019-10-13T12:49:39.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:39.599+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.605560+0000) 2019-10-13T12:49:39.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:39.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:39.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:39.607+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:39.610505+0000) 2019-10-13T12:49:40.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:39.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:40.003884+0000) 2019-10-13T12:49:41.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:41.379+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.384977+0000) 2019-10-13T12:49:42.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:42.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:42.261326+0000) 2019-10-13T12:49:44.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:44.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.613140+0000) 2019-10-13T12:49:44.615 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:44.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.614006+0000) 2019-10-13T12:49:44.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:44.611+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.614211+0000) 2019-10-13T12:49:44.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:44.607+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.614488+0000) 2019-10-13T12:49:44.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:44.607+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.614846+0000) 2019-10-13T12:49:44.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:44.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.615803+0000) 2019-10-13T12:49:44.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:44.611+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.617307+0000) 2019-10-13T12:49:44.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:44.615+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:44.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:44.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.622789+0000) 2019-10-13T12:49:45.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:45.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:45.536239+0000) 2019-10-13T12:49:45.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:45.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:45.821904+0000) 2019-10-13T12:49:46.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:46.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.504244+0000) 2019-10-13T12:49:48.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:48.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:48.769600+0000) 2019-10-13T12:49:48.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:48.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:48.772552+0000) 2019-10-13T12:49:49.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:49.619+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.625225+0000) 2019-10-13T12:49:49.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:49.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.625904+0000) 2019-10-13T12:49:49.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:49.623+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.626789+0000) 2019-10-13T12:49:49.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:49.619+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.627145+0000) 2019-10-13T12:49:49.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:49.623+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.627643+0000) 2019-10-13T12:49:49.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:49.623+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.628490+0000) 2019-10-13T12:49:49.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:49.623+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.629022+0000) 2019-10-13T12:49:49.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:49.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:49.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:49.627+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:49.633694+0000) 2019-10-13T12:49:50.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:49.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:50.004018+0000) 2019-10-13T12:49:51.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:51.387+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.391696+0000) 2019-10-13T12:49:52.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:52.255+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:52.261560+0000) 2019-10-13T12:49:54.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:54.631+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.635202+0000) 2019-10-13T12:49:54.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:54.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.636927+0000) 2019-10-13T12:49:54.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:54.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.637667+0000) 2019-10-13T12:49:54.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:54.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.638087+0000) 2019-10-13T12:49:54.640 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:54.635+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.638474+0000) 2019-10-13T12:49:54.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:54.631+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.639150+0000) 2019-10-13T12:49:54.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:54.635+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.639477+0000) 2019-10-13T12:49:54.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:54.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:49:54.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:54.639+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.645374+0000) 2019-10-13T12:49:55.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:55.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:55.536412+0000) 2019-10-13T12:49:55.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:55.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:55.822110+0000) 2019-10-13T12:49:56.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:56.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.504468+0000) 2019-10-13T12:49:58.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:58.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:58.769801+0000) 2019-10-13T12:49:58.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:58.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:58.772743+0000) 2019-10-13T12:49:59.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:49:59.643+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.647578+0000) 2019-10-13T12:49:59.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:49:59.643+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.648165+0000) 2019-10-13T12:49:59.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:59.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.648941+0000) 2019-10-13T12:49:59.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:49:59.643+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.648519+0000) 2019-10-13T12:49:59.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:49:59.643+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.649672+0000) 2019-10-13T12:49:59.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:49:59.643+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.650456+0000) 2019-10-13T12:49:59.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:49:59.643+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.650796+0000) 2019-10-13T12:49:59.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:49:59.651+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:59.655726+0000) 2019-10-13T12:49:59.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:49:59.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157330 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:00.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:49:59.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:00.004213+0000) 2019-10-13T12:50:01.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:01.395+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.398543+0000) 2019-10-13T12:50:02.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:02.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:02.261775+0000) 2019-10-13T12:50:04.023 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:50:04.024 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:04.027 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:50:04.027 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:04.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:04.655+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.659066+0000) 2019-10-13T12:50:04.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:04.659+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.659861+0000) 2019-10-13T12:50:04.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:04.659+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.660112+0000) 2019-10-13T12:50:04.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:04.655+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.660581+0000) 2019-10-13T12:50:04.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:04.659+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.660727+0000) 2019-10-13T12:50:04.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:04.659+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.662015+0000) 2019-10-13T12:50:04.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:04.659+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.662543+0000) 2019-10-13T12:50:04.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:04.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:04.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:04.663+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.668029+0000) 2019-10-13T12:50:05.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:05.535+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:05.536585+0000) 2019-10-13T12:50:05.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:05.819+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:05.822341+0000) 2019-10-13T12:50:06.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:06.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.504660+0000) 2019-10-13T12:50:08.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:08.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:08.770049+0000) 2019-10-13T12:50:08.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:08.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:08.772944+0000) 2019-10-13T12:50:09.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:09.667+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.670305+0000) 2019-10-13T12:50:09.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:09.667+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.670814+0000) 2019-10-13T12:50:09.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:09.671+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.671597+0000) 2019-10-13T12:50:09.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:09.671+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.671333+0000) 2019-10-13T12:50:09.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:09.671+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.672059+0000) 2019-10-13T12:50:09.675 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:09.671+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.673334+0000) 2019-10-13T12:50:09.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:09.671+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.673347+0000) 2019-10-13T12:50:09.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:09.675+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157399 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:09.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:09.675+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:09.679359+0000) 2019-10-13T12:50:10.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:09.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:10.004408+0000) 2019-10-13T12:50:11.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:11.407+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.410010+0000) 2019-10-13T12:50:12.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:12.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:12.261971+0000) 2019-10-13T12:50:14.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:14.678+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.681783+0000) 2019-10-13T12:50:14.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:14.678+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.682219+0000) 2019-10-13T12:50:14.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:14.678+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.682506+0000) 2019-10-13T12:50:14.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:14.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.684464+0000) 2019-10-13T12:50:14.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:14.682+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.684627+0000) 2019-10-13T12:50:14.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:14.682+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.686191+0000) 2019-10-13T12:50:14.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:14.682+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.686586+0000) 2019-10-13T12:50:14.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:14.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157432 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:14.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:14.687+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.690664+0000) 2019-10-13T12:50:15.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:15.534+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:15.536723+0000) 2019-10-13T12:50:15.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:15.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:15.822543+0000) 2019-10-13T12:50:16.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:16.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.504853+0000) 2019-10-13T12:50:18.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:18.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:18.770244+0000) 2019-10-13T12:50:18.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:18.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:18.773133+0000) 2019-10-13T12:50:19.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:19.690+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.693318+0000) 2019-10-13T12:50:19.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:19.690+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.693851+0000) 2019-10-13T12:50:19.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:19.690+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.694403+0000) 2019-10-13T12:50:19.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:19.691+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.695245+0000) 2019-10-13T12:50:19.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:19.694+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.695925+0000) 2019-10-13T12:50:19.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:19.694+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.696821+0000) 2019-10-13T12:50:19.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:19.694+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.697289+0000) 2019-10-13T12:50:19.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:19.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:19.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:19.703+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:19.706320+0000) 2019-10-13T12:50:20.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:19.999+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:20.004610+0000) 2019-10-13T12:50:21.419 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:21.411+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.417216+0000) 2019-10-13T12:50:22.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:22.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:22.262172+0000) 2019-10-13T12:50:24.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:24.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.705841+0000) 2019-10-13T12:50:24.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:24.702+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.707260+0000) 2019-10-13T12:50:24.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:24.706+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.707494+0000) 2019-10-13T12:50:24.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:24.706+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.708163+0000) 2019-10-13T12:50:24.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:24.703+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.708551+0000) 2019-10-13T12:50:24.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:24.706+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.709221+0000) 2019-10-13T12:50:24.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:24.706+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.709695+0000) 2019-10-13T12:50:24.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:24.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:24.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:24.711+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.715821+0000) 2019-10-13T12:50:25.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:25.534+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:25.536940+0000) 2019-10-13T12:50:25.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:25.818+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:25.822733+0000) 2019-10-13T12:50:26.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:26.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.505048+0000) 2019-10-13T12:50:28.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:28.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:28.770478+0000) 2019-10-13T12:50:28.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:28.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:28.773333+0000) 2019-10-13T12:50:29.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:29.714+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.717292+0000) 2019-10-13T12:50:29.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:29.710+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.717779+0000) 2019-10-13T12:50:29.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:29.714+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.718116+0000) 2019-10-13T12:50:29.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:29.714+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.718336+0000) 2019-10-13T12:50:29.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:29.718+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.719591+0000) 2019-10-13T12:50:29.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:29.718+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.720069+0000) 2019-10-13T12:50:29.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:29.718+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.720429+0000) 2019-10-13T12:50:29.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:29.722+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:29.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:29.718+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:29.726014+0000) 2019-10-13T12:50:30.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:29.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:30.004816+0000) 2019-10-13T12:50:31.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:31.418+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.423212+0000) 2019-10-13T12:50:32.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:32.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:32.262398+0000) 2019-10-13T12:50:34.150 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:50:34.150 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:34.154 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:50:34.154 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:34.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:34.726+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.728768+0000) 2019-10-13T12:50:34.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:34.726+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.729325+0000) 2019-10-13T12:50:34.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:34.726+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.730118+0000) 2019-10-13T12:50:34.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:34.726+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.730576+0000) 2019-10-13T12:50:34.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:34.726+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.730940+0000) 2019-10-13T12:50:34.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:34.730+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.731600+0000) 2019-10-13T12:50:34.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:34.730+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.731939+0000) 2019-10-13T12:50:34.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:34.730+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.737162+0000) 2019-10-13T12:50:34.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:34.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:35.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:35.534+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:35.537119+0000) 2019-10-13T12:50:35.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:35.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:35.822938+0000) 2019-10-13T12:50:36.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:36.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.505199+0000) 2019-10-13T12:50:38.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:38.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:38.770703+0000) 2019-10-13T12:50:38.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:38.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:38.773554+0000) 2019-10-13T12:50:39.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:39.738+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.740505+0000) 2019-10-13T12:50:39.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:39.734+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.741731+0000) 2019-10-13T12:50:39.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:39.738+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.741552+0000) 2019-10-13T12:50:39.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:39.738+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.742167+0000) 2019-10-13T12:50:39.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:39.738+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.742627+0000) 2019-10-13T12:50:39.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:39.738+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.742968+0000) 2019-10-13T12:50:39.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:39.742+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.743593+0000) 2019-10-13T12:50:39.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:39.742+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:39.748108+0000) 2019-10-13T12:50:39.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:39.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:40.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:39.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:40.005006+0000) 2019-10-13T12:50:41.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:41.422+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.429701+0000) 2019-10-13T12:50:42.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:42.262+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:42.262625+0000) 2019-10-13T12:50:44.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:44.750+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.751731+0000) 2019-10-13T12:50:44.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:44.750+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.751393+0000) 2019-10-13T12:50:44.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:44.750+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.753144+0000) 2019-10-13T12:50:44.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:44.750+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.753425+0000) 2019-10-13T12:50:44.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:44.750+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.754201+0000) 2019-10-13T12:50:44.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:44.750+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.754525+0000) 2019-10-13T12:50:44.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:44.750+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.754934+0000) 2019-10-13T12:50:44.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:44.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:44.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:44.758+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.763497+0000) 2019-10-13T12:50:45.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:45.534+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:45.537317+0000) 2019-10-13T12:50:45.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:45.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:45.823148+0000) 2019-10-13T12:50:46.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:46.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.505410+0000) 2019-10-13T12:50:48.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:48.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:48.770928+0000) 2019-10-13T12:50:48.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:48.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:48.773750+0000) 2019-10-13T12:50:49.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:49.758+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.762247+0000) 2019-10-13T12:50:49.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:49.758+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.762647+0000) 2019-10-13T12:50:49.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:49.758+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.763153+0000) 2019-10-13T12:50:49.765 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:49.762+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.763998+0000) 2019-10-13T12:50:49.766 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:49.758+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.764757+0000) 2019-10-13T12:50:49.767 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:49.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.765536+0000) 2019-10-13T12:50:49.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:49.762+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.765677+0000) 2019-10-13T12:50:49.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:49.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:49.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:49.766+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:49.771447+0000) 2019-10-13T12:50:50.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:49.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:50.005176+0000) 2019-10-13T12:50:51.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:51.430+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.436101+0000) 2019-10-13T12:50:52.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:52.258+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:52.262838+0000) 2019-10-13T12:50:54.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:54.770+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.774231+0000) 2019-10-13T12:50:54.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:54.770+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.774922+0000) 2019-10-13T12:50:54.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:54.774+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.775400+0000) 2019-10-13T12:50:54.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:54.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.775743+0000) 2019-10-13T12:50:54.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:54.774+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.776408+0000) 2019-10-13T12:50:54.778 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:54.770+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.777218+0000) 2019-10-13T12:50:54.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:54.774+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.777579+0000) 2019-10-13T12:50:54.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:54.778+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:54.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:54.778+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.782703+0000) 2019-10-13T12:50:55.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:55.534+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:55.537506+0000) 2019-10-13T12:50:55.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:55.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:55.823363+0000) 2019-10-13T12:50:56.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:56.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.505608+0000) 2019-10-13T12:50:58.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:58.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:58.771125+0000) 2019-10-13T12:50:58.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:58.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:58.773941+0000) 2019-10-13T12:50:59.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:50:59.781+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.784323+0000) 2019-10-13T12:50:59.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:50:59.781+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.786234+0000) 2019-10-13T12:50:59.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:50:59.781+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.786351+0000) 2019-10-13T12:50:59.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:50:59.781+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.787174+0000) 2019-10-13T12:50:59.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:59.782+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.787849+0000) 2019-10-13T12:50:59.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:50:59.785+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.788000+0000) 2019-10-13T12:50:59.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:50:59.785+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.788242+0000) 2019-10-13T12:50:59.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:50:59.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:50:59.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:50:59.786+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:59.792692+0000) 2019-10-13T12:51:00.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:50:59.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:00.005370+0000) 2019-10-13T12:51:01.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:01.442+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.445268+0000) 2019-10-13T12:51:02.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:02.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:02.263049+0000) 2019-10-13T12:51:04.276 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:51:04.277 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:04.280 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:51:04.281 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:04.798 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:04.793+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.796248+0000) 2019-10-13T12:51:04.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:04.793+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.796573+0000) 2019-10-13T12:51:04.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:04.793+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.797000+0000) 2019-10-13T12:51:04.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:04.793+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.798194+0000) 2019-10-13T12:51:04.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:04.794+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.798491+0000) 2019-10-13T12:51:04.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:04.793+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.798637+0000) 2019-10-13T12:51:04.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:04.797+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.799624+0000) 2019-10-13T12:51:04.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:04.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157771 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:04.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:04.798+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.804279+0000) 2019-10-13T12:51:05.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:05.533+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:05.537715+0000) 2019-10-13T12:51:05.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:05.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:05.823574+0000) 2019-10-13T12:51:06.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:06.501+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.505806+0000) 2019-10-13T12:51:08.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:08.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:08.771351+0000) 2019-10-13T12:51:08.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:08.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:08.774136+0000) 2019-10-13T12:51:09.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:09.801+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.807256+0000) 2019-10-13T12:51:09.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:09.805+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.808151+0000) 2019-10-13T12:51:09.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:09.805+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.808889+0000) 2019-10-13T12:51:09.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:09.805+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.809360+0000) 2019-10-13T12:51:09.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:09.805+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.809688+0000) 2019-10-13T12:51:09.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:09.806+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.810198+0000) 2019-10-13T12:51:09.812 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:09.805+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.810888+0000) 2019-10-13T12:51:09.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:09.810+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:09.814637+0000) 2019-10-13T12:51:09.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:09.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:10.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:09.998+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:10.005567+0000) 2019-10-13T12:51:11.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:11.450+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.451912+0000) 2019-10-13T12:51:12.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:12.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:12.263278+0000) 2019-10-13T12:51:14.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:14.813+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.818580+0000) 2019-10-13T12:51:14.821 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:14.818+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.819601+0000) 2019-10-13T12:51:14.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:14.817+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.819408+0000) 2019-10-13T12:51:14.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:14.817+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.820693+0000) 2019-10-13T12:51:14.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:14.817+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.821171+0000) 2019-10-13T12:51:14.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:14.817+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.821810+0000) 2019-10-13T12:51:14.823 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:14.817+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.821623+0000) 2019-10-13T12:51:14.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:14.826+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.826827+0000) 2019-10-13T12:51:14.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:14.821+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:15.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:15.533+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:15.537885+0000) 2019-10-13T12:51:15.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:15.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:15.823788+0000) 2019-10-13T12:51:16.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:16.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.506000+0000) 2019-10-13T12:51:18.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:18.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:18.771569+0000) 2019-10-13T12:51:18.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:18.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:18.774331+0000) 2019-10-13T12:51:19.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:19.825+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.830159+0000) 2019-10-13T12:51:19.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:19.830+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.831693+0000) 2019-10-13T12:51:19.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:19.829+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.832217+0000) 2019-10-13T12:51:19.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:19.829+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.832349+0000) 2019-10-13T12:51:19.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:19.829+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.832647+0000) 2019-10-13T12:51:19.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:19.829+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.833114+0000) 2019-10-13T12:51:19.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:19.829+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.833602+0000) 2019-10-13T12:51:19.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:19.833+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:19.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:19.838+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:19.840373+0000) 2019-10-13T12:51:20.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:20.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:20.005782+0000) 2019-10-13T12:51:21.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:21.457+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.458489+0000) 2019-10-13T12:51:22.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:22.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:22.263527+0000) 2019-10-13T12:51:24.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:24.837+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.841526+0000) 2019-10-13T12:51:24.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:24.841+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.842337+0000) 2019-10-13T12:51:24.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:24.837+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.842958+0000) 2019-10-13T12:51:24.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:24.841+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.843610+0000) 2019-10-13T12:51:24.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:24.841+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.844082+0000) 2019-10-13T12:51:24.846 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:24.841+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.844015+0000) 2019-10-13T12:51:24.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:24.841+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.844554+0000) 2019-10-13T12:51:24.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:24.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:24.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:24.849+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.851955+0000) 2019-10-13T12:51:25.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:25.533+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:25.538068+0000) 2019-10-13T12:51:25.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:25.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:25.824032+0000) 2019-10-13T12:51:26.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:26.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.506196+0000) 2019-10-13T12:51:28.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:28.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:28.771835+0000) 2019-10-13T12:51:28.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:28.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:28.774550+0000) 2019-10-13T12:51:29.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:29.849+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.852338+0000) 2019-10-13T12:51:29.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:29.849+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.853579+0000) 2019-10-13T12:51:29.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:29.849+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.853864+0000) 2019-10-13T12:51:29.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:29.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.854145+0000) 2019-10-13T12:51:29.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:29.849+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.854553+0000) 2019-10-13T12:51:29.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:29.853+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.855437+0000) 2019-10-13T12:51:29.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:29.853+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.856008+0000) 2019-10-13T12:51:29.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:29.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:29.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:29.861+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:29.863530+0000) 2019-10-13T12:51:30.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:30.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:30.005962+0000) 2019-10-13T12:51:31.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:31.461+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.464933+0000) 2019-10-13T12:51:32.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:32.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:32.263727+0000) 2019-10-13T12:51:34.389 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:51:34.389 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:34.392 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:51:34.393 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:34.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:34.857+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.862983+0000) 2019-10-13T12:51:34.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:34.861+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.863849+0000) 2019-10-13T12:51:34.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:34.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.864378+0000) 2019-10-13T12:51:34.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:34.861+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.865048+0000) 2019-10-13T12:51:34.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:34.861+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.865674+0000) 2019-10-13T12:51:34.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:34.861+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.865864+0000) 2019-10-13T12:51:34.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:34.861+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.866006+0000) 2019-10-13T12:51:34.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:34.865+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 157972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:34.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:34.869+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.872732+0000) 2019-10-13T12:51:35.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:35.537+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:35.538245+0000) 2019-10-13T12:51:35.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:35.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:35.824235+0000) 2019-10-13T12:51:36.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:36.501+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.506392+0000) 2019-10-13T12:51:38.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:38.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:38.772049+0000) 2019-10-13T12:51:38.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:38.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:38.774768+0000) 2019-10-13T12:51:39.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:39.868+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.874150+0000) 2019-10-13T12:51:39.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:39.868+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.874878+0000) 2019-10-13T12:51:39.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:39.868+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.875184+0000) 2019-10-13T12:51:39.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:39.873+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.876021+0000) 2019-10-13T12:51:39.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:39.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.876629+0000) 2019-10-13T12:51:39.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:39.872+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.877105+0000) 2019-10-13T12:51:39.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:39.872+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.877810+0000) 2019-10-13T12:51:39.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:39.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:39.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:39.881+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:39.884872+0000) 2019-10-13T12:51:40.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:40.001+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:40.006114+0000) 2019-10-13T12:51:41.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:41.469+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.471319+0000) 2019-10-13T12:51:42.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:42.260+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:42.263944+0000) 2019-10-13T12:51:44.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:44.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.886234+0000) 2019-10-13T12:51:44.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:44.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.886596+0000) 2019-10-13T12:51:44.888 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:44.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.886837+0000) 2019-10-13T12:51:44.889 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:44.884+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.887941+0000) 2019-10-13T12:51:44.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:44.884+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.888294+0000) 2019-10-13T12:51:44.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:44.884+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.889518+0000) 2019-10-13T12:51:44.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:44.884+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.890079+0000) 2019-10-13T12:51:44.896 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:44.893+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.895239+0000) 2019-10-13T12:51:44.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:44.892+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:45.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:45.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:45.538426+0000) 2019-10-13T12:51:45.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:45.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:45.824429+0000) 2019-10-13T12:51:46.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:46.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.506583+0000) 2019-10-13T12:51:48.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:48.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:48.772253+0000) 2019-10-13T12:51:48.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:48.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:48.774990+0000) 2019-10-13T12:51:49.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:49.892+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.899043+0000) 2019-10-13T12:51:49.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:49.896+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.899788+0000) 2019-10-13T12:51:49.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:49.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.899868+0000) 2019-10-13T12:51:49.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:49.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.901092+0000) 2019-10-13T12:51:49.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:49.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.901830+0000) 2019-10-13T12:51:49.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:49.896+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.902422+0000) 2019-10-13T12:51:49.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:49.896+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.902763+0000) 2019-10-13T12:51:49.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:49.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:49.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:49.905+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:49.907165+0000) 2019-10-13T12:51:50.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:50.005+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:50.006309+0000) 2019-10-13T12:51:51.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:51.477+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.477806+0000) 2019-10-13T12:51:52.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:52.260+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:52.264183+0000) 2019-10-13T12:51:54.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:54.904+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.910339+0000) 2019-10-13T12:51:54.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:54.904+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.910927+0000) 2019-10-13T12:51:54.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:54.908+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.912481+0000) 2019-10-13T12:51:54.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:54.908+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.912209+0000) 2019-10-13T12:51:54.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:54.908+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.912918+0000) 2019-10-13T12:51:54.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:54.908+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.911940+0000) 2019-10-13T12:51:54.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:54.909+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.913132+0000) 2019-10-13T12:51:54.919 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:54.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:54.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:54.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.919766+0000) 2019-10-13T12:51:55.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:55.532+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:55.538605+0000) 2019-10-13T12:51:55.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:55.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:55.824616+0000) 2019-10-13T12:51:56.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:56.504+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.506779+0000) 2019-10-13T12:51:58.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:58.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:58.772450+0000) 2019-10-13T12:51:58.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:58.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:58.775213+0000) 2019-10-13T12:51:59.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:51:59.916+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.920996+0000) 2019-10-13T12:51:59.923 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:51:59.916+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.921700+0000) 2019-10-13T12:51:59.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:51:59.916+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.922639+0000) 2019-10-13T12:51:59.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:51:59.916+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.923241+0000) 2019-10-13T12:51:59.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:51:59.920+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.923891+0000) 2019-10-13T12:51:59.926 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:51:59.920+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.924225+0000) 2019-10-13T12:51:59.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:51:59.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.924677+0000) 2019-10-13T12:51:59.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:51:59.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:51:59.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:51:59.929+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:59.931181+0000) 2019-10-13T12:52:00.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:00.005+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:00.006492+0000) 2019-10-13T12:52:01.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:01.481+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.484441+0000) 2019-10-13T12:52:02.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:02.260+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:02.264384+0000) 2019-10-13T12:52:04.483 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:52:04.484 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:04.487 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:52:04.487 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:04.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:04.928+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.932613+0000) 2019-10-13T12:52:04.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:04.928+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.933000+0000) 2019-10-13T12:52:04.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:04.929+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.933445+0000) 2019-10-13T12:52:04.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:04.928+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.934057+0000) 2019-10-13T12:52:04.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:04.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.935269+0000) 2019-10-13T12:52:04.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:04.932+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.936165+0000) 2019-10-13T12:52:04.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:04.932+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.936963+0000) 2019-10-13T12:52:04.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:04.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:04.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:04.941+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.942980+0000) 2019-10-13T12:52:05.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:05.536+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:05.538775+0000) 2019-10-13T12:52:05.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:05.820+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:05.824812+0000) 2019-10-13T12:52:06.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:06.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.507005+0000) 2019-10-13T12:52:08.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:08.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:08.772672+0000) 2019-10-13T12:52:08.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:08.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:08.775423+0000) 2019-10-13T12:52:09.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:09.940+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.944312+0000) 2019-10-13T12:52:09.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:09.940+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.945108+0000) 2019-10-13T12:52:09.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:09.940+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.945744+0000) 2019-10-13T12:52:09.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:09.940+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.945984+0000) 2019-10-13T12:52:09.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:09.945+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.947053+0000) 2019-10-13T12:52:09.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:09.944+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.947379+0000) 2019-10-13T12:52:09.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:09.944+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.948230+0000) 2019-10-13T12:52:09.953 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:09.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:09.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:09.949+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:09.952845+0000) 2019-10-13T12:52:10.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:10.005+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:10.006648+0000) 2019-10-13T12:52:11.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:11.489+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.490785+0000) 2019-10-13T12:52:12.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:12.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:12.264614+0000) 2019-10-13T12:52:14.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:14.948+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.955042+0000) 2019-10-13T12:52:14.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:14.952+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.955644+0000) 2019-10-13T12:52:14.958 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:14.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.956202+0000) 2019-10-13T12:52:14.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:14.952+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.956563+0000) 2019-10-13T12:52:14.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:14.952+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.957604+0000) 2019-10-13T12:52:14.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:14.952+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.958163+0000) 2019-10-13T12:52:14.960 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:14.952+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.958545+0000) 2019-10-13T12:52:14.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:14.956+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:14.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:14.961+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.963687+0000) 2019-10-13T12:52:15.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:15.536+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:15.538961+0000) 2019-10-13T12:52:15.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:15.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:15.825025+0000) 2019-10-13T12:52:16.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:16.500+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.507152+0000) 2019-10-13T12:52:18.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:18.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:18.772877+0000) 2019-10-13T12:52:18.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:18.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:18.775617+0000) 2019-10-13T12:52:19.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:19.960+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.965524+0000) 2019-10-13T12:52:19.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:19.964+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.966257+0000) 2019-10-13T12:52:19.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:19.960+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.966570+0000) 2019-10-13T12:52:19.969 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:19.960+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.966945+0000) 2019-10-13T12:52:19.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:19.964+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.967440+0000) 2019-10-13T12:52:19.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:19.964+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.967716+0000) 2019-10-13T12:52:19.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:19.964+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.968081+0000) 2019-10-13T12:52:19.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:19.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:19.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:19.972+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:19.975104+0000) 2019-10-13T12:52:20.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:20.004+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:20.006776+0000) 2019-10-13T12:52:21.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:21.492+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.497230+0000) 2019-10-13T12:52:22.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:22.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:22.264817+0000) 2019-10-13T12:52:24.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:24.971+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.977125+0000) 2019-10-13T12:52:24.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:24.971+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.978403+0000) 2019-10-13T12:52:24.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:24.972+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.977888+0000) 2019-10-13T12:52:24.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:24.971+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.978798+0000) 2019-10-13T12:52:24.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:24.975+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.979354+0000) 2019-10-13T12:52:24.982 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:24.975+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.979759+0000) 2019-10-13T12:52:24.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:24.975+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.979487+0000) 2019-10-13T12:52:24.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:24.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:24.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:24.984+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.986527+0000) 2019-10-13T12:52:25.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:25.531+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:25.539149+0000) 2019-10-13T12:52:25.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:25.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:25.825226+0000) 2019-10-13T12:52:26.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:26.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.507401+0000) 2019-10-13T12:52:28.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:28.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:28.773073+0000) 2019-10-13T12:52:28.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:28.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:28.775810+0000) 2019-10-13T12:52:29.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:29.984+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.989596+0000) 2019-10-13T12:52:29.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:29.983+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.989616+0000) 2019-10-13T12:52:29.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:29.983+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.990335+0000) 2019-10-13T12:52:29.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:29.983+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.990016+0000) 2019-10-13T12:52:29.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:29.983+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.991051+0000) 2019-10-13T12:52:29.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:29.983+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.991012+0000) 2019-10-13T12:52:29.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:29.987+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.991802+0000) 2019-10-13T12:52:29.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:29.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:30.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:29.996+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:29.998855+0000) 2019-10-13T12:52:30.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:30.004+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:30.006874+0000) 2019-10-13T12:52:31.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:31.500+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.503727+0000) 2019-10-13T12:52:32.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:32.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:32.265014+0000) 2019-10-13T12:52:34.552 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:52:34.553 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:34.556 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:52:34.556 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:35.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:34.996+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.000249+0000) 2019-10-13T12:52:35.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:34.995+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.000679+0000) 2019-10-13T12:52:35.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:34.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.000963+0000) 2019-10-13T12:52:35.004 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:34.995+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.002185+0000) 2019-10-13T12:52:35.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:34.995+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.002425+0000) 2019-10-13T12:52:35.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:34.995+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.003029+0000) 2019-10-13T12:52:35.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:34.999+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.003564+0000) 2019-10-13T12:52:35.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:35.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:35.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:35.008+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.010545+0000) 2019-10-13T12:52:35.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:35.535+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.539353+0000) 2019-10-13T12:52:35.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:35.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:35.825403+0000) 2019-10-13T12:52:36.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:36.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.507609+0000) 2019-10-13T12:52:38.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:38.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:38.773277+0000) 2019-10-13T12:52:38.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:38.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:38.776032+0000) 2019-10-13T12:52:40.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:40.004+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.007025+0000) 2019-10-13T12:52:40.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:40.007+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.011413+0000) 2019-10-13T12:52:40.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:40.007+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.011728+0000) 2019-10-13T12:52:40.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:40.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.012242+0000) 2019-10-13T12:52:40.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:40.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.013068+0000) 2019-10-13T12:52:40.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:40.007+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.013242+0000) 2019-10-13T12:52:40.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:40.007+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.014068+0000) 2019-10-13T12:52:40.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:40.007+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.014693+0000) 2019-10-13T12:52:40.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:40.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:40.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:40.016+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:40.020072+0000) 2019-10-13T12:52:41.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:41.508+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.510118+0000) 2019-10-13T12:52:42.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:42.259+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:42.265209+0000) 2019-10-13T12:52:45.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:45.016+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.021826+0000) 2019-10-13T12:52:45.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:45.015+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.022207+0000) 2019-10-13T12:52:45.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:45.015+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.022619+0000) 2019-10-13T12:52:45.024 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:45.019+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.023355+0000) 2019-10-13T12:52:45.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:45.019+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.024292+0000) 2019-10-13T12:52:45.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:45.019+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.024726+0000) 2019-10-13T12:52:45.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:45.019+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.024980+0000) 2019-10-13T12:52:45.030 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:45.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:45.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:45.028+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.030961+0000) 2019-10-13T12:52:45.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:45.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.539533+0000) 2019-10-13T12:52:45.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:45.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:45.825601+0000) 2019-10-13T12:52:46.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:46.507+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.507820+0000) 2019-10-13T12:52:48.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:48.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:48.773492+0000) 2019-10-13T12:52:48.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:48.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:48.776260+0000) 2019-10-13T12:52:50.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:50.004+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.007216+0000) 2019-10-13T12:52:50.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:50.027+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.033093+0000) 2019-10-13T12:52:50.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:50.028+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.033850+0000) 2019-10-13T12:52:50.036 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:50.027+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.034177+0000) 2019-10-13T12:52:50.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:50.027+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.034482+0000) 2019-10-13T12:52:50.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:50.031+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.035601+0000) 2019-10-13T12:52:50.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:50.031+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.035807+0000) 2019-10-13T12:52:50.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:50.031+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.036167+0000) 2019-10-13T12:52:50.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:50.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:50.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:50.040+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:50.042262+0000) 2019-10-13T12:52:51.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:51.516+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.516436+0000) 2019-10-13T12:52:52.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:52.263+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:52.265442+0000) 2019-10-13T12:52:55.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:52:55.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.044082+0000) 2019-10-13T12:52:55.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:55.043+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.044390+0000) 2019-10-13T12:52:55.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:55.043+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.044585+0000) 2019-10-13T12:52:55.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:52:55.043+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.044872+0000) 2019-10-13T12:52:55.047 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:55.043+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.045247+0000) 2019-10-13T12:52:55.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:55.043+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.045586+0000) 2019-10-13T12:52:55.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:55.043+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.046366+0000) 2019-10-13T12:52:55.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:52:55.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.050127+0000) 2019-10-13T12:52:55.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:52:55.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158508 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:52:55.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:52:55.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.539709+0000) 2019-10-13T12:52:55.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:52:55.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:55.825779+0000) 2019-10-13T12:52:56.509 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:52:56.507+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.508020+0000) 2019-10-13T12:52:58.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:52:58.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:58.773726+0000) 2019-10-13T12:52:58.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:52:58.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:58.776456+0000) 2019-10-13T12:53:00.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:00.004+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.007401+0000) 2019-10-13T12:53:00.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:00.048+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.053982+0000) 2019-10-13T12:53:00.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:00.051+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.054597+0000) 2019-10-13T12:53:00.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:00.055+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.056019+0000) 2019-10-13T12:53:00.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:00.055+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.056271+0000) 2019-10-13T12:53:00.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:00.055+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.056905+0000) 2019-10-13T12:53:00.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:00.055+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.056934+0000) 2019-10-13T12:53:00.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:00.055+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.057281+0000) 2019-10-13T12:53:00.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:00.056+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:00.061520+0000) 2019-10-13T12:53:00.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:00.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:01.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:01.520+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.522987+0000) 2019-10-13T12:53:02.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:02.263+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:02.265707+0000) 2019-10-13T12:53:04.655 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:53:04.656 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:04.659 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:53:04.659 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:05.067 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:05.063+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.065506+0000) 2019-10-13T12:53:05.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:05.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.066290+0000) 2019-10-13T12:53:05.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:05.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.066332+0000) 2019-10-13T12:53:05.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:05.063+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.066630+0000) 2019-10-13T12:53:05.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:05.067+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.067785+0000) 2019-10-13T12:53:05.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:05.067+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.068170+0000) 2019-10-13T12:53:05.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:05.067+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.068641+0000) 2019-10-13T12:53:05.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:05.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:05.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:05.072+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.074187+0000) 2019-10-13T12:53:05.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:05.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.539892+0000) 2019-10-13T12:53:05.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:05.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:05.825995+0000) 2019-10-13T12:53:06.510 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:06.506+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.508218+0000) 2019-10-13T12:53:08.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:08.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:08.773948+0000) 2019-10-13T12:53:08.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:08.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:08.776651+0000) 2019-10-13T12:53:10.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:10.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.007552+0000) 2019-10-13T12:53:10.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:10.074+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.076543+0000) 2019-10-13T12:53:10.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:10.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.077164+0000) 2019-10-13T12:53:10.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:10.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.077576+0000) 2019-10-13T12:53:10.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:10.074+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.077927+0000) 2019-10-13T12:53:10.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:10.075+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.078313+0000) 2019-10-13T12:53:10.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:10.074+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.078802+0000) 2019-10-13T12:53:10.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:10.078+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.079551+0000) 2019-10-13T12:53:10.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:10.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:10.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:10.079+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:10.085883+0000) 2019-10-13T12:53:11.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:11.523+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.529540+0000) 2019-10-13T12:53:12.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:12.262+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:12.265909+0000) 2019-10-13T12:53:15.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:15.086+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.088114+0000) 2019-10-13T12:53:15.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:15.083+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.088709+0000) 2019-10-13T12:53:15.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:15.086+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.089650+0000) 2019-10-13T12:53:15.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:15.086+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.089939+0000) 2019-10-13T12:53:15.092 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:15.086+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.090296+0000) 2019-10-13T12:53:15.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:15.086+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.090766+0000) 2019-10-13T12:53:15.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:15.086+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.091089+0000) 2019-10-13T12:53:15.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:15.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:15.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:15.091+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.097036+0000) 2019-10-13T12:53:15.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:15.538+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.540062+0000) 2019-10-13T12:53:15.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:15.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:15.826180+0000) 2019-10-13T12:53:16.510 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:16.506+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.508465+0000) 2019-10-13T12:53:18.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:18.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:18.774173+0000) 2019-10-13T12:53:18.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:18.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:18.776853+0000) 2019-10-13T12:53:20.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:20.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.007747+0000) 2019-10-13T12:53:20.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:20.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.100114+0000) 2019-10-13T12:53:20.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:20.098+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.100759+0000) 2019-10-13T12:53:20.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:20.098+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.101182+0000) 2019-10-13T12:53:20.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:20.098+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.102147+0000) 2019-10-13T12:53:20.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:20.099+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.102499+0000) 2019-10-13T12:53:20.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:20.098+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.102638+0000) 2019-10-13T12:53:20.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:20.102+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.103732+0000) 2019-10-13T12:53:20.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:20.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158676 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:20.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:20.107+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:20.110348+0000) 2019-10-13T12:53:21.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:21.535+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.536009+0000) 2019-10-13T12:53:22.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:22.262+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:22.266083+0000) 2019-10-13T12:53:25.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:25.106+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.110594+0000) 2019-10-13T12:53:25.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:25.110+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.112410+0000) 2019-10-13T12:53:25.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:25.110+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.112722+0000) 2019-10-13T12:53:25.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:25.110+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.113099+0000) 2019-10-13T12:53:25.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:25.110+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.114012+0000) 2019-10-13T12:53:25.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:25.110+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.114826+0000) 2019-10-13T12:53:25.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:25.111+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.115159+0000) 2019-10-13T12:53:25.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:25.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:25.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:25.115+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.119944+0000) 2019-10-13T12:53:25.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:25.538+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.540237+0000) 2019-10-13T12:53:25.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:25.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:25.826359+0000) 2019-10-13T12:53:26.510 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:26.506+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.508679+0000) 2019-10-13T12:53:28.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:28.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:28.774370+0000) 2019-10-13T12:53:28.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:28.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:28.777038+0000) 2019-10-13T12:53:30.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:30.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.007943+0000) 2019-10-13T12:53:30.125 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:30.122+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.123629+0000) 2019-10-13T12:53:30.126 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:30.122+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.124177+0000) 2019-10-13T12:53:30.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:30.122+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.124320+0000) 2019-10-13T12:53:30.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:30.122+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.125552+0000) 2019-10-13T12:53:30.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:30.122+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.125694+0000) 2019-10-13T12:53:30.128 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:30.122+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.126127+0000) 2019-10-13T12:53:30.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:30.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.126996+0000) 2019-10-13T12:53:30.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:30.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:30.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:30.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:30.131711+0000) 2019-10-13T12:53:31.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:31.539+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.542358+0000) 2019-10-13T12:53:32.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:32.262+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:32.266245+0000) 2019-10-13T12:53:34.775 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:53:34.775 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:34.779 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:53:34.779 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:35.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:35.130+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.133649+0000) 2019-10-13T12:53:35.136 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:35.130+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.134032+0000) 2019-10-13T12:53:35.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:35.130+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.134549+0000) 2019-10-13T12:53:35.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:35.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.135495+0000) 2019-10-13T12:53:35.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:35.131+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.136060+0000) 2019-10-13T12:53:35.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:35.134+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.136145+0000) 2019-10-13T12:53:35.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:35.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.136455+0000) 2019-10-13T12:53:35.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:35.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:35.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:35.135+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.141879+0000) 2019-10-13T12:53:35.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:35.538+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.540445+0000) 2019-10-13T12:53:35.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:35.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:35.826540+0000) 2019-10-13T12:53:36.510 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:36.506+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.508898+0000) 2019-10-13T12:53:38.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:38.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:38.774573+0000) 2019-10-13T12:53:38.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:38.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:38.777229+0000) 2019-10-13T12:53:40.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:40.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.008136+0000) 2019-10-13T12:53:40.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:40.142+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.144105+0000) 2019-10-13T12:53:40.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:40.142+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.144928+0000) 2019-10-13T12:53:40.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:40.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.146255+0000) 2019-10-13T12:53:40.148 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:40.142+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.146491+0000) 2019-10-13T12:53:40.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:40.142+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.146831+0000) 2019-10-13T12:53:40.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:40.146+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.147537+0000) 2019-10-13T12:53:40.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:40.146+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.147790+0000) 2019-10-13T12:53:40.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:40.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:40.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:40.151+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:40.155201+0000) 2019-10-13T12:53:41.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:41.543+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.548792+0000) 2019-10-13T12:53:42.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:42.262+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:42.266438+0000) 2019-10-13T12:53:45.159 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:45.154+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.157654+0000) 2019-10-13T12:53:45.160 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:45.154+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.158013+0000) 2019-10-13T12:53:45.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:45.154+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.158433+0000) 2019-10-13T12:53:45.161 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:45.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.159412+0000) 2019-10-13T12:53:45.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:45.154+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.158805+0000) 2019-10-13T12:53:45.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:45.158+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.159569+0000) 2019-10-13T12:53:45.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:45.158+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.159889+0000) 2019-10-13T12:53:45.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:45.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:45.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:45.163+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.166626+0000) 2019-10-13T12:53:45.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:45.538+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.540623+0000) 2019-10-13T12:53:45.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:45.822+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:45.826716+0000) 2019-10-13T12:53:46.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:46.506+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.509110+0000) 2019-10-13T12:53:48.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:48.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:48.774766+0000) 2019-10-13T12:53:48.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:48.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:48.777386+0000) 2019-10-13T12:53:50.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:50.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.008329+0000) 2019-10-13T12:53:50.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:50.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.169175+0000) 2019-10-13T12:53:50.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:50.165+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.169528+0000) 2019-10-13T12:53:50.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:50.165+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.170214+0000) 2019-10-13T12:53:50.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:50.165+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.170259+0000) 2019-10-13T12:53:50.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:50.165+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.170644+0000) 2019-10-13T12:53:50.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:50.165+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.170985+0000) 2019-10-13T12:53:50.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:50.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.171856+0000) 2019-10-13T12:53:50.178 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:50.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:50.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:50.175+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:50.178296+0000) 2019-10-13T12:53:51.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:51.555+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.558849+0000) 2019-10-13T12:53:52.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:52.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:52.266637+0000) 2019-10-13T12:53:55.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:55.177+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.179724+0000) 2019-10-13T12:53:55.182 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:55.177+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.180210+0000) 2019-10-13T12:53:55.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:53:55.177+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.180546+0000) 2019-10-13T12:53:55.183 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:53:55.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.181376+0000) 2019-10-13T12:53:55.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:55.177+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.181315+0000) 2019-10-13T12:53:55.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:55.177+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.182192+0000) 2019-10-13T12:53:55.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:55.181+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.183367+0000) 2019-10-13T12:53:55.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:53:55.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:53:55.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:53:55.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.194854+0000) 2019-10-13T12:53:55.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:53:55.537+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.540833+0000) 2019-10-13T12:53:55.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:53:55.821+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:55.826930+0000) 2019-10-13T12:53:56.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:53:56.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.509325+0000) 2019-10-13T12:53:58.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:53:58.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:58.774995+0000) 2019-10-13T12:53:58.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:53:58.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:58.777597+0000) 2019-10-13T12:54:00.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:00.003+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.008521+0000) 2019-10-13T12:54:00.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:00.185+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.190885+0000) 2019-10-13T12:54:00.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:00.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.192127+0000) 2019-10-13T12:54:00.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:00.189+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.193002+0000) 2019-10-13T12:54:00.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:00.189+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.193651+0000) 2019-10-13T12:54:00.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:00.189+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.193395+0000) 2019-10-13T12:54:00.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:00.189+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.194516+0000) 2019-10-13T12:54:00.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:00.191+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.194960+0000) 2019-10-13T12:54:00.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:00.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:00.202 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:00.195+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:00.200866+0000) 2019-10-13T12:54:01.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:01.559+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.565200+0000) 2019-10-13T12:54:02.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:02.265+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:02.266872+0000) 2019-10-13T12:54:04.887 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:54:04.887 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:04.890 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:54:04.891 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:05.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:05.197+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.202970+0000) 2019-10-13T12:54:05.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:05.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.204155+0000) 2019-10-13T12:54:05.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:05.201+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.204118+0000) 2019-10-13T12:54:05.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:05.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.204590+0000) 2019-10-13T12:54:05.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:05.201+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.205010+0000) 2019-10-13T12:54:05.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:05.201+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.205388+0000) 2019-10-13T12:54:05.207 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:05.198+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.205948+0000) 2019-10-13T12:54:05.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:05.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 158973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:05.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:05.206+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.214085+0000) 2019-10-13T12:54:05.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:05.537+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.541024+0000) 2019-10-13T12:54:05.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:05.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:05.827149+0000) 2019-10-13T12:54:06.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:06.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.509558+0000) 2019-10-13T12:54:08.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:08.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:08.775194+0000) 2019-10-13T12:54:08.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:08.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:08.777823+0000) 2019-10-13T12:54:10.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:10.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.008756+0000) 2019-10-13T12:54:10.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:10.213+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.215542+0000) 2019-10-13T12:54:10.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:10.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.216247+0000) 2019-10-13T12:54:10.219 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:10.213+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.217680+0000) 2019-10-13T12:54:10.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:10.213+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.218091+0000) 2019-10-13T12:54:10.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:10.213+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.218375+0000) 2019-10-13T12:54:10.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:10.213+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.219072+0000) 2019-10-13T12:54:10.221 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:10.214+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.219869+0000) 2019-10-13T12:54:10.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:10.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:10.222793+0000) 2019-10-13T12:54:10.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:10.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:11.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:11.570+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.570986+0000) 2019-10-13T12:54:12.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:12.261+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:12.267077+0000) 2019-10-13T12:54:15.228 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:15.221+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.226930+0000) 2019-10-13T12:54:15.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:15.222+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.227439+0000) 2019-10-13T12:54:15.229 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:15.225+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.227865+0000) 2019-10-13T12:54:15.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:15.225+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.228243+0000) 2019-10-13T12:54:15.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:15.225+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.228631+0000) 2019-10-13T12:54:15.230 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:15.225+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.229146+0000) 2019-10-13T12:54:15.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:15.225+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.229847+0000) 2019-10-13T12:54:15.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:15.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:15.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:15.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.238173+0000) 2019-10-13T12:54:15.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:15.537+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.541201+0000) 2019-10-13T12:54:15.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:15.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:15.827366+0000) 2019-10-13T12:54:16.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:16.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.509746+0000) 2019-10-13T12:54:18.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:18.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:18.775422+0000) 2019-10-13T12:54:18.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:18.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:18.778020+0000) 2019-10-13T12:54:20.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:20.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.008966+0000) 2019-10-13T12:54:20.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:20.233+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.237453+0000) 2019-10-13T12:54:20.240 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:20.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.238294+0000) 2019-10-13T12:54:20.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:20.233+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.239090+0000) 2019-10-13T12:54:20.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:20.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.239339+0000) 2019-10-13T12:54:20.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:20.237+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.239747+0000) 2019-10-13T12:54:20.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:20.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.240152+0000) 2019-10-13T12:54:20.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:20.237+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.240330+0000) 2019-10-13T12:54:20.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:20.238+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:20.244815+0000) 2019-10-13T12:54:20.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:20.241+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:21.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:21.570+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.577587+0000) 2019-10-13T12:54:22.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:22.265+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:22.267341+0000) 2019-10-13T12:54:25.251 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:25.245+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.249297+0000) 2019-10-13T12:54:25.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:25.245+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.250092+0000) 2019-10-13T12:54:25.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:25.245+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.250515+0000) 2019-10-13T12:54:25.252 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:25.246+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.251232+0000) 2019-10-13T12:54:25.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:25.249+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.251564+0000) 2019-10-13T12:54:25.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:25.249+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.252444+0000) 2019-10-13T12:54:25.255 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:25.249+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.253592+0000) 2019-10-13T12:54:25.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:25.250+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.255443+0000) 2019-10-13T12:54:25.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:25.253+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:25.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:25.541+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.541377+0000) 2019-10-13T12:54:25.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:25.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:25.827579+0000) 2019-10-13T12:54:26.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:26.505+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.509940+0000) 2019-10-13T12:54:28.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:28.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:28.775685+0000) 2019-10-13T12:54:28.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:28.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:28.778211+0000) 2019-10-13T12:54:30.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:30.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.009155+0000) 2019-10-13T12:54:30.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:30.257+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.262856+0000) 2019-10-13T12:54:30.265 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:30.261+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.263487+0000) 2019-10-13T12:54:30.265 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:30.261+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.264008+0000) 2019-10-13T12:54:30.266 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:30.258+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.264975+0000) 2019-10-13T12:54:30.267 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:30.261+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.264964+0000) 2019-10-13T12:54:30.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:30.261+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.265337+0000) 2019-10-13T12:54:30.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:30.261+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.265760+0000) 2019-10-13T12:54:30.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:30.265+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159138 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:30.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:30.266+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:30.270644+0000) 2019-10-13T12:54:31.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:31.578+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.584177+0000) 2019-10-13T12:54:32.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:32.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:32.267514+0000) 2019-10-13T12:54:35.017 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:54:35.018 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:35.021 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:54:35.022 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:35.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:35.268+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.272145+0000) 2019-10-13T12:54:35.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:35.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.273349+0000) 2019-10-13T12:54:35.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:35.268+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.274086+0000) 2019-10-13T12:54:35.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:35.268+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.273872+0000) 2019-10-13T12:54:35.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:35.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.274761+0000) 2019-10-13T12:54:35.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:35.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.275350+0000) 2019-10-13T12:54:35.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:35.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.275519+0000) 2019-10-13T12:54:35.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:35.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:35.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:35.274+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.280298+0000) 2019-10-13T12:54:35.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:35.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.541585+0000) 2019-10-13T12:54:35.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:35.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:35.827766+0000) 2019-10-13T12:54:36.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:36.508+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.510139+0000) 2019-10-13T12:54:38.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:38.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:38.775891+0000) 2019-10-13T12:54:38.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:38.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:38.778404+0000) 2019-10-13T12:54:40.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:40.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.009355+0000) 2019-10-13T12:54:40.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:40.276+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.283061+0000) 2019-10-13T12:54:40.285 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:40.280+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.284205+0000) 2019-10-13T12:54:40.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:40.280+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.284551+0000) 2019-10-13T12:54:40.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:40.280+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.285260+0000) 2019-10-13T12:54:40.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:40.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.285866+0000) 2019-10-13T12:54:40.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:40.280+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.286658+0000) 2019-10-13T12:54:40.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:40.280+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.287143+0000) 2019-10-13T12:54:40.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:40.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:40.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:40.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:40.292598+0000) 2019-10-13T12:54:41.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:41.590+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.590564+0000) 2019-10-13T12:54:42.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:42.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:42.267730+0000) 2019-10-13T12:54:45.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:45.292+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.295481+0000) 2019-10-13T12:54:45.298 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:45.292+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.296413+0000) 2019-10-13T12:54:45.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:45.292+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.297130+0000) 2019-10-13T12:54:45.299 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:45.292+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.297848+0000) 2019-10-13T12:54:45.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:45.292+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.298263+0000) 2019-10-13T12:54:45.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:45.294+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.298887+0000) 2019-10-13T12:54:45.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:45.296+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.299791+0000) 2019-10-13T12:54:45.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:45.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.302507+0000) 2019-10-13T12:54:45.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:45.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:45.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:45.536+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.541764+0000) 2019-10-13T12:54:45.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:45.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:45.827947+0000) 2019-10-13T12:54:46.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:46.504+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.510370+0000) 2019-10-13T12:54:48.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:48.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:48.776088+0000) 2019-10-13T12:54:48.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:48.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:48.778609+0000) 2019-10-13T12:54:50.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:50.002+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.009544+0000) 2019-10-13T12:54:50.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:50.302+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.306296+0000) 2019-10-13T12:54:50.308 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:50.300+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.306601+0000) 2019-10-13T12:54:50.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:50.304+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.307654+0000) 2019-10-13T12:54:50.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:50.304+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.307964+0000) 2019-10-13T12:54:50.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:50.304+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.309593+0000) 2019-10-13T12:54:50.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:50.304+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.308246+0000) 2019-10-13T12:54:50.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:50.304+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.310237+0000) 2019-10-13T12:54:50.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:50.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:50.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:50.310+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:50.315753+0000) 2019-10-13T12:54:51.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:51.594+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.596893+0000) 2019-10-13T12:54:52.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:52.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:52.267947+0000) 2019-10-13T12:54:55.319 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:55.312+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.317752+0000) 2019-10-13T12:54:55.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:55.312+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.318476+0000) 2019-10-13T12:54:55.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:55.312+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.319252+0000) 2019-10-13T12:54:55.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:55.316+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.320060+0000) 2019-10-13T12:54:55.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:54:55.318+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.320752+0000) 2019-10-13T12:54:55.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:54:55.316+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.320864+0000) 2019-10-13T12:54:55.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:55.316+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.321252+0000) 2019-10-13T12:54:55.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:54:55.322+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.325757+0000) 2019-10-13T12:54:55.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:54:55.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:54:55.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:54:55.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.541942+0000) 2019-10-13T12:54:55.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:54:55.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:55.828162+0000) 2019-10-13T12:54:56.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:54:56.504+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.510581+0000) 2019-10-13T12:54:58.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:54:58.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:58.776283+0000) 2019-10-13T12:54:58.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:54:58.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:58.778796+0000) 2019-10-13T12:55:00.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:00.005+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.009745+0000) 2019-10-13T12:55:00.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:00.324+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.329310+0000) 2019-10-13T12:55:00.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:00.325+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.329913+0000) 2019-10-13T12:55:00.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:00.324+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.330055+0000) 2019-10-13T12:55:00.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:00.324+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.330501+0000) 2019-10-13T12:55:00.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:00.324+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.331284+0000) 2019-10-13T12:55:00.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:00.328+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.332045+0000) 2019-10-13T12:55:00.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:00.328+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.332397+0000) 2019-10-13T12:55:00.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:00.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:00.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:00.337+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:00.338260+0000) 2019-10-13T12:55:01.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:01.601+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.603427+0000) 2019-10-13T12:55:02.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:02.264+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:02.268155+0000) 2019-10-13T12:55:05.117 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:55:05.117 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:05.121 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:55:05.121 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:05.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:05.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.341050+0000) 2019-10-13T12:55:05.343 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:05.336+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.341250+0000) 2019-10-13T12:55:05.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:05.336+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.341662+0000) 2019-10-13T12:55:05.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:05.336+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.342369+0000) 2019-10-13T12:55:05.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:05.336+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.342743+0000) 2019-10-13T12:55:05.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:05.336+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.342956+0000) 2019-10-13T12:55:05.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:05.340+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.343517+0000) 2019-10-13T12:55:05.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:05.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:05.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:05.349+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.350476+0000) 2019-10-13T12:55:05.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:05.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.542136+0000) 2019-10-13T12:55:05.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:05.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:05.828378+0000) 2019-10-13T12:55:06.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:06.508+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.510784+0000) 2019-10-13T12:55:08.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:08.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:08.776486+0000) 2019-10-13T12:55:08.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:08.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:08.779000+0000) 2019-10-13T12:55:10.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:10.005+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.009941+0000) 2019-10-13T12:55:10.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:10.344+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.350492+0000) 2019-10-13T12:55:10.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:10.344+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.350765+0000) 2019-10-13T12:55:10.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:10.344+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.351264+0000) 2019-10-13T12:55:10.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:10.348+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.351905+0000) 2019-10-13T12:55:10.354 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:10.348+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.352270+0000) 2019-10-13T12:55:10.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:10.348+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.353057+0000) 2019-10-13T12:55:10.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:10.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.354253+0000) 2019-10-13T12:55:10.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:10.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:10.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:10.357+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:10.360579+0000) 2019-10-13T12:55:11.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:11.605+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.610097+0000) 2019-10-13T12:55:12.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:12.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:12.268373+0000) 2019-10-13T12:55:15.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:15.355+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.361566+0000) 2019-10-13T12:55:15.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:15.359+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.363531+0000) 2019-10-13T12:55:15.365 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:15.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.363973+0000) 2019-10-13T12:55:15.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:15.359+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.364219+0000) 2019-10-13T12:55:15.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:15.359+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.364844+0000) 2019-10-13T12:55:15.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:15.359+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.365645+0000) 2019-10-13T12:55:15.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:15.359+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.365254+0000) 2019-10-13T12:55:15.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:15.369+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.370372+0000) 2019-10-13T12:55:15.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:15.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:15.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:15.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.542327+0000) 2019-10-13T12:55:15.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:15.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:15.828567+0000) 2019-10-13T12:55:16.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:16.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.511012+0000) 2019-10-13T12:55:18.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:18.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:18.776689+0000) 2019-10-13T12:55:18.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:18.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:18.779240+0000) 2019-10-13T12:55:20.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:20.009+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.010188+0000) 2019-10-13T12:55:20.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:20.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.373797+0000) 2019-10-13T12:55:20.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:20.367+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.374444+0000) 2019-10-13T12:55:20.376 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:20.367+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.374687+0000) 2019-10-13T12:55:20.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:20.367+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.375239+0000) 2019-10-13T12:55:20.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:20.371+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.375747+0000) 2019-10-13T12:55:20.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:20.371+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.376640+0000) 2019-10-13T12:55:20.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:20.371+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.375966+0000) 2019-10-13T12:55:20.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:20.377+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:20.380839+0000) 2019-10-13T12:55:20.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:20.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159472 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:21.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:21.613+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.616905+0000) 2019-10-13T12:55:22.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:22.267+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:22.268605+0000) 2019-10-13T12:55:25.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:25.379+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.383996+0000) 2019-10-13T12:55:25.386 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:25.381+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.384673+0000) 2019-10-13T12:55:25.387 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:25.379+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.386131+0000) 2019-10-13T12:55:25.388 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:25.379+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.386569+0000) 2019-10-13T12:55:25.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:25.379+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.386964+0000) 2019-10-13T12:55:25.389 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:25.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.387586+0000) 2019-10-13T12:55:25.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:25.383+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.387953+0000) 2019-10-13T12:55:25.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:25.389+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.391667+0000) 2019-10-13T12:55:25.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:25.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:25.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:25.535+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.542501+0000) 2019-10-13T12:55:25.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:25.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:25.828769+0000) 2019-10-13T12:55:26.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:26.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.511270+0000) 2019-10-13T12:55:28.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:28.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:28.776886+0000) 2019-10-13T12:55:28.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:28.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:28.779503+0000) 2019-10-13T12:55:30.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:30.009+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.010379+0000) 2019-10-13T12:55:30.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:30.391+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.394889+0000) 2019-10-13T12:55:30.398 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:30.391+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.396175+0000) 2019-10-13T12:55:30.398 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:30.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.396729+0000) 2019-10-13T12:55:30.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:30.391+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.398358+0000) 2019-10-13T12:55:30.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:30.391+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.398472+0000) 2019-10-13T12:55:30.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:30.397+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.399261+0000) 2019-10-13T12:55:30.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:30.395+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.399845+0000) 2019-10-13T12:55:30.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:30.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:30.406 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:30.401+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:30.404388+0000) 2019-10-13T12:55:31.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:31.621+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.623452+0000) 2019-10-13T12:55:32.270 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:32.267+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:32.268810+0000) 2019-10-13T12:55:35.257 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:55:35.257 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:35.261 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:55:35.261 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:35.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:35.403+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.407486+0000) 2019-10-13T12:55:35.409 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:35.403+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.408045+0000) 2019-10-13T12:55:35.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:35.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.408379+0000) 2019-10-13T12:55:35.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:35.403+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.408960+0000) 2019-10-13T12:55:35.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:35.403+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.409565+0000) 2019-10-13T12:55:35.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:35.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.409975+0000) 2019-10-13T12:55:35.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:35.409+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.411145+0000) 2019-10-13T12:55:35.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:35.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:35.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:35.413+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.415943+0000) 2019-10-13T12:55:35.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:35.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.542673+0000) 2019-10-13T12:55:35.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:35.823+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:35.828951+0000) 2019-10-13T12:55:36.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:36.507+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.511500+0000) 2019-10-13T12:55:38.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:38.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:38.777110+0000) 2019-10-13T12:55:38.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:38.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:38.779701+0000) 2019-10-13T12:55:40.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:40.009+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.010584+0000) 2019-10-13T12:55:40.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:40.413+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.418069+0000) 2019-10-13T12:55:40.420 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:40.411+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.418273+0000) 2019-10-13T12:55:40.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:40.411+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.418661+0000) 2019-10-13T12:55:40.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:40.411+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.419170+0000) 2019-10-13T12:55:40.421 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:40.415+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.419953+0000) 2019-10-13T12:55:40.422 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:40.415+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.420426+0000) 2019-10-13T12:55:40.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:40.415+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.420579+0000) 2019-10-13T12:55:40.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:40.421+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:40.425223+0000) 2019-10-13T12:55:40.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:40.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159606 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:41.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:41.625+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.629818+0000) 2019-10-13T12:55:42.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:42.267+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:42.269032+0000) 2019-10-13T12:55:45.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:45.425+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.428854+0000) 2019-10-13T12:55:45.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:45.427+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.429205+0000) 2019-10-13T12:55:45.431 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:45.427+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.429561+0000) 2019-10-13T12:55:45.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:45.427+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.430283+0000) 2019-10-13T12:55:45.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:45.427+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.430705+0000) 2019-10-13T12:55:45.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:45.431+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.432019+0000) 2019-10-13T12:55:45.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:45.427+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.431081+0000) 2019-10-13T12:55:45.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:45.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159640 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:45.440 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:45.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.439213+0000) 2019-10-13T12:55:45.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:45.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.542854+0000) 2019-10-13T12:55:45.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:45.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:45.829145+0000) 2019-10-13T12:55:46.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:46.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.511696+0000) 2019-10-13T12:55:48.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:48.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:48.777311+0000) 2019-10-13T12:55:48.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:48.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:48.779889+0000) 2019-10-13T12:55:50.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:50.009+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.010755+0000) 2019-10-13T12:55:50.440 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:50.435+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.438728+0000) 2019-10-13T12:55:50.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:50.439+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.439385+0000) 2019-10-13T12:55:50.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:50.439+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.439605+0000) 2019-10-13T12:55:50.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:50.437+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.440527+0000) 2019-10-13T12:55:50.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:50.439+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.440966+0000) 2019-10-13T12:55:50.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:50.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.441227+0000) 2019-10-13T12:55:50.443 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:50.439+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.441678+0000) 2019-10-13T12:55:50.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:50.443+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:50.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:50.445+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:50.447108+0000) 2019-10-13T12:55:51.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:51.633+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:51.636420+0000) 2019-10-13T12:55:52.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:52.267+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.269263+0000) 2019-10-13T12:55:55.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:55.447+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.449653+0000) 2019-10-13T12:55:55.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:55:55.447+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.450529+0000) 2019-10-13T12:55:55.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:55.447+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.450920+0000) 2019-10-13T12:55:55.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:55:55.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.451450+0000) 2019-10-13T12:55:55.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:55.451+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.451648+0000) 2019-10-13T12:55:55.454 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:55.451+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.452392+0000) 2019-10-13T12:55:55.454 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:55.451+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.452333+0000) 2019-10-13T12:55:55.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:55:55.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:55:55.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:55:55.456+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.459099+0000) 2019-10-13T12:55:55.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:55:55.539+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.543078+0000) 2019-10-13T12:55:55.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:55:55.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:55.829390+0000) 2019-10-13T12:55:56.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:55:56.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:56.511916+0000) 2019-10-13T12:55:58.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:55:58.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:58.777526+0000) 2019-10-13T12:55:58.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:55:58.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:58.780085+0000) 2019-10-13T12:56:00.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:00.008+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.010974+0000) 2019-10-13T12:56:00.462 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:00.458+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.460481+0000) 2019-10-13T12:56:00.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:00.456+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.461897+0000) 2019-10-13T12:56:00.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:00.458+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.462181+0000) 2019-10-13T12:56:00.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:00.458+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.462367+0000) 2019-10-13T12:56:00.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:00.462+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.463749+0000) 2019-10-13T12:56:00.466 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:00.462+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.464266+0000) 2019-10-13T12:56:00.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:00.462+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.464295+0000) 2019-10-13T12:56:00.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:00.466+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:00.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:00.468+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:00.470260+0000) 2019-10-13T12:56:01.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:01.640+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:01.642729+0000) 2019-10-13T12:56:02.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:02.266+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.269495+0000) 2019-10-13T12:56:05.388 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:56:05.388 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:05.392 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:56:05.392 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:05.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:05.468+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.471423+0000) 2019-10-13T12:56:05.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:05.470+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.472213+0000) 2019-10-13T12:56:05.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:05.470+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.472793+0000) 2019-10-13T12:56:05.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:05.470+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.472985+0000) 2019-10-13T12:56:05.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:05.470+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.474290+0000) 2019-10-13T12:56:05.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:05.470+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.474755+0000) 2019-10-13T12:56:05.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:05.470+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.475033+0000) 2019-10-13T12:56:05.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:05.478+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:05.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:05.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.479718+0000) 2019-10-13T12:56:05.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:05.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.543273+0000) 2019-10-13T12:56:05.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:05.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:05.829570+0000) 2019-10-13T12:56:06.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:06.510+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:06.512142+0000) 2019-10-13T12:56:08.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:08.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:08.777721+0000) 2019-10-13T12:56:08.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:08.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:08.780305+0000) 2019-10-13T12:56:10.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:10.008+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.011169+0000) 2019-10-13T12:56:10.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:10.478+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.483273+0000) 2019-10-13T12:56:10.486 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:10.482+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.483585+0000) 2019-10-13T12:56:10.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:10.482+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.483975+0000) 2019-10-13T12:56:10.486 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:10.482+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.484659+0000) 2019-10-13T12:56:10.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:10.482+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.484900+0000) 2019-10-13T12:56:10.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:10.482+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.485278+0000) 2019-10-13T12:56:10.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:10.480+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.485724+0000) 2019-10-13T12:56:10.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:10.488+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:10.492469+0000) 2019-10-13T12:56:10.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:10.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:11.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:11.648+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:11.648970+0000) 2019-10-13T12:56:12.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:12.266+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.269712+0000) 2019-10-13T12:56:15.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:15.494+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.495376+0000) 2019-10-13T12:56:15.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:15.494+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.496097+0000) 2019-10-13T12:56:15.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:15.494+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.496448+0000) 2019-10-13T12:56:15.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:15.494+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.497268+0000) 2019-10-13T12:56:15.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:15.492+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.497998+0000) 2019-10-13T12:56:15.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:15.494+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.498052+0000) 2019-10-13T12:56:15.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:15.494+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.498357+0000) 2019-10-13T12:56:15.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:15.496+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.499406+0000) 2019-10-13T12:56:15.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:15.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:15.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:15.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.543481+0000) 2019-10-13T12:56:15.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:15.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:15.829756+0000) 2019-10-13T12:56:16.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:16.510+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:16.512339+0000) 2019-10-13T12:56:18.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:18.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:18.777947+0000) 2019-10-13T12:56:18.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:18.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:18.780504+0000) 2019-10-13T12:56:20.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:20.008+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.011365+0000) 2019-10-13T12:56:20.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:20.502+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.507082+0000) 2019-10-13T12:56:20.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:20.506+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.508309+0000) 2019-10-13T12:56:20.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:20.506+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.508751+0000) 2019-10-13T12:56:20.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:20.506+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.509178+0000) 2019-10-13T12:56:20.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:20.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.509753+0000) 2019-10-13T12:56:20.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:20.506+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.510006+0000) 2019-10-13T12:56:20.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:20.506+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.510113+0000) 2019-10-13T12:56:20.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:20.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:20.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:20.512+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:20.516527+0000) 2019-10-13T12:56:21.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:21.652+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:21.655247+0000) 2019-10-13T12:56:22.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:22.266+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.269907+0000) 2019-10-13T12:56:25.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:25.518+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.519418+0000) 2019-10-13T12:56:25.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:25.518+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.519943+0000) 2019-10-13T12:56:25.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:25.518+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.520525+0000) 2019-10-13T12:56:25.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:25.518+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.520840+0000) 2019-10-13T12:56:25.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:25.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.520572+0000) 2019-10-13T12:56:25.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:25.516+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.521823+0000) 2019-10-13T12:56:25.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:25.518+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.522786+0000) 2019-10-13T12:56:25.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:25.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:25.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:25.524+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.528786+0000) 2019-10-13T12:56:25.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:25.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.543640+0000) 2019-10-13T12:56:25.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:25.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:25.829942+0000) 2019-10-13T12:56:26.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:26.510+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:26.512531+0000) 2019-10-13T12:56:28.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:28.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:28.778149+0000) 2019-10-13T12:56:28.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:28.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:28.780699+0000) 2019-10-13T12:56:30.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:30.008+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.011562+0000) 2019-10-13T12:56:30.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:30.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.531056+0000) 2019-10-13T12:56:30.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:30.530+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.531456+0000) 2019-10-13T12:56:30.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:30.530+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.531912+0000) 2019-10-13T12:56:30.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:30.530+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.532219+0000) 2019-10-13T12:56:30.535 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:30.530+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.532490+0000) 2019-10-13T12:56:30.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:30.530+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.532774+0000) 2019-10-13T12:56:30.535 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:30.530+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.533993+0000) 2019-10-13T12:56:30.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:30.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:30.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:30.536+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:30.540891+0000) 2019-10-13T12:56:31.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:31.660+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:31.661591+0000) 2019-10-13T12:56:32.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:32.266+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.270106+0000) 2019-10-13T12:56:35.501 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:56:35.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:35.505 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:56:35.505 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:35.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:35.538+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.542177+0000) 2019-10-13T12:56:35.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:35.538+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.542501+0000) 2019-10-13T12:56:35.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:35.542+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.543392+0000) 2019-10-13T12:56:35.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:35.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.543771+0000) 2019-10-13T12:56:35.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:35.542+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.544269+0000) 2019-10-13T12:56:35.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:35.542+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.545038+0000) 2019-10-13T12:56:35.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:35.540+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.545382+0000) 2019-10-13T12:56:35.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:35.542+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.545809+0000) 2019-10-13T12:56:35.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:35.546+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 159973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:35.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:35.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.553651+0000) 2019-10-13T12:56:35.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:35.826+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:35.830171+0000) 2019-10-13T12:56:36.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:36.510+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:36.512696+0000) 2019-10-13T12:56:38.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:38.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:38.778352+0000) 2019-10-13T12:56:38.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:38.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:38.780894+0000) 2019-10-13T12:56:40.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:40.008+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.011787+0000) 2019-10-13T12:56:40.557 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:40.549+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.553446+0000) 2019-10-13T12:56:40.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:40.549+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.553492+0000) 2019-10-13T12:56:40.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:40.549+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.553983+0000) 2019-10-13T12:56:40.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:40.549+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.554469+0000) 2019-10-13T12:56:40.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:40.553+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.555330+0000) 2019-10-13T12:56:40.558 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:40.553+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.555364+0000) 2019-10-13T12:56:40.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:40.552+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.555340+0000) 2019-10-13T12:56:40.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:40.556+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:40.561919+0000) 2019-10-13T12:56:40.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:40.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:41.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:41.664+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:41.668202+0000) 2019-10-13T12:56:42.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:42.269+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.270336+0000) 2019-10-13T12:56:45.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:45.541+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.543959+0000) 2019-10-13T12:56:45.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:45.561+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.564972+0000) 2019-10-13T12:56:45.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:45.561+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.565247+0000) 2019-10-13T12:56:45.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:45.561+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.565981+0000) 2019-10-13T12:56:45.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:45.561+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.566282+0000) 2019-10-13T12:56:45.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:45.561+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.566722+0000) 2019-10-13T12:56:45.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:45.565+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.568082+0000) 2019-10-13T12:56:45.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:45.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.568920+0000) 2019-10-13T12:56:45.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:45.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:45.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:45.572+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.575112+0000) 2019-10-13T12:56:45.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:45.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:45.830356+0000) 2019-10-13T12:56:46.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:46.509+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:46.512869+0000) 2019-10-13T12:56:48.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:48.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:48.778551+0000) 2019-10-13T12:56:48.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:48.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:48.781098+0000) 2019-10-13T12:56:50.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:50.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.011992+0000) 2019-10-13T12:56:50.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:50.573+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.576630+0000) 2019-10-13T12:56:50.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:50.573+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.576981+0000) 2019-10-13T12:56:50.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:50.573+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.577253+0000) 2019-10-13T12:56:50.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:50.575+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.578236+0000) 2019-10-13T12:56:50.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:50.573+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.578358+0000) 2019-10-13T12:56:50.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:50.573+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.578577+0000) 2019-10-13T12:56:50.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:50.573+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.578921+0000) 2019-10-13T12:56:50.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:50.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:50.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:50.579+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:50.585168+0000) 2019-10-13T12:56:51.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:51.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:51.674587+0000) 2019-10-13T12:56:52.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:52.269+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.270554+0000) 2019-10-13T12:56:55.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:55.541+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.544199+0000) 2019-10-13T12:56:55.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:56:55.579+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.586040+0000) 2019-10-13T12:56:55.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:55.581+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.587077+0000) 2019-10-13T12:56:55.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:56:55.581+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.587019+0000) 2019-10-13T12:56:55.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:55.581+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.586600+0000) 2019-10-13T12:56:55.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:55.581+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.586697+0000) 2019-10-13T12:56:55.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:55.585+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.588954+0000) 2019-10-13T12:56:55.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:56:55.585+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.589426+0000) 2019-10-13T12:56:55.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:56:55.587+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.594067+0000) 2019-10-13T12:56:55.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:56:55.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:56:55.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:56:55.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:55.830538+0000) 2019-10-13T12:56:56.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:56:56.509+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:56.513034+0000) 2019-10-13T12:56:58.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:56:58.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:58.778751+0000) 2019-10-13T12:56:58.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:56:58.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:58.781309+0000) 2019-10-13T12:57:00.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:00.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.012212+0000) 2019-10-13T12:57:00.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:00.591+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.597772+0000) 2019-10-13T12:57:00.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:00.593+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.598060+0000) 2019-10-13T12:57:00.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:00.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.598430+0000) 2019-10-13T12:57:00.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:00.593+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.598898+0000) 2019-10-13T12:57:00.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:00.593+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.598668+0000) 2019-10-13T12:57:00.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:00.593+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.598697+0000) 2019-10-13T12:57:00.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:00.593+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.599241+0000) 2019-10-13T12:57:00.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:00.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:00.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:00.603+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:00.606923+0000) 2019-10-13T12:57:01.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:01.683+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:01.686169+0000) 2019-10-13T12:57:02.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:02.269+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.270753+0000) 2019-10-13T12:57:05.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:05.541+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.544427+0000) 2019-10-13T12:57:05.596 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:57:05.596 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:05.600 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:57:05.600 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:05.611 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:05.603+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.609930+0000) 2019-10-13T12:57:05.612 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:05.605+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.609654+0000) 2019-10-13T12:57:05.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:05.605+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.609917+0000) 2019-10-13T12:57:05.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:05.605+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.610233+0000) 2019-10-13T12:57:05.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:05.605+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.610770+0000) 2019-10-13T12:57:05.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:05.609+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.611484+0000) 2019-10-13T12:57:05.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:05.609+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.612107+0000) 2019-10-13T12:57:05.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:05.611+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.616691+0000) 2019-10-13T12:57:05.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:05.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:05.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:05.825+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:05.830717+0000) 2019-10-13T12:57:06.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:06.509+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:06.513242+0000) 2019-10-13T12:57:08.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:08.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:08.778954+0000) 2019-10-13T12:57:08.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:08.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:08.781463+0000) 2019-10-13T12:57:10.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:10.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.012424+0000) 2019-10-13T12:57:10.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:10.617+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.621412+0000) 2019-10-13T12:57:10.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:10.617+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.621762+0000) 2019-10-13T12:57:10.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:10.617+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.621567+0000) 2019-10-13T12:57:10.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:10.617+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.622330+0000) 2019-10-13T12:57:10.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:10.617+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.622601+0000) 2019-10-13T12:57:10.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:10.619+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.623328+0000) 2019-10-13T12:57:10.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:10.617+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.623236+0000) 2019-10-13T12:57:10.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:10.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:10.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:10.627+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:10.630660+0000) 2019-10-13T12:57:11.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:11.691+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:11.692604+0000) 2019-10-13T12:57:12.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:12.265+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.270906+0000) 2019-10-13T12:57:15.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:15.541+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.544687+0000) 2019-10-13T12:57:15.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:15.629+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.632649+0000) 2019-10-13T12:57:15.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:15.629+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.633430+0000) 2019-10-13T12:57:15.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:15.629+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.634854+0000) 2019-10-13T12:57:15.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:15.629+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.635059+0000) 2019-10-13T12:57:15.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:15.631+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.635593+0000) 2019-10-13T12:57:15.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:15.633+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.635412+0000) 2019-10-13T12:57:15.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:15.633+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.635569+0000) 2019-10-13T12:57:15.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:15.631+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.636616+0000) 2019-10-13T12:57:15.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:15.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:15.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:15.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:15.830901+0000) 2019-10-13T12:57:16.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:16.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:16.513418+0000) 2019-10-13T12:57:18.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:18.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:18.779165+0000) 2019-10-13T12:57:18.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:18.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:18.781660+0000) 2019-10-13T12:57:20.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:20.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.012629+0000) 2019-10-13T12:57:20.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:20.641+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.644114+0000) 2019-10-13T12:57:20.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:20.641+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.644686+0000) 2019-10-13T12:57:20.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:20.641+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.644945+0000) 2019-10-13T12:57:20.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:20.641+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.645415+0000) 2019-10-13T12:57:20.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:20.641+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.645735+0000) 2019-10-13T12:57:20.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:20.641+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.646118+0000) 2019-10-13T12:57:20.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:20.643+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.646724+0000) 2019-10-13T12:57:20.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:20.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:20.646877+0000) 2019-10-13T12:57:20.654 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:20.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:21.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:21.695+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:21.699028+0000) 2019-10-13T12:57:22.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:22.265+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.271090+0000) 2019-10-13T12:57:25.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:25.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.544911+0000) 2019-10-13T12:57:25.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:25.648+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.654107+0000) 2019-10-13T12:57:25.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:25.648+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.654613+0000) 2019-10-13T12:57:25.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:25.651+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.655251+0000) 2019-10-13T12:57:25.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:25.652+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.655494+0000) 2019-10-13T12:57:25.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:25.652+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.656642+0000) 2019-10-13T12:57:25.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:25.652+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.657522+0000) 2019-10-13T12:57:25.659 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:25.652+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.657968+0000) 2019-10-13T12:57:25.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:25.656+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:25.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:25.659+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.663528+0000) 2019-10-13T12:57:25.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:25.824+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:25.831083+0000) 2019-10-13T12:57:26.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:26.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:26.513581+0000) 2019-10-13T12:57:28.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:28.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:28.779363+0000) 2019-10-13T12:57:28.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:28.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:28.781882+0000) 2019-10-13T12:57:30.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:30.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.012864+0000) 2019-10-13T12:57:30.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:30.660+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.663731+0000) 2019-10-13T12:57:30.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:30.659+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.664859+0000) 2019-10-13T12:57:30.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:30.660+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.665101+0000) 2019-10-13T12:57:30.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:30.660+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.665472+0000) 2019-10-13T12:57:30.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:30.660+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.665844+0000) 2019-10-13T12:57:30.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:30.660+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.666868+0000) 2019-10-13T12:57:30.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:30.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.667217+0000) 2019-10-13T12:57:30.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:30.663+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:30.669730+0000) 2019-10-13T12:57:30.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:30.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:31.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:31.699+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:31.705450+0000) 2019-10-13T12:57:32.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:32.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.271289+0000) 2019-10-13T12:57:35.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:35.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.545117+0000) 2019-10-13T12:57:35.648 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:57:35.648 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:35.652 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:57:35.652 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:35.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:35.668+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.674482+0000) 2019-10-13T12:57:35.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:35.668+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.674956+0000) 2019-10-13T12:57:35.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:35.668+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.675281+0000) 2019-10-13T12:57:35.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:35.671+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.676427+0000) 2019-10-13T12:57:35.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:35.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.676008+0000) 2019-10-13T12:57:35.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:35.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.676759+0000) 2019-10-13T12:57:35.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:35.672+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.676916+0000) 2019-10-13T12:57:35.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:35.675+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.680914+0000) 2019-10-13T12:57:35.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:35.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160376 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:35.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:35.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:35.831268+0000) 2019-10-13T12:57:36.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:36.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:36.513755+0000) 2019-10-13T12:57:38.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:38.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:38.779531+0000) 2019-10-13T12:57:38.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:38.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:38.782072+0000) 2019-10-13T12:57:40.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:40.007+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.013056+0000) 2019-10-13T12:57:40.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:40.680+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.685063+0000) 2019-10-13T12:57:40.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:40.680+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.685363+0000) 2019-10-13T12:57:40.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:40.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.685925+0000) 2019-10-13T12:57:40.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:40.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.686567+0000) 2019-10-13T12:57:40.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:40.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.688159+0000) 2019-10-13T12:57:40.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:40.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.687432+0000) 2019-10-13T12:57:40.690 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:40.684+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.688499+0000) 2019-10-13T12:57:40.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:40.687+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:40.691127+0000) 2019-10-13T12:57:40.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:40.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:41.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:41.711+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:41.711881+0000) 2019-10-13T12:57:42.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:42.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.271495+0000) 2019-10-13T12:57:45.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:45.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.545316+0000) 2019-10-13T12:57:45.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:45.688+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.695199+0000) 2019-10-13T12:57:45.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:45.692+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.695746+0000) 2019-10-13T12:57:45.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:45.692+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.695881+0000) 2019-10-13T12:57:45.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:45.690+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.696498+0000) 2019-10-13T12:57:45.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:45.692+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.696521+0000) 2019-10-13T12:57:45.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:45.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.697655+0000) 2019-10-13T12:57:45.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:45.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.698560+0000) 2019-10-13T12:57:45.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:45.694+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.701449+0000) 2019-10-13T12:57:45.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:45.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:45.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:45.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:45.831481+0000) 2019-10-13T12:57:46.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:46.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:46.513932+0000) 2019-10-13T12:57:48.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:48.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:48.779744+0000) 2019-10-13T12:57:48.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:48.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:48.782265+0000) 2019-10-13T12:57:50.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:50.006+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.013245+0000) 2019-10-13T12:57:50.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:50.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.705117+0000) 2019-10-13T12:57:50.707 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:50.700+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.706111+0000) 2019-10-13T12:57:50.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:50.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.707075+0000) 2019-10-13T12:57:50.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:50.704+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.707526+0000) 2019-10-13T12:57:50.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:50.702+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.708180+0000) 2019-10-13T12:57:50.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:50.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.708300+0000) 2019-10-13T12:57:50.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:50.704+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.709369+0000) 2019-10-13T12:57:50.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:50.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:50.716 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:50.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:50.715310+0000) 2019-10-13T12:57:51.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:51.714+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:51.718098+0000) 2019-10-13T12:57:52.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:52.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.271696+0000) 2019-10-13T12:57:55.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:55.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.545511+0000) 2019-10-13T12:57:55.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:57:55.712+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.716436+0000) 2019-10-13T12:57:55.719 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:55.712+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.717116+0000) 2019-10-13T12:57:55.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:55.712+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.717459+0000) 2019-10-13T12:57:55.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:57:55.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.718219+0000) 2019-10-13T12:57:55.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:55.712+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.718853+0000) 2019-10-13T12:57:55.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:57:55.714+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.719664+0000) 2019-10-13T12:57:55.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:55.716+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.719719+0000) 2019-10-13T12:57:55.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:57:55.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:57:55.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:57:55.722+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.727492+0000) 2019-10-13T12:57:55.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:57:55.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:55.831716+0000) 2019-10-13T12:57:56.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:57:56.508+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:56.514103+0000) 2019-10-13T12:57:58.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:57:58.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:58.779954+0000) 2019-10-13T12:57:58.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:57:58.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:58.782420+0000) 2019-10-13T12:58:00.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:00.006+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.013441+0000) 2019-10-13T12:58:00.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:00.720+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.725541+0000) 2019-10-13T12:58:00.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:00.720+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.726370+0000) 2019-10-13T12:58:00.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:00.720+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.726611+0000) 2019-10-13T12:58:00.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:00.722+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.727474+0000) 2019-10-13T12:58:00.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:00.720+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.727003+0000) 2019-10-13T12:58:00.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:00.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.727702+0000) 2019-10-13T12:58:00.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:00.724+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.729136+0000) 2019-10-13T12:58:00.735 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:00.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:00.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:00.730+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:00.735375+0000) 2019-10-13T12:58:01.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:01.718+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:01.724151+0000) 2019-10-13T12:58:02.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:02.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.271878+0000) 2019-10-13T12:58:05.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:05.540+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.545707+0000) 2019-10-13T12:58:05.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:05.730+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.736475+0000) 2019-10-13T12:58:05.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:05.732+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.737021+0000) 2019-10-13T12:58:05.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:05.732+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.737285+0000) 2019-10-13T12:58:05.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:05.732+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.737888+0000) 2019-10-13T12:58:05.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:05.732+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.738439+0000) 2019-10-13T12:58:05.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:05.732+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.738700+0000) 2019-10-13T12:58:05.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:05.732+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.739233+0000) 2019-10-13T12:58:05.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:05.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:05.748 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:05.742+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.747241+0000) 2019-10-13T12:58:05.768 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:58:05.768 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:05.772 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:58:05.772 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:05.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:05.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:05.831860+0000) 2019-10-13T12:58:06.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:06.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:06.514307+0000) 2019-10-13T12:58:08.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:08.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:08.780170+0000) 2019-10-13T12:58:08.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:08.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:08.782623+0000) 2019-10-13T12:58:10.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:10.006+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.013670+0000) 2019-10-13T12:58:10.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:10.743+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.747456+0000) 2019-10-13T12:58:10.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:10.742+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.748244+0000) 2019-10-13T12:58:10.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:10.743+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.748354+0000) 2019-10-13T12:58:10.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:10.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.748941+0000) 2019-10-13T12:58:10.751 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:10.743+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.749178+0000) 2019-10-13T12:58:10.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:10.743+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.749929+0000) 2019-10-13T12:58:10.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:10.743+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.750658+0000) 2019-10-13T12:58:10.757 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:10.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:10.760 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:10.754+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:10.758802+0000) 2019-10-13T12:58:11.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:11.730+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:11.730611+0000) 2019-10-13T12:58:12.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:12.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.272074+0000) 2019-10-13T12:58:15.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:15.543+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.545902+0000) 2019-10-13T12:58:15.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:15.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.758948+0000) 2019-10-13T12:58:15.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:15.754+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.759422+0000) 2019-10-13T12:58:15.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:15.755+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.759793+0000) 2019-10-13T12:58:15.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:15.755+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.760725+0000) 2019-10-13T12:58:15.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:15.755+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.760809+0000) 2019-10-13T12:58:15.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:15.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.761574+0000) 2019-10-13T12:58:15.764 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:15.755+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.762233+0000) 2019-10-13T12:58:15.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:15.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:15.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:15.762+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.767563+0000) 2019-10-13T12:58:15.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:15.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:15.832007+0000) 2019-10-13T12:58:16.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:16.507+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:16.514482+0000) 2019-10-13T12:58:18.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:18.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:18.780367+0000) 2019-10-13T12:58:18.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:18.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:18.782823+0000) 2019-10-13T12:58:20.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:20.006+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.013878+0000) 2019-10-13T12:58:20.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:20.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.770175+0000) 2019-10-13T12:58:20.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:20.767+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.771689+0000) 2019-10-13T12:58:20.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:20.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.771786+0000) 2019-10-13T12:58:20.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:20.767+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.772559+0000) 2019-10-13T12:58:20.774 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:20.767+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.773027+0000) 2019-10-13T12:58:20.775 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:20.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.773986+0000) 2019-10-13T12:58:20.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:20.767+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.774149+0000) 2019-10-13T12:58:20.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:20.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:20.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:20.778+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:20.782205+0000) 2019-10-13T12:58:21.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:21.734+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:21.736948+0000) 2019-10-13T12:58:22.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:22.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.272276+0000) 2019-10-13T12:58:25.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:25.543+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.546100+0000) 2019-10-13T12:58:25.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:25.775+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.781157+0000) 2019-10-13T12:58:25.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:25.775+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.782702+0000) 2019-10-13T12:58:25.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:25.779+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.783432+0000) 2019-10-13T12:58:25.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:25.779+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.783989+0000) 2019-10-13T12:58:25.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:25.778+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.784719+0000) 2019-10-13T12:58:25.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:25.779+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.784166+0000) 2019-10-13T12:58:25.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:25.779+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.784817+0000) 2019-10-13T12:58:25.792 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:25.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:25.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:25.786+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.790558+0000) 2019-10-13T12:58:25.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:25.827+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:25.832157+0000) 2019-10-13T12:58:26.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:26.507+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:26.514656+0000) 2019-10-13T12:58:28.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:28.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:28.780566+0000) 2019-10-13T12:58:28.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:28.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:28.783062+0000) 2019-10-13T12:58:30.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:30.010+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.014078+0000) 2019-10-13T12:58:30.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:30.787+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.792581+0000) 2019-10-13T12:58:30.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:30.787+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.793193+0000) 2019-10-13T12:58:30.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:30.787+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.793658+0000) 2019-10-13T12:58:30.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:30.787+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.794750+0000) 2019-10-13T12:58:30.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:30.787+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.795007+0000) 2019-10-13T12:58:30.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:30.791+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.795927+0000) 2019-10-13T12:58:30.798 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:30.794+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.797030+0000) 2019-10-13T12:58:30.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:30.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:30.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:30.802+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:30.805990+0000) 2019-10-13T12:58:31.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:31.742+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:31.742741+0000) 2019-10-13T12:58:32.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:32.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.272504+0000) 2019-10-13T12:58:35.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:35.543+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.546334+0000) 2019-10-13T12:58:35.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:35.799+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.804276+0000) 2019-10-13T12:58:35.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:35.799+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.804883+0000) 2019-10-13T12:58:35.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:35.799+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.805557+0000) 2019-10-13T12:58:35.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:35.806+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.806308+0000) 2019-10-13T12:58:35.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:35.799+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.806487+0000) 2019-10-13T12:58:35.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:35.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.806849+0000) 2019-10-13T12:58:35.809 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:35.799+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.807244+0000) 2019-10-13T12:58:35.814 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:35.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:35.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:35.814+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.814552+0000) 2019-10-13T12:58:35.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:35.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:35.832351+0000) 2019-10-13T12:58:35.879 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:58:35.879 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:35.884 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:58:35.884 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:36.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:36.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:36.514830+0000) 2019-10-13T12:58:38.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:38.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:38.780797+0000) 2019-10-13T12:58:38.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:38.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:38.783253+0000) 2019-10-13T12:58:40.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:40.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.014301+0000) 2019-10-13T12:58:40.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:40.815+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.815800+0000) 2019-10-13T12:58:40.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:40.815+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.816081+0000) 2019-10-13T12:58:40.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:40.813+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.817116+0000) 2019-10-13T12:58:40.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:40.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.816694+0000) 2019-10-13T12:58:40.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:40.815+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.818332+0000) 2019-10-13T12:58:40.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:40.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.818659+0000) 2019-10-13T12:58:40.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:40.819+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.819737+0000) 2019-10-13T12:58:40.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:40.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160809 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:40.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:40.821+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:40.825972+0000) 2019-10-13T12:58:41.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:41.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:41.749400+0000) 2019-10-13T12:58:42.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:42.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.272738+0000) 2019-10-13T12:58:45.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:45.543+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.546537+0000) 2019-10-13T12:58:45.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:45.827+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.828730+0000) 2019-10-13T12:58:45.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:45.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.828944+0000) 2019-10-13T12:58:45.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:45.827+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.829455+0000) 2019-10-13T12:58:45.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:45.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.830599+0000) 2019-10-13T12:58:45.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:45.829+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.831143+0000) 2019-10-13T12:58:45.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:45.831+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.831397+0000) 2019-10-13T12:58:45.833 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:45.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.832071+0000) 2019-10-13T12:58:45.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:45.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.832489+0000) 2019-10-13T12:58:45.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:45.835+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160845 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:45.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:45.837+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:45.839081+0000) 2019-10-13T12:58:46.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:46.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:46.515035+0000) 2019-10-13T12:58:48.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:48.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:48.781002+0000) 2019-10-13T12:58:48.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:48.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:48.783478+0000) 2019-10-13T12:58:50.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:50.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.014516+0000) 2019-10-13T12:58:50.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:50.838+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.840373+0000) 2019-10-13T12:58:50.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:50.838+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.841241+0000) 2019-10-13T12:58:50.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:50.838+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.841760+0000) 2019-10-13T12:58:50.844 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:50.838+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.842106+0000) 2019-10-13T12:58:50.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:50.841+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.843267+0000) 2019-10-13T12:58:50.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:50.842+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.843981+0000) 2019-10-13T12:58:50.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:50.842+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.844380+0000) 2019-10-13T12:58:50.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:50.845+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:50.847388+0000) 2019-10-13T12:58:50.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:50.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:51.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:51.753+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:51.755801+0000) 2019-10-13T12:58:52.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:52.270+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.272943+0000) 2019-10-13T12:58:55.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:55.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.546753+0000) 2019-10-13T12:58:55.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:55.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.832691+0000) 2019-10-13T12:58:55.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:55.850+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.852042+0000) 2019-10-13T12:58:55.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:58:55.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.852444+0000) 2019-10-13T12:58:55.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:58:55.849+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.852977+0000) 2019-10-13T12:58:55.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:58:55.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.853248+0000) 2019-10-13T12:58:55.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:55.850+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.854364+0000) 2019-10-13T12:58:55.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:58:55.850+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.854986+0000) 2019-10-13T12:58:55.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:55.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.856062+0000) 2019-10-13T12:58:55.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:58:55.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:58:55.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:58:55.857+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:55.860536+0000) 2019-10-13T12:58:56.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:58:56.510+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:56.515230+0000) 2019-10-13T12:58:58.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:58:58.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:58.781203+0000) 2019-10-13T12:58:58.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:58:58.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:58.783660+0000) 2019-10-13T12:59:00.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:00.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.014745+0000) 2019-10-13T12:59:00.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:00.858+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.862520+0000) 2019-10-13T12:59:00.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:00.862+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.863732+0000) 2019-10-13T12:59:00.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:00.862+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.864584+0000) 2019-10-13T12:59:00.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:00.862+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.865007+0000) 2019-10-13T12:59:00.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:00.861+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.865576+0000) 2019-10-13T12:59:00.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:00.862+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.865908+0000) 2019-10-13T12:59:00.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:00.862+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.866213+0000) 2019-10-13T12:59:00.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:00.869+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:00.870684+0000) 2019-10-13T12:59:00.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:00.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:01.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:01.761+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:01.762240+0000) 2019-10-13T12:59:02.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:02.270+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.273141+0000) 2019-10-13T12:59:05.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:05.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.546954+0000) 2019-10-13T12:59:05.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:05.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.832914+0000) 2019-10-13T12:59:05.876 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:05.870+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.873962+0000) 2019-10-13T12:59:05.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:05.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.874446+0000) 2019-10-13T12:59:05.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:05.870+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.875144+0000) 2019-10-13T12:59:05.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:05.874+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.875725+0000) 2019-10-13T12:59:05.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:05.874+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.875885+0000) 2019-10-13T12:59:05.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:05.873+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.876373+0000) 2019-10-13T12:59:05.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:05.874+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.876884+0000) 2019-10-13T12:59:05.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:05.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 160978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:05.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:05.881+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:05.883069+0000) 2019-10-13T12:59:05.990 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:59:05.990 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:05.994 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:59:05.994 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:06.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:06.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:06.515416+0000) 2019-10-13T12:59:08.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:08.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:08.781401+0000) 2019-10-13T12:59:08.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:08.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:08.783854+0000) 2019-10-13T12:59:10.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:10.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.014960+0000) 2019-10-13T12:59:10.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:10.882+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.884619+0000) 2019-10-13T12:59:10.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:10.882+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.885278+0000) 2019-10-13T12:59:10.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:10.882+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.885774+0000) 2019-10-13T12:59:10.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:10.882+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.886319+0000) 2019-10-13T12:59:10.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:10.885+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.887362+0000) 2019-10-13T12:59:10.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:10.882+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.886940+0000) 2019-10-13T12:59:10.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:10.886+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.887760+0000) 2019-10-13T12:59:10.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:10.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:10.896 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:10.893+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:10.894756+0000) 2019-10-13T12:59:11.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:11.765+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:11.768493+0000) 2019-10-13T12:59:12.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:12.270+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.273341+0000) 2019-10-13T12:59:15.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:15.542+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.547154+0000) 2019-10-13T12:59:15.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:15.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.833079+0000) 2019-10-13T12:59:15.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:15.894+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.896439+0000) 2019-10-13T12:59:15.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:15.894+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.896815+0000) 2019-10-13T12:59:15.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:15.894+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.897243+0000) 2019-10-13T12:59:15.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:15.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.898129+0000) 2019-10-13T12:59:15.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:15.894+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.898551+0000) 2019-10-13T12:59:15.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:15.894+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.899148+0000) 2019-10-13T12:59:15.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:15.898+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.899793+0000) 2019-10-13T12:59:15.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:15.901+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:15.905061+0000) 2019-10-13T12:59:15.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:15.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:16.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:16.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:16.515607+0000) 2019-10-13T12:59:18.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:18.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:18.781604+0000) 2019-10-13T12:59:18.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:18.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:18.784052+0000) 2019-10-13T12:59:20.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:20.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.015162+0000) 2019-10-13T12:59:20.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:20.906+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.908735+0000) 2019-10-13T12:59:20.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:20.905+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.909639+0000) 2019-10-13T12:59:20.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:20.906+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.910096+0000) 2019-10-13T12:59:20.912 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:20.906+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.910897+0000) 2019-10-13T12:59:20.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:20.906+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.911246+0000) 2019-10-13T12:59:20.913 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:20.910+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.911552+0000) 2019-10-13T12:59:20.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:20.910+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.911882+0000) 2019-10-13T12:59:20.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:20.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:20.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:20.913+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:20.917209+0000) 2019-10-13T12:59:21.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:21.773+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:21.774809+0000) 2019-10-13T12:59:22.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:22.270+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.273547+0000) 2019-10-13T12:59:25.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:25.546+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.547356+0000) 2019-10-13T12:59:25.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:25.830+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.833281+0000) 2019-10-13T12:59:25.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:25.918+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.920488+0000) 2019-10-13T12:59:25.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:25.918+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.921015+0000) 2019-10-13T12:59:25.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:25.918+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.921265+0000) 2019-10-13T12:59:25.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:25.917+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.921746+0000) 2019-10-13T12:59:25.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:25.918+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.922297+0000) 2019-10-13T12:59:25.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:25.918+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.922909+0000) 2019-10-13T12:59:25.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:25.922+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.923828+0000) 2019-10-13T12:59:25.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:25.925+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:25.928261+0000) 2019-10-13T12:59:25.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:25.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:26.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:26.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:26.515780+0000) 2019-10-13T12:59:28.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:28.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:28.781804+0000) 2019-10-13T12:59:28.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:28.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:28.784248+0000) 2019-10-13T12:59:30.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:30.013+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.015358+0000) 2019-10-13T12:59:30.933 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:30.930+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.931789+0000) 2019-10-13T12:59:30.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:30.928+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.932331+0000) 2019-10-13T12:59:30.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:30.930+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.932695+0000) 2019-10-13T12:59:30.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:30.930+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.933569+0000) 2019-10-13T12:59:30.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:30.930+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.933910+0000) 2019-10-13T12:59:30.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:30.930+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.934162+0000) 2019-10-13T12:59:30.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:30.930+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.935261+0000) 2019-10-13T12:59:30.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:30.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161143 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:30.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:30.936+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:30.941491+0000) 2019-10-13T12:59:31.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:31.780+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:31.781212+0000) 2019-10-13T12:59:32.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:32.270+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.273746+0000) 2019-10-13T12:59:35.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:35.545+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.547560+0000) 2019-10-13T12:59:35.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:35.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.833482+0000) 2019-10-13T12:59:35.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:35.937+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.942783+0000) 2019-10-13T12:59:35.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:35.937+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.942466+0000) 2019-10-13T12:59:35.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:35.941+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.943758+0000) 2019-10-13T12:59:35.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:35.941+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.943469+0000) 2019-10-13T12:59:35.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:35.941+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.944469+0000) 2019-10-13T12:59:35.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:35.940+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.945244+0000) 2019-10-13T12:59:35.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:35.941+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.946288+0000) 2019-10-13T12:59:35.951 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:35.945+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:35.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:35.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:35.950909+0000) 2019-10-13T12:59:36.126 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T12:59:36.126 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:36.130 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T12:59:36.130 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:36.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:36.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:36.515973+0000) 2019-10-13T12:59:38.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:38.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:38.782028+0000) 2019-10-13T12:59:38.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:38.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:38.784469+0000) 2019-10-13T12:59:40.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:40.012+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.015553+0000) 2019-10-13T12:59:40.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:40.949+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.953239+0000) 2019-10-13T12:59:40.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:40.948+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.953707+0000) 2019-10-13T12:59:40.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:40.949+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.954520+0000) 2019-10-13T12:59:40.956 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:40.949+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.954644+0000) 2019-10-13T12:59:40.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:40.953+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.955594+0000) 2019-10-13T12:59:40.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:40.953+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.956190+0000) 2019-10-13T12:59:40.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:40.953+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.956551+0000) 2019-10-13T12:59:40.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:40.956+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:40.960666+0000) 2019-10-13T12:59:40.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:40.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:41.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:41.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:41.787647+0000) 2019-10-13T12:59:42.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:42.273+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.273982+0000) 2019-10-13T12:59:45.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:45.545+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.547769+0000) 2019-10-13T12:59:45.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:45.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.833687+0000) 2019-10-13T12:59:45.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:45.960+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.964464+0000) 2019-10-13T12:59:45.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:45.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.965291+0000) 2019-10-13T12:59:45.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:45.961+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.965749+0000) 2019-10-13T12:59:45.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:45.961+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.966679+0000) 2019-10-13T12:59:45.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:45.961+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.966874+0000) 2019-10-13T12:59:45.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:45.965+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.967354+0000) 2019-10-13T12:59:45.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:45.965+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.968953+0000) 2019-10-13T12:59:45.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:45.968+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:45.972831+0000) 2019-10-13T12:59:45.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:45.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:46.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:46.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:46.516099+0000) 2019-10-13T12:59:48.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:48.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:48.782224+0000) 2019-10-13T12:59:48.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:48.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:48.784675+0000) 2019-10-13T12:59:50.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:50.012+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.015738+0000) 2019-10-13T12:59:50.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:50.973+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.976663+0000) 2019-10-13T12:59:50.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:50.972+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.977209+0000) 2019-10-13T12:59:50.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:50.973+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.977688+0000) 2019-10-13T12:59:50.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:50.973+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.978088+0000) 2019-10-13T12:59:50.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:50.973+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.978580+0000) 2019-10-13T12:59:50.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:50.977+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.979333+0000) 2019-10-13T12:59:50.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:50.977+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.979664+0000) 2019-10-13T12:59:50.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:50.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:50.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:50.980+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:50.985156+0000) 2019-10-13T12:59:51.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:51.792+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:51.794289+0000) 2019-10-13T12:59:52.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:52.273+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.274140+0000) 2019-10-13T12:59:55.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:55.545+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.548028+0000) 2019-10-13T12:59:55.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:55.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.833842+0000) 2019-10-13T12:59:55.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T12:59:55.984+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.987521+0000) 2019-10-13T12:59:55.989 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:55.985+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.987855+0000) 2019-10-13T12:59:55.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:55.985+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.988511+0000) 2019-10-13T12:59:55.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T12:59:55.985+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.988876+0000) 2019-10-13T12:59:55.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T12:59:55.985+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.989754+0000) 2019-10-13T12:59:55.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T12:59:55.985+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.990375+0000) 2019-10-13T12:59:55.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:55.985+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.990653+0000) 2019-10-13T12:59:55.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T12:59:55.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T12:59:56.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T12:59:55.996+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:55.998770+0000) 2019-10-13T12:59:56.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T12:59:56.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:56.516302+0000) 2019-10-13T12:59:58.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T12:59:58.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:58.782419+0000) 2019-10-13T12:59:58.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T12:59:58.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:58.784870+0000) 2019-10-13T13:00:00.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:00.012+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:00.015920+0000) 2019-10-13T13:00:01.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:00.993+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:00.999076+0000) 2019-10-13T13:00:01.001 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:00.997+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:00.999379+0000) 2019-10-13T13:00:01.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:00.997+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.000200+0000) 2019-10-13T13:00:01.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:00.997+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:00.999891+0000) 2019-10-13T13:00:01.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:00.997+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.000711+0000) 2019-10-13T13:00:01.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:00.996+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.000911+0000) 2019-10-13T13:00:01.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:00.997+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.001725+0000) 2019-10-13T13:00:01.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:01.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:01.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:01.004+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.008365+0000) 2019-10-13T13:00:01.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:01.796+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.801103+0000) 2019-10-13T13:00:02.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:02.269+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.274369+0000) 2019-10-13T13:00:05.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:05.545+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:05.548226+0000) 2019-10-13T13:00:05.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:05.829+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:05.834073+0000) 2019-10-13T13:00:06.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:06.005+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.009742+0000) 2019-10-13T13:00:06.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:06.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.011210+0000) 2019-10-13T13:00:06.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:06.009+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.011455+0000) 2019-10-13T13:00:06.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:06.009+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.012004+0000) 2019-10-13T13:00:06.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:06.008+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.013073+0000) 2019-10-13T13:00:06.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:06.009+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.013159+0000) 2019-10-13T13:00:06.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:06.009+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.013417+0000) 2019-10-13T13:00:06.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:06.016+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.018843+0000) 2019-10-13T13:00:06.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:06.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:06.247 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:00:06.247 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:06.250 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:00:06.250 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:06.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:06.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.516474+0000) 2019-10-13T13:00:08.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:08.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:08.782648+0000) 2019-10-13T13:00:08.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:08.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:08.785094+0000) 2019-10-13T13:00:10.017 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:10.012+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:10.016102+0000) 2019-10-13T13:00:11.024 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:11.017+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.022167+0000) 2019-10-13T13:00:11.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:11.017+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.023204+0000) 2019-10-13T13:00:11.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:11.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.023416+0000) 2019-10-13T13:00:11.026 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:11.021+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.023870+0000) 2019-10-13T13:00:11.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:11.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.024179+0000) 2019-10-13T13:00:11.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:11.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.025750+0000) 2019-10-13T13:00:11.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:11.021+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.025836+0000) 2019-10-13T13:00:11.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:11.028+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.030640+0000) 2019-10-13T13:00:11.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:11.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:11.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:11.804+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.807754+0000) 2019-10-13T13:00:12.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:12.273+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.274578+0000) 2019-10-13T13:00:15.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:15.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:15.548388+0000) 2019-10-13T13:00:15.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:15.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:15.834270+0000) 2019-10-13T13:00:16.036 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:16.028+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.034185+0000) 2019-10-13T13:00:16.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:16.028+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.034843+0000) 2019-10-13T13:00:16.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:16.028+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.034888+0000) 2019-10-13T13:00:16.037 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:16.028+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.035268+0000) 2019-10-13T13:00:16.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:16.032+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.036196+0000) 2019-10-13T13:00:16.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:16.032+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.036730+0000) 2019-10-13T13:00:16.038 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:16.032+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.037134+0000) 2019-10-13T13:00:16.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:16.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161442 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:16.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:16.040+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.042776+0000) 2019-10-13T13:00:16.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:16.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.516664+0000) 2019-10-13T13:00:18.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:18.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:18.782875+0000) 2019-10-13T13:00:18.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:18.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:18.785290+0000) 2019-10-13T13:00:20.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:20.012+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:20.016268+0000) 2019-10-13T13:00:21.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:21.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.045755+0000) 2019-10-13T13:00:21.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:21.040+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.046020+0000) 2019-10-13T13:00:21.048 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:21.040+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.046606+0000) 2019-10-13T13:00:21.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:21.040+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.046685+0000) 2019-10-13T13:00:21.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:21.040+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.047139+0000) 2019-10-13T13:00:21.049 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:21.044+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.047941+0000) 2019-10-13T13:00:21.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:21.044+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.049022+0000) 2019-10-13T13:00:21.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:21.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:21.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:21.052+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.054698+0000) 2019-10-13T13:00:21.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:21.812+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.814111+0000) 2019-10-13T13:00:22.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:22.268+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.274788+0000) 2019-10-13T13:00:25.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:25.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:25.548544+0000) 2019-10-13T13:00:25.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:25.828+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:25.834468+0000) 2019-10-13T13:00:26.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:26.051+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.057156+0000) 2019-10-13T13:00:26.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:26.052+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.057212+0000) 2019-10-13T13:00:26.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:26.052+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.058361+0000) 2019-10-13T13:00:26.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:26.052+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.058780+0000) 2019-10-13T13:00:26.061 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:26.052+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.058954+0000) 2019-10-13T13:00:26.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:26.056+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.059288+0000) 2019-10-13T13:00:26.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:26.056+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.060516+0000) 2019-10-13T13:00:26.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:26.059+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.064383+0000) 2019-10-13T13:00:26.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:26.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:26.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:26.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.516839+0000) 2019-10-13T13:00:28.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:28.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:28.783098+0000) 2019-10-13T13:00:28.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:28.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:28.785487+0000) 2019-10-13T13:00:30.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:30.011+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:30.016465+0000) 2019-10-13T13:00:31.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:31.063+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.068411+0000) 2019-10-13T13:00:31.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:31.064+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.068756+0000) 2019-10-13T13:00:31.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:31.064+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.068862+0000) 2019-10-13T13:00:31.071 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:31.064+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.069244+0000) 2019-10-13T13:00:31.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:31.064+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.069378+0000) 2019-10-13T13:00:31.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:31.064+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.069975+0000) 2019-10-13T13:00:31.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:31.064+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.070867+0000) 2019-10-13T13:00:31.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:31.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:31.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:31.075+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.079195+0000) 2019-10-13T13:00:31.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:31.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.820532+0000) 2019-10-13T13:00:32.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:32.272+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.274988+0000) 2019-10-13T13:00:35.550 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:35.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:35.548774+0000) 2019-10-13T13:00:35.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:35.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:35.834670+0000) 2019-10-13T13:00:36.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:36.076+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.079482+0000) 2019-10-13T13:00:36.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:36.076+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.079834+0000) 2019-10-13T13:00:36.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:36.076+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.080825+0000) 2019-10-13T13:00:36.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:36.075+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.081227+0000) 2019-10-13T13:00:36.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:36.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.081383+0000) 2019-10-13T13:00:36.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:36.076+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.081801+0000) 2019-10-13T13:00:36.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:36.076+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.082637+0000) 2019-10-13T13:00:36.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:36.080+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:36.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:36.083+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.088734+0000) 2019-10-13T13:00:36.348 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:00:36.348 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:36.352 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:00:36.352 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:36.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:36.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.517013+0000) 2019-10-13T13:00:38.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:38.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:38.783337+0000) 2019-10-13T13:00:38.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:38.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:38.785676+0000) 2019-10-13T13:00:40.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:40.011+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:40.016656+0000) 2019-10-13T13:00:41.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:41.087+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.091540+0000) 2019-10-13T13:00:41.094 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:41.088+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.091686+0000) 2019-10-13T13:00:41.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:41.088+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.092311+0000) 2019-10-13T13:00:41.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:41.088+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.092584+0000) 2019-10-13T13:00:41.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:41.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.092304+0000) 2019-10-13T13:00:41.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:41.088+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.093437+0000) 2019-10-13T13:00:41.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:41.088+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.093835+0000) 2019-10-13T13:00:41.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:41.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:41.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:41.099+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.103346+0000) 2019-10-13T13:00:41.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:41.823+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.826935+0000) 2019-10-13T13:00:42.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:42.272+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.275217+0000) 2019-10-13T13:00:45.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:45.548+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:45.548997+0000) 2019-10-13T13:00:45.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:45.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:45.834870+0000) 2019-10-13T13:00:46.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:46.099+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.102369+0000) 2019-10-13T13:00:46.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:46.096+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.102641+0000) 2019-10-13T13:00:46.105 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:46.096+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.103066+0000) 2019-10-13T13:00:46.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:46.100+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.103649+0000) 2019-10-13T13:00:46.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:46.100+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.103996+0000) 2019-10-13T13:00:46.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:46.100+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.104245+0000) 2019-10-13T13:00:46.106 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:46.100+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.104889+0000) 2019-10-13T13:00:46.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:46.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:46.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:46.107+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.111086+0000) 2019-10-13T13:00:46.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:46.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.517213+0000) 2019-10-13T13:00:48.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:48.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:48.783542+0000) 2019-10-13T13:00:48.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:48.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:48.785876+0000) 2019-10-13T13:00:50.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:50.011+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:50.016879+0000) 2019-10-13T13:00:51.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:51.108+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.113619+0000) 2019-10-13T13:00:51.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:51.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.113993+0000) 2019-10-13T13:00:51.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:51.111+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.114471+0000) 2019-10-13T13:00:51.117 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:51.108+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.115132+0000) 2019-10-13T13:00:51.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:51.108+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.114791+0000) 2019-10-13T13:00:51.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:51.112+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.116304+0000) 2019-10-13T13:00:51.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:51.112+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.116073+0000) 2019-10-13T13:00:51.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:51.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:51.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:51.119+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.122191+0000) 2019-10-13T13:00:51.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:51.827+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.833147+0000) 2019-10-13T13:00:52.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:52.272+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.275420+0000) 2019-10-13T13:00:55.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:55.544+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:55.549201+0000) 2019-10-13T13:00:55.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:55.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:55.835078+0000) 2019-10-13T13:00:56.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:56.120+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.124385+0000) 2019-10-13T13:00:56.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:00:56.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.125373+0000) 2019-10-13T13:00:56.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:00:56.120+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.125636+0000) 2019-10-13T13:00:56.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:56.120+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.126317+0000) 2019-10-13T13:00:56.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:00:56.123+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.126818+0000) 2019-10-13T13:00:56.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:00:56.120+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.127000+0000) 2019-10-13T13:00:56.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:56.124+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.127651+0000) 2019-10-13T13:00:56.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:00:56.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:00:56.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:00:56.131+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.134920+0000) 2019-10-13T13:00:56.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:00:56.512+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.517388+0000) 2019-10-13T13:00:58.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:00:58.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:58.783770+0000) 2019-10-13T13:00:58.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:00:58.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:58.786071+0000) 2019-10-13T13:01:00.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:00.011+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:00.017075+0000) 2019-10-13T13:01:01.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:01.131+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.136827+0000) 2019-10-13T13:01:01.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:01.131+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.136942+0000) 2019-10-13T13:01:01.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:01.131+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.137214+0000) 2019-10-13T13:01:01.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:01.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.138481+0000) 2019-10-13T13:01:01.140 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:01.131+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.138633+0000) 2019-10-13T13:01:01.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:01.131+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.138978+0000) 2019-10-13T13:01:01.141 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:01.135+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.139915+0000) 2019-10-13T13:01:01.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:01.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:01.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:01.139+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.145969+0000) 2019-10-13T13:01:01.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:01.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.839459+0000) 2019-10-13T13:01:02.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:02.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.275585+0000) 2019-10-13T13:01:05.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:05.543+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:05.549397+0000) 2019-10-13T13:01:05.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:05.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:05.835344+0000) 2019-10-13T13:01:06.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:06.143+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.147879+0000) 2019-10-13T13:01:06.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:06.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.148364+0000) 2019-10-13T13:01:06.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:06.143+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.148790+0000) 2019-10-13T13:01:06.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:06.143+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.149413+0000) 2019-10-13T13:01:06.151 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:06.143+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.149911+0000) 2019-10-13T13:01:06.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:06.143+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.150613+0000) 2019-10-13T13:01:06.152 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:06.143+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.151177+0000) 2019-10-13T13:01:06.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:06.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:06.159 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:06.151+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.157802+0000) 2019-10-13T13:01:06.478 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:01:06.478 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:06.482 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:01:06.482 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:06.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:06.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.517571+0000) 2019-10-13T13:01:08.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:08.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:08.783967+0000) 2019-10-13T13:01:08.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:08.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:08.786271+0000) 2019-10-13T13:01:10.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:10.011+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:10.017285+0000) 2019-10-13T13:01:11.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:11.155+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.158885+0000) 2019-10-13T13:01:11.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:11.155+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.160096+0000) 2019-10-13T13:01:11.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:11.155+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.160805+0000) 2019-10-13T13:01:11.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:11.155+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.161446+0000) 2019-10-13T13:01:11.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:11.155+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.161835+0000) 2019-10-13T13:01:11.164 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:11.159+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.162631+0000) 2019-10-13T13:01:11.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:11.155+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.163266+0000) 2019-10-13T13:01:11.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:11.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:11.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:11.163+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.169486+0000) 2019-10-13T13:01:11.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:11.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.845839+0000) 2019-10-13T13:01:12.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:12.271+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.275785+0000) 2019-10-13T13:01:15.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:15.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:15.549625+0000) 2019-10-13T13:01:15.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:15.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:15.835574+0000) 2019-10-13T13:01:16.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:16.163+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.171232+0000) 2019-10-13T13:01:16.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:16.167+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.171643+0000) 2019-10-13T13:01:16.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:16.167+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.172075+0000) 2019-10-13T13:01:16.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:16.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.172654+0000) 2019-10-13T13:01:16.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:16.167+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.173011+0000) 2019-10-13T13:01:16.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:16.167+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.173607+0000) 2019-10-13T13:01:16.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:16.167+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.174170+0000) 2019-10-13T13:01:16.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:16.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:16.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:16.175+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.179838+0000) 2019-10-13T13:01:16.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:16.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.517763+0000) 2019-10-13T13:01:18.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:18.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:18.784178+0000) 2019-10-13T13:01:18.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:18.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:18.786471+0000) 2019-10-13T13:01:20.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:20.010+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:20.017512+0000) 2019-10-13T13:01:21.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:21.175+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.182432+0000) 2019-10-13T13:01:21.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:21.179+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.183483+0000) 2019-10-13T13:01:21.186 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:21.178+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.184352+0000) 2019-10-13T13:01:21.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:21.179+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.184812+0000) 2019-10-13T13:01:21.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:21.179+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.185050+0000) 2019-10-13T13:01:21.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:21.179+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.185424+0000) 2019-10-13T13:01:21.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:21.179+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.186481+0000) 2019-10-13T13:01:21.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:21.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:21.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:21.186+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.191788+0000) 2019-10-13T13:01:21.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:21.850+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.852298+0000) 2019-10-13T13:01:22.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:22.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.276013+0000) 2019-10-13T13:01:25.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:25.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:25.549831+0000) 2019-10-13T13:01:25.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:25.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:25.835773+0000) 2019-10-13T13:01:26.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:26.190+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.194262+0000) 2019-10-13T13:01:26.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:26.187+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.194430+0000) 2019-10-13T13:01:26.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:26.191+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.195369+0000) 2019-10-13T13:01:26.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:26.191+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.196307+0000) 2019-10-13T13:01:26.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:26.191+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.196916+0000) 2019-10-13T13:01:26.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:26.191+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.196943+0000) 2019-10-13T13:01:26.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:26.191+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.197207+0000) 2019-10-13T13:01:26.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:26.194+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.202010+0000) 2019-10-13T13:01:26.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:26.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:26.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:26.511+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.517936+0000) 2019-10-13T13:01:28.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:28.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:28.784436+0000) 2019-10-13T13:01:28.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:28.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:28.786666+0000) 2019-10-13T13:01:30.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:30.010+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:30.017702+0000) 2019-10-13T13:01:31.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:31.203+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.206015+0000) 2019-10-13T13:01:31.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:31.203+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.206445+0000) 2019-10-13T13:01:31.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:31.203+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.207079+0000) 2019-10-13T13:01:31.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:31.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.207547+0000) 2019-10-13T13:01:31.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:31.207+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.208046+0000) 2019-10-13T13:01:31.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:31.202+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.208769+0000) 2019-10-13T13:01:31.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:31.207+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.208769+0000) 2019-10-13T13:01:31.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:31.206+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.211925+0000) 2019-10-13T13:01:31.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:31.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:31.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:31.854+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.858675+0000) 2019-10-13T13:01:32.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:32.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.276248+0000) 2019-10-13T13:01:35.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:35.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:35.550030+0000) 2019-10-13T13:01:35.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:35.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:35.835924+0000) 2019-10-13T13:01:36.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:36.215+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.216643+0000) 2019-10-13T13:01:36.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:36.214+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.218204+0000) 2019-10-13T13:01:36.220 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:36.215+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.218333+0000) 2019-10-13T13:01:36.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:36.215+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.218463+0000) 2019-10-13T13:01:36.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:36.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.219196+0000) 2019-10-13T13:01:36.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:36.219+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.219428+0000) 2019-10-13T13:01:36.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:36.219+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.219958+0000) 2019-10-13T13:01:36.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:36.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.224309+0000) 2019-10-13T13:01:36.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:36.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 161975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:36.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:36.515+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.518113+0000) 2019-10-13T13:01:36.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:01:36.535 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:36.538 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:01:36.538 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:38.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:38.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:38.784636+0000) 2019-10-13T13:01:38.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:38.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:38.786888+0000) 2019-10-13T13:01:40.019 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:40.010+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:40.017894+0000) 2019-10-13T13:01:41.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:41.227+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.229002+0000) 2019-10-13T13:01:41.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:41.227+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.229388+0000) 2019-10-13T13:01:41.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:41.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.229834+0000) 2019-10-13T13:01:41.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:41.227+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.230063+0000) 2019-10-13T13:01:41.232 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:41.226+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.230897+0000) 2019-10-13T13:01:41.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:41.231+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.231374+0000) 2019-10-13T13:01:41.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:41.231+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.231577+0000) 2019-10-13T13:01:41.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:41.230+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.234285+0000) 2019-10-13T13:01:41.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:41.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:41.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:41.862+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.865166+0000) 2019-10-13T13:01:42.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:42.274+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.276443+0000) 2019-10-13T13:01:45.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:45.546+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:45.550260+0000) 2019-10-13T13:01:45.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:45.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:45.836129+0000) 2019-10-13T13:01:46.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:46.234+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.238546+0000) 2019-10-13T13:01:46.241 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:46.234+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.240189+0000) 2019-10-13T13:01:46.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:46.238+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.240572+0000) 2019-10-13T13:01:46.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:46.238+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.240935+0000) 2019-10-13T13:01:46.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:46.238+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.241265+0000) 2019-10-13T13:01:46.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:46.238+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.242022+0000) 2019-10-13T13:01:46.244 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:46.238+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.242909+0000) 2019-10-13T13:01:46.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:46.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:46.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:46.246+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.250760+0000) 2019-10-13T13:01:46.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:46.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.518261+0000) 2019-10-13T13:01:48.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:48.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:48.784843+0000) 2019-10-13T13:01:48.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:48.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:48.787111+0000) 2019-10-13T13:01:50.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:50.010+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:50.018089+0000) 2019-10-13T13:01:51.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:51.246+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.251831+0000) 2019-10-13T13:01:51.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:51.250+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.252271+0000) 2019-10-13T13:01:51.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:51.250+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.252648+0000) 2019-10-13T13:01:51.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:51.250+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.253182+0000) 2019-10-13T13:01:51.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:51.250+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.253458+0000) 2019-10-13T13:01:51.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:51.254+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.255451+0000) 2019-10-13T13:01:51.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:51.254+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.255703+0000) 2019-10-13T13:01:51.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:51.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:51.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:51.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.264701+0000) 2019-10-13T13:01:51.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:51.870+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.872711+0000) 2019-10-13T13:01:52.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:52.274+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.276687+0000) 2019-10-13T13:01:55.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:55.546+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:55.550455+0000) 2019-10-13T13:01:55.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:55.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:55.836386+0000) 2019-10-13T13:01:56.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:56.258+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.261808+0000) 2019-10-13T13:01:56.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:56.258+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.262465+0000) 2019-10-13T13:01:56.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:01:56.258+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.262752+0000) 2019-10-13T13:01:56.265 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:01:56.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.263422+0000) 2019-10-13T13:01:56.265 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:56.262+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.263606+0000) 2019-10-13T13:01:56.266 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:01:56.262+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.263979+0000) 2019-10-13T13:01:56.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:01:56.262+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.264570+0000) 2019-10-13T13:01:56.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:01:56.266+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:01:56.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:01:56.266+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.272738+0000) 2019-10-13T13:01:56.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:01:56.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.518430+0000) 2019-10-13T13:01:58.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:01:58.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:58.785076+0000) 2019-10-13T13:01:58.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:01:58.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:58.787334+0000) 2019-10-13T13:02:00.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:00.014+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:00.018274+0000) 2019-10-13T13:02:01.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:01.270+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.273251+0000) 2019-10-13T13:02:01.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:01.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.273879+0000) 2019-10-13T13:02:01.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:01.270+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.274065+0000) 2019-10-13T13:02:01.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:01.270+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.274297+0000) 2019-10-13T13:02:01.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:01.270+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.275062+0000) 2019-10-13T13:02:01.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:01.274+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.275762+0000) 2019-10-13T13:02:01.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:01.274+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.276475+0000) 2019-10-13T13:02:01.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:01.278+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:01.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:01.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.283592+0000) 2019-10-13T13:02:01.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:01.878+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.879227+0000) 2019-10-13T13:02:02.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:02.274+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.276889+0000) 2019-10-13T13:02:05.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:05.546+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:05.550652+0000) 2019-10-13T13:02:05.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:05.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:05.836602+0000) 2019-10-13T13:02:06.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:06.282+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.285218+0000) 2019-10-13T13:02:06.288 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:06.282+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.286075+0000) 2019-10-13T13:02:06.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:06.282+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.286342+0000) 2019-10-13T13:02:06.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:06.282+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.286510+0000) 2019-10-13T13:02:06.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:06.282+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.286994+0000) 2019-10-13T13:02:06.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:06.286+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.287349+0000) 2019-10-13T13:02:06.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:06.286+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.287770+0000) 2019-10-13T13:02:06.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:06.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:06.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:06.290+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.295610+0000) 2019-10-13T13:02:06.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:06.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.518602+0000) 2019-10-13T13:02:06.799 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:02:06.800 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:06.803 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:02:06.804 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:08.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:08.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:08.785232+0000) 2019-10-13T13:02:08.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:08.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:08.787494+0000) 2019-10-13T13:02:10.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:10.018+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:10.018467+0000) 2019-10-13T13:02:11.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:11.294+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.298168+0000) 2019-10-13T13:02:11.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:11.294+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.298711+0000) 2019-10-13T13:02:11.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:11.294+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.298939+0000) 2019-10-13T13:02:11.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:11.298+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.299563+0000) 2019-10-13T13:02:11.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:11.298+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.299918+0000) 2019-10-13T13:02:11.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:11.298+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.300456+0000) 2019-10-13T13:02:11.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:11.298+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.301783+0000) 2019-10-13T13:02:11.307 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:11.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:11.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:11.306+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.306358+0000) 2019-10-13T13:02:11.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:11.882+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.885644+0000) 2019-10-13T13:02:12.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:12.274+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.277124+0000) 2019-10-13T13:02:15.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:15.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:15.550852+0000) 2019-10-13T13:02:15.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:15.834+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:15.836815+0000) 2019-10-13T13:02:16.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:16.306+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.308915+0000) 2019-10-13T13:02:16.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:16.306+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.310220+0000) 2019-10-13T13:02:16.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:16.306+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.310690+0000) 2019-10-13T13:02:16.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:16.310+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.311452+0000) 2019-10-13T13:02:16.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:16.310+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.311757+0000) 2019-10-13T13:02:16.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:16.310+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.312333+0000) 2019-10-13T13:02:16.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:16.309+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.312663+0000) 2019-10-13T13:02:16.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:16.313+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.317283+0000) 2019-10-13T13:02:16.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:16.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:16.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:16.514+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.518777+0000) 2019-10-13T13:02:18.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:18.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:18.785430+0000) 2019-10-13T13:02:18.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:18.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:18.787705+0000) 2019-10-13T13:02:20.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:20.017+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:20.018683+0000) 2019-10-13T13:02:21.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:21.318+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.321380+0000) 2019-10-13T13:02:21.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:21.318+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.321947+0000) 2019-10-13T13:02:21.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:21.318+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.322403+0000) 2019-10-13T13:02:21.324 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:21.318+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.322682+0000) 2019-10-13T13:02:21.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:21.322+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.323457+0000) 2019-10-13T13:02:21.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:21.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.324200+0000) 2019-10-13T13:02:21.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:21.321+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.324916+0000) 2019-10-13T13:02:21.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:21.325+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.329475+0000) 2019-10-13T13:02:21.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:21.326+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:21.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:21.889+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.892190+0000) 2019-10-13T13:02:22.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:22.274+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.277308+0000) 2019-10-13T13:02:25.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:25.545+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:25.551047+0000) 2019-10-13T13:02:25.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:25.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:25.837045+0000) 2019-10-13T13:02:26.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:26.329+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.333023+0000) 2019-10-13T13:02:26.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:26.329+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.334053+0000) 2019-10-13T13:02:26.336 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:26.329+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.334223+0000) 2019-10-13T13:02:26.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:26.329+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.334704+0000) 2019-10-13T13:02:26.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:26.333+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.335294+0000) 2019-10-13T13:02:26.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:26.333+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.336657+0000) 2019-10-13T13:02:26.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:26.333+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.335919+0000) 2019-10-13T13:02:26.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:26.337+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:26.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:26.337+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.341835+0000) 2019-10-13T13:02:26.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:26.513+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.518911+0000) 2019-10-13T13:02:28.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:28.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:28.785662+0000) 2019-10-13T13:02:28.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:28.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:28.787916+0000) 2019-10-13T13:02:30.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:30.017+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:30.018839+0000) 2019-10-13T13:02:31.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:31.341+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.344310+0000) 2019-10-13T13:02:31.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:31.341+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.344629+0000) 2019-10-13T13:02:31.347 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:31.341+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.345241+0000) 2019-10-13T13:02:31.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:31.341+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.345610+0000) 2019-10-13T13:02:31.348 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:31.341+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.346064+0000) 2019-10-13T13:02:31.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:31.341+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.346530+0000) 2019-10-13T13:02:31.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:31.341+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.347240+0000) 2019-10-13T13:02:31.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:31.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:31.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:31.349+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.353627+0000) 2019-10-13T13:02:31.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:31.897+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.898871+0000) 2019-10-13T13:02:32.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:32.273+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.277482+0000) 2019-10-13T13:02:35.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:35.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:35.551280+0000) 2019-10-13T13:02:35.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:35.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:35.837296+0000) 2019-10-13T13:02:36.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:36.353+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.355765+0000) 2019-10-13T13:02:36.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:36.353+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.356208+0000) 2019-10-13T13:02:36.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:36.353+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.356663+0000) 2019-10-13T13:02:36.359 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:36.353+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.356967+0000) 2019-10-13T13:02:36.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:36.353+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.357329+0000) 2019-10-13T13:02:36.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:36.357+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.358623+0000) 2019-10-13T13:02:36.361 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:36.353+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.359272+0000) 2019-10-13T13:02:36.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:36.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:36.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:36.361+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.364953+0000) 2019-10-13T13:02:36.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:36.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.519128+0000) 2019-10-13T13:02:36.860 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:02:36.861 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:36.865 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:02:36.865 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:38.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:38.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:38.785865+0000) 2019-10-13T13:02:38.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:38.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:38.788114+0000) 2019-10-13T13:02:40.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:40.017+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:40.019091+0000) 2019-10-13T13:02:41.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:41.361+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.366907+0000) 2019-10-13T13:02:41.369 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:41.365+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.367624+0000) 2019-10-13T13:02:41.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:41.365+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.368533+0000) 2019-10-13T13:02:41.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:41.365+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.369150+0000) 2019-10-13T13:02:41.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:41.365+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.369474+0000) 2019-10-13T13:02:41.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:41.365+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.369927+0000) 2019-10-13T13:02:41.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:41.365+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.370274+0000) 2019-10-13T13:02:41.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:41.373+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:41.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:41.373+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.377007+0000) 2019-10-13T13:02:41.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:41.901+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.905081+0000) 2019-10-13T13:02:42.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:42.277+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.277665+0000) 2019-10-13T13:02:45.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:45.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:45.551523+0000) 2019-10-13T13:02:45.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:45.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:45.837494+0000) 2019-10-13T13:02:46.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:46.373+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.379068+0000) 2019-10-13T13:02:46.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:46.377+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.379525+0000) 2019-10-13T13:02:46.383 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:46.377+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.381299+0000) 2019-10-13T13:02:46.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:46.377+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.381851+0000) 2019-10-13T13:02:46.384 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:46.377+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.382097+0000) 2019-10-13T13:02:46.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:46.377+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.382852+0000) 2019-10-13T13:02:46.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:46.381+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.383365+0000) 2019-10-13T13:02:46.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:46.385+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.387605+0000) 2019-10-13T13:02:46.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:46.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:46.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:46.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.519343+0000) 2019-10-13T13:02:48.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:48.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.786099+0000) 2019-10-13T13:02:48.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:48.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.788341+0000) 2019-10-13T13:02:50.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:50.017+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:50.019309+0000) 2019-10-13T13:02:51.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:51.389+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.392038+0000) 2019-10-13T13:02:51.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:51.389+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.392227+0000) 2019-10-13T13:02:51.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:51.389+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.392615+0000) 2019-10-13T13:02:51.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:51.389+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.393278+0000) 2019-10-13T13:02:51.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:51.389+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.393424+0000) 2019-10-13T13:02:51.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:51.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.393928+0000) 2019-10-13T13:02:51.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:51.393+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.395902+0000) 2019-10-13T13:02:51.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:51.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:51.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:51.401+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.403102+0000) 2019-10-13T13:02:51.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:51.909+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.911436+0000) 2019-10-13T13:02:52.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:52.273+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:52.277873+0000) 2019-10-13T13:02:55.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:55.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:55.551714+0000) 2019-10-13T13:02:55.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:55.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:55.837693+0000) 2019-10-13T13:02:56.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:56.397+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.402919+0000) 2019-10-13T13:02:56.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:56.401+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.403690+0000) 2019-10-13T13:02:56.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:56.401+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.404368+0000) 2019-10-13T13:02:56.406 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:02:56.401+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.405117+0000) 2019-10-13T13:02:56.407 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:02:56.401+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.405408+0000) 2019-10-13T13:02:56.408 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:02:56.401+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.405568+0000) 2019-10-13T13:02:56.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:02:56.401+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.406001+0000) 2019-10-13T13:02:56.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:02:56.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:02:56.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:02:56.409+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.413715+0000) 2019-10-13T13:02:56.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:02:56.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.519581+0000) 2019-10-13T13:02:58.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:02:58.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.786304+0000) 2019-10-13T13:02:58.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:02:58.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.788570+0000) 2019-10-13T13:03:00.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:00.017+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:00.019524+0000) 2019-10-13T13:03:01.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:01.409+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.414899+0000) 2019-10-13T13:03:01.417 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:01.413+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.415984+0000) 2019-10-13T13:03:01.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:01.413+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.416462+0000) 2019-10-13T13:03:01.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:01.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.417235+0000) 2019-10-13T13:03:01.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:01.413+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.416723+0000) 2019-10-13T13:03:01.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:01.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.417367+0000) 2019-10-13T13:03:01.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:01.413+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.418092+0000) 2019-10-13T13:03:01.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:01.421+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:01.429 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:01.425+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.428152+0000) 2019-10-13T13:03:01.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:01.913+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.917613+0000) 2019-10-13T13:03:02.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:02.277+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:02.278058+0000) 2019-10-13T13:03:05.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:05.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:05.551934+0000) 2019-10-13T13:03:05.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:05.833+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:05.837888+0000) 2019-10-13T13:03:06.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:06.421+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.426466+0000) 2019-10-13T13:03:06.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:06.425+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.427362+0000) 2019-10-13T13:03:06.430 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:06.425+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.428575+0000) 2019-10-13T13:03:06.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:06.425+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.428949+0000) 2019-10-13T13:03:06.431 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:06.425+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.429539+0000) 2019-10-13T13:03:06.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:06.425+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.429791+0000) 2019-10-13T13:03:06.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:06.425+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.430283+0000) 2019-10-13T13:03:06.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:06.433+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.434784+0000) 2019-10-13T13:03:06.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:06.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:06.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:06.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.519796+0000) 2019-10-13T13:03:06.972 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:03:06.972 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:06.976 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:03:06.976 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:08.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:08.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.786549+0000) 2019-10-13T13:03:08.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:08.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.788782+0000) 2019-10-13T13:03:10.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:10.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:10.019748+0000) 2019-10-13T13:03:11.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:11.432+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.439151+0000) 2019-10-13T13:03:11.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:11.436+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.439709+0000) 2019-10-13T13:03:11.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:11.436+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.440282+0000) 2019-10-13T13:03:11.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:11.436+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.440279+0000) 2019-10-13T13:03:11.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:11.436+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.440695+0000) 2019-10-13T13:03:11.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:11.436+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.441644+0000) 2019-10-13T13:03:11.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:11.440+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.442815+0000) 2019-10-13T13:03:11.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:11.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:11.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:11.448+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.450808+0000) 2019-10-13T13:03:11.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:11.920+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.923947+0000) 2019-10-13T13:03:12.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:12.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:12.278240+0000) 2019-10-13T13:03:15.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:15.548+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:15.552133+0000) 2019-10-13T13:03:15.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:15.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:15.838085+0000) 2019-10-13T13:03:16.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:16.444+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.449828+0000) 2019-10-13T13:03:16.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:16.444+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.450295+0000) 2019-10-13T13:03:16.453 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:16.448+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.451551+0000) 2019-10-13T13:03:16.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:16.448+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.452150+0000) 2019-10-13T13:03:16.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:16.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.453171+0000) 2019-10-13T13:03:16.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:16.448+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.453388+0000) 2019-10-13T13:03:16.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:16.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.453828+0000) 2019-10-13T13:03:16.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:16.452+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:16.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:16.456+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.459328+0000) 2019-10-13T13:03:16.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:16.516+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.519991+0000) 2019-10-13T13:03:18.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:18.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.786749+0000) 2019-10-13T13:03:18.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:18.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.788982+0000) 2019-10-13T13:03:20.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:20.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:20.019961+0000) 2019-10-13T13:03:21.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:21.460+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.462838+0000) 2019-10-13T13:03:21.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:21.456+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.462947+0000) 2019-10-13T13:03:21.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:21.460+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.463361+0000) 2019-10-13T13:03:21.465 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:21.460+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.463594+0000) 2019-10-13T13:03:21.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:21.460+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.464102+0000) 2019-10-13T13:03:21.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:21.460+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.464510+0000) 2019-10-13T13:03:21.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:21.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.465643+0000) 2019-10-13T13:03:21.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:21.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:21.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:21.468+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.472807+0000) 2019-10-13T13:03:21.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:21.928+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.930437+0000) 2019-10-13T13:03:22.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:22.272+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:22.278421+0000) 2019-10-13T13:03:25.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:25.548+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:25.552290+0000) 2019-10-13T13:03:25.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:25.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:25.838303+0000) 2019-10-13T13:03:26.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:26.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.472651+0000) 2019-10-13T13:03:26.475 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:26.468+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.473077+0000) 2019-10-13T13:03:26.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:26.468+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.474192+0000) 2019-10-13T13:03:26.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:26.468+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.474617+0000) 2019-10-13T13:03:26.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:26.472+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.475283+0000) 2019-10-13T13:03:26.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:26.468+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.475271+0000) 2019-10-13T13:03:26.477 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:26.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.475706+0000) 2019-10-13T13:03:26.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:26.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:26.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:26.480+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.482826+0000) 2019-10-13T13:03:26.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:26.516+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.520162+0000) 2019-10-13T13:03:28.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:28.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.786974+0000) 2019-10-13T13:03:28.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:28.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.789202+0000) 2019-10-13T13:03:30.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:30.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:30.020186+0000) 2019-10-13T13:03:31.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:31.480+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.483984+0000) 2019-10-13T13:03:31.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:31.480+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.484929+0000) 2019-10-13T13:03:31.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:31.480+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.485805+0000) 2019-10-13T13:03:31.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:31.484+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.486420+0000) 2019-10-13T13:03:31.488 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:31.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.486493+0000) 2019-10-13T13:03:31.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:31.484+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.487767+0000) 2019-10-13T13:03:31.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:31.484+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.488231+0000) 2019-10-13T13:03:31.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:31.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162744 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:31.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:31.492+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.495805+0000) 2019-10-13T13:03:31.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:31.936+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.937132+0000) 2019-10-13T13:03:32.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:32.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:32.278608+0000) 2019-10-13T13:03:35.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:35.552+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:35.552509+0000) 2019-10-13T13:03:35.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:35.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:35.838503+0000) 2019-10-13T13:03:36.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:36.492+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.496060+0000) 2019-10-13T13:03:36.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:36.492+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.496495+0000) 2019-10-13T13:03:36.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:36.492+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.497041+0000) 2019-10-13T13:03:36.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:36.492+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.497709+0000) 2019-10-13T13:03:36.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:36.492+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.498128+0000) 2019-10-13T13:03:36.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:36.492+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.498291+0000) 2019-10-13T13:03:36.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:36.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.498912+0000) 2019-10-13T13:03:36.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:36.500+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.503639+0000) 2019-10-13T13:03:36.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:36.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:36.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:36.516+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.520305+0000) 2019-10-13T13:03:37.083 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:03:37.083 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:37.087 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:03:37.087 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:38.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:38.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.787172+0000) 2019-10-13T13:03:38.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:38.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.789397+0000) 2019-10-13T13:03:40.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:40.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:40.020419+0000) 2019-10-13T13:03:41.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:41.504+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.507775+0000) 2019-10-13T13:03:41.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:41.504+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.508013+0000) 2019-10-13T13:03:41.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:41.504+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.509182+0000) 2019-10-13T13:03:41.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:41.504+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.509809+0000) 2019-10-13T13:03:41.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:41.504+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.509640+0000) 2019-10-13T13:03:41.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:41.504+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.510597+0000) 2019-10-13T13:03:41.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:41.504+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.510910+0000) 2019-10-13T13:03:41.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:41.512+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:41.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:41.512+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.516907+0000) 2019-10-13T13:03:41.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:41.948+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.948854+0000) 2019-10-13T13:03:42.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:42.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:42.278812+0000) 2019-10-13T13:03:45.554 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:45.548+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:45.552709+0000) 2019-10-13T13:03:45.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:45.832+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:45.838720+0000) 2019-10-13T13:03:46.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:46.516+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.519517+0000) 2019-10-13T13:03:46.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:46.516+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.520040+0000) 2019-10-13T13:03:46.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:46.516+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.520445+0000) 2019-10-13T13:03:46.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:46.516+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.520455+0000) 2019-10-13T13:03:46.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:46.516+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.520982+0000) 2019-10-13T13:03:46.523 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:46.516+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.521554+0000) 2019-10-13T13:03:46.524 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:46.516+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.521865+0000) 2019-10-13T13:03:46.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:46.520+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.522745+0000) 2019-10-13T13:03:46.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:46.524+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:46.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:46.524+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.529552+0000) 2019-10-13T13:03:48.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:48.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.787407+0000) 2019-10-13T13:03:48.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:48.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.789590+0000) 2019-10-13T13:03:50.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:50.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:50.020614+0000) 2019-10-13T13:03:51.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:51.527+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.532407+0000) 2019-10-13T13:03:51.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:51.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.532889+0000) 2019-10-13T13:03:51.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:51.527+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.533473+0000) 2019-10-13T13:03:51.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:51.527+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.533812+0000) 2019-10-13T13:03:51.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:51.532+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.534343+0000) 2019-10-13T13:03:51.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:51.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.534582+0000) 2019-10-13T13:03:51.537 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:51.527+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.535179+0000) 2019-10-13T13:03:51.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:51.535+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162878 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:51.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:51.540+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.542391+0000) 2019-10-13T13:03:51.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:51.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.960302+0000) 2019-10-13T13:03:52.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:52.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:52.278993+0000) 2019-10-13T13:03:55.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:55.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:55.552930+0000) 2019-10-13T13:03:55.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:55.831+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:55.838953+0000) 2019-10-13T13:03:56.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:56.515+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.520633+0000) 2019-10-13T13:03:56.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:03:56.540+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.543027+0000) 2019-10-13T13:03:56.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:03:56.539+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.543536+0000) 2019-10-13T13:03:56.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:56.539+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.544046+0000) 2019-10-13T13:03:56.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:03:56.539+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.544573+0000) 2019-10-13T13:03:56.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:03:56.539+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.545216+0000) 2019-10-13T13:03:56.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:03:56.539+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.546108+0000) 2019-10-13T13:03:56.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:56.539+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.546474+0000) 2019-10-13T13:03:56.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:03:56.547+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:03:56.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:03:56.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.552525+0000) 2019-10-13T13:03:58.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:03:58.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.787608+0000) 2019-10-13T13:03:58.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:03:58.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.789791+0000) 2019-10-13T13:04:00.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:00.016+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:00.020778+0000) 2019-10-13T13:04:01.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:01.547+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.555223+0000) 2019-10-13T13:04:01.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:01.552+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.555797+0000) 2019-10-13T13:04:01.558 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:01.551+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.556320+0000) 2019-10-13T13:04:01.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:01.551+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.556948+0000) 2019-10-13T13:04:01.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:01.551+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.557133+0000) 2019-10-13T13:04:01.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:01.551+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.557484+0000) 2019-10-13T13:04:01.560 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:01.551+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.558695+0000) 2019-10-13T13:04:01.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:01.560+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.564387+0000) 2019-10-13T13:04:01.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:01.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162946 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:01.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:01.968+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.971878+0000) 2019-10-13T13:04:02.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:02.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:02.279173+0000) 2019-10-13T13:04:05.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:05.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:05.553155+0000) 2019-10-13T13:04:05.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:05.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:05.839161+0000) 2019-10-13T13:04:06.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:06.519+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.520830+0000) 2019-10-13T13:04:06.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:06.563+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.567637+0000) 2019-10-13T13:04:06.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:06.563+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.567964+0000) 2019-10-13T13:04:06.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:06.563+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.568367+0000) 2019-10-13T13:04:06.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:06.563+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.568915+0000) 2019-10-13T13:04:06.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:06.563+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.569294+0000) 2019-10-13T13:04:06.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:06.563+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.570110+0000) 2019-10-13T13:04:06.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:06.563+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.570309+0000) 2019-10-13T13:04:06.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:06.571+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.576073+0000) 2019-10-13T13:04:06.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:06.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 162980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:07.203 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:04:07.203 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:07.207 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:04:07.207 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:08.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:08.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.787790+0000) 2019-10-13T13:04:08.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:08.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.790022+0000) 2019-10-13T13:04:10.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:10.015+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:10.021031+0000) 2019-10-13T13:04:11.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:11.571+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.579259+0000) 2019-10-13T13:04:11.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:11.575+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.579722+0000) 2019-10-13T13:04:11.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:11.575+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.581302+0000) 2019-10-13T13:04:11.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:11.575+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.580733+0000) 2019-10-13T13:04:11.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:11.575+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.580685+0000) 2019-10-13T13:04:11.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:11.575+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.581679+0000) 2019-10-13T13:04:11.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:11.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.582701+0000) 2019-10-13T13:04:11.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:11.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163012 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:11.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:11.583+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.588588+0000) 2019-10-13T13:04:11.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:11.975+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.978335+0000) 2019-10-13T13:04:12.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:12.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:12.279381+0000) 2019-10-13T13:04:15.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:15.547+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:15.553353+0000) 2019-10-13T13:04:15.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:15.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:15.839413+0000) 2019-10-13T13:04:16.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:16.519+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.521057+0000) 2019-10-13T13:04:16.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:16.583+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.590281+0000) 2019-10-13T13:04:16.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:16.583+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.590812+0000) 2019-10-13T13:04:16.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:16.587+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.591566+0000) 2019-10-13T13:04:16.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:16.587+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.592438+0000) 2019-10-13T13:04:16.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:16.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.592398+0000) 2019-10-13T13:04:16.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:16.587+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.592704+0000) 2019-10-13T13:04:16.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:16.587+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.593383+0000) 2019-10-13T13:04:16.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:16.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:16.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:16.595+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.600490+0000) 2019-10-13T13:04:18.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:18.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.787953+0000) 2019-10-13T13:04:18.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:18.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.790213+0000) 2019-10-13T13:04:20.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:20.015+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:20.021229+0000) 2019-10-13T13:04:21.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:21.603+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.603917+0000) 2019-10-13T13:04:21.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:21.603+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.604005+0000) 2019-10-13T13:04:21.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:21.603+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.604621+0000) 2019-10-13T13:04:21.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:21.599+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.604976+0000) 2019-10-13T13:04:21.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:21.603+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.605504+0000) 2019-10-13T13:04:21.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:21.603+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.606178+0000) 2019-10-13T13:04:21.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:21.603+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.606924+0000) 2019-10-13T13:04:21.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:21.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:21.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:21.607+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.611739+0000) 2019-10-13T13:04:21.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:21.979+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.984272+0000) 2019-10-13T13:04:22.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:22.279+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:22.279574+0000) 2019-10-13T13:04:25.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:25.551+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:25.553566+0000) 2019-10-13T13:04:25.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:25.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:25.839618+0000) 2019-10-13T13:04:26.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:26.519+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.521284+0000) 2019-10-13T13:04:26.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:26.611+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.614431+0000) 2019-10-13T13:04:26.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:26.611+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.614518+0000) 2019-10-13T13:04:26.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:26.611+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.615078+0000) 2019-10-13T13:04:26.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:26.611+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.615201+0000) 2019-10-13T13:04:26.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:26.615+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.615911+0000) 2019-10-13T13:04:26.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:26.615+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.616223+0000) 2019-10-13T13:04:26.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:26.615+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.616775+0000) 2019-10-13T13:04:26.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:26.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.623049+0000) 2019-10-13T13:04:26.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:26.619+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163112 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:28.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:28.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.788119+0000) 2019-10-13T13:04:28.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:28.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.790413+0000) 2019-10-13T13:04:30.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:30.015+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:30.021428+0000) 2019-10-13T13:04:31.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:31.623+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.626211+0000) 2019-10-13T13:04:31.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:31.623+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.626746+0000) 2019-10-13T13:04:31.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:31.627+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.627508+0000) 2019-10-13T13:04:31.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:31.627+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.627998+0000) 2019-10-13T13:04:31.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:31.627+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.628104+0000) 2019-10-13T13:04:31.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:31.627+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.628901+0000) 2019-10-13T13:04:31.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:31.623+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.629518+0000) 2019-10-13T13:04:31.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:31.631+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:31.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:31.631+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.635994+0000) 2019-10-13T13:04:31.992 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:31.987+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.990730+0000) 2019-10-13T13:04:32.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:32.279+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:32.279756+0000) 2019-10-13T13:04:35.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:35.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:35.553766+0000) 2019-10-13T13:04:35.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:35.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:35.839857+0000) 2019-10-13T13:04:36.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:36.518+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.521499+0000) 2019-10-13T13:04:36.640 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:36.634+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.638311+0000) 2019-10-13T13:04:36.641 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:36.635+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.638908+0000) 2019-10-13T13:04:36.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:36.634+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.639194+0000) 2019-10-13T13:04:36.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:36.638+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.639882+0000) 2019-10-13T13:04:36.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:36.638+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.640285+0000) 2019-10-13T13:04:36.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:36.638+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.641082+0000) 2019-10-13T13:04:36.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:36.638+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.641773+0000) 2019-10-13T13:04:36.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:36.642+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:36.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:36.647+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.650217+0000) 2019-10-13T13:04:37.316 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:04:37.317 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:37.321 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:04:37.322 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:38.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:38.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.788362+0000) 2019-10-13T13:04:38.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:38.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.790599+0000) 2019-10-13T13:04:40.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:40.015+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:40.021642+0000) 2019-10-13T13:04:41.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:41.646+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.650154+0000) 2019-10-13T13:04:41.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:41.650+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.651496+0000) 2019-10-13T13:04:41.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:41.650+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.651737+0000) 2019-10-13T13:04:41.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:41.650+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.652509+0000) 2019-10-13T13:04:41.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:41.647+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.653223+0000) 2019-10-13T13:04:41.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:41.650+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.653391+0000) 2019-10-13T13:04:41.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:41.650+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.654346+0000) 2019-10-13T13:04:41.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:41.651+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.657182+0000) 2019-10-13T13:04:41.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:41.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163211 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:41.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:41.995+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.997027+0000) 2019-10-13T13:04:42.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:42.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:42.279955+0000) 2019-10-13T13:04:45.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:45.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:45.553986+0000) 2019-10-13T13:04:45.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:45.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:45.840083+0000) 2019-10-13T13:04:46.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:46.518+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.521716+0000) 2019-10-13T13:04:46.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:46.658+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.661463+0000) 2019-10-13T13:04:46.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:46.658+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.662427+0000) 2019-10-13T13:04:46.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:46.662+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.663413+0000) 2019-10-13T13:04:46.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:46.662+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.664050+0000) 2019-10-13T13:04:46.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:46.662+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.664703+0000) 2019-10-13T13:04:46.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:46.662+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.665143+0000) 2019-10-13T13:04:46.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:46.663+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.666377+0000) 2019-10-13T13:04:46.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:46.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:46.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:46.667+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.672052+0000) 2019-10-13T13:04:48.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:48.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.788565+0000) 2019-10-13T13:04:48.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:48.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.790815+0000) 2019-10-13T13:04:50.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:50.015+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:50.021840+0000) 2019-10-13T13:04:51.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:51.670+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.673931+0000) 2019-10-13T13:04:51.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:51.670+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.674289+0000) 2019-10-13T13:04:51.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:51.670+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.675163+0000) 2019-10-13T13:04:51.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:51.674+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.675377+0000) 2019-10-13T13:04:51.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:51.674+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.675807+0000) 2019-10-13T13:04:51.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:51.674+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.676662+0000) 2019-10-13T13:04:51.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:51.671+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.677395+0000) 2019-10-13T13:04:51.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:51.679+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.682936+0000) 2019-10-13T13:04:51.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:51.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:52.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:52.003+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:52.003382+0000) 2019-10-13T13:04:52.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:52.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:52.280168+0000) 2019-10-13T13:04:55.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:55.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:55.554187+0000) 2019-10-13T13:04:55.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:55.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:55.840299+0000) 2019-10-13T13:04:56.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:56.518+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.521913+0000) 2019-10-13T13:04:56.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:56.682+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.686787+0000) 2019-10-13T13:04:56.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:04:56.682+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.687219+0000) 2019-10-13T13:04:56.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:04:56.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.688492+0000) 2019-10-13T13:04:56.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:04:56.686+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.688690+0000) 2019-10-13T13:04:56.691 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:04:56.686+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.689381+0000) 2019-10-13T13:04:56.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:56.686+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.689636+0000) 2019-10-13T13:04:56.692 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:04:56.686+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.690415+0000) 2019-10-13T13:04:56.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:04:56.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:04:56.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:04:56.695+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.698579+0000) 2019-10-13T13:04:58.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:04:58.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.788790+0000) 2019-10-13T13:04:58.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:04:58.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.791011+0000) 2019-10-13T13:05:00.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:00.014+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:00.022043+0000) 2019-10-13T13:05:01.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:01.694+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.697934+0000) 2019-10-13T13:05:01.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:01.698+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.699910+0000) 2019-10-13T13:05:01.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:01.698+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.700487+0000) 2019-10-13T13:05:01.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:01.694+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.701089+0000) 2019-10-13T13:05:01.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:01.698+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.701497+0000) 2019-10-13T13:05:01.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:01.698+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.702002+0000) 2019-10-13T13:05:01.704 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:01.698+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.702898+0000) 2019-10-13T13:05:01.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:01.702+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.707867+0000) 2019-10-13T13:05:01.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:01.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:02.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:02.006+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:02.009535+0000) 2019-10-13T13:05:02.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:02.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:02.280341+0000) 2019-10-13T13:05:05.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:05.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:05.554384+0000) 2019-10-13T13:05:05.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:05.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:05.840459+0000) 2019-10-13T13:05:06.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:06.518+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.522107+0000) 2019-10-13T13:05:06.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:06.706+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.711227+0000) 2019-10-13T13:05:06.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:06.710+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.711554+0000) 2019-10-13T13:05:06.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:06.710+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.712072+0000) 2019-10-13T13:05:06.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:06.710+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.712383+0000) 2019-10-13T13:05:06.715 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:06.710+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.712906+0000) 2019-10-13T13:05:06.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:06.706+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.714092+0000) 2019-10-13T13:05:06.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:06.710+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.714253+0000) 2019-10-13T13:05:06.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:06.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.717531+0000) 2019-10-13T13:05:06.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:06.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:07.441 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:05:07.441 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:07.445 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:05:07.445 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:08.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:08.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.788992+0000) 2019-10-13T13:05:08.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:08.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.791200+0000) 2019-10-13T13:05:10.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:10.018+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:10.022238+0000) 2019-10-13T13:05:11.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:11.714+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.721603+0000) 2019-10-13T13:05:11.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:11.718+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.721846+0000) 2019-10-13T13:05:11.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:11.718+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.722556+0000) 2019-10-13T13:05:11.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:11.718+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.722865+0000) 2019-10-13T13:05:11.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:11.722+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.723622+0000) 2019-10-13T13:05:11.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:11.722+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.724185+0000) 2019-10-13T13:05:11.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:11.722+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.724486+0000) 2019-10-13T13:05:11.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:11.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:11.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:11.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.731850+0000) 2019-10-13T13:05:12.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:12.010+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:12.015986+0000) 2019-10-13T13:05:12.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:12.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:12.280530+0000) 2019-10-13T13:05:15.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:15.550+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:15.554614+0000) 2019-10-13T13:05:15.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:15.838+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:15.840656+0000) 2019-10-13T13:05:16.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:16.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.522294+0000) 2019-10-13T13:05:16.735 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:16.729+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.732908+0000) 2019-10-13T13:05:16.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:16.729+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.733971+0000) 2019-10-13T13:05:16.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:16.729+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.733861+0000) 2019-10-13T13:05:16.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:16.730+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.734567+0000) 2019-10-13T13:05:16.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:16.729+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.734730+0000) 2019-10-13T13:05:16.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:16.729+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.735108+0000) 2019-10-13T13:05:16.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:16.733+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.735667+0000) 2019-10-13T13:05:16.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:16.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:16.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:16.738+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.743162+0000) 2019-10-13T13:05:18.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:18.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.789197+0000) 2019-10-13T13:05:18.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:18.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.791398+0000) 2019-10-13T13:05:20.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:20.018+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:20.022466+0000) 2019-10-13T13:05:21.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:21.741+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.745623+0000) 2019-10-13T13:05:21.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:21.741+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.746432+0000) 2019-10-13T13:05:21.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:21.741+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.746732+0000) 2019-10-13T13:05:21.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:21.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.747248+0000) 2019-10-13T13:05:21.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:21.742+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.747738+0000) 2019-10-13T13:05:21.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:21.745+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.748307+0000) 2019-10-13T13:05:21.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:21.745+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.749464+0000) 2019-10-13T13:05:21.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:21.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:21.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:21.750+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.758092+0000) 2019-10-13T13:05:22.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:22.022+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:22.023034+0000) 2019-10-13T13:05:22.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:22.277+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:22.280743+0000) 2019-10-13T13:05:25.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:25.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:25.554823+0000) 2019-10-13T13:05:25.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:25.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:25.840857+0000) 2019-10-13T13:05:26.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:26.517+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.522485+0000) 2019-10-13T13:05:26.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:26.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.756407+0000) 2019-10-13T13:05:26.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:26.750+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.756992+0000) 2019-10-13T13:05:26.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:26.753+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.757414+0000) 2019-10-13T13:05:26.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:26.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.758315+0000) 2019-10-13T13:05:26.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:26.753+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.758533+0000) 2019-10-13T13:05:26.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:26.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.759010+0000) 2019-10-13T13:05:26.761 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:26.757+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.759374+0000) 2019-10-13T13:05:26.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:26.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:26.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:26.762+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.766697+0000) 2019-10-13T13:05:28.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:28.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.789413+0000) 2019-10-13T13:05:28.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:28.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.791564+0000) 2019-10-13T13:05:30.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:30.018+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:30.022703+0000) 2019-10-13T13:05:31.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:31.765+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.768521+0000) 2019-10-13T13:05:31.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:31.765+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.768874+0000) 2019-10-13T13:05:31.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:31.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.769198+0000) 2019-10-13T13:05:31.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:31.765+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.769737+0000) 2019-10-13T13:05:31.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:31.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.770699+0000) 2019-10-13T13:05:31.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:31.765+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.770704+0000) 2019-10-13T13:05:31.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:31.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.771244+0000) 2019-10-13T13:05:31.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:31.770+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.775870+0000) 2019-10-13T13:05:31.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:31.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:32.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:32.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:32.029911+0000) 2019-10-13T13:05:32.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:32.277+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:32.280926+0000) 2019-10-13T13:05:35.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:35.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:35.555078+0000) 2019-10-13T13:05:35.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:35.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:35.841054+0000) 2019-10-13T13:05:36.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:36.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.522698+0000) 2019-10-13T13:05:36.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:36.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.780968+0000) 2019-10-13T13:05:36.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:36.777+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.781135+0000) 2019-10-13T13:05:36.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:36.777+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.781402+0000) 2019-10-13T13:05:36.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:36.777+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.781869+0000) 2019-10-13T13:05:36.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:36.777+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.782139+0000) 2019-10-13T13:05:36.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:36.777+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.782581+0000) 2019-10-13T13:05:36.785 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:36.778+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.783731+0000) 2019-10-13T13:05:36.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:36.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:36.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:36.782+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.789887+0000) 2019-10-13T13:05:37.571 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:05:37.571 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:37.575 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:05:37.575 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:38.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:38.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.789636+0000) 2019-10-13T13:05:38.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:38.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.791791+0000) 2019-10-13T13:05:40.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:40.018+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:40.022921+0000) 2019-10-13T13:05:41.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:41.789+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.792354+0000) 2019-10-13T13:05:41.795 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:41.786+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.793591+0000) 2019-10-13T13:05:41.795 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:41.789+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.792779+0000) 2019-10-13T13:05:41.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:41.789+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.794457+0000) 2019-10-13T13:05:41.796 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:41.789+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.794377+0000) 2019-10-13T13:05:41.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:41.789+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.794865+0000) 2019-10-13T13:05:41.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:41.789+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.795265+0000) 2019-10-13T13:05:41.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:41.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:41.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:41.798+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.802359+0000) 2019-10-13T13:05:42.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:42.034+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:42.036806+0000) 2019-10-13T13:05:42.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:42.277+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:42.281104+0000) 2019-10-13T13:05:45.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:45.549+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:45.555280+0000) 2019-10-13T13:05:45.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:45.837+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:45.841253+0000) 2019-10-13T13:05:46.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:46.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.522892+0000) 2019-10-13T13:05:46.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:46.801+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.803818+0000) 2019-10-13T13:05:46.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:46.801+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.805069+0000) 2019-10-13T13:05:46.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:46.806+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.806267+0000) 2019-10-13T13:05:46.808 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:46.801+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.806447+0000) 2019-10-13T13:05:46.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:46.801+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.806778+0000) 2019-10-13T13:05:46.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:46.801+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.807235+0000) 2019-10-13T13:05:46.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:46.805+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.807960+0000) 2019-10-13T13:05:46.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:46.810+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.812460+0000) 2019-10-13T13:05:46.814 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:46.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:48.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:48.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.789835+0000) 2019-10-13T13:05:48.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:48.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.792018+0000) 2019-10-13T13:05:50.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:50.022+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:50.023110+0000) 2019-10-13T13:05:51.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:51.813+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.815381+0000) 2019-10-13T13:05:51.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:51.813+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.815989+0000) 2019-10-13T13:05:51.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:51.813+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.816805+0000) 2019-10-13T13:05:51.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:51.818+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.818745+0000) 2019-10-13T13:05:51.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:51.813+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.818802+0000) 2019-10-13T13:05:51.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:51.813+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.819225+0000) 2019-10-13T13:05:51.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:51.817+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.819683+0000) 2019-10-13T13:05:51.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:51.821+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:51.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:51.822+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.824203+0000) 2019-10-13T13:05:52.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:52.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:52.043221+0000) 2019-10-13T13:05:52.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:52.281+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:52.281285+0000) 2019-10-13T13:05:55.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:55.553+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:55.555468+0000) 2019-10-13T13:05:55.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:55.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:55.841456+0000) 2019-10-13T13:05:56.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:56.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.523119+0000) 2019-10-13T13:05:56.829 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:05:56.825+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.827434+0000) 2019-10-13T13:05:56.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:05:56.825+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.828201+0000) 2019-10-13T13:05:56.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:56.825+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.828676+0000) 2019-10-13T13:05:56.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:05:56.825+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.829295+0000) 2019-10-13T13:05:56.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:56.825+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.829634+0000) 2019-10-13T13:05:56.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:05:56.829+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.830856+0000) 2019-10-13T13:05:56.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:05:56.825+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.830982+0000) 2019-10-13T13:05:56.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:05:56.833+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.835108+0000) 2019-10-13T13:05:56.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:05:56.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163706 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:05:58.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:05:58.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.790037+0000) 2019-10-13T13:05:58.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:05:58.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.792213+0000) 2019-10-13T13:06:00.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:00.021+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:00.023311+0000) 2019-10-13T13:06:01.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:01.832+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.838250+0000) 2019-10-13T13:06:01.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:01.832+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.838645+0000) 2019-10-13T13:06:01.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:01.836+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.840011+0000) 2019-10-13T13:06:01.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:01.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.840327+0000) 2019-10-13T13:06:01.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:01.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.840992+0000) 2019-10-13T13:06:01.843 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:01.836+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.841671+0000) 2019-10-13T13:06:01.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:01.841+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.842283+0000) 2019-10-13T13:06:01.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:01.841+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.846010+0000) 2019-10-13T13:06:01.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:01.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:02.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:02.045+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:02.049569+0000) 2019-10-13T13:06:02.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:02.280+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:02.281423+0000) 2019-10-13T13:06:05.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:05.552+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:05.555618+0000) 2019-10-13T13:06:05.843 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:05.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:05.841660+0000) 2019-10-13T13:06:06.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:06.520+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.523343+0000) 2019-10-13T13:06:06.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:06.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.850582+0000) 2019-10-13T13:06:06.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:06.848+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.851401+0000) 2019-10-13T13:06:06.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:06.848+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.851783+0000) 2019-10-13T13:06:06.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:06.848+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.852076+0000) 2019-10-13T13:06:06.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:06.848+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.852774+0000) 2019-10-13T13:06:06.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:06.848+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.853194+0000) 2019-10-13T13:06:06.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:06.853+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.854593+0000) 2019-10-13T13:06:06.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:06.853+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.857643+0000) 2019-10-13T13:06:06.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:06.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:07.673 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:06:07.673 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:07.677 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:06:07.677 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:08.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:08.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.790231+0000) 2019-10-13T13:06:08.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:08.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.792407+0000) 2019-10-13T13:06:10.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:10.021+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:10.023516+0000) 2019-10-13T13:06:11.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:11.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.861769+0000) 2019-10-13T13:06:11.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:11.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.862376+0000) 2019-10-13T13:06:11.865 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:11.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.863186+0000) 2019-10-13T13:06:11.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:11.860+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.863685+0000) 2019-10-13T13:06:11.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:11.860+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.864878+0000) 2019-10-13T13:06:11.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:11.860+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.865159+0000) 2019-10-13T13:06:11.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:11.861+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.865718+0000) 2019-10-13T13:06:11.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:11.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:11.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:11.869+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.873433+0000) 2019-10-13T13:06:12.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:12.053+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:12.055335+0000) 2019-10-13T13:06:12.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:12.280+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:12.281607+0000) 2019-10-13T13:06:15.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:15.552+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:15.555833+0000) 2019-10-13T13:06:15.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:15.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:15.841854+0000) 2019-10-13T13:06:16.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:16.520+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.523577+0000) 2019-10-13T13:06:16.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:16.868+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.873999+0000) 2019-10-13T13:06:16.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:16.873+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.874728+0000) 2019-10-13T13:06:16.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:16.868+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.874985+0000) 2019-10-13T13:06:16.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:16.872+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.876485+0000) 2019-10-13T13:06:16.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:16.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.876903+0000) 2019-10-13T13:06:16.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:16.872+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.877288+0000) 2019-10-13T13:06:16.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:16.872+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.877682+0000) 2019-10-13T13:06:16.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:16.877+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.882090+0000) 2019-10-13T13:06:16.885 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:16.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:18.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:18.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.790430+0000) 2019-10-13T13:06:18.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:18.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.792636+0000) 2019-10-13T13:06:20.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:20.021+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:20.023722+0000) 2019-10-13T13:06:21.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:21.880+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.886122+0000) 2019-10-13T13:06:21.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:21.880+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.886562+0000) 2019-10-13T13:06:21.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:21.880+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.887220+0000) 2019-10-13T13:06:21.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:21.884+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.888243+0000) 2019-10-13T13:06:21.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:21.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.888757+0000) 2019-10-13T13:06:21.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:21.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.888989+0000) 2019-10-13T13:06:21.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:21.884+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.889850+0000) 2019-10-13T13:06:21.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:21.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:21.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:21.897+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.899546+0000) 2019-10-13T13:06:22.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:22.065+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:22.066768+0000) 2019-10-13T13:06:22.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:22.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:22.281844+0000) 2019-10-13T13:06:25.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:25.552+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:25.555985+0000) 2019-10-13T13:06:25.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:25.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:25.842056+0000) 2019-10-13T13:06:26.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:26.520+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.523769+0000) 2019-10-13T13:06:26.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:26.892+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.897707+0000) 2019-10-13T13:06:26.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:26.892+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.898511+0000) 2019-10-13T13:06:26.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:26.892+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.898876+0000) 2019-10-13T13:06:26.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:26.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.899526+0000) 2019-10-13T13:06:26.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:26.896+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.899613+0000) 2019-10-13T13:06:26.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:26.896+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.899904+0000) 2019-10-13T13:06:26.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:26.896+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.900494+0000) 2019-10-13T13:06:26.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:26.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:26.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:26.905+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.908059+0000) 2019-10-13T13:06:28.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:28.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.790660+0000) 2019-10-13T13:06:28.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:28.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.792859+0000) 2019-10-13T13:06:30.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:30.021+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:30.023946+0000) 2019-10-13T13:06:31.910 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:31.904+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.908421+0000) 2019-10-13T13:06:31.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:31.905+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.909403+0000) 2019-10-13T13:06:31.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:31.904+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.909567+0000) 2019-10-13T13:06:31.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:31.904+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.910410+0000) 2019-10-13T13:06:31.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:31.904+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.910588+0000) 2019-10-13T13:06:31.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:31.908+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.911631+0000) 2019-10-13T13:06:31.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:31.908+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.912592+0000) 2019-10-13T13:06:31.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:31.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:31.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:31.913+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.917766+0000) 2019-10-13T13:06:32.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:32.069+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:32.073271+0000) 2019-10-13T13:06:32.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:32.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:32.282017+0000) 2019-10-13T13:06:35.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:35.552+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:35.556181+0000) 2019-10-13T13:06:35.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:35.836+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:35.842254+0000) 2019-10-13T13:06:36.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:36.520+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.523971+0000) 2019-10-13T13:06:36.922 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:36.916+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.920280+0000) 2019-10-13T13:06:36.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:36.916+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.920679+0000) 2019-10-13T13:06:36.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:36.916+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.921547+0000) 2019-10-13T13:06:36.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:36.916+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.922243+0000) 2019-10-13T13:06:36.925 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:36.916+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.922268+0000) 2019-10-13T13:06:36.925 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:36.916+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.922824+0000) 2019-10-13T13:06:36.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:36.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.923709+0000) 2019-10-13T13:06:36.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:36.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 163974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:36.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:36.925+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.929997+0000) 2019-10-13T13:06:37.767 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:06:37.768 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:37.771 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:06:37.771 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:38.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:38.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.790879+0000) 2019-10-13T13:06:38.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:38.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.793050+0000) 2019-10-13T13:06:40.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:40.021+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:40.024139+0000) 2019-10-13T13:06:41.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:41.929+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.932465+0000) 2019-10-13T13:06:41.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:41.928+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.932928+0000) 2019-10-13T13:06:41.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:41.928+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.933599+0000) 2019-10-13T13:06:41.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:41.928+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.933854+0000) 2019-10-13T13:06:41.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:41.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.934315+0000) 2019-10-13T13:06:41.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:41.928+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.934829+0000) 2019-10-13T13:06:41.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:41.932+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.935804+0000) 2019-10-13T13:06:41.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:41.937+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.939186+0000) 2019-10-13T13:06:41.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:41.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:42.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:42.077+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:42.079312+0000) 2019-10-13T13:06:42.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:42.276+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:42.282196+0000) 2019-10-13T13:06:45.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:45.551+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:45.556365+0000) 2019-10-13T13:06:45.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:45.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:45.842465+0000) 2019-10-13T13:06:46.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:46.523+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.524179+0000) 2019-10-13T13:06:46.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:46.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.943591+0000) 2019-10-13T13:06:46.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:46.939+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.944071+0000) 2019-10-13T13:06:46.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:46.940+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.945006+0000) 2019-10-13T13:06:46.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:46.939+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.945292+0000) 2019-10-13T13:06:46.948 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:46.939+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.946175+0000) 2019-10-13T13:06:46.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:46.939+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.947132+0000) 2019-10-13T13:06:46.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:46.943+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.947645+0000) 2019-10-13T13:06:46.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:46.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:46.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:46.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.953356+0000) 2019-10-13T13:06:48.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:48.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.791080+0000) 2019-10-13T13:06:48.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:48.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.793262+0000) 2019-10-13T13:06:50.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:50.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:50.024332+0000) 2019-10-13T13:06:51.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:51.951+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.954976+0000) 2019-10-13T13:06:51.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:51.951+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.955727+0000) 2019-10-13T13:06:51.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:51.951+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.956393+0000) 2019-10-13T13:06:51.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:51.951+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.957120+0000) 2019-10-13T13:06:51.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:51.951+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.957556+0000) 2019-10-13T13:06:51.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:51.952+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.958025+0000) 2019-10-13T13:06:51.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:51.951+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.958682+0000) 2019-10-13T13:06:51.965 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:51.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:51.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:51.964+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.966440+0000) 2019-10-13T13:06:52.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:52.084+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:52.085772+0000) 2019-10-13T13:06:52.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:52.279+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:52.282328+0000) 2019-10-13T13:06:55.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:55.551+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:55.556565+0000) 2019-10-13T13:06:55.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:55.835+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:55.842664+0000) 2019-10-13T13:06:56.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:56.519+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.524417+0000) 2019-10-13T13:06:56.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:06:56.959+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.966917+0000) 2019-10-13T13:06:56.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:06:56.963+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.967356+0000) 2019-10-13T13:06:56.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:56.963+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.967620+0000) 2019-10-13T13:06:56.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:06:56.963+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.968380+0000) 2019-10-13T13:06:56.970 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:56.963+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.969027+0000) 2019-10-13T13:06:56.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:06:56.964+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.969666+0000) 2019-10-13T13:06:56.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:06:56.963+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.970220+0000) 2019-10-13T13:06:56.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:06:56.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164108 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:06:56.978 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:06:56.972+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.977018+0000) 2019-10-13T13:06:58.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:06:58.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.791291+0000) 2019-10-13T13:06:58.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:06:58.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.793458+0000) 2019-10-13T13:07:00.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:00.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:00.024534+0000) 2019-10-13T13:07:01.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:01.975+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.979373+0000) 2019-10-13T13:07:01.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:01.975+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.979656+0000) 2019-10-13T13:07:01.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:01.976+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.980256+0000) 2019-10-13T13:07:01.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:01.975+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.980393+0000) 2019-10-13T13:07:01.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:01.975+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.980935+0000) 2019-10-13T13:07:01.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:01.975+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.981056+0000) 2019-10-13T13:07:01.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:01.975+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.982007+0000) 2019-10-13T13:07:01.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:01.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:01.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:01.984+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.988120+0000) 2019-10-13T13:07:02.093 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:02.088+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:02.091679+0000) 2019-10-13T13:07:02.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:02.275+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:02.282507+0000) 2019-10-13T13:07:05.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:05.555+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:05.556765+0000) 2019-10-13T13:07:05.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:05.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:05.842901+0000) 2019-10-13T13:07:06.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:06.523+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.524607+0000) 2019-10-13T13:07:06.993 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:06.987+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.990912+0000) 2019-10-13T13:07:06.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:06.987+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.991443+0000) 2019-10-13T13:07:06.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:06.987+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.991720+0000) 2019-10-13T13:07:06.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:06.987+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.992655+0000) 2019-10-13T13:07:06.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:06.987+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.992969+0000) 2019-10-13T13:07:06.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:06.988+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.993896+0000) 2019-10-13T13:07:06.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:06.987+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.993723+0000) 2019-10-13T13:07:07.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:06.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:07.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:06.996+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.999743+0000) 2019-10-13T13:07:07.870 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:07:07.871 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:07.874 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:07:07.874 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:08.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:08.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.791529+0000) 2019-10-13T13:07:08.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:08.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.793702+0000) 2019-10-13T13:07:10.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:10.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:10.024737+0000) 2019-10-13T13:07:12.004 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:12.000+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.002600+0000) 2019-10-13T13:07:12.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:11.995+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.002691+0000) 2019-10-13T13:07:12.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:11.999+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.003719+0000) 2019-10-13T13:07:12.006 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:11.999+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.004394+0000) 2019-10-13T13:07:12.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:11.999+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.003926+0000) 2019-10-13T13:07:12.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:11.999+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.004673+0000) 2019-10-13T13:07:12.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:11.999+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.005089+0000) 2019-10-13T13:07:12.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:12.004+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.009737+0000) 2019-10-13T13:07:12.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:12.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:12.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:12.092+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.095681+0000) 2019-10-13T13:07:12.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:12.279+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:12.282695+0000) 2019-10-13T13:07:15.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:15.555+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:15.557003+0000) 2019-10-13T13:07:15.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:15.839+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:15.843118+0000) 2019-10-13T13:07:16.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:16.523+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:16.524799+0000) 2019-10-13T13:07:17.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:17.011+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.012659+0000) 2019-10-13T13:07:17.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:17.011+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.013369+0000) 2019-10-13T13:07:17.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:17.012+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.014543+0000) 2019-10-13T13:07:17.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:17.011+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.014725+0000) 2019-10-13T13:07:17.017 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:17.011+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.015199+0000) 2019-10-13T13:07:17.017 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:17.015+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.015462+0000) 2019-10-13T13:07:17.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:17.015+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.015950+0000) 2019-10-13T13:07:17.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:17.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:17.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:17.020+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.023390+0000) 2019-10-13T13:07:18.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:18.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.791759+0000) 2019-10-13T13:07:18.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:18.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.793933+0000) 2019-10-13T13:07:20.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:20.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:20.024955+0000) 2019-10-13T13:07:22.027 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:22.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.025725+0000) 2019-10-13T13:07:22.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:22.023+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.026577+0000) 2019-10-13T13:07:22.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:22.023+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.026582+0000) 2019-10-13T13:07:22.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:22.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.027644+0000) 2019-10-13T13:07:22.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:22.027+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.027916+0000) 2019-10-13T13:07:22.030 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:22.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.027976+0000) 2019-10-13T13:07:22.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:22.027+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.028537+0000) 2019-10-13T13:07:22.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:22.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:22.036 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:22.032+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.035070+0000) 2019-10-13T13:07:22.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:22.104+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.105785+0000) 2019-10-13T13:07:22.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:22.279+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:22.282902+0000) 2019-10-13T13:07:25.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:25.554+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:25.557199+0000) 2019-10-13T13:07:25.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:25.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:25.843368+0000) 2019-10-13T13:07:26.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:26.522+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:26.525012+0000) 2019-10-13T13:07:27.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:27.034+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.038004+0000) 2019-10-13T13:07:27.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:27.034+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.038395+0000) 2019-10-13T13:07:27.041 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:27.034+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.038729+0000) 2019-10-13T13:07:27.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:27.038+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.039344+0000) 2019-10-13T13:07:27.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:27.038+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.039286+0000) 2019-10-13T13:07:27.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:27.038+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.040039+0000) 2019-10-13T13:07:27.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:27.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.041395+0000) 2019-10-13T13:07:27.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:27.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:27.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:27.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.049471+0000) 2019-10-13T13:07:28.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:28.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.791987+0000) 2019-10-13T13:07:28.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:28.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.794154+0000) 2019-10-13T13:07:30.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:30.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:30.025176+0000) 2019-10-13T13:07:32.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:32.046+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.049666+0000) 2019-10-13T13:07:32.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:32.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.050693+0000) 2019-10-13T13:07:32.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:32.046+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.050929+0000) 2019-10-13T13:07:32.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:32.050+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.051253+0000) 2019-10-13T13:07:32.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:32.050+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.051904+0000) 2019-10-13T13:07:32.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:32.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.052725+0000) 2019-10-13T13:07:32.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:32.050+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.053050+0000) 2019-10-13T13:07:32.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:32.052+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.056490+0000) 2019-10-13T13:07:32.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:32.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:32.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:32.112+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.112445+0000) 2019-10-13T13:07:32.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:32.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:32.283097+0000) 2019-10-13T13:07:35.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:35.554+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:35.557401+0000) 2019-10-13T13:07:35.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:35.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:35.843574+0000) 2019-10-13T13:07:36.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:36.522+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:36.525207+0000) 2019-10-13T13:07:37.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:37.058+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.061744+0000) 2019-10-13T13:07:37.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:37.058+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.062457+0000) 2019-10-13T13:07:37.065 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:37.058+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.062696+0000) 2019-10-13T13:07:37.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:37.060+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.063396+0000) 2019-10-13T13:07:37.065 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:37.062+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.063505+0000) 2019-10-13T13:07:37.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:37.062+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.064235+0000) 2019-10-13T13:07:37.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:37.062+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.063956+0000) 2019-10-13T13:07:37.071 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:37.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164376 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:37.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:37.068+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.071935+0000) 2019-10-13T13:07:37.938 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:07:37.938 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:37.942 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:07:37.942 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:38.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:38.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.792203+0000) 2019-10-13T13:07:38.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:38.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.794354+0000) 2019-10-13T13:07:40.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:40.020+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:40.025400+0000) 2019-10-13T13:07:42.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:42.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.072981+0000) 2019-10-13T13:07:42.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:42.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.073605+0000) 2019-10-13T13:07:42.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:42.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.074173+0000) 2019-10-13T13:07:42.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:42.070+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.074819+0000) 2019-10-13T13:07:42.077 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:42.074+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.075367+0000) 2019-10-13T13:07:42.077 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:42.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.075952+0000) 2019-10-13T13:07:42.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:42.074+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.076121+0000) 2019-10-13T13:07:42.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:42.075+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.081171+0000) 2019-10-13T13:07:42.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:42.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:42.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:42.111+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.115939+0000) 2019-10-13T13:07:42.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:42.278+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:42.283235+0000) 2019-10-13T13:07:45.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:45.554+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:45.557592+0000) 2019-10-13T13:07:45.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:45.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:45.843807+0000) 2019-10-13T13:07:46.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:46.522+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:46.525431+0000) 2019-10-13T13:07:47.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:47.079+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.084649+0000) 2019-10-13T13:07:47.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:47.082+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.085028+0000) 2019-10-13T13:07:47.088 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:47.082+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.085544+0000) 2019-10-13T13:07:47.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:47.082+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.086270+0000) 2019-10-13T13:07:47.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:47.082+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.086705+0000) 2019-10-13T13:07:47.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:47.086+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.087362+0000) 2019-10-13T13:07:47.090 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:47.086+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.087698+0000) 2019-10-13T13:07:47.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:47.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:47.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:47.087+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.093682+0000) 2019-10-13T13:07:48.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:48.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.792407+0000) 2019-10-13T13:07:48.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:48.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.794566+0000) 2019-10-13T13:07:50.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:50.019+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:50.025592+0000) 2019-10-13T13:07:52.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:52.094+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.096576+0000) 2019-10-13T13:07:52.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:52.091+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.097163+0000) 2019-10-13T13:07:52.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:52.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.097396+0000) 2019-10-13T13:07:52.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:52.094+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.097798+0000) 2019-10-13T13:07:52.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:52.094+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.098059+0000) 2019-10-13T13:07:52.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:52.098+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.099867+0000) 2019-10-13T13:07:52.102 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:52.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.100672+0000) 2019-10-13T13:07:52.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:52.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:52.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:52.099+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.105818+0000) 2019-10-13T13:07:52.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:52.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.122308+0000) 2019-10-13T13:07:52.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:52.282+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:52.283427+0000) 2019-10-13T13:07:55.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:55.554+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:55.557788+0000) 2019-10-13T13:07:55.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:55.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:55.844007+0000) 2019-10-13T13:07:56.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:56.522+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:56.525621+0000) 2019-10-13T13:07:57.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:07:57.106+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.108971+0000) 2019-10-13T13:07:57.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:07:57.107+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.110128+0000) 2019-10-13T13:07:57.112 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:57.106+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.110477+0000) 2019-10-13T13:07:57.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:57.106+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.110756+0000) 2019-10-13T13:07:57.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:07:57.106+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.111026+0000) 2019-10-13T13:07:57.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:07:57.110+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.111687+0000) 2019-10-13T13:07:57.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:07:57.110+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.111597+0000) 2019-10-13T13:07:57.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:07:57.111+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.116359+0000) 2019-10-13T13:07:57.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:07:57.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:07:58.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:07:58.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.792637+0000) 2019-10-13T13:07:58.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:07:58.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.794758+0000) 2019-10-13T13:08:00.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:00.019+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:00.025828+0000) 2019-10-13T13:08:02.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:02.114+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.119115+0000) 2019-10-13T13:08:02.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:02.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.120341+0000) 2019-10-13T13:08:02.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:02.118+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.119620+0000) 2019-10-13T13:08:02.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:02.118+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.120884+0000) 2019-10-13T13:08:02.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:02.118+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.122300+0000) 2019-10-13T13:08:02.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:02.118+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.122743+0000) 2019-10-13T13:08:02.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:02.118+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.123197+0000) 2019-10-13T13:08:02.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:02.123+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.127846+0000) 2019-10-13T13:08:02.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:02.123+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.127894+0000) 2019-10-13T13:08:02.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:02.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:02.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:02.282+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:02.283596+0000) 2019-10-13T13:08:05.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:05.554+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:05.557990+0000) 2019-10-13T13:08:05.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:05.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:05.844205+0000) 2019-10-13T13:08:06.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:06.522+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:06.525821+0000) 2019-10-13T13:08:07.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:07.130+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.132054+0000) 2019-10-13T13:08:07.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:07.130+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.132417+0000) 2019-10-13T13:08:07.135 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:07.130+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.133366+0000) 2019-10-13T13:08:07.136 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:07.130+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.134554+0000) 2019-10-13T13:08:07.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:07.131+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.135460+0000) 2019-10-13T13:08:07.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:07.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.136311+0000) 2019-10-13T13:08:07.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:07.134+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.136823+0000) 2019-10-13T13:08:07.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:07.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164576 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:07.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:07.139+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.142569+0000) 2019-10-13T13:08:08.051 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:08:08.051 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:08.055 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:08:08.055 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:08.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:08.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.792838+0000) 2019-10-13T13:08:08.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:08.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.794948+0000) 2019-10-13T13:08:10.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:10.019+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:10.026033+0000) 2019-10-13T13:08:12.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:12.123+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.128111+0000) 2019-10-13T13:08:12.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:12.141+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.143518+0000) 2019-10-13T13:08:12.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:12.141+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.144883+0000) 2019-10-13T13:08:12.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:12.141+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.145140+0000) 2019-10-13T13:08:12.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:12.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.146179+0000) 2019-10-13T13:08:12.148 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:12.141+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.145638+0000) 2019-10-13T13:08:12.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:12.141+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.146398+0000) 2019-10-13T13:08:12.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:12.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.146502+0000) 2019-10-13T13:08:12.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:12.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:12.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:12.147+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.153630+0000) 2019-10-13T13:08:12.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:12.281+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:12.283779+0000) 2019-10-13T13:08:15.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:15.557+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:15.558190+0000) 2019-10-13T13:08:15.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:15.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:15.844406+0000) 2019-10-13T13:08:16.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:16.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:16.526044+0000) 2019-10-13T13:08:17.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:17.153+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.156061+0000) 2019-10-13T13:08:17.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:17.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.156571+0000) 2019-10-13T13:08:17.159 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:17.151+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.157157+0000) 2019-10-13T13:08:17.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:17.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.157730+0000) 2019-10-13T13:08:17.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:17.153+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.158188+0000) 2019-10-13T13:08:17.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:17.157+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.159608+0000) 2019-10-13T13:08:17.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:17.157+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.160179+0000) 2019-10-13T13:08:17.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:17.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:17.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:17.159+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.164576+0000) 2019-10-13T13:08:18.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:18.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.793069+0000) 2019-10-13T13:08:18.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:18.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.795166+0000) 2019-10-13T13:08:20.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:20.023+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:20.026228+0000) 2019-10-13T13:08:22.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:22.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.134676+0000) 2019-10-13T13:08:22.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:22.165+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.167696+0000) 2019-10-13T13:08:22.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:22.165+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.168066+0000) 2019-10-13T13:08:22.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:22.165+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.168715+0000) 2019-10-13T13:08:22.171 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:22.165+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.168901+0000) 2019-10-13T13:08:22.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:22.163+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.169445+0000) 2019-10-13T13:08:22.172 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:22.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.169512+0000) 2019-10-13T13:08:22.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:22.165+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.170191+0000) 2019-10-13T13:08:22.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:22.167+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.172725+0000) 2019-10-13T13:08:22.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:22.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:22.287 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:22.281+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:22.283996+0000) 2019-10-13T13:08:25.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:25.553+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:25.558389+0000) 2019-10-13T13:08:25.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:25.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:25.844611+0000) 2019-10-13T13:08:26.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:26.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:26.526299+0000) 2019-10-13T13:08:27.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:27.173+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.178892+0000) 2019-10-13T13:08:27.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:27.173+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.179270+0000) 2019-10-13T13:08:27.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:27.177+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.179827+0000) 2019-10-13T13:08:27.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:27.177+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.180298+0000) 2019-10-13T13:08:27.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:27.175+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.180844+0000) 2019-10-13T13:08:27.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:27.177+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.181457+0000) 2019-10-13T13:08:27.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:27.177+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.181840+0000) 2019-10-13T13:08:27.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:27.183+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.186927+0000) 2019-10-13T13:08:27.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:27.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:28.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:28.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.793274+0000) 2019-10-13T13:08:28.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:28.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.795359+0000) 2019-10-13T13:08:30.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:30.023+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:30.026430+0000) 2019-10-13T13:08:32.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:32.135+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.141058+0000) 2019-10-13T13:08:32.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:32.187+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.190655+0000) 2019-10-13T13:08:32.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:32.185+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.190966+0000) 2019-10-13T13:08:32.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:32.189+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.191935+0000) 2019-10-13T13:08:32.194 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:32.189+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.192438+0000) 2019-10-13T13:08:32.195 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:32.189+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.192651+0000) 2019-10-13T13:08:32.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:32.189+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.193407+0000) 2019-10-13T13:08:32.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:32.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.193700+0000) 2019-10-13T13:08:32.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:32.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:32.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:32.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.203312+0000) 2019-10-13T13:08:32.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:32.281+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:32.284218+0000) 2019-10-13T13:08:35.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:35.553+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:35.558588+0000) 2019-10-13T13:08:35.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:35.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:35.844811+0000) 2019-10-13T13:08:36.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:36.525+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:36.526554+0000) 2019-10-13T13:08:37.205 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:37.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.203219+0000) 2019-10-13T13:08:37.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:37.201+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.203670+0000) 2019-10-13T13:08:37.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:37.198+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.204342+0000) 2019-10-13T13:08:37.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:37.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.204318+0000) 2019-10-13T13:08:37.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:37.201+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.204856+0000) 2019-10-13T13:08:37.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:37.201+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.205080+0000) 2019-10-13T13:08:37.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:37.201+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.205485+0000) 2019-10-13T13:08:37.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:37.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164776 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:37.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:37.206+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.211919+0000) 2019-10-13T13:08:38.170 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:08:38.170 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:38.174 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:08:38.174 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:38.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:38.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.793499+0000) 2019-10-13T13:08:38.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:38.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.795508+0000) 2019-10-13T13:08:40.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:40.022+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:40.026642+0000) 2019-10-13T13:08:42.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:42.146+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.147739+0000) 2019-10-13T13:08:42.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:42.209+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.213564+0000) 2019-10-13T13:08:42.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:42.209+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.214000+0000) 2019-10-13T13:08:42.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:42.210+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.215306+0000) 2019-10-13T13:08:42.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:42.209+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.214900+0000) 2019-10-13T13:08:42.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:42.213+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.215562+0000) 2019-10-13T13:08:42.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:42.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.216648+0000) 2019-10-13T13:08:42.218 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:42.213+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.216811+0000) 2019-10-13T13:08:42.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:42.217+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:42.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:42.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.222641+0000) 2019-10-13T13:08:42.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:42.281+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:42.284398+0000) 2019-10-13T13:08:45.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:45.557+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:45.558788+0000) 2019-10-13T13:08:45.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:45.841+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:45.845010+0000) 2019-10-13T13:08:46.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:46.521+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:46.526774+0000) 2019-10-13T13:08:47.228 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:47.221+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.225924+0000) 2019-10-13T13:08:47.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:47.221+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.226794+0000) 2019-10-13T13:08:47.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:47.221+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.227250+0000) 2019-10-13T13:08:47.229 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:47.225+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.227486+0000) 2019-10-13T13:08:47.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:47.225+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.227805+0000) 2019-10-13T13:08:47.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:47.225+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.228212+0000) 2019-10-13T13:08:47.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:47.222+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.229778+0000) 2019-10-13T13:08:47.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:47.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:47.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:47.230+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.235605+0000) 2019-10-13T13:08:48.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:48.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.793697+0000) 2019-10-13T13:08:48.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:48.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.795707+0000) 2019-10-13T13:08:50.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:50.022+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:50.026833+0000) 2019-10-13T13:08:52.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:52.150+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.154217+0000) 2019-10-13T13:08:52.239 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:52.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.237879+0000) 2019-10-13T13:08:52.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:52.232+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.238209+0000) 2019-10-13T13:08:52.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:52.232+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.238392+0000) 2019-10-13T13:08:52.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:52.232+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.238702+0000) 2019-10-13T13:08:52.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:52.232+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.238933+0000) 2019-10-13T13:08:52.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:52.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.239553+0000) 2019-10-13T13:08:52.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:52.236+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.240200+0000) 2019-10-13T13:08:52.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:52.238+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.243519+0000) 2019-10-13T13:08:52.248 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:52.240+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:52.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:52.280+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:52.284582+0000) 2019-10-13T13:08:55.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:55.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:55.558986+0000) 2019-10-13T13:08:55.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:55.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:55.845245+0000) 2019-10-13T13:08:56.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:56.520+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:56.526959+0000) 2019-10-13T13:08:57.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:57.244+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.250047+0000) 2019-10-13T13:08:57.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:08:57.244+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.250407+0000) 2019-10-13T13:08:57.253 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:08:57.248+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.251527+0000) 2019-10-13T13:08:57.254 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:08:57.246+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.252272+0000) 2019-10-13T13:08:57.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:08:57.248+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.251693+0000) 2019-10-13T13:08:57.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:08:57.248+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.252431+0000) 2019-10-13T13:08:57.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:57.248+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.253147+0000) 2019-10-13T13:08:57.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:08:57.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:08:57.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:08:57.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.259242+0000) 2019-10-13T13:08:58.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:08:58.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.793894+0000) 2019-10-13T13:08:58.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:08:58.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.795907+0000) 2019-10-13T13:09:00.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:00.022+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:00.027050+0000) 2019-10-13T13:09:02.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:02.158+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.165639+0000) 2019-10-13T13:09:02.262 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:02.256+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.260638+0000) 2019-10-13T13:09:02.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:02.256+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.261735+0000) 2019-10-13T13:09:02.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:02.256+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.262314+0000) 2019-10-13T13:09:02.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:02.256+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.262884+0000) 2019-10-13T13:09:02.265 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:02.256+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.263176+0000) 2019-10-13T13:09:02.265 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:02.260+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.263830+0000) 2019-10-13T13:09:02.266 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:02.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.264390+0000) 2019-10-13T13:09:02.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:02.262+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.268860+0000) 2019-10-13T13:09:02.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:02.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:02.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:02.280+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:02.284742+0000) 2019-10-13T13:09:05.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:05.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:05.559189+0000) 2019-10-13T13:09:05.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:05.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:05.845443+0000) 2019-10-13T13:09:06.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:06.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:06.527160+0000) 2019-10-13T13:09:07.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:07.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.273057+0000) 2019-10-13T13:09:07.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:07.268+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.273467+0000) 2019-10-13T13:09:07.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:07.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.273701+0000) 2019-10-13T13:09:07.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:07.268+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.274316+0000) 2019-10-13T13:09:07.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:07.268+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.274762+0000) 2019-10-13T13:09:07.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:07.270+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.275610+0000) 2019-10-13T13:09:07.278 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:07.272+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.275745+0000) 2019-10-13T13:09:07.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:07.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 164976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:07.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:07.278+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.282847+0000) 2019-10-13T13:09:08.273 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:09:08.273 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:08.277 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:09:08.277 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:08.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:08.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.794102+0000) 2019-10-13T13:09:08.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:08.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.796103+0000) 2019-10-13T13:09:10.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:10.022+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:10.027281+0000) 2019-10-13T13:09:12.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:12.170+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.172266+0000) 2019-10-13T13:09:12.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:12.280+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.284656+0000) 2019-10-13T13:09:12.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:12.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.285220+0000) 2019-10-13T13:09:12.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:12.280+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.286052+0000) 2019-10-13T13:09:12.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:12.280+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.286314+0000) 2019-10-13T13:09:12.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:12.280+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.287013+0000) 2019-10-13T13:09:12.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:12.284+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.287561+0000) 2019-10-13T13:09:12.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:12.284+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.288309+0000) 2019-10-13T13:09:12.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:12.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:12.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:12.290+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:12.295044+0000) 2019-10-13T13:09:15.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:15.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:15.559382+0000) 2019-10-13T13:09:15.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:15.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:15.845644+0000) 2019-10-13T13:09:16.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:16.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:16.527350+0000) 2019-10-13T13:09:17.298 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:17.292+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.296456+0000) 2019-10-13T13:09:17.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:17.292+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.296915+0000) 2019-10-13T13:09:17.299 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:17.292+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.297185+0000) 2019-10-13T13:09:17.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:17.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.297713+0000) 2019-10-13T13:09:17.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:17.292+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.297911+0000) 2019-10-13T13:09:17.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:17.292+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.298002+0000) 2019-10-13T13:09:17.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:17.292+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.298230+0000) 2019-10-13T13:09:17.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:17.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:17.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:17.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.305717+0000) 2019-10-13T13:09:18.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:18.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.794305+0000) 2019-10-13T13:09:18.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:18.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.796299+0000) 2019-10-13T13:09:19.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:19.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:19.035450+0000) 2019-10-13T13:09:20.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:20.026+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:20.027477+0000) 2019-10-13T13:09:22.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:22.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.178753+0000) 2019-10-13T13:09:22.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:22.304+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.308063+0000) 2019-10-13T13:09:22.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:22.304+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.308138+0000) 2019-10-13T13:09:22.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:22.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.309380+0000) 2019-10-13T13:09:22.311 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:22.304+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.308797+0000) 2019-10-13T13:09:22.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:22.304+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.309495+0000) 2019-10-13T13:09:22.312 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:22.304+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.309776+0000) 2019-10-13T13:09:22.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:22.304+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.310492+0000) 2019-10-13T13:09:22.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:22.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:22.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:22.314+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:22.316641+0000) 2019-10-13T13:09:25.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:25.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:25.559583+0000) 2019-10-13T13:09:25.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:25.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:25.845850+0000) 2019-10-13T13:09:26.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:26.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:26.527543+0000) 2019-10-13T13:09:27.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:27.312+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.318646+0000) 2019-10-13T13:09:27.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:27.316+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.319461+0000) 2019-10-13T13:09:27.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:27.318+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.320204+0000) 2019-10-13T13:09:27.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:27.316+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.320629+0000) 2019-10-13T13:09:27.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:27.316+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.321640+0000) 2019-10-13T13:09:27.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:27.316+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.322067+0000) 2019-10-13T13:09:27.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:27.316+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.322250+0000) 2019-10-13T13:09:27.328 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:27.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:27.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:27.326+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.327973+0000) 2019-10-13T13:09:28.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:28.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.794506+0000) 2019-10-13T13:09:28.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:28.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.796495+0000) 2019-10-13T13:09:29.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:29.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:29.035679+0000) 2019-10-13T13:09:30.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:30.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:30.027669+0000) 2019-10-13T13:09:32.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:32.181+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.185353+0000) 2019-10-13T13:09:32.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:32.324+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.329842+0000) 2019-10-13T13:09:32.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:32.329+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.331082+0000) 2019-10-13T13:09:32.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:32.328+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.331752+0000) 2019-10-13T13:09:32.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:32.328+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.332308+0000) 2019-10-13T13:09:32.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:32.328+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.332599+0000) 2019-10-13T13:09:32.335 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:32.328+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.332883+0000) 2019-10-13T13:09:32.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:32.328+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.333598+0000) 2019-10-13T13:09:32.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:32.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:32.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:32.337+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:32.339682+0000) 2019-10-13T13:09:35.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:35.555+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:35.559784+0000) 2019-10-13T13:09:35.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:35.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:35.846056+0000) 2019-10-13T13:09:36.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:36.523+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:36.527734+0000) 2019-10-13T13:09:37.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:37.335+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.342729+0000) 2019-10-13T13:09:37.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:37.335+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.343013+0000) 2019-10-13T13:09:37.345 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:37.341+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.343695+0000) 2019-10-13T13:09:37.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:37.339+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.344028+0000) 2019-10-13T13:09:37.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:37.339+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.344249+0000) 2019-10-13T13:09:37.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:37.339+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.344451+0000) 2019-10-13T13:09:37.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:37.339+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.345293+0000) 2019-10-13T13:09:37.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:37.345+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.348813+0000) 2019-10-13T13:09:37.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:37.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:38.401 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:09:38.401 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:38.405 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:09:38.405 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:38.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:38.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:38.794658+0000) 2019-10-13T13:09:38.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:38.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:38.796690+0000) 2019-10-13T13:09:39.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:39.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.035911+0000) 2019-10-13T13:09:40.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:40.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:40.027883+0000) 2019-10-13T13:09:42.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:42.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.191724+0000) 2019-10-13T13:09:42.356 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:42.347+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.354247+0000) 2019-10-13T13:09:42.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:42.347+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.354582+0000) 2019-10-13T13:09:42.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:42.351+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.355972+0000) 2019-10-13T13:09:42.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:42.351+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.356458+0000) 2019-10-13T13:09:42.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:42.351+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.356437+0000) 2019-10-13T13:09:42.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:42.353+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.357160+0000) 2019-10-13T13:09:42.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:42.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.357224+0000) 2019-10-13T13:09:42.363 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:42.355+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:42.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:42.361+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:42.362712+0000) 2019-10-13T13:09:45.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:45.559+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:45.560000+0000) 2019-10-13T13:09:45.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:45.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:45.846260+0000) 2019-10-13T13:09:46.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:46.523+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:46.527928+0000) 2019-10-13T13:09:47.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:47.359+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.365048+0000) 2019-10-13T13:09:47.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:47.361+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.365818+0000) 2019-10-13T13:09:47.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:47.359+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.366248+0000) 2019-10-13T13:09:47.369 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:47.359+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.367209+0000) 2019-10-13T13:09:47.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:47.359+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.367017+0000) 2019-10-13T13:09:47.369 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:47.363+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.367516+0000) 2019-10-13T13:09:47.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:47.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.368260+0000) 2019-10-13T13:09:47.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:47.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:47.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:47.369+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.373157+0000) 2019-10-13T13:09:48.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:48.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:48.794882+0000) 2019-10-13T13:09:48.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:48.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:48.796923+0000) 2019-10-13T13:09:49.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:49.031+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.036142+0000) 2019-10-13T13:09:50.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:50.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:50.028068+0000) 2019-10-13T13:09:52.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:52.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.198328+0000) 2019-10-13T13:09:52.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:52.371+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.375192+0000) 2019-10-13T13:09:52.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:52.371+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.375591+0000) 2019-10-13T13:09:52.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:52.371+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.376237+0000) 2019-10-13T13:09:52.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:52.371+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.377006+0000) 2019-10-13T13:09:52.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:52.373+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.378081+0000) 2019-10-13T13:09:52.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:52.371+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.378402+0000) 2019-10-13T13:09:52.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:52.371+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.379217+0000) 2019-10-13T13:09:52.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:52.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165279 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:52.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:52.381+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:52.385369+0000) 2019-10-13T13:09:55.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:55.559+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:55.560205+0000) 2019-10-13T13:09:55.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:55.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:55.846459+0000) 2019-10-13T13:09:56.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:56.527+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:56.528148+0000) 2019-10-13T13:09:57.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:09:57.383+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.387137+0000) 2019-10-13T13:09:57.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:57.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.387504+0000) 2019-10-13T13:09:57.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:09:57.385+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.388507+0000) 2019-10-13T13:09:57.390 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:57.383+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.388996+0000) 2019-10-13T13:09:57.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:57.383+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.389588+0000) 2019-10-13T13:09:57.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:09:57.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.390614+0000) 2019-10-13T13:09:57.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:09:57.383+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.390973+0000) 2019-10-13T13:09:57.396 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:09:57.393+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.394924+0000) 2019-10-13T13:09:57.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:09:57.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:09:58.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:09:58.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:58.795127+0000) 2019-10-13T13:09:58.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:09:58.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:58.797147+0000) 2019-10-13T13:09:59.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:09:59.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.036347+0000) 2019-10-13T13:10:00.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:00.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:00.028255+0000) 2019-10-13T13:10:02.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:02.209+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.209601+0000) 2019-10-13T13:10:02.402 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:02.395+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.400322+0000) 2019-10-13T13:10:02.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:02.395+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.400642+0000) 2019-10-13T13:10:02.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:02.395+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.401067+0000) 2019-10-13T13:10:02.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:02.395+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.401680+0000) 2019-10-13T13:10:02.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:02.401+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.402609+0000) 2019-10-13T13:10:02.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:02.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.402686+0000) 2019-10-13T13:10:02.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:02.399+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.403735+0000) 2019-10-13T13:10:02.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:02.405+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:02.406459+0000) 2019-10-13T13:10:02.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:02.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:05.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:05.559+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:05.560403+0000) 2019-10-13T13:10:05.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:05.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:05.846696+0000) 2019-10-13T13:10:06.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:06.527+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:06.528347+0000) 2019-10-13T13:10:07.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:07.411+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.412162+0000) 2019-10-13T13:10:07.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:07.411+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.412567+0000) 2019-10-13T13:10:07.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:07.411+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.413207+0000) 2019-10-13T13:10:07.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:07.411+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.413628+0000) 2019-10-13T13:10:07.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:07.411+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.414541+0000) 2019-10-13T13:10:07.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:07.411+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.414664+0000) 2019-10-13T13:10:07.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:07.413+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.415776+0000) 2019-10-13T13:10:07.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:07.417+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.419357+0000) 2019-10-13T13:10:07.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:07.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:08.545 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:10:08.546 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:08.549 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:10:08.549 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:08.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:08.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:08.795342+0000) 2019-10-13T13:10:08.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:08.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:08.797376+0000) 2019-10-13T13:10:09.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:09.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.036538+0000) 2019-10-13T13:10:10.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:10.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:10.028454+0000) 2019-10-13T13:10:12.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:12.213+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.216116+0000) 2019-10-13T13:10:12.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:12.419+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.422703+0000) 2019-10-13T13:10:12.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:12.419+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.423258+0000) 2019-10-13T13:10:12.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:12.423+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.424039+0000) 2019-10-13T13:10:12.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:12.423+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.424373+0000) 2019-10-13T13:10:12.427 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:12.421+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.425529+0000) 2019-10-13T13:10:12.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:12.423+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.425427+0000) 2019-10-13T13:10:12.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:12.423+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.426414+0000) 2019-10-13T13:10:12.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:12.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:12.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:12.429+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:12.433594+0000) 2019-10-13T13:10:15.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:15.559+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:15.560626+0000) 2019-10-13T13:10:15.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:15.843+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:15.846894+0000) 2019-10-13T13:10:16.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:16.527+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:16.528564+0000) 2019-10-13T13:10:17.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:17.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.434519+0000) 2019-10-13T13:10:17.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:17.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.434708+0000) 2019-10-13T13:10:17.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:17.435+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.435375+0000) 2019-10-13T13:10:17.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:17.435+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.436355+0000) 2019-10-13T13:10:17.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:17.435+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.436230+0000) 2019-10-13T13:10:17.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:17.435+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.437167+0000) 2019-10-13T13:10:17.439 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:17.435+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.437514+0000) 2019-10-13T13:10:17.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:17.441+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.442376+0000) 2019-10-13T13:10:17.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:17.443+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:18.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:18.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:18.795546+0000) 2019-10-13T13:10:18.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:18.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:18.797575+0000) 2019-10-13T13:10:19.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:19.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.036733+0000) 2019-10-13T13:10:20.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:20.025+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:20.028655+0000) 2019-10-13T13:10:22.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:22.221+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.221935+0000) 2019-10-13T13:10:22.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:22.446+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.447516+0000) 2019-10-13T13:10:22.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:22.446+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.448425+0000) 2019-10-13T13:10:22.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:22.445+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.448913+0000) 2019-10-13T13:10:22.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:22.446+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.449205+0000) 2019-10-13T13:10:22.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:22.446+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.449419+0000) 2019-10-13T13:10:22.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:22.446+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.449838+0000) 2019-10-13T13:10:22.452 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:22.446+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.450060+0000) 2019-10-13T13:10:22.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:22.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165481 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:22.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:22.453+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:22.457137+0000) 2019-10-13T13:10:25.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:25.558+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:25.560827+0000) 2019-10-13T13:10:25.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:25.842+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:25.847091+0000) 2019-10-13T13:10:26.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:26.526+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:26.528760+0000) 2019-10-13T13:10:27.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:27.454+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.459254+0000) 2019-10-13T13:10:27.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:27.458+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.459720+0000) 2019-10-13T13:10:27.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:27.458+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.460428+0000) 2019-10-13T13:10:27.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:27.458+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.460651+0000) 2019-10-13T13:10:27.463 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:27.456+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.461984+0000) 2019-10-13T13:10:27.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:27.458+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.462344+0000) 2019-10-13T13:10:27.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:27.458+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.462806+0000) 2019-10-13T13:10:27.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:27.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:27.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:27.464+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.467551+0000) 2019-10-13T13:10:28.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:28.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:28.795754+0000) 2019-10-13T13:10:28.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:28.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:28.797762+0000) 2019-10-13T13:10:29.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:29.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.036928+0000) 2019-10-13T13:10:30.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:30.024+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:30.028859+0000) 2019-10-13T13:10:32.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:32.224+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.228170+0000) 2019-10-13T13:10:32.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:32.466+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.470220+0000) 2019-10-13T13:10:32.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:32.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.471175+0000) 2019-10-13T13:10:32.474 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:32.470+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.471707+0000) 2019-10-13T13:10:32.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:32.470+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.472612+0000) 2019-10-13T13:10:32.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:32.470+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.473013+0000) 2019-10-13T13:10:32.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:32.468+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.473550+0000) 2019-10-13T13:10:32.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:32.470+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.473630+0000) 2019-10-13T13:10:32.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:32.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:32.478562+0000) 2019-10-13T13:10:32.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:32.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:35.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:35.558+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:35.561026+0000) 2019-10-13T13:10:35.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:35.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:35.847353+0000) 2019-10-13T13:10:36.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:36.526+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:36.528951+0000) 2019-10-13T13:10:37.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:37.478+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.482596+0000) 2019-10-13T13:10:37.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:37.478+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.482798+0000) 2019-10-13T13:10:37.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:37.482+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.483439+0000) 2019-10-13T13:10:37.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:37.480+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.484316+0000) 2019-10-13T13:10:37.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:37.482+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.484635+0000) 2019-10-13T13:10:37.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:37.482+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.485431+0000) 2019-10-13T13:10:37.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:37.482+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.485939+0000) 2019-10-13T13:10:37.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:37.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:37.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:37.488+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.493191+0000) 2019-10-13T13:10:38.660 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:10:38.660 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:38.664 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:10:38.664 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:38.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:38.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:38.795962+0000) 2019-10-13T13:10:38.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:38.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:38.797953+0000) 2019-10-13T13:10:39.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:39.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.037151+0000) 2019-10-13T13:10:40.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:40.024+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:40.029078+0000) 2019-10-13T13:10:42.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:42.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.234502+0000) 2019-10-13T13:10:42.496 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:42.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.494239+0000) 2019-10-13T13:10:42.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:42.490+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.494593+0000) 2019-10-13T13:10:42.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:42.494+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.495813+0000) 2019-10-13T13:10:42.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:42.492+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.496816+0000) 2019-10-13T13:10:42.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:42.494+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.496794+0000) 2019-10-13T13:10:42.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:42.494+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.497103+0000) 2019-10-13T13:10:42.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:42.494+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.498187+0000) 2019-10-13T13:10:42.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:42.498+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:42.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:42.500+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:42.504146+0000) 2019-10-13T13:10:45.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:45.558+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:45.561224+0000) 2019-10-13T13:10:45.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:45.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:45.847552+0000) 2019-10-13T13:10:46.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:46.526+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:46.529161+0000) 2019-10-13T13:10:47.508 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:47.502+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.506434+0000) 2019-10-13T13:10:47.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:47.502+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.506738+0000) 2019-10-13T13:10:47.509 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:47.506+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.507828+0000) 2019-10-13T13:10:47.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:47.504+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.508971+0000) 2019-10-13T13:10:47.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:47.506+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.509117+0000) 2019-10-13T13:10:47.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:47.506+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.509521+0000) 2019-10-13T13:10:47.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:47.506+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.510173+0000) 2019-10-13T13:10:47.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:47.508+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.512612+0000) 2019-10-13T13:10:47.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:47.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:48.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:48.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:48.796195+0000) 2019-10-13T13:10:48.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:48.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:48.798151+0000) 2019-10-13T13:10:49.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:49.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.037369+0000) 2019-10-13T13:10:50.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:50.024+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:50.029274+0000) 2019-10-13T13:10:52.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:52.236+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.240792+0000) 2019-10-13T13:10:52.521 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:52.518+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.519379+0000) 2019-10-13T13:10:52.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:52.518+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.520045+0000) 2019-10-13T13:10:52.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:52.518+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.520080+0000) 2019-10-13T13:10:52.522 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:52.518+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.520363+0000) 2019-10-13T13:10:52.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:52.518+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.520696+0000) 2019-10-13T13:10:52.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:52.516+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.521575+0000) 2019-10-13T13:10:52.523 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:52.518+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.521568+0000) 2019-10-13T13:10:52.529 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:52.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:52.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:52.524+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:52.529591+0000) 2019-10-13T13:10:55.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:55.558+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:55.561442+0000) 2019-10-13T13:10:55.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:55.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:55.847752+0000) 2019-10-13T13:10:56.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:56.526+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:56.529409+0000) 2019-10-13T13:10:57.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:10:57.530+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.531664+0000) 2019-10-13T13:10:57.534 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:57.530+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.531974+0000) 2019-10-13T13:10:57.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:57.530+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.532546+0000) 2019-10-13T13:10:57.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:57.530+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.533003+0000) 2019-10-13T13:10:57.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:10:57.530+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.533252+0000) 2019-10-13T13:10:57.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:10:57.530+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.533835+0000) 2019-10-13T13:10:57.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:10:57.532+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.534478+0000) 2019-10-13T13:10:57.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:10:57.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:10:57.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:10:57.540+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.542608+0000) 2019-10-13T13:10:58.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:10:58.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:58.796397+0000) 2019-10-13T13:10:58.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:10:58.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:58.798360+0000) 2019-10-13T13:10:59.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:10:59.034+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.037570+0000) 2019-10-13T13:11:00.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:00.024+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:00.029497+0000) 2019-10-13T13:11:02.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:02.244+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.247250+0000) 2019-10-13T13:11:02.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:02.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.542198+0000) 2019-10-13T13:11:02.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:02.541+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.543523+0000) 2019-10-13T13:11:02.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:02.540+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.544246+0000) 2019-10-13T13:11:02.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:02.541+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.544807+0000) 2019-10-13T13:11:02.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:02.541+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.544440+0000) 2019-10-13T13:11:02.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:02.541+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.545152+0000) 2019-10-13T13:11:02.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:02.541+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.546477+0000) 2019-10-13T13:11:02.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:02.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165753 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:02.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:02.548+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:02.553630+0000) 2019-10-13T13:11:05.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:05.557+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:05.561677+0000) 2019-10-13T13:11:05.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:05.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:05.847951+0000) 2019-10-13T13:11:06.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:06.525+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:06.529600+0000) 2019-10-13T13:11:07.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:07.553+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.555370+0000) 2019-10-13T13:11:07.558 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:07.552+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.555957+0000) 2019-10-13T13:11:07.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:07.553+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.555950+0000) 2019-10-13T13:11:07.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:07.553+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.556443+0000) 2019-10-13T13:11:07.559 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:07.553+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.556578+0000) 2019-10-13T13:11:07.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:07.553+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.557238+0000) 2019-10-13T13:11:07.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:07.553+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.557749+0000) 2019-10-13T13:11:07.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:07.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:07.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:07.560+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.564242+0000) 2019-10-13T13:11:08.784 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:11:08.784 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:08.788 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:11:08.788 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:08.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:08.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:08.796587+0000) 2019-10-13T13:11:08.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:08.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:08.798547+0000) 2019-10-13T13:11:09.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:09.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.037765+0000) 2019-10-13T13:11:10.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:10.024+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:10.029697+0000) 2019-10-13T13:11:12.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:12.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.253739+0000) 2019-10-13T13:11:12.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:12.561+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.565352+0000) 2019-10-13T13:11:12.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:12.561+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.566664+0000) 2019-10-13T13:11:12.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:12.561+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.566982+0000) 2019-10-13T13:11:12.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:12.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.567869+0000) 2019-10-13T13:11:12.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:12.564+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.568462+0000) 2019-10-13T13:11:12.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:12.565+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.568487+0000) 2019-10-13T13:11:12.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:12.565+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.568863+0000) 2019-10-13T13:11:12.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:12.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165819 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:12.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:12.572+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:12.575942+0000) 2019-10-13T13:11:15.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:15.557+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:15.561898+0000) 2019-10-13T13:11:15.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:15.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:15.848156+0000) 2019-10-13T13:11:16.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:16.525+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:16.529823+0000) 2019-10-13T13:11:17.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:17.573+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.578419+0000) 2019-10-13T13:11:17.581 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:17.577+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.579911+0000) 2019-10-13T13:11:17.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:17.577+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.580118+0000) 2019-10-13T13:11:17.582 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:17.575+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.580629+0000) 2019-10-13T13:11:17.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:17.577+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.580844+0000) 2019-10-13T13:11:17.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:17.577+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.581471+0000) 2019-10-13T13:11:17.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:17.577+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.581196+0000) 2019-10-13T13:11:17.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:17.579+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.585310+0000) 2019-10-13T13:11:17.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:17.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:18.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:18.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:18.796814+0000) 2019-10-13T13:11:18.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:18.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:18.798751+0000) 2019-10-13T13:11:19.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:19.033+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.037983+0000) 2019-10-13T13:11:20.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:20.023+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:20.029898+0000) 2019-10-13T13:11:22.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:22.255+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.260092+0000) 2019-10-13T13:11:22.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:22.585+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.590031+0000) 2019-10-13T13:11:22.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:22.587+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.591048+0000) 2019-10-13T13:11:22.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:22.589+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.591396+0000) 2019-10-13T13:11:22.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:22.589+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.591866+0000) 2019-10-13T13:11:22.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:22.589+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.592328+0000) 2019-10-13T13:11:22.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:22.589+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.592603+0000) 2019-10-13T13:11:22.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:22.589+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.593842+0000) 2019-10-13T13:11:22.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:22.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:22.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:22.599+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:22.602947+0000) 2019-10-13T13:11:25.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:25.557+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:25.562079+0000) 2019-10-13T13:11:25.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:25.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:25.848355+0000) 2019-10-13T13:11:26.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:26.529+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:26.530017+0000) 2019-10-13T13:11:27.604 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:27.599+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.602548+0000) 2019-10-13T13:11:27.605 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:27.597+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.603248+0000) 2019-10-13T13:11:27.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:27.601+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.603380+0000) 2019-10-13T13:11:27.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:27.601+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.604067+0000) 2019-10-13T13:11:27.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:27.601+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.604744+0000) 2019-10-13T13:11:27.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:27.601+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.604936+0000) 2019-10-13T13:11:27.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:27.601+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.605934+0000) 2019-10-13T13:11:27.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:27.607+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.610926+0000) 2019-10-13T13:11:27.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:27.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165922 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:28.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:28.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:28.797044+0000) 2019-10-13T13:11:28.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:28.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:28.798961+0000) 2019-10-13T13:11:29.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:29.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.038183+0000) 2019-10-13T13:11:30.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:30.023+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:30.030119+0000) 2019-10-13T13:11:32.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:32.263+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.266650+0000) 2019-10-13T13:11:32.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:32.613+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.615645+0000) 2019-10-13T13:11:32.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:32.613+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.616844+0000) 2019-10-13T13:11:32.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:32.611+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.617577+0000) 2019-10-13T13:11:32.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:32.613+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.617843+0000) 2019-10-13T13:11:32.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:32.613+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.618527+0000) 2019-10-13T13:11:32.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:32.613+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.618955+0000) 2019-10-13T13:11:32.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:32.617+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.619893+0000) 2019-10-13T13:11:32.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:32.619+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:32.622754+0000) 2019-10-13T13:11:32.625 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:32.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165957 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:35.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:35.561+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:35.562297+0000) 2019-10-13T13:11:35.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:35.845+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:35.848589+0000) 2019-10-13T13:11:36.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:36.529+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:36.530217+0000) 2019-10-13T13:11:37.629 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:37.623+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.627278+0000) 2019-10-13T13:11:37.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:37.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.628014+0000) 2019-10-13T13:11:37.630 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:37.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.628474+0000) 2019-10-13T13:11:37.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:37.625+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.628787+0000) 2019-10-13T13:11:37.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:37.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.629476+0000) 2019-10-13T13:11:37.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:37.625+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.630150+0000) 2019-10-13T13:11:37.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:37.625+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.630708+0000) 2019-10-13T13:11:37.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:37.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 165992 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:37.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:37.631+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.637137+0000) 2019-10-13T13:11:38.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:38.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:38.797250+0000) 2019-10-13T13:11:38.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:38.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:38.799168+0000) 2019-10-13T13:11:38.907 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:11:38.907 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:38.910 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:11:38.911 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:39.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:39.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.038405+0000) 2019-10-13T13:11:40.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:40.027+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:40.030353+0000) 2019-10-13T13:11:42.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:42.271+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.273042+0000) 2019-10-13T13:11:42.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:42.633+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.639274+0000) 2019-10-13T13:11:42.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:42.637+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.639685+0000) 2019-10-13T13:11:42.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:42.637+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.640393+0000) 2019-10-13T13:11:42.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:42.637+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.641063+0000) 2019-10-13T13:11:42.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:42.637+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.641690+0000) 2019-10-13T13:11:42.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:42.639+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.642640+0000) 2019-10-13T13:11:42.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:42.637+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.642653+0000) 2019-10-13T13:11:42.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:42.643+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:42.646844+0000) 2019-10-13T13:11:42.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:42.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:45.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:45.560+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:45.562494+0000) 2019-10-13T13:11:45.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:45.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:45.848803+0000) 2019-10-13T13:11:46.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:46.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:46.530412+0000) 2019-10-13T13:11:47.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:47.644+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.650421+0000) 2019-10-13T13:11:47.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:47.644+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.650616+0000) 2019-10-13T13:11:47.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:47.648+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.651281+0000) 2019-10-13T13:11:47.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:47.648+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.651601+0000) 2019-10-13T13:11:47.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:47.648+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.651958+0000) 2019-10-13T13:11:47.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:47.647+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.652500+0000) 2019-10-13T13:11:47.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:47.648+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.653427+0000) 2019-10-13T13:11:47.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:47.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:47.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:47.655+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.660314+0000) 2019-10-13T13:11:48.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:48.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:48.797476+0000) 2019-10-13T13:11:48.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:48.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:48.799378+0000) 2019-10-13T13:11:49.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:49.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.038658+0000) 2019-10-13T13:11:50.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:50.027+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:50.030547+0000) 2019-10-13T13:11:52.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:52.275+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.279347+0000) 2019-10-13T13:11:52.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:52.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.661201+0000) 2019-10-13T13:11:52.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:52.655+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.662109+0000) 2019-10-13T13:11:52.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:52.656+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.661670+0000) 2019-10-13T13:11:52.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:52.656+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.663091+0000) 2019-10-13T13:11:52.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:52.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.662677+0000) 2019-10-13T13:11:52.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:52.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.663717+0000) 2019-10-13T13:11:52.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:52.660+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.664041+0000) 2019-10-13T13:11:52.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:52.667+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:52.670507+0000) 2019-10-13T13:11:52.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:52.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:55.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:55.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:55.562720+0000) 2019-10-13T13:11:55.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:55.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:55.849036+0000) 2019-10-13T13:11:56.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:56.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:56.530612+0000) 2019-10-13T13:11:57.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:11:57.668+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.673051+0000) 2019-10-13T13:11:57.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:11:57.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.673584+0000) 2019-10-13T13:11:57.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:57.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.674923+0000) 2019-10-13T13:11:57.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:57.672+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.675823+0000) 2019-10-13T13:11:57.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:11:57.672+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.676219+0000) 2019-10-13T13:11:57.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:57.672+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.676392+0000) 2019-10-13T13:11:57.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:11:57.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.677146+0000) 2019-10-13T13:11:57.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:11:57.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166130 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:11:57.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:11:57.675+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.682016+0000) 2019-10-13T13:11:58.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:11:58.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:58.797673+0000) 2019-10-13T13:11:58.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:11:58.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:58.799604+0000) 2019-10-13T13:11:59.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:11:59.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.038867+0000) 2019-10-13T13:12:00.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:00.027+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:00.030759+0000) 2019-10-13T13:12:02.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:02.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.285730+0000) 2019-10-13T13:12:02.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:02.680+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.685149+0000) 2019-10-13T13:12:02.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:02.680+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.685469+0000) 2019-10-13T13:12:02.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:02.680+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.685857+0000) 2019-10-13T13:12:02.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:02.680+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.686312+0000) 2019-10-13T13:12:02.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:02.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.686841+0000) 2019-10-13T13:12:02.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:02.684+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.687403+0000) 2019-10-13T13:12:02.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:02.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.688002+0000) 2019-10-13T13:12:02.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:02.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:02.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:02.691+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:02.694612+0000) 2019-10-13T13:12:05.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:05.556+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:05.562915+0000) 2019-10-13T13:12:05.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:05.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:05.849236+0000) 2019-10-13T13:12:06.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:06.528+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:06.530812+0000) 2019-10-13T13:12:07.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:07.692+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.697576+0000) 2019-10-13T13:12:07.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:07.692+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.697287+0000) 2019-10-13T13:12:07.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:07.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.698209+0000) 2019-10-13T13:12:07.701 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:07.692+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.698273+0000) 2019-10-13T13:12:07.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:07.692+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.698891+0000) 2019-10-13T13:12:07.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:07.692+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.698500+0000) 2019-10-13T13:12:07.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:07.695+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.700395+0000) 2019-10-13T13:12:07.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:07.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:07.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:07.703+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.707046+0000) 2019-10-13T13:12:08.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:08.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:08.797873+0000) 2019-10-13T13:12:08.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:08.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:08.799801+0000) 2019-10-13T13:12:09.007 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:12:09.008 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:09.012 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:12:09.012 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:09.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:09.032+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.039091+0000) 2019-10-13T13:12:10.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:10.027+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:10.030919+0000) 2019-10-13T13:12:12.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:12.286+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.292104+0000) 2019-10-13T13:12:12.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:12.704+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.708875+0000) 2019-10-13T13:12:12.711 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:12.704+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.709571+0000) 2019-10-13T13:12:12.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:12.704+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.710301+0000) 2019-10-13T13:12:12.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:12.704+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.710425+0000) 2019-10-13T13:12:12.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:12.706+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.711011+0000) 2019-10-13T13:12:12.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:12.704+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.710976+0000) 2019-10-13T13:12:12.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:12.708+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.711374+0000) 2019-10-13T13:12:12.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:12.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:12.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:12.714+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:12.718885+0000) 2019-10-13T13:12:15.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:15.560+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:15.563145+0000) 2019-10-13T13:12:15.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:15.844+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:15.849402+0000) 2019-10-13T13:12:16.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:16.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:16.531033+0000) 2019-10-13T13:12:17.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:17.716+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.722309+0000) 2019-10-13T13:12:17.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:17.718+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.722859+0000) 2019-10-13T13:12:17.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:17.716+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.722879+0000) 2019-10-13T13:12:17.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:17.716+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.723254+0000) 2019-10-13T13:12:17.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:17.720+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.723947+0000) 2019-10-13T13:12:17.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:17.720+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.723602+0000) 2019-10-13T13:12:17.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:17.720+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.724477+0000) 2019-10-13T13:12:17.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:17.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:17.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:17.726+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.731901+0000) 2019-10-13T13:12:18.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:18.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:18.798073+0000) 2019-10-13T13:12:18.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:18.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:18.800041+0000) 2019-10-13T13:12:19.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:19.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.039339+0000) 2019-10-13T13:12:20.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:20.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:20.031119+0000) 2019-10-13T13:12:22.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:22.298+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.299096+0000) 2019-10-13T13:12:22.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:22.728+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.734092+0000) 2019-10-13T13:12:22.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:22.728+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.734785+0000) 2019-10-13T13:12:22.739 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:22.728+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.735229+0000) 2019-10-13T13:12:22.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:22.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.735654+0000) 2019-10-13T13:12:22.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:22.732+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.736610+0000) 2019-10-13T13:12:22.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:22.730+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.737180+0000) 2019-10-13T13:12:22.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:22.732+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.737469+0000) 2019-10-13T13:12:22.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:22.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:22.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:22.738+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:22.744975+0000) 2019-10-13T13:12:25.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:25.560+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:25.563390+0000) 2019-10-13T13:12:25.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:25.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:25.849642+0000) 2019-10-13T13:12:26.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:26.524+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:26.531217+0000) 2019-10-13T13:12:27.747 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:27.739+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.745336+0000) 2019-10-13T13:12:27.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:27.739+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.745955+0000) 2019-10-13T13:12:27.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:27.742+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.746553+0000) 2019-10-13T13:12:27.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:27.743+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.747391+0000) 2019-10-13T13:12:27.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:27.743+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.747926+0000) 2019-10-13T13:12:27.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:27.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.748510+0000) 2019-10-13T13:12:27.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:27.743+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.748688+0000) 2019-10-13T13:12:27.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:27.746+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.753384+0000) 2019-10-13T13:12:27.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:27.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166336 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:28.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:28.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:28.798273+0000) 2019-10-13T13:12:28.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:28.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:28.800265+0000) 2019-10-13T13:12:29.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:29.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.039530+0000) 2019-10-13T13:12:30.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:30.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:30.031341+0000) 2019-10-13T13:12:32.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:32.302+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.306201+0000) 2019-10-13T13:12:32.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:32.751+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.756968+0000) 2019-10-13T13:12:32.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:32.751+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.757436+0000) 2019-10-13T13:12:32.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:32.751+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.757854+0000) 2019-10-13T13:12:32.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:32.751+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.758399+0000) 2019-10-13T13:12:32.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:32.754+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.758947+0000) 2019-10-13T13:12:32.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:32.755+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.759811+0000) 2019-10-13T13:12:32.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:32.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.760381+0000) 2019-10-13T13:12:32.767 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:32.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:32.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:32.762+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:32.767035+0000) 2019-10-13T13:12:35.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:35.559+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:35.563587+0000) 2019-10-13T13:12:35.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:35.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:35.849839+0000) 2019-10-13T13:12:36.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:36.527+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:36.531435+0000) 2019-10-13T13:12:37.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:37.763+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.768804+0000) 2019-10-13T13:12:37.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:37.763+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.769490+0000) 2019-10-13T13:12:37.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:37.763+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.769770+0000) 2019-10-13T13:12:37.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:37.763+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.770407+0000) 2019-10-13T13:12:37.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:37.763+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.770734+0000) 2019-10-13T13:12:37.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:37.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.771367+0000) 2019-10-13T13:12:37.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:37.767+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.772011+0000) 2019-10-13T13:12:37.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:37.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:37.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:37.770+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.778062+0000) 2019-10-13T13:12:38.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:38.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:38.798462+0000) 2019-10-13T13:12:38.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:38.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:38.800504+0000) 2019-10-13T13:12:39.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:39.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.039702+0000) 2019-10-13T13:12:39.128 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:12:39.128 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:39.132 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:12:39.132 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:40.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:40.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:40.031603+0000) 2019-10-13T13:12:42.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:42.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.313075+0000) 2019-10-13T13:12:42.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:42.775+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.779953+0000) 2019-10-13T13:12:42.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:42.775+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.780613+0000) 2019-10-13T13:12:42.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:42.775+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.781796+0000) 2019-10-13T13:12:42.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:42.775+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.781854+0000) 2019-10-13T13:12:42.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:42.775+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.783159+0000) 2019-10-13T13:12:42.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:42.778+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.784831+0000) 2019-10-13T13:12:42.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:42.779+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.784741+0000) 2019-10-13T13:12:42.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:42.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:42.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:42.786+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:42.791882+0000) 2019-10-13T13:12:45.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:45.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:45.563791+0000) 2019-10-13T13:12:45.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:45.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:45.850060+0000) 2019-10-13T13:12:46.533 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:46.527+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:46.531652+0000) 2019-10-13T13:12:47.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:47.787+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.791786+0000) 2019-10-13T13:12:47.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:47.786+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.792690+0000) 2019-10-13T13:12:47.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:47.787+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.791993+0000) 2019-10-13T13:12:47.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:47.787+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.792975+0000) 2019-10-13T13:12:47.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:47.787+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.793483+0000) 2019-10-13T13:12:47.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:47.787+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.794572+0000) 2019-10-13T13:12:47.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:47.791+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.795642+0000) 2019-10-13T13:12:47.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:47.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:47.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:47.798+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.802191+0000) 2019-10-13T13:12:48.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:48.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:48.798677+0000) 2019-10-13T13:12:48.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:48.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:48.800701+0000) 2019-10-13T13:12:49.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:49.035+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.039898+0000) 2019-10-13T13:12:50.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:50.026+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:50.031811+0000) 2019-10-13T13:12:52.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:52.318+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.320123+0000) 2019-10-13T13:12:52.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:52.799+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.804047+0000) 2019-10-13T13:12:52.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:52.799+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.804497+0000) 2019-10-13T13:12:52.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:52.798+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.805702+0000) 2019-10-13T13:12:52.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:52.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.806181+0000) 2019-10-13T13:12:52.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:52.799+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.806580+0000) 2019-10-13T13:12:52.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:52.799+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.807062+0000) 2019-10-13T13:12:52.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:52.803+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.807563+0000) 2019-10-13T13:12:52.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:52.802+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:52.809917+0000) 2019-10-13T13:12:52.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:52.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:55.566 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:55.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:55.564033+0000) 2019-10-13T13:12:55.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:55.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:55.850262+0000) 2019-10-13T13:12:56.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:56.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:56.531865+0000) 2019-10-13T13:12:57.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:12:57.815+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.815537+0000) 2019-10-13T13:12:57.818 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:12:57.815+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.815999+0000) 2019-10-13T13:12:57.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:57.815+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.816298+0000) 2019-10-13T13:12:57.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:12:57.815+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.817098+0000) 2019-10-13T13:12:57.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:12:57.818+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.818427+0000) 2019-10-13T13:12:57.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:57.815+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.818492+0000) 2019-10-13T13:12:57.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:57.815+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.819191+0000) 2019-10-13T13:12:57.825 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:12:57.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:12:57.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:12:57.826+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.828637+0000) 2019-10-13T13:12:58.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:12:58.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:58.798892+0000) 2019-10-13T13:12:58.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:12:58.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:58.800897+0000) 2019-10-13T13:12:59.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:12:59.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.040091+0000) 2019-10-13T13:13:00.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:00.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:00.032040+0000) 2019-10-13T13:13:02.328 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:02.326+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.326923+0000) 2019-10-13T13:13:02.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:02.823+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.826365+0000) 2019-10-13T13:13:02.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:02.823+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.827161+0000) 2019-10-13T13:13:02.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:02.827+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.828186+0000) 2019-10-13T13:13:02.830 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:02.826+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.828927+0000) 2019-10-13T13:13:02.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:02.827+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.828873+0000) 2019-10-13T13:13:02.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:02.827+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.829381+0000) 2019-10-13T13:13:02.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:02.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.829824+0000) 2019-10-13T13:13:02.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:02.834+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:02.834451+0000) 2019-10-13T13:13:02.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:02.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:05.566 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:05.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:05.564258+0000) 2019-10-13T13:13:05.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:05.847+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:05.850466+0000) 2019-10-13T13:13:06.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:06.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:06.532063+0000) 2019-10-13T13:13:07.839 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:07.835+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.837495+0000) 2019-10-13T13:13:07.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:07.835+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.838591+0000) 2019-10-13T13:13:07.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:07.839+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.839417+0000) 2019-10-13T13:13:07.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:07.837+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.840098+0000) 2019-10-13T13:13:07.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:07.839+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.840175+0000) 2019-10-13T13:13:07.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:07.839+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.840663+0000) 2019-10-13T13:13:07.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:07.839+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.841515+0000) 2019-10-13T13:13:07.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:07.841+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.843425+0000) 2019-10-13T13:13:07.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:07.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166611 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:08.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:08.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:08.799088+0000) 2019-10-13T13:13:08.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:08.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:08.801104+0000) 2019-10-13T13:13:09.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:09.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.040300+0000) 2019-10-13T13:13:09.285 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:13:09.285 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:09.289 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:13:09.289 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:10.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:10.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:10.032236+0000) 2019-10-13T13:13:12.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:12.329+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.333838+0000) 2019-10-13T13:13:12.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:12.846+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.849449+0000) 2019-10-13T13:13:12.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:12.846+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.850072+0000) 2019-10-13T13:13:12.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:12.846+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.850660+0000) 2019-10-13T13:13:12.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:12.849+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.851539+0000) 2019-10-13T13:13:12.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:12.850+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.851685+0000) 2019-10-13T13:13:12.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:12.850+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.852715+0000) 2019-10-13T13:13:12.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:12.850+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.853045+0000) 2019-10-13T13:13:12.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:12.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166648 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:12.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:12.857+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:12.860582+0000) 2019-10-13T13:13:15.566 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:15.562+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:15.564481+0000) 2019-10-13T13:13:15.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:15.846+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:15.850690+0000) 2019-10-13T13:13:16.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:16.530+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:16.532257+0000) 2019-10-13T13:13:17.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:17.858+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.861727+0000) 2019-10-13T13:13:17.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:17.858+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.862367+0000) 2019-10-13T13:13:17.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:17.861+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.863012+0000) 2019-10-13T13:13:17.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:17.862+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.863738+0000) 2019-10-13T13:13:17.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:17.862+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.864213+0000) 2019-10-13T13:13:17.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:17.862+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.864982+0000) 2019-10-13T13:13:17.867 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:17.862+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.865285+0000) 2019-10-13T13:13:17.871 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:17.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:17.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:17.869+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.870851+0000) 2019-10-13T13:13:18.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:18.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:18.799216+0000) 2019-10-13T13:13:18.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:18.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:18.801274+0000) 2019-10-13T13:13:19.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:19.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.040495+0000) 2019-10-13T13:13:20.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:20.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:20.032440+0000) 2019-10-13T13:13:22.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:22.337+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.340953+0000) 2019-10-13T13:13:22.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:22.870+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.872501+0000) 2019-10-13T13:13:22.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:22.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.874147+0000) 2019-10-13T13:13:22.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:22.870+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.874505+0000) 2019-10-13T13:13:22.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:22.870+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.875274+0000) 2019-10-13T13:13:22.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:22.874+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.875486+0000) 2019-10-13T13:13:22.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:22.873+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.876044+0000) 2019-10-13T13:13:22.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:22.874+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.876053+0000) 2019-10-13T13:13:22.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:22.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:22.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:22.881+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:22.882510+0000) 2019-10-13T13:13:25.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:25.562+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:25.564689+0000) 2019-10-13T13:13:25.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:25.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:25.850893+0000) 2019-10-13T13:13:26.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:26.530+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:26.532449+0000) 2019-10-13T13:13:27.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:27.882+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.884523+0000) 2019-10-13T13:13:27.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:27.882+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.885032+0000) 2019-10-13T13:13:27.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:27.882+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.885219+0000) 2019-10-13T13:13:27.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:27.882+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.886217+0000) 2019-10-13T13:13:27.889 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:27.882+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.886695+0000) 2019-10-13T13:13:27.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:27.885+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.887674+0000) 2019-10-13T13:13:27.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:27.886+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.887885+0000) 2019-10-13T13:13:27.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:27.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166748 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:27.895 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:27.889+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.893331+0000) 2019-10-13T13:13:28.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:28.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:28.799395+0000) 2019-10-13T13:13:28.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:28.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:28.801483+0000) 2019-10-13T13:13:29.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:29.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.040641+0000) 2019-10-13T13:13:30.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:30.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:30.032642+0000) 2019-10-13T13:13:32.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:32.345+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.347750+0000) 2019-10-13T13:13:32.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:32.894+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.896924+0000) 2019-10-13T13:13:32.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:32.894+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.897443+0000) 2019-10-13T13:13:32.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:32.894+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.897682+0000) 2019-10-13T13:13:32.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:32.897+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.898698+0000) 2019-10-13T13:13:32.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:32.894+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.898689+0000) 2019-10-13T13:13:32.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:32.894+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.899279+0000) 2019-10-13T13:13:32.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:32.898+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.899635+0000) 2019-10-13T13:13:32.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:32.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:32.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:32.901+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:32.905784+0000) 2019-10-13T13:13:35.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:35.562+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:35.564925+0000) 2019-10-13T13:13:35.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:35.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:35.851114+0000) 2019-10-13T13:13:36.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:36.530+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:36.532681+0000) 2019-10-13T13:13:37.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:37.906+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.908381+0000) 2019-10-13T13:13:37.911 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:37.906+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.908978+0000) 2019-10-13T13:13:37.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:37.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.909378+0000) 2019-10-13T13:13:37.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:37.906+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.909536+0000) 2019-10-13T13:13:37.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:37.906+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.910639+0000) 2019-10-13T13:13:37.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:37.909+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.911952+0000) 2019-10-13T13:13:37.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:37.910+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.912009+0000) 2019-10-13T13:13:37.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:37.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:37.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:37.913+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.917693+0000) 2019-10-13T13:13:38.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:38.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:38.799594+0000) 2019-10-13T13:13:38.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:38.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:38.801657+0000) 2019-10-13T13:13:39.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:39.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.040830+0000) 2019-10-13T13:13:39.399 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:13:39.400 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:39.403 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:13:39.403 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:40.034 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:40.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:40.032848+0000) 2019-10-13T13:13:42.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:42.353+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.354539+0000) 2019-10-13T13:13:42.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:42.918+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.919818+0000) 2019-10-13T13:13:42.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:42.918+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.920367+0000) 2019-10-13T13:13:42.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:42.918+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.920991+0000) 2019-10-13T13:13:42.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:42.918+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.921344+0000) 2019-10-13T13:13:42.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:42.918+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.921756+0000) 2019-10-13T13:13:42.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:42.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.923553+0000) 2019-10-13T13:13:42.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:42.922+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.923798+0000) 2019-10-13T13:13:42.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:42.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:42.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:42.925+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:42.929661+0000) 2019-10-13T13:13:45.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:45.562+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:45.565121+0000) 2019-10-13T13:13:45.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:45.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:45.851337+0000) 2019-10-13T13:13:46.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:46.530+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:46.532904+0000) 2019-10-13T13:13:47.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:47.926+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.931006+0000) 2019-10-13T13:13:47.933 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:47.930+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.931926+0000) 2019-10-13T13:13:47.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:47.930+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.932688+0000) 2019-10-13T13:13:47.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:47.930+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.933088+0000) 2019-10-13T13:13:47.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:47.930+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.933730+0000) 2019-10-13T13:13:47.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:47.930+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.933848+0000) 2019-10-13T13:13:47.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:47.933+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.935505+0000) 2019-10-13T13:13:47.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:47.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:47.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:47.941+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.942426+0000) 2019-10-13T13:13:48.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:48.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:48.799779+0000) 2019-10-13T13:13:48.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:48.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:48.801835+0000) 2019-10-13T13:13:49.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:49.038+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.041039+0000) 2019-10-13T13:13:50.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:50.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:50.033050+0000) 2019-10-13T13:13:52.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:52.365+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.366357+0000) 2019-10-13T13:13:52.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:52.941+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.944187+0000) 2019-10-13T13:13:52.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:52.941+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.944926+0000) 2019-10-13T13:13:52.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:52.941+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.945385+0000) 2019-10-13T13:13:52.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:52.941+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.945928+0000) 2019-10-13T13:13:52.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:52.941+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.946261+0000) 2019-10-13T13:13:52.948 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:52.941+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.946016+0000) 2019-10-13T13:13:52.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:52.945+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.948048+0000) 2019-10-13T13:13:52.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:52.949+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:52.951169+0000) 2019-10-13T13:13:52.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:52.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:55.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:55.561+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:55.565331+0000) 2019-10-13T13:13:55.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:55.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:55.851534+0000) 2019-10-13T13:13:56.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:56.529+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:56.533098+0000) 2019-10-13T13:13:57.957 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:57.953+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.955439+0000) 2019-10-13T13:13:57.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:13:57.953+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.955945+0000) 2019-10-13T13:13:57.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:13:57.953+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.956198+0000) 2019-10-13T13:13:57.959 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:57.953+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.956552+0000) 2019-10-13T13:13:57.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:13:57.953+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.957234+0000) 2019-10-13T13:13:57.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:13:57.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.957869+0000) 2019-10-13T13:13:57.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:57.953+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.958678+0000) 2019-10-13T13:13:57.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:13:57.961+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.964759+0000) 2019-10-13T13:13:57.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:13:57.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:13:58.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:13:58.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:58.799959+0000) 2019-10-13T13:13:58.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:13:58.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:58.802008+0000) 2019-10-13T13:13:59.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:13:59.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.041263+0000) 2019-10-13T13:14:00.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:00.029+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:00.033243+0000) 2019-10-13T13:14:02.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:02.368+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.372715+0000) 2019-10-13T13:14:02.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:02.961+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.967178+0000) 2019-10-13T13:14:02.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:02.965+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.968577+0000) 2019-10-13T13:14:02.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:02.965+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.968934+0000) 2019-10-13T13:14:02.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:02.964+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.969698+0000) 2019-10-13T13:14:02.972 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:02.965+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.969474+0000) 2019-10-13T13:14:02.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:02.965+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.970046+0000) 2019-10-13T13:14:02.972 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:02.965+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.970533+0000) 2019-10-13T13:14:02.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:02.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 166993 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:02.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:02.976+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:02.980235+0000) 2019-10-13T13:14:05.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:05.561+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:05.565535+0000) 2019-10-13T13:14:05.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:05.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:05.851698+0000) 2019-10-13T13:14:06.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:06.529+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:06.533293+0000) 2019-10-13T13:14:07.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:07.977+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.980430+0000) 2019-10-13T13:14:07.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:07.977+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.980936+0000) 2019-10-13T13:14:07.983 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:07.977+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.981862+0000) 2019-10-13T13:14:07.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:07.980+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.982402+0000) 2019-10-13T13:14:07.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:07.977+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.982959+0000) 2019-10-13T13:14:07.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:07.981+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.983551+0000) 2019-10-13T13:14:07.985 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:07.981+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.983905+0000) 2019-10-13T13:14:07.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:07.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:07.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:07.984+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.989556+0000) 2019-10-13T13:14:08.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:08.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:08.800134+0000) 2019-10-13T13:14:08.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:08.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:08.802196+0000) 2019-10-13T13:14:09.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:09.037+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.041452+0000) 2019-10-13T13:14:09.519 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:14:09.519 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:09.523 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:14:09.523 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:10.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:10.032+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:10.033455+0000) 2019-10-13T13:14:12.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:12.376+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.378959+0000) 2019-10-13T13:14:12.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:12.989+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.992632+0000) 2019-10-13T13:14:12.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:12.989+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.992881+0000) 2019-10-13T13:14:12.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:12.988+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.993951+0000) 2019-10-13T13:14:12.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:12.989+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.994244+0000) 2019-10-13T13:14:12.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:12.989+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.994325+0000) 2019-10-13T13:14:12.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:12.989+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.994804+0000) 2019-10-13T13:14:12.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:12.993+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:12.995773+0000) 2019-10-13T13:14:13.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:12.997+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:13.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:13.000+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:13.005008+0000) 2019-10-13T13:14:15.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:15.565+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:15.565764+0000) 2019-10-13T13:14:15.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:15.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:15.851921+0000) 2019-10-13T13:14:16.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:16.529+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:16.533488+0000) 2019-10-13T13:14:18.005 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:18.001+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.003759+0000) 2019-10-13T13:14:18.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:18.001+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.004374+0000) 2019-10-13T13:14:18.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:18.001+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.004401+0000) 2019-10-13T13:14:18.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:18.001+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.005133+0000) 2019-10-13T13:14:18.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:18.000+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.005695+0000) 2019-10-13T13:14:18.008 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:18.001+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.006477+0000) 2019-10-13T13:14:18.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:18.001+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.006725+0000) 2019-10-13T13:14:18.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:18.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:18.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:18.008+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.012470+0000) 2019-10-13T13:14:18.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:18.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.800306+0000) 2019-10-13T13:14:18.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:18.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.802331+0000) 2019-10-13T13:14:19.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:19.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.041655+0000) 2019-10-13T13:14:20.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:20.032+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:20.033672+0000) 2019-10-13T13:14:22.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:22.380+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:22.385404+0000) 2019-10-13T13:14:23.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:23.009+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.014972+0000) 2019-10-13T13:14:23.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:23.013+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.015613+0000) 2019-10-13T13:14:23.018 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:23.013+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.016644+0000) 2019-10-13T13:14:23.019 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:23.013+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.016744+0000) 2019-10-13T13:14:23.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:23.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.017654+0000) 2019-10-13T13:14:23.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:23.016+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.018370+0000) 2019-10-13T13:14:23.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:23.013+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.018148+0000) 2019-10-13T13:14:23.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:23.020+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:23.022948+0000) 2019-10-13T13:14:23.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:23.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:25.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:25.561+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:25.565989+0000) 2019-10-13T13:14:25.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:25.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:25.852127+0000) 2019-10-13T13:14:26.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:26.533+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:26.533686+0000) 2019-10-13T13:14:28.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:28.025+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.027425+0000) 2019-10-13T13:14:28.030 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:28.025+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.027858+0000) 2019-10-13T13:14:28.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:28.025+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.028518+0000) 2019-10-13T13:14:28.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:28.025+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.029346+0000) 2019-10-13T13:14:28.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:28.025+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.029923+0000) 2019-10-13T13:14:28.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:28.025+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.030514+0000) 2019-10-13T13:14:28.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:28.028+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.031795+0000) 2019-10-13T13:14:28.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:28.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:28.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:28.032+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.037543+0000) 2019-10-13T13:14:28.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:28.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.800503+0000) 2019-10-13T13:14:28.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:28.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.802504+0000) 2019-10-13T13:14:29.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:29.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.041840+0000) 2019-10-13T13:14:30.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:30.032+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:30.033870+0000) 2019-10-13T13:14:32.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:32.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:32.392082+0000) 2019-10-13T13:14:33.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:33.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.038150+0000) 2019-10-13T13:14:33.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:33.033+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.038742+0000) 2019-10-13T13:14:33.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:33.037+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.039545+0000) 2019-10-13T13:14:33.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:33.037+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.040186+0000) 2019-10-13T13:14:33.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:33.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.040721+0000) 2019-10-13T13:14:33.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:33.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.041895+0000) 2019-10-13T13:14:33.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:33.037+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.041904+0000) 2019-10-13T13:14:33.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:33.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:33.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:33.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:33.048393+0000) 2019-10-13T13:14:35.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:35.565+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:35.566186+0000) 2019-10-13T13:14:35.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:35.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:35.852348+0000) 2019-10-13T13:14:36.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:36.528+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:36.533893+0000) 2019-10-13T13:14:38.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:38.044+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.049838+0000) 2019-10-13T13:14:38.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:38.044+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.050661+0000) 2019-10-13T13:14:38.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:38.048+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.051733+0000) 2019-10-13T13:14:38.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:38.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.052341+0000) 2019-10-13T13:14:38.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:38.048+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.052415+0000) 2019-10-13T13:14:38.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:38.048+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.053104+0000) 2019-10-13T13:14:38.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:38.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.053797+0000) 2019-10-13T13:14:38.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:38.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167232 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:38.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:38.056+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.059879+0000) 2019-10-13T13:14:38.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:38.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.800681+0000) 2019-10-13T13:14:38.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:38.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.802679+0000) 2019-10-13T13:14:39.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:39.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.042055+0000) 2019-10-13T13:14:39.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:14:39.638 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:39.642 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:14:39.642 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:40.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:40.028+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:40.034084+0000) 2019-10-13T13:14:42.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:42.396+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:42.398591+0000) 2019-10-13T13:14:43.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:43.056+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.062464+0000) 2019-10-13T13:14:43.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:43.060+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.063338+0000) 2019-10-13T13:14:43.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:43.060+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.064406+0000) 2019-10-13T13:14:43.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:43.060+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.064545+0000) 2019-10-13T13:14:43.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:43.060+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.064885+0000) 2019-10-13T13:14:43.067 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:43.060+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.065800+0000) 2019-10-13T13:14:43.068 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:43.060+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.066384+0000) 2019-10-13T13:14:43.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:43.068+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:43.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:43.068+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:43.074025+0000) 2019-10-13T13:14:45.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:45.560+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:45.566384+0000) 2019-10-13T13:14:45.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:45.848+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:45.852573+0000) 2019-10-13T13:14:46.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:46.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:46.534087+0000) 2019-10-13T13:14:48.077 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:48.068+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.074766+0000) 2019-10-13T13:14:48.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:48.068+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.075172+0000) 2019-10-13T13:14:48.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:48.072+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.077383+0000) 2019-10-13T13:14:48.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:48.072+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.077819+0000) 2019-10-13T13:14:48.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:48.076+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.078614+0000) 2019-10-13T13:14:48.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:48.072+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.078763+0000) 2019-10-13T13:14:48.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:48.076+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.079369+0000) 2019-10-13T13:14:48.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:48.080+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:48.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:48.080+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.084813+0000) 2019-10-13T13:14:48.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:48.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.800864+0000) 2019-10-13T13:14:48.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:48.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.802859+0000) 2019-10-13T13:14:49.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:49.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.042241+0000) 2019-10-13T13:14:50.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:50.032+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:50.034312+0000) 2019-10-13T13:14:52.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:52.400+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:52.405245+0000) 2019-10-13T13:14:53.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:53.084+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.087714+0000) 2019-10-13T13:14:53.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:53.084+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.088194+0000) 2019-10-13T13:14:53.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:53.084+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.088838+0000) 2019-10-13T13:14:53.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:53.084+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.089358+0000) 2019-10-13T13:14:53.091 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:53.084+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.089558+0000) 2019-10-13T13:14:53.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:53.084+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.089925+0000) 2019-10-13T13:14:53.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:53.084+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.090402+0000) 2019-10-13T13:14:53.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:53.088+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:53.093606+0000) 2019-10-13T13:14:53.097 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:53.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:55.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:55.564+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:55.566594+0000) 2019-10-13T13:14:55.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:55.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:55.852773+0000) 2019-10-13T13:14:56.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:56.528+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:56.534276+0000) 2019-10-13T13:14:58.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:58.092+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.098510+0000) 2019-10-13T13:14:58.101 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:58.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.098926+0000) 2019-10-13T13:14:58.101 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:14:58.096+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.099868+0000) 2019-10-13T13:14:58.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:14:58.096+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.100593+0000) 2019-10-13T13:14:58.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:14:58.095+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.100739+0000) 2019-10-13T13:14:58.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:14:58.096+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.100978+0000) 2019-10-13T13:14:58.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:58.096+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.101293+0000) 2019-10-13T13:14:58.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:14:58.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:14:58.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:14:58.103+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.108648+0000) 2019-10-13T13:14:58.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:14:58.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.801099+0000) 2019-10-13T13:14:58.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:14:58.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.803032+0000) 2019-10-13T13:14:59.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:14:59.036+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.042416+0000) 2019-10-13T13:15:00.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:00.031+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:00.034481+0000) 2019-10-13T13:15:02.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:02.407+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:02.411691+0000) 2019-10-13T13:15:03.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:03.104+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.111079+0000) 2019-10-13T13:15:03.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:03.107+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.111796+0000) 2019-10-13T13:15:03.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:03.108+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.111945+0000) 2019-10-13T13:15:03.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:03.108+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.112183+0000) 2019-10-13T13:15:03.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:03.108+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.112886+0000) 2019-10-13T13:15:03.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:03.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.113310+0000) 2019-10-13T13:15:03.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:03.108+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.114379+0000) 2019-10-13T13:15:03.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:03.112+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:03.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:03.115+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:03.119850+0000) 2019-10-13T13:15:05.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:05.560+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:05.566787+0000) 2019-10-13T13:15:05.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:05.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:05.852978+0000) 2019-10-13T13:15:06.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:06.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:06.534470+0000) 2019-10-13T13:15:08.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:08.116+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.121315+0000) 2019-10-13T13:15:08.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:08.119+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.122763+0000) 2019-10-13T13:15:08.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:08.116+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.122862+0000) 2019-10-13T13:15:08.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:08.120+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.123598+0000) 2019-10-13T13:15:08.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:08.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.124117+0000) 2019-10-13T13:15:08.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:08.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.124596+0000) 2019-10-13T13:15:08.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:08.120+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.124933+0000) 2019-10-13T13:15:08.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:08.127+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.130648+0000) 2019-10-13T13:15:08.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:08.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:08.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:08.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.801284+0000) 2019-10-13T13:15:08.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:08.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.803226+0000) 2019-10-13T13:15:09.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:09.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.042594+0000) 2019-10-13T13:15:09.716 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:15:09.717 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:09.720 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:15:09.721 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:10.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:10.031+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:10.034722+0000) 2019-10-13T13:15:12.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:12.419+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:12.422790+0000) 2019-10-13T13:15:13.136 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:13.128+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.134102+0000) 2019-10-13T13:15:13.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:13.131+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.134990+0000) 2019-10-13T13:15:13.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:13.132+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.135492+0000) 2019-10-13T13:15:13.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:13.132+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.135795+0000) 2019-10-13T13:15:13.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:13.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.136265+0000) 2019-10-13T13:15:13.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:13.132+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.136554+0000) 2019-10-13T13:15:13.139 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:13.132+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.137129+0000) 2019-10-13T13:15:13.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:13.135+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:13.141521+0000) 2019-10-13T13:15:13.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:13.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:15.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:15.564+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:15.566985+0000) 2019-10-13T13:15:15.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:15.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:15.853210+0000) 2019-10-13T13:15:16.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:16.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:16.534668+0000) 2019-10-13T13:15:18.147 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:18.140+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.145253+0000) 2019-10-13T13:15:18.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:18.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.145884+0000) 2019-10-13T13:15:18.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:18.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.146520+0000) 2019-10-13T13:15:18.149 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:18.140+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.146569+0000) 2019-10-13T13:15:18.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:18.144+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.147557+0000) 2019-10-13T13:15:18.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:18.144+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.147876+0000) 2019-10-13T13:15:18.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:18.144+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.148881+0000) 2019-10-13T13:15:18.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:18.147+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.153682+0000) 2019-10-13T13:15:18.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:18.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:18.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:18.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.801490+0000) 2019-10-13T13:15:18.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:18.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.803417+0000) 2019-10-13T13:15:19.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:19.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.042771+0000) 2019-10-13T13:15:20.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:20.031+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:20.034952+0000) 2019-10-13T13:15:22.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:22.423+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:22.429329+0000) 2019-10-13T13:15:23.160 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:23.151+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.158292+0000) 2019-10-13T13:15:23.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:23.151+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.159126+0000) 2019-10-13T13:15:23.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:23.155+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.159755+0000) 2019-10-13T13:15:23.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:23.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.160215+0000) 2019-10-13T13:15:23.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:23.155+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.160468+0000) 2019-10-13T13:15:23.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:23.155+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.160784+0000) 2019-10-13T13:15:23.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:23.155+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.161396+0000) 2019-10-13T13:15:23.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:23.159+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:23.165240+0000) 2019-10-13T13:15:23.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:23.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167538 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:25.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:25.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:25.567198+0000) 2019-10-13T13:15:25.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:25.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:25.853412+0000) 2019-10-13T13:15:26.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:26.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:26.534865+0000) 2019-10-13T13:15:28.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:28.167+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.170611+0000) 2019-10-13T13:15:28.173 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:28.167+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.171585+0000) 2019-10-13T13:15:28.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:28.167+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.171965+0000) 2019-10-13T13:15:28.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:28.167+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.172572+0000) 2019-10-13T13:15:28.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:28.167+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.172974+0000) 2019-10-13T13:15:28.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:28.167+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.172772+0000) 2019-10-13T13:15:28.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:28.167+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.173651+0000) 2019-10-13T13:15:28.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:28.171+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.177448+0000) 2019-10-13T13:15:28.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:28.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:28.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:28.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.801664+0000) 2019-10-13T13:15:28.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:28.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.803597+0000) 2019-10-13T13:15:29.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:29.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.043008+0000) 2019-10-13T13:15:30.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:30.031+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:30.035165+0000) 2019-10-13T13:15:32.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:32.435+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:32.435831+0000) 2019-10-13T13:15:33.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:33.179+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.183088+0000) 2019-10-13T13:15:33.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:33.179+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.183627+0000) 2019-10-13T13:15:33.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:33.179+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.184458+0000) 2019-10-13T13:15:33.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:33.179+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.184910+0000) 2019-10-13T13:15:33.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:33.179+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.185567+0000) 2019-10-13T13:15:33.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:33.179+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.185761+0000) 2019-10-13T13:15:33.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:33.179+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.186243+0000) 2019-10-13T13:15:33.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:33.187+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:33.190418+0000) 2019-10-13T13:15:33.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:33.187+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:35.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:35.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:35.567394+0000) 2019-10-13T13:15:35.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:35.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:35.853604+0000) 2019-10-13T13:15:36.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:36.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:36.535087+0000) 2019-10-13T13:15:38.197 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:38.191+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.195228+0000) 2019-10-13T13:15:38.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:38.191+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.196076+0000) 2019-10-13T13:15:38.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:38.191+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.196583+0000) 2019-10-13T13:15:38.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:38.191+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.196820+0000) 2019-10-13T13:15:38.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:38.191+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.197446+0000) 2019-10-13T13:15:38.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:38.191+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.197541+0000) 2019-10-13T13:15:38.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:38.191+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.198560+0000) 2019-10-13T13:15:38.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:38.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:38.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:38.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.205570+0000) 2019-10-13T13:15:38.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:38.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.801837+0000) 2019-10-13T13:15:38.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:38.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.803766+0000) 2019-10-13T13:15:39.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:39.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.043181+0000) 2019-10-13T13:15:39.871 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:15:39.871 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:39.875 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:15:39.875 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:40.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:40.031+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:40.035402+0000) 2019-10-13T13:15:42.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:42.439+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:42.442357+0000) 2019-10-13T13:15:43.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:43.203+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.206948+0000) 2019-10-13T13:15:43.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:43.203+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.208119+0000) 2019-10-13T13:15:43.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:43.203+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.208342+0000) 2019-10-13T13:15:43.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:43.203+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.208980+0000) 2019-10-13T13:15:43.211 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:43.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.209607+0000) 2019-10-13T13:15:43.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:43.203+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.209694+0000) 2019-10-13T13:15:43.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:43.203+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.210086+0000) 2019-10-13T13:15:43.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:43.211+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:43.215079+0000) 2019-10-13T13:15:43.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:43.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:45.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:45.567+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:45.567595+0000) 2019-10-13T13:15:45.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:45.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:45.853802+0000) 2019-10-13T13:15:46.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:46.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:46.535282+0000) 2019-10-13T13:15:48.222 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:48.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.220524+0000) 2019-10-13T13:15:48.223 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:48.215+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.221458+0000) 2019-10-13T13:15:48.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:48.215+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.222034+0000) 2019-10-13T13:15:48.224 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:48.215+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.222413+0000) 2019-10-13T13:15:48.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:48.215+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.222167+0000) 2019-10-13T13:15:48.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:48.215+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.222762+0000) 2019-10-13T13:15:48.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:48.215+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.223168+0000) 2019-10-13T13:15:48.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:48.223+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.226979+0000) 2019-10-13T13:15:48.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:48.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:48.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:48.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.802016+0000) 2019-10-13T13:15:48.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:48.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.803941+0000) 2019-10-13T13:15:49.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:49.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.043387+0000) 2019-10-13T13:15:50.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:50.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:50.035601+0000) 2019-10-13T13:15:52.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:52.446+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:52.448434+0000) 2019-10-13T13:15:53.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:53.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.232213+0000) 2019-10-13T13:15:53.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:53.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.232895+0000) 2019-10-13T13:15:53.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:53.231+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.232995+0000) 2019-10-13T13:15:53.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:53.231+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.233717+0000) 2019-10-13T13:15:53.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:53.231+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.234102+0000) 2019-10-13T13:15:53.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:53.230+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.235666+0000) 2019-10-13T13:15:53.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:53.235+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.236043+0000) 2019-10-13T13:15:53.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:53.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:53.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:53.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:53.241391+0000) 2019-10-13T13:15:55.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:55.567+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:55.567811+0000) 2019-10-13T13:15:55.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:55.851+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:55.854012+0000) 2019-10-13T13:15:56.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:56.535+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:56.535530+0000) 2019-10-13T13:15:58.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:15:58.238+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.243529+0000) 2019-10-13T13:15:58.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:15:58.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.244391+0000) 2019-10-13T13:15:58.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:58.243+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.244643+0000) 2019-10-13T13:15:58.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:15:58.243+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.245027+0000) 2019-10-13T13:15:58.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:58.243+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.245389+0000) 2019-10-13T13:15:58.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:58.243+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.246189+0000) 2019-10-13T13:15:58.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:15:58.243+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.246696+0000) 2019-10-13T13:15:58.254 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:15:58.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:15:58.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:15:58.250+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.255888+0000) 2019-10-13T13:15:58.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:15:58.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.802193+0000) 2019-10-13T13:15:58.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:15:58.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.804117+0000) 2019-10-13T13:15:59.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:15:59.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.043560+0000) 2019-10-13T13:16:00.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:00.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:00.035798+0000) 2019-10-13T13:16:02.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:02.450+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:02.454734+0000) 2019-10-13T13:16:03.258 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:03.254+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.255916+0000) 2019-10-13T13:16:03.259 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:03.250+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.256344+0000) 2019-10-13T13:16:03.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:03.254+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.256544+0000) 2019-10-13T13:16:03.259 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:03.254+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.256930+0000) 2019-10-13T13:16:03.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:03.254+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.257604+0000) 2019-10-13T13:16:03.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:03.254+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.258551+0000) 2019-10-13T13:16:03.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:03.254+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.258728+0000) 2019-10-13T13:16:03.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:03.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:03.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:03.258+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:03.264632+0000) 2019-10-13T13:16:05.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:05.566+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:05.568054+0000) 2019-10-13T13:16:05.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:05.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:05.854278+0000) 2019-10-13T13:16:06.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:06.534+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:06.535752+0000) 2019-10-13T13:16:08.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:08.266+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.267341+0000) 2019-10-13T13:16:08.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:08.266+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.268023+0000) 2019-10-13T13:16:08.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:08.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.268900+0000) 2019-10-13T13:16:08.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:08.266+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.269009+0000) 2019-10-13T13:16:08.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:08.266+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.269578+0000) 2019-10-13T13:16:08.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:08.266+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.269855+0000) 2019-10-13T13:16:08.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:08.266+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.270158+0000) 2019-10-13T13:16:08.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:08.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:08.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:08.270+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.277171+0000) 2019-10-13T13:16:08.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:08.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.802370+0000) 2019-10-13T13:16:08.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:08.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.804323+0000) 2019-10-13T13:16:09.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:09.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.043739+0000) 2019-10-13T13:16:09.989 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:16:09.989 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:09.993 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:16:09.993 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:10.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:10.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:10.036028+0000) 2019-10-13T13:16:12.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:12.458+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:12.461109+0000) 2019-10-13T13:16:13.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:13.278+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.280015+0000) 2019-10-13T13:16:13.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:13.278+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.280309+0000) 2019-10-13T13:16:13.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:13.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.280686+0000) 2019-10-13T13:16:13.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:13.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.281543+0000) 2019-10-13T13:16:13.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:13.278+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.282560+0000) 2019-10-13T13:16:13.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:13.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.282392+0000) 2019-10-13T13:16:13.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:13.282+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.284197+0000) 2019-10-13T13:16:13.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:13.282+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:13.288561+0000) 2019-10-13T13:16:13.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:13.286+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:15.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:15.566+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:15.568312+0000) 2019-10-13T13:16:15.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:15.850+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:15.854474+0000) 2019-10-13T13:16:16.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:16.534+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:16.535974+0000) 2019-10-13T13:16:18.293 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:18.290+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.291673+0000) 2019-10-13T13:16:18.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:18.290+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.292303+0000) 2019-10-13T13:16:18.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:18.290+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.293794+0000) 2019-10-13T13:16:18.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:18.290+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.294486+0000) 2019-10-13T13:16:18.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:18.290+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.294867+0000) 2019-10-13T13:16:18.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:18.290+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.295274+0000) 2019-10-13T13:16:18.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:18.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.295792+0000) 2019-10-13T13:16:18.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:18.294+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.300869+0000) 2019-10-13T13:16:18.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:18.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:18.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:18.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.802580+0000) 2019-10-13T13:16:18.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:18.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.804504+0000) 2019-10-13T13:16:19.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:19.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:19.043905+0000) 2019-10-13T13:16:20.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:20.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.036249+0000) 2019-10-13T13:16:22.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:22.466+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:22.467975+0000) 2019-10-13T13:16:23.306 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:23.302+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.304358+0000) 2019-10-13T13:16:23.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:23.302+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.304115+0000) 2019-10-13T13:16:23.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:23.302+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.304827+0000) 2019-10-13T13:16:23.307 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:23.302+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.305792+0000) 2019-10-13T13:16:23.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:23.302+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.306407+0000) 2019-10-13T13:16:23.308 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:23.302+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.306367+0000) 2019-10-13T13:16:23.308 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:23.302+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.306776+0000) 2019-10-13T13:16:23.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:23.306+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:23.311234+0000) 2019-10-13T13:16:23.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:23.310+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:25.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:25.566+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.568515+0000) 2019-10-13T13:16:25.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:25.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.854683+0000) 2019-10-13T13:16:26.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:26.534+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:26.536194+0000) 2019-10-13T13:16:28.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:28.314+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.315931+0000) 2019-10-13T13:16:28.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:28.310+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.316517+0000) 2019-10-13T13:16:28.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:28.314+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.316799+0000) 2019-10-13T13:16:28.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:28.314+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.317427+0000) 2019-10-13T13:16:28.319 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:28.314+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.317812+0000) 2019-10-13T13:16:28.320 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:28.314+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.318123+0000) 2019-10-13T13:16:28.321 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:28.318+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.319442+0000) 2019-10-13T13:16:28.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:28.322+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 167972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:28.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:28.318+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.325935+0000) 2019-10-13T13:16:28.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:28.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.802761+0000) 2019-10-13T13:16:28.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:28.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.804704+0000) 2019-10-13T13:16:29.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:29.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:29.044107+0000) 2019-10-13T13:16:30.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:30.030+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.036476+0000) 2019-10-13T13:16:32.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:32.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:32.473660+0000) 2019-10-13T13:16:33.329 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:33.322+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.326834+0000) 2019-10-13T13:16:33.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:33.326+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.327621+0000) 2019-10-13T13:16:33.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:33.326+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.328250+0000) 2019-10-13T13:16:33.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:33.326+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.328982+0000) 2019-10-13T13:16:33.331 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:33.326+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.330005+0000) 2019-10-13T13:16:33.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:33.326+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.330989+0000) 2019-10-13T13:16:33.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:33.326+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.331074+0000) 2019-10-13T13:16:33.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:33.334+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:33.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:33.334+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:33.339527+0000) 2019-10-13T13:16:35.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:35.566+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.568718+0000) 2019-10-13T13:16:35.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:35.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.854904+0000) 2019-10-13T13:16:36.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:36.534+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:36.536384+0000) 2019-10-13T13:16:38.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:38.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.339218+0000) 2019-10-13T13:16:38.341 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:38.338+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.339656+0000) 2019-10-13T13:16:38.342 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:38.338+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.339927+0000) 2019-10-13T13:16:38.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:38.338+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.340256+0000) 2019-10-13T13:16:38.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:38.338+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.341332+0000) 2019-10-13T13:16:38.343 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:38.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.341628+0000) 2019-10-13T13:16:38.344 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:38.338+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.341641+0000) 2019-10-13T13:16:38.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:38.346+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.346830+0000) 2019-10-13T13:16:38.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:38.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:38.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:38.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.802959+0000) 2019-10-13T13:16:38.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:38.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.804905+0000) 2019-10-13T13:16:39.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:39.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:39.044288+0000) 2019-10-13T13:16:40.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:40.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.036693+0000) 2019-10-13T13:16:40.075 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:16:40.075 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:40.078 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:16:40.079 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:42.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:42.478+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:42.480128+0000) 2019-10-13T13:16:43.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:43.346+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.350400+0000) 2019-10-13T13:16:43.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:43.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.351246+0000) 2019-10-13T13:16:43.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:43.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.351829+0000) 2019-10-13T13:16:43.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:43.349+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.352703+0000) 2019-10-13T13:16:43.354 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:43.350+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.352693+0000) 2019-10-13T13:16:43.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:43.350+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.353304+0000) 2019-10-13T13:16:43.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:43.350+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.354427+0000) 2019-10-13T13:16:43.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:43.357+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:43.358950+0000) 2019-10-13T13:16:43.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:43.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:45.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:45.565+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.568941+0000) 2019-10-13T13:16:45.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:45.849+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.855124+0000) 2019-10-13T13:16:46.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:46.533+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:46.536575+0000) 2019-10-13T13:16:48.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:48.357+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.362461+0000) 2019-10-13T13:16:48.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:48.357+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.363068+0000) 2019-10-13T13:16:48.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:48.361+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.363386+0000) 2019-10-13T13:16:48.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:48.361+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.364384+0000) 2019-10-13T13:16:48.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:48.361+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.364929+0000) 2019-10-13T13:16:48.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:48.361+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.365402+0000) 2019-10-13T13:16:48.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:48.361+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.366044+0000) 2019-10-13T13:16:48.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:48.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:48.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:48.369+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.373811+0000) 2019-10-13T13:16:48.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:48.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.803169+0000) 2019-10-13T13:16:48.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:48.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.805082+0000) 2019-10-13T13:16:49.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:49.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:49.044469+0000) 2019-10-13T13:16:50.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:50.033+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.036908+0000) 2019-10-13T13:16:52.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:52.485+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:52.486739+0000) 2019-10-13T13:16:53.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:53.369+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.374323+0000) 2019-10-13T13:16:53.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:53.369+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.374657+0000) 2019-10-13T13:16:53.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:53.373+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.376332+0000) 2019-10-13T13:16:53.378 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:53.373+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.376732+0000) 2019-10-13T13:16:53.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:53.373+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.377228+0000) 2019-10-13T13:16:53.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:53.373+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.377852+0000) 2019-10-13T13:16:53.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:53.373+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.377947+0000) 2019-10-13T13:16:53.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:53.381+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:53.382805+0000) 2019-10-13T13:16:53.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:53.377+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:55.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:55.565+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.569152+0000) 2019-10-13T13:16:55.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:55.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.855360+0000) 2019-10-13T13:16:56.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:56.533+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:56.536820+0000) 2019-10-13T13:16:58.388 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:58.381+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.386016+0000) 2019-10-13T13:16:58.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:16:58.381+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.386725+0000) 2019-10-13T13:16:58.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:58.381+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.386894+0000) 2019-10-13T13:16:58.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:58.381+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.387204+0000) 2019-10-13T13:16:58.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:16:58.385+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.387669+0000) 2019-10-13T13:16:58.391 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:16:58.385+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.389779+0000) 2019-10-13T13:16:58.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:16:58.385+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.390073+0000) 2019-10-13T13:16:58.396 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:16:58.389+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:16:58.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:16:58.393+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.395299+0000) 2019-10-13T13:16:58.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:16:58.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.803346+0000) 2019-10-13T13:16:58.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:16:58.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.805271+0000) 2019-10-13T13:16:59.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:16:59.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:59.044646+0000) 2019-10-13T13:17:00.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:00.033+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.037124+0000) 2019-10-13T13:17:02.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:02.489+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:02.493234+0000) 2019-10-13T13:17:03.400 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:03.393+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.397188+0000) 2019-10-13T13:17:03.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:03.393+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.397747+0000) 2019-10-13T13:17:03.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:03.393+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.398537+0000) 2019-10-13T13:17:03.401 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:03.393+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.399076+0000) 2019-10-13T13:17:03.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:03.397+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.399376+0000) 2019-10-13T13:17:03.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:03.397+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.399820+0000) 2019-10-13T13:17:03.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:03.397+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.400738+0000) 2019-10-13T13:17:03.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:03.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:03.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:03.405+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:03.406660+0000) 2019-10-13T13:17:05.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:05.565+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.569366+0000) 2019-10-13T13:17:05.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:05.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.855594+0000) 2019-10-13T13:17:06.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:06.533+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:06.537014+0000) 2019-10-13T13:17:08.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:08.405+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.408187+0000) 2019-10-13T13:17:08.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:08.409+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.410310+0000) 2019-10-13T13:17:08.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:08.405+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.410509+0000) 2019-10-13T13:17:08.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:08.405+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.410904+0000) 2019-10-13T13:17:08.413 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:08.409+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.411677+0000) 2019-10-13T13:17:08.414 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:08.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.411518+0000) 2019-10-13T13:17:08.414 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:08.409+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.412710+0000) 2019-10-13T13:17:08.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:08.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168236 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:08.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:08.417+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.419600+0000) 2019-10-13T13:17:08.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:08.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.803529+0000) 2019-10-13T13:17:08.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:08.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.805446+0000) 2019-10-13T13:17:09.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:09.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:09.044837+0000) 2019-10-13T13:17:10.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:10.033+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.037345+0000) 2019-10-13T13:17:10.187 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:17:10.187 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:10.191 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:17:10.191 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:12.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:12.497+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:12.499621+0000) 2019-10-13T13:17:13.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:13.417+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.421152+0000) 2019-10-13T13:17:13.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:13.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.421532+0000) 2019-10-13T13:17:13.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:13.417+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.422374+0000) 2019-10-13T13:17:13.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:13.417+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.422724+0000) 2019-10-13T13:17:13.425 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:13.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.423139+0000) 2019-10-13T13:17:13.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:13.421+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.423684+0000) 2019-10-13T13:17:13.426 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:13.421+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.424304+0000) 2019-10-13T13:17:13.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:13.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:13.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:13.429+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:13.433122+0000) 2019-10-13T13:17:15.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:15.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.569567+0000) 2019-10-13T13:17:15.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:15.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.855794+0000) 2019-10-13T13:17:16.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:16.533+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:16.537211+0000) 2019-10-13T13:17:18.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:18.429+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.433137+0000) 2019-10-13T13:17:18.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:18.429+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.433423+0000) 2019-10-13T13:17:18.436 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:18.429+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.434051+0000) 2019-10-13T13:17:18.436 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:18.429+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.434397+0000) 2019-10-13T13:17:18.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:18.433+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.435207+0000) 2019-10-13T13:17:18.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:18.433+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.435292+0000) 2019-10-13T13:17:18.438 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:18.433+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.435961+0000) 2019-10-13T13:17:18.443 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:18.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:18.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:18.441+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.442938+0000) 2019-10-13T13:17:18.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:18.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.803706+0000) 2019-10-13T13:17:18.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:18.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.805619+0000) 2019-10-13T13:17:19.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:19.041+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:19.045017+0000) 2019-10-13T13:17:20.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:20.033+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.037544+0000) 2019-10-13T13:17:22.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:22.505+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:22.506023+0000) 2019-10-13T13:17:23.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:23.445+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.447238+0000) 2019-10-13T13:17:23.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:23.441+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.445100+0000) 2019-10-13T13:17:23.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:23.441+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.445899+0000) 2019-10-13T13:17:23.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:23.441+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.446208+0000) 2019-10-13T13:17:23.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:23.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.446688+0000) 2019-10-13T13:17:23.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:23.441+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.447283+0000) 2019-10-13T13:17:23.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:23.445+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.447611+0000) 2019-10-13T13:17:23.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:23.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168336 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:23.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:23.453+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:23.455224+0000) 2019-10-13T13:17:25.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:25.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.569802+0000) 2019-10-13T13:17:25.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:25.853+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.856015+0000) 2019-10-13T13:17:26.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:26.537+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:26.537407+0000) 2019-10-13T13:17:28.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:28.452+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.455977+0000) 2019-10-13T13:17:28.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:28.452+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.456516+0000) 2019-10-13T13:17:28.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:28.452+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.456942+0000) 2019-10-13T13:17:28.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:28.452+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.457581+0000) 2019-10-13T13:17:28.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:28.452+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.458249+0000) 2019-10-13T13:17:28.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:28.456+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.459401+0000) 2019-10-13T13:17:28.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:28.457+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.459737+0000) 2019-10-13T13:17:28.466 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:28.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168369 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:28.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:28.461+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.465586+0000) 2019-10-13T13:17:28.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:28.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.803902+0000) 2019-10-13T13:17:28.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:28.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.805813+0000) 2019-10-13T13:17:29.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:29.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:29.045190+0000) 2019-10-13T13:17:30.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:30.033+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.037762+0000) 2019-10-13T13:17:32.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:32.509+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:32.512385+0000) 2019-10-13T13:17:33.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:33.464+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.467407+0000) 2019-10-13T13:17:33.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:33.464+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.469324+0000) 2019-10-13T13:17:33.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:33.464+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.469671+0000) 2019-10-13T13:17:33.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:33.464+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.470001+0000) 2019-10-13T13:17:33.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:33.469+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.470896+0000) 2019-10-13T13:17:33.473 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:33.464+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.470272+0000) 2019-10-13T13:17:33.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:33.464+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.471021+0000) 2019-10-13T13:17:33.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:33.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168403 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:33.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:33.473+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:33.477832+0000) 2019-10-13T13:17:35.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:35.568+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.570007+0000) 2019-10-13T13:17:35.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:35.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.856255+0000) 2019-10-13T13:17:36.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:36.536+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:36.537648+0000) 2019-10-13T13:17:38.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:38.476+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.480588+0000) 2019-10-13T13:17:38.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:38.476+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.481118+0000) 2019-10-13T13:17:38.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:38.476+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.481602+0000) 2019-10-13T13:17:38.484 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:38.480+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.482620+0000) 2019-10-13T13:17:38.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:38.476+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.482278+0000) 2019-10-13T13:17:38.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:38.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.482604+0000) 2019-10-13T13:17:38.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:38.480+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.483615+0000) 2019-10-13T13:17:38.491 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:38.484+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:38.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:38.484+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.489592+0000) 2019-10-13T13:17:38.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:38.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.804084+0000) 2019-10-13T13:17:38.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:38.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.806009+0000) 2019-10-13T13:17:39.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:39.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:39.045370+0000) 2019-10-13T13:17:40.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:40.032+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.037977+0000) 2019-10-13T13:17:40.247 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:17:40.247 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:40.250 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:17:40.251 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:42.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:42.516+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:42.518820+0000) 2019-10-13T13:17:43.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:43.488+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.492775+0000) 2019-10-13T13:17:43.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:43.488+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.493315+0000) 2019-10-13T13:17:43.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:43.488+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.493516+0000) 2019-10-13T13:17:43.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:43.488+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.493855+0000) 2019-10-13T13:17:43.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:43.488+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.494489+0000) 2019-10-13T13:17:43.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:43.492+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.495318+0000) 2019-10-13T13:17:43.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:43.492+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.496202+0000) 2019-10-13T13:17:43.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:43.496+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168468 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:43.503 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:43.496+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:43.501595+0000) 2019-10-13T13:17:45.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:45.568+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.570204+0000) 2019-10-13T13:17:45.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:45.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.856455+0000) 2019-10-13T13:17:46.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:46.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:46.537859+0000) 2019-10-13T13:17:48.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:48.500+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.504085+0000) 2019-10-13T13:17:48.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:48.500+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.505247+0000) 2019-10-13T13:17:48.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:48.500+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.504560+0000) 2019-10-13T13:17:48.508 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:48.500+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.504874+0000) 2019-10-13T13:17:48.508 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:48.500+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.505755+0000) 2019-10-13T13:17:48.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:48.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.506645+0000) 2019-10-13T13:17:48.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:48.504+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.507683+0000) 2019-10-13T13:17:48.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:48.508+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.510335+0000) 2019-10-13T13:17:48.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:48.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:48.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:48.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.804263+0000) 2019-10-13T13:17:48.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:48.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.806209+0000) 2019-10-13T13:17:49.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:49.040+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:49.045561+0000) 2019-10-13T13:17:50.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:50.036+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.038200+0000) 2019-10-13T13:17:52.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:52.520+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:52.525140+0000) 2019-10-13T13:17:53.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:53.512+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.515859+0000) 2019-10-13T13:17:53.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:53.512+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.516877+0000) 2019-10-13T13:17:53.519 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:53.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.517322+0000) 2019-10-13T13:17:53.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:53.512+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.517904+0000) 2019-10-13T13:17:53.520 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:53.512+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.518846+0000) 2019-10-13T13:17:53.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:53.516+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.519353+0000) 2019-10-13T13:17:53.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:53.516+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.519671+0000) 2019-10-13T13:17:53.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:53.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:53.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:53.520+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:53.525522+0000) 2019-10-13T13:17:55.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:55.564+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.570403+0000) 2019-10-13T13:17:55.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:55.852+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.856652+0000) 2019-10-13T13:17:56.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:56.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:56.538002+0000) 2019-10-13T13:17:58.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:17:58.524+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.527807+0000) 2019-10-13T13:17:58.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:58.524+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.528691+0000) 2019-10-13T13:17:58.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:17:58.524+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.529785+0000) 2019-10-13T13:17:58.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:17:58.524+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.530248+0000) 2019-10-13T13:17:58.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:58.524+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.531036+0000) 2019-10-13T13:17:58.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:58.528+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.531599+0000) 2019-10-13T13:17:58.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:17:58.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.532536+0000) 2019-10-13T13:17:58.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:17:58.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:17:58.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:17:58.532+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.537527+0000) 2019-10-13T13:17:58.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:17:58.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.804475+0000) 2019-10-13T13:17:58.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:17:58.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.806414+0000) 2019-10-13T13:17:59.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:17:59.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:59.045738+0000) 2019-10-13T13:18:00.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:00.036+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.038382+0000) 2019-10-13T13:18:02.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:02.528+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:02.531696+0000) 2019-10-13T13:18:03.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:03.536+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.541303+0000) 2019-10-13T13:18:03.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:03.536+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.541751+0000) 2019-10-13T13:18:03.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:03.536+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.542374+0000) 2019-10-13T13:18:03.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:03.536+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.542120+0000) 2019-10-13T13:18:03.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:03.536+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.542822+0000) 2019-10-13T13:18:03.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:03.540+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.543332+0000) 2019-10-13T13:18:03.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:03.540+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.543912+0000) 2019-10-13T13:18:03.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:03.544+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:03.547007+0000) 2019-10-13T13:18:03.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:03.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:05.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:05.564+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.570619+0000) 2019-10-13T13:18:05.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:05.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.856853+0000) 2019-10-13T13:18:06.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:06.532+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:06.538195+0000) 2019-10-13T13:18:08.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:08.548+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.551793+0000) 2019-10-13T13:18:08.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:08.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.552729+0000) 2019-10-13T13:18:08.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:08.548+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.553532+0000) 2019-10-13T13:18:08.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:08.548+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.553830+0000) 2019-10-13T13:18:08.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:08.548+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.554596+0000) 2019-10-13T13:18:08.557 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:08.548+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.554639+0000) 2019-10-13T13:18:08.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:08.552+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.555908+0000) 2019-10-13T13:18:08.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:08.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:08.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:08.560+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.563615+0000) 2019-10-13T13:18:08.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:08.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.804657+0000) 2019-10-13T13:18:08.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:08.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.806588+0000) 2019-10-13T13:18:09.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:09.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:09.045918+0000) 2019-10-13T13:18:10.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:10.036+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.038590+0000) 2019-10-13T13:18:10.307 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:18:10.308 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:10.311 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:18:10.311 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:12.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:12.536+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:12.538133+0000) 2019-10-13T13:18:13.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:13.555+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.562928+0000) 2019-10-13T13:18:13.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:13.559+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.563933+0000) 2019-10-13T13:18:13.566 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:13.560+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.564854+0000) 2019-10-13T13:18:13.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:13.559+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.564864+0000) 2019-10-13T13:18:13.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:13.559+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.565247+0000) 2019-10-13T13:18:13.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:13.559+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.565773+0000) 2019-10-13T13:18:13.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:13.559+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.566535+0000) 2019-10-13T13:18:13.574 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:13.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:13.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:13.568+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:13.572684+0000) 2019-10-13T13:18:15.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:15.567+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.570815+0000) 2019-10-13T13:18:15.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:15.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.857014+0000) 2019-10-13T13:18:16.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:16.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:16.538392+0000) 2019-10-13T13:18:18.577 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:18.567+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.575196+0000) 2019-10-13T13:18:18.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:18.571+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.575651+0000) 2019-10-13T13:18:18.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:18.571+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.575914+0000) 2019-10-13T13:18:18.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:18.571+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.576185+0000) 2019-10-13T13:18:18.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:18.571+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.577302+0000) 2019-10-13T13:18:18.579 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:18.572+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.577951+0000) 2019-10-13T13:18:18.580 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:18.571+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.578503+0000) 2019-10-13T13:18:18.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:18.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:18.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:18.580+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.584673+0000) 2019-10-13T13:18:18.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:18.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.804855+0000) 2019-10-13T13:18:18.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:18.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.806762+0000) 2019-10-13T13:18:19.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:19.039+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:19.046095+0000) 2019-10-13T13:18:20.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:20.036+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.038805+0000) 2019-10-13T13:18:22.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:22.544+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:22.544603+0000) 2019-10-13T13:18:23.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:23.583+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.587459+0000) 2019-10-13T13:18:23.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:23.583+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.587821+0000) 2019-10-13T13:18:23.590 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:23.583+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.588342+0000) 2019-10-13T13:18:23.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:23.583+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.588774+0000) 2019-10-13T13:18:23.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:23.583+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.589352+0000) 2019-10-13T13:18:23.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:23.583+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.590030+0000) 2019-10-13T13:18:23.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:23.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.591162+0000) 2019-10-13T13:18:23.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:23.592+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:23.596721+0000) 2019-10-13T13:18:23.598 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:23.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:25.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:25.563+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.571008+0000) 2019-10-13T13:18:25.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:25.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.857244+0000) 2019-10-13T13:18:26.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:26.531+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:26.538586+0000) 2019-10-13T13:18:28.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:28.596+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.599868+0000) 2019-10-13T13:18:28.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:28.595+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.600036+0000) 2019-10-13T13:18:28.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:28.595+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.600488+0000) 2019-10-13T13:18:28.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:28.595+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.600860+0000) 2019-10-13T13:18:28.603 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:28.595+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.601616+0000) 2019-10-13T13:18:28.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:28.595+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.602175+0000) 2019-10-13T13:18:28.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:28.595+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.602798+0000) 2019-10-13T13:18:28.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:28.603+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:28.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:28.608+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.610231+0000) 2019-10-13T13:18:28.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:28.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.805055+0000) 2019-10-13T13:18:28.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:28.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.806942+0000) 2019-10-13T13:18:29.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:29.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:29.046271+0000) 2019-10-13T13:18:30.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:30.036+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.039017+0000) 2019-10-13T13:18:32.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:32.547+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:32.551155+0000) 2019-10-13T13:18:33.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:33.607+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.612371+0000) 2019-10-13T13:18:33.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:33.607+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.612904+0000) 2019-10-13T13:18:33.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:33.607+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.613558+0000) 2019-10-13T13:18:33.615 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:33.607+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.613920+0000) 2019-10-13T13:18:33.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:33.611+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.614450+0000) 2019-10-13T13:18:33.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:33.607+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.614736+0000) 2019-10-13T13:18:33.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:33.611+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.616602+0000) 2019-10-13T13:18:33.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:33.615+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:33.621672+0000) 2019-10-13T13:18:33.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:33.615+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:35.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:35.567+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.571211+0000) 2019-10-13T13:18:35.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:35.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.857442+0000) 2019-10-13T13:18:36.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:36.535+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:36.538792+0000) 2019-10-13T13:18:38.627 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:38.619+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.625145+0000) 2019-10-13T13:18:38.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:38.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.625354+0000) 2019-10-13T13:18:38.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:38.623+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.626369+0000) 2019-10-13T13:18:38.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:38.619+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.626497+0000) 2019-10-13T13:18:38.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:38.619+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.626941+0000) 2019-10-13T13:18:38.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:38.623+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.627353+0000) 2019-10-13T13:18:38.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:38.623+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.627784+0000) 2019-10-13T13:18:38.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:38.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:38.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:38.631+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.635185+0000) 2019-10-13T13:18:38.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:38.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.805265+0000) 2019-10-13T13:18:38.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:38.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.807166+0000) 2019-10-13T13:18:39.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:39.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:39.046449+0000) 2019-10-13T13:18:40.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:40.035+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.039214+0000) 2019-10-13T13:18:40.355 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:18:40.355 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:40.358 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:18:40.359 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:42.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:42.555+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:42.557727+0000) 2019-10-13T13:18:43.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:43.635+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.636259+0000) 2019-10-13T13:18:43.639 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:43.631+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.637991+0000) 2019-10-13T13:18:43.640 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:43.635+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.637939+0000) 2019-10-13T13:18:43.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:43.635+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.638375+0000) 2019-10-13T13:18:43.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:43.635+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.638986+0000) 2019-10-13T13:18:43.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:43.639+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.639780+0000) 2019-10-13T13:18:43.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:43.639+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.640517+0000) 2019-10-13T13:18:43.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:43.639+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:43.643097+0000) 2019-10-13T13:18:43.647 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:43.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:45.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:45.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.571345+0000) 2019-10-13T13:18:45.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:45.855+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.857635+0000) 2019-10-13T13:18:46.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:46.535+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:46.539034+0000) 2019-10-13T13:18:48.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:48.647+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.649331+0000) 2019-10-13T13:18:48.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:48.647+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.649870+0000) 2019-10-13T13:18:48.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:48.647+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.650231+0000) 2019-10-13T13:18:48.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:48.647+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.650978+0000) 2019-10-13T13:18:48.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:48.651+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.651290+0000) 2019-10-13T13:18:48.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:48.651+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.652105+0000) 2019-10-13T13:18:48.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:48.651+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.652885+0000) 2019-10-13T13:18:48.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:48.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:48.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:48.655+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.659356+0000) 2019-10-13T13:18:48.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:48.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.805470+0000) 2019-10-13T13:18:48.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:48.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.807342+0000) 2019-10-13T13:18:49.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:49.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:49.046671+0000) 2019-10-13T13:18:50.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:50.035+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.039408+0000) 2019-10-13T13:18:52.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:52.563+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:52.564230+0000) 2019-10-13T13:18:53.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:53.659+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.661291+0000) 2019-10-13T13:18:53.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:53.659+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.661817+0000) 2019-10-13T13:18:53.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:53.659+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.663227+0000) 2019-10-13T13:18:53.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:53.659+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.662698+0000) 2019-10-13T13:18:53.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:53.663+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.663349+0000) 2019-10-13T13:18:53.666 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:53.663+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.663692+0000) 2019-10-13T13:18:53.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:53.663+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.664769+0000) 2019-10-13T13:18:53.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:53.663+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:53.668661+0000) 2019-10-13T13:18:53.672 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:53.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168931 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:55.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:55.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.571542+0000) 2019-10-13T13:18:55.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:55.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.857865+0000) 2019-10-13T13:18:56.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:56.534+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:56.539220+0000) 2019-10-13T13:18:58.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:58.670+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.673110+0000) 2019-10-13T13:18:58.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:18:58.670+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.674018+0000) 2019-10-13T13:18:58.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:58.670+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.674711+0000) 2019-10-13T13:18:58.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:18:58.671+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.675494+0000) 2019-10-13T13:18:58.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:18:58.674+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.676018+0000) 2019-10-13T13:18:58.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:58.674+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.676699+0000) 2019-10-13T13:18:58.679 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:18:58.674+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.677324+0000) 2019-10-13T13:18:58.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:18:58.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168963 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:18:58.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:18:58.679+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.683227+0000) 2019-10-13T13:18:58.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:18:58.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.805631+0000) 2019-10-13T13:18:58.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:18:58.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.807511+0000) 2019-10-13T13:18:59.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:18:59.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:59.046848+0000) 2019-10-13T13:19:00.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:00.035+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.039632+0000) 2019-10-13T13:19:02.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:02.567+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:02.570646+0000) 2019-10-13T13:19:03.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:03.682+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.685048+0000) 2019-10-13T13:19:03.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:03.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.686176+0000) 2019-10-13T13:19:03.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:03.682+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.686287+0000) 2019-10-13T13:19:03.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:03.682+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.686694+0000) 2019-10-13T13:19:03.689 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:03.686+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.687362+0000) 2019-10-13T13:19:03.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:03.686+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.688274+0000) 2019-10-13T13:19:03.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:03.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.689317+0000) 2019-10-13T13:19:03.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:03.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 168996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:03.697 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:03.691+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:03.695900+0000) 2019-10-13T13:19:05.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:05.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.571784+0000) 2019-10-13T13:19:05.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:05.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.858061+0000) 2019-10-13T13:19:06.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:06.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:06.539413+0000) 2019-10-13T13:19:08.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:08.694+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.696672+0000) 2019-10-13T13:19:08.699 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:08.694+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.697128+0000) 2019-10-13T13:19:08.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:08.694+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.697790+0000) 2019-10-13T13:19:08.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:08.694+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.698453+0000) 2019-10-13T13:19:08.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:08.698+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.699457+0000) 2019-10-13T13:19:08.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:08.695+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.700216+0000) 2019-10-13T13:19:08.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:08.698+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.700180+0000) 2019-10-13T13:19:08.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:08.699+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.704179+0000) 2019-10-13T13:19:08.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:08.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:08.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:08.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.805801+0000) 2019-10-13T13:19:08.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:08.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.807734+0000) 2019-10-13T13:19:09.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:09.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:09.047026+0000) 2019-10-13T13:19:10.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:10.039+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.039860+0000) 2019-10-13T13:19:10.403 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:19:10.403 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:10.406 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:19:10.407 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:12.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:12.571+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:12.576933+0000) 2019-10-13T13:19:13.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:13.706+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.708030+0000) 2019-10-13T13:19:13.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:13.706+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.708964+0000) 2019-10-13T13:19:13.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:13.706+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.709195+0000) 2019-10-13T13:19:13.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:13.706+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.709437+0000) 2019-10-13T13:19:13.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:13.706+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.710835+0000) 2019-10-13T13:19:13.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:13.710+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.711434+0000) 2019-10-13T13:19:13.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:13.707+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.712075+0000) 2019-10-13T13:19:13.716 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:13.711+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:13.714546+0000) 2019-10-13T13:19:13.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:13.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:15.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:15.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.571984+0000) 2019-10-13T13:19:15.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:15.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.858262+0000) 2019-10-13T13:19:16.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:16.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:16.539597+0000) 2019-10-13T13:19:18.722 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:18.718+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.719711+0000) 2019-10-13T13:19:18.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:18.718+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.720814+0000) 2019-10-13T13:19:18.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:18.715+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.721539+0000) 2019-10-13T13:19:18.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:18.718+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.722161+0000) 2019-10-13T13:19:18.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:18.718+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.722221+0000) 2019-10-13T13:19:18.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:18.718+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.722766+0000) 2019-10-13T13:19:18.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:18.722+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.723406+0000) 2019-10-13T13:19:18.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:18.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:18.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:18.723+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.728202+0000) 2019-10-13T13:19:18.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:18.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.806015+0000) 2019-10-13T13:19:18.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:18.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.807924+0000) 2019-10-13T13:19:19.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:19.042+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:19.047231+0000) 2019-10-13T13:19:20.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:20.035+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.040058+0000) 2019-10-13T13:19:22.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:22.579+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:22.582830+0000) 2019-10-13T13:19:23.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:23.726+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.730830+0000) 2019-10-13T13:19:23.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:23.730+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.731894+0000) 2019-10-13T13:19:23.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:23.730+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.732094+0000) 2019-10-13T13:19:23.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:23.730+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.732607+0000) 2019-10-13T13:19:23.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:23.727+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.734051+0000) 2019-10-13T13:19:23.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:23.730+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.734021+0000) 2019-10-13T13:19:23.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:23.730+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.734604+0000) 2019-10-13T13:19:23.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:23.731+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:23.737919+0000) 2019-10-13T13:19:23.742 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:23.738+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169127 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:25.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:25.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.572183+0000) 2019-10-13T13:19:25.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:25.854+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.858441+0000) 2019-10-13T13:19:26.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:26.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:26.539803+0000) 2019-10-13T13:19:28.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:28.738+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.743037+0000) 2019-10-13T13:19:28.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:28.742+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.743953+0000) 2019-10-13T13:19:28.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:28.738+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.744747+0000) 2019-10-13T13:19:28.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:28.742+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.745223+0000) 2019-10-13T13:19:28.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:28.742+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.746192+0000) 2019-10-13T13:19:28.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:28.742+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.746550+0000) 2019-10-13T13:19:28.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:28.742+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.747087+0000) 2019-10-13T13:19:28.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:28.746+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.750816+0000) 2019-10-13T13:19:28.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:28.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:28.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:28.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.806194+0000) 2019-10-13T13:19:28.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:28.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.808093+0000) 2019-10-13T13:19:29.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:29.046+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:29.047409+0000) 2019-10-13T13:19:30.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:30.038+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.040257+0000) 2019-10-13T13:19:32.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:32.586+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:32.589242+0000) 2019-10-13T13:19:33.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:33.754+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.756298+0000) 2019-10-13T13:19:33.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:33.750+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.757378+0000) 2019-10-13T13:19:33.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:33.754+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.757377+0000) 2019-10-13T13:19:33.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:33.754+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.757800+0000) 2019-10-13T13:19:33.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:33.754+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.758595+0000) 2019-10-13T13:19:33.761 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:33.758+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.759430+0000) 2019-10-13T13:19:33.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:33.758+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.759726+0000) 2019-10-13T13:19:33.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:33.762+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:33.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:33.758+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:33.765162+0000) 2019-10-13T13:19:35.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:35.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.572401+0000) 2019-10-13T13:19:35.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:35.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.858640+0000) 2019-10-13T13:19:36.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:36.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:36.539988+0000) 2019-10-13T13:19:38.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:38.761+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.767170+0000) 2019-10-13T13:19:38.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:38.765+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.767563+0000) 2019-10-13T13:19:38.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:38.765+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.769324+0000) 2019-10-13T13:19:38.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:38.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.769844+0000) 2019-10-13T13:19:38.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:38.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.770563+0000) 2019-10-13T13:19:38.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:38.765+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.770460+0000) 2019-10-13T13:19:38.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:38.765+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.770817+0000) 2019-10-13T13:19:38.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:38.770+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.775391+0000) 2019-10-13T13:19:38.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:38.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:38.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:38.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.806428+0000) 2019-10-13T13:19:38.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:38.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.808301+0000) 2019-10-13T13:19:39.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:39.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:39.047608+0000) 2019-10-13T13:19:40.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:40.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.040472+0000) 2019-10-13T13:19:40.452 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:19:40.452 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:40.457 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:19:40.457 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:42.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:42.594+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:42.595783+0000) 2019-10-13T13:19:43.781 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:43.774+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.779622+0000) 2019-10-13T13:19:43.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:43.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.779782+0000) 2019-10-13T13:19:43.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:43.777+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.780795+0000) 2019-10-13T13:19:43.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:43.777+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.780919+0000) 2019-10-13T13:19:43.783 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:43.777+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.781481+0000) 2019-10-13T13:19:43.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:43.777+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.782350+0000) 2019-10-13T13:19:43.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:43.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.782939+0000) 2019-10-13T13:19:43.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:43.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:43.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:43.782+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:43.788279+0000) 2019-10-13T13:19:45.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:45.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.572603+0000) 2019-10-13T13:19:45.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:45.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.858864+0000) 2019-10-13T13:19:46.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:46.537+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:46.540186+0000) 2019-10-13T13:19:48.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:48.786+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.791244+0000) 2019-10-13T13:19:48.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:48.789+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.791953+0000) 2019-10-13T13:19:48.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:48.789+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.792306+0000) 2019-10-13T13:19:48.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:48.789+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.792861+0000) 2019-10-13T13:19:48.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:48.789+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.792877+0000) 2019-10-13T13:19:48.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:48.789+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.793787+0000) 2019-10-13T13:19:48.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:48.789+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.794848+0000) 2019-10-13T13:19:48.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:48.790+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.798084+0000) 2019-10-13T13:19:48.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:48.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:48.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:48.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.806568+0000) 2019-10-13T13:19:48.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:48.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.808433+0000) 2019-10-13T13:19:49.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:49.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:49.047783+0000) 2019-10-13T13:19:50.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:50.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.040662+0000) 2019-10-13T13:19:52.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:52.598+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:52.602174+0000) 2019-10-13T13:19:53.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:53.797+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.801633+0000) 2019-10-13T13:19:53.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:53.797+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.802511+0000) 2019-10-13T13:19:53.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:53.801+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.803803+0000) 2019-10-13T13:19:53.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:53.798+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.804524+0000) 2019-10-13T13:19:53.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:53.801+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.804720+0000) 2019-10-13T13:19:53.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:53.801+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.805049+0000) 2019-10-13T13:19:53.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:53.801+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.805229+0000) 2019-10-13T13:19:53.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:53.806+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:53.810573+0000) 2019-10-13T13:19:53.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:53.805+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:55.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:55.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.572800+0000) 2019-10-13T13:19:55.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:55.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.859067+0000) 2019-10-13T13:19:56.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:56.537+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:56.540384+0000) 2019-10-13T13:19:58.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:58.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.806727+0000) 2019-10-13T13:19:58.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:58.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.808588+0000) 2019-10-13T13:19:58.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:19:58.809+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.814314+0000) 2019-10-13T13:19:58.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:19:58.809+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.814723+0000) 2019-10-13T13:19:58.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:19:58.813+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.815331+0000) 2019-10-13T13:19:58.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:19:58.813+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.815780+0000) 2019-10-13T13:19:58.818 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:19:58.810+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.816282+0000) 2019-10-13T13:19:58.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:58.813+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.816583+0000) 2019-10-13T13:19:58.819 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:19:58.813+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.817404+0000) 2019-10-13T13:19:58.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:19:58.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:19:58.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:19:58.822+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.827296+0000) 2019-10-13T13:19:59.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:19:59.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:59.047991+0000) 2019-10-13T13:20:00.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:00.034+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.040867+0000) 2019-10-13T13:20:02.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:02.606+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:02.608661+0000) 2019-10-13T13:20:03.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:03.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.825516+0000) 2019-10-13T13:20:03.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:03.821+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.826453+0000) 2019-10-13T13:20:03.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:03.821+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.827192+0000) 2019-10-13T13:20:03.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:03.821+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.827238+0000) 2019-10-13T13:20:03.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:03.825+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.827676+0000) 2019-10-13T13:20:03.830 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:03.822+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.828397+0000) 2019-10-13T13:20:03.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:03.825+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.828784+0000) 2019-10-13T13:20:03.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:03.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:03.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:03.830+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:03.835741+0000) 2019-10-13T13:20:05.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:05.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.572996+0000) 2019-10-13T13:20:05.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:05.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.859266+0000) 2019-10-13T13:20:06.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:06.537+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:06.540620+0000) 2019-10-13T13:20:08.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:08.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.806935+0000) 2019-10-13T13:20:08.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:08.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.808784+0000) 2019-10-13T13:20:08.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:08.833+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.837827+0000) 2019-10-13T13:20:08.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:08.833+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.838349+0000) 2019-10-13T13:20:08.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:08.837+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.839387+0000) 2019-10-13T13:20:08.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:08.837+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.839925+0000) 2019-10-13T13:20:08.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:08.837+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.840151+0000) 2019-10-13T13:20:08.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:08.834+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.841670+0000) 2019-10-13T13:20:08.843 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:08.837+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.841460+0000) 2019-10-13T13:20:08.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:08.842+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.847425+0000) 2019-10-13T13:20:08.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:08.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:09.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:09.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:09.048188+0000) 2019-10-13T13:20:10.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:10.038+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.041110+0000) 2019-10-13T13:20:10.592 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:20:10.592 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:10.596 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:20:10.596 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:12.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:12.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:12.615091+0000) 2019-10-13T13:20:13.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:13.845+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.850392+0000) 2019-10-13T13:20:13.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:13.845+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.851184+0000) 2019-10-13T13:20:13.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:13.849+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.851847+0000) 2019-10-13T13:20:13.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:13.850+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.852515+0000) 2019-10-13T13:20:13.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:13.849+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.852431+0000) 2019-10-13T13:20:13.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:13.849+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.852812+0000) 2019-10-13T13:20:13.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:13.849+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.853244+0000) 2019-10-13T13:20:13.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:13.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169457 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:13.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:13.858+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:13.859526+0000) 2019-10-13T13:20:15.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:15.569+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.573216+0000) 2019-10-13T13:20:15.861 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:15.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.859507+0000) 2019-10-13T13:20:16.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:16.537+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:16.540849+0000) 2019-10-13T13:20:18.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:18.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.807125+0000) 2019-10-13T13:20:18.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:18.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.808980+0000) 2019-10-13T13:20:18.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:18.857+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.861862+0000) 2019-10-13T13:20:18.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:18.862+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.862441+0000) 2019-10-13T13:20:18.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:18.857+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.862578+0000) 2019-10-13T13:20:18.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:18.857+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.863164+0000) 2019-10-13T13:20:18.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:18.861+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.863950+0000) 2019-10-13T13:20:18.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:18.861+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.864223+0000) 2019-10-13T13:20:18.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:18.861+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.864533+0000) 2019-10-13T13:20:18.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:18.866+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.870102+0000) 2019-10-13T13:20:18.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:18.865+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:19.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:19.045+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:19.048374+0000) 2019-10-13T13:20:20.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:20.037+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.041358+0000) 2019-10-13T13:20:22.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:22.617+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:22.621510+0000) 2019-10-13T13:20:23.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:23.869+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.873225+0000) 2019-10-13T13:20:23.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:23.868+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.873649+0000) 2019-10-13T13:20:23.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:23.868+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.874760+0000) 2019-10-13T13:20:23.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:23.872+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.875704+0000) 2019-10-13T13:20:23.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:23.872+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.876259+0000) 2019-10-13T13:20:23.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:23.872+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.876740+0000) 2019-10-13T13:20:23.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:23.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.877121+0000) 2019-10-13T13:20:23.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:23.877+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:23.880091+0000) 2019-10-13T13:20:23.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:23.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:25.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:25.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.573417+0000) 2019-10-13T13:20:25.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:25.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.859710+0000) 2019-10-13T13:20:26.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:26.540+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:26.541048+0000) 2019-10-13T13:20:28.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:28.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.807343+0000) 2019-10-13T13:20:28.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:28.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.809150+0000) 2019-10-13T13:20:28.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:28.880+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.886193+0000) 2019-10-13T13:20:28.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:28.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.886799+0000) 2019-10-13T13:20:28.889 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:28.880+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.887103+0000) 2019-10-13T13:20:28.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:28.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.887612+0000) 2019-10-13T13:20:28.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:28.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.888890+0000) 2019-10-13T13:20:28.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:28.884+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.889149+0000) 2019-10-13T13:20:28.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:28.884+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.889611+0000) 2019-10-13T13:20:28.896 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:28.893+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.894969+0000) 2019-10-13T13:20:28.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:28.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:29.050 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:29.044+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:29.048555+0000) 2019-10-13T13:20:30.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:30.037+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.041564+0000) 2019-10-13T13:20:32.630 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:32.625+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:32.627984+0000) 2019-10-13T13:20:33.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:33.892+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.897757+0000) 2019-10-13T13:20:33.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:33.892+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.898481+0000) 2019-10-13T13:20:33.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:33.896+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.899565+0000) 2019-10-13T13:20:33.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:33.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.900190+0000) 2019-10-13T13:20:33.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:33.896+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.900596+0000) 2019-10-13T13:20:33.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:33.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.901064+0000) 2019-10-13T13:20:33.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:33.896+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.901600+0000) 2019-10-13T13:20:33.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:33.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:33.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:33.905+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:33.907724+0000) 2019-10-13T13:20:35.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:35.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.573620+0000) 2019-10-13T13:20:35.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:35.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.859910+0000) 2019-10-13T13:20:36.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:36.540+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:36.541238+0000) 2019-10-13T13:20:38.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:38.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.807500+0000) 2019-10-13T13:20:38.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:38.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.809333+0000) 2019-10-13T13:20:38.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:38.909+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.910672+0000) 2019-10-13T13:20:38.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:38.904+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.910958+0000) 2019-10-13T13:20:38.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:38.908+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.911830+0000) 2019-10-13T13:20:38.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:38.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.912194+0000) 2019-10-13T13:20:38.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:38.908+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.913463+0000) 2019-10-13T13:20:38.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:38.908+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.914036+0000) 2019-10-13T13:20:38.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:38.908+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.914444+0000) 2019-10-13T13:20:38.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:38.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.918316+0000) 2019-10-13T13:20:38.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:38.912+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:39.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:39.048+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:39.048767+0000) 2019-10-13T13:20:40.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:40.037+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.041759+0000) 2019-10-13T13:20:40.690 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:20:40.690 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:40.694 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:20:40.694 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:42.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:42.633+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:42.634457+0000) 2019-10-13T13:20:43.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:43.916+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.921712+0000) 2019-10-13T13:20:43.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:43.916+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.922088+0000) 2019-10-13T13:20:43.925 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:43.921+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.922924+0000) 2019-10-13T13:20:43.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:43.916+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.923273+0000) 2019-10-13T13:20:43.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:43.920+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.923832+0000) 2019-10-13T13:20:43.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:43.920+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.924465+0000) 2019-10-13T13:20:43.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:43.920+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.925748+0000) 2019-10-13T13:20:43.931 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:43.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:43.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:43.929+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:43.930940+0000) 2019-10-13T13:20:45.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:45.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.573833+0000) 2019-10-13T13:20:45.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:45.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.860049+0000) 2019-10-13T13:20:46.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:46.540+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:46.541430+0000) 2019-10-13T13:20:48.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:48.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.807698+0000) 2019-10-13T13:20:48.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:48.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.809558+0000) 2019-10-13T13:20:48.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:48.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.933699+0000) 2019-10-13T13:20:48.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:48.933+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.934237+0000) 2019-10-13T13:20:48.937 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:48.928+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.934190+0000) 2019-10-13T13:20:48.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:48.928+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.934608+0000) 2019-10-13T13:20:48.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:48.928+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.934913+0000) 2019-10-13T13:20:48.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:48.932+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.936241+0000) 2019-10-13T13:20:48.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:48.932+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.936954+0000) 2019-10-13T13:20:48.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:48.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:48.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:48.941+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.942601+0000) 2019-10-13T13:20:49.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:49.048+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:49.048949+0000) 2019-10-13T13:20:50.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:50.037+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.041949+0000) 2019-10-13T13:20:52.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:52.637+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:52.640876+0000) 2019-10-13T13:20:53.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:53.940+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.945034+0000) 2019-10-13T13:20:53.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:53.940+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.945118+0000) 2019-10-13T13:20:53.948 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:53.940+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.946289+0000) 2019-10-13T13:20:53.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:53.945+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.947383+0000) 2019-10-13T13:20:53.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:53.944+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.947663+0000) 2019-10-13T13:20:53.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:53.944+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.948182+0000) 2019-10-13T13:20:53.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:53.944+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.948452+0000) 2019-10-13T13:20:53.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:53.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:53.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:53.953+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:53.955592+0000) 2019-10-13T13:20:55.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:55.568+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.574029+0000) 2019-10-13T13:20:55.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:55.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.860247+0000) 2019-10-13T13:20:56.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:56.536+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:56.541643+0000) 2019-10-13T13:20:58.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:58.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.807898+0000) 2019-10-13T13:20:58.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:58.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.809750+0000) 2019-10-13T13:20:58.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:20:58.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.956402+0000) 2019-10-13T13:20:58.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:20:58.952+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.957029+0000) 2019-10-13T13:20:58.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:20:58.952+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.957351+0000) 2019-10-13T13:20:58.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:20:58.952+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.958530+0000) 2019-10-13T13:20:58.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:20:58.957+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.959391+0000) 2019-10-13T13:20:58.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:58.956+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.959494+0000) 2019-10-13T13:20:58.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:20:58.956+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.959823+0000) 2019-10-13T13:20:58.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:20:58.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:20:58.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:20:58.965+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.967540+0000) 2019-10-13T13:20:59.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:20:59.048+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:59.049147+0000) 2019-10-13T13:21:00.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:00.041+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.042173+0000) 2019-10-13T13:21:02.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:02.645+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:02.647418+0000) 2019-10-13T13:21:03.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:03.963+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.968216+0000) 2019-10-13T13:21:03.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:03.963+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.968977+0000) 2019-10-13T13:21:03.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:03.963+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.969287+0000) 2019-10-13T13:21:03.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:03.969+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.970307+0000) 2019-10-13T13:21:03.972 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:03.963+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.969707+0000) 2019-10-13T13:21:03.973 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:03.967+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.971794+0000) 2019-10-13T13:21:03.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:03.967+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.972189+0000) 2019-10-13T13:21:03.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:03.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:03.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:03.973+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:03.977959+0000) 2019-10-13T13:21:05.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:05.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.574262+0000) 2019-10-13T13:21:05.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:05.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.860472+0000) 2019-10-13T13:21:06.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:06.539+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:06.541842+0000) 2019-10-13T13:21:08.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:08.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.808051+0000) 2019-10-13T13:21:08.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:08.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.809935+0000) 2019-10-13T13:21:08.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:08.975+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.980681+0000) 2019-10-13T13:21:08.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:08.975+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.981491+0000) 2019-10-13T13:21:08.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:08.977+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.982006+0000) 2019-10-13T13:21:08.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:08.975+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.982178+0000) 2019-10-13T13:21:08.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:08.975+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.982575+0000) 2019-10-13T13:21:08.985 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:08.975+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.982969+0000) 2019-10-13T13:21:08.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:08.979+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.983603+0000) 2019-10-13T13:21:08.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:08.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:08.992 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:08.989+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.990519+0000) 2019-10-13T13:21:09.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:09.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:09.049348+0000) 2019-10-13T13:21:10.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:10.041+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.042363+0000) 2019-10-13T13:21:10.796 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:21:10.797 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:10.800 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:21:10.800 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:12.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:12.653+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:12.653786+0000) 2019-10-13T13:21:13.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:13.987+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.992243+0000) 2019-10-13T13:21:13.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:13.987+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.992461+0000) 2019-10-13T13:21:13.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:13.992+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.994176+0000) 2019-10-13T13:21:13.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:13.987+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.994149+0000) 2019-10-13T13:21:13.996 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:13.987+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.994681+0000) 2019-10-13T13:21:13.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:13.987+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.995051+0000) 2019-10-13T13:21:13.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:13.991+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:13.995385+0000) 2019-10-13T13:21:14.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:13.996+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:14.000373+0000) 2019-10-13T13:21:14.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:13.995+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:15.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:15.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.574464+0000) 2019-10-13T13:21:15.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:15.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.860688+0000) 2019-10-13T13:21:16.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:16.539+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:16.542037+0000) 2019-10-13T13:21:18.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:18.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:18.808247+0000) 2019-10-13T13:21:18.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:18.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:18.810132+0000) 2019-10-13T13:21:19.006 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:18.999+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.004350+0000) 2019-10-13T13:21:19.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:18.999+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.004932+0000) 2019-10-13T13:21:19.007 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:18.999+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.005643+0000) 2019-10-13T13:21:19.008 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:19.004+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.006436+0000) 2019-10-13T13:21:19.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:18.999+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.006470+0000) 2019-10-13T13:21:19.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:18.999+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.006709+0000) 2019-10-13T13:21:19.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:18.999+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.007105+0000) 2019-10-13T13:21:19.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:19.008+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.011793+0000) 2019-10-13T13:21:19.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:19.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:19.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:19.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:19.049568+0000) 2019-10-13T13:21:20.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:20.040+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.042554+0000) 2019-10-13T13:21:22.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:22.656+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:22.660139+0000) 2019-10-13T13:21:24.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:24.011+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.016954+0000) 2019-10-13T13:21:24.019 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:24.011+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.017948+0000) 2019-10-13T13:21:24.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:24.016+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.018521+0000) 2019-10-13T13:21:24.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:24.011+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.018632+0000) 2019-10-13T13:21:24.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:24.011+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.019098+0000) 2019-10-13T13:21:24.021 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:24.011+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.018877+0000) 2019-10-13T13:21:24.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:24.015+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.019524+0000) 2019-10-13T13:21:24.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:24.020+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.023870+0000) 2019-10-13T13:21:24.026 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:24.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:25.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:25.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.574672+0000) 2019-10-13T13:21:25.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:25.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.860921+0000) 2019-10-13T13:21:26.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:26.535+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:26.542230+0000) 2019-10-13T13:21:28.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:28.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:28.808403+0000) 2019-10-13T13:21:28.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:28.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:28.810337+0000) 2019-10-13T13:21:29.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:29.023+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.027296+0000) 2019-10-13T13:21:29.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:29.024+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.029221+0000) 2019-10-13T13:21:29.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:29.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.029014+0000) 2019-10-13T13:21:29.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:29.023+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.029504+0000) 2019-10-13T13:21:29.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:29.023+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.029840+0000) 2019-10-13T13:21:29.032 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:29.023+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.030672+0000) 2019-10-13T13:21:29.033 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:29.023+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.031058+0000) 2019-10-13T13:21:29.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:29.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:29.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:29.032+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.037484+0000) 2019-10-13T13:21:29.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:29.043+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:29.049707+0000) 2019-10-13T13:21:30.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:30.040+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.042782+0000) 2019-10-13T13:21:32.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:32.664+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:32.666644+0000) 2019-10-13T13:21:34.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:34.035+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.039076+0000) 2019-10-13T13:21:34.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:34.039+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.039973+0000) 2019-10-13T13:21:34.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:34.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.040566+0000) 2019-10-13T13:21:34.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:34.039+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.040881+0000) 2019-10-13T13:21:34.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:34.039+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.041103+0000) 2019-10-13T13:21:34.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:34.039+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.042096+0000) 2019-10-13T13:21:34.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:34.039+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.042341+0000) 2019-10-13T13:21:34.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:34.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 169985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:34.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:34.044+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.048283+0000) 2019-10-13T13:21:35.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:35.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.574876+0000) 2019-10-13T13:21:35.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:35.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.861117+0000) 2019-10-13T13:21:36.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:36.539+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:36.542418+0000) 2019-10-13T13:21:38.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:38.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:38.808621+0000) 2019-10-13T13:21:38.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:38.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:38.810522+0000) 2019-10-13T13:21:39.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:39.048+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.051265+0000) 2019-10-13T13:21:39.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:39.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.051451+0000) 2019-10-13T13:21:39.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:39.051+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.052045+0000) 2019-10-13T13:21:39.056 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:39.051+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.052535+0000) 2019-10-13T13:21:39.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:39.051+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.053049+0000) 2019-10-13T13:21:39.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:39.051+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.053342+0000) 2019-10-13T13:21:39.057 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:39.051+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.054824+0000) 2019-10-13T13:21:39.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:39.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:39.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:39.060+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:39.062811+0000) 2019-10-13T13:21:40.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:40.040+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.042994+0000) 2019-10-13T13:21:40.853 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:21:40.853 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:40.857 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:21:40.857 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:42.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:42.672+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:42.673190+0000) 2019-10-13T13:21:44.065 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:44.059+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.062878+0000) 2019-10-13T13:21:44.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:44.060+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.063785+0000) 2019-10-13T13:21:44.066 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:44.063+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.063923+0000) 2019-10-13T13:21:44.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:44.063+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.065083+0000) 2019-10-13T13:21:44.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:44.063+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.064877+0000) 2019-10-13T13:21:44.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:44.063+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.065820+0000) 2019-10-13T13:21:44.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:44.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.066765+0000) 2019-10-13T13:21:44.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:44.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:44.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:44.072+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.074193+0000) 2019-10-13T13:21:45.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:45.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.575069+0000) 2019-10-13T13:21:45.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:45.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.800538+0000) 2019-10-13T13:21:45.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:45.859+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.861328+0000) 2019-10-13T13:21:46.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:46.539+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:46.542619+0000) 2019-10-13T13:21:48.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:48.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:48.808778+0000) 2019-10-13T13:21:48.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:48.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:48.810673+0000) 2019-10-13T13:21:49.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:49.072+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.075477+0000) 2019-10-13T13:21:49.078 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:49.074+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.076195+0000) 2019-10-13T13:21:49.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:49.074+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.076659+0000) 2019-10-13T13:21:49.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:49.074+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.077199+0000) 2019-10-13T13:21:49.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:49.074+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.077927+0000) 2019-10-13T13:21:49.080 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:49.074+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.078283+0000) 2019-10-13T13:21:49.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:49.078+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.079399+0000) 2019-10-13T13:21:49.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:49.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170084 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:49.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:49.084+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:49.088245+0000) 2019-10-13T13:21:50.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:50.040+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.043160+0000) 2019-10-13T13:21:52.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:52.676+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:52.679602+0000) 2019-10-13T13:21:54.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:54.082+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.087157+0000) 2019-10-13T13:21:54.090 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:54.086+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.087854+0000) 2019-10-13T13:21:54.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:54.086+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.087986+0000) 2019-10-13T13:21:54.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:54.086+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.088378+0000) 2019-10-13T13:21:54.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:54.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.090390+0000) 2019-10-13T13:21:54.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:54.086+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.090378+0000) 2019-10-13T13:21:54.093 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:54.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.090663+0000) 2019-10-13T13:21:54.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:54.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170117 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:54.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:54.092+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.096530+0000) 2019-10-13T13:21:55.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:55.570+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.575270+0000) 2019-10-13T13:21:55.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:55.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.800739+0000) 2019-10-13T13:21:55.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:55.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.861525+0000) 2019-10-13T13:21:56.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:56.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:56.542813+0000) 2019-10-13T13:21:58.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:58.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:58.808982+0000) 2019-10-13T13:21:58.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:58.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:58.810896+0000) 2019-10-13T13:21:59.102 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:21:59.098+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.099725+0000) 2019-10-13T13:21:59.102 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:21:59.098+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.099930+0000) 2019-10-13T13:21:59.102 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:21:59.098+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.100332+0000) 2019-10-13T13:21:59.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:21:59.098+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.100877+0000) 2019-10-13T13:21:59.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:21:59.098+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.101176+0000) 2019-10-13T13:21:59.103 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:21:59.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.101894+0000) 2019-10-13T13:21:59.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:21:59.098+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.102054+0000) 2019-10-13T13:21:59.111 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:21:59.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:21:59.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:21:59.104+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:59.109556+0000) 2019-10-13T13:22:00.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:00.040+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.043379+0000) 2019-10-13T13:22:02.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:02.684+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:02.686010+0000) 2019-10-13T13:22:04.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:04.110+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.111949+0000) 2019-10-13T13:22:04.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:04.110+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.112832+0000) 2019-10-13T13:22:04.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:04.110+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.112924+0000) 2019-10-13T13:22:04.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:04.110+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.113266+0000) 2019-10-13T13:22:04.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:04.112+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.114375+0000) 2019-10-13T13:22:04.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:04.110+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.115184+0000) 2019-10-13T13:22:04.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:04.114+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.115505+0000) 2019-10-13T13:22:04.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:04.116+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.121053+0000) 2019-10-13T13:22:04.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:04.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:05.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:05.574+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.575503+0000) 2019-10-13T13:22:05.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:05.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.800933+0000) 2019-10-13T13:22:05.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:05.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.861758+0000) 2019-10-13T13:22:06.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:06.538+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:06.543055+0000) 2019-10-13T13:22:08.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:08.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:08.809180+0000) 2019-10-13T13:22:08.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:08.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:08.811105+0000) 2019-10-13T13:22:09.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:09.122+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.124386+0000) 2019-10-13T13:22:09.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:09.122+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.125061+0000) 2019-10-13T13:22:09.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:09.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.125244+0000) 2019-10-13T13:22:09.128 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:09.122+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.126150+0000) 2019-10-13T13:22:09.128 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:09.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.126643+0000) 2019-10-13T13:22:09.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:09.122+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.126843+0000) 2019-10-13T13:22:09.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:09.122+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.127125+0000) 2019-10-13T13:22:09.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:09.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170220 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:09.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:09.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:09.133865+0000) 2019-10-13T13:22:10.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:10.039+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.043551+0000) 2019-10-13T13:22:10.918 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:22:10.918 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:10.921 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:22:10.922 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:12.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:12.687+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:12.691776+0000) 2019-10-13T13:22:14.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:14.134+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.136022+0000) 2019-10-13T13:22:14.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:14.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.137341+0000) 2019-10-13T13:22:14.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:14.131+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.137951+0000) 2019-10-13T13:22:14.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:14.134+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.138321+0000) 2019-10-13T13:22:14.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:14.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.138816+0000) 2019-10-13T13:22:14.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:14.134+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.138279+0000) 2019-10-13T13:22:14.141 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:14.134+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.139189+0000) 2019-10-13T13:22:14.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:14.139+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.143265+0000) 2019-10-13T13:22:14.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:14.142+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:15.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:15.574+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.575656+0000) 2019-10-13T13:22:15.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:15.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.801150+0000) 2019-10-13T13:22:15.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:15.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.861955+0000) 2019-10-13T13:22:16.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:16.542+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:16.543251+0000) 2019-10-13T13:22:18.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:18.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:18.809393+0000) 2019-10-13T13:22:18.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:18.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:18.811299+0000) 2019-10-13T13:22:19.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:19.146+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.148314+0000) 2019-10-13T13:22:19.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:19.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.148953+0000) 2019-10-13T13:22:19.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:19.146+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.149044+0000) 2019-10-13T13:22:19.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:19.146+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.149682+0000) 2019-10-13T13:22:19.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:19.146+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.150701+0000) 2019-10-13T13:22:19.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:19.146+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.150400+0000) 2019-10-13T13:22:19.153 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:19.150+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.151829+0000) 2019-10-13T13:22:19.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:19.154+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:19.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:19.151+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:19.156527+0000) 2019-10-13T13:22:20.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:20.039+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.043756+0000) 2019-10-13T13:22:22.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:22.695+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:22.698197+0000) 2019-10-13T13:22:24.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:24.154+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.158918+0000) 2019-10-13T13:22:24.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:24.158+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.160311+0000) 2019-10-13T13:22:24.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:24.158+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.160748+0000) 2019-10-13T13:22:24.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:24.158+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.161445+0000) 2019-10-13T13:22:24.163 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:24.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.162059+0000) 2019-10-13T13:22:24.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:24.158+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.162700+0000) 2019-10-13T13:22:24.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:24.162+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.163567+0000) 2019-10-13T13:22:24.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:24.163+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.166576+0000) 2019-10-13T13:22:24.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:24.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:25.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:25.574+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.575854+0000) 2019-10-13T13:22:25.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:25.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.801347+0000) 2019-10-13T13:22:25.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:25.858+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.862186+0000) 2019-10-13T13:22:26.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:26.542+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:26.543508+0000) 2019-10-13T13:22:28.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:28.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:28.809588+0000) 2019-10-13T13:22:28.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:28.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:28.811542+0000) 2019-10-13T13:22:29.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:29.170+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.172060+0000) 2019-10-13T13:22:29.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:29.170+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.172718+0000) 2019-10-13T13:22:29.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:29.170+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.173192+0000) 2019-10-13T13:22:29.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:29.167+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.174023+0000) 2019-10-13T13:22:29.176 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:29.170+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.174012+0000) 2019-10-13T13:22:29.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:29.170+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.174286+0000) 2019-10-13T13:22:29.179 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:29.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.175012+0000) 2019-10-13T13:22:29.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:29.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:29.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:29.175+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:29.181153+0000) 2019-10-13T13:22:30.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:30.039+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.043954+0000) 2019-10-13T13:22:32.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:32.699+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:32.705148+0000) 2019-10-13T13:22:34.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:34.181+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.184273+0000) 2019-10-13T13:22:34.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:34.181+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.184767+0000) 2019-10-13T13:22:34.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:34.181+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.185103+0000) 2019-10-13T13:22:34.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:34.181+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.186275+0000) 2019-10-13T13:22:34.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:34.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.187242+0000) 2019-10-13T13:22:34.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:34.185+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.187404+0000) 2019-10-13T13:22:34.190 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:34.185+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.188022+0000) 2019-10-13T13:22:34.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:34.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:34.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:34.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.195024+0000) 2019-10-13T13:22:35.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:35.573+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.576087+0000) 2019-10-13T13:22:35.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:35.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.801577+0000) 2019-10-13T13:22:35.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:35.861+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.862389+0000) 2019-10-13T13:22:36.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:36.541+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:36.543737+0000) 2019-10-13T13:22:38.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:38.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:38.809785+0000) 2019-10-13T13:22:38.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:38.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:38.811740+0000) 2019-10-13T13:22:39.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:39.193+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.196870+0000) 2019-10-13T13:22:39.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:39.193+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.197301+0000) 2019-10-13T13:22:39.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:39.191+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.198085+0000) 2019-10-13T13:22:39.202 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:39.193+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.198224+0000) 2019-10-13T13:22:39.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:39.193+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.198723+0000) 2019-10-13T13:22:39.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:39.193+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.199108+0000) 2019-10-13T13:22:39.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:39.197+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.199705+0000) 2019-10-13T13:22:39.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:39.199+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:39.204199+0000) 2019-10-13T13:22:39.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:39.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:40.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:40.043+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.044117+0000) 2019-10-13T13:22:41.049 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:22:41.049 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:41.053 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:22:41.053 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:42.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:42.707+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:42.712195+0000) 2019-10-13T13:22:44.210 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:44.205+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.207523+0000) 2019-10-13T13:22:44.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:44.205+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.207847+0000) 2019-10-13T13:22:44.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:44.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.208848+0000) 2019-10-13T13:22:44.211 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:44.205+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.209295+0000) 2019-10-13T13:22:44.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:44.205+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.209430+0000) 2019-10-13T13:22:44.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:44.207+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.210203+0000) 2019-10-13T13:22:44.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:44.205+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.210834+0000) 2019-10-13T13:22:44.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:44.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.216735+0000) 2019-10-13T13:22:44.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:44.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:45.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:45.573+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.576278+0000) 2019-10-13T13:22:45.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:45.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.801798+0000) 2019-10-13T13:22:45.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:45.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.862594+0000) 2019-10-13T13:22:46.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:46.541+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:46.543984+0000) 2019-10-13T13:22:48.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:48.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:48.809989+0000) 2019-10-13T13:22:48.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:48.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:48.811932+0000) 2019-10-13T13:22:49.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:49.217+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.219416+0000) 2019-10-13T13:22:49.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:49.217+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.220130+0000) 2019-10-13T13:22:49.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:49.217+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.220780+0000) 2019-10-13T13:22:49.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:49.217+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.220999+0000) 2019-10-13T13:22:49.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:49.219+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.222277+0000) 2019-10-13T13:22:49.225 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:49.217+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.223160+0000) 2019-10-13T13:22:49.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:49.217+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.223199+0000) 2019-10-13T13:22:49.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:49.223+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:49.229314+0000) 2019-10-13T13:22:49.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:49.225+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:50.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:50.039+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.044272+0000) 2019-10-13T13:22:52.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:52.715+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:52.718581+0000) 2019-10-13T13:22:54.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:54.229+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.232761+0000) 2019-10-13T13:22:54.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:54.229+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.233812+0000) 2019-10-13T13:22:54.236 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:54.229+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.234132+0000) 2019-10-13T13:22:54.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:54.229+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.234456+0000) 2019-10-13T13:22:54.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:54.231+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.235063+0000) 2019-10-13T13:22:54.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:54.229+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.235258+0000) 2019-10-13T13:22:54.238 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:54.233+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.236085+0000) 2019-10-13T13:22:54.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:54.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:54.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:54.239+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.243020+0000) 2019-10-13T13:22:55.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:55.573+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.576483+0000) 2019-10-13T13:22:55.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:55.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.802021+0000) 2019-10-13T13:22:55.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:55.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.862838+0000) 2019-10-13T13:22:56.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:56.541+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:56.544173+0000) 2019-10-13T13:22:58.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:58.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:58.810217+0000) 2019-10-13T13:22:58.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:58.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:58.812125+0000) 2019-10-13T13:22:59.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:22:59.241+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.245382+0000) 2019-10-13T13:22:59.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:22:59.241+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.246107+0000) 2019-10-13T13:22:59.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:22:59.241+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.246485+0000) 2019-10-13T13:22:59.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:22:59.241+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.246845+0000) 2019-10-13T13:22:59.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:22:59.245+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.247521+0000) 2019-10-13T13:22:59.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:22:59.245+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.247914+0000) 2019-10-13T13:22:59.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:22:59.243+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.248611+0000) 2019-10-13T13:22:59.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:22:59.249+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:22:59.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:22:59.251+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:59.256908+0000) 2019-10-13T13:23:00.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:00.043+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:00.044453+0000) 2019-10-13T13:23:02.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:02.718+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:02.725684+0000) 2019-10-13T13:23:04.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:04.253+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.257240+0000) 2019-10-13T13:23:04.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:04.253+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.257558+0000) 2019-10-13T13:23:04.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:04.253+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.258858+0000) 2019-10-13T13:23:04.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:04.254+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.259364+0000) 2019-10-13T13:23:04.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:04.257+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.259558+0000) 2019-10-13T13:23:04.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:04.257+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.260292+0000) 2019-10-13T13:23:04.262 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:04.257+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.260555+0000) 2019-10-13T13:23:04.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:04.262+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.266447+0000) 2019-10-13T13:23:04.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:04.261+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:05.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:05.573+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:05.576722+0000) 2019-10-13T13:23:05.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:05.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:05.802219+0000) 2019-10-13T13:23:05.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:05.857+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:05.863001+0000) 2019-10-13T13:23:06.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:06.541+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.544370+0000) 2019-10-13T13:23:08.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:08.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:08.810444+0000) 2019-10-13T13:23:08.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:08.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:08.812337+0000) 2019-10-13T13:23:09.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:09.265+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.269208+0000) 2019-10-13T13:23:09.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:09.262+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.270061+0000) 2019-10-13T13:23:09.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:09.265+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.271075+0000) 2019-10-13T13:23:09.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:09.265+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.270782+0000) 2019-10-13T13:23:09.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:09.269+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.272313+0000) 2019-10-13T13:23:09.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:09.269+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.273082+0000) 2019-10-13T13:23:09.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:09.269+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.273174+0000) 2019-10-13T13:23:09.279 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:09.273+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:09.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:09.274+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:09.278473+0000) 2019-10-13T13:23:10.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:10.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:10.044579+0000) 2019-10-13T13:23:11.173 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:23:11.173 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:11.177 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:23:11.177 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:12.734 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:12.730+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:12.732576+0000) 2019-10-13T13:23:14.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:14.276+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.280479+0000) 2019-10-13T13:23:14.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:14.276+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.280960+0000) 2019-10-13T13:23:14.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:14.276+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.281616+0000) 2019-10-13T13:23:14.284 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:14.278+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.282539+0000) 2019-10-13T13:23:14.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:14.276+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.283102+0000) 2019-10-13T13:23:14.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:14.276+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.283084+0000) 2019-10-13T13:23:14.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:14.280+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.283480+0000) 2019-10-13T13:23:14.291 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:14.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:14.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:14.286+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.290765+0000) 2019-10-13T13:23:15.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:15.576+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:15.576985+0000) 2019-10-13T13:23:15.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:15.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:15.802433+0000) 2019-10-13T13:23:15.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:15.856+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:15.863241+0000) 2019-10-13T13:23:16.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:16.540+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.544569+0000) 2019-10-13T13:23:18.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:18.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:18.810699+0000) 2019-10-13T13:23:18.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:18.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:18.812532+0000) 2019-10-13T13:23:19.294 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:19.288+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.292620+0000) 2019-10-13T13:23:19.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:19.288+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.293003+0000) 2019-10-13T13:23:19.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:19.288+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.293645+0000) 2019-10-13T13:23:19.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:19.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.294758+0000) 2019-10-13T13:23:19.297 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:19.290+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.295920+0000) 2019-10-13T13:23:19.298 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:19.292+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.295921+0000) 2019-10-13T13:23:19.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:19.292+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.296418+0000) 2019-10-13T13:23:19.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:19.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:19.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:19.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:19.302210+0000) 2019-10-13T13:23:20.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:20.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:20.044749+0000) 2019-10-13T13:23:22.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:22.738+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:22.739751+0000) 2019-10-13T13:23:24.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:24.300+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.305081+0000) 2019-10-13T13:23:24.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:24.300+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.305658+0000) 2019-10-13T13:23:24.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:24.300+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.306104+0000) 2019-10-13T13:23:24.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:24.300+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.306511+0000) 2019-10-13T13:23:24.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:24.302+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.307268+0000) 2019-10-13T13:23:24.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:24.304+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.307903+0000) 2019-10-13T13:23:24.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:24.304+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.307971+0000) 2019-10-13T13:23:24.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:24.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:24.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:24.310+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.316371+0000) 2019-10-13T13:23:25.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:25.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:25.577198+0000) 2019-10-13T13:23:25.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:25.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:25.802639+0000) 2019-10-13T13:23:25.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:25.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:25.863505+0000) 2019-10-13T13:23:26.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:26.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.544764+0000) 2019-10-13T13:23:28.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:28.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:28.810893+0000) 2019-10-13T13:23:28.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:28.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:28.812729+0000) 2019-10-13T13:23:29.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:29.312+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.316420+0000) 2019-10-13T13:23:29.319 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:29.312+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.316731+0000) 2019-10-13T13:23:29.319 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:29.310+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.317906+0000) 2019-10-13T13:23:29.320 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:29.312+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.318566+0000) 2019-10-13T13:23:29.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:29.312+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.319137+0000) 2019-10-13T13:23:29.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:29.316+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.319377+0000) 2019-10-13T13:23:29.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:29.316+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.319895+0000) 2019-10-13T13:23:29.328 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:29.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:29.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:29.322+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:29.327401+0000) 2019-10-13T13:23:30.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:30.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:30.044941+0000) 2019-10-13T13:23:32.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:32.746+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:32.746920+0000) 2019-10-13T13:23:34.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:34.324+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.329021+0000) 2019-10-13T13:23:34.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:34.324+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.329626+0000) 2019-10-13T13:23:34.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:34.324+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.330602+0000) 2019-10-13T13:23:34.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:34.326+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.331346+0000) 2019-10-13T13:23:34.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:34.328+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.331390+0000) 2019-10-13T13:23:34.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:34.328+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.331890+0000) 2019-10-13T13:23:34.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:34.328+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.332149+0000) 2019-10-13T13:23:34.339 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:34.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:34.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:34.334+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.338602+0000) 2019-10-13T13:23:35.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:35.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:35.577396+0000) 2019-10-13T13:23:35.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:35.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:35.802854+0000) 2019-10-13T13:23:35.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:35.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:35.863740+0000) 2019-10-13T13:23:36.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:36.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.544957+0000) 2019-10-13T13:23:38.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:38.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:38.811093+0000) 2019-10-13T13:23:38.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:38.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:38.812925+0000) 2019-10-13T13:23:39.343 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:39.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.340968+0000) 2019-10-13T13:23:39.343 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:39.336+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.341448+0000) 2019-10-13T13:23:39.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:39.336+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.341791+0000) 2019-10-13T13:23:39.344 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:39.336+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.342113+0000) 2019-10-13T13:23:39.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:39.336+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.342274+0000) 2019-10-13T13:23:39.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:39.336+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.342629+0000) 2019-10-13T13:23:39.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:39.340+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.344069+0000) 2019-10-13T13:23:39.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:39.342+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:39.349573+0000) 2019-10-13T13:23:39.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:39.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:40.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:40.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:40.045113+0000) 2019-10-13T13:23:41.278 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:23:41.279 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:41.282 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:23:41.282 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:42.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:42.750+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:42.753750+0000) 2019-10-13T13:23:44.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:44.348+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.354888+0000) 2019-10-13T13:23:44.357 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:44.350+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.355596+0000) 2019-10-13T13:23:44.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:44.352+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.355813+0000) 2019-10-13T13:23:44.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:44.352+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.356213+0000) 2019-10-13T13:23:44.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:44.352+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.356785+0000) 2019-10-13T13:23:44.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:44.352+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.356734+0000) 2019-10-13T13:23:44.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:44.352+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.357043+0000) 2019-10-13T13:23:44.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:44.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170862 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:44.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:44.358+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.364477+0000) 2019-10-13T13:23:45.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:45.576+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:45.577607+0000) 2019-10-13T13:23:45.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:45.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:45.803080+0000) 2019-10-13T13:23:45.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:45.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:45.863940+0000) 2019-10-13T13:23:46.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:46.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.545142+0000) 2019-10-13T13:23:48.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:48.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:48.811284+0000) 2019-10-13T13:23:48.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:48.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:48.813114+0000) 2019-10-13T13:23:49.368 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:49.360+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.366615+0000) 2019-10-13T13:23:49.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:49.366+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.367274+0000) 2019-10-13T13:23:49.369 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:49.364+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.367640+0000) 2019-10-13T13:23:49.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:49.364+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.368139+0000) 2019-10-13T13:23:49.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:49.364+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.368593+0000) 2019-10-13T13:23:49.371 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:49.364+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.368843+0000) 2019-10-13T13:23:49.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:49.364+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.369948+0000) 2019-10-13T13:23:49.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:49.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170900 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:49.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:49.374+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:49.375916+0000) 2019-10-13T13:23:50.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:50.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:50.045281+0000) 2019-10-13T13:23:52.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:52.758+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:52.760255+0000) 2019-10-13T13:23:54.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:54.372+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.378585+0000) 2019-10-13T13:23:54.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:54.372+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.379224+0000) 2019-10-13T13:23:54.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:54.376+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.379755+0000) 2019-10-13T13:23:54.382 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:54.376+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.380480+0000) 2019-10-13T13:23:54.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:54.376+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.381154+0000) 2019-10-13T13:23:54.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:54.378+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.381949+0000) 2019-10-13T13:23:54.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:54.376+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.382462+0000) 2019-10-13T13:23:54.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:54.380+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170931 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:54.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:54.386+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.388637+0000) 2019-10-13T13:23:55.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:55.572+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:55.577836+0000) 2019-10-13T13:23:55.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:55.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:55.803349+0000) 2019-10-13T13:23:55.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:55.860+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:55.864159+0000) 2019-10-13T13:23:56.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:56.543+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.545351+0000) 2019-10-13T13:23:58.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:58.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:58.811512+0000) 2019-10-13T13:23:58.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:58.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:58.813338+0000) 2019-10-13T13:23:59.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:23:59.383+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.390609+0000) 2019-10-13T13:23:59.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:23:59.389+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.391299+0000) 2019-10-13T13:23:59.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:23:59.387+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.391699+0000) 2019-10-13T13:23:59.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:23:59.387+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.392296+0000) 2019-10-13T13:23:59.395 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:23:59.387+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.393345+0000) 2019-10-13T13:23:59.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:23:59.387+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.393684+0000) 2019-10-13T13:23:59.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:23:59.387+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.394503+0000) 2019-10-13T13:23:59.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:23:59.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:23:59.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:23:59.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:59.401645+0000) 2019-10-13T13:24:00.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:00.041+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:00.045458+0000) 2019-10-13T13:24:02.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:02.761+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:02.766093+0000) 2019-10-13T13:24:04.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:04.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.403107+0000) 2019-10-13T13:24:04.405 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:04.399+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.403805+0000) 2019-10-13T13:24:04.406 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:04.399+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.404085+0000) 2019-10-13T13:24:04.407 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:04.399+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.404828+0000) 2019-10-13T13:24:04.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:04.401+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.405607+0000) 2019-10-13T13:24:04.407 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:04.399+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.405861+0000) 2019-10-13T13:24:04.408 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:04.399+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.406850+0000) 2019-10-13T13:24:04.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:04.405+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.410004+0000) 2019-10-13T13:24:04.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:04.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 170998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:05.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:05.575+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:05.578072+0000) 2019-10-13T13:24:05.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:05.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:05.803550+0000) 2019-10-13T13:24:05.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:05.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:05.864362+0000) 2019-10-13T13:24:06.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:06.539+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.545542+0000) 2019-10-13T13:24:08.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:08.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:08.811703+0000) 2019-10-13T13:24:08.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:08.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:08.813515+0000) 2019-10-13T13:24:09.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:09.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.414362+0000) 2019-10-13T13:24:09.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:09.407+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.414810+0000) 2019-10-13T13:24:09.417 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:09.411+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.415360+0000) 2019-10-13T13:24:09.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:09.411+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.415746+0000) 2019-10-13T13:24:09.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:09.411+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.416911+0000) 2019-10-13T13:24:09.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:09.411+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.417277+0000) 2019-10-13T13:24:09.419 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:09.411+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.417753+0000) 2019-10-13T13:24:09.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:09.421+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:09.423147+0000) 2019-10-13T13:24:09.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:09.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:10.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:10.041+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:10.045627+0000) 2019-10-13T13:24:11.410 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:24:11.411 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:11.415 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:24:11.415 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:12.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:12.769+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:12.772626+0000) 2019-10-13T13:24:14.429 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:14.423+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.427289+0000) 2019-10-13T13:24:14.430 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:14.423+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.427646+0000) 2019-10-13T13:24:14.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:14.425+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.428448+0000) 2019-10-13T13:24:14.430 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:14.423+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.428507+0000) 2019-10-13T13:24:14.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:14.423+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.429156+0000) 2019-10-13T13:24:14.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:14.423+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.429620+0000) 2019-10-13T13:24:14.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:14.423+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.430133+0000) 2019-10-13T13:24:14.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:14.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:14.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:14.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.440338+0000) 2019-10-13T13:24:15.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:15.571+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:15.578329+0000) 2019-10-13T13:24:15.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:15.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:15.803754+0000) 2019-10-13T13:24:15.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:15.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:15.864585+0000) 2019-10-13T13:24:16.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:16.543+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.545753+0000) 2019-10-13T13:24:18.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:18.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:18.811936+0000) 2019-10-13T13:24:18.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:18.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:18.813710+0000) 2019-10-13T13:24:19.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:19.435+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.439164+0000) 2019-10-13T13:24:19.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:19.435+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.439908+0000) 2019-10-13T13:24:19.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:19.435+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.441181+0000) 2019-10-13T13:24:19.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:19.435+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.441931+0000) 2019-10-13T13:24:19.444 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:19.435+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.442296+0000) 2019-10-13T13:24:19.444 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:19.435+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.442687+0000) 2019-10-13T13:24:19.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:19.441+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.443484+0000) 2019-10-13T13:24:19.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:19.445+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:19.448630+0000) 2019-10-13T13:24:19.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:19.443+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:20.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:20.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:20.045835+0000) 2019-10-13T13:24:22.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:22.777+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:22.779291+0000) 2019-10-13T13:24:24.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:24.447+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.452191+0000) 2019-10-13T13:24:24.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:24.447+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.453512+0000) 2019-10-13T13:24:24.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:24.449+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.453843+0000) 2019-10-13T13:24:24.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:24.447+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.454070+0000) 2019-10-13T13:24:24.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:24.447+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.454279+0000) 2019-10-13T13:24:24.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:24.447+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.454847+0000) 2019-10-13T13:24:24.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:24.451+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.455602+0000) 2019-10-13T13:24:24.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:24.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:24.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:24.461+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.463062+0000) 2019-10-13T13:24:25.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:25.575+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:25.578524+0000) 2019-10-13T13:24:25.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:25.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:25.803941+0000) 2019-10-13T13:24:25.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:25.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:25.864801+0000) 2019-10-13T13:24:26.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:26.543+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.545948+0000) 2019-10-13T13:24:28.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:28.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:28.812135+0000) 2019-10-13T13:24:28.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:28.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:28.813919+0000) 2019-10-13T13:24:29.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:29.463+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.463965+0000) 2019-10-13T13:24:29.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:29.463+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.464835+0000) 2019-10-13T13:24:29.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:29.461+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.465928+0000) 2019-10-13T13:24:29.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:29.463+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.466409+0000) 2019-10-13T13:24:29.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:29.463+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.466900+0000) 2019-10-13T13:24:29.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:29.467+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.467455+0000) 2019-10-13T13:24:29.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:29.467+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.467658+0000) 2019-10-13T13:24:29.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:29.469+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:29.471908+0000) 2019-10-13T13:24:29.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:29.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:30.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:30.041+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:30.046048+0000) 2019-10-13T13:24:32.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:32.785+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:32.785623+0000) 2019-10-13T13:24:34.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:34.475+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.476974+0000) 2019-10-13T13:24:34.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:34.475+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.478079+0000) 2019-10-13T13:24:34.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:34.477+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.478563+0000) 2019-10-13T13:24:34.481 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:34.475+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.478589+0000) 2019-10-13T13:24:34.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:34.475+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.479068+0000) 2019-10-13T13:24:34.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:34.479+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.479422+0000) 2019-10-13T13:24:34.482 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:34.479+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.480047+0000) 2019-10-13T13:24:34.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:34.481+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.484138+0000) 2019-10-13T13:24:34.486 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:34.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:35.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:35.575+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:35.578729+0000) 2019-10-13T13:24:35.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:35.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:35.804142+0000) 2019-10-13T13:24:35.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:35.863+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:35.865010+0000) 2019-10-13T13:24:36.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:36.543+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.546153+0000) 2019-10-13T13:24:38.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:38.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:38.812347+0000) 2019-10-13T13:24:38.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:38.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:38.814107+0000) 2019-10-13T13:24:39.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:39.482+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.486885+0000) 2019-10-13T13:24:39.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:39.486+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.487836+0000) 2019-10-13T13:24:39.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:39.486+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.488707+0000) 2019-10-13T13:24:39.491 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:39.485+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.489380+0000) 2019-10-13T13:24:39.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:39.486+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.489647+0000) 2019-10-13T13:24:39.492 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:39.486+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.490261+0000) 2019-10-13T13:24:39.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:39.486+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.490603+0000) 2019-10-13T13:24:39.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:39.494+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:39.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:39.493+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:39.496639+0000) 2019-10-13T13:24:40.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:40.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:40.046257+0000) 2019-10-13T13:24:41.542 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:24:41.542 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:41.546 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:24:41.546 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:42.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:42.789+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:42.792119+0000) 2019-10-13T13:24:44.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:44.497+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.499587+0000) 2019-10-13T13:24:44.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:44.498+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.499717+0000) 2019-10-13T13:24:44.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:44.498+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.500000+0000) 2019-10-13T13:24:44.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:44.498+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.500410+0000) 2019-10-13T13:24:44.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:44.498+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.501711+0000) 2019-10-13T13:24:44.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:44.498+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.502039+0000) 2019-10-13T13:24:44.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:44.498+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.502777+0000) 2019-10-13T13:24:44.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:44.505+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.507267+0000) 2019-10-13T13:24:44.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:44.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:45.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:45.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:45.579004+0000) 2019-10-13T13:24:45.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:45.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:45.804365+0000) 2019-10-13T13:24:45.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:45.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:45.865187+0000) 2019-10-13T13:24:46.548 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:46.542+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.546377+0000) 2019-10-13T13:24:48.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:48.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:48.812572+0000) 2019-10-13T13:24:48.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:48.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:48.814314+0000) 2019-10-13T13:24:49.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:49.510+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.511558+0000) 2019-10-13T13:24:49.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:49.510+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.511860+0000) 2019-10-13T13:24:49.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:49.509+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.512570+0000) 2019-10-13T13:24:49.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:49.510+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.513525+0000) 2019-10-13T13:24:49.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:49.510+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.513128+0000) 2019-10-13T13:24:49.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:49.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.513823+0000) 2019-10-13T13:24:49.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:49.510+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.515211+0000) 2019-10-13T13:24:49.522 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:49.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:49.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:49.517+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:49.521029+0000) 2019-10-13T13:24:50.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:50.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:50.046444+0000) 2019-10-13T13:24:52.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:52.796+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:52.798596+0000) 2019-10-13T13:24:54.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:54.522+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.523395+0000) 2019-10-13T13:24:54.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:54.522+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.523752+0000) 2019-10-13T13:24:54.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:54.522+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.524047+0000) 2019-10-13T13:24:54.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:54.522+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.524472+0000) 2019-10-13T13:24:54.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:54.522+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.526298+0000) 2019-10-13T13:24:54.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:54.524+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.527315+0000) 2019-10-13T13:24:54.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:54.526+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.527469+0000) 2019-10-13T13:24:54.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:54.528+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.533060+0000) 2019-10-13T13:24:54.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:54.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:55.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:55.574+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:55.579201+0000) 2019-10-13T13:24:55.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:55.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:55.804593+0000) 2019-10-13T13:24:55.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:55.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:55.865399+0000) 2019-10-13T13:24:56.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:56.542+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.546588+0000) 2019-10-13T13:24:58.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:58.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:58.812809+0000) 2019-10-13T13:24:58.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:58.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:58.814505+0000) 2019-10-13T13:24:59.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:24:59.534+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.536592+0000) 2019-10-13T13:24:59.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:24:59.534+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.537067+0000) 2019-10-13T13:24:59.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:24:59.534+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.537424+0000) 2019-10-13T13:24:59.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:24:59.536+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.538349+0000) 2019-10-13T13:24:59.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:24:59.534+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.538642+0000) 2019-10-13T13:24:59.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:24:59.534+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.538623+0000) 2019-10-13T13:24:59.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:24:59.534+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.539223+0000) 2019-10-13T13:24:59.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:24:59.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:24:59.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:24:59.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:59.546425+0000) 2019-10-13T13:25:00.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:00.044+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:00.046661+0000) 2019-10-13T13:25:02.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:02.804+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:02.805040+0000) 2019-10-13T13:25:04.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:04.546+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.548065+0000) 2019-10-13T13:25:04.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:04.544+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.548905+0000) 2019-10-13T13:25:04.551 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:04.546+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.548983+0000) 2019-10-13T13:25:04.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:04.546+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.549448+0000) 2019-10-13T13:25:04.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:04.546+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.549507+0000) 2019-10-13T13:25:04.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:04.546+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.549963+0000) 2019-10-13T13:25:04.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:04.546+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.550453+0000) 2019-10-13T13:25:04.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:04.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:04.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:04.552+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.556440+0000) 2019-10-13T13:25:05.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:05.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:05.579405+0000) 2019-10-13T13:25:05.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:05.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:05.804748+0000) 2019-10-13T13:25:05.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:05.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:05.865616+0000) 2019-10-13T13:25:06.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:06.546+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.546774+0000) 2019-10-13T13:25:08.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:08.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:08.813004+0000) 2019-10-13T13:25:08.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:08.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:08.814666+0000) 2019-10-13T13:25:09.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:09.558+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.559364+0000) 2019-10-13T13:25:09.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:09.558+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.559926+0000) 2019-10-13T13:25:09.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:09.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.560802+0000) 2019-10-13T13:25:09.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:09.558+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.561310+0000) 2019-10-13T13:25:09.564 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:09.558+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.561597+0000) 2019-10-13T13:25:09.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:09.558+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.561888+0000) 2019-10-13T13:25:09.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:09.558+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.562296+0000) 2019-10-13T13:25:09.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:09.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:09.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:09.564+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:09.569468+0000) 2019-10-13T13:25:10.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:10.044+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:10.046895+0000) 2019-10-13T13:25:11.657 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:25:11.657 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:11.661 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:25:11.661 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:12.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:12.808+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:12.811551+0000) 2019-10-13T13:25:14.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:14.570+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.571565+0000) 2019-10-13T13:25:14.574 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:14.568+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.572525+0000) 2019-10-13T13:25:14.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:14.570+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.572991+0000) 2019-10-13T13:25:14.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:14.570+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.573322+0000) 2019-10-13T13:25:14.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:14.570+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.574023+0000) 2019-10-13T13:25:14.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:14.570+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.574670+0000) 2019-10-13T13:25:14.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:14.574+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.575704+0000) 2019-10-13T13:25:14.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:14.576+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.579860+0000) 2019-10-13T13:25:14.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:14.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:15.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:15.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:15.579623+0000) 2019-10-13T13:25:15.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:15.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:15.804945+0000) 2019-10-13T13:25:15.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:15.862+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:15.865848+0000) 2019-10-13T13:25:16.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:16.542+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.546995+0000) 2019-10-13T13:25:18.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:18.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:18.813197+0000) 2019-10-13T13:25:18.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:18.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:18.814813+0000) 2019-10-13T13:25:19.586 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:19.582+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.584415+0000) 2019-10-13T13:25:19.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:19.582+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.585072+0000) 2019-10-13T13:25:19.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:19.580+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.585870+0000) 2019-10-13T13:25:19.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:19.582+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.586340+0000) 2019-10-13T13:25:19.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:19.582+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.586751+0000) 2019-10-13T13:25:19.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:19.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.587456+0000) 2019-10-13T13:25:19.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:19.586+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.587863+0000) 2019-10-13T13:25:19.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:19.590+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:19.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:19.588+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:19.593615+0000) 2019-10-13T13:25:20.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:20.044+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:20.047071+0000) 2019-10-13T13:25:22.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:22.812+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:22.817940+0000) 2019-10-13T13:25:24.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:24.593+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.596438+0000) 2019-10-13T13:25:24.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:24.593+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.597680+0000) 2019-10-13T13:25:24.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:24.593+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.598022+0000) 2019-10-13T13:25:24.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:24.593+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.598414+0000) 2019-10-13T13:25:24.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:24.596+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.599662+0000) 2019-10-13T13:25:24.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:24.597+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.600050+0000) 2019-10-13T13:25:24.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:24.597+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.600388+0000) 2019-10-13T13:25:24.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:24.600+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.605692+0000) 2019-10-13T13:25:24.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:24.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:25.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:25.577+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:25.579820+0000) 2019-10-13T13:25:25.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:25.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:25.805167+0000) 2019-10-13T13:25:25.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:25.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:25.866088+0000) 2019-10-13T13:25:26.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:26.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.547204+0000) 2019-10-13T13:25:28.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:28.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:28.813437+0000) 2019-10-13T13:25:28.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:28.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:28.815028+0000) 2019-10-13T13:25:29.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:29.605+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.609478+0000) 2019-10-13T13:25:29.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:29.608+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.610382+0000) 2019-10-13T13:25:29.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:29.605+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.610347+0000) 2019-10-13T13:25:29.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:29.605+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.610687+0000) 2019-10-13T13:25:29.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:29.609+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.611445+0000) 2019-10-13T13:25:29.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:29.609+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.612220+0000) 2019-10-13T13:25:29.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:29.609+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.613050+0000) 2019-10-13T13:25:29.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:29.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:29.620 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:29.616+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:29.618267+0000) 2019-10-13T13:25:30.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:30.044+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:30.047273+0000) 2019-10-13T13:25:32.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:32.820+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:32.824231+0000) 2019-10-13T13:25:34.623 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:34.616+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.621752+0000) 2019-10-13T13:25:34.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:34.617+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.621361+0000) 2019-10-13T13:25:34.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:34.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.622124+0000) 2019-10-13T13:25:34.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:34.617+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.622484+0000) 2019-10-13T13:25:34.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:34.617+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.622826+0000) 2019-10-13T13:25:34.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:34.621+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.623972+0000) 2019-10-13T13:25:34.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:34.621+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.624074+0000) 2019-10-13T13:25:34.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:34.624+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.630028+0000) 2019-10-13T13:25:34.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:34.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:35.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:35.577+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:35.580020+0000) 2019-10-13T13:25:35.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:35.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:35.805391+0000) 2019-10-13T13:25:35.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:35.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:35.866320+0000) 2019-10-13T13:25:36.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:36.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.547417+0000) 2019-10-13T13:25:38.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:38.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:38.813634+0000) 2019-10-13T13:25:38.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:38.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:38.815223+0000) 2019-10-13T13:25:39.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:39.628+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.633543+0000) 2019-10-13T13:25:39.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:39.629+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.633999+0000) 2019-10-13T13:25:39.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:39.629+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.635016+0000) 2019-10-13T13:25:39.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:39.629+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.635176+0000) 2019-10-13T13:25:39.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:39.633+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.635526+0000) 2019-10-13T13:25:39.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:39.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.635862+0000) 2019-10-13T13:25:39.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:39.633+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.637015+0000) 2019-10-13T13:25:39.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:39.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171647 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:39.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:39.640+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:39.643763+0000) 2019-10-13T13:25:40.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:40.044+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:40.047474+0000) 2019-10-13T13:25:41.796 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:25:41.796 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:41.800 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:25:41.800 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:42.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:42.832+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:42.835576+0000) 2019-10-13T13:25:44.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:44.641+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.646282+0000) 2019-10-13T13:25:44.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:44.641+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.646628+0000) 2019-10-13T13:25:44.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:44.644+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.647256+0000) 2019-10-13T13:25:44.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:44.645+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.647597+0000) 2019-10-13T13:25:44.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:44.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.648139+0000) 2019-10-13T13:25:44.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:44.645+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.648849+0000) 2019-10-13T13:25:44.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:44.645+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.649164+0000) 2019-10-13T13:25:44.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:44.652+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.654487+0000) 2019-10-13T13:25:44.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:44.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:45.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:45.577+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:45.580212+0000) 2019-10-13T13:25:45.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:45.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:45.805609+0000) 2019-10-13T13:25:45.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:45.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:45.866522+0000) 2019-10-13T13:25:46.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:46.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.547608+0000) 2019-10-13T13:25:48.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:48.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:48.813859+0000) 2019-10-13T13:25:48.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:48.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:48.815450+0000) 2019-10-13T13:25:49.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:49.653+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.658209+0000) 2019-10-13T13:25:49.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:49.653+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.658788+0000) 2019-10-13T13:25:49.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:49.655+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.659537+0000) 2019-10-13T13:25:49.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:49.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.659671+0000) 2019-10-13T13:25:49.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:49.657+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.660775+0000) 2019-10-13T13:25:49.663 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:49.657+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.661285+0000) 2019-10-13T13:25:49.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:49.657+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.661643+0000) 2019-10-13T13:25:49.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:49.659+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:49.665633+0000) 2019-10-13T13:25:49.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:49.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171715 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:50.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:50.043+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:50.047629+0000) 2019-10-13T13:25:52.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:52.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:52.842196+0000) 2019-10-13T13:25:54.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:54.665+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.669862+0000) 2019-10-13T13:25:54.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:54.665+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.670511+0000) 2019-10-13T13:25:54.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:54.669+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.671839+0000) 2019-10-13T13:25:54.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:54.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.672468+0000) 2019-10-13T13:25:54.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:54.669+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.672695+0000) 2019-10-13T13:25:54.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:54.669+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.673165+0000) 2019-10-13T13:25:54.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:54.669+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.673435+0000) 2019-10-13T13:25:54.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:54.671+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.677858+0000) 2019-10-13T13:25:54.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:54.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:55.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:55.577+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:55.580385+0000) 2019-10-13T13:25:55.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:55.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:55.805805+0000) 2019-10-13T13:25:55.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:55.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:55.866742+0000) 2019-10-13T13:25:56.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:56.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.547800+0000) 2019-10-13T13:25:58.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:58.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:58.814003+0000) 2019-10-13T13:25:58.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:58.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:58.815628+0000) 2019-10-13T13:25:59.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:25:59.677+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.681915+0000) 2019-10-13T13:25:59.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:25:59.679+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.682439+0000) 2019-10-13T13:25:59.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:25:59.677+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.682907+0000) 2019-10-13T13:25:59.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:25:59.681+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.683395+0000) 2019-10-13T13:25:59.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:25:59.681+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.683771+0000) 2019-10-13T13:25:59.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:25:59.681+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.684247+0000) 2019-10-13T13:25:59.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:25:59.681+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.684486+0000) 2019-10-13T13:25:59.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:25:59.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171782 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:25:59.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:25:59.687+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:59.691240+0000) 2019-10-13T13:26:00.049 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:00.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:00.047791+0000) 2019-10-13T13:26:02.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:02.847+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:02.848685+0000) 2019-10-13T13:26:04.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:04.689+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.693186+0000) 2019-10-13T13:26:04.696 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:04.689+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.693736+0000) 2019-10-13T13:26:04.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:04.691+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.694383+0000) 2019-10-13T13:26:04.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:04.689+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.694372+0000) 2019-10-13T13:26:04.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:04.693+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.695644+0000) 2019-10-13T13:26:04.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:04.693+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.696385+0000) 2019-10-13T13:26:04.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:04.693+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.696673+0000) 2019-10-13T13:26:04.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:04.695+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.701689+0000) 2019-10-13T13:26:04.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:04.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:05.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:05.576+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:05.580555+0000) 2019-10-13T13:26:05.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:05.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:05.805970+0000) 2019-10-13T13:26:05.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:05.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:05.866940+0000) 2019-10-13T13:26:06.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:06.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.547992+0000) 2019-10-13T13:26:08.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:08.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:08.814214+0000) 2019-10-13T13:26:08.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:08.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:08.815844+0000) 2019-10-13T13:26:09.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:09.700+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.705521+0000) 2019-10-13T13:26:09.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:09.700+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.705954+0000) 2019-10-13T13:26:09.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:09.704+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.707555+0000) 2019-10-13T13:26:09.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:09.704+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.707917+0000) 2019-10-13T13:26:09.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:09.703+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.708572+0000) 2019-10-13T13:26:09.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:09.704+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.708639+0000) 2019-10-13T13:26:09.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:09.704+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.709803+0000) 2019-10-13T13:26:09.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:09.707+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:09.713330+0000) 2019-10-13T13:26:09.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:09.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171851 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:10.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:10.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:10.047968+0000) 2019-10-13T13:26:11.861 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:26:11.861 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:11.867 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:26:11.867 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:12.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:12.859+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:12.860459+0000) 2019-10-13T13:26:14.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:14.712+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.717062+0000) 2019-10-13T13:26:14.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:14.715+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.718990+0000) 2019-10-13T13:26:14.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:14.712+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.719040+0000) 2019-10-13T13:26:14.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:14.716+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.719596+0000) 2019-10-13T13:26:14.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:14.716+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.719571+0000) 2019-10-13T13:26:14.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:14.716+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.720163+0000) 2019-10-13T13:26:14.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:14.716+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.721340+0000) 2019-10-13T13:26:14.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:14.719+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.723216+0000) 2019-10-13T13:26:14.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:14.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:15.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:15.576+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:15.580731+0000) 2019-10-13T13:26:15.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:15.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:15.806186+0000) 2019-10-13T13:26:15.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:15.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:15.867138+0000) 2019-10-13T13:26:16.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:16.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.548203+0000) 2019-10-13T13:26:18.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:18.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:18.814431+0000) 2019-10-13T13:26:18.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:18.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:18.816040+0000) 2019-10-13T13:26:19.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:19.724+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.729388+0000) 2019-10-13T13:26:19.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:19.724+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.729954+0000) 2019-10-13T13:26:19.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:19.724+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.730635+0000) 2019-10-13T13:26:19.733 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:19.724+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.731240+0000) 2019-10-13T13:26:19.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:19.727+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.732026+0000) 2019-10-13T13:26:19.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:19.728+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.732801+0000) 2019-10-13T13:26:19.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:19.728+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.733175+0000) 2019-10-13T13:26:19.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:19.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:19.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:19.735+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:19.739112+0000) 2019-10-13T13:26:20.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:20.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:20.048164+0000) 2019-10-13T13:26:22.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:22.863+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:22.866420+0000) 2019-10-13T13:26:24.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:24.736+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.742051+0000) 2019-10-13T13:26:24.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:24.736+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.743273+0000) 2019-10-13T13:26:24.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:24.740+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.743992+0000) 2019-10-13T13:26:24.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:24.740+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.744277+0000) 2019-10-13T13:26:24.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:24.740+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.744668+0000) 2019-10-13T13:26:24.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:24.739+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.745242+0000) 2019-10-13T13:26:24.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:24.740+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.745366+0000) 2019-10-13T13:26:24.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:24.743+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.749617+0000) 2019-10-13T13:26:24.753 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:24.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:25.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:25.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:25.580908+0000) 2019-10-13T13:26:25.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:25.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:25.806421+0000) 2019-10-13T13:26:25.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:25.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:25.867358+0000) 2019-10-13T13:26:26.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:26.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.548400+0000) 2019-10-13T13:26:28.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:28.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:28.814667+0000) 2019-10-13T13:26:28.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:28.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:28.816213+0000) 2019-10-13T13:26:29.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:29.748+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.754792+0000) 2019-10-13T13:26:29.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:29.752+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.755348+0000) 2019-10-13T13:26:29.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:29.752+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.755615+0000) 2019-10-13T13:26:29.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:29.752+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.756468+0000) 2019-10-13T13:26:29.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:29.751+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.757034+0000) 2019-10-13T13:26:29.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:29.752+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.757650+0000) 2019-10-13T13:26:29.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:29.752+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.757997+0000) 2019-10-13T13:26:29.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:29.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 171987 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:29.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:29.759+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:29.764219+0000) 2019-10-13T13:26:30.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:30.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:30.048333+0000) 2019-10-13T13:26:32.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:32.871+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:32.872955+0000) 2019-10-13T13:26:34.768 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:34.763+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.766276+0000) 2019-10-13T13:26:34.769 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:34.760+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.766860+0000) 2019-10-13T13:26:34.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:34.764+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.767480+0000) 2019-10-13T13:26:34.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:34.764+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.768055+0000) 2019-10-13T13:26:34.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:34.764+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.768197+0000) 2019-10-13T13:26:34.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:34.760+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.767134+0000) 2019-10-13T13:26:34.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:34.764+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.769059+0000) 2019-10-13T13:26:34.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:34.768+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:34.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:34.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.775434+0000) 2019-10-13T13:26:35.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:35.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:35.581105+0000) 2019-10-13T13:26:35.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:35.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:35.806641+0000) 2019-10-13T13:26:35.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:35.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:35.867563+0000) 2019-10-13T13:26:36.550 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:36.548+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.548600+0000) 2019-10-13T13:26:38.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:38.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:38.814889+0000) 2019-10-13T13:26:38.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:38.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:38.816408+0000) 2019-10-13T13:26:39.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:39.772+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.777983+0000) 2019-10-13T13:26:39.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:39.772+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.778441+0000) 2019-10-13T13:26:39.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:39.772+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.779051+0000) 2019-10-13T13:26:39.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:39.776+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.779576+0000) 2019-10-13T13:26:39.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:39.774+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.780699+0000) 2019-10-13T13:26:39.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:39.776+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.781025+0000) 2019-10-13T13:26:39.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:39.776+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.781193+0000) 2019-10-13T13:26:39.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:39.780+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:39.790 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:39.782+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:39.788634+0000) 2019-10-13T13:26:40.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:40.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:40.048516+0000) 2019-10-13T13:26:41.984 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:26:41.984 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:41.988 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:26:41.988 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:42.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:42.878+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:42.879081+0000) 2019-10-13T13:26:44.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:44.784+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.789734+0000) 2019-10-13T13:26:44.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:44.786+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.791594+0000) 2019-10-13T13:26:44.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:44.788+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.792168+0000) 2019-10-13T13:26:44.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:44.788+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.792613+0000) 2019-10-13T13:26:44.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:44.788+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.792921+0000) 2019-10-13T13:26:44.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:44.788+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.793245+0000) 2019-10-13T13:26:44.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:44.788+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.793561+0000) 2019-10-13T13:26:44.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:44.794+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.798352+0000) 2019-10-13T13:26:44.801 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:44.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:45.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:45.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:45.581314+0000) 2019-10-13T13:26:45.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:45.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:45.806850+0000) 2019-10-13T13:26:45.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:45.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:45.867761+0000) 2019-10-13T13:26:46.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:46.544+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.548818+0000) 2019-10-13T13:26:48.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:48.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:48.815110+0000) 2019-10-13T13:26:48.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:48.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:48.816607+0000) 2019-10-13T13:26:49.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:49.795+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.802095+0000) 2019-10-13T13:26:49.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:49.799+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.804161+0000) 2019-10-13T13:26:49.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:49.799+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.804889+0000) 2019-10-13T13:26:49.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:49.799+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.804981+0000) 2019-10-13T13:26:49.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:49.798+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.805743+0000) 2019-10-13T13:26:49.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:49.799+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.805881+0000) 2019-10-13T13:26:49.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:49.799+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.806263+0000) 2019-10-13T13:26:49.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:49.806+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:49.810575+0000) 2019-10-13T13:26:49.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:49.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:50.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:50.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:50.048700+0000) 2019-10-13T13:26:52.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:52.882+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:52.885515+0000) 2019-10-13T13:26:54.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:54.807+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.814378+0000) 2019-10-13T13:26:54.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:54.810+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.816137+0000) 2019-10-13T13:26:54.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:54.811+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.816489+0000) 2019-10-13T13:26:54.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:54.811+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.817244+0000) 2019-10-13T13:26:54.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:54.811+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.817050+0000) 2019-10-13T13:26:54.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:54.811+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.817773+0000) 2019-10-13T13:26:54.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:54.811+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.818369+0000) 2019-10-13T13:26:54.825 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:54.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:54.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:54.818+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.825637+0000) 2019-10-13T13:26:55.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:55.579+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:55.581491+0000) 2019-10-13T13:26:55.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:55.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:55.807071+0000) 2019-10-13T13:26:55.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:55.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:55.867959+0000) 2019-10-13T13:26:56.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:56.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.549012+0000) 2019-10-13T13:26:58.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:58.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:58.815328+0000) 2019-10-13T13:26:58.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:58.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:58.816803+0000) 2019-10-13T13:26:59.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:26:59.823+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.827204+0000) 2019-10-13T13:26:59.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:26:59.823+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.827699+0000) 2019-10-13T13:26:59.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:26:59.823+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.828336+0000) 2019-10-13T13:26:59.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:26:59.823+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.828867+0000) 2019-10-13T13:26:59.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:26:59.823+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.829564+0000) 2019-10-13T13:26:59.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:26:59.826+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.830268+0000) 2019-10-13T13:26:59.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:26:59.823+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.830312+0000) 2019-10-13T13:26:59.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:26:59.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:26:59.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:26:59.834+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:59.840792+0000) 2019-10-13T13:27:00.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:00.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:00.048915+0000) 2019-10-13T13:27:02.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:02.890+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:02.891922+0000) 2019-10-13T13:27:04.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:04.835+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.839807+0000) 2019-10-13T13:27:04.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:04.835+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.840302+0000) 2019-10-13T13:27:04.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:04.835+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.840544+0000) 2019-10-13T13:27:04.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:04.835+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.841718+0000) 2019-10-13T13:27:04.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:04.835+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.842246+0000) 2019-10-13T13:27:04.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:04.835+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.842439+0000) 2019-10-13T13:27:04.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:04.838+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.844097+0000) 2019-10-13T13:27:04.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:04.842+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.847135+0000) 2019-10-13T13:27:04.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:04.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:05.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:05.575+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:05.581674+0000) 2019-10-13T13:27:05.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:05.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:05.807249+0000) 2019-10-13T13:27:05.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:05.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:05.868131+0000) 2019-10-13T13:27:06.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:06.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.549217+0000) 2019-10-13T13:27:08.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:08.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:08.815546+0000) 2019-10-13T13:27:08.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:08.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:08.817009+0000) 2019-10-13T13:27:09.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:09.847+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.851572+0000) 2019-10-13T13:27:09.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:09.847+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.852282+0000) 2019-10-13T13:27:09.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:09.846+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.852788+0000) 2019-10-13T13:27:09.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:09.847+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.853060+0000) 2019-10-13T13:27:09.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:09.847+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.853256+0000) 2019-10-13T13:27:09.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:09.847+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.854173+0000) 2019-10-13T13:27:09.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:09.847+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.854731+0000) 2019-10-13T13:27:09.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:09.854+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:09.860239+0000) 2019-10-13T13:27:09.862 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:09.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:10.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:10.042+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:10.049091+0000) 2019-10-13T13:27:12.047 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:27:12.047 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:12.051 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:27:12.051 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:12.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:12.894+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:12.898256+0000) 2019-10-13T13:27:14.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:14.859+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.863868+0000) 2019-10-13T13:27:14.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:14.859+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.864247+0000) 2019-10-13T13:27:14.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:14.859+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.864549+0000) 2019-10-13T13:27:14.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:14.859+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.865380+0000) 2019-10-13T13:27:14.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:14.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.865912+0000) 2019-10-13T13:27:14.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:14.859+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.866363+0000) 2019-10-13T13:27:14.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:14.862+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.867011+0000) 2019-10-13T13:27:14.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:14.866+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.870994+0000) 2019-10-13T13:27:14.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:14.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:15.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:15.575+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:15.581856+0000) 2019-10-13T13:27:15.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:15.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:15.807425+0000) 2019-10-13T13:27:15.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:15.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:15.868305+0000) 2019-10-13T13:27:16.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:16.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.549410+0000) 2019-10-13T13:27:18.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:18.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:18.815741+0000) 2019-10-13T13:27:18.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:18.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:18.817201+0000) 2019-10-13T13:27:19.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:19.875+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.875296+0000) 2019-10-13T13:27:19.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:19.875+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.875705+0000) 2019-10-13T13:27:19.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:19.875+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.877008+0000) 2019-10-13T13:27:19.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:19.875+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.877837+0000) 2019-10-13T13:27:19.880 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:19.875+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.878015+0000) 2019-10-13T13:27:19.880 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:19.875+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.878683+0000) 2019-10-13T13:27:19.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:19.874+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.879590+0000) 2019-10-13T13:27:19.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:19.883+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:19.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:19.878+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:19.885006+0000) 2019-10-13T13:27:20.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:20.046+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:20.049261+0000) 2019-10-13T13:27:22.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:22.898+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:22.904750+0000) 2019-10-13T13:27:24.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:24.887+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.887361+0000) 2019-10-13T13:27:24.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:24.887+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.887983+0000) 2019-10-13T13:27:24.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:24.887+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.888310+0000) 2019-10-13T13:27:24.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:24.887+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.888658+0000) 2019-10-13T13:27:24.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:24.882+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.889836+0000) 2019-10-13T13:27:24.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:24.887+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.889788+0000) 2019-10-13T13:27:24.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:24.887+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.891122+0000) 2019-10-13T13:27:24.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:24.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:24.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:24.890+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.897134+0000) 2019-10-13T13:27:25.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:25.579+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:25.582065+0000) 2019-10-13T13:27:25.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:25.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:25.807598+0000) 2019-10-13T13:27:25.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:25.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:25.868529+0000) 2019-10-13T13:27:26.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:26.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.549598+0000) 2019-10-13T13:27:28.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:28.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:28.815935+0000) 2019-10-13T13:27:28.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:28.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:28.817390+0000) 2019-10-13T13:27:29.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:29.895+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.899169+0000) 2019-10-13T13:27:29.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:29.899+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.900038+0000) 2019-10-13T13:27:29.902 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:29.899+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.900619+0000) 2019-10-13T13:27:29.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:29.899+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.901226+0000) 2019-10-13T13:27:29.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:29.899+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.901623+0000) 2019-10-13T13:27:29.904 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:29.899+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.901992+0000) 2019-10-13T13:27:29.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:29.902+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.902813+0000) 2019-10-13T13:27:29.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:29.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:29.911 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:29.906+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:29.909789+0000) 2019-10-13T13:27:30.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:30.046+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:30.049439+0000) 2019-10-13T13:27:32.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:32.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:32.911172+0000) 2019-10-13T13:27:34.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:34.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.911103+0000) 2019-10-13T13:27:34.914 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:34.910+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.912599+0000) 2019-10-13T13:27:34.915 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:34.910+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.912925+0000) 2019-10-13T13:27:34.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:34.910+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.913347+0000) 2019-10-13T13:27:34.916 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:34.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.913828+0000) 2019-10-13T13:27:34.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:34.910+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.914923+0000) 2019-10-13T13:27:34.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:34.913+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.915709+0000) 2019-10-13T13:27:34.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:34.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:34.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:34.921+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.923203+0000) 2019-10-13T13:27:35.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:35.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:35.582265+0000) 2019-10-13T13:27:35.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:35.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:35.807817+0000) 2019-10-13T13:27:35.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:35.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:35.868722+0000) 2019-10-13T13:27:36.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:36.546+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.549790+0000) 2019-10-13T13:27:38.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:38.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:38.816133+0000) 2019-10-13T13:27:38.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:38.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:38.817582+0000) 2019-10-13T13:27:39.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:39.918+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.923132+0000) 2019-10-13T13:27:39.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:39.922+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.924064+0000) 2019-10-13T13:27:39.926 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:39.922+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.924480+0000) 2019-10-13T13:27:39.927 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:39.922+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.924934+0000) 2019-10-13T13:27:39.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:39.922+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.926044+0000) 2019-10-13T13:27:39.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:39.922+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.925792+0000) 2019-10-13T13:27:39.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:39.925+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.927061+0000) 2019-10-13T13:27:39.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:39.929+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:39.930191+0000) 2019-10-13T13:27:39.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:39.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:40.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:40.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:40.049642+0000) 2019-10-13T13:27:42.106 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:27:42.106 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:42.109 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:27:42.110 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:42.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:42.921+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:42.922663+0000) 2019-10-13T13:27:44.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:44.934+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.936168+0000) 2019-10-13T13:27:44.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:44.934+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.937050+0000) 2019-10-13T13:27:44.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:44.934+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.937755+0000) 2019-10-13T13:27:44.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:44.934+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.938204+0000) 2019-10-13T13:27:44.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:44.937+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.938933+0000) 2019-10-13T13:27:44.941 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:44.934+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.938935+0000) 2019-10-13T13:27:44.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:44.938+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.939468+0000) 2019-10-13T13:27:44.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:44.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:44.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:44.941+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.946043+0000) 2019-10-13T13:27:45.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:45.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:45.582446+0000) 2019-10-13T13:27:45.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:45.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:45.807996+0000) 2019-10-13T13:27:45.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:45.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:45.868869+0000) 2019-10-13T13:27:46.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:46.546+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.549979+0000) 2019-10-13T13:27:48.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:48.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:48.816354+0000) 2019-10-13T13:27:48.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:48.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:48.817766+0000) 2019-10-13T13:27:49.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:49.946+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.948424+0000) 2019-10-13T13:27:49.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:49.945+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.949235+0000) 2019-10-13T13:27:49.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:49.946+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.949555+0000) 2019-10-13T13:27:49.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:49.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.950008+0000) 2019-10-13T13:27:49.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:49.946+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.951015+0000) 2019-10-13T13:27:49.953 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:49.950+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.951592+0000) 2019-10-13T13:27:49.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:49.950+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.952076+0000) 2019-10-13T13:27:49.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:49.953+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:49.956952+0000) 2019-10-13T13:27:49.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:49.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:50.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:50.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:50.049795+0000) 2019-10-13T13:27:52.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:52.925+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:52.929217+0000) 2019-10-13T13:27:54.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:54.957+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.960644+0000) 2019-10-13T13:27:54.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:54.958+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.960683+0000) 2019-10-13T13:27:54.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:54.958+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.961198+0000) 2019-10-13T13:27:54.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:54.958+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.962231+0000) 2019-10-13T13:27:54.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:54.958+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.962695+0000) 2019-10-13T13:27:54.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:54.962+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.963697+0000) 2019-10-13T13:27:54.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:54.962+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.964620+0000) 2019-10-13T13:27:54.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:54.969+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.970585+0000) 2019-10-13T13:27:54.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:54.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:27:55.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:55.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:55.582640+0000) 2019-10-13T13:27:55.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:55.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:55.808189+0000) 2019-10-13T13:27:55.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:55.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:55.869055+0000) 2019-10-13T13:27:56.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:56.546+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.550206+0000) 2019-10-13T13:27:58.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:58.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:58.816567+0000) 2019-10-13T13:27:58.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:58.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:58.817964+0000) 2019-10-13T13:27:59.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:27:59.970+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.974391+0000) 2019-10-13T13:27:59.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:27:59.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.975360+0000) 2019-10-13T13:27:59.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:27:59.973+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.975939+0000) 2019-10-13T13:27:59.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:27:59.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.976024+0000) 2019-10-13T13:27:59.978 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:27:59.974+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.976679+0000) 2019-10-13T13:27:59.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:27:59.974+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.976583+0000) 2019-10-13T13:27:59.979 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:27:59.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.977638+0000) 2019-10-13T13:27:59.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:27:59.977+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:59.980321+0000) 2019-10-13T13:27:59.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:27:59.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:00.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:00.045+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:00.049979+0000) 2019-10-13T13:28:02.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:02.933+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:02.935630+0000) 2019-10-13T13:28:04.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:04.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.985977+0000) 2019-10-13T13:28:04.988 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:04.982+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.986009+0000) 2019-10-13T13:28:04.988 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:04.982+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.986470+0000) 2019-10-13T13:28:04.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:04.982+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.986885+0000) 2019-10-13T13:28:04.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:04.986+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.987483+0000) 2019-10-13T13:28:04.990 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:04.986+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.987987+0000) 2019-10-13T13:28:04.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:04.986+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.988584+0000) 2019-10-13T13:28:04.997 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:04.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:04.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:04.993+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.996143+0000) 2019-10-13T13:28:05.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:05.578+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:05.582832+0000) 2019-10-13T13:28:05.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:05.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:05.808404+0000) 2019-10-13T13:28:05.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:05.866+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:05.869241+0000) 2019-10-13T13:28:06.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:06.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.550401+0000) 2019-10-13T13:28:08.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:08.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:08.816762+0000) 2019-10-13T13:28:08.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:08.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:08.818186+0000) 2019-10-13T13:28:10.000 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:09.994+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:09.998176+0000) 2019-10-13T13:28:10.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:09.997+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:09.999284+0000) 2019-10-13T13:28:10.001 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:09.998+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:09.999553+0000) 2019-10-13T13:28:10.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:09.998+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.000041+0000) 2019-10-13T13:28:10.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:09.998+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.001446+0000) 2019-10-13T13:28:10.003 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:09.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.001922+0000) 2019-10-13T13:28:10.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:09.998+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.002552+0000) 2019-10-13T13:28:10.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:10.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.007026+0000) 2019-10-13T13:28:10.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:10.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:10.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:10.049+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:10.050185+0000) 2019-10-13T13:28:12.235 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:28:12.235 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:12.239 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:28:12.239 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:12.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:12.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:12.941951+0000) 2019-10-13T13:28:15.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:15.005+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.009958+0000) 2019-10-13T13:28:15.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:15.009+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.011375+0000) 2019-10-13T13:28:15.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:15.009+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.012243+0000) 2019-10-13T13:28:15.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:15.009+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.011574+0000) 2019-10-13T13:28:15.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:15.009+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.012311+0000) 2019-10-13T13:28:15.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:15.009+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.013075+0000) 2019-10-13T13:28:15.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:15.009+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.013321+0000) 2019-10-13T13:28:15.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:15.017+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.019594+0000) 2019-10-13T13:28:15.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:15.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:15.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:15.581+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.583009+0000) 2019-10-13T13:28:15.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:15.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.808639+0000) 2019-10-13T13:28:15.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:15.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:15.869455+0000) 2019-10-13T13:28:16.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:16.549+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.550597+0000) 2019-10-13T13:28:18.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:18.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:18.816990+0000) 2019-10-13T13:28:18.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:18.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:18.818378+0000) 2019-10-13T13:28:20.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:20.017+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.021868+0000) 2019-10-13T13:28:20.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:20.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.023802+0000) 2019-10-13T13:28:20.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:20.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.023915+0000) 2019-10-13T13:28:20.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:20.021+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.024646+0000) 2019-10-13T13:28:20.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:20.021+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.024491+0000) 2019-10-13T13:28:20.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:20.021+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.024776+0000) 2019-10-13T13:28:20.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:20.021+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.025623+0000) 2019-10-13T13:28:20.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:20.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:20.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:20.029+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.033755+0000) 2019-10-13T13:28:20.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:20.049+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:20.050339+0000) 2019-10-13T13:28:22.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:22.945+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:22.948220+0000) 2019-10-13T13:28:25.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:25.029+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.034485+0000) 2019-10-13T13:28:25.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:25.029+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.035085+0000) 2019-10-13T13:28:25.038 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:25.033+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.036282+0000) 2019-10-13T13:28:25.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:25.033+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.035853+0000) 2019-10-13T13:28:25.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:25.033+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.037044+0000) 2019-10-13T13:28:25.039 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:25.033+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.037112+0000) 2019-10-13T13:28:25.040 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:25.033+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.038260+0000) 2019-10-13T13:28:25.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:25.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:25.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:25.041+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.044607+0000) 2019-10-13T13:28:25.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:25.581+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.583212+0000) 2019-10-13T13:28:25.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:25.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.808818+0000) 2019-10-13T13:28:25.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:25.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:25.869662+0000) 2019-10-13T13:28:26.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:26.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.550817+0000) 2019-10-13T13:28:28.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:28.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:28.817184+0000) 2019-10-13T13:28:28.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:28.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:28.818567+0000) 2019-10-13T13:28:30.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:30.041+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.046653+0000) 2019-10-13T13:28:30.049 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:30.041+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.047252+0000) 2019-10-13T13:28:30.049 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:30.045+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.047449+0000) 2019-10-13T13:28:30.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:30.045+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.047877+0000) 2019-10-13T13:28:30.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:30.044+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.048535+0000) 2019-10-13T13:28:30.051 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:30.045+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.048678+0000) 2019-10-13T13:28:30.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:30.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.048908+0000) 2019-10-13T13:28:30.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:30.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.050509+0000) 2019-10-13T13:28:30.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:30.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:30.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:30.052+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:30.057044+0000) 2019-10-13T13:28:32.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:32.952+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:32.954483+0000) 2019-10-13T13:28:35.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:35.053+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.058775+0000) 2019-10-13T13:28:35.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:35.057+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.059472+0000) 2019-10-13T13:28:35.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:35.057+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.059790+0000) 2019-10-13T13:28:35.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:35.057+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.060380+0000) 2019-10-13T13:28:35.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:35.056+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.061121+0000) 2019-10-13T13:28:35.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:35.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.061858+0000) 2019-10-13T13:28:35.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:35.057+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.062172+0000) 2019-10-13T13:28:35.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:35.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172827 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:35.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:35.064+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.068181+0000) 2019-10-13T13:28:35.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:35.581+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.583414+0000) 2019-10-13T13:28:35.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:35.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.808994+0000) 2019-10-13T13:28:35.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:35.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:35.869860+0000) 2019-10-13T13:28:36.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:36.545+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.551030+0000) 2019-10-13T13:28:38.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:38.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:38.817380+0000) 2019-10-13T13:28:38.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:38.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:38.818782+0000) 2019-10-13T13:28:40.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:40.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.050711+0000) 2019-10-13T13:28:40.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:40.065+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.070089+0000) 2019-10-13T13:28:40.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:40.065+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.070580+0000) 2019-10-13T13:28:40.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:40.069+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.071284+0000) 2019-10-13T13:28:40.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:40.069+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.071907+0000) 2019-10-13T13:28:40.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:40.069+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.072965+0000) 2019-10-13T13:28:40.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:40.069+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.073751+0000) 2019-10-13T13:28:40.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:40.072+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.074385+0000) 2019-10-13T13:28:40.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:40.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:40.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:40.076+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:40.081425+0000) 2019-10-13T13:28:42.300 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:28:42.300 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:42.304 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:28:42.304 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:42.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:42.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:42.960915+0000) 2019-10-13T13:28:45.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:45.081+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.083747+0000) 2019-10-13T13:28:45.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:45.081+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.084214+0000) 2019-10-13T13:28:45.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:45.081+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.084896+0000) 2019-10-13T13:28:45.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:45.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.085250+0000) 2019-10-13T13:28:45.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:45.081+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.085711+0000) 2019-10-13T13:28:45.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:45.081+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.086173+0000) 2019-10-13T13:28:45.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:45.084+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.087102+0000) 2019-10-13T13:28:45.093 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:45.088+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.091394+0000) 2019-10-13T13:28:45.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:45.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:45.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:45.581+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.583544+0000) 2019-10-13T13:28:45.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:45.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.809215+0000) 2019-10-13T13:28:45.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:45.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:45.870056+0000) 2019-10-13T13:28:46.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:46.549+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.551243+0000) 2019-10-13T13:28:48.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:48.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:48.817607+0000) 2019-10-13T13:28:48.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:48.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:48.818981+0000) 2019-10-13T13:28:50.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:50.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.050899+0000) 2019-10-13T13:28:50.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:50.089+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.093858+0000) 2019-10-13T13:28:50.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:50.089+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.094518+0000) 2019-10-13T13:28:50.096 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:50.089+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.094632+0000) 2019-10-13T13:28:50.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:50.089+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.095174+0000) 2019-10-13T13:28:50.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:50.093+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.097147+0000) 2019-10-13T13:28:50.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:50.093+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.097183+0000) 2019-10-13T13:28:50.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:50.092+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.098001+0000) 2019-10-13T13:28:50.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:50.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:50.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:50.100+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:50.102785+0000) 2019-10-13T13:28:52.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:52.964+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:52.967411+0000) 2019-10-13T13:28:55.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:55.097+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.103075+0000) 2019-10-13T13:28:55.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:55.101+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.103641+0000) 2019-10-13T13:28:55.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:28:55.100+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.105162+0000) 2019-10-13T13:28:55.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:55.101+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.105396+0000) 2019-10-13T13:28:55.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:55.101+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.105804+0000) 2019-10-13T13:28:55.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:55.101+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.106285+0000) 2019-10-13T13:28:55.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:55.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.106544+0000) 2019-10-13T13:28:55.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:28:55.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172960 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:28:55.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:28:55.112+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.116454+0000) 2019-10-13T13:28:55.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:28:55.581+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.583740+0000) 2019-10-13T13:28:55.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:28:55.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.809388+0000) 2019-10-13T13:28:55.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:28:55.865+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:55.870268+0000) 2019-10-13T13:28:56.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:28:56.549+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.551443+0000) 2019-10-13T13:28:58.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:28:58.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:58.817833+0000) 2019-10-13T13:28:58.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:28:58.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:58.819174+0000) 2019-10-13T13:29:00.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:00.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.051135+0000) 2019-10-13T13:29:00.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:00.112+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.116222+0000) 2019-10-13T13:29:00.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:00.112+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.116705+0000) 2019-10-13T13:29:00.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:00.112+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.118423+0000) 2019-10-13T13:29:00.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:00.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.119129+0000) 2019-10-13T13:29:00.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:00.112+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.119176+0000) 2019-10-13T13:29:00.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:00.116+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.119523+0000) 2019-10-13T13:29:00.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:00.116+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.119787+0000) 2019-10-13T13:29:00.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:00.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 172994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:00.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:00.124+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:00.126460+0000) 2019-10-13T13:29:02.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:02.968+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:02.973769+0000) 2019-10-13T13:29:05.129 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:05.120+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.127258+0000) 2019-10-13T13:29:05.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:05.124+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.128042+0000) 2019-10-13T13:29:05.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:05.124+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.128180+0000) 2019-10-13T13:29:05.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:05.124+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.129285+0000) 2019-10-13T13:29:05.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:05.124+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.129838+0000) 2019-10-13T13:29:05.132 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:05.124+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.130577+0000) 2019-10-13T13:29:05.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:05.124+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.131105+0000) 2019-10-13T13:29:05.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:05.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.136417+0000) 2019-10-13T13:29:05.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:05.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:05.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:05.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.583918+0000) 2019-10-13T13:29:05.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:05.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.809525+0000) 2019-10-13T13:29:05.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:05.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:05.870473+0000) 2019-10-13T13:29:06.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:06.548+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.551643+0000) 2019-10-13T13:29:08.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:08.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:08.818027+0000) 2019-10-13T13:29:08.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:08.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:08.819375+0000) 2019-10-13T13:29:10.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:10.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.051334+0000) 2019-10-13T13:29:10.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:10.136+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.141288+0000) 2019-10-13T13:29:10.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:10.136+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.141249+0000) 2019-10-13T13:29:10.144 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:10.136+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.141512+0000) 2019-10-13T13:29:10.144 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:10.136+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.142363+0000) 2019-10-13T13:29:10.144 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:10.136+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.142162+0000) 2019-10-13T13:29:10.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:10.136+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.142904+0000) 2019-10-13T13:29:10.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:10.140+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.143284+0000) 2019-10-13T13:29:10.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:10.144+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:10.147605+0000) 2019-10-13T13:29:10.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:10.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:12.415 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:29:12.415 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:12.418 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:29:12.419 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:12.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:12.980+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:12.980262+0000) 2019-10-13T13:29:15.154 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:15.148+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.152098+0000) 2019-10-13T13:29:15.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:15.148+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.152540+0000) 2019-10-13T13:29:15.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:15.148+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.153731+0000) 2019-10-13T13:29:15.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:15.148+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.154264+0000) 2019-10-13T13:29:15.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:15.152+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.154738+0000) 2019-10-13T13:29:15.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:15.148+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.154737+0000) 2019-10-13T13:29:15.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:15.148+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.155117+0000) 2019-10-13T13:29:15.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:15.156+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.159923+0000) 2019-10-13T13:29:15.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:15.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:15.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:15.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.584105+0000) 2019-10-13T13:29:15.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:15.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.809709+0000) 2019-10-13T13:29:15.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:15.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:15.870685+0000) 2019-10-13T13:29:16.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:16.548+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.551858+0000) 2019-10-13T13:29:18.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:18.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:18.818222+0000) 2019-10-13T13:29:18.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:18.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:18.819571+0000) 2019-10-13T13:29:20.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:20.048+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.051570+0000) 2019-10-13T13:29:20.166 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:20.160+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.164537+0000) 2019-10-13T13:29:20.168 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:20.160+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.165921+0000) 2019-10-13T13:29:20.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:20.160+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.165741+0000) 2019-10-13T13:29:20.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:20.160+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.166155+0000) 2019-10-13T13:29:20.169 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:20.164+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.166928+0000) 2019-10-13T13:29:20.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:20.160+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.166903+0000) 2019-10-13T13:29:20.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:20.164+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.168519+0000) 2019-10-13T13:29:20.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:20.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:20.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:20.172+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:20.174278+0000) 2019-10-13T13:29:22.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:22.983+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:22.986682+0000) 2019-10-13T13:29:25.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:25.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.177569+0000) 2019-10-13T13:29:25.180 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:25.172+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.177809+0000) 2019-10-13T13:29:25.181 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:25.172+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.178037+0000) 2019-10-13T13:29:25.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:25.175+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.179230+0000) 2019-10-13T13:29:25.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:25.172+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.178656+0000) 2019-10-13T13:29:25.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:25.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.179451+0000) 2019-10-13T13:29:25.183 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:25.176+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.180480+0000) 2019-10-13T13:29:25.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:25.179+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.183664+0000) 2019-10-13T13:29:25.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:25.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:25.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:25.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.584283+0000) 2019-10-13T13:29:25.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:25.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.809873+0000) 2019-10-13T13:29:25.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:25.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:25.870868+0000) 2019-10-13T13:29:26.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:26.548+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.552074+0000) 2019-10-13T13:29:28.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:28.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:28.818415+0000) 2019-10-13T13:29:28.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:28.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:28.819766+0000) 2019-10-13T13:29:30.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:30.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.051774+0000) 2019-10-13T13:29:30.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:30.184+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.189264+0000) 2019-10-13T13:29:30.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:30.184+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.189725+0000) 2019-10-13T13:29:30.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:30.184+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.190315+0000) 2019-10-13T13:29:30.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:30.184+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.190963+0000) 2019-10-13T13:29:30.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:30.184+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.191030+0000) 2019-10-13T13:29:30.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:30.188+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.191908+0000) 2019-10-13T13:29:30.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:30.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.192648+0000) 2019-10-13T13:29:30.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:30.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:30.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:30.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:30.198328+0000) 2019-10-13T13:29:32.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:32.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:32.993003+0000) 2019-10-13T13:29:35.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:35.195+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.201413+0000) 2019-10-13T13:29:35.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:35.196+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.202548+0000) 2019-10-13T13:29:35.205 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:35.196+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.203127+0000) 2019-10-13T13:29:35.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:35.200+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.203459+0000) 2019-10-13T13:29:35.206 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:35.200+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.203843+0000) 2019-10-13T13:29:35.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:35.200+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.204143+0000) 2019-10-13T13:29:35.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:35.200+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.204451+0000) 2019-10-13T13:29:35.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:35.203+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.208852+0000) 2019-10-13T13:29:35.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:35.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:35.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:35.584+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.584467+0000) 2019-10-13T13:29:35.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:35.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.810043+0000) 2019-10-13T13:29:35.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:35.864+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:35.871053+0000) 2019-10-13T13:29:36.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:36.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.552323+0000) 2019-10-13T13:29:38.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:38.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:38.818627+0000) 2019-10-13T13:29:38.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:38.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:38.819954+0000) 2019-10-13T13:29:40.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:40.051+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.051970+0000) 2019-10-13T13:29:40.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:40.208+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.212889+0000) 2019-10-13T13:29:40.216 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:40.211+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.214306+0000) 2019-10-13T13:29:40.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:40.208+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.215139+0000) 2019-10-13T13:29:40.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:40.212+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.215804+0000) 2019-10-13T13:29:40.218 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:40.212+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.215896+0000) 2019-10-13T13:29:40.218 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:40.212+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.216123+0000) 2019-10-13T13:29:40.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:40.212+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.216765+0000) 2019-10-13T13:29:40.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:40.215+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:40.221645+0000) 2019-10-13T13:29:40.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:40.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:42.522 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:29:42.523 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:42.526 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:29:42.526 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:43.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:42.995+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.999451+0000) 2019-10-13T13:29:45.227 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:45.219+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.225993+0000) 2019-10-13T13:29:45.228 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:45.219+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.226144+0000) 2019-10-13T13:29:45.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:45.219+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.226704+0000) 2019-10-13T13:29:45.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:45.219+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.226867+0000) 2019-10-13T13:29:45.230 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:45.223+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.228068+0000) 2019-10-13T13:29:45.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:45.223+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.228287+0000) 2019-10-13T13:29:45.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:45.223+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.228861+0000) 2019-10-13T13:29:45.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:45.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:45.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:45.231+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.237407+0000) 2019-10-13T13:29:45.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:45.583+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.584644+0000) 2019-10-13T13:29:45.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:45.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.810216+0000) 2019-10-13T13:29:45.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:45.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:45.871239+0000) 2019-10-13T13:29:46.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:46.551+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:46.552517+0000) 2019-10-13T13:29:48.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:48.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:48.818832+0000) 2019-10-13T13:29:48.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:48.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:48.820150+0000) 2019-10-13T13:29:50.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:50.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.052183+0000) 2019-10-13T13:29:50.239 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:50.231+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.236911+0000) 2019-10-13T13:29:50.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:50.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.237584+0000) 2019-10-13T13:29:50.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:50.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.238268+0000) 2019-10-13T13:29:50.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:50.231+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.238873+0000) 2019-10-13T13:29:50.241 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:50.235+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.239574+0000) 2019-10-13T13:29:50.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:50.235+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.240208+0000) 2019-10-13T13:29:50.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:50.235+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.239917+0000) 2019-10-13T13:29:50.247 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:50.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:50.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:50.243+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:50.248254+0000) 2019-10-13T13:29:53.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:52.999+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:53.005868+0000) 2019-10-13T13:29:55.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:29:55.243+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.248209+0000) 2019-10-13T13:29:55.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:55.243+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.248573+0000) 2019-10-13T13:29:55.251 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:55.243+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.248992+0000) 2019-10-13T13:29:55.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:55.243+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.250364+0000) 2019-10-13T13:29:55.253 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:55.243+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.250698+0000) 2019-10-13T13:29:55.253 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:55.247+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.251417+0000) 2019-10-13T13:29:55.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:55.247+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.251565+0000) 2019-10-13T13:29:55.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:29:55.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:29:55.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:29:55.255+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.258620+0000) 2019-10-13T13:29:55.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:29:55.583+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.584847+0000) 2019-10-13T13:29:55.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:29:55.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.810397+0000) 2019-10-13T13:29:55.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:29:55.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:55.871423+0000) 2019-10-13T13:29:56.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:29:56.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:56.552740+0000) 2019-10-13T13:29:58.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:29:58.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:58.819029+0000) 2019-10-13T13:29:58.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:29:58.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:58.820350+0000) 2019-10-13T13:30:00.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:00.047+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.052389+0000) 2019-10-13T13:30:00.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:00.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.260562+0000) 2019-10-13T13:30:00.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:00.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.260946+0000) 2019-10-13T13:30:00.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:00.255+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.261688+0000) 2019-10-13T13:30:00.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:00.259+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.262338+0000) 2019-10-13T13:30:00.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:00.255+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.262513+0000) 2019-10-13T13:30:00.265 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:00.255+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.263044+0000) 2019-10-13T13:30:00.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:00.259+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.264127+0000) 2019-10-13T13:30:00.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:00.263+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:00.269374+0000) 2019-10-13T13:30:00.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:00.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:03.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:03.007+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:03.011812+0000) 2019-10-13T13:30:05.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:05.267+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.273634+0000) 2019-10-13T13:30:05.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:05.267+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.274617+0000) 2019-10-13T13:30:05.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:05.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.275040+0000) 2019-10-13T13:30:05.278 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:05.271+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.276250+0000) 2019-10-13T13:30:05.278 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:05.271+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.276277+0000) 2019-10-13T13:30:05.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:05.271+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.276972+0000) 2019-10-13T13:30:05.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:05.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.277984+0000) 2019-10-13T13:30:05.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:05.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:05.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:05.279+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.283792+0000) 2019-10-13T13:30:05.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:05.583+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.585032+0000) 2019-10-13T13:30:05.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:05.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.810597+0000) 2019-10-13T13:30:05.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:05.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:05.871578+0000) 2019-10-13T13:30:06.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:06.547+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:06.552933+0000) 2019-10-13T13:30:08.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:08.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:08.819221+0000) 2019-10-13T13:30:08.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:08.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:08.820543+0000) 2019-10-13T13:30:10.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:10.051+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.052583+0000) 2019-10-13T13:30:10.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:10.279+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.285420+0000) 2019-10-13T13:30:10.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:10.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.286823+0000) 2019-10-13T13:30:10.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:10.283+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.287776+0000) 2019-10-13T13:30:10.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:10.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.288025+0000) 2019-10-13T13:30:10.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:10.283+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.288662+0000) 2019-10-13T13:30:10.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:10.283+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.289551+0000) 2019-10-13T13:30:10.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:10.283+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.289924+0000) 2019-10-13T13:30:10.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:10.287+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:10.293042+0000) 2019-10-13T13:30:10.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:10.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:12.579 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:30:12.579 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:12.583 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:30:12.583 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:13.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:13.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:13.018173+0000) 2019-10-13T13:30:15.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:15.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.298188+0000) 2019-10-13T13:30:15.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:15.295+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.299590+0000) 2019-10-13T13:30:15.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:15.299+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.299630+0000) 2019-10-13T13:30:15.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:15.299+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.300127+0000) 2019-10-13T13:30:15.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:15.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.300267+0000) 2019-10-13T13:30:15.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:15.299+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.301403+0000) 2019-10-13T13:30:15.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:15.299+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.301731+0000) 2019-10-13T13:30:15.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:15.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:15.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:15.303+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.306916+0000) 2019-10-13T13:30:15.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:15.583+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.585212+0000) 2019-10-13T13:30:15.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:15.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.810773+0000) 2019-10-13T13:30:15.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:15.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:15.871759+0000) 2019-10-13T13:30:16.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:16.551+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:16.553169+0000) 2019-10-13T13:30:18.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:18.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:18.819414+0000) 2019-10-13T13:30:18.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:18.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:18.820764+0000) 2019-10-13T13:30:20.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:20.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.052808+0000) 2019-10-13T13:30:20.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:20.307+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.310286+0000) 2019-10-13T13:30:20.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:20.307+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.310795+0000) 2019-10-13T13:30:20.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:20.311+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.311418+0000) 2019-10-13T13:30:20.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:20.311+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.311654+0000) 2019-10-13T13:30:20.314 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:20.311+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.312117+0000) 2019-10-13T13:30:20.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:20.306+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.312249+0000) 2019-10-13T13:30:20.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:20.311+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.313204+0000) 2019-10-13T13:30:20.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:20.314+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:20.318814+0000) 2019-10-13T13:30:20.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:20.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:23.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:23.022+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:23.024542+0000) 2019-10-13T13:30:25.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:25.318+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.322602+0000) 2019-10-13T13:30:25.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:25.318+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.323177+0000) 2019-10-13T13:30:25.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:25.322+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.323369+0000) 2019-10-13T13:30:25.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:25.322+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.324083+0000) 2019-10-13T13:30:25.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:25.322+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.324483+0000) 2019-10-13T13:30:25.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:25.322+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.325058+0000) 2019-10-13T13:30:25.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:25.322+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.325170+0000) 2019-10-13T13:30:25.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:25.326+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.332122+0000) 2019-10-13T13:30:25.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:25.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:25.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:25.582+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.585396+0000) 2019-10-13T13:30:25.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:25.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.810949+0000) 2019-10-13T13:30:25.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:25.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:25.871956+0000) 2019-10-13T13:30:26.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:26.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:26.553358+0000) 2019-10-13T13:30:28.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:28.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:28.819646+0000) 2019-10-13T13:30:28.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:28.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:28.820959+0000) 2019-10-13T13:30:30.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:30.046+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.053022+0000) 2019-10-13T13:30:30.337 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:30.334+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.335582+0000) 2019-10-13T13:30:30.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:30.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.336012+0000) 2019-10-13T13:30:30.338 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:30.334+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.336780+0000) 2019-10-13T13:30:30.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:30.334+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.337343+0000) 2019-10-13T13:30:30.339 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:30.334+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.337849+0000) 2019-10-13T13:30:30.340 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:30.334+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.338471+0000) 2019-10-13T13:30:30.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:30.334+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.339260+0000) 2019-10-13T13:30:30.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:30.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:30.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:30.338+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:30.344500+0000) 2019-10-13T13:30:33.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:33.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:33.030857+0000) 2019-10-13T13:30:35.350 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:35.346+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.348092+0000) 2019-10-13T13:30:35.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:35.346+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.348750+0000) 2019-10-13T13:30:35.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:35.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.348922+0000) 2019-10-13T13:30:35.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:35.342+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.349716+0000) 2019-10-13T13:30:35.352 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:35.346+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.349604+0000) 2019-10-13T13:30:35.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:35.346+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.349939+0000) 2019-10-13T13:30:35.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:35.346+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.350709+0000) 2019-10-13T13:30:35.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:35.350+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.354878+0000) 2019-10-13T13:30:35.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:35.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:35.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:35.582+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.585598+0000) 2019-10-13T13:30:35.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:35.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.811122+0000) 2019-10-13T13:30:35.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:35.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:35.872109+0000) 2019-10-13T13:30:36.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:36.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:36.553610+0000) 2019-10-13T13:30:38.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:38.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:38.819841+0000) 2019-10-13T13:30:38.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:38.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:38.821149+0000) 2019-10-13T13:30:40.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:40.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.053214+0000) 2019-10-13T13:30:40.361 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:40.354+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.359199+0000) 2019-10-13T13:30:40.363 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:40.354+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.361314+0000) 2019-10-13T13:30:40.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:40.358+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.360906+0000) 2019-10-13T13:30:40.364 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:40.358+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.361515+0000) 2019-10-13T13:30:40.364 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:40.358+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.361714+0000) 2019-10-13T13:30:40.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:40.358+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.362407+0000) 2019-10-13T13:30:40.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:40.362+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.363686+0000) 2019-10-13T13:30:40.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:40.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:40.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:40.362+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:40.370049+0000) 2019-10-13T13:30:42.625 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:30:42.626 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:42.629 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:30:42.629 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:43.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:43.034+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:43.037585+0000) 2019-10-13T13:30:45.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:45.370+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.371892+0000) 2019-10-13T13:30:45.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:45.370+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.372298+0000) 2019-10-13T13:30:45.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:45.366+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.373892+0000) 2019-10-13T13:30:45.376 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:45.370+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.374049+0000) 2019-10-13T13:30:45.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:45.370+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.374640+0000) 2019-10-13T13:30:45.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:45.370+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.374869+0000) 2019-10-13T13:30:45.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:45.374+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.375716+0000) 2019-10-13T13:30:45.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:45.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173680 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:45.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:45.378+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.384243+0000) 2019-10-13T13:30:45.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:45.582+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.585766+0000) 2019-10-13T13:30:45.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:45.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.811350+0000) 2019-10-13T13:30:45.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:45.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:45.872347+0000) 2019-10-13T13:30:46.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:46.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:46.553817+0000) 2019-10-13T13:30:48.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:48.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:48.820063+0000) 2019-10-13T13:30:48.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:48.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:48.821357+0000) 2019-10-13T13:30:50.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:50.046+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.053420+0000) 2019-10-13T13:30:50.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:50.378+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.385466+0000) 2019-10-13T13:30:50.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:50.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.385738+0000) 2019-10-13T13:30:50.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:50.382+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.385808+0000) 2019-10-13T13:30:50.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:50.382+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.386203+0000) 2019-10-13T13:30:50.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:50.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.386958+0000) 2019-10-13T13:30:50.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:50.386+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.387417+0000) 2019-10-13T13:30:50.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:50.386+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.388005+0000) 2019-10-13T13:30:50.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:50.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:50.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:50.386+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:50.393467+0000) 2019-10-13T13:30:53.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:53.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:53.044091+0000) 2019-10-13T13:30:55.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:55.394+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.396171+0000) 2019-10-13T13:30:55.399 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:55.394+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.396324+0000) 2019-10-13T13:30:55.399 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:55.394+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.397674+0000) 2019-10-13T13:30:55.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:55.394+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.398817+0000) 2019-10-13T13:30:55.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:30:55.394+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.399774+0000) 2019-10-13T13:30:55.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:55.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.399951+0000) 2019-10-13T13:30:55.402 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:55.394+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.397930+0000) 2019-10-13T13:30:55.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:30:55.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:30:55.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:30:55.398+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.404787+0000) 2019-10-13T13:30:55.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:30:55.582+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.585961+0000) 2019-10-13T13:30:55.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:30:55.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.811525+0000) 2019-10-13T13:30:55.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:30:55.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:55.872532+0000) 2019-10-13T13:30:56.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:30:56.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:56.554019+0000) 2019-10-13T13:30:58.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:30:58.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:58.820290+0000) 2019-10-13T13:30:58.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:30:58.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:58.821551+0000) 2019-10-13T13:31:00.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:00.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.053647+0000) 2019-10-13T13:31:00.411 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:00.406+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.409245+0000) 2019-10-13T13:31:00.412 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:00.406+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.410151+0000) 2019-10-13T13:31:00.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:00.406+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.411107+0000) 2019-10-13T13:31:00.413 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:00.410+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.411399+0000) 2019-10-13T13:31:00.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:00.406+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.412227+0000) 2019-10-13T13:31:00.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:00.410+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.412241+0000) 2019-10-13T13:31:00.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:00.410+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.412847+0000) 2019-10-13T13:31:00.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:00.410+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:00.415039+0000) 2019-10-13T13:31:00.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:00.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173778 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:03.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:03.050+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:03.050499+0000) 2019-10-13T13:31:05.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:05.418+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.420332+0000) 2019-10-13T13:31:05.423 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:05.418+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.420703+0000) 2019-10-13T13:31:05.423 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:05.418+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.421574+0000) 2019-10-13T13:31:05.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:05.418+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.421943+0000) 2019-10-13T13:31:05.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:05.422+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.423087+0000) 2019-10-13T13:31:05.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:05.422+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.423297+0000) 2019-10-13T13:31:05.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:05.422+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.423990+0000) 2019-10-13T13:31:05.430 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:05.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:05.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:05.426+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.428746+0000) 2019-10-13T13:31:05.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:05.582+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.586168+0000) 2019-10-13T13:31:05.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:05.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.811675+0000) 2019-10-13T13:31:05.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:05.870+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:05.872735+0000) 2019-10-13T13:31:06.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:06.550+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:06.554214+0000) 2019-10-13T13:31:08.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:08.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:08.820486+0000) 2019-10-13T13:31:08.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:08.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:08.821756+0000) 2019-10-13T13:31:10.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:10.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.053842+0000) 2019-10-13T13:31:10.433 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:10.425+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.430979+0000) 2019-10-13T13:31:10.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:10.429+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.432218+0000) 2019-10-13T13:31:10.435 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:10.429+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.432589+0000) 2019-10-13T13:31:10.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:10.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.433802+0000) 2019-10-13T13:31:10.436 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:10.429+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.434172+0000) 2019-10-13T13:31:10.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:10.434+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.434974+0000) 2019-10-13T13:31:10.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:10.429+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.434643+0000) 2019-10-13T13:31:10.440 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:10.438+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:10.438369+0000) 2019-10-13T13:31:10.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:10.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:12.745 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:31:12.745 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:12.749 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:31:12.749 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:13.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:13.053+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:13.056991+0000) 2019-10-13T13:31:15.446 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:15.441+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.443781+0000) 2019-10-13T13:31:15.447 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:15.441+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.444337+0000) 2019-10-13T13:31:15.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:15.441+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.445252+0000) 2019-10-13T13:31:15.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:15.441+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.445365+0000) 2019-10-13T13:31:15.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:15.441+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.446214+0000) 2019-10-13T13:31:15.449 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:15.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.446027+0000) 2019-10-13T13:31:15.450 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:15.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.447992+0000) 2019-10-13T13:31:15.454 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:15.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:15.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:15.449+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.453428+0000) 2019-10-13T13:31:15.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:15.585+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.586406+0000) 2019-10-13T13:31:15.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:15.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.811818+0000) 2019-10-13T13:31:15.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:15.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:15.872949+0000) 2019-10-13T13:31:16.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:16.553+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:16.554414+0000) 2019-10-13T13:31:18.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:18.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:18.820683+0000) 2019-10-13T13:31:18.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:18.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:18.821952+0000) 2019-10-13T13:31:20.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:20.049+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.054081+0000) 2019-10-13T13:31:20.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:20.453+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.456587+0000) 2019-10-13T13:31:20.459 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:20.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.456885+0000) 2019-10-13T13:31:20.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:20.453+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.457661+0000) 2019-10-13T13:31:20.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:20.453+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.457812+0000) 2019-10-13T13:31:20.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:20.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.458332+0000) 2019-10-13T13:31:20.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:20.453+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.458613+0000) 2019-10-13T13:31:20.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:20.457+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.459512+0000) 2019-10-13T13:31:20.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:20.461+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:20.462850+0000) 2019-10-13T13:31:20.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:20.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:23.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:23.061+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:23.063482+0000) 2019-10-13T13:31:25.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:25.461+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.467184+0000) 2019-10-13T13:31:25.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:25.465+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.467685+0000) 2019-10-13T13:31:25.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:25.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.467814+0000) 2019-10-13T13:31:25.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:25.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.468606+0000) 2019-10-13T13:31:25.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:25.465+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.469768+0000) 2019-10-13T13:31:25.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:25.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.470061+0000) 2019-10-13T13:31:25.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:25.465+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.470975+0000) 2019-10-13T13:31:25.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:25.473+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.475011+0000) 2019-10-13T13:31:25.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:25.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:25.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:25.585+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.586644+0000) 2019-10-13T13:31:25.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:25.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.811949+0000) 2019-10-13T13:31:25.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:25.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:25.873136+0000) 2019-10-13T13:31:26.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:26.553+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:26.554639+0000) 2019-10-13T13:31:28.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:28.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:28.820914+0000) 2019-10-13T13:31:28.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:28.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:28.822141+0000) 2019-10-13T13:31:30.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:30.053+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.054268+0000) 2019-10-13T13:31:30.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:30.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.479883+0000) 2019-10-13T13:31:30.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:30.477+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.480496+0000) 2019-10-13T13:31:30.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:30.477+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.480733+0000) 2019-10-13T13:31:30.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:30.477+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.481541+0000) 2019-10-13T13:31:30.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:30.477+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.481923+0000) 2019-10-13T13:31:30.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:30.477+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.482345+0000) 2019-10-13T13:31:30.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:30.477+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.483116+0000) 2019-10-13T13:31:30.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:30.485+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:30.486570+0000) 2019-10-13T13:31:30.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:30.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 173976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:33.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:33.065+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:33.069406+0000) 2019-10-13T13:31:35.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:35.489+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.493862+0000) 2019-10-13T13:31:35.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:35.489+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.494212+0000) 2019-10-13T13:31:35.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:35.489+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.494474+0000) 2019-10-13T13:31:35.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:35.493+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.495580+0000) 2019-10-13T13:31:35.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:35.493+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.495940+0000) 2019-10-13T13:31:35.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:35.493+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.496928+0000) 2019-10-13T13:31:35.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:35.493+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.497450+0000) 2019-10-13T13:31:35.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:35.497+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.500793+0000) 2019-10-13T13:31:35.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:35.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:35.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:35.585+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.586823+0000) 2019-10-13T13:31:35.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:35.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.812162+0000) 2019-10-13T13:31:35.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:35.869+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:35.873379+0000) 2019-10-13T13:31:36.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:36.549+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:36.554827+0000) 2019-10-13T13:31:38.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:38.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:38.821110+0000) 2019-10-13T13:31:38.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:38.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:38.822364+0000) 2019-10-13T13:31:40.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:40.053+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.054467+0000) 2019-10-13T13:31:40.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:40.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.506187+0000) 2019-10-13T13:31:40.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:40.501+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.506504+0000) 2019-10-13T13:31:40.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:40.505+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.507986+0000) 2019-10-13T13:31:40.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:40.505+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.507413+0000) 2019-10-13T13:31:40.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:40.505+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.508131+0000) 2019-10-13T13:31:40.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:40.505+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.508786+0000) 2019-10-13T13:31:40.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:40.505+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.509008+0000) 2019-10-13T13:31:40.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:40.509+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:40.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:40.513+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:40.514588+0000) 2019-10-13T13:31:42.798 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:31:42.798 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:42.802 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:31:42.802 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:43.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:43.077+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:43.079631+0000) 2019-10-13T13:31:45.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:45.513+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.517333+0000) 2019-10-13T13:31:45.520 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:45.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.518396+0000) 2019-10-13T13:31:45.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:45.517+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.519133+0000) 2019-10-13T13:31:45.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:45.513+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.519113+0000) 2019-10-13T13:31:45.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:45.517+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.520404+0000) 2019-10-13T13:31:45.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:45.517+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.520470+0000) 2019-10-13T13:31:45.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:45.517+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.520804+0000) 2019-10-13T13:31:45.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:45.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:45.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:45.521+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.526053+0000) 2019-10-13T13:31:45.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:45.585+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.587015+0000) 2019-10-13T13:31:45.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:45.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.812346+0000) 2019-10-13T13:31:45.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:45.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:45.873585+0000) 2019-10-13T13:31:46.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:46.549+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:46.555033+0000) 2019-10-13T13:31:48.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:48.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:48.821336+0000) 2019-10-13T13:31:48.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:48.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:48.822559+0000) 2019-10-13T13:31:50.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:50.053+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.054652+0000) 2019-10-13T13:31:50.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:50.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.529031+0000) 2019-10-13T13:31:50.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:50.524+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.529579+0000) 2019-10-13T13:31:50.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:50.524+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.530365+0000) 2019-10-13T13:31:50.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:50.524+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.529884+0000) 2019-10-13T13:31:50.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:50.528+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.531509+0000) 2019-10-13T13:31:50.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:50.528+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.531775+0000) 2019-10-13T13:31:50.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:50.528+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.532376+0000) 2019-10-13T13:31:50.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:50.533+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:50.535699+0000) 2019-10-13T13:31:50.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:50.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174108 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:53.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:53.081+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:53.085967+0000) 2019-10-13T13:31:55.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:55.536+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.541164+0000) 2019-10-13T13:31:55.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:55.536+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.541776+0000) 2019-10-13T13:31:55.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:55.536+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.542184+0000) 2019-10-13T13:31:55.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:55.536+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.542646+0000) 2019-10-13T13:31:55.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:31:55.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.543665+0000) 2019-10-13T13:31:55.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:55.540+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.543605+0000) 2019-10-13T13:31:55.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:55.540+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.544152+0000) 2019-10-13T13:31:55.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:31:55.545+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.549685+0000) 2019-10-13T13:31:55.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:31:55.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:31:55.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:31:55.580+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.587223+0000) 2019-10-13T13:31:55.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:31:55.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.812568+0000) 2019-10-13T13:31:55.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:31:55.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:55.873771+0000) 2019-10-13T13:31:56.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:31:56.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:56.555243+0000) 2019-10-13T13:31:58.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:31:58.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:58.821524+0000) 2019-10-13T13:31:58.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:31:58.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:58.822712+0000) 2019-10-13T13:32:00.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:00.053+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.054861+0000) 2019-10-13T13:32:00.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:00.548+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.554594+0000) 2019-10-13T13:32:00.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:00.548+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.555187+0000) 2019-10-13T13:32:00.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:00.552+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.555823+0000) 2019-10-13T13:32:00.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:00.552+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.555488+0000) 2019-10-13T13:32:00.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:00.552+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.556078+0000) 2019-10-13T13:32:00.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:00.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.556644+0000) 2019-10-13T13:32:00.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:00.552+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.557429+0000) 2019-10-13T13:32:00.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:00.553+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:00.558045+0000) 2019-10-13T13:32:00.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:00.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:03.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:03.089+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:03.092011+0000) 2019-10-13T13:32:05.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:05.560+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.565386+0000) 2019-10-13T13:32:05.568 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:05.565+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.566984+0000) 2019-10-13T13:32:05.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:05.564+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.567699+0000) 2019-10-13T13:32:05.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:05.564+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.567970+0000) 2019-10-13T13:32:05.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:05.564+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.568445+0000) 2019-10-13T13:32:05.571 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:05.564+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.568971+0000) 2019-10-13T13:32:05.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:05.564+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.569100+0000) 2019-10-13T13:32:05.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:05.569+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.573737+0000) 2019-10-13T13:32:05.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:05.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:05.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:05.584+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.587410+0000) 2019-10-13T13:32:05.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:05.812+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.812744+0000) 2019-10-13T13:32:05.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:05.868+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:05.873956+0000) 2019-10-13T13:32:06.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:06.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:06.555473+0000) 2019-10-13T13:32:08.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:08.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:08.821752+0000) 2019-10-13T13:32:08.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:08.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:08.822943+0000) 2019-10-13T13:32:10.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:10.052+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.055101+0000) 2019-10-13T13:32:10.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:10.572+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.578316+0000) 2019-10-13T13:32:10.581 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:10.576+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.578986+0000) 2019-10-13T13:32:10.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:10.576+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.579467+0000) 2019-10-13T13:32:10.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:10.576+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.579879+0000) 2019-10-13T13:32:10.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:10.584+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.588766+0000) 2019-10-13T13:32:10.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:10.576+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.580680+0000) 2019-10-13T13:32:10.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:10.576+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.580839+0000) 2019-10-13T13:32:10.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:10.576+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:10.581040+0000) 2019-10-13T13:32:10.594 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:10.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:12.863 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:32:12.863 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:12.867 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:32:12.867 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:13.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:13.100+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:13.102928+0000) 2019-10-13T13:32:15.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:15.584+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.587589+0000) 2019-10-13T13:32:15.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:15.588+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.591413+0000) 2019-10-13T13:32:15.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:15.588+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.591674+0000) 2019-10-13T13:32:15.594 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:15.588+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.592304+0000) 2019-10-13T13:32:15.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:15.588+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.592364+0000) 2019-10-13T13:32:15.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:15.588+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.592777+0000) 2019-10-13T13:32:15.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:15.588+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.594039+0000) 2019-10-13T13:32:15.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:15.588+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.594642+0000) 2019-10-13T13:32:15.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:15.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:15.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:15.596+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.600136+0000) 2019-10-13T13:32:15.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:15.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.812941+0000) 2019-10-13T13:32:15.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:15.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:15.874159+0000) 2019-10-13T13:32:16.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:16.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:16.555660+0000) 2019-10-13T13:32:18.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:18.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:18.821950+0000) 2019-10-13T13:32:18.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:18.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:18.823114+0000) 2019-10-13T13:32:20.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:20.052+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.055324+0000) 2019-10-13T13:32:20.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:20.596+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.602580+0000) 2019-10-13T13:32:20.605 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:20.596+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.603139+0000) 2019-10-13T13:32:20.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:20.600+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.603791+0000) 2019-10-13T13:32:20.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:20.600+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.605077+0000) 2019-10-13T13:32:20.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:20.600+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.605523+0000) 2019-10-13T13:32:20.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:20.600+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.605775+0000) 2019-10-13T13:32:20.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:20.600+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.606588+0000) 2019-10-13T13:32:20.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:20.608+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:20.610982+0000) 2019-10-13T13:32:20.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:20.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:23.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:23.108+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:23.109402+0000) 2019-10-13T13:32:25.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:25.584+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.587796+0000) 2019-10-13T13:32:25.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:25.608+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.614673+0000) 2019-10-13T13:32:25.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:25.612+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.615453+0000) 2019-10-13T13:32:25.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:25.612+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.615905+0000) 2019-10-13T13:32:25.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:25.612+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.617061+0000) 2019-10-13T13:32:25.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:25.612+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.617349+0000) 2019-10-13T13:32:25.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:25.612+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.618667+0000) 2019-10-13T13:32:25.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:25.612+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.619282+0000) 2019-10-13T13:32:25.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:25.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:25.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:25.620+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.625016+0000) 2019-10-13T13:32:25.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:25.812+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.813114+0000) 2019-10-13T13:32:25.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:25.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:25.874348+0000) 2019-10-13T13:32:26.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:26.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:26.555825+0000) 2019-10-13T13:32:28.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:28.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:28.822145+0000) 2019-10-13T13:32:28.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:28.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:28.823340+0000) 2019-10-13T13:32:30.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:30.052+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.055517+0000) 2019-10-13T13:32:30.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:30.624+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.627706+0000) 2019-10-13T13:32:30.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:30.624+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.628451+0000) 2019-10-13T13:32:30.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:30.624+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.629285+0000) 2019-10-13T13:32:30.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:30.624+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.630029+0000) 2019-10-13T13:32:30.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:30.624+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.630046+0000) 2019-10-13T13:32:30.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:30.628+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.631385+0000) 2019-10-13T13:32:30.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:30.628+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.631967+0000) 2019-10-13T13:32:30.638 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:30.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:30.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:30.636+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:30.639236+0000) 2019-10-13T13:32:33.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:33.112+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:33.115809+0000) 2019-10-13T13:32:35.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:35.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.588028+0000) 2019-10-13T13:32:35.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:35.635+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.639580+0000) 2019-10-13T13:32:35.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:35.636+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.640698+0000) 2019-10-13T13:32:35.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:35.635+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.641038+0000) 2019-10-13T13:32:35.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:35.635+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.641681+0000) 2019-10-13T13:32:35.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:35.635+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.642030+0000) 2019-10-13T13:32:35.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:35.635+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.642626+0000) 2019-10-13T13:32:35.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:35.635+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.643224+0000) 2019-10-13T13:32:35.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:35.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:35.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:35.644+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.649893+0000) 2019-10-13T13:32:35.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:35.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.813269+0000) 2019-10-13T13:32:35.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:35.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:35.874537+0000) 2019-10-13T13:32:36.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:36.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:36.556001+0000) 2019-10-13T13:32:38.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:38.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:38.822360+0000) 2019-10-13T13:32:38.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:38.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:38.823547+0000) 2019-10-13T13:32:40.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:40.052+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.055717+0000) 2019-10-13T13:32:40.666 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:40.647+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.651716+0000) 2019-10-13T13:32:40.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:40.647+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.652492+0000) 2019-10-13T13:32:40.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:40.647+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.652916+0000) 2019-10-13T13:32:40.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:40.647+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.654274+0000) 2019-10-13T13:32:40.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:40.647+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.654869+0000) 2019-10-13T13:32:40.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:40.651+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.655449+0000) 2019-10-13T13:32:40.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:40.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:40.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:40.648+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.651236+0000) 2019-10-13T13:32:40.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:40.656+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:40.658348+0000) 2019-10-13T13:32:43.011 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:32:43.011 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:43.015 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:32:43.015 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:43.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:43.120+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:43.122207+0000) 2019-10-13T13:32:45.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:45.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.588264+0000) 2019-10-13T13:32:45.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:45.659+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.663572+0000) 2019-10-13T13:32:45.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:45.659+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.665068+0000) 2019-10-13T13:32:45.668 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:45.659+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.666020+0000) 2019-10-13T13:32:45.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:45.664+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.666653+0000) 2019-10-13T13:32:45.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:45.659+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.666819+0000) 2019-10-13T13:32:45.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:45.663+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.667475+0000) 2019-10-13T13:32:45.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:45.663+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.668053+0000) 2019-10-13T13:32:45.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:45.668+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.671253+0000) 2019-10-13T13:32:45.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:45.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:45.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:45.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.813451+0000) 2019-10-13T13:32:45.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:45.867+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:45.874729+0000) 2019-10-13T13:32:46.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:46.551+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:46.556133+0000) 2019-10-13T13:32:48.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:48.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:48.822556+0000) 2019-10-13T13:32:48.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:48.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:48.823758+0000) 2019-10-13T13:32:50.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:50.052+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.055904+0000) 2019-10-13T13:32:50.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:50.671+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.676113+0000) 2019-10-13T13:32:50.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:50.672+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.676683+0000) 2019-10-13T13:32:50.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:50.671+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.677716+0000) 2019-10-13T13:32:50.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:50.671+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.677651+0000) 2019-10-13T13:32:50.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:50.671+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.677832+0000) 2019-10-13T13:32:50.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:50.671+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.678452+0000) 2019-10-13T13:32:50.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:50.671+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.678738+0000) 2019-10-13T13:32:50.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:50.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:50.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:50.684+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:50.686325+0000) 2019-10-13T13:32:53.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:53.128+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:53.128553+0000) 2019-10-13T13:32:55.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:55.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.588481+0000) 2019-10-13T13:32:55.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:55.683+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.688962+0000) 2019-10-13T13:32:55.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:55.683+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.689653+0000) 2019-10-13T13:32:55.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:32:55.688+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.690539+0000) 2019-10-13T13:32:55.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:32:55.683+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.691156+0000) 2019-10-13T13:32:55.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:55.687+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.691480+0000) 2019-10-13T13:32:55.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:55.687+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.691843+0000) 2019-10-13T13:32:55.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:55.687+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.692241+0000) 2019-10-13T13:32:55.699 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:32:55.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:32:55.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:32:55.692+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.697779+0000) 2019-10-13T13:32:55.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:32:55.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.813633+0000) 2019-10-13T13:32:55.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:32:55.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:55.874975+0000) 2019-10-13T13:32:56.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:32:56.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:56.556309+0000) 2019-10-13T13:32:58.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:32:58.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:58.822759+0000) 2019-10-13T13:32:58.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:32:58.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:58.823979+0000) 2019-10-13T13:33:00.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:00.056+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.056107+0000) 2019-10-13T13:33:00.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:00.695+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.701163+0000) 2019-10-13T13:33:00.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:00.695+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.702133+0000) 2019-10-13T13:33:00.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:00.695+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.702392+0000) 2019-10-13T13:33:00.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:00.695+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.702791+0000) 2019-10-13T13:33:00.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:00.699+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.704152+0000) 2019-10-13T13:33:00.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:00.695+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.703243+0000) 2019-10-13T13:33:00.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:00.699+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.704213+0000) 2019-10-13T13:33:00.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:00.703+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:00.707863+0000) 2019-10-13T13:33:00.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:00.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:03.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:03.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:03.135022+0000) 2019-10-13T13:33:05.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:05.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.588678+0000) 2019-10-13T13:33:05.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:05.711+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.713437+0000) 2019-10-13T13:33:05.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:05.711+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.714166+0000) 2019-10-13T13:33:05.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:05.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.714600+0000) 2019-10-13T13:33:05.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:05.715+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.715417+0000) 2019-10-13T13:33:05.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:05.715+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.716030+0000) 2019-10-13T13:33:05.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:05.715+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.716246+0000) 2019-10-13T13:33:05.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:05.711+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.716901+0000) 2019-10-13T13:33:05.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:05.715+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.719413+0000) 2019-10-13T13:33:05.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:05.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:05.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:05.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.813806+0000) 2019-10-13T13:33:05.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:05.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:05.875159+0000) 2019-10-13T13:33:06.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:06.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:06.556482+0000) 2019-10-13T13:33:08.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:08.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:08.822976+0000) 2019-10-13T13:33:08.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:08.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:08.824189+0000) 2019-10-13T13:33:10.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:10.051+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.056301+0000) 2019-10-13T13:33:10.727 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:10.723+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.724902+0000) 2019-10-13T13:33:10.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:10.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.725420+0000) 2019-10-13T13:33:10.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:10.723+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.725805+0000) 2019-10-13T13:33:10.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:10.723+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.726167+0000) 2019-10-13T13:33:10.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:10.723+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.726549+0000) 2019-10-13T13:33:10.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:10.727+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.727547+0000) 2019-10-13T13:33:10.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:10.723+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.728981+0000) 2019-10-13T13:33:10.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:10.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174636 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:10.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:10.731+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:10.734497+0000) 2019-10-13T13:33:13.093 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:33:13.094 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:13.097 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:33:13.098 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:13.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:13.143+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:13.146626+0000) 2019-10-13T13:33:15.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:15.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.588884+0000) 2019-10-13T13:33:15.739 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:15.735+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.736895+0000) 2019-10-13T13:33:15.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:15.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.737916+0000) 2019-10-13T13:33:15.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:15.735+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.738563+0000) 2019-10-13T13:33:15.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:15.735+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.738859+0000) 2019-10-13T13:33:15.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:15.735+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.739134+0000) 2019-10-13T13:33:15.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:15.739+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.739824+0000) 2019-10-13T13:33:15.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:15.739+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.739891+0000) 2019-10-13T13:33:15.745 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:15.739+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.743769+0000) 2019-10-13T13:33:15.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:15.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:15.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:15.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.813955+0000) 2019-10-13T13:33:15.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:15.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:15.875404+0000) 2019-10-13T13:33:16.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:16.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:16.556655+0000) 2019-10-13T13:33:18.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:18.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:18.823175+0000) 2019-10-13T13:33:18.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:18.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:18.824367+0000) 2019-10-13T13:33:20.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:20.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.056528+0000) 2019-10-13T13:33:20.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:20.746+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.749745+0000) 2019-10-13T13:33:20.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:20.746+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.750140+0000) 2019-10-13T13:33:20.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:20.746+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.750899+0000) 2019-10-13T13:33:20.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:20.750+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.751364+0000) 2019-10-13T13:33:20.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:20.750+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.752438+0000) 2019-10-13T13:33:20.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:20.747+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.753022+0000) 2019-10-13T13:33:20.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:20.750+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.753508+0000) 2019-10-13T13:33:20.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:20.755+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:20.758881+0000) 2019-10-13T13:33:20.762 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:20.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:23.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:23.147+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:23.152809+0000) 2019-10-13T13:33:25.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:25.586+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.589145+0000) 2019-10-13T13:33:25.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:25.759+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.762231+0000) 2019-10-13T13:33:25.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:25.758+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.762364+0000) 2019-10-13T13:33:25.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:25.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.763440+0000) 2019-10-13T13:33:25.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:25.762+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.763649+0000) 2019-10-13T13:33:25.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:25.762+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.764372+0000) 2019-10-13T13:33:25.767 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:25.762+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.765184+0000) 2019-10-13T13:33:25.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:25.762+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.765772+0000) 2019-10-13T13:33:25.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:25.767+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.773082+0000) 2019-10-13T13:33:25.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:25.770+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:25.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:25.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.814125+0000) 2019-10-13T13:33:25.877 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:25.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:25.875539+0000) 2019-10-13T13:33:26.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:26.554+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:26.556846+0000) 2019-10-13T13:33:28.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:28.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:28.823333+0000) 2019-10-13T13:33:28.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:28.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:28.824560+0000) 2019-10-13T13:33:30.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:30.051+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.056724+0000) 2019-10-13T13:33:30.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:30.770+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.775084+0000) 2019-10-13T13:33:30.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:30.774+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.775629+0000) 2019-10-13T13:33:30.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:30.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.775947+0000) 2019-10-13T13:33:30.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:30.771+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.777290+0000) 2019-10-13T13:33:30.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:30.774+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.777271+0000) 2019-10-13T13:33:30.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:30.774+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.777816+0000) 2019-10-13T13:33:30.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:30.774+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.777838+0000) 2019-10-13T13:33:30.786 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:30.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:30.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:30.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:30.785283+0000) 2019-10-13T13:33:33.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:33.155+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:33.159260+0000) 2019-10-13T13:33:35.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:35.586+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.589345+0000) 2019-10-13T13:33:35.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:35.786+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.787689+0000) 2019-10-13T13:33:35.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:35.786+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.787953+0000) 2019-10-13T13:33:35.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:35.783+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.789844+0000) 2019-10-13T13:33:35.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:35.786+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.789247+0000) 2019-10-13T13:33:35.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:35.786+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.790015+0000) 2019-10-13T13:33:35.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:35.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.790765+0000) 2019-10-13T13:33:35.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:35.790+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.791353+0000) 2019-10-13T13:33:35.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:35.791+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.797328+0000) 2019-10-13T13:33:35.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:35.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:35.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:35.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.814320+0000) 2019-10-13T13:33:35.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:35.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:35.875718+0000) 2019-10-13T13:33:36.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:36.554+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:36.557072+0000) 2019-10-13T13:33:38.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:38.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:38.823502+0000) 2019-10-13T13:33:38.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:38.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:38.824756+0000) 2019-10-13T13:33:40.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:40.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.056921+0000) 2019-10-13T13:33:40.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:40.798+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.800608+0000) 2019-10-13T13:33:40.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:40.795+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.801708+0000) 2019-10-13T13:33:40.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:40.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.801218+0000) 2019-10-13T13:33:40.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:40.798+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.801853+0000) 2019-10-13T13:33:40.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:40.802+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.803357+0000) 2019-10-13T13:33:40.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:40.802+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.803657+0000) 2019-10-13T13:33:40.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:40.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.803972+0000) 2019-10-13T13:33:40.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:40.803+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:40.808715+0000) 2019-10-13T13:33:40.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:40.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:43.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:43.159+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:43.165650+0000) 2019-10-13T13:33:43.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:33:43.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:43.203 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:33:43.204 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:45.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:45.586+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.589545+0000) 2019-10-13T13:33:45.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:45.810+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.812888+0000) 2019-10-13T13:33:45.815 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:45.807+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.813905+0000) 2019-10-13T13:33:45.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:45.810+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.813386+0000) 2019-10-13T13:33:45.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:45.810+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.814032+0000) 2019-10-13T13:33:45.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:45.810+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.814743+0000) 2019-10-13T13:33:45.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:45.814+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.815714+0000) 2019-10-13T13:33:45.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:45.814+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.817014+0000) 2019-10-13T13:33:45.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:45.815+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.822016+0000) 2019-10-13T13:33:45.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:45.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:45.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:45.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:45.875900+0000) 2019-10-13T13:33:46.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:46.554+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:46.557241+0000) 2019-10-13T13:33:48.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:48.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:48.823664+0000) 2019-10-13T13:33:48.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:48.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:48.824965+0000) 2019-10-13T13:33:50.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:50.051+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.057111+0000) 2019-10-13T13:33:50.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:50.822+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.825232+0000) 2019-10-13T13:33:50.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:50.819+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.826013+0000) 2019-10-13T13:33:50.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:50.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.826679+0000) 2019-10-13T13:33:50.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:50.822+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.827168+0000) 2019-10-13T13:33:50.830 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:50.826+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.827665+0000) 2019-10-13T13:33:50.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:50.826+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.827892+0000) 2019-10-13T13:33:50.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:50.826+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.828340+0000) 2019-10-13T13:33:50.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:50.827+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:50.832822+0000) 2019-10-13T13:33:50.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:50.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174900 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:52.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:52.562+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.565006+0000) 2019-10-13T13:33:53.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:53.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:53.171913+0000) 2019-10-13T13:33:55.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:55.586+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.589742+0000) 2019-10-13T13:33:55.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:55.834+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.837810+0000) 2019-10-13T13:33:55.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:55.834+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.837920+0000) 2019-10-13T13:33:55.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:55.834+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.838713+0000) 2019-10-13T13:33:55.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:55.834+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.839159+0000) 2019-10-13T13:33:55.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:33:55.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.839822+0000) 2019-10-13T13:33:55.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:33:55.838+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.839874+0000) 2019-10-13T13:33:55.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:33:55.838+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.840325+0000) 2019-10-13T13:33:55.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:33:55.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174931 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:33:55.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:33:55.842+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.847154+0000) 2019-10-13T13:33:55.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:33:55.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:55.876116+0000) 2019-10-13T13:33:56.559 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:33:56.554+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:56.557418+0000) 2019-10-13T13:33:58.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:33:58.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:58.823891+0000) 2019-10-13T13:33:58.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:33:58.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:58.825180+0000) 2019-10-13T13:34:00.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:00.054+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.057331+0000) 2019-10-13T13:34:00.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:00.846+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.851592+0000) 2019-10-13T13:34:00.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:00.845+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.849491+0000) 2019-10-13T13:34:00.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:00.845+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.850419+0000) 2019-10-13T13:34:00.857 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:00.845+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.850867+0000) 2019-10-13T13:34:00.857 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:00.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.852131+0000) 2019-10-13T13:34:00.857 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:00.849+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.852538+0000) 2019-10-13T13:34:00.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:00.849+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.852998+0000) 2019-10-13T13:34:00.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:00.854+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:00.858593+0000) 2019-10-13T13:34:00.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:00.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174966 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:02.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:02.561+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.565232+0000) 2019-10-13T13:34:03.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:03.174+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:03.178368+0000) 2019-10-13T13:34:05.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:05.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.589966+0000) 2019-10-13T13:34:05.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:05.857+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.863132+0000) 2019-10-13T13:34:05.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:05.861+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.863781+0000) 2019-10-13T13:34:05.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:05.861+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.864126+0000) 2019-10-13T13:34:05.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:05.858+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.864539+0000) 2019-10-13T13:34:05.871 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:05.861+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.864645+0000) 2019-10-13T13:34:05.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:05.861+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.865071+0000) 2019-10-13T13:34:05.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:05.861+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.865723+0000) 2019-10-13T13:34:05.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:05.862+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.870045+0000) 2019-10-13T13:34:05.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:05.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 174998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:05.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:05.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:05.876252+0000) 2019-10-13T13:34:06.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:06.553+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:06.557621+0000) 2019-10-13T13:34:08.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:08.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:08.824107+0000) 2019-10-13T13:34:08.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:08.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:08.825372+0000) 2019-10-13T13:34:10.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:10.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.057519+0000) 2019-10-13T13:34:10.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:10.869+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.874999+0000) 2019-10-13T13:34:10.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:10.873+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.875455+0000) 2019-10-13T13:34:10.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:10.870+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.876148+0000) 2019-10-13T13:34:10.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:10.873+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.876286+0000) 2019-10-13T13:34:10.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:10.873+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.876685+0000) 2019-10-13T13:34:10.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:10.873+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.877198+0000) 2019-10-13T13:34:10.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:10.873+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.877733+0000) 2019-10-13T13:34:10.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:10.878+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:10.884343+0000) 2019-10-13T13:34:10.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:10.881+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:12.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:12.561+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.565432+0000) 2019-10-13T13:34:13.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:13.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:13.184924+0000) 2019-10-13T13:34:13.281 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:34:13.281 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:13.285 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:34:13.285 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:15.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:15.585+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.590123+0000) 2019-10-13T13:34:15.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:15.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.876436+0000) 2019-10-13T13:34:15.889 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:15.885+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.887559+0000) 2019-10-13T13:34:15.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:15.885+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.887933+0000) 2019-10-13T13:34:15.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:15.885+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.888363+0000) 2019-10-13T13:34:15.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:15.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.889453+0000) 2019-10-13T13:34:15.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:15.885+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.889019+0000) 2019-10-13T13:34:15.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:15.885+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.889565+0000) 2019-10-13T13:34:15.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:15.885+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.890058+0000) 2019-10-13T13:34:15.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:15.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:15.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:15.890+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:15.896991+0000) 2019-10-13T13:34:16.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:16.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:16.557806+0000) 2019-10-13T13:34:18.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:18.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:18.824340+0000) 2019-10-13T13:34:18.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:18.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:18.825563+0000) 2019-10-13T13:34:20.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:20.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.057735+0000) 2019-10-13T13:34:20.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:20.897+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.899368+0000) 2019-10-13T13:34:20.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:20.894+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.900480+0000) 2019-10-13T13:34:20.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:20.897+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.900601+0000) 2019-10-13T13:34:20.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:20.897+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.900913+0000) 2019-10-13T13:34:20.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:20.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.902055+0000) 2019-10-13T13:34:20.904 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:20.897+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.902736+0000) 2019-10-13T13:34:20.905 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:20.897+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.903166+0000) 2019-10-13T13:34:20.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:20.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:20.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:20.902+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:20.908747+0000) 2019-10-13T13:34:22.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:22.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.565633+0000) 2019-10-13T13:34:23.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:23.190+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:23.191268+0000) 2019-10-13T13:34:25.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:25.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.590354+0000) 2019-10-13T13:34:25.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:25.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.876678+0000) 2019-10-13T13:34:25.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:25.906+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.911907+0000) 2019-10-13T13:34:25.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:25.909+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.911925+0000) 2019-10-13T13:34:25.914 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:25.909+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.912529+0000) 2019-10-13T13:34:25.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:25.909+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.912788+0000) 2019-10-13T13:34:25.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:25.909+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.913840+0000) 2019-10-13T13:34:25.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:25.909+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.914798+0000) 2019-10-13T13:34:25.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:25.909+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.915128+0000) 2019-10-13T13:34:25.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:25.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:25.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:25.914+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:25.920946+0000) 2019-10-13T13:34:26.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:26.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:26.557994+0000) 2019-10-13T13:34:28.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:28.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:28.824489+0000) 2019-10-13T13:34:28.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:28.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:28.825747+0000) 2019-10-13T13:34:30.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:30.050+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.057934+0000) 2019-10-13T13:34:30.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:30.921+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.924091+0000) 2019-10-13T13:34:30.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:30.921+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.924855+0000) 2019-10-13T13:34:30.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:30.921+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.924931+0000) 2019-10-13T13:34:30.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:30.918+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.925734+0000) 2019-10-13T13:34:30.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:30.921+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.925282+0000) 2019-10-13T13:34:30.928 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:30.921+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.926221+0000) 2019-10-13T13:34:30.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:30.921+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.926877+0000) 2019-10-13T13:34:30.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:30.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:30.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:30.926+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:30.932283+0000) 2019-10-13T13:34:32.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:32.561+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.565851+0000) 2019-10-13T13:34:33.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:33.190+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:33.197859+0000) 2019-10-13T13:34:35.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:35.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.590581+0000) 2019-10-13T13:34:35.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:35.873+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.876914+0000) 2019-10-13T13:34:35.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:35.933+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.935747+0000) 2019-10-13T13:34:35.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:35.933+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.936326+0000) 2019-10-13T13:34:35.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:35.933+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.936577+0000) 2019-10-13T13:34:35.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:35.933+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.936964+0000) 2019-10-13T13:34:35.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:35.933+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.937457+0000) 2019-10-13T13:34:35.940 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:35.933+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.937833+0000) 2019-10-13T13:34:35.941 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:35.934+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.939350+0000) 2019-10-13T13:34:35.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:35.941+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:35.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:35.938+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:35.945297+0000) 2019-10-13T13:34:36.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:36.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:36.558175+0000) 2019-10-13T13:34:38.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:38.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:38.824707+0000) 2019-10-13T13:34:38.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:38.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:38.825951+0000) 2019-10-13T13:34:40.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:40.054+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.058179+0000) 2019-10-13T13:34:40.950 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:40.945+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.948323+0000) 2019-10-13T13:34:40.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:40.945+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.948898+0000) 2019-10-13T13:34:40.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:40.945+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.949714+0000) 2019-10-13T13:34:40.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:40.945+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.950464+0000) 2019-10-13T13:34:40.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:40.945+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.950815+0000) 2019-10-13T13:34:40.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:40.946+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.951639+0000) 2019-10-13T13:34:40.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:40.949+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.951939+0000) 2019-10-13T13:34:40.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:40.950+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:40.957049+0000) 2019-10-13T13:34:40.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:40.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:42.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:42.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.566057+0000) 2019-10-13T13:34:43.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:43.206+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:43.209399+0000) 2019-10-13T13:34:43.396 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:34:43.396 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:43.400 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:34:43.400 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:45.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:45.584+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.590782+0000) 2019-10-13T13:34:45.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:45.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.877120+0000) 2019-10-13T13:34:45.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:45.956+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.961207+0000) 2019-10-13T13:34:45.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:45.956+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.962239+0000) 2019-10-13T13:34:45.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:45.956+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.962662+0000) 2019-10-13T13:34:45.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:45.960+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.963465+0000) 2019-10-13T13:34:45.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:45.960+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.964008+0000) 2019-10-13T13:34:45.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:45.962+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.963989+0000) 2019-10-13T13:34:45.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:45.960+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.964292+0000) 2019-10-13T13:34:45.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:45.966+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:45.969615+0000) 2019-10-13T13:34:45.972 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:45.964+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175261 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:46.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:46.556+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:46.558357+0000) 2019-10-13T13:34:48.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:48.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:48.824907+0000) 2019-10-13T13:34:48.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:48.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:48.826144+0000) 2019-10-13T13:34:50.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:50.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.058364+0000) 2019-10-13T13:34:50.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:50.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.973882+0000) 2019-10-13T13:34:50.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:50.968+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.974054+0000) 2019-10-13T13:34:50.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:50.968+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.974803+0000) 2019-10-13T13:34:50.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:50.972+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.976049+0000) 2019-10-13T13:34:50.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:50.972+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.976417+0000) 2019-10-13T13:34:50.979 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:50.972+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.976700+0000) 2019-10-13T13:34:50.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:50.972+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.977160+0000) 2019-10-13T13:34:50.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:50.977+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:50.981803+0000) 2019-10-13T13:34:50.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:50.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175293 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:52.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:52.564+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.566261+0000) 2019-10-13T13:34:53.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:53.213+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:53.215782+0000) 2019-10-13T13:34:55.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:55.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.590999+0000) 2019-10-13T13:34:55.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:55.872+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.877314+0000) 2019-10-13T13:34:55.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:34:55.980+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.985906+0000) 2019-10-13T13:34:55.988 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:55.980+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.986286+0000) 2019-10-13T13:34:55.989 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:55.984+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.987457+0000) 2019-10-13T13:34:55.991 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:55.980+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.986658+0000) 2019-10-13T13:34:55.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:34:55.984+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.988734+0000) 2019-10-13T13:34:55.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:34:55.984+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.989108+0000) 2019-10-13T13:34:55.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:34:55.985+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.988772+0000) 2019-10-13T13:34:55.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:34:55.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:34:56.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:34:55.993+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:55.997849+0000) 2019-10-13T13:34:56.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:34:56.556+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:56.558541+0000) 2019-10-13T13:34:58.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:34:58.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:58.825134+0000) 2019-10-13T13:34:58.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:34:58.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:58.826334+0000) 2019-10-13T13:35:00.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:00.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.058560+0000) 2019-10-13T13:35:00.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:00.992+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.996992+0000) 2019-10-13T13:35:00.999 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:00.992+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.997399+0000) 2019-10-13T13:35:01.000 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:00.992+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.998699+0000) 2019-10-13T13:35:01.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:00.992+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.999011+0000) 2019-10-13T13:35:01.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:00.996+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.999363+0000) 2019-10-13T13:35:01.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:00.997+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:00.999947+0000) 2019-10-13T13:35:01.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:00.996+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:01.000069+0000) 2019-10-13T13:35:01.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:01.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:01.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:01.005+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:01.006569+0000) 2019-10-13T13:35:02.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:02.560+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.566494+0000) 2019-10-13T13:35:03.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:03.221+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:03.222179+0000) 2019-10-13T13:35:05.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:05.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:05.591202+0000) 2019-10-13T13:35:05.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:05.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:05.877563+0000) 2019-10-13T13:35:06.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:06.004+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.009397+0000) 2019-10-13T13:35:06.012 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:06.004+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.010337+0000) 2019-10-13T13:35:06.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:06.009+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.011362+0000) 2019-10-13T13:35:06.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:06.008+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.011346+0000) 2019-10-13T13:35:06.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:06.008+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.011733+0000) 2019-10-13T13:35:06.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:06.008+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.012693+0000) 2019-10-13T13:35:06.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:06.008+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.013230+0000) 2019-10-13T13:35:06.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:06.013+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.016608+0000) 2019-10-13T13:35:06.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:06.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:06.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:06.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:06.558736+0000) 2019-10-13T13:35:08.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:08.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:08.825326+0000) 2019-10-13T13:35:08.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:08.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:08.826508+0000) 2019-10-13T13:35:10.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:10.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:10.058812+0000) 2019-10-13T13:35:11.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:11.016+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.021462+0000) 2019-10-13T13:35:11.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:11.016+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.022070+0000) 2019-10-13T13:35:11.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:11.020+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.024126+0000) 2019-10-13T13:35:11.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:11.021+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.024787+0000) 2019-10-13T13:35:11.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:11.020+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.024841+0000) 2019-10-13T13:35:11.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:11.020+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.025327+0000) 2019-10-13T13:35:11.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:11.020+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.025501+0000) 2019-10-13T13:35:11.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:11.025+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.029804+0000) 2019-10-13T13:35:11.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:11.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:12.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:12.564+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.566714+0000) 2019-10-13T13:35:13.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:13.225+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:13.228419+0000) 2019-10-13T13:35:13.520 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:35:13.520 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:13.523 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:35:13.524 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:15.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:15.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:15.591425+0000) 2019-10-13T13:35:15.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:15.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:15.877764+0000) 2019-10-13T13:35:16.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:16.028+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.034718+0000) 2019-10-13T13:35:16.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:16.028+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.035194+0000) 2019-10-13T13:35:16.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:16.032+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.035482+0000) 2019-10-13T13:35:16.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:16.032+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.035728+0000) 2019-10-13T13:35:16.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:16.033+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.036419+0000) 2019-10-13T13:35:16.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:16.032+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.036408+0000) 2019-10-13T13:35:16.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:16.032+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.036876+0000) 2019-10-13T13:35:16.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:16.037+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.041496+0000) 2019-10-13T13:35:16.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:16.036+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:16.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:16.552+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:16.558946+0000) 2019-10-13T13:35:18.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:18.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:18.825484+0000) 2019-10-13T13:35:18.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:18.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:18.826728+0000) 2019-10-13T13:35:20.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:20.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:20.059005+0000) 2019-10-13T13:35:21.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:21.040+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.045771+0000) 2019-10-13T13:35:21.048 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:21.040+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.046181+0000) 2019-10-13T13:35:21.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:21.044+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.047516+0000) 2019-10-13T13:35:21.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:21.044+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.048048+0000) 2019-10-13T13:35:21.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:21.044+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.048406+0000) 2019-10-13T13:35:21.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:21.044+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.049046+0000) 2019-10-13T13:35:21.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:21.045+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.049491+0000) 2019-10-13T13:35:21.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:21.049+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.053698+0000) 2019-10-13T13:35:21.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:21.052+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:22.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:22.560+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.566946+0000) 2019-10-13T13:35:23.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:23.233+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:23.234862+0000) 2019-10-13T13:35:25.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:25.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:25.591625+0000) 2019-10-13T13:35:25.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:25.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:25.877984+0000) 2019-10-13T13:35:26.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:26.052+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.058014+0000) 2019-10-13T13:35:26.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:26.052+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.059009+0000) 2019-10-13T13:35:26.061 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:26.056+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.059757+0000) 2019-10-13T13:35:26.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:26.056+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.060139+0000) 2019-10-13T13:35:26.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:26.056+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.060359+0000) 2019-10-13T13:35:26.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:26.056+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.061203+0000) 2019-10-13T13:35:26.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:26.057+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.061811+0000) 2019-10-13T13:35:26.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:26.064+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:26.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:26.065+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.068692+0000) 2019-10-13T13:35:26.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:26.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:26.559149+0000) 2019-10-13T13:35:28.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:28.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:28.825705+0000) 2019-10-13T13:35:28.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:28.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:28.826927+0000) 2019-10-13T13:35:30.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:30.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:30.059203+0000) 2019-10-13T13:35:31.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:31.067+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.071532+0000) 2019-10-13T13:35:31.076 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:31.067+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.072257+0000) 2019-10-13T13:35:31.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:31.067+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.072667+0000) 2019-10-13T13:35:31.077 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:31.067+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.073297+0000) 2019-10-13T13:35:31.077 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:31.067+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.074087+0000) 2019-10-13T13:35:31.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:31.069+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.073874+0000) 2019-10-13T13:35:31.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:31.067+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.074510+0000) 2019-10-13T13:35:31.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:31.077+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.078567+0000) 2019-10-13T13:35:31.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:31.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:32.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:32.559+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.567167+0000) 2019-10-13T13:35:33.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:33.237+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:33.241229+0000) 2019-10-13T13:35:35.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:35.587+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:35.591843+0000) 2019-10-13T13:35:35.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:35.871+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:35.878206+0000) 2019-10-13T13:35:36.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:36.081+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.083854+0000) 2019-10-13T13:35:36.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:36.079+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.084244+0000) 2019-10-13T13:35:36.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:36.079+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.084729+0000) 2019-10-13T13:35:36.087 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:36.079+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.085056+0000) 2019-10-13T13:35:36.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:36.079+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.085576+0000) 2019-10-13T13:35:36.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:36.079+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.086639+0000) 2019-10-13T13:35:36.089 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:36.079+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.087182+0000) 2019-10-13T13:35:36.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:36.089+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.092317+0000) 2019-10-13T13:35:36.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:36.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:36.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:36.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:36.559356+0000) 2019-10-13T13:35:38.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:38.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:38.825907+0000) 2019-10-13T13:35:38.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:38.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:38.827100+0000) 2019-10-13T13:35:40.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:40.057+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:40.059416+0000) 2019-10-13T13:35:41.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:41.091+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.096257+0000) 2019-10-13T13:35:41.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:41.091+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.097214+0000) 2019-10-13T13:35:41.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:41.093+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.097890+0000) 2019-10-13T13:35:41.100 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:41.091+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.098029+0000) 2019-10-13T13:35:41.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:41.091+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.098727+0000) 2019-10-13T13:35:41.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:41.095+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.099357+0000) 2019-10-13T13:35:41.102 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:41.095+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.100450+0000) 2019-10-13T13:35:41.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:41.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:41.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:41.105+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.107068+0000) 2019-10-13T13:35:42.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:42.563+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.567368+0000) 2019-10-13T13:35:43.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:43.249+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:43.252753+0000) 2019-10-13T13:35:43.651 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:35:43.651 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:43.654 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:35:43.655 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:45.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:45.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:45.592047+0000) 2019-10-13T13:35:45.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:45.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:45.878411+0000) 2019-10-13T13:35:46.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:46.103+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.109049+0000) 2019-10-13T13:35:46.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:46.103+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.109390+0000) 2019-10-13T13:35:46.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:46.103+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.109897+0000) 2019-10-13T13:35:46.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:46.108+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.110982+0000) 2019-10-13T13:35:46.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:46.103+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.110938+0000) 2019-10-13T13:35:46.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:46.107+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.111369+0000) 2019-10-13T13:35:46.114 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:46.107+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.111754+0000) 2019-10-13T13:35:46.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:46.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:46.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:46.116+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.118850+0000) 2019-10-13T13:35:46.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:46.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:46.559526+0000) 2019-10-13T13:35:48.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:48.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:48.826107+0000) 2019-10-13T13:35:48.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:48.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:48.827295+0000) 2019-10-13T13:35:50.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:50.056+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:50.059608+0000) 2019-10-13T13:35:51.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:51.115+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.121123+0000) 2019-10-13T13:35:51.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:51.115+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.121877+0000) 2019-10-13T13:35:51.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:51.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.121883+0000) 2019-10-13T13:35:51.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:51.115+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.122359+0000) 2019-10-13T13:35:51.126 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:51.119+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.124000+0000) 2019-10-13T13:35:51.126 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:51.119+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.124292+0000) 2019-10-13T13:35:51.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:51.120+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.125100+0000) 2019-10-13T13:35:51.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:51.124+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.129627+0000) 2019-10-13T13:35:51.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:51.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:52.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:52.567+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.567539+0000) 2019-10-13T13:35:53.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:53.260+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:53.263816+0000) 2019-10-13T13:35:55.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:55.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:55.592268+0000) 2019-10-13T13:35:55.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:55.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:55.878639+0000) 2019-10-13T13:35:56.136 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:56.127+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.133883+0000) 2019-10-13T13:35:56.137 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:35:56.127+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.134840+0000) 2019-10-13T13:35:56.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:35:56.127+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.135227+0000) 2019-10-13T13:35:56.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:35:56.132+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.136125+0000) 2019-10-13T13:35:56.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:56.131+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.136188+0000) 2019-10-13T13:35:56.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:56.131+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.136577+0000) 2019-10-13T13:35:56.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:35:56.131+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.137208+0000) 2019-10-13T13:35:56.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:35:56.140+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.142923+0000) 2019-10-13T13:35:56.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:35:56.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:35:56.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:35:56.559+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:56.559700+0000) 2019-10-13T13:35:58.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:35:58.823+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:58.826308+0000) 2019-10-13T13:35:58.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:35:58.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:58.827532+0000) 2019-10-13T13:36:00.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:00.056+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:00.059813+0000) 2019-10-13T13:36:01.149 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:01.143+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.146674+0000) 2019-10-13T13:36:01.149 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:01.143+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.147203+0000) 2019-10-13T13:36:01.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:01.147+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.147639+0000) 2019-10-13T13:36:01.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:01.144+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.148239+0000) 2019-10-13T13:36:01.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:01.147+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.148345+0000) 2019-10-13T13:36:01.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:01.147+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.149205+0000) 2019-10-13T13:36:01.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:01.147+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.149241+0000) 2019-10-13T13:36:01.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:01.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:01.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:01.152+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.156265+0000) 2019-10-13T13:36:02.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:02.567+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.567758+0000) 2019-10-13T13:36:03.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:03.268+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:03.270371+0000) 2019-10-13T13:36:05.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:05.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:05.592472+0000) 2019-10-13T13:36:05.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:05.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:05.878842+0000) 2019-10-13T13:36:06.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:06.156+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.158509+0000) 2019-10-13T13:36:06.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:06.155+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.158696+0000) 2019-10-13T13:36:06.161 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:06.155+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.159262+0000) 2019-10-13T13:36:06.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:06.159+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.159637+0000) 2019-10-13T13:36:06.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:06.159+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.160069+0000) 2019-10-13T13:36:06.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:06.159+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.160446+0000) 2019-10-13T13:36:06.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:06.159+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.161433+0000) 2019-10-13T13:36:06.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:06.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:06.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:06.164+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.168865+0000) 2019-10-13T13:36:06.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:06.559+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:06.559885+0000) 2019-10-13T13:36:08.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:08.823+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:08.826537+0000) 2019-10-13T13:36:08.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:08.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:08.827740+0000) 2019-10-13T13:36:10.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:10.056+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:10.060017+0000) 2019-10-13T13:36:11.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:11.170+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.171895+0000) 2019-10-13T13:36:11.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:11.170+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.172211+0000) 2019-10-13T13:36:11.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:11.170+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.172724+0000) 2019-10-13T13:36:11.175 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:11.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.173632+0000) 2019-10-13T13:36:11.176 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:11.170+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.173690+0000) 2019-10-13T13:36:11.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:11.172+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.174878+0000) 2019-10-13T13:36:11.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:11.170+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.174965+0000) 2019-10-13T13:36:11.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:11.176+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.178560+0000) 2019-10-13T13:36:11.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:11.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:12.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:12.566+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:12.568003+0000) 2019-10-13T13:36:13.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:13.272+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.276746+0000) 2019-10-13T13:36:13.757 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:36:13.757 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:13.761 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:36:13.761 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:15.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:15.590+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:15.592630+0000) 2019-10-13T13:36:15.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:15.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:15.879040+0000) 2019-10-13T13:36:16.187 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:16.182+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.184588+0000) 2019-10-13T13:36:16.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:16.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.184915+0000) 2019-10-13T13:36:16.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:16.182+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.185562+0000) 2019-10-13T13:36:16.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:16.182+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.186074+0000) 2019-10-13T13:36:16.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:16.182+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.186448+0000) 2019-10-13T13:36:16.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:16.184+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.187031+0000) 2019-10-13T13:36:16.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:16.186+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.187734+0000) 2019-10-13T13:36:16.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:16.188+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.192596+0000) 2019-10-13T13:36:16.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:16.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:16.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:16.558+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:16.560069+0000) 2019-10-13T13:36:18.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:18.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.826737+0000) 2019-10-13T13:36:18.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:18.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.827949+0000) 2019-10-13T13:36:20.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:20.056+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:20.060231+0000) 2019-10-13T13:36:21.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:21.194+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.195903+0000) 2019-10-13T13:36:21.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:21.192+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.197026+0000) 2019-10-13T13:36:21.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:21.194+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.196672+0000) 2019-10-13T13:36:21.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:21.194+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.197690+0000) 2019-10-13T13:36:21.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:21.194+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.198662+0000) 2019-10-13T13:36:21.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:21.194+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.198239+0000) 2019-10-13T13:36:21.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:21.194+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.199064+0000) 2019-10-13T13:36:21.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:21.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:21.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:21.204+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.206548+0000) 2019-10-13T13:36:22.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:22.566+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:22.568227+0000) 2019-10-13T13:36:23.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:23.280+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.282852+0000) 2019-10-13T13:36:25.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:25.590+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:25.592834+0000) 2019-10-13T13:36:25.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:25.874+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:25.879268+0000) 2019-10-13T13:36:26.211 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:26.206+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.209260+0000) 2019-10-13T13:36:26.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:26.204+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.209807+0000) 2019-10-13T13:36:26.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:26.206+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.209827+0000) 2019-10-13T13:36:26.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:26.206+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.210679+0000) 2019-10-13T13:36:26.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:26.206+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.210407+0000) 2019-10-13T13:36:26.214 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:26.210+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.211435+0000) 2019-10-13T13:36:26.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:26.210+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.211873+0000) 2019-10-13T13:36:26.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:26.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:26.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:26.212+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.217746+0000) 2019-10-13T13:36:26.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:26.558+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:26.560247+0000) 2019-10-13T13:36:28.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:28.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.826936+0000) 2019-10-13T13:36:28.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:28.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.828162+0000) 2019-10-13T13:36:30.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:30.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:30.060450+0000) 2019-10-13T13:36:31.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:31.218+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.220318+0000) 2019-10-13T13:36:31.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:31.220+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.222856+0000) 2019-10-13T13:36:31.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:31.218+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.221371+0000) 2019-10-13T13:36:31.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:31.218+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.221967+0000) 2019-10-13T13:36:31.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:31.218+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.222278+0000) 2019-10-13T13:36:31.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:31.218+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.223138+0000) 2019-10-13T13:36:31.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:31.222+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.223506+0000) 2019-10-13T13:36:31.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:31.224+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.230064+0000) 2019-10-13T13:36:31.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:31.226+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:32.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:32.566+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:32.568458+0000) 2019-10-13T13:36:33.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:33.284+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.288823+0000) 2019-10-13T13:36:35.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:35.590+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:35.593034+0000) 2019-10-13T13:36:35.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:35.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:35.879496+0000) 2019-10-13T13:36:36.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:36.230+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.233120+0000) 2019-10-13T13:36:36.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:36.230+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.234058+0000) 2019-10-13T13:36:36.237 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:36.230+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.234454+0000) 2019-10-13T13:36:36.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:36.232+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.234957+0000) 2019-10-13T13:36:36.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:36.230+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.235172+0000) 2019-10-13T13:36:36.238 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:36.234+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.235727+0000) 2019-10-13T13:36:36.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:36.234+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.236636+0000) 2019-10-13T13:36:36.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:36.238+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 175993 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:36.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:36.240+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.242867+0000) 2019-10-13T13:36:36.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:36.558+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:36.560442+0000) 2019-10-13T13:36:38.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:38.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.827121+0000) 2019-10-13T13:36:38.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:38.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.828371+0000) 2019-10-13T13:36:40.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:40.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:40.060657+0000) 2019-10-13T13:36:41.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:41.242+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.245176+0000) 2019-10-13T13:36:41.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:41.243+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.246415+0000) 2019-10-13T13:36:41.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:41.242+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.246411+0000) 2019-10-13T13:36:41.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:41.242+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.247067+0000) 2019-10-13T13:36:41.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:41.246+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.247826+0000) 2019-10-13T13:36:41.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:41.246+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.248585+0000) 2019-10-13T13:36:41.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:41.246+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.248930+0000) 2019-10-13T13:36:41.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:41.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:41.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:41.251+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.257959+0000) 2019-10-13T13:36:42.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:42.566+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:42.568615+0000) 2019-10-13T13:36:43.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:43.291+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.295221+0000) 2019-10-13T13:36:43.904 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:36:43.904 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:43.908 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:36:43.908 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:45.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:45.590+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:45.593232+0000) 2019-10-13T13:36:45.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:45.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:45.879706+0000) 2019-10-13T13:36:46.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:46.254+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.258026+0000) 2019-10-13T13:36:46.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:46.254+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.258356+0000) 2019-10-13T13:36:46.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:46.254+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.258676+0000) 2019-10-13T13:36:46.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:46.258+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.259360+0000) 2019-10-13T13:36:46.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:46.255+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.259892+0000) 2019-10-13T13:36:46.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:46.258+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.260712+0000) 2019-10-13T13:36:46.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:46.258+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.261006+0000) 2019-10-13T13:36:46.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:46.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:46.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:46.263+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.267333+0000) 2019-10-13T13:36:46.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:46.558+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:46.560624+0000) 2019-10-13T13:36:48.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:48.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.827337+0000) 2019-10-13T13:36:48.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:48.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.828601+0000) 2019-10-13T13:36:50.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:50.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:50.060843+0000) 2019-10-13T13:36:51.271 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:51.263+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.269412+0000) 2019-10-13T13:36:51.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:51.266+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.270040+0000) 2019-10-13T13:36:51.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:51.266+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.270422+0000) 2019-10-13T13:36:51.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:51.266+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.270775+0000) 2019-10-13T13:36:51.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:51.270+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.271692+0000) 2019-10-13T13:36:51.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:51.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.272447+0000) 2019-10-13T13:36:51.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:51.270+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.272068+0000) 2019-10-13T13:36:51.280 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:51.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:51.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:51.279+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.283577+0000) 2019-10-13T13:36:52.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:52.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:52.568812+0000) 2019-10-13T13:36:53.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:53.299+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.305823+0000) 2019-10-13T13:36:55.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:55.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:55.593425+0000) 2019-10-13T13:36:55.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:55.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:55.879896+0000) 2019-10-13T13:36:56.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:56.277+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.281174+0000) 2019-10-13T13:36:56.284 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:36:56.277+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.281684+0000) 2019-10-13T13:36:56.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:36:56.277+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.282624+0000) 2019-10-13T13:36:56.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:56.281+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.283592+0000) 2019-10-13T13:36:56.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:56.281+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.284363+0000) 2019-10-13T13:36:56.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:36:56.281+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.285676+0000) 2019-10-13T13:36:56.288 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:36:56.279+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.284315+0000) 2019-10-13T13:36:56.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:36:56.283+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.289202+0000) 2019-10-13T13:36:56.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:36:56.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176128 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:36:56.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:36:56.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:56.560766+0000) 2019-10-13T13:36:58.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:36:58.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.827542+0000) 2019-10-13T13:36:58.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:36:58.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.828821+0000) 2019-10-13T13:37:00.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:00.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:00.061038+0000) 2019-10-13T13:37:01.295 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:01.289+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.293241+0000) 2019-10-13T13:37:01.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:01.289+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.293670+0000) 2019-10-13T13:37:01.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:01.289+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.293930+0000) 2019-10-13T13:37:01.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:01.289+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.294524+0000) 2019-10-13T13:37:01.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:01.289+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.294838+0000) 2019-10-13T13:37:01.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:01.293+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.295547+0000) 2019-10-13T13:37:01.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:01.291+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.295863+0000) 2019-10-13T13:37:01.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:01.299+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.302574+0000) 2019-10-13T13:37:01.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:01.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:02.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:02.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:02.569018+0000) 2019-10-13T13:37:03.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:03.311+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.312391+0000) 2019-10-13T13:37:05.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:05.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:05.593624+0000) 2019-10-13T13:37:05.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:05.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:05.880093+0000) 2019-10-13T13:37:06.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:06.301+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.306072+0000) 2019-10-13T13:37:06.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:06.301+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.306990+0000) 2019-10-13T13:37:06.309 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:06.305+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.307585+0000) 2019-10-13T13:37:06.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:06.303+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.308197+0000) 2019-10-13T13:37:06.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:06.305+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.308721+0000) 2019-10-13T13:37:06.311 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:06.305+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.308949+0000) 2019-10-13T13:37:06.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:06.305+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.309674+0000) 2019-10-13T13:37:06.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:06.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176197 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:06.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:06.311+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.316065+0000) 2019-10-13T13:37:06.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:06.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:06.560957+0000) 2019-10-13T13:37:08.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:08.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.827731+0000) 2019-10-13T13:37:08.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:08.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.829027+0000) 2019-10-13T13:37:10.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:10.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:10.061265+0000) 2019-10-13T13:37:11.321 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:11.313+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.318967+0000) 2019-10-13T13:37:11.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:11.317+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.319705+0000) 2019-10-13T13:37:11.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:11.317+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.319784+0000) 2019-10-13T13:37:11.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:11.315+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.320431+0000) 2019-10-13T13:37:11.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:11.317+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.320589+0000) 2019-10-13T13:37:11.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:11.317+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.320965+0000) 2019-10-13T13:37:11.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:11.317+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.321518+0000) 2019-10-13T13:37:11.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:11.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176230 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:11.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:11.323+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.329110+0000) 2019-10-13T13:37:12.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:12.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:12.569224+0000) 2019-10-13T13:37:13.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:13.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.323022+0000) 2019-10-13T13:37:13.961 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:37:13.961 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:13.965 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:37:13.965 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:15.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:15.593+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:15.593828+0000) 2019-10-13T13:37:15.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:15.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:15.880308+0000) 2019-10-13T13:37:16.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:16.325+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.330784+0000) 2019-10-13T13:37:16.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:16.329+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.332065+0000) 2019-10-13T13:37:16.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:16.329+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.332685+0000) 2019-10-13T13:37:16.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:16.327+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.333880+0000) 2019-10-13T13:37:16.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:16.329+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.333905+0000) 2019-10-13T13:37:16.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:16.329+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.334488+0000) 2019-10-13T13:37:16.337 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:16.329+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.334975+0000) 2019-10-13T13:37:16.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:16.335+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.339815+0000) 2019-10-13T13:37:16.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:16.337+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176264 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:16.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:16.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:16.561154+0000) 2019-10-13T13:37:18.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:18.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.827976+0000) 2019-10-13T13:37:18.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:18.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.829200+0000) 2019-10-13T13:37:20.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:20.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:20.061496+0000) 2019-10-13T13:37:21.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:21.337+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.343067+0000) 2019-10-13T13:37:21.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:21.341+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.343531+0000) 2019-10-13T13:37:21.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:21.339+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.344018+0000) 2019-10-13T13:37:21.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:21.341+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.345105+0000) 2019-10-13T13:37:21.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:21.341+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.345329+0000) 2019-10-13T13:37:21.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:21.341+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.346277+0000) 2019-10-13T13:37:21.349 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:21.341+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.346688+0000) 2019-10-13T13:37:21.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:21.347+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.350759+0000) 2019-10-13T13:37:21.354 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:21.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:22.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:22.565+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:22.569422+0000) 2019-10-13T13:37:23.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:23.327+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.329501+0000) 2019-10-13T13:37:25.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:25.593+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:25.594025+0000) 2019-10-13T13:37:25.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:25.877+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:25.880539+0000) 2019-10-13T13:37:26.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:26.349+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.355253+0000) 2019-10-13T13:37:26.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:26.353+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.355773+0000) 2019-10-13T13:37:26.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:26.353+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.356493+0000) 2019-10-13T13:37:26.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:26.353+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.357181+0000) 2019-10-13T13:37:26.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:26.353+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.357534+0000) 2019-10-13T13:37:26.360 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:26.355+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.358129+0000) 2019-10-13T13:37:26.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:26.353+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.358618+0000) 2019-10-13T13:37:26.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:26.359+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.364808+0000) 2019-10-13T13:37:26.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:26.361+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176333 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:26.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:26.557+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:26.561351+0000) 2019-10-13T13:37:28.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:28.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.828177+0000) 2019-10-13T13:37:28.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:28.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.829390+0000) 2019-10-13T13:37:30.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:30.055+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:30.061707+0000) 2019-10-13T13:37:31.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:31.365+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.368451+0000) 2019-10-13T13:37:31.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:31.365+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.369107+0000) 2019-10-13T13:37:31.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:31.365+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.369650+0000) 2019-10-13T13:37:31.372 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:31.365+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.370265+0000) 2019-10-13T13:37:31.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:31.365+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.370965+0000) 2019-10-13T13:37:31.373 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:31.367+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.371478+0000) 2019-10-13T13:37:31.374 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:31.369+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.371493+0000) 2019-10-13T13:37:31.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:31.373+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:31.380 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:31.375+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.378531+0000) 2019-10-13T13:37:32.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:32.569+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:32.569652+0000) 2019-10-13T13:37:33.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:33.338+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.340800+0000) 2019-10-13T13:37:35.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:35.589+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:35.594236+0000) 2019-10-13T13:37:35.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:35.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:35.880737+0000) 2019-10-13T13:37:36.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:36.376+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.380850+0000) 2019-10-13T13:37:36.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:36.376+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.381082+0000) 2019-10-13T13:37:36.384 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:36.376+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.381733+0000) 2019-10-13T13:37:36.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:36.376+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.382444+0000) 2019-10-13T13:37:36.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:36.378+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.383109+0000) 2019-10-13T13:37:36.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:36.376+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.383146+0000) 2019-10-13T13:37:36.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:36.380+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.384470+0000) 2019-10-13T13:37:36.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:36.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:36.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:36.386+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.390608+0000) 2019-10-13T13:37:36.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:36.560+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:36.561525+0000) 2019-10-13T13:37:38.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:38.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.828397+0000) 2019-10-13T13:37:38.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:38.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.829603+0000) 2019-10-13T13:37:40.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:40.054+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:40.061941+0000) 2019-10-13T13:37:41.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:41.388+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.392272+0000) 2019-10-13T13:37:41.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:41.388+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.393167+0000) 2019-10-13T13:37:41.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:41.388+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.394400+0000) 2019-10-13T13:37:41.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:41.390+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.394734+0000) 2019-10-13T13:37:41.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:41.392+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.395388+0000) 2019-10-13T13:37:41.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:41.392+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.395486+0000) 2019-10-13T13:37:41.398 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:41.392+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.395819+0000) 2019-10-13T13:37:41.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:41.394+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.400395+0000) 2019-10-13T13:37:41.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:41.396+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:42.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:42.568+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:42.569851+0000) 2019-10-13T13:37:43.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:43.342+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.347234+0000) 2019-10-13T13:37:44.018 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:37:44.018 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:44.021 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:37:44.021 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:45.596 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:45.592+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:45.594435+0000) 2019-10-13T13:37:45.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:45.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:45.880945+0000) 2019-10-13T13:37:46.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:46.402+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.405595+0000) 2019-10-13T13:37:46.408 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:46.400+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.406337+0000) 2019-10-13T13:37:46.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:46.400+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.406876+0000) 2019-10-13T13:37:46.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:46.404+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.407581+0000) 2019-10-13T13:37:46.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:46.404+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.407746+0000) 2019-10-13T13:37:46.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:46.404+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.408002+0000) 2019-10-13T13:37:46.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:46.404+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.408466+0000) 2019-10-13T13:37:46.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:46.406+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.413486+0000) 2019-10-13T13:37:46.415 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:46.408+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:46.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:46.556+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:46.561735+0000) 2019-10-13T13:37:48.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:48.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.828622+0000) 2019-10-13T13:37:48.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:48.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.829802+0000) 2019-10-13T13:37:50.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:50.058+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:50.062127+0000) 2019-10-13T13:37:51.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:51.412+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.416195+0000) 2019-10-13T13:37:51.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:51.412+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.416839+0000) 2019-10-13T13:37:51.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:51.412+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.417648+0000) 2019-10-13T13:37:51.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:51.412+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.417884+0000) 2019-10-13T13:37:51.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:51.414+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.419028+0000) 2019-10-13T13:37:51.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:51.412+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.418902+0000) 2019-10-13T13:37:51.422 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:51.416+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.419776+0000) 2019-10-13T13:37:51.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:51.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:51.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:51.418+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.424980+0000) 2019-10-13T13:37:52.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:52.568+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:52.570082+0000) 2019-10-13T13:37:53.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:53.346+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.353740+0000) 2019-10-13T13:37:55.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:55.592+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:55.594635+0000) 2019-10-13T13:37:55.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:55.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:55.881171+0000) 2019-10-13T13:37:56.430 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:37:56.424+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.427958+0000) 2019-10-13T13:37:56.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:37:56.424+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.428595+0000) 2019-10-13T13:37:56.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:56.424+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.429374+0000) 2019-10-13T13:37:56.431 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:56.424+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.428926+0000) 2019-10-13T13:37:56.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:37:56.426+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.430268+0000) 2019-10-13T13:37:56.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:37:56.424+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.430428+0000) 2019-10-13T13:37:56.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:56.424+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.430983+0000) 2019-10-13T13:37:56.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:37:56.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.436612+0000) 2019-10-13T13:37:56.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:37:56.432+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176535 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:37:56.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:37:56.560+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:56.561917+0000) 2019-10-13T13:37:58.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:37:58.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.828819+0000) 2019-10-13T13:37:58.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:37:58.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.829957+0000) 2019-10-13T13:38:00.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:00.058+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:00.062374+0000) 2019-10-13T13:38:01.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:01.436+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.439864+0000) 2019-10-13T13:38:01.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:01.434+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.441039+0000) 2019-10-13T13:38:01.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:01.436+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.440295+0000) 2019-10-13T13:38:01.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:01.436+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.441367+0000) 2019-10-13T13:38:01.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:01.436+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.441656+0000) 2019-10-13T13:38:01.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:01.436+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.442752+0000) 2019-10-13T13:38:01.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:01.440+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.443425+0000) 2019-10-13T13:38:01.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:01.442+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.448637+0000) 2019-10-13T13:38:01.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:01.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176571 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:02.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:02.568+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:02.570312+0000) 2019-10-13T13:38:03.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:03.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.360217+0000) 2019-10-13T13:38:05.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:05.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:05.594825+0000) 2019-10-13T13:38:05.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:05.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:05.881379+0000) 2019-10-13T13:38:06.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:06.446+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.452617+0000) 2019-10-13T13:38:06.455 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:06.448+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.452664+0000) 2019-10-13T13:38:06.456 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:06.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.453048+0000) 2019-10-13T13:38:06.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:06.448+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.453728+0000) 2019-10-13T13:38:06.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:06.448+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.453953+0000) 2019-10-13T13:38:06.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:06.448+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.455232+0000) 2019-10-13T13:38:06.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:06.452+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.455600+0000) 2019-10-13T13:38:06.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:06.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:06.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:06.454+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.461467+0000) 2019-10-13T13:38:06.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:06.560+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:06.562093+0000) 2019-10-13T13:38:08.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:08.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.829047+0000) 2019-10-13T13:38:08.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:08.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.830138+0000) 2019-10-13T13:38:10.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:10.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:10.062612+0000) 2019-10-13T13:38:11.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:11.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.465024+0000) 2019-10-13T13:38:11.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:11.460+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.465152+0000) 2019-10-13T13:38:11.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:11.460+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.465501+0000) 2019-10-13T13:38:11.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:11.460+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.465801+0000) 2019-10-13T13:38:11.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:11.460+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.466545+0000) 2019-10-13T13:38:11.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:11.462+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.467138+0000) 2019-10-13T13:38:11.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:11.464+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.467873+0000) 2019-10-13T13:38:11.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:11.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:11.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:11.470+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.474566+0000) 2019-10-13T13:38:12.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:12.568+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:12.570534+0000) 2019-10-13T13:38:13.369 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:13.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.366765+0000) 2019-10-13T13:38:14.136 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:38:14.136 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:14.140 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:38:14.140 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:15.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:15.588+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:15.595034+0000) 2019-10-13T13:38:15.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:15.876+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:15.881595+0000) 2019-10-13T13:38:16.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:16.470+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.476386+0000) 2019-10-13T13:38:16.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:16.472+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.476628+0000) 2019-10-13T13:38:16.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:16.472+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.477767+0000) 2019-10-13T13:38:16.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:16.472+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.477863+0000) 2019-10-13T13:38:16.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:16.472+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.478273+0000) 2019-10-13T13:38:16.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:16.472+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.479125+0000) 2019-10-13T13:38:16.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:16.476+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.479336+0000) 2019-10-13T13:38:16.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:16.478+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.484140+0000) 2019-10-13T13:38:16.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:16.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176671 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:16.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:16.560+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:16.562293+0000) 2019-10-13T13:38:18.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:18.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.829285+0000) 2019-10-13T13:38:18.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:18.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.830339+0000) 2019-10-13T13:38:20.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:20.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:20.062803+0000) 2019-10-13T13:38:21.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:21.483+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.488634+0000) 2019-10-13T13:38:21.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:21.483+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.489155+0000) 2019-10-13T13:38:21.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:21.490+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.490753+0000) 2019-10-13T13:38:21.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:21.483+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.490781+0000) 2019-10-13T13:38:21.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:21.483+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.491031+0000) 2019-10-13T13:38:21.494 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:21.487+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.491692+0000) 2019-10-13T13:38:21.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:21.487+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.492367+0000) 2019-10-13T13:38:21.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:21.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:21.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:21.498+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.499057+0000) 2019-10-13T13:38:22.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:22.563+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:22.570741+0000) 2019-10-13T13:38:23.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:23.370+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.373198+0000) 2019-10-13T13:38:25.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:25.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:25.595237+0000) 2019-10-13T13:38:25.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:25.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:25.881793+0000) 2019-10-13T13:38:26.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:26.495+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.502036+0000) 2019-10-13T13:38:26.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:26.495+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.502769+0000) 2019-10-13T13:38:26.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:26.499+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.503704+0000) 2019-10-13T13:38:26.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:26.499+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.503961+0000) 2019-10-13T13:38:26.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:26.499+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.504487+0000) 2019-10-13T13:38:26.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:26.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.505071+0000) 2019-10-13T13:38:26.507 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:26.499+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.505390+0000) 2019-10-13T13:38:26.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:26.506+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.510008+0000) 2019-10-13T13:38:26.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:26.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:26.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:26.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:26.562489+0000) 2019-10-13T13:38:28.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:28.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.829509+0000) 2019-10-13T13:38:28.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:28.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.830569+0000) 2019-10-13T13:38:30.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:30.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:30.063018+0000) 2019-10-13T13:38:31.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:31.511+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.515485+0000) 2019-10-13T13:38:31.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:31.511+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.515877+0000) 2019-10-13T13:38:31.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:31.511+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.516465+0000) 2019-10-13T13:38:31.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:31.511+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.517166+0000) 2019-10-13T13:38:31.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:31.513+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.517770+0000) 2019-10-13T13:38:31.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:31.511+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.517758+0000) 2019-10-13T13:38:31.520 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:31.511+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.518411+0000) 2019-10-13T13:38:31.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:31.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:31.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:31.521+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.525522+0000) 2019-10-13T13:38:32.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:32.567+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:32.570940+0000) 2019-10-13T13:38:33.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:33.381+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.384971+0000) 2019-10-13T13:38:35.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:35.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:35.595491+0000) 2019-10-13T13:38:35.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:35.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:35.882016+0000) 2019-10-13T13:38:36.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:36.519+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.527249+0000) 2019-10-13T13:38:36.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:36.523+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.527814+0000) 2019-10-13T13:38:36.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:36.523+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.527981+0000) 2019-10-13T13:38:36.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:36.523+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.528665+0000) 2019-10-13T13:38:36.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:36.523+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.529334+0000) 2019-10-13T13:38:36.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:36.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.529841+0000) 2019-10-13T13:38:36.532 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:36.523+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.530381+0000) 2019-10-13T13:38:36.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:36.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.535740+0000) 2019-10-13T13:38:36.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:36.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:36.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:36.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:36.562667+0000) 2019-10-13T13:38:38.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:38.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.829761+0000) 2019-10-13T13:38:38.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:38.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.830792+0000) 2019-10-13T13:38:40.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:40.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:40.063202+0000) 2019-10-13T13:38:41.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:41.535+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.540574+0000) 2019-10-13T13:38:41.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:41.535+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.541361+0000) 2019-10-13T13:38:41.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:41.535+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.541785+0000) 2019-10-13T13:38:41.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:41.535+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.542292+0000) 2019-10-13T13:38:41.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:41.535+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.542979+0000) 2019-10-13T13:38:41.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:41.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.543358+0000) 2019-10-13T13:38:41.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:41.539+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.543702+0000) 2019-10-13T13:38:41.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:41.545+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.549925+0000) 2019-10-13T13:38:41.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:41.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:42.573 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:42.567+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:42.571165+0000) 2019-10-13T13:38:43.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:43.389+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.391233+0000) 2019-10-13T13:38:44.255 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:38:44.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:44.259 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:38:44.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:45.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:45.591+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:45.595691+0000) 2019-10-13T13:38:45.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:45.875+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:45.882200+0000) 2019-10-13T13:38:46.558 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:46.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.555986+0000) 2019-10-13T13:38:46.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:46.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.554190+0000) 2019-10-13T13:38:46.559 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:46.547+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.554635+0000) 2019-10-13T13:38:46.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:46.547+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.554952+0000) 2019-10-13T13:38:46.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:46.551+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.556653+0000) 2019-10-13T13:38:46.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:46.551+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.556785+0000) 2019-10-13T13:38:46.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:46.551+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.557023+0000) 2019-10-13T13:38:46.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:46.557+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.560237+0000) 2019-10-13T13:38:46.564 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:46.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:46.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:46.555+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:46.562824+0000) 2019-10-13T13:38:48.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:48.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.829963+0000) 2019-10-13T13:38:48.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:48.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.830996+0000) 2019-10-13T13:38:50.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:50.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:50.063391+0000) 2019-10-13T13:38:51.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:51.561+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.565695+0000) 2019-10-13T13:38:51.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:51.563+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.566019+0000) 2019-10-13T13:38:51.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:51.563+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.566233+0000) 2019-10-13T13:38:51.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:51.563+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.566569+0000) 2019-10-13T13:38:51.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:51.567+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.567565+0000) 2019-10-13T13:38:51.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:51.567+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.568856+0000) 2019-10-13T13:38:51.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:51.567+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.569131+0000) 2019-10-13T13:38:51.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:51.573+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.575359+0000) 2019-10-13T13:38:51.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:51.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176910 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:52.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:52.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:52.571394+0000) 2019-10-13T13:38:53.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:53.393+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.397692+0000) 2019-10-13T13:38:55.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:55.595+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:55.595888+0000) 2019-10-13T13:38:55.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:55.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:55.882413+0000) 2019-10-13T13:38:56.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:56.559+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.563010+0000) 2019-10-13T13:38:56.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:38:56.575+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.578117+0000) 2019-10-13T13:38:56.581 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:38:56.575+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.578865+0000) 2019-10-13T13:38:56.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:56.579+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.579533+0000) 2019-10-13T13:38:56.582 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:38:56.577+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.580117+0000) 2019-10-13T13:38:56.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:56.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.580192+0000) 2019-10-13T13:38:56.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:38:56.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.580999+0000) 2019-10-13T13:38:56.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:38:56.579+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.581777+0000) 2019-10-13T13:38:56.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:38:56.581+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:56.585814+0000) 2019-10-13T13:38:56.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:38:56.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:38:58.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:38:58.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.830158+0000) 2019-10-13T13:38:58.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:38:58.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.831176+0000) 2019-10-13T13:39:00.065 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:00.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:00.063588+0000) 2019-10-13T13:39:01.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:01.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.591088+0000) 2019-10-13T13:39:01.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:01.591+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.591550+0000) 2019-10-13T13:39:01.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:01.591+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.592234+0000) 2019-10-13T13:39:01.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:01.591+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.593034+0000) 2019-10-13T13:39:01.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:01.591+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.593326+0000) 2019-10-13T13:39:01.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:01.593+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.594477+0000) 2019-10-13T13:39:01.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:01.591+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.594690+0000) 2019-10-13T13:39:01.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:01.597+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.598432+0000) 2019-10-13T13:39:01.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:01.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 176979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:02.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:02.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:02.571600+0000) 2019-10-13T13:39:03.406 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:03.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.404283+0000) 2019-10-13T13:39:05.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:05.594+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:05.596117+0000) 2019-10-13T13:39:05.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:05.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:05.882607+0000) 2019-10-13T13:39:06.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:06.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.563245+0000) 2019-10-13T13:39:06.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:06.602+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.603813+0000) 2019-10-13T13:39:06.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:06.602+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.604825+0000) 2019-10-13T13:39:06.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:06.602+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.605112+0000) 2019-10-13T13:39:06.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:06.602+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.605453+0000) 2019-10-13T13:39:06.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:06.602+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.605729+0000) 2019-10-13T13:39:06.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:06.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.606614+0000) 2019-10-13T13:39:06.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:06.605+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.607410+0000) 2019-10-13T13:39:06.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:06.609+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:06.610989+0000) 2019-10-13T13:39:06.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:06.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177012 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:08.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:08.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.830357+0000) 2019-10-13T13:39:08.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:08.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.831398+0000) 2019-10-13T13:39:10.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:10.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:10.063800+0000) 2019-10-13T13:39:11.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:11.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.615665+0000) 2019-10-13T13:39:11.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:11.614+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.616441+0000) 2019-10-13T13:39:11.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:11.614+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.616704+0000) 2019-10-13T13:39:11.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:11.614+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.617300+0000) 2019-10-13T13:39:11.620 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:11.617+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.618273+0000) 2019-10-13T13:39:11.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:11.614+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.617697+0000) 2019-10-13T13:39:11.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:11.614+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.619245+0000) 2019-10-13T13:39:11.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:11.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:11.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:11.621+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.625914+0000) 2019-10-13T13:39:12.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:12.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:12.571813+0000) 2019-10-13T13:39:13.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:13.409+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.410771+0000) 2019-10-13T13:39:14.303 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:39:14.303 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:14.307 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:39:14.307 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:15.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:15.594+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:15.596340+0000) 2019-10-13T13:39:15.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:15.878+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:15.882820+0000) 2019-10-13T13:39:16.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:16.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.563484+0000) 2019-10-13T13:39:16.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:16.626+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.628966+0000) 2019-10-13T13:39:16.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:16.626+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.628718+0000) 2019-10-13T13:39:16.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:16.626+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.629374+0000) 2019-10-13T13:39:16.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:16.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.629959+0000) 2019-10-13T13:39:16.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:16.626+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.630597+0000) 2019-10-13T13:39:16.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:16.630+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.632143+0000) 2019-10-13T13:39:16.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:16.629+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.633142+0000) 2019-10-13T13:39:16.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:16.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:16.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:16.641+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:16.642461+0000) 2019-10-13T13:39:18.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:18.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.830561+0000) 2019-10-13T13:39:18.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:18.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.831604+0000) 2019-10-13T13:39:20.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:20.061+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:20.064027+0000) 2019-10-13T13:39:21.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:21.638+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.640579+0000) 2019-10-13T13:39:21.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:21.638+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.641336+0000) 2019-10-13T13:39:21.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:21.638+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.641473+0000) 2019-10-13T13:39:21.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:21.638+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.641935+0000) 2019-10-13T13:39:21.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:21.638+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.642475+0000) 2019-10-13T13:39:21.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:21.642+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.644019+0000) 2019-10-13T13:39:21.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:21.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:21.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:21.640+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.644072+0000) 2019-10-13T13:39:21.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:21.644+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.649279+0000) 2019-10-13T13:39:22.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:22.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:22.572029+0000) 2019-10-13T13:39:23.419 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:23.412+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.417103+0000) 2019-10-13T13:39:25.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:25.594+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:25.596543+0000) 2019-10-13T13:39:25.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:25.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:25.883050+0000) 2019-10-13T13:39:26.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:26.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.563675+0000) 2019-10-13T13:39:26.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:26.650+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.653584+0000) 2019-10-13T13:39:26.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:26.650+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.653977+0000) 2019-10-13T13:39:26.656 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:26.650+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.654340+0000) 2019-10-13T13:39:26.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:26.650+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.654988+0000) 2019-10-13T13:39:26.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:26.654+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.655640+0000) 2019-10-13T13:39:26.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:26.654+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.656410+0000) 2019-10-13T13:39:26.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:26.652+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.657178+0000) 2019-10-13T13:39:26.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:26.660+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:26.662401+0000) 2019-10-13T13:39:26.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:26.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:28.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:28.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.830763+0000) 2019-10-13T13:39:28.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:28.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.831800+0000) 2019-10-13T13:39:30.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:30.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:30.064217+0000) 2019-10-13T13:39:31.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:31.660+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.666091+0000) 2019-10-13T13:39:31.668 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:31.662+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.666483+0000) 2019-10-13T13:39:31.669 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:31.662+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.666748+0000) 2019-10-13T13:39:31.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:31.662+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.667156+0000) 2019-10-13T13:39:31.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:31.666+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.667520+0000) 2019-10-13T13:39:31.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:31.666+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.668213+0000) 2019-10-13T13:39:31.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:31.666+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.668883+0000) 2019-10-13T13:39:31.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:31.672+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.674620+0000) 2019-10-13T13:39:31.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:31.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:32.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:32.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:32.572204+0000) 2019-10-13T13:39:33.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:33.420+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.422938+0000) 2019-10-13T13:39:35.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:35.594+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:35.596750+0000) 2019-10-13T13:39:35.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:35.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:35.883250+0000) 2019-10-13T13:39:36.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:36.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.563903+0000) 2019-10-13T13:39:36.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:36.674+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.678200+0000) 2019-10-13T13:39:36.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:36.674+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.678623+0000) 2019-10-13T13:39:36.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:36.674+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.679170+0000) 2019-10-13T13:39:36.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:36.678+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.679589+0000) 2019-10-13T13:39:36.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:36.676+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.680716+0000) 2019-10-13T13:39:36.683 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:36.678+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.680817+0000) 2019-10-13T13:39:36.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:36.678+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.681511+0000) 2019-10-13T13:39:36.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:36.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:36.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:36.684+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:36.687347+0000) 2019-10-13T13:39:38.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:38.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.830960+0000) 2019-10-13T13:39:38.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:38.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.831989+0000) 2019-10-13T13:39:40.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:40.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:40.064414+0000) 2019-10-13T13:39:41.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:41.686+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.690158+0000) 2019-10-13T13:39:41.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:41.686+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.690619+0000) 2019-10-13T13:39:41.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:41.690+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.691288+0000) 2019-10-13T13:39:41.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:41.690+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.691955+0000) 2019-10-13T13:39:41.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:41.690+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.692276+0000) 2019-10-13T13:39:41.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:41.690+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.692564+0000) 2019-10-13T13:39:41.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:41.688+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.693283+0000) 2019-10-13T13:39:41.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:41.692+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.698050+0000) 2019-10-13T13:39:41.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:41.698+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:42.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:42.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:42.572385+0000) 2019-10-13T13:39:43.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:43.428+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.429264+0000) 2019-10-13T13:39:44.419 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:39:44.419 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:44.422 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:39:44.423 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:45.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:45.593+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:45.596977+0000) 2019-10-13T13:39:45.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:45.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:45.883499+0000) 2019-10-13T13:39:46.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:46.561+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.564097+0000) 2019-10-13T13:39:46.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:46.697+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.702882+0000) 2019-10-13T13:39:46.706 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:46.701+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.703829+0000) 2019-10-13T13:39:46.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:46.701+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.704269+0000) 2019-10-13T13:39:46.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:46.701+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.704808+0000) 2019-10-13T13:39:46.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:46.701+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.705049+0000) 2019-10-13T13:39:46.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:46.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.705947+0000) 2019-10-13T13:39:46.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:46.701+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.706084+0000) 2019-10-13T13:39:46.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:46.709+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:46.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:46.708+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:46.712976+0000) 2019-10-13T13:39:48.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:48.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.831193+0000) 2019-10-13T13:39:48.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:48.829+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.832199+0000) 2019-10-13T13:39:50.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:50.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:50.064601+0000) 2019-10-13T13:39:51.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:51.709+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.715165+0000) 2019-10-13T13:39:51.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:51.713+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.715697+0000) 2019-10-13T13:39:51.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:51.713+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.716683+0000) 2019-10-13T13:39:51.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:51.712+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.717875+0000) 2019-10-13T13:39:51.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:51.713+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.717928+0000) 2019-10-13T13:39:51.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:51.713+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.718459+0000) 2019-10-13T13:39:51.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:51.713+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.719279+0000) 2019-10-13T13:39:51.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:51.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:51.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:51.720+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.725353+0000) 2019-10-13T13:39:52.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:52.569+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:52.572566+0000) 2019-10-13T13:39:53.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:53.432+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.435611+0000) 2019-10-13T13:39:55.599 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:55.593+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:55.597230+0000) 2019-10-13T13:39:55.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:55.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:55.883724+0000) 2019-10-13T13:39:56.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:56.561+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.564294+0000) 2019-10-13T13:39:56.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:56.725+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.727939+0000) 2019-10-13T13:39:56.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:39:56.725+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.729346+0000) 2019-10-13T13:39:56.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:39:56.725+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.729904+0000) 2019-10-13T13:39:56.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:56.725+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.730304+0000) 2019-10-13T13:39:56.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:39:56.728+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.731064+0000) 2019-10-13T13:39:56.733 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:39:56.725+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.731161+0000) 2019-10-13T13:39:56.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:39:56.729+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.732204+0000) 2019-10-13T13:39:56.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:39:56.732+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:56.737128+0000) 2019-10-13T13:39:56.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:39:56.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:39:58.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:39:58.829+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.831397+0000) 2019-10-13T13:39:58.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:39:58.829+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.832371+0000) 2019-10-13T13:40:00.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:00.060+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:00.064818+0000) 2019-10-13T13:40:01.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:01.737+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.739841+0000) 2019-10-13T13:40:01.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:01.737+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.740371+0000) 2019-10-13T13:40:01.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:01.737+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.741710+0000) 2019-10-13T13:40:01.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:01.736+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.742053+0000) 2019-10-13T13:40:01.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:01.737+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.742262+0000) 2019-10-13T13:40:01.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:01.741+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.743340+0000) 2019-10-13T13:40:01.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:01.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.743720+0000) 2019-10-13T13:40:01.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:01.744+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.748229+0000) 2019-10-13T13:40:01.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:01.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:02.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:02.569+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:02.572773+0000) 2019-10-13T13:40:03.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:03.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.446823+0000) 2019-10-13T13:40:05.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:05.593+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:05.597424+0000) 2019-10-13T13:40:05.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:05.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:05.883922+0000) 2019-10-13T13:40:06.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:06.561+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.564491+0000) 2019-10-13T13:40:06.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:06.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.753743+0000) 2019-10-13T13:40:06.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:06.749+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.754181+0000) 2019-10-13T13:40:06.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:06.749+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.754605+0000) 2019-10-13T13:40:06.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:06.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.755289+0000) 2019-10-13T13:40:06.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:06.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.756053+0000) 2019-10-13T13:40:06.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:06.752+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.756516+0000) 2019-10-13T13:40:06.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:06.753+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.756684+0000) 2019-10-13T13:40:06.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:06.756+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:06.759798+0000) 2019-10-13T13:40:06.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:06.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177420 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:08.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:08.829+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.831619+0000) 2019-10-13T13:40:08.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:08.829+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.832576+0000) 2019-10-13T13:40:10.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:10.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:10.065005+0000) 2019-10-13T13:40:11.768 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:11.764+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.766260+0000) 2019-10-13T13:40:11.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:11.761+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.766729+0000) 2019-10-13T13:40:11.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:11.765+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.767374+0000) 2019-10-13T13:40:11.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:11.761+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.766919+0000) 2019-10-13T13:40:11.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:11.765+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.767851+0000) 2019-10-13T13:40:11.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:11.765+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.768258+0000) 2019-10-13T13:40:11.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:11.765+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.768746+0000) 2019-10-13T13:40:11.775 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:11.768+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.772919+0000) 2019-10-13T13:40:11.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:11.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177452 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:12.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:12.569+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:12.572953+0000) 2019-10-13T13:40:13.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:13.452+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.453170+0000) 2019-10-13T13:40:14.522 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:40:14.522 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:14.525 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:40:14.525 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:15.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:15.597+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:15.597626+0000) 2019-10-13T13:40:15.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:15.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:15.884115+0000) 2019-10-13T13:40:16.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:16.561+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.564689+0000) 2019-10-13T13:40:16.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:16.773+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.778304+0000) 2019-10-13T13:40:16.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:16.773+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.778752+0000) 2019-10-13T13:40:16.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:16.775+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.779551+0000) 2019-10-13T13:40:16.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:16.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.779830+0000) 2019-10-13T13:40:16.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:16.777+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.779890+0000) 2019-10-13T13:40:16.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:16.777+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.780652+0000) 2019-10-13T13:40:16.783 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:16.777+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.781459+0000) 2019-10-13T13:40:16.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:16.783+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:16.786934+0000) 2019-10-13T13:40:16.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:16.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:18.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:18.829+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.831838+0000) 2019-10-13T13:40:18.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:18.829+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.832753+0000) 2019-10-13T13:40:20.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:20.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:20.065198+0000) 2019-10-13T13:40:21.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:21.785+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.789762+0000) 2019-10-13T13:40:21.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:21.785+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.790198+0000) 2019-10-13T13:40:21.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:21.787+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.791028+0000) 2019-10-13T13:40:21.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:21.789+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.792184+0000) 2019-10-13T13:40:21.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:21.789+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.792484+0000) 2019-10-13T13:40:21.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:21.789+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.793079+0000) 2019-10-13T13:40:21.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:21.789+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.793308+0000) 2019-10-13T13:40:21.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:21.795+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.798413+0000) 2019-10-13T13:40:21.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:21.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:22.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:22.569+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:22.573134+0000) 2019-10-13T13:40:23.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:23.455+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.459529+0000) 2019-10-13T13:40:25.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:25.597+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:25.597833+0000) 2019-10-13T13:40:25.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:25.881+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:25.884362+0000) 2019-10-13T13:40:26.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:26.561+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.564881+0000) 2019-10-13T13:40:26.805 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:26.799+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.802871+0000) 2019-10-13T13:40:26.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:26.801+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.803561+0000) 2019-10-13T13:40:26.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:26.801+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.804028+0000) 2019-10-13T13:40:26.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:26.801+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.804449+0000) 2019-10-13T13:40:26.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:26.801+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.805201+0000) 2019-10-13T13:40:26.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:26.801+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.806210+0000) 2019-10-13T13:40:26.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:26.801+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.806569+0000) 2019-10-13T13:40:26.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:26.807+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:26.812140+0000) 2019-10-13T13:40:26.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:26.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:28.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:28.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.832037+0000) 2019-10-13T13:40:28.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:28.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.832976+0000) 2019-10-13T13:40:30.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:30.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:30.065398+0000) 2019-10-13T13:40:31.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:31.812+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.815876+0000) 2019-10-13T13:40:31.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:31.812+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.816529+0000) 2019-10-13T13:40:31.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:31.812+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.817410+0000) 2019-10-13T13:40:31.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:31.812+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.817143+0000) 2019-10-13T13:40:31.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:31.812+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.818089+0000) 2019-10-13T13:40:31.821 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:31.815+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.818877+0000) 2019-10-13T13:40:31.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:31.812+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.819169+0000) 2019-10-13T13:40:31.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:31.819+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.824785+0000) 2019-10-13T13:40:31.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:31.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:32.575 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:32.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:32.573316+0000) 2019-10-13T13:40:33.468 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:33.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.465854+0000) 2019-10-13T13:40:35.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:35.596+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:35.598062+0000) 2019-10-13T13:40:35.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:35.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:35.884523+0000) 2019-10-13T13:40:36.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:36.560+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.565130+0000) 2019-10-13T13:40:36.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:36.824+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.828135+0000) 2019-10-13T13:40:36.831 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:36.823+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.828956+0000) 2019-10-13T13:40:36.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:36.824+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.829121+0000) 2019-10-13T13:40:36.832 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:36.824+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.829384+0000) 2019-10-13T13:40:36.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:36.824+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.829678+0000) 2019-10-13T13:40:36.833 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:36.824+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.830619+0000) 2019-10-13T13:40:36.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:36.828+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.831676+0000) 2019-10-13T13:40:36.840 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:36.832+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:36.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:36.835+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:36.839815+0000) 2019-10-13T13:40:38.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:38.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.832259+0000) 2019-10-13T13:40:38.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:38.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.833194+0000) 2019-10-13T13:40:40.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:40.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:40.065597+0000) 2019-10-13T13:40:41.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:41.836+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.841475+0000) 2019-10-13T13:40:41.844 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:41.836+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.842024+0000) 2019-10-13T13:40:41.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:41.836+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.842514+0000) 2019-10-13T13:40:41.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:41.839+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.843196+0000) 2019-10-13T13:40:41.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:41.840+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.843800+0000) 2019-10-13T13:40:41.846 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:41.840+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.843954+0000) 2019-10-13T13:40:41.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:41.840+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.844744+0000) 2019-10-13T13:40:41.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:41.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:41.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:41.847+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.851488+0000) 2019-10-13T13:40:42.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:42.568+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:42.573496+0000) 2019-10-13T13:40:43.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:43.471+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.472298+0000) 2019-10-13T13:40:44.633 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:40:44.633 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:44.637 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:40:44.637 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:45.600 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:45.592+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:45.598267+0000) 2019-10-13T13:40:45.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:45.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:45.884773+0000) 2019-10-13T13:40:46.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:46.564+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.565320+0000) 2019-10-13T13:40:46.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:46.848+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.853393+0000) 2019-10-13T13:40:46.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:46.848+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.853991+0000) 2019-10-13T13:40:46.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:46.848+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.855155+0000) 2019-10-13T13:40:46.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:46.852+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.855606+0000) 2019-10-13T13:40:46.858 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:46.852+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.855920+0000) 2019-10-13T13:40:46.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:46.852+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.856414+0000) 2019-10-13T13:40:46.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:46.851+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.857100+0000) 2019-10-13T13:40:46.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:46.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:46.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:46.859+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:46.864245+0000) 2019-10-13T13:40:48.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:48.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.832491+0000) 2019-10-13T13:40:48.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:48.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.833388+0000) 2019-10-13T13:40:50.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:50.059+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:50.065781+0000) 2019-10-13T13:40:51.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:51.860+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.865230+0000) 2019-10-13T13:40:51.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:51.860+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.865707+0000) 2019-10-13T13:40:51.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:51.860+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.866618+0000) 2019-10-13T13:40:51.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:51.860+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.866645+0000) 2019-10-13T13:40:51.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:51.864+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.868084+0000) 2019-10-13T13:40:51.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:51.864+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.868612+0000) 2019-10-13T13:40:51.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:51.867+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.870164+0000) 2019-10-13T13:40:51.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:51.871+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.875406+0000) 2019-10-13T13:40:51.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:51.868+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:52.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:52.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:52.573739+0000) 2019-10-13T13:40:53.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:53.479+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.483233+0000) 2019-10-13T13:40:55.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:55.596+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:55.598503+0000) 2019-10-13T13:40:55.887 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:55.880+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:55.884988+0000) 2019-10-13T13:40:56.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:56.564+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.565546+0000) 2019-10-13T13:40:56.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:40:56.872+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.878462+0000) 2019-10-13T13:40:56.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:56.876+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.879871+0000) 2019-10-13T13:40:56.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:40:56.876+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.880216+0000) 2019-10-13T13:40:56.883 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:40:56.876+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.880595+0000) 2019-10-13T13:40:56.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:40:56.875+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.881438+0000) 2019-10-13T13:40:56.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:56.876+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.880888+0000) 2019-10-13T13:40:56.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:40:56.876+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.882116+0000) 2019-10-13T13:40:56.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:40:56.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:40:56.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:40:56.883+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:56.888318+0000) 2019-10-13T13:40:58.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:40:58.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.832725+0000) 2019-10-13T13:40:58.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:40:58.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.833576+0000) 2019-10-13T13:41:00.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:00.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:00.065977+0000) 2019-10-13T13:41:01.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:01.884+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.890345+0000) 2019-10-13T13:41:01.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:01.888+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.891349+0000) 2019-10-13T13:41:01.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:01.888+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.891869+0000) 2019-10-13T13:41:01.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:01.888+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.892772+0000) 2019-10-13T13:41:01.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:01.888+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.893878+0000) 2019-10-13T13:41:01.896 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:01.888+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.894263+0000) 2019-10-13T13:41:01.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:01.891+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.895007+0000) 2019-10-13T13:41:01.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:01.895+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.899610+0000) 2019-10-13T13:41:01.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:01.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177794 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:02.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:02.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:02.573949+0000) 2019-10-13T13:41:03.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:03.487+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.489633+0000) 2019-10-13T13:41:05.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:05.596+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:05.598716+0000) 2019-10-13T13:41:05.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:05.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:05.885195+0000) 2019-10-13T13:41:06.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:06.564+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.565749+0000) 2019-10-13T13:41:06.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:06.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.902614+0000) 2019-10-13T13:41:06.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:06.900+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.903582+0000) 2019-10-13T13:41:06.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:06.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.903959+0000) 2019-10-13T13:41:06.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:06.900+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.905275+0000) 2019-10-13T13:41:06.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:06.899+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.905898+0000) 2019-10-13T13:41:06.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:06.900+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.905974+0000) 2019-10-13T13:41:06.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:06.900+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.906250+0000) 2019-10-13T13:41:06.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:06.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:06.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:06.907+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:06.913787+0000) 2019-10-13T13:41:08.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:08.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.832926+0000) 2019-10-13T13:41:08.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:08.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.833791+0000) 2019-10-13T13:41:10.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:10.063+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:10.066128+0000) 2019-10-13T13:41:11.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:11.910+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.916084+0000) 2019-10-13T13:41:11.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:11.911+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.916109+0000) 2019-10-13T13:41:11.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:11.911+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.916409+0000) 2019-10-13T13:41:11.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:11.911+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.916826+0000) 2019-10-13T13:41:11.920 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:11.911+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.917671+0000) 2019-10-13T13:41:11.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:11.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.918598+0000) 2019-10-13T13:41:11.921 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:11.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.919106+0000) 2019-10-13T13:41:11.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:11.918+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.924849+0000) 2019-10-13T13:41:11.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:11.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:12.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:12.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:12.574088+0000) 2019-10-13T13:41:13.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:13.494+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.496112+0000) 2019-10-13T13:41:14.769 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:41:14.769 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:14.773 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:41:14.773 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:15.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:15.595+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:15.598939+0000) 2019-10-13T13:41:15.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:15.879+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:15.885431+0000) 2019-10-13T13:41:16.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:16.563+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.565939+0000) 2019-10-13T13:41:16.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:16.923+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.928139+0000) 2019-10-13T13:41:16.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:16.923+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.929101+0000) 2019-10-13T13:41:16.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:16.923+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.929801+0000) 2019-10-13T13:41:16.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:16.926+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.930622+0000) 2019-10-13T13:41:16.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:16.923+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.931123+0000) 2019-10-13T13:41:16.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:16.927+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.931508+0000) 2019-10-13T13:41:16.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:16.927+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.932211+0000) 2019-10-13T13:41:16.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:16.930+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:16.937121+0000) 2019-10-13T13:41:16.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:16.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:18.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:18.831+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.833130+0000) 2019-10-13T13:41:18.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:18.831+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.833989+0000) 2019-10-13T13:41:20.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:20.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:20.066398+0000) 2019-10-13T13:41:21.943 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:21.935+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.940574+0000) 2019-10-13T13:41:21.943 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:21.935+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.941208+0000) 2019-10-13T13:41:21.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:21.935+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.941529+0000) 2019-10-13T13:41:21.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:21.935+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.942158+0000) 2019-10-13T13:41:21.947 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:21.935+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.942944+0000) 2019-10-13T13:41:21.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:21.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.943661+0000) 2019-10-13T13:41:21.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:21.939+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.944163+0000) 2019-10-13T13:41:21.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:21.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:21.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:21.946+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.950439+0000) 2019-10-13T13:41:22.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:22.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:22.574308+0000) 2019-10-13T13:41:23.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:23.498+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.502567+0000) 2019-10-13T13:41:25.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:25.595+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:25.599144+0000) 2019-10-13T13:41:25.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:25.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:25.885630+0000) 2019-10-13T13:41:26.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:26.559+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.566157+0000) 2019-10-13T13:41:26.955 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:26.947+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.953268+0000) 2019-10-13T13:41:26.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:26.947+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.953698+0000) 2019-10-13T13:41:26.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:26.950+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.954311+0000) 2019-10-13T13:41:26.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:26.947+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.954708+0000) 2019-10-13T13:41:26.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:26.951+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.955710+0000) 2019-10-13T13:41:26.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:26.951+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.956393+0000) 2019-10-13T13:41:26.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:26.951+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.956771+0000) 2019-10-13T13:41:26.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:26.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:26.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:26.958+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:26.963807+0000) 2019-10-13T13:41:28.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:28.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.833326+0000) 2019-10-13T13:41:28.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:28.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.834182+0000) 2019-10-13T13:41:30.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:30.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:30.066593+0000) 2019-10-13T13:41:31.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:31.959+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.965652+0000) 2019-10-13T13:41:31.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:31.959+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.966881+0000) 2019-10-13T13:41:31.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:31.962+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.967846+0000) 2019-10-13T13:41:31.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:31.963+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.968403+0000) 2019-10-13T13:41:31.971 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:31.963+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.969115+0000) 2019-10-13T13:41:31.971 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:31.963+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.969227+0000) 2019-10-13T13:41:31.972 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:31.963+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.969594+0000) 2019-10-13T13:41:31.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:31.970+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.974695+0000) 2019-10-13T13:41:31.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:31.967+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 177997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:32.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:32.567+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:32.574509+0000) 2019-10-13T13:41:33.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:33.510+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.511730+0000) 2019-10-13T13:41:35.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:35.595+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:35.599376+0000) 2019-10-13T13:41:35.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:35.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:35.885861+0000) 2019-10-13T13:41:36.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:36.563+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.566370+0000) 2019-10-13T13:41:36.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:36.971+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.977994+0000) 2019-10-13T13:41:36.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:36.971+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.978424+0000) 2019-10-13T13:41:36.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:36.975+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.979912+0000) 2019-10-13T13:41:36.982 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:36.975+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.980191+0000) 2019-10-13T13:41:36.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:36.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.981495+0000) 2019-10-13T13:41:36.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:36.975+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.981576+0000) 2019-10-13T13:41:36.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:36.975+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.982472+0000) 2019-10-13T13:41:36.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:36.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:36.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:36.982+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:36.989617+0000) 2019-10-13T13:41:38.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:38.831+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.833527+0000) 2019-10-13T13:41:38.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:38.831+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.834371+0000) 2019-10-13T13:41:40.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:40.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:40.066792+0000) 2019-10-13T13:41:41.993 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:41.987+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.990399+0000) 2019-10-13T13:41:41.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:41.987+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.990795+0000) 2019-10-13T13:41:41.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:41.986+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.991357+0000) 2019-10-13T13:41:41.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:41.991+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.991638+0000) 2019-10-13T13:41:41.995 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:41.991+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.992174+0000) 2019-10-13T13:41:41.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:41.991+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.992695+0000) 2019-10-13T13:41:41.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:41.991+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.993343+0000) 2019-10-13T13:41:42.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:41.994+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.998831+0000) 2019-10-13T13:41:42.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:41.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:42.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:42.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:42.574703+0000) 2019-10-13T13:41:43.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:43.514+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.518181+0000) 2019-10-13T13:41:44.883 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:41:44.883 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:44.887 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:41:44.887 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:45.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:45.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:45.599601+0000) 2019-10-13T13:41:45.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:45.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:45.886071+0000) 2019-10-13T13:41:46.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:46.563+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:46.566570+0000) 2019-10-13T13:41:47.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:46.998+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.003505+0000) 2019-10-13T13:41:47.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:47.003+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.003721+0000) 2019-10-13T13:41:47.006 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:47.003+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.004238+0000) 2019-10-13T13:41:47.007 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:47.003+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.005253+0000) 2019-10-13T13:41:47.008 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:47.003+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.005591+0000) 2019-10-13T13:41:47.008 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:47.003+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.005940+0000) 2019-10-13T13:41:47.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:47.003+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.007027+0000) 2019-10-13T13:41:47.014 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:47.011+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:47.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:47.006+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.013779+0000) 2019-10-13T13:41:48.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:48.831+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.833730+0000) 2019-10-13T13:41:48.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:48.831+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.834564+0000) 2019-10-13T13:41:50.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:50.062+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:50.066981+0000) 2019-10-13T13:41:52.018 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:52.011+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.015144+0000) 2019-10-13T13:41:52.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:52.015+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.015590+0000) 2019-10-13T13:41:52.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:52.015+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.016237+0000) 2019-10-13T13:41:52.019 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:52.015+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.016805+0000) 2019-10-13T13:41:52.020 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:52.015+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.017176+0000) 2019-10-13T13:41:52.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:52.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.018589+0000) 2019-10-13T13:41:52.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:52.015+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.018970+0000) 2019-10-13T13:41:52.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:52.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:52.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:52.018+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.024507+0000) 2019-10-13T13:41:52.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:52.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:52.574884+0000) 2019-10-13T13:41:53.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:53.522+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.524499+0000) 2019-10-13T13:41:55.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:55.598+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:55.599804+0000) 2019-10-13T13:41:55.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:55.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:55.886302+0000) 2019-10-13T13:41:56.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:56.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:56.566783+0000) 2019-10-13T13:41:57.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:41:57.026+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.027577+0000) 2019-10-13T13:41:57.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:41:57.026+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.028502+0000) 2019-10-13T13:41:57.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:57.026+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.028916+0000) 2019-10-13T13:41:57.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:41:57.026+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.029316+0000) 2019-10-13T13:41:57.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:41:57.022+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.030024+0000) 2019-10-13T13:41:57.032 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:57.026+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.030136+0000) 2019-10-13T13:41:57.033 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:41:57.030+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.031578+0000) 2019-10-13T13:41:57.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:41:57.030+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.036886+0000) 2019-10-13T13:41:57.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:41:57.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:41:58.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:41:58.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.833928+0000) 2019-10-13T13:41:58.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:41:58.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.834758+0000) 2019-10-13T13:42:00.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:00.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:00.067205+0000) 2019-10-13T13:42:02.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:02.038+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.041367+0000) 2019-10-13T13:42:02.044 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:02.038+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.041651+0000) 2019-10-13T13:42:02.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:02.042+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.042546+0000) 2019-10-13T13:42:02.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:02.038+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.042692+0000) 2019-10-13T13:42:02.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:02.038+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.043080+0000) 2019-10-13T13:42:02.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:02.042+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.043513+0000) 2019-10-13T13:42:02.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:02.042+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.044146+0000) 2019-10-13T13:42:02.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:02.046+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.048836+0000) 2019-10-13T13:42:02.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:02.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178201 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:02.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:02.570+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:02.575079+0000) 2019-10-13T13:42:03.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:03.530+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.531207+0000) 2019-10-13T13:42:05.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:05.598+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:05.600008+0000) 2019-10-13T13:42:05.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:05.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:05.886493+0000) 2019-10-13T13:42:06.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:06.562+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:06.566981+0000) 2019-10-13T13:42:07.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:07.050+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.054012+0000) 2019-10-13T13:42:07.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:07.050+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.054476+0000) 2019-10-13T13:42:07.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:07.050+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.054881+0000) 2019-10-13T13:42:07.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:07.054+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.055836+0000) 2019-10-13T13:42:07.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:07.054+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.056330+0000) 2019-10-13T13:42:07.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:07.053+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.057115+0000) 2019-10-13T13:42:07.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:07.054+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.057641+0000) 2019-10-13T13:42:07.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:07.057+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.061468+0000) 2019-10-13T13:42:07.065 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:07.058+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:08.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:08.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.834128+0000) 2019-10-13T13:42:08.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:08.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.834988+0000) 2019-10-13T13:42:10.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:10.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:10.067406+0000) 2019-10-13T13:42:12.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:12.062+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.066691+0000) 2019-10-13T13:42:12.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:12.062+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.067043+0000) 2019-10-13T13:42:12.071 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:12.066+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.068039+0000) 2019-10-13T13:42:12.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:12.066+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.068108+0000) 2019-10-13T13:42:12.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:12.066+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.068731+0000) 2019-10-13T13:42:12.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:12.065+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.069718+0000) 2019-10-13T13:42:12.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:12.066+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.069882+0000) 2019-10-13T13:42:12.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:12.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:12.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:12.073+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.077030+0000) 2019-10-13T13:42:12.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:12.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:12.575264+0000) 2019-10-13T13:42:13.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:13.533+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.537804+0000) 2019-10-13T13:42:14.984 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:42:14.984 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:14.988 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:42:14.988 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:15.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:15.598+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:15.600271+0000) 2019-10-13T13:42:15.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:15.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:15.886696+0000) 2019-10-13T13:42:16.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:16.566+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:16.567177+0000) 2019-10-13T13:42:17.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:17.077+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.081867+0000) 2019-10-13T13:42:17.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:17.074+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.078321+0000) 2019-10-13T13:42:17.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:17.074+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.079218+0000) 2019-10-13T13:42:17.089 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:17.078+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.079744+0000) 2019-10-13T13:42:17.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:17.078+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.080376+0000) 2019-10-13T13:42:17.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:17.078+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.081000+0000) 2019-10-13T13:42:17.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:17.078+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.081933+0000) 2019-10-13T13:42:17.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:17.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:17.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:17.085+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.089251+0000) 2019-10-13T13:42:18.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:18.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.834336+0000) 2019-10-13T13:42:18.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:18.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.835190+0000) 2019-10-13T13:42:20.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:20.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:20.067601+0000) 2019-10-13T13:42:22.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:22.086+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.090693+0000) 2019-10-13T13:42:22.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:22.086+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.091185+0000) 2019-10-13T13:42:22.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:22.090+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.091612+0000) 2019-10-13T13:42:22.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:22.090+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.092090+0000) 2019-10-13T13:42:22.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:22.090+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.092913+0000) 2019-10-13T13:42:22.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:22.089+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.093695+0000) 2019-10-13T13:42:22.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:22.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.094025+0000) 2019-10-13T13:42:22.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:22.097+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.100220+0000) 2019-10-13T13:42:22.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:22.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:22.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:22.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:22.575466+0000) 2019-10-13T13:42:23.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:23.541+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.544246+0000) 2019-10-13T13:42:25.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:25.598+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:25.600501+0000) 2019-10-13T13:42:25.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:25.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:25.886927+0000) 2019-10-13T13:42:26.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:26.566+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:26.567373+0000) 2019-10-13T13:42:27.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:27.098+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.103239+0000) 2019-10-13T13:42:27.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:27.102+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.104463+0000) 2019-10-13T13:42:27.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:27.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.104823+0000) 2019-10-13T13:42:27.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:27.101+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.105821+0000) 2019-10-13T13:42:27.108 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:27.102+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.105842+0000) 2019-10-13T13:42:27.109 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:27.102+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.106159+0000) 2019-10-13T13:42:27.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:27.102+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.106854+0000) 2019-10-13T13:42:27.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:27.109+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.111968+0000) 2019-10-13T13:42:27.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:27.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:28.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:28.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.834558+0000) 2019-10-13T13:42:28.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:28.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.835415+0000) 2019-10-13T13:42:30.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:30.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:30.067804+0000) 2019-10-13T13:42:32.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:32.114+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.115596+0000) 2019-10-13T13:42:32.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:32.114+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.116590+0000) 2019-10-13T13:42:32.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:32.114+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.117124+0000) 2019-10-13T13:42:32.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:32.114+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.117786+0000) 2019-10-13T13:42:32.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:32.114+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.116673+0000) 2019-10-13T13:42:32.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:32.117+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.118852+0000) 2019-10-13T13:42:32.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:32.114+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.118950+0000) 2019-10-13T13:42:32.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:32.121+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.124428+0000) 2019-10-13T13:42:32.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:32.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:32.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:32.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:32.575677+0000) 2019-10-13T13:42:33.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:33.549+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.550715+0000) 2019-10-13T13:42:35.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:35.598+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:35.600720+0000) 2019-10-13T13:42:35.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:35.882+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:35.887127+0000) 2019-10-13T13:42:36.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:36.566+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:36.567581+0000) 2019-10-13T13:42:37.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:37.126+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.128296+0000) 2019-10-13T13:42:37.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:37.126+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.128830+0000) 2019-10-13T13:42:37.132 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:37.126+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.129315+0000) 2019-10-13T13:42:37.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:37.126+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.129737+0000) 2019-10-13T13:42:37.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:37.129+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.130437+0000) 2019-10-13T13:42:37.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:37.126+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.130734+0000) 2019-10-13T13:42:37.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:37.130+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.131400+0000) 2019-10-13T13:42:37.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:37.134+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:37.140 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:37.137+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.138781+0000) 2019-10-13T13:42:38.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:38.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.834754+0000) 2019-10-13T13:42:38.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:38.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.835626+0000) 2019-10-13T13:42:40.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:40.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:40.068020+0000) 2019-10-13T13:42:42.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:42.137+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.140855+0000) 2019-10-13T13:42:42.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:42.137+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.141232+0000) 2019-10-13T13:42:42.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:42.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.141944+0000) 2019-10-13T13:42:42.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:42.137+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.142581+0000) 2019-10-13T13:42:42.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:42.137+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.142678+0000) 2019-10-13T13:42:42.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:42.141+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.143499+0000) 2019-10-13T13:42:42.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:42.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.143880+0000) 2019-10-13T13:42:42.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:42.149+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.152093+0000) 2019-10-13T13:42:42.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:42.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:42.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:42.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:42.575873+0000) 2019-10-13T13:42:43.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:43.553+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:43.556836+0000) 2019-10-13T13:42:45.122 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:42:45.122 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:45.125 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:42:45.126 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:45.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:45.597+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:45.600951+0000) 2019-10-13T13:42:45.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:45.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:45.887371+0000) 2019-10-13T13:42:46.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:46.565+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:46.567777+0000) 2019-10-13T13:42:47.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:47.149+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.153558+0000) 2019-10-13T13:42:47.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:47.149+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.154337+0000) 2019-10-13T13:42:47.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:47.149+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.154673+0000) 2019-10-13T13:42:47.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:47.153+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.155326+0000) 2019-10-13T13:42:47.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:47.153+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.155633+0000) 2019-10-13T13:42:47.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:47.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.156188+0000) 2019-10-13T13:42:47.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:47.153+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.156536+0000) 2019-10-13T13:42:47.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:47.161+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.162686+0000) 2019-10-13T13:42:47.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:47.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:48.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:48.829+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:48.834959+0000) 2019-10-13T13:42:48.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:48.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:48.835802+0000) 2019-10-13T13:42:50.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:50.065+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:50.068206+0000) 2019-10-13T13:42:52.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:52.161+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.165814+0000) 2019-10-13T13:42:52.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:52.161+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.167065+0000) 2019-10-13T13:42:52.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:52.165+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.167705+0000) 2019-10-13T13:42:52.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:52.165+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.168474+0000) 2019-10-13T13:42:52.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:52.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.168017+0000) 2019-10-13T13:42:52.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:52.165+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.168995+0000) 2019-10-13T13:42:52.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:52.165+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.169584+0000) 2019-10-13T13:42:52.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:52.169+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.173203+0000) 2019-10-13T13:42:52.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:52.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:52.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:52.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:52.576053+0000) 2019-10-13T13:42:53.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:53.561+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:53.563202+0000) 2019-10-13T13:42:55.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:55.597+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:55.601151+0000) 2019-10-13T13:42:55.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:55.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:55.887603+0000) 2019-10-13T13:42:56.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:56.565+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:56.567972+0000) 2019-10-13T13:42:57.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:42:57.173+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.178304+0000) 2019-10-13T13:42:57.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:42:57.173+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.178725+0000) 2019-10-13T13:42:57.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:42:57.177+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.179399+0000) 2019-10-13T13:42:57.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:42:57.177+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.179909+0000) 2019-10-13T13:42:57.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:42:57.177+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.180358+0000) 2019-10-13T13:42:57.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:57.177+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.180835+0000) 2019-10-13T13:42:57.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:57.177+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.181408+0000) 2019-10-13T13:42:57.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:42:57.185+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.187253+0000) 2019-10-13T13:42:57.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:42:57.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:42:58.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:42:58.829+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:58.835155+0000) 2019-10-13T13:42:58.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:42:58.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:58.835947+0000) 2019-10-13T13:43:00.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:00.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:00.068380+0000) 2019-10-13T13:43:02.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:02.185+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.190784+0000) 2019-10-13T13:43:02.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:02.185+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.191129+0000) 2019-10-13T13:43:02.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:02.188+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.192090+0000) 2019-10-13T13:43:02.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:02.189+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.192559+0000) 2019-10-13T13:43:02.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:02.189+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.192185+0000) 2019-10-13T13:43:02.195 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:02.189+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.192962+0000) 2019-10-13T13:43:02.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:02.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.193373+0000) 2019-10-13T13:43:02.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:02.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:02.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:02.200+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.203663+0000) 2019-10-13T13:43:02.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:02.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:02.576228+0000) 2019-10-13T13:43:03.572 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:03.568+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:03.569652+0000) 2019-10-13T13:43:05.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:05.597+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:05.601354+0000) 2019-10-13T13:43:05.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:05.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:05.887806+0000) 2019-10-13T13:43:06.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:06.565+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:06.568172+0000) 2019-10-13T13:43:07.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:07.197+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.202324+0000) 2019-10-13T13:43:07.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:07.197+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.202961+0000) 2019-10-13T13:43:07.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:07.200+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.203970+0000) 2019-10-13T13:43:07.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:07.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.204240+0000) 2019-10-13T13:43:07.207 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:07.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.204870+0000) 2019-10-13T13:43:07.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:07.201+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.205054+0000) 2019-10-13T13:43:07.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:07.201+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.205793+0000) 2019-10-13T13:43:07.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:07.208+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.211538+0000) 2019-10-13T13:43:07.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:07.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:08.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:08.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:08.835347+0000) 2019-10-13T13:43:08.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:08.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:08.836108+0000) 2019-10-13T13:43:10.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:10.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:10.068577+0000) 2019-10-13T13:43:12.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:12.212+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.214683+0000) 2019-10-13T13:43:12.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:12.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.215755+0000) 2019-10-13T13:43:12.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:12.213+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.216228+0000) 2019-10-13T13:43:12.219 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:12.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.216764+0000) 2019-10-13T13:43:12.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:12.213+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.216884+0000) 2019-10-13T13:43:12.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:12.213+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.217744+0000) 2019-10-13T13:43:12.220 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:12.213+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.218364+0000) 2019-10-13T13:43:12.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:12.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178676 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:12.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:12.220+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.224058+0000) 2019-10-13T13:43:12.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:12.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:12.576411+0000) 2019-10-13T13:43:13.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:13.572+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:13.576183+0000) 2019-10-13T13:43:15.177 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:43:15.178 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:15.181 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:43:15.181 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:15.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:15.601+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:15.601571+0000) 2019-10-13T13:43:15.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:15.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:15.888041+0000) 2019-10-13T13:43:16.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:16.565+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:16.568362+0000) 2019-10-13T13:43:17.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:17.224+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.226952+0000) 2019-10-13T13:43:17.230 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:17.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.228033+0000) 2019-10-13T13:43:17.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:17.225+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.227361+0000) 2019-10-13T13:43:17.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:17.225+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.229135+0000) 2019-10-13T13:43:17.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:17.225+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.229393+0000) 2019-10-13T13:43:17.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:17.225+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.229995+0000) 2019-10-13T13:43:17.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:17.225+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.230494+0000) 2019-10-13T13:43:17.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:17.232+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.235617+0000) 2019-10-13T13:43:17.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:17.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178710 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:18.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:18.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:18.835569+0000) 2019-10-13T13:43:18.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:18.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:18.836256+0000) 2019-10-13T13:43:20.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:20.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:20.068790+0000) 2019-10-13T13:43:22.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:22.236+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.240126+0000) 2019-10-13T13:43:22.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:22.236+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.240012+0000) 2019-10-13T13:43:22.244 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:22.236+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.241530+0000) 2019-10-13T13:43:22.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:22.236+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.240972+0000) 2019-10-13T13:43:22.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:22.236+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.242027+0000) 2019-10-13T13:43:22.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:22.236+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.242045+0000) 2019-10-13T13:43:22.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:22.236+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.242342+0000) 2019-10-13T13:43:22.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:22.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.247352+0000) 2019-10-13T13:43:22.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:22.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:22.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:22.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:22.576591+0000) 2019-10-13T13:43:23.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:23.580+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:23.582843+0000) 2019-10-13T13:43:25.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:25.600+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:25.601773+0000) 2019-10-13T13:43:25.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:25.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:25.888272+0000) 2019-10-13T13:43:26.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:26.564+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:26.568556+0000) 2019-10-13T13:43:27.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:27.248+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.252102+0000) 2019-10-13T13:43:27.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:27.248+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.252880+0000) 2019-10-13T13:43:27.255 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:27.248+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.253227+0000) 2019-10-13T13:43:27.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:27.248+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.254171+0000) 2019-10-13T13:43:27.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:27.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.254614+0000) 2019-10-13T13:43:27.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:27.248+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.254767+0000) 2019-10-13T13:43:27.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:27.248+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.255043+0000) 2019-10-13T13:43:27.263 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:27.256+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178778 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:27.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:27.260+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.262482+0000) 2019-10-13T13:43:28.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:28.832+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:28.835782+0000) 2019-10-13T13:43:28.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:28.832+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:28.836441+0000) 2019-10-13T13:43:30.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:30.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:30.068989+0000) 2019-10-13T13:43:32.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:32.260+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.264249+0000) 2019-10-13T13:43:32.267 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:32.260+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.264597+0000) 2019-10-13T13:43:32.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:32.260+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.265669+0000) 2019-10-13T13:43:32.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:32.260+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.265243+0000) 2019-10-13T13:43:32.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:32.260+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.266285+0000) 2019-10-13T13:43:32.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:32.260+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.266484+0000) 2019-10-13T13:43:32.270 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:32.260+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.267150+0000) 2019-10-13T13:43:32.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:32.272+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.274477+0000) 2019-10-13T13:43:32.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:32.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:32.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:32.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:32.576772+0000) 2019-10-13T13:43:33.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:33.584+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:33.589497+0000) 2019-10-13T13:43:35.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:35.600+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:35.601978+0000) 2019-10-13T13:43:35.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:35.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:35.888474+0000) 2019-10-13T13:43:36.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:36.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:36.568786+0000) 2019-10-13T13:43:37.279 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:37.272+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.277199+0000) 2019-10-13T13:43:37.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:37.272+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.277395+0000) 2019-10-13T13:43:37.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:37.272+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.278021+0000) 2019-10-13T13:43:37.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:37.272+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.278321+0000) 2019-10-13T13:43:37.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:37.272+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.278782+0000) 2019-10-13T13:43:37.281 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:37.272+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.278995+0000) 2019-10-13T13:43:37.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:37.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.280373+0000) 2019-10-13T13:43:37.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:37.284+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.286207+0000) 2019-10-13T13:43:37.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:37.280+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:38.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:38.832+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:38.835982+0000) 2019-10-13T13:43:38.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:38.832+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:38.836629+0000) 2019-10-13T13:43:40.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:40.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:40.069183+0000) 2019-10-13T13:43:42.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:42.284+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.290554+0000) 2019-10-13T13:43:42.293 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:42.284+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.291037+0000) 2019-10-13T13:43:42.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:42.288+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.291561+0000) 2019-10-13T13:43:42.294 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:42.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.292669+0000) 2019-10-13T13:43:42.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:42.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.292987+0000) 2019-10-13T13:43:42.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:42.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.293496+0000) 2019-10-13T13:43:42.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:42.288+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.294269+0000) 2019-10-13T13:43:42.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:42.296+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.299240+0000) 2019-10-13T13:43:42.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:42.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:42.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:42.576+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:42.576951+0000) 2019-10-13T13:43:43.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:43.592+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:43.595932+0000) 2019-10-13T13:43:45.292 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:43:45.292 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:45.295 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:43:45.296 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:45.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:45.600+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:45.602179+0000) 2019-10-13T13:43:45.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:45.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:45.888676+0000) 2019-10-13T13:43:46.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:46.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:46.568982+0000) 2019-10-13T13:43:47.305 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:47.296+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.302633+0000) 2019-10-13T13:43:47.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:47.296+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.303089+0000) 2019-10-13T13:43:47.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:47.300+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.303574+0000) 2019-10-13T13:43:47.306 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:47.300+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.304054+0000) 2019-10-13T13:43:47.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:47.300+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.304119+0000) 2019-10-13T13:43:47.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:47.300+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.305233+0000) 2019-10-13T13:43:47.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:47.300+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.305723+0000) 2019-10-13T13:43:47.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:47.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178914 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:47.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:47.308+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.313732+0000) 2019-10-13T13:43:48.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:48.832+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:48.836205+0000) 2019-10-13T13:43:48.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:48.832+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:48.836845+0000) 2019-10-13T13:43:50.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:50.064+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:50.069377+0000) 2019-10-13T13:43:52.318 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:52.312+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.315210+0000) 2019-10-13T13:43:52.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:52.312+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.315692+0000) 2019-10-13T13:43:52.319 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:52.312+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.317357+0000) 2019-10-13T13:43:52.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:52.312+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.318145+0000) 2019-10-13T13:43:52.321 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:52.312+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.318379+0000) 2019-10-13T13:43:52.321 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:52.312+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.318749+0000) 2019-10-13T13:43:52.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:52.312+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.319150+0000) 2019-10-13T13:43:52.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:52.320+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.324178+0000) 2019-10-13T13:43:52.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:52.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:52.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:52.576+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:52.577132+0000) 2019-10-13T13:43:53.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:53.599+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:53.602369+0000) 2019-10-13T13:43:55.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:55.600+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:55.602380+0000) 2019-10-13T13:43:55.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:55.884+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:55.888870+0000) 2019-10-13T13:43:56.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:56.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:56.569178+0000) 2019-10-13T13:43:57.331 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:43:57.324+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.329046+0000) 2019-10-13T13:43:57.332 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:57.324+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.329566+0000) 2019-10-13T13:43:57.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:43:57.323+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.330089+0000) 2019-10-13T13:43:57.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:43:57.324+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.330320+0000) 2019-10-13T13:43:57.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:43:57.324+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.330557+0000) 2019-10-13T13:43:57.334 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:43:57.324+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.330832+0000) 2019-10-13T13:43:57.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:57.328+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.331824+0000) 2019-10-13T13:43:57.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:43:57.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 178982 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:43:57.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:43:57.335+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.339393+0000) 2019-10-13T13:43:58.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:43:58.832+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:58.836390+0000) 2019-10-13T13:43:58.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:43:58.832+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:58.837050+0000) 2019-10-13T13:44:00.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:00.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:00.069576+0000) 2019-10-13T13:44:02.344 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:02.336+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.341539+0000) 2019-10-13T13:44:02.344 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:02.336+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.341893+0000) 2019-10-13T13:44:02.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:02.336+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.342184+0000) 2019-10-13T13:44:02.345 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:02.339+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.342928+0000) 2019-10-13T13:44:02.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:02.336+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.342803+0000) 2019-10-13T13:44:02.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:02.340+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.343285+0000) 2019-10-13T13:44:02.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:02.340+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.344085+0000) 2019-10-13T13:44:02.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:02.347+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.350355+0000) 2019-10-13T13:44:02.353 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:02.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179016 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:02.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:02.572+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:02.577311+0000) 2019-10-13T13:44:03.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:03.603+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:03.608874+0000) 2019-10-13T13:44:05.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:05.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:05.602588+0000) 2019-10-13T13:44:05.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:05.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:05.889064+0000) 2019-10-13T13:44:06.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:06.567+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:06.569375+0000) 2019-10-13T13:44:07.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:07.347+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.353227+0000) 2019-10-13T13:44:07.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:07.347+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.354284+0000) 2019-10-13T13:44:07.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:07.347+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.354656+0000) 2019-10-13T13:44:07.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:07.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.355480+0000) 2019-10-13T13:44:07.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:07.351+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.355795+0000) 2019-10-13T13:44:07.358 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:07.351+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.356517+0000) 2019-10-13T13:44:07.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:07.351+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.356896+0000) 2019-10-13T13:44:07.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:07.355+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.361696+0000) 2019-10-13T13:44:07.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:07.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:08.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:08.835+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:08.836595+0000) 2019-10-13T13:44:08.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:08.835+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:08.837204+0000) 2019-10-13T13:44:10.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:10.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:10.069799+0000) 2019-10-13T13:44:12.369 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:12.363+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.366828+0000) 2019-10-13T13:44:12.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:12.363+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.367570+0000) 2019-10-13T13:44:12.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:12.363+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.367625+0000) 2019-10-13T13:44:12.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:12.363+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.367943+0000) 2019-10-13T13:44:12.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:12.363+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.368302+0000) 2019-10-13T13:44:12.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:12.363+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.369074+0000) 2019-10-13T13:44:12.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:12.363+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.369658+0000) 2019-10-13T13:44:12.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:12.371+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.376567+0000) 2019-10-13T13:44:12.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:12.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:12.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:12.571+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:12.577513+0000) 2019-10-13T13:44:13.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:13.611+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:13.615374+0000) 2019-10-13T13:44:15.411 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:44:15.411 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:15.415 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:44:15.415 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:15.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:15.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:15.602803+0000) 2019-10-13T13:44:15.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:15.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:15.889255+0000) 2019-10-13T13:44:16.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:16.563+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:16.569569+0000) 2019-10-13T13:44:17.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:17.375+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.378961+0000) 2019-10-13T13:44:17.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:17.375+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.379738+0000) 2019-10-13T13:44:17.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:17.375+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.380369+0000) 2019-10-13T13:44:17.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:17.375+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.380693+0000) 2019-10-13T13:44:17.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:17.375+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.381391+0000) 2019-10-13T13:44:17.384 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:17.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.381558+0000) 2019-10-13T13:44:17.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:17.375+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.382274+0000) 2019-10-13T13:44:17.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:17.383+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.388436+0000) 2019-10-13T13:44:17.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:17.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:18.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:18.835+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:18.836828+0000) 2019-10-13T13:44:18.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:18.835+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:18.837353+0000) 2019-10-13T13:44:20.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:20.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:20.069995+0000) 2019-10-13T13:44:22.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:22.387+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.393547+0000) 2019-10-13T13:44:22.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:22.387+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.394025+0000) 2019-10-13T13:44:22.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:22.387+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.394757+0000) 2019-10-13T13:44:22.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:22.387+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.395212+0000) 2019-10-13T13:44:22.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:22.391+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.395739+0000) 2019-10-13T13:44:22.398 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:22.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.395818+0000) 2019-10-13T13:44:22.399 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:22.391+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.396822+0000) 2019-10-13T13:44:22.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:22.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:22.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:22.399+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.403151+0000) 2019-10-13T13:44:22.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:22.575+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:22.577727+0000) 2019-10-13T13:44:23.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:23.619+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:23.621838+0000) 2019-10-13T13:44:25.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:25.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:25.603001+0000) 2019-10-13T13:44:25.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:25.883+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:25.889461+0000) 2019-10-13T13:44:26.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:26.567+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:26.569764+0000) 2019-10-13T13:44:27.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:27.399+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.405549+0000) 2019-10-13T13:44:27.408 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:27.399+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.405954+0000) 2019-10-13T13:44:27.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:27.399+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.406927+0000) 2019-10-13T13:44:27.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:27.403+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.407496+0000) 2019-10-13T13:44:27.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:27.403+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.407937+0000) 2019-10-13T13:44:27.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:27.403+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.408365+0000) 2019-10-13T13:44:27.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:27.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.409048+0000) 2019-10-13T13:44:27.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:27.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:27.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:27.411+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.415330+0000) 2019-10-13T13:44:28.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:28.835+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:28.837070+0000) 2019-10-13T13:44:28.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:28.835+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:28.837515+0000) 2019-10-13T13:44:30.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:30.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:30.070223+0000) 2019-10-13T13:44:32.421 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:32.411+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.418654+0000) 2019-10-13T13:44:32.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:32.411+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.418247+0000) 2019-10-13T13:44:32.422 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:32.415+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.419337+0000) 2019-10-13T13:44:32.422 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:32.415+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.419894+0000) 2019-10-13T13:44:32.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:32.415+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.420398+0000) 2019-10-13T13:44:32.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:32.415+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.420723+0000) 2019-10-13T13:44:32.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:32.415+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.421093+0000) 2019-10-13T13:44:32.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:32.419+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.423459+0000) 2019-10-13T13:44:32.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:32.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:32.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:32.575+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:32.577910+0000) 2019-10-13T13:44:33.630 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:33.623+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:33.628248+0000) 2019-10-13T13:44:35.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:35.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:35.603227+0000) 2019-10-13T13:44:35.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:35.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:35.889697+0000) 2019-10-13T13:44:36.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:36.567+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:36.569950+0000) 2019-10-13T13:44:37.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:37.427+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.430221+0000) 2019-10-13T13:44:37.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:37.427+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.430307+0000) 2019-10-13T13:44:37.433 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:37.427+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.431036+0000) 2019-10-13T13:44:37.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:37.427+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.430899+0000) 2019-10-13T13:44:37.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:37.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.432028+0000) 2019-10-13T13:44:37.435 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:37.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.432548+0000) 2019-10-13T13:44:37.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:37.431+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.432908+0000) 2019-10-13T13:44:37.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:37.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:37.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:37.435+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.440646+0000) 2019-10-13T13:44:38.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:38.835+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:38.837273+0000) 2019-10-13T13:44:38.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:38.835+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:38.837678+0000) 2019-10-13T13:44:40.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:40.067+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:40.070416+0000) 2019-10-13T13:44:42.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:42.443+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.443847+0000) 2019-10-13T13:44:42.447 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:42.443+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.444037+0000) 2019-10-13T13:44:42.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:42.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.444391+0000) 2019-10-13T13:44:42.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:42.443+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.444840+0000) 2019-10-13T13:44:42.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:42.443+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.446493+0000) 2019-10-13T13:44:42.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:42.443+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.446752+0000) 2019-10-13T13:44:42.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:42.443+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.446923+0000) 2019-10-13T13:44:42.453 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:42.447+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.451617+0000) 2019-10-13T13:44:42.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:42.451+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:42.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:42.575+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:42.578138+0000) 2019-10-13T13:44:43.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:43.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:43.634660+0000) 2019-10-13T13:44:45.545 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:44:45.545 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:45.549 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:44:45.549 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:45.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:45.603+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:45.603459+0000) 2019-10-13T13:44:45.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:45.887+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:45.889899+0000) 2019-10-13T13:44:46.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:46.567+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:46.570101+0000) 2019-10-13T13:44:47.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:47.455+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.456986+0000) 2019-10-13T13:44:47.460 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:47.455+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.457465+0000) 2019-10-13T13:44:47.460 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:47.455+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.457742+0000) 2019-10-13T13:44:47.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:47.455+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.458136+0000) 2019-10-13T13:44:47.461 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:47.455+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.458711+0000) 2019-10-13T13:44:47.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:47.455+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.459303+0000) 2019-10-13T13:44:47.462 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:47.459+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.460305+0000) 2019-10-13T13:44:47.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:47.463+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:47.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:47.463+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.466842+0000) 2019-10-13T13:44:48.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:48.834+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:48.837476+0000) 2019-10-13T13:44:48.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:48.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:48.837835+0000) 2019-10-13T13:44:50.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:50.066+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:50.070622+0000) 2019-10-13T13:44:52.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:52.466+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.468727+0000) 2019-10-13T13:44:52.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:52.466+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.469277+0000) 2019-10-13T13:44:52.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:52.466+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.470182+0000) 2019-10-13T13:44:52.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:52.466+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.470569+0000) 2019-10-13T13:44:52.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:52.466+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.471087+0000) 2019-10-13T13:44:52.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:52.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.471973+0000) 2019-10-13T13:44:52.474 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:52.470+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.471960+0000) 2019-10-13T13:44:52.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:52.474+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.478560+0000) 2019-10-13T13:44:52.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:52.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:52.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:52.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:52.578323+0000) 2019-10-13T13:44:53.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:53.634+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:53.641167+0000) 2019-10-13T13:44:55.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:55.602+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:55.603699+0000) 2019-10-13T13:44:55.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:55.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:55.890124+0000) 2019-10-13T13:44:56.572 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:56.566+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:56.570294+0000) 2019-10-13T13:44:57.483 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:57.478+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.481114+0000) 2019-10-13T13:44:57.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:44:57.478+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.481663+0000) 2019-10-13T13:44:57.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:44:57.478+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.482720+0000) 2019-10-13T13:44:57.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:44:57.478+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.483953+0000) 2019-10-13T13:44:57.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:44:57.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.484770+0000) 2019-10-13T13:44:57.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:44:57.482+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.484848+0000) 2019-10-13T13:44:57.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:57.482+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.485256+0000) 2019-10-13T13:44:57.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:44:57.486+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.490833+0000) 2019-10-13T13:44:57.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:44:57.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:44:58.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:44:58.834+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:58.837704+0000) 2019-10-13T13:44:58.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:44:58.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:58.838019+0000) 2019-10-13T13:45:00.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:00.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:00.070828+0000) 2019-10-13T13:45:02.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:02.490+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.494534+0000) 2019-10-13T13:45:02.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:02.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.494495+0000) 2019-10-13T13:45:02.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:02.490+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.495376+0000) 2019-10-13T13:45:02.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:02.494+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.495705+0000) 2019-10-13T13:45:02.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:02.494+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.495960+0000) 2019-10-13T13:45:02.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:02.494+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.497020+0000) 2019-10-13T13:45:02.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:02.494+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.497403+0000) 2019-10-13T13:45:02.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:02.498+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.504535+0000) 2019-10-13T13:45:02.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:02.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:02.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:02.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:02.578501+0000) 2019-10-13T13:45:03.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:03.642+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:03.647523+0000) 2019-10-13T13:45:05.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:05.602+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:05.603933+0000) 2019-10-13T13:45:05.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:05.886+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:05.890335+0000) 2019-10-13T13:45:06.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:06.566+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:06.570480+0000) 2019-10-13T13:45:07.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:07.506+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.507809+0000) 2019-10-13T13:45:07.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:07.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.508639+0000) 2019-10-13T13:45:07.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:07.506+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.508989+0000) 2019-10-13T13:45:07.511 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:07.506+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.509044+0000) 2019-10-13T13:45:07.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:07.506+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.509440+0000) 2019-10-13T13:45:07.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:07.506+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.510357+0000) 2019-10-13T13:45:07.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:07.506+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.511271+0000) 2019-10-13T13:45:07.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:07.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:07.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:07.510+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.517507+0000) 2019-10-13T13:45:08.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:08.834+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:08.837934+0000) 2019-10-13T13:45:08.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:08.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:08.838199+0000) 2019-10-13T13:45:10.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:10.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:10.071026+0000) 2019-10-13T13:45:12.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:12.514+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.520880+0000) 2019-10-13T13:45:12.523 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:12.518+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.521011+0000) 2019-10-13T13:45:12.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:12.518+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.521461+0000) 2019-10-13T13:45:12.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:12.518+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.522637+0000) 2019-10-13T13:45:12.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:12.518+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.523037+0000) 2019-10-13T13:45:12.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:12.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.523277+0000) 2019-10-13T13:45:12.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:12.522+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.523612+0000) 2019-10-13T13:45:12.530 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:12.522+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.528699+0000) 2019-10-13T13:45:12.531 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:12.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:12.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:12.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:12.578683+0000) 2019-10-13T13:45:13.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:13.650+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:13.654114+0000) 2019-10-13T13:45:15.606 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:15.602+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:15.604132+0000) 2019-10-13T13:45:15.677 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:45:15.677 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:15.681 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:45:15.681 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:15.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:15.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:15.890545+0000) 2019-10-13T13:45:16.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:16.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:16.570653+0000) 2019-10-13T13:45:17.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:17.526+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.533330+0000) 2019-10-13T13:45:17.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:17.530+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.533715+0000) 2019-10-13T13:45:17.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:17.530+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.534364+0000) 2019-10-13T13:45:17.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:17.530+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.534528+0000) 2019-10-13T13:45:17.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:17.530+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.534851+0000) 2019-10-13T13:45:17.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:17.534+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.535768+0000) 2019-10-13T13:45:17.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:17.534+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.536085+0000) 2019-10-13T13:45:17.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:17.534+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.541725+0000) 2019-10-13T13:45:17.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:17.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:18.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:18.834+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:18.838138+0000) 2019-10-13T13:45:18.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:18.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:18.838384+0000) 2019-10-13T13:45:20.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:20.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:20.071225+0000) 2019-10-13T13:45:22.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:22.542+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.545285+0000) 2019-10-13T13:45:22.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:22.542+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.546562+0000) 2019-10-13T13:45:22.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:22.542+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.547025+0000) 2019-10-13T13:45:22.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:22.542+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.547286+0000) 2019-10-13T13:45:22.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:22.546+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.548407+0000) 2019-10-13T13:45:22.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:22.546+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.548368+0000) 2019-10-13T13:45:22.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:22.546+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.548711+0000) 2019-10-13T13:45:22.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:22.550+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.554425+0000) 2019-10-13T13:45:22.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:22.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:22.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:22.574+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:22.578871+0000) 2019-10-13T13:45:23.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:23.658+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:23.660657+0000) 2019-10-13T13:45:25.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:25.602+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:25.604336+0000) 2019-10-13T13:45:25.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:25.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:25.890755+0000) 2019-10-13T13:45:26.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:26.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:26.570853+0000) 2019-10-13T13:45:27.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:27.554+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.558055+0000) 2019-10-13T13:45:27.561 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:27.554+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.559434+0000) 2019-10-13T13:45:27.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:27.558+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.559558+0000) 2019-10-13T13:45:27.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:27.558+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.559980+0000) 2019-10-13T13:45:27.563 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:27.558+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.560532+0000) 2019-10-13T13:45:27.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:27.558+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.561100+0000) 2019-10-13T13:45:27.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:27.558+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.561564+0000) 2019-10-13T13:45:27.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:27.562+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.567480+0000) 2019-10-13T13:45:27.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:27.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179594 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:28.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:28.834+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:28.838334+0000) 2019-10-13T13:45:28.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:28.834+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:28.838554+0000) 2019-10-13T13:45:30.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:30.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:30.071455+0000) 2019-10-13T13:45:32.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:32.565+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.570965+0000) 2019-10-13T13:45:32.574 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:32.569+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.571932+0000) 2019-10-13T13:45:32.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:32.569+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.572774+0000) 2019-10-13T13:45:32.575 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:32.569+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.572438+0000) 2019-10-13T13:45:32.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:32.569+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.573074+0000) 2019-10-13T13:45:32.576 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:32.566+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.573674+0000) 2019-10-13T13:45:32.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:32.569+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.574080+0000) 2019-10-13T13:45:32.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:32.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.579043+0000) 2019-10-13T13:45:32.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:32.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:32.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:32.574+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:32.580575+0000) 2019-10-13T13:45:33.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:33.666+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:33.667068+0000) 2019-10-13T13:45:35.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:35.601+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:35.604491+0000) 2019-10-13T13:45:35.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:35.885+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:35.890961+0000) 2019-10-13T13:45:36.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:36.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:36.571005+0000) 2019-10-13T13:45:37.586 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:37.577+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.583151+0000) 2019-10-13T13:45:37.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:37.581+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.583888+0000) 2019-10-13T13:45:37.587 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:37.581+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.584354+0000) 2019-10-13T13:45:37.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:37.581+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.584641+0000) 2019-10-13T13:45:37.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:37.581+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.585080+0000) 2019-10-13T13:45:37.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:37.581+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.586041+0000) 2019-10-13T13:45:37.589 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:37.586+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.586730+0000) 2019-10-13T13:45:37.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:37.590+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.592345+0000) 2019-10-13T13:45:37.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:37.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:38.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:38.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:38.838560+0000) 2019-10-13T13:45:38.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:38.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:38.838745+0000) 2019-10-13T13:45:40.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:40.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:40.071682+0000) 2019-10-13T13:45:42.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:42.573+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.579251+0000) 2019-10-13T13:45:42.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:42.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.595607+0000) 2019-10-13T13:45:42.598 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:42.593+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.596349+0000) 2019-10-13T13:45:42.600 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:42.593+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.597892+0000) 2019-10-13T13:45:42.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:42.593+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.598296+0000) 2019-10-13T13:45:42.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:42.593+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.598698+0000) 2019-10-13T13:45:42.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:42.597+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.599345+0000) 2019-10-13T13:45:42.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:42.597+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.599392+0000) 2019-10-13T13:45:42.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:42.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:42.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:42.605+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:42.606536+0000) 2019-10-13T13:45:43.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:43.669+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:43.673611+0000) 2019-10-13T13:45:45.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:45.601+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:45.604727+0000) 2019-10-13T13:45:45.796 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:45:45.796 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:45.799 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:45:45.799 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:45.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:45.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:45.891201+0000) 2019-10-13T13:45:46.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:46.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:46.571176+0000) 2019-10-13T13:45:47.611 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:47.605+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.608914+0000) 2019-10-13T13:45:47.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:47.605+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.609375+0000) 2019-10-13T13:45:47.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:47.605+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.610057+0000) 2019-10-13T13:45:47.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:47.605+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.610183+0000) 2019-10-13T13:45:47.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:47.605+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.610735+0000) 2019-10-13T13:45:47.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:47.609+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.611809+0000) 2019-10-13T13:45:47.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:47.609+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.612180+0000) 2019-10-13T13:45:47.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:47.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:47.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:47.617+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.620002+0000) 2019-10-13T13:45:48.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:48.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:48.838786+0000) 2019-10-13T13:45:48.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:48.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:48.838947+0000) 2019-10-13T13:45:50.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:50.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:50.071877+0000) 2019-10-13T13:45:52.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:52.577+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.579516+0000) 2019-10-13T13:45:52.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:52.617+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.620960+0000) 2019-10-13T13:45:52.624 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:52.617+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.621587+0000) 2019-10-13T13:45:52.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:52.617+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.622462+0000) 2019-10-13T13:45:52.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:52.617+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.622814+0000) 2019-10-13T13:45:52.626 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:52.621+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.623757+0000) 2019-10-13T13:45:52.626 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:52.621+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.624081+0000) 2019-10-13T13:45:52.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:52.621+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.624379+0000) 2019-10-13T13:45:52.631 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:52.625+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:52.628862+0000) 2019-10-13T13:45:52.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:52.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179762 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:53.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:53.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:53.679973+0000) 2019-10-13T13:45:55.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:55.601+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:55.604944+0000) 2019-10-13T13:45:55.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:55.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:55.891431+0000) 2019-10-13T13:45:56.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:56.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:56.571339+0000) 2019-10-13T13:45:57.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:45:57.629+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.633255+0000) 2019-10-13T13:45:57.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:45:57.629+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.633704+0000) 2019-10-13T13:45:57.637 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:57.629+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.634723+0000) 2019-10-13T13:45:57.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:57.629+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.635086+0000) 2019-10-13T13:45:57.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:45:57.633+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.635408+0000) 2019-10-13T13:45:57.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:45:57.633+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.636015+0000) 2019-10-13T13:45:57.639 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:45:57.633+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.636814+0000) 2019-10-13T13:45:57.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:45:57.637+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.640984+0000) 2019-10-13T13:45:57.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:45:57.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:45:58.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:45:58.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:58.838986+0000) 2019-10-13T13:45:58.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:45:58.833+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:58.839139+0000) 2019-10-13T13:46:00.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:00.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:00.072079+0000) 2019-10-13T13:46:02.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:02.577+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.579675+0000) 2019-10-13T13:46:02.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:02.641+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.645496+0000) 2019-10-13T13:46:02.648 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:02.641+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.646218+0000) 2019-10-13T13:46:02.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:02.641+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.646560+0000) 2019-10-13T13:46:02.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:02.641+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.646861+0000) 2019-10-13T13:46:02.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:02.645+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.647687+0000) 2019-10-13T13:46:02.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:02.645+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.647966+0000) 2019-10-13T13:46:02.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:02.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.648680+0000) 2019-10-13T13:46:02.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:02.653+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:02.654671+0000) 2019-10-13T13:46:02.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:02.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:03.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:03.685+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:03.686741+0000) 2019-10-13T13:46:05.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:05.601+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:05.605135+0000) 2019-10-13T13:46:05.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:05.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:05.891632+0000) 2019-10-13T13:46:06.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:06.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:06.571514+0000) 2019-10-13T13:46:07.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:07.653+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.658450+0000) 2019-10-13T13:46:07.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:07.653+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.658969+0000) 2019-10-13T13:46:07.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:07.657+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.659425+0000) 2019-10-13T13:46:07.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:07.657+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.660053+0000) 2019-10-13T13:46:07.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:07.657+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.660734+0000) 2019-10-13T13:46:07.663 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:07.657+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.661003+0000) 2019-10-13T13:46:07.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:07.657+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.661656+0000) 2019-10-13T13:46:07.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:07.665+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.666665+0000) 2019-10-13T13:46:07.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:07.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179866 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:08.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:08.833+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:08.839190+0000) 2019-10-13T13:46:08.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:08.837+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:08.839347+0000) 2019-10-13T13:46:10.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:10.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:10.072281+0000) 2019-10-13T13:46:12.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:12.577+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.579895+0000) 2019-10-13T13:46:12.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:12.669+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.671370+0000) 2019-10-13T13:46:12.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:12.669+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.671814+0000) 2019-10-13T13:46:12.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:12.669+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.672958+0000) 2019-10-13T13:46:12.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:12.669+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.672625+0000) 2019-10-13T13:46:12.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:12.669+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.673628+0000) 2019-10-13T13:46:12.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:12.669+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.674315+0000) 2019-10-13T13:46:12.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:12.673+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.674695+0000) 2019-10-13T13:46:12.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:12.677+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:12.680128+0000) 2019-10-13T13:46:12.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:12.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:13.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:13.689+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:13.693350+0000) 2019-10-13T13:46:15.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:15.604+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:15.605342+0000) 2019-10-13T13:46:15.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:15.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:15.891836+0000) 2019-10-13T13:46:15.923 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:46:15.923 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:15.926 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:46:15.927 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:16.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:16.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:16.571717+0000) 2019-10-13T13:46:17.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:17.680+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.683497+0000) 2019-10-13T13:46:17.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:17.680+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.683701+0000) 2019-10-13T13:46:17.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:17.680+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.684459+0000) 2019-10-13T13:46:17.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:17.680+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.684771+0000) 2019-10-13T13:46:17.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:17.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.685253+0000) 2019-10-13T13:46:17.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:17.685+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.686940+0000) 2019-10-13T13:46:17.689 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:17.680+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.687076+0000) 2019-10-13T13:46:17.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:17.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:17.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:17.693+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.694626+0000) 2019-10-13T13:46:18.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:18.836+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:18.839419+0000) 2019-10-13T13:46:18.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:18.836+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:18.839538+0000) 2019-10-13T13:46:20.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:20.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:20.072514+0000) 2019-10-13T13:46:22.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:22.576+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.580116+0000) 2019-10-13T13:46:22.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:22.688+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.695121+0000) 2019-10-13T13:46:22.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:22.693+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.696439+0000) 2019-10-13T13:46:22.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:22.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.696675+0000) 2019-10-13T13:46:22.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:22.692+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.697100+0000) 2019-10-13T13:46:22.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:22.692+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.697464+0000) 2019-10-13T13:46:22.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:22.692+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.697966+0000) 2019-10-13T13:46:22.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:22.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.699200+0000) 2019-10-13T13:46:22.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:22.701+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:22.704011+0000) 2019-10-13T13:46:22.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:22.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 179967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:23.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:23.697+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:23.699889+0000) 2019-10-13T13:46:25.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:25.604+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:25.605581+0000) 2019-10-13T13:46:25.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:25.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:25.892047+0000) 2019-10-13T13:46:26.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:26.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:26.571909+0000) 2019-10-13T13:46:27.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:27.704+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.708002+0000) 2019-10-13T13:46:27.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:27.704+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.708617+0000) 2019-10-13T13:46:27.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:27.704+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.709039+0000) 2019-10-13T13:46:27.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:27.705+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.709624+0000) 2019-10-13T13:46:27.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:27.704+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.710476+0000) 2019-10-13T13:46:27.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:27.704+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.711087+0000) 2019-10-13T13:46:27.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:27.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.711466+0000) 2019-10-13T13:46:27.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:27.713+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.716432+0000) 2019-10-13T13:46:27.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:27.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:28.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:28.836+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:28.839577+0000) 2019-10-13T13:46:28.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:28.836+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:28.839730+0000) 2019-10-13T13:46:30.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:30.069+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:30.072696+0000) 2019-10-13T13:46:32.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:32.576+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.580335+0000) 2019-10-13T13:46:32.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:32.716+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.720096+0000) 2019-10-13T13:46:32.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:32.716+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.721633+0000) 2019-10-13T13:46:32.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:32.716+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.722228+0000) 2019-10-13T13:46:32.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:32.716+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.722365+0000) 2019-10-13T13:46:32.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:32.716+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.722621+0000) 2019-10-13T13:46:32.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:32.721+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.723172+0000) 2019-10-13T13:46:32.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:32.720+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.723663+0000) 2019-10-13T13:46:32.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:32.725+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:32.727314+0000) 2019-10-13T13:46:32.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:32.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180034 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:33.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:33.705+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:33.706247+0000) 2019-10-13T13:46:35.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:35.600+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:35.605779+0000) 2019-10-13T13:46:35.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:35.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:35.892243+0000) 2019-10-13T13:46:36.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:36.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:36.572075+0000) 2019-10-13T13:46:37.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:37.728+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.732365+0000) 2019-10-13T13:46:37.735 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:37.728+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.732739+0000) 2019-10-13T13:46:37.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:37.728+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.733018+0000) 2019-10-13T13:46:37.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:37.728+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.734139+0000) 2019-10-13T13:46:37.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:37.728+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.734157+0000) 2019-10-13T13:46:37.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:37.732+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.734910+0000) 2019-10-13T13:46:37.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:37.728+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.734752+0000) 2019-10-13T13:46:37.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:37.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.738458+0000) 2019-10-13T13:46:37.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:37.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:38.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:38.836+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:38.839851+0000) 2019-10-13T13:46:38.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:38.836+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:38.839944+0000) 2019-10-13T13:46:40.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:40.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:40.072886+0000) 2019-10-13T13:46:42.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:42.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.580537+0000) 2019-10-13T13:46:42.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:42.740+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.743548+0000) 2019-10-13T13:46:42.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:42.740+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.744120+0000) 2019-10-13T13:46:42.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:42.740+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.745605+0000) 2019-10-13T13:46:42.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:42.740+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.746112+0000) 2019-10-13T13:46:42.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:42.740+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.746113+0000) 2019-10-13T13:46:42.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:42.740+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.747006+0000) 2019-10-13T13:46:42.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:42.744+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.747353+0000) 2019-10-13T13:46:42.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:42.748+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:42.753021+0000) 2019-10-13T13:46:42.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:42.748+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:43.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:43.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:43.712704+0000) 2019-10-13T13:46:45.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:45.604+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:45.605977+0000) 2019-10-13T13:46:45.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:45.888+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:45.892447+0000) 2019-10-13T13:46:46.084 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:46:46.084 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:46.088 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:46:46.088 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:46.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:46.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:46.572271+0000) 2019-10-13T13:46:47.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:47.752+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.756132+0000) 2019-10-13T13:46:47.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:47.752+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.756430+0000) 2019-10-13T13:46:47.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:47.752+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.757439+0000) 2019-10-13T13:46:47.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:47.752+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.757847+0000) 2019-10-13T13:46:47.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:47.752+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.758149+0000) 2019-10-13T13:46:47.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:47.752+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.758724+0000) 2019-10-13T13:46:47.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:47.752+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.758846+0000) 2019-10-13T13:46:47.766 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:47.760+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.763716+0000) 2019-10-13T13:46:47.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:47.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:48.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:48.836+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:48.840083+0000) 2019-10-13T13:46:48.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:48.836+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:48.840169+0000) 2019-10-13T13:46:50.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:50.068+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:50.073080+0000) 2019-10-13T13:46:52.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:52.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.580787+0000) 2019-10-13T13:46:52.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:52.764+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.770391+0000) 2019-10-13T13:46:52.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:52.764+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.771044+0000) 2019-10-13T13:46:52.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:52.764+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.770683+0000) 2019-10-13T13:46:52.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:52.768+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.771576+0000) 2019-10-13T13:46:52.775 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:52.768+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.771556+0000) 2019-10-13T13:46:52.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:52.768+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.771999+0000) 2019-10-13T13:46:52.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:52.768+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.772771+0000) 2019-10-13T13:46:52.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:52.772+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:52.776673+0000) 2019-10-13T13:46:52.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:52.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:53.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:53.716+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:53.719227+0000) 2019-10-13T13:46:55.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:55.604+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:55.606177+0000) 2019-10-13T13:46:55.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:55.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:55.892674+0000) 2019-10-13T13:46:56.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:56.568+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:56.572469+0000) 2019-10-13T13:46:57.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:46:57.775+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.782141+0000) 2019-10-13T13:46:57.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:46:57.775+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.782609+0000) 2019-10-13T13:46:57.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:46:57.780+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.784043+0000) 2019-10-13T13:46:57.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:57.779+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.784102+0000) 2019-10-13T13:46:57.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:46:57.779+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.784803+0000) 2019-10-13T13:46:57.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:46:57.779+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.784836+0000) 2019-10-13T13:46:57.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:57.779+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.786043+0000) 2019-10-13T13:46:57.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:46:57.788+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.791367+0000) 2019-10-13T13:46:57.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:46:57.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:46:58.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:46:58.835+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:58.840303+0000) 2019-10-13T13:46:58.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:46:58.835+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:58.840386+0000) 2019-10-13T13:47:00.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:00.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:00.073272+0000) 2019-10-13T13:47:02.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:02.579+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.580988+0000) 2019-10-13T13:47:02.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:02.787+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.794601+0000) 2019-10-13T13:47:02.798 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:02.792+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.795921+0000) 2019-10-13T13:47:02.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:02.791+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.795996+0000) 2019-10-13T13:47:02.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:02.791+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.796742+0000) 2019-10-13T13:47:02.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:02.791+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.797310+0000) 2019-10-13T13:47:02.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:02.791+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.797919+0000) 2019-10-13T13:47:02.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:02.791+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.798310+0000) 2019-10-13T13:47:02.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:02.800+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:02.802899+0000) 2019-10-13T13:47:02.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:02.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:03.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:03.724+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:03.725690+0000) 2019-10-13T13:47:05.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:05.603+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:05.606401+0000) 2019-10-13T13:47:05.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:05.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:05.892870+0000) 2019-10-13T13:47:06.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:06.571+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:06.572698+0000) 2019-10-13T13:47:07.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:07.799+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.805482+0000) 2019-10-13T13:47:07.808 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:07.804+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.806493+0000) 2019-10-13T13:47:07.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:07.803+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.807419+0000) 2019-10-13T13:47:07.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:07.803+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.807937+0000) 2019-10-13T13:47:07.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:07.803+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.808302+0000) 2019-10-13T13:47:07.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:07.803+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.809085+0000) 2019-10-13T13:47:07.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:07.803+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.809656+0000) 2019-10-13T13:47:07.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:07.808+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.813134+0000) 2019-10-13T13:47:07.818 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:07.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:08.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:08.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:08.840538+0000) 2019-10-13T13:47:08.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:08.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:08.840609+0000) 2019-10-13T13:47:10.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:10.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:10.073503+0000) 2019-10-13T13:47:12.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:12.575+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.581188+0000) 2019-10-13T13:47:12.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:12.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.819560+0000) 2019-10-13T13:47:12.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:12.815+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.820419+0000) 2019-10-13T13:47:12.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:12.816+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.821270+0000) 2019-10-13T13:47:12.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:12.815+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.821230+0000) 2019-10-13T13:47:12.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:12.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.821847+0000) 2019-10-13T13:47:12.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:12.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.822101+0000) 2019-10-13T13:47:12.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:12.815+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.823021+0000) 2019-10-13T13:47:12.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:12.824+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:12.826538+0000) 2019-10-13T13:47:12.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:12.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:13.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:13.736+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:13.736878+0000) 2019-10-13T13:47:15.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:15.599+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:15.606620+0000) 2019-10-13T13:47:15.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:15.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:15.893066+0000) 2019-10-13T13:47:16.468 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:47:16.468 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:16.472 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:47:16.472 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:16.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:16.571+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:16.572920+0000) 2019-10-13T13:47:17.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:17.827+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.832636+0000) 2019-10-13T13:47:17.835 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:17.828+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.833395+0000) 2019-10-13T13:47:17.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:17.827+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.833399+0000) 2019-10-13T13:47:17.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:17.827+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.833705+0000) 2019-10-13T13:47:17.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:17.827+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.834049+0000) 2019-10-13T13:47:17.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:17.827+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.834401+0000) 2019-10-13T13:47:17.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:17.831+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.835438+0000) 2019-10-13T13:47:17.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:17.840+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.842508+0000) 2019-10-13T13:47:17.846 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:17.835+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:18.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:18.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:18.840764+0000) 2019-10-13T13:47:18.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:18.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:18.840818+0000) 2019-10-13T13:47:20.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:20.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:20.073702+0000) 2019-10-13T13:47:22.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:22.579+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.581388+0000) 2019-10-13T13:47:22.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:22.839+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.844246+0000) 2019-10-13T13:47:22.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:22.839+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.844493+0000) 2019-10-13T13:47:22.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:22.840+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.845050+0000) 2019-10-13T13:47:22.848 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:22.839+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.845297+0000) 2019-10-13T13:47:22.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:22.839+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.846283+0000) 2019-10-13T13:47:22.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:22.839+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.846928+0000) 2019-10-13T13:47:22.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:22.843+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.847352+0000) 2019-10-13T13:47:22.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:22.848+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:22.852254+0000) 2019-10-13T13:47:22.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:22.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:23.745 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:23.740+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:23.743375+0000) 2019-10-13T13:47:25.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:25.603+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:25.606846+0000) 2019-10-13T13:47:25.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:25.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:25.893273+0000) 2019-10-13T13:47:26.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:26.571+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:26.573127+0000) 2019-10-13T13:47:27.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:27.855+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.857253+0000) 2019-10-13T13:47:27.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:27.855+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.857578+0000) 2019-10-13T13:47:27.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:27.855+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.857973+0000) 2019-10-13T13:47:27.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:27.855+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.858339+0000) 2019-10-13T13:47:27.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:27.856+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.859831+0000) 2019-10-13T13:47:27.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:27.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.860483+0000) 2019-10-13T13:47:27.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:27.859+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.860796+0000) 2019-10-13T13:47:27.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:27.860+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.865633+0000) 2019-10-13T13:47:27.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:27.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:28.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:28.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:28.840950+0000) 2019-10-13T13:47:28.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:28.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:28.841018+0000) 2019-10-13T13:47:30.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:30.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:30.073906+0000) 2019-10-13T13:47:32.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:32.579+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.581592+0000) 2019-10-13T13:47:32.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:32.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.869943+0000) 2019-10-13T13:47:32.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:32.867+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.869406+0000) 2019-10-13T13:47:32.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:32.867+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.870243+0000) 2019-10-13T13:47:32.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:32.867+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.870652+0000) 2019-10-13T13:47:32.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:32.867+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.870852+0000) 2019-10-13T13:47:32.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:32.871+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.872095+0000) 2019-10-13T13:47:32.875 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:32.871+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.873385+0000) 2019-10-13T13:47:32.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:32.875+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:32.878901+0000) 2019-10-13T13:47:32.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:32.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:33.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:33.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:33.749981+0000) 2019-10-13T13:47:35.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:35.603+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:35.607043+0000) 2019-10-13T13:47:35.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:35.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:35.893507+0000) 2019-10-13T13:47:36.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:36.571+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:36.573321+0000) 2019-10-13T13:47:37.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:37.879+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.883085+0000) 2019-10-13T13:47:37.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:37.883+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.883296+0000) 2019-10-13T13:47:37.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:37.883+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.883663+0000) 2019-10-13T13:47:37.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:37.883+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.884696+0000) 2019-10-13T13:47:37.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:37.879+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.885445+0000) 2019-10-13T13:47:37.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:37.883+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.885775+0000) 2019-10-13T13:47:37.889 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:37.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.886860+0000) 2019-10-13T13:47:37.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:37.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.890781+0000) 2019-10-13T13:47:37.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:37.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:38.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:38.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:38.841142+0000) 2019-10-13T13:47:38.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:38.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:38.841226+0000) 2019-10-13T13:47:40.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:40.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:40.074173+0000) 2019-10-13T13:47:42.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:42.578+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.581816+0000) 2019-10-13T13:47:42.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:42.891+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.896383+0000) 2019-10-13T13:47:42.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:42.894+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.896542+0000) 2019-10-13T13:47:42.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:42.894+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.897125+0000) 2019-10-13T13:47:42.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:42.894+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.897607+0000) 2019-10-13T13:47:42.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:42.894+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.898031+0000) 2019-10-13T13:47:42.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:42.894+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.898667+0000) 2019-10-13T13:47:42.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:42.898+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.899463+0000) 2019-10-13T13:47:42.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:42.899+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:42.905024+0000) 2019-10-13T13:47:42.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:42.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:43.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:43.755+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:43.756067+0000) 2019-10-13T13:47:45.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:45.602+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:45.607198+0000) 2019-10-13T13:47:45.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:45.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:45.893702+0000) 2019-10-13T13:47:46.524 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:47:46.524 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:46.528 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:47:46.528 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:46.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:46.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:46.573547+0000) 2019-10-13T13:47:47.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:47.906+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.908775+0000) 2019-10-13T13:47:47.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:47.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.909822+0000) 2019-10-13T13:47:47.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:47.906+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.910535+0000) 2019-10-13T13:47:47.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:47.906+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.910783+0000) 2019-10-13T13:47:47.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:47.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.911518+0000) 2019-10-13T13:47:47.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:47.910+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.911584+0000) 2019-10-13T13:47:47.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:47.910+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.912266+0000) 2019-10-13T13:47:47.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:47.911+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.916714+0000) 2019-10-13T13:47:47.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:47.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180546 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:48.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:48.838+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:48.841325+0000) 2019-10-13T13:47:48.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:48.838+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:48.841412+0000) 2019-10-13T13:47:50.076 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:50.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:50.074392+0000) 2019-10-13T13:47:52.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:52.578+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.582034+0000) 2019-10-13T13:47:52.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:52.918+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.921745+0000) 2019-10-13T13:47:52.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:52.918+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.922159+0000) 2019-10-13T13:47:52.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:52.918+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.922486+0000) 2019-10-13T13:47:52.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:52.918+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.923073+0000) 2019-10-13T13:47:52.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:52.919+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.923172+0000) 2019-10-13T13:47:52.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:52.922+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.923506+0000) 2019-10-13T13:47:52.927 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:52.922+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.924583+0000) 2019-10-13T13:47:52.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:52.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:52.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:52.927+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:52.933326+0000) 2019-10-13T13:47:53.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:53.759+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:53.762492+0000) 2019-10-13T13:47:55.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:55.606+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:55.607461+0000) 2019-10-13T13:47:55.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:55.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:55.893927+0000) 2019-10-13T13:47:56.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:56.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:56.573771+0000) 2019-10-13T13:47:57.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:57.930+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.933944+0000) 2019-10-13T13:47:57.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:47:57.930+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.934281+0000) 2019-10-13T13:47:57.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:47:57.931+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.934970+0000) 2019-10-13T13:47:57.938 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:47:57.930+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.935146+0000) 2019-10-13T13:47:57.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:47:57.934+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.935687+0000) 2019-10-13T13:47:57.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:47:57.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.936999+0000) 2019-10-13T13:47:57.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:57.934+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.937477+0000) 2019-10-13T13:47:57.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:47:57.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180616 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:47:57.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:47:57.939+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.943740+0000) 2019-10-13T13:47:58.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:47:58.838+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:58.841498+0000) 2019-10-13T13:47:58.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:47:58.838+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:58.841580+0000) 2019-10-13T13:48:00.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:00.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:00.074617+0000) 2019-10-13T13:48:02.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:02.578+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.582271+0000) 2019-10-13T13:48:02.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:02.942+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.946298+0000) 2019-10-13T13:48:02.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:02.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.947343+0000) 2019-10-13T13:48:02.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:02.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.947893+0000) 2019-10-13T13:48:02.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:02.943+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.949038+0000) 2019-10-13T13:48:02.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:02.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.949203+0000) 2019-10-13T13:48:02.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:02.946+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.949415+0000) 2019-10-13T13:48:02.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:02.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.950411+0000) 2019-10-13T13:48:02.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:02.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:02.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:02.951+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:02.956509+0000) 2019-10-13T13:48:03.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:03.767+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:03.769098+0000) 2019-10-13T13:48:05.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:05.606+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:05.607651+0000) 2019-10-13T13:48:05.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:05.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:05.894124+0000) 2019-10-13T13:48:06.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:06.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:06.573952+0000) 2019-10-13T13:48:07.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:07.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.958600+0000) 2019-10-13T13:48:07.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:07.954+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.958847+0000) 2019-10-13T13:48:07.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:07.958+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.960923+0000) 2019-10-13T13:48:07.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:07.958+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.961105+0000) 2019-10-13T13:48:07.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:07.958+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.961516+0000) 2019-10-13T13:48:07.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:07.959+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.962188+0000) 2019-10-13T13:48:07.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:07.958+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.962304+0000) 2019-10-13T13:48:07.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:07.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:07.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:07.963+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.969732+0000) 2019-10-13T13:48:08.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:08.838+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:08.841674+0000) 2019-10-13T13:48:08.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:08.838+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:08.841774+0000) 2019-10-13T13:48:10.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:10.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:10.074827+0000) 2019-10-13T13:48:12.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:12.582+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.582469+0000) 2019-10-13T13:48:12.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:12.966+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.970910+0000) 2019-10-13T13:48:12.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:12.970+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.971984+0000) 2019-10-13T13:48:12.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:12.970+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.972935+0000) 2019-10-13T13:48:12.976 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:12.970+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.973868+0000) 2019-10-13T13:48:12.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:12.970+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.974026+0000) 2019-10-13T13:48:12.977 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:12.971+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.974977+0000) 2019-10-13T13:48:12.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:12.974+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.975714+0000) 2019-10-13T13:48:12.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:12.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:12.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:12.975+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:12.982010+0000) 2019-10-13T13:48:13.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:13.775+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:13.775518+0000) 2019-10-13T13:48:15.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:15.606+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:15.607866+0000) 2019-10-13T13:48:15.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:15.890+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:15.894340+0000) 2019-10-13T13:48:16.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:16.570+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:16.574142+0000) 2019-10-13T13:48:16.634 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:48:16.634 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:16.638 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:48:16.638 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:17.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:17.979+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.984315+0000) 2019-10-13T13:48:17.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:17.982+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.984585+0000) 2019-10-13T13:48:17.988 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:17.982+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.985563+0000) 2019-10-13T13:48:17.989 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:17.982+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.986539+0000) 2019-10-13T13:48:17.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:17.982+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.986956+0000) 2019-10-13T13:48:17.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:17.986+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.987824+0000) 2019-10-13T13:48:17.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:17.986+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.988476+0000) 2019-10-13T13:48:17.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:17.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:17.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:17.991+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.994522+0000) 2019-10-13T13:48:18.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:18.838+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:18.841852+0000) 2019-10-13T13:48:18.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:18.838+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:18.841960+0000) 2019-10-13T13:48:20.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:20.071+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:20.075031+0000) 2019-10-13T13:48:22.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:22.578+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.582672+0000) 2019-10-13T13:48:22.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:22.994+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.996902+0000) 2019-10-13T13:48:23.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:22.994+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.997314+0000) 2019-10-13T13:48:23.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:22.994+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.997861+0000) 2019-10-13T13:48:23.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:22.995+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.998911+0000) 2019-10-13T13:48:23.001 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:22.994+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.999035+0000) 2019-10-13T13:48:23.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:22.998+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:22.999475+0000) 2019-10-13T13:48:23.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:22.998+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:23.000023+0000) 2019-10-13T13:48:23.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:22.999+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:23.005168+0000) 2019-10-13T13:48:23.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:23.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:23.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:23.779+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:23.781996+0000) 2019-10-13T13:48:25.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:25.605+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:25.608082+0000) 2019-10-13T13:48:25.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:25.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:25.894541+0000) 2019-10-13T13:48:26.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:26.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:26.574357+0000) 2019-10-13T13:48:28.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:28.002+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.008967+0000) 2019-10-13T13:48:28.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:28.005+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.010355+0000) 2019-10-13T13:48:28.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:28.005+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.011093+0000) 2019-10-13T13:48:28.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:28.005+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.011075+0000) 2019-10-13T13:48:28.014 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:28.009+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.011540+0000) 2019-10-13T13:48:28.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:28.009+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.012208+0000) 2019-10-13T13:48:28.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:28.009+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.011899+0000) 2019-10-13T13:48:28.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:28.010+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.016365+0000) 2019-10-13T13:48:28.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:28.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:28.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:28.837+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.842062+0000) 2019-10-13T13:48:28.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:28.837+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.842146+0000) 2019-10-13T13:48:30.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:30.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:30.075228+0000) 2019-10-13T13:48:32.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:32.577+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:32.582823+0000) 2019-10-13T13:48:33.024 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:33.017+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.021534+0000) 2019-10-13T13:48:33.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:33.018+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.022874+0000) 2019-10-13T13:48:33.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:33.017+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.022945+0000) 2019-10-13T13:48:33.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:33.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.023478+0000) 2019-10-13T13:48:33.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:33.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.023736+0000) 2019-10-13T13:48:33.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:33.021+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.024118+0000) 2019-10-13T13:48:33.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:33.021+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.025375+0000) 2019-10-13T13:48:33.030 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:33.022+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.028167+0000) 2019-10-13T13:48:33.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:33.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:33.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:33.786+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:33.788429+0000) 2019-10-13T13:48:35.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:35.605+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:35.608317+0000) 2019-10-13T13:48:35.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:35.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:35.894747+0000) 2019-10-13T13:48:36.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:36.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:36.574552+0000) 2019-10-13T13:48:38.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:38.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.033896+0000) 2019-10-13T13:48:38.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:38.029+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.034908+0000) 2019-10-13T13:48:38.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:38.029+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.035125+0000) 2019-10-13T13:48:38.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:38.033+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.036123+0000) 2019-10-13T13:48:38.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:38.033+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.037038+0000) 2019-10-13T13:48:38.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:38.033+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.037512+0000) 2019-10-13T13:48:38.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:38.034+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.038244+0000) 2019-10-13T13:48:38.045 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:38.038+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.043430+0000) 2019-10-13T13:48:38.046 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:38.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180890 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:38.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:38.841+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.842245+0000) 2019-10-13T13:48:38.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:38.841+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.842313+0000) 2019-10-13T13:48:40.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:40.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:40.075429+0000) 2019-10-13T13:48:42.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:42.581+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:42.583015+0000) 2019-10-13T13:48:43.050 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:43.041+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.047216+0000) 2019-10-13T13:48:43.050 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:43.045+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.047611+0000) 2019-10-13T13:48:43.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:43.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.048353+0000) 2019-10-13T13:48:43.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:43.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.049029+0000) 2019-10-13T13:48:43.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:43.045+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.049164+0000) 2019-10-13T13:48:43.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:43.045+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.049873+0000) 2019-10-13T13:48:43.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:43.046+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.050567+0000) 2019-10-13T13:48:43.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:43.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:43.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:43.050+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.057233+0000) 2019-10-13T13:48:43.797 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:43.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:43.794866+0000) 2019-10-13T13:48:45.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:45.605+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:45.608525+0000) 2019-10-13T13:48:45.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:45.889+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:45.894975+0000) 2019-10-13T13:48:46.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:46.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:46.574746+0000) 2019-10-13T13:48:46.687 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:48:46.687 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:46.691 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:48:46.691 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:48.061 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:48.053+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.058601+0000) 2019-10-13T13:48:48.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:48.053+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.058992+0000) 2019-10-13T13:48:48.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:48.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.059940+0000) 2019-10-13T13:48:48.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:48.057+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.060181+0000) 2019-10-13T13:48:48.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:48.057+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.061080+0000) 2019-10-13T13:48:48.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:48.057+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.061350+0000) 2019-10-13T13:48:48.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:48.058+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.062454+0000) 2019-10-13T13:48:48.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:48.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180957 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:48.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:48.062+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.069221+0000) 2019-10-13T13:48:48.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:48.837+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.842412+0000) 2019-10-13T13:48:48.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:48.837+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.842508+0000) 2019-10-13T13:48:50.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:50.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:50.075622+0000) 2019-10-13T13:48:52.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:52.577+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:52.583212+0000) 2019-10-13T13:48:53.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:53.069+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.071898+0000) 2019-10-13T13:48:53.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:53.066+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.072777+0000) 2019-10-13T13:48:53.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:53.069+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.072823+0000) 2019-10-13T13:48:53.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:53.069+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.073193+0000) 2019-10-13T13:48:53.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:53.069+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.073716+0000) 2019-10-13T13:48:53.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:53.069+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.074302+0000) 2019-10-13T13:48:53.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:53.069+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.074850+0000) 2019-10-13T13:48:53.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:53.074+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.078863+0000) 2019-10-13T13:48:53.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:53.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 180991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:53.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:53.798+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:53.801302+0000) 2019-10-13T13:48:55.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:55.605+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:55.608731+0000) 2019-10-13T13:48:55.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:55.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:55.895180+0000) 2019-10-13T13:48:56.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:56.569+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:56.574935+0000) 2019-10-13T13:48:58.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:48:58.078+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.084522+0000) 2019-10-13T13:48:58.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:48:58.081+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.084911+0000) 2019-10-13T13:48:58.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:48:58.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.085425+0000) 2019-10-13T13:48:58.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:48:58.081+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.086212+0000) 2019-10-13T13:48:58.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:58.081+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.086379+0000) 2019-10-13T13:48:58.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:48:58.081+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.086795+0000) 2019-10-13T13:48:58.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:58.081+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.087032+0000) 2019-10-13T13:48:58.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:48:58.090+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.094374+0000) 2019-10-13T13:48:58.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:48:58.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181026 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:48:58.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:48:58.841+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.842606+0000) 2019-10-13T13:48:58.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:48:58.841+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.842696+0000) 2019-10-13T13:49:00.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:00.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:00.075818+0000) 2019-10-13T13:49:02.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:02.581+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:02.583404+0000) 2019-10-13T13:49:03.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:03.093+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.096303+0000) 2019-10-13T13:49:03.099 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:03.093+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.096708+0000) 2019-10-13T13:49:03.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:03.093+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.097442+0000) 2019-10-13T13:49:03.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:03.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.097697+0000) 2019-10-13T13:49:03.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:03.093+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.098150+0000) 2019-10-13T13:49:03.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:03.097+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.100817+0000) 2019-10-13T13:49:03.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:03.094+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.100130+0000) 2019-10-13T13:49:03.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:03.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181058 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:03.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:03.102+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.106699+0000) 2019-10-13T13:49:03.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:03.806+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:03.808148+0000) 2019-10-13T13:49:05.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:05.605+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:05.608926+0000) 2019-10-13T13:49:05.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:05.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:05.895381+0000) 2019-10-13T13:49:06.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:06.573+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:06.575083+0000) 2019-10-13T13:49:08.112 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:08.104+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.109570+0000) 2019-10-13T13:49:08.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:08.104+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.110049+0000) 2019-10-13T13:49:08.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:08.104+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.110437+0000) 2019-10-13T13:49:08.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:08.104+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.110758+0000) 2019-10-13T13:49:08.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:08.108+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.112035+0000) 2019-10-13T13:49:08.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:08.106+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.113944+0000) 2019-10-13T13:49:08.116 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:08.108+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.114215+0000) 2019-10-13T13:49:08.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:08.112+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181093 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:08.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:08.114+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.119312+0000) 2019-10-13T13:49:08.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:08.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.842801+0000) 2019-10-13T13:49:08.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:08.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.842928+0000) 2019-10-13T13:49:10.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:10.070+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:10.076011+0000) 2019-10-13T13:49:12.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:12.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:12.583604+0000) 2019-10-13T13:49:13.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:13.116+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.122169+0000) 2019-10-13T13:49:13.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:13.116+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.122628+0000) 2019-10-13T13:49:13.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:13.120+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.123827+0000) 2019-10-13T13:49:13.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:13.122+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.124736+0000) 2019-10-13T13:49:13.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:13.120+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.125037+0000) 2019-10-13T13:49:13.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:13.120+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.125740+0000) 2019-10-13T13:49:13.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:13.120+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.125909+0000) 2019-10-13T13:49:13.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:13.130+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.130245+0000) 2019-10-13T13:49:13.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:13.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:13.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:13.814+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:13.814736+0000) 2019-10-13T13:49:15.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:15.604+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:15.609152+0000) 2019-10-13T13:49:15.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:15.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:15.895584+0000) 2019-10-13T13:49:16.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:16.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:16.575281+0000) 2019-10-13T13:49:16.795 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:49:16.795 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:16.798 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:49:16.799 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:18.137 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:18.128+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.134472+0000) 2019-10-13T13:49:18.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:18.132+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.135538+0000) 2019-10-13T13:49:18.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:18.132+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.136570+0000) 2019-10-13T13:49:18.139 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:18.132+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.136746+0000) 2019-10-13T13:49:18.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:18.132+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.137220+0000) 2019-10-13T13:49:18.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:18.132+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.137753+0000) 2019-10-13T13:49:18.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:18.138+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.138444+0000) 2019-10-13T13:49:18.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:18.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:18.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:18.142+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.145830+0000) 2019-10-13T13:49:18.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:18.836+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.843006+0000) 2019-10-13T13:49:18.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:18.836+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.843111+0000) 2019-10-13T13:49:20.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:20.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.076207+0000) 2019-10-13T13:49:22.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:22.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:22.583750+0000) 2019-10-13T13:49:23.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:23.144+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.147433+0000) 2019-10-13T13:49:23.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:23.144+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.147929+0000) 2019-10-13T13:49:23.151 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:23.144+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.148479+0000) 2019-10-13T13:49:23.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:23.144+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.149747+0000) 2019-10-13T13:49:23.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:23.144+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.150170+0000) 2019-10-13T13:49:23.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:23.144+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.150590+0000) 2019-10-13T13:49:23.153 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:23.149+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.151610+0000) 2019-10-13T13:49:23.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:23.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:23.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:23.153+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.156270+0000) 2019-10-13T13:49:23.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:23.817+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:23.821452+0000) 2019-10-13T13:49:25.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:25.608+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.609376+0000) 2019-10-13T13:49:25.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:25.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.895775+0000) 2019-10-13T13:49:26.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:26.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:26.575532+0000) 2019-10-13T13:49:28.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:28.152+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.159268+0000) 2019-10-13T13:49:28.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:28.157+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.160364+0000) 2019-10-13T13:49:28.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:28.156+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.160430+0000) 2019-10-13T13:49:28.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:28.156+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.161079+0000) 2019-10-13T13:49:28.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:28.156+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.161587+0000) 2019-10-13T13:49:28.164 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:28.156+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.162052+0000) 2019-10-13T13:49:28.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:28.156+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.162449+0000) 2019-10-13T13:49:28.170 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:28.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181229 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:28.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:28.165+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.169460+0000) 2019-10-13T13:49:28.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:28.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.843181+0000) 2019-10-13T13:49:28.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:28.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.843346+0000) 2019-10-13T13:49:30.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:30.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.076404+0000) 2019-10-13T13:49:32.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:32.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:32.583963+0000) 2019-10-13T13:49:33.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:33.164+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.171087+0000) 2019-10-13T13:49:33.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:33.168+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.172174+0000) 2019-10-13T13:49:33.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:33.168+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.173562+0000) 2019-10-13T13:49:33.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:33.173+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.174305+0000) 2019-10-13T13:49:33.177 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:33.168+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.174294+0000) 2019-10-13T13:49:33.177 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:33.168+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.174699+0000) 2019-10-13T13:49:33.177 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:33.172+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.175346+0000) 2019-10-13T13:49:33.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:33.177+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.179083+0000) 2019-10-13T13:49:33.182 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:33.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:33.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:33.825+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:33.827498+0000) 2019-10-13T13:49:35.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:35.608+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.609578+0000) 2019-10-13T13:49:35.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:35.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.896014+0000) 2019-10-13T13:49:36.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:36.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:36.575723+0000) 2019-10-13T13:49:38.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:38.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.183213+0000) 2019-10-13T13:49:38.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:38.180+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.183539+0000) 2019-10-13T13:49:38.187 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:38.180+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.184593+0000) 2019-10-13T13:49:38.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:38.181+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.185396+0000) 2019-10-13T13:49:38.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:38.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.185583+0000) 2019-10-13T13:49:38.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:38.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.186066+0000) 2019-10-13T13:49:38.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:38.180+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.186427+0000) 2019-10-13T13:49:38.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:38.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:38.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:38.189+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.193534+0000) 2019-10-13T13:49:38.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:38.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.843378+0000) 2019-10-13T13:49:38.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:38.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.843531+0000) 2019-10-13T13:49:40.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:40.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.076596+0000) 2019-10-13T13:49:42.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:42.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:42.584169+0000) 2019-10-13T13:49:43.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:43.188+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.194288+0000) 2019-10-13T13:49:43.197 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:43.188+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.194941+0000) 2019-10-13T13:49:43.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:43.192+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.195386+0000) 2019-10-13T13:49:43.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:43.192+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.195995+0000) 2019-10-13T13:49:43.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:43.193+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.197308+0000) 2019-10-13T13:49:43.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:43.192+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.197852+0000) 2019-10-13T13:49:43.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:43.192+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.198013+0000) 2019-10-13T13:49:43.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:43.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:43.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:43.201+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.205458+0000) 2019-10-13T13:49:43.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:43.833+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:43.834626+0000) 2019-10-13T13:49:45.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:45.608+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.609781+0000) 2019-10-13T13:49:45.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:45.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.896170+0000) 2019-10-13T13:49:46.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:46.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:46.575921+0000) 2019-10-13T13:49:46.910 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:49:46.910 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:46.913 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:49:46.914 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:48.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:48.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.206169+0000) 2019-10-13T13:49:48.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:48.200+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.206886+0000) 2019-10-13T13:49:48.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:48.205+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.207660+0000) 2019-10-13T13:49:48.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:48.204+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.207754+0000) 2019-10-13T13:49:48.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:48.204+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.208976+0000) 2019-10-13T13:49:48.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:48.204+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.209415+0000) 2019-10-13T13:49:48.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:48.204+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.209514+0000) 2019-10-13T13:49:48.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:48.213+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.215471+0000) 2019-10-13T13:49:48.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:48.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:48.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:48.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.843578+0000) 2019-10-13T13:49:48.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:48.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.843737+0000) 2019-10-13T13:49:50.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:50.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.076801+0000) 2019-10-13T13:49:52.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:52.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:52.584368+0000) 2019-10-13T13:49:53.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:53.215+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.220693+0000) 2019-10-13T13:49:53.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:53.215+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.221330+0000) 2019-10-13T13:49:53.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:53.217+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.222078+0000) 2019-10-13T13:49:53.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:53.215+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.222312+0000) 2019-10-13T13:49:53.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:53.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.222721+0000) 2019-10-13T13:49:53.225 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:53.215+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.223249+0000) 2019-10-13T13:49:53.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:53.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.223949+0000) 2019-10-13T13:49:53.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:53.225+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.227901+0000) 2019-10-13T13:49:53.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:53.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:53.843 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:53.837+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:53.840730+0000) 2019-10-13T13:49:55.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:55.607+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.609986+0000) 2019-10-13T13:49:55.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:55.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.896385+0000) 2019-10-13T13:49:56.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:56.575+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:56.576128+0000) 2019-10-13T13:49:58.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:49:58.229+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.232332+0000) 2019-10-13T13:49:58.235 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:49:58.227+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.232550+0000) 2019-10-13T13:49:58.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:49:58.227+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.232985+0000) 2019-10-13T13:49:58.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:58.227+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.233173+0000) 2019-10-13T13:49:58.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:49:58.227+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.234256+0000) 2019-10-13T13:49:58.237 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:58.227+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.234813+0000) 2019-10-13T13:49:58.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:49:58.231+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.235517+0000) 2019-10-13T13:49:58.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:49:58.237+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.240616+0000) 2019-10-13T13:49:58.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:49:58.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:49:58.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:49:58.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.843760+0000) 2019-10-13T13:49:58.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:49:58.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.843883+0000) 2019-10-13T13:50:00.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:00.073+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.077018+0000) 2019-10-13T13:50:02.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:02.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:02.584565+0000) 2019-10-13T13:50:03.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:03.239+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.244192+0000) 2019-10-13T13:50:03.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:03.239+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.244473+0000) 2019-10-13T13:50:03.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:03.239+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.245253+0000) 2019-10-13T13:50:03.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:03.239+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.245706+0000) 2019-10-13T13:50:03.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:03.239+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.246176+0000) 2019-10-13T13:50:03.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:03.245+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.247149+0000) 2019-10-13T13:50:03.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:03.243+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.247724+0000) 2019-10-13T13:50:03.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:03.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:03.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:03.253+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.254999+0000) 2019-10-13T13:50:03.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:03.845+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:03.847665+0000) 2019-10-13T13:50:05.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:05.607+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.610186+0000) 2019-10-13T13:50:05.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:05.891+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.896608+0000) 2019-10-13T13:50:06.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:06.575+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:06.576324+0000) 2019-10-13T13:50:08.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:08.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.258231+0000) 2019-10-13T13:50:08.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:08.251+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.258358+0000) 2019-10-13T13:50:08.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:08.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.260296+0000) 2019-10-13T13:50:08.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:08.255+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.260712+0000) 2019-10-13T13:50:08.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:08.257+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.261483+0000) 2019-10-13T13:50:08.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:08.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.261575+0000) 2019-10-13T13:50:08.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:08.255+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.261875+0000) 2019-10-13T13:50:08.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:08.261+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.265813+0000) 2019-10-13T13:50:08.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:08.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181493 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:08.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:08.843+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.843943+0000) 2019-10-13T13:50:08.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:08.843+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.844067+0000) 2019-10-13T13:50:10.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:10.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.077221+0000) 2019-10-13T13:50:12.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:12.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:12.584788+0000) 2019-10-13T13:50:13.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:13.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.269677+0000) 2019-10-13T13:50:13.273 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:13.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.271260+0000) 2019-10-13T13:50:13.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:13.267+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.271602+0000) 2019-10-13T13:50:13.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:13.267+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.272145+0000) 2019-10-13T13:50:13.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:13.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.272329+0000) 2019-10-13T13:50:13.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:13.267+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.272641+0000) 2019-10-13T13:50:13.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:13.267+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.273492+0000) 2019-10-13T13:50:13.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:13.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181524 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:13.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:13.277+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.281072+0000) 2019-10-13T13:50:13.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:13.853+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:13.854610+0000) 2019-10-13T13:50:15.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:15.603+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.610386+0000) 2019-10-13T13:50:15.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:15.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.896841+0000) 2019-10-13T13:50:16.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:16.571+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:16.576521+0000) 2019-10-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:18.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:18.280+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.282882+0000) 2019-10-13T13:50:18.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:18.279+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.283458+0000) 2019-10-13T13:50:18.286 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:18.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.284067+0000) 2019-10-13T13:50:18.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:18.279+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.284435+0000) 2019-10-13T13:50:18.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:18.279+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.285204+0000) 2019-10-13T13:50:18.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:18.279+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.284969+0000) 2019-10-13T13:50:18.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:18.279+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.285483+0000) 2019-10-13T13:50:18.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:18.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:18.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:18.292+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.295083+0000) 2019-10-13T13:50:18.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:18.843+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.844066+0000) 2019-10-13T13:50:18.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:18.843+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.844225+0000) 2019-10-13T13:50:20.079 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:20.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.077416+0000) 2019-10-13T13:50:22.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:22.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:22.584989+0000) 2019-10-13T13:50:23.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:23.295+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.295712+0000) 2019-10-13T13:50:23.299 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:23.295+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.296962+0000) 2019-10-13T13:50:23.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:23.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.297293+0000) 2019-10-13T13:50:23.300 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:23.292+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.297963+0000) 2019-10-13T13:50:23.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:23.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.297978+0000) 2019-10-13T13:50:23.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:23.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.298518+0000) 2019-10-13T13:50:23.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:23.295+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.299148+0000) 2019-10-13T13:50:23.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:23.300+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.302735+0000) 2019-10-13T13:50:23.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:23.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:23.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:23.864+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:23.866389+0000) 2019-10-13T13:50:25.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:25.607+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.610598+0000) 2019-10-13T13:50:25.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:25.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.897036+0000) 2019-10-13T13:50:26.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:26.575+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:26.576721+0000) 2019-10-13T13:50:28.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:28.307+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.308432+0000) 2019-10-13T13:50:28.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:28.307+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.309158+0000) 2019-10-13T13:50:28.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:28.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.309640+0000) 2019-10-13T13:50:28.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:28.308+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.310497+0000) 2019-10-13T13:50:28.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:28.311+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.311645+0000) 2019-10-13T13:50:28.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:28.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.311977+0000) 2019-10-13T13:50:28.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:28.311+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.312749+0000) 2019-10-13T13:50:28.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:28.312+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.317245+0000) 2019-10-13T13:50:28.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:28.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:28.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:28.843+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.844254+0000) 2019-10-13T13:50:28.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:28.843+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.844404+0000) 2019-10-13T13:50:30.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:30.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.077616+0000) 2019-10-13T13:50:32.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:32.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:32.585193+0000) 2019-10-13T13:50:33.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:33.318+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.321556+0000) 2019-10-13T13:50:33.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:33.320+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.322460+0000) 2019-10-13T13:50:33.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:33.318+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.322022+0000) 2019-10-13T13:50:33.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:33.318+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.322637+0000) 2019-10-13T13:50:33.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:33.322+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.324279+0000) 2019-10-13T13:50:33.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:33.322+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.324456+0000) 2019-10-13T13:50:33.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:33.322+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.325109+0000) 2019-10-13T13:50:33.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:33.324+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.329950+0000) 2019-10-13T13:50:33.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:33.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:33.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:33.872+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:33.873077+0000) 2019-10-13T13:50:35.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:35.606+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.610793+0000) 2019-10-13T13:50:35.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:35.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.897234+0000) 2019-10-13T13:50:36.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:36.574+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:36.576945+0000) 2019-10-13T13:50:38.337 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:38.330+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.334195+0000) 2019-10-13T13:50:38.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:38.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.335346+0000) 2019-10-13T13:50:38.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:38.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.335670+0000) 2019-10-13T13:50:38.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:38.332+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.336256+0000) 2019-10-13T13:50:38.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:38.334+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.336612+0000) 2019-10-13T13:50:38.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:38.334+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.337003+0000) 2019-10-13T13:50:38.340 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:38.334+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.337292+0000) 2019-10-13T13:50:38.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:38.340+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.344995+0000) 2019-10-13T13:50:38.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:38.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:38.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:38.842+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.844448+0000) 2019-10-13T13:50:38.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:38.842+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.844551+0000) 2019-10-13T13:50:40.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:40.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.077814+0000) 2019-10-13T13:50:42.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:42.582+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:42.585392+0000) 2019-10-13T13:50:43.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:43.346+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.348725+0000) 2019-10-13T13:50:43.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:43.346+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.349671+0000) 2019-10-13T13:50:43.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:43.346+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.350395+0000) 2019-10-13T13:50:43.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:43.348+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.351111+0000) 2019-10-13T13:50:43.353 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:43.346+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.351100+0000) 2019-10-13T13:50:43.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:43.350+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.351529+0000) 2019-10-13T13:50:43.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:43.350+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.351779+0000) 2019-10-13T13:50:43.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:43.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:43.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:43.356+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.359117+0000) 2019-10-13T13:50:43.882 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:43.876+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:43.879617+0000) 2019-10-13T13:50:45.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:45.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.610950+0000) 2019-10-13T13:50:45.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:45.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.897433+0000) 2019-10-13T13:50:46.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:46.574+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:46.577161+0000) 2019-10-13T13:50:47.107 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:50:47.107 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:47.110 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:50:47.110 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:48.364 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:48.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.361989+0000) 2019-10-13T13:50:48.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:48.358+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.362364+0000) 2019-10-13T13:50:48.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:48.358+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.362819+0000) 2019-10-13T13:50:48.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:48.358+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.363208+0000) 2019-10-13T13:50:48.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:48.362+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.364143+0000) 2019-10-13T13:50:48.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:48.362+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.363705+0000) 2019-10-13T13:50:48.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:48.362+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.364967+0000) 2019-10-13T13:50:48.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:48.368+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.370609+0000) 2019-10-13T13:50:48.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:48.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:48.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:48.842+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.844623+0000) 2019-10-13T13:50:48.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:48.842+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.844712+0000) 2019-10-13T13:50:50.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:50.072+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.078033+0000) 2019-10-13T13:50:52.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:52.582+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:52.585587+0000) 2019-10-13T13:50:53.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:53.372+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.375331+0000) 2019-10-13T13:50:53.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:53.374+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.375729+0000) 2019-10-13T13:50:53.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:53.374+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.375744+0000) 2019-10-13T13:50:53.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:53.374+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.376041+0000) 2019-10-13T13:50:53.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:53.374+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.376901+0000) 2019-10-13T13:50:53.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:53.374+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.376693+0000) 2019-10-13T13:50:53.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:53.374+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.378085+0000) 2019-10-13T13:50:53.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:53.376+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.380664+0000) 2019-10-13T13:50:53.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:53.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:53.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:53.884+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:53.886187+0000) 2019-10-13T13:50:55.613 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:55.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.611141+0000) 2019-10-13T13:50:55.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:55.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.897646+0000) 2019-10-13T13:50:56.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:56.574+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:56.577356+0000) 2019-10-13T13:50:58.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:58.386+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.387625+0000) 2019-10-13T13:50:58.391 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:50:58.384+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.388858+0000) 2019-10-13T13:50:58.391 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:50:58.386+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.388563+0000) 2019-10-13T13:50:58.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:50:58.386+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.389175+0000) 2019-10-13T13:50:58.392 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:50:58.386+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.389887+0000) 2019-10-13T13:50:58.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:50:58.386+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.390215+0000) 2019-10-13T13:50:58.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:58.386+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.390972+0000) 2019-10-13T13:50:58.396 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:50:58.392+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.394448+0000) 2019-10-13T13:50:58.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:50:58.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:50:58.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:50:58.842+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.844834+0000) 2019-10-13T13:50:58.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:50:58.842+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.844869+0000) 2019-10-13T13:51:00.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:00.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.078246+0000) 2019-10-13T13:51:02.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:02.582+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:02.585801+0000) 2019-10-13T13:51:03.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:03.398+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.400132+0000) 2019-10-13T13:51:03.403 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:03.396+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.400762+0000) 2019-10-13T13:51:03.403 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:03.398+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.400870+0000) 2019-10-13T13:51:03.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:03.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.401374+0000) 2019-10-13T13:51:03.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:03.398+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.401757+0000) 2019-10-13T13:51:03.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:03.398+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.402715+0000) 2019-10-13T13:51:03.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:03.402+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.404513+0000) 2019-10-13T13:51:03.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:03.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181854 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:03.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:03.408+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.410890+0000) 2019-10-13T13:51:03.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:03.892+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:03.892149+0000) 2019-10-13T13:51:05.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:05.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.611341+0000) 2019-10-13T13:51:05.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:05.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.897873+0000) 2019-10-13T13:51:06.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:06.574+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:06.577575+0000) 2019-10-13T13:51:08.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:08.410+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.413411+0000) 2019-10-13T13:51:08.416 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:08.410+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.413864+0000) 2019-10-13T13:51:08.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:08.410+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.414405+0000) 2019-10-13T13:51:08.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:08.412+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.415216+0000) 2019-10-13T13:51:08.419 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:08.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.416085+0000) 2019-10-13T13:51:08.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:08.414+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.416223+0000) 2019-10-13T13:51:08.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:08.414+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.416494+0000) 2019-10-13T13:51:08.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:08.416+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.419682+0000) 2019-10-13T13:51:08.424 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:08.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:08.847 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:08.842+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.845031+0000) 2019-10-13T13:51:08.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:08.842+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.845031+0000) 2019-10-13T13:51:10.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:10.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.078442+0000) 2019-10-13T13:51:12.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:12.582+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:12.585999+0000) 2019-10-13T13:51:13.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:13.422+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.424907+0000) 2019-10-13T13:51:13.428 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:13.422+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.425445+0000) 2019-10-13T13:51:13.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:13.422+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.425985+0000) 2019-10-13T13:51:13.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:13.422+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.426443+0000) 2019-10-13T13:51:13.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:13.422+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.427172+0000) 2019-10-13T13:51:13.431 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:13.423+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.428493+0000) 2019-10-13T13:51:13.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:13.426+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.427925+0000) 2019-10-13T13:51:13.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:13.427+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.431814+0000) 2019-10-13T13:51:13.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:13.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181920 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:13.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:13.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:13.898682+0000) 2019-10-13T13:51:15.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:15.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.611539+0000) 2019-10-13T13:51:15.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:15.894+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.898098+0000) 2019-10-13T13:51:16.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:16.573+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:16.577804+0000) 2019-10-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:18.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:18.433+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.437611+0000) 2019-10-13T13:51:18.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:18.433+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.438580+0000) 2019-10-13T13:51:18.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:18.437+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.439649+0000) 2019-10-13T13:51:18.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:18.437+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.439765+0000) 2019-10-13T13:51:18.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:18.437+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.440199+0000) 2019-10-13T13:51:18.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:18.435+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.440944+0000) 2019-10-13T13:51:18.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:18.437+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.440969+0000) 2019-10-13T13:51:18.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:18.439+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.444479+0000) 2019-10-13T13:51:18.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:18.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:18.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:18.841+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.845223+0000) 2019-10-13T13:51:18.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:18.841+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.845223+0000) 2019-10-13T13:51:20.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:20.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.078635+0000) 2019-10-13T13:51:22.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:22.585+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:22.586202+0000) 2019-10-13T13:51:23.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:23.445+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.449137+0000) 2019-10-13T13:51:23.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:23.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.449285+0000) 2019-10-13T13:51:23.452 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:23.445+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.449721+0000) 2019-10-13T13:51:23.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:23.449+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.451401+0000) 2019-10-13T13:51:23.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:23.449+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.451792+0000) 2019-10-13T13:51:23.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:23.449+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.452427+0000) 2019-10-13T13:51:23.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:23.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.453061+0000) 2019-10-13T13:51:23.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:23.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 181986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:23.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:23.459+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.462389+0000) 2019-10-13T13:51:23.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:23.903+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:23.905275+0000) 2019-10-13T13:51:25.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:25.609+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.611735+0000) 2019-10-13T13:51:25.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:25.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.898331+0000) 2019-10-13T13:51:26.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:26.573+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:26.578022+0000) 2019-10-13T13:51:28.465 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:28.457+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.462319+0000) 2019-10-13T13:51:28.465 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:28.457+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.462458+0000) 2019-10-13T13:51:28.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:28.457+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.462908+0000) 2019-10-13T13:51:28.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:28.461+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.463427+0000) 2019-10-13T13:51:28.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:28.461+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.463749+0000) 2019-10-13T13:51:28.467 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:28.461+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.464807+0000) 2019-10-13T13:51:28.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:28.459+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.465515+0000) 2019-10-13T13:51:28.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:28.463+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.469085+0000) 2019-10-13T13:51:28.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:28.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182019 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:28.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:28.841+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.845453+0000) 2019-10-13T13:51:28.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:28.841+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.845453+0000) 2019-10-13T13:51:30.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:30.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.078835+0000) 2019-10-13T13:51:32.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:32.581+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:32.586424+0000) 2019-10-13T13:51:33.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:33.469+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.473541+0000) 2019-10-13T13:51:33.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:33.469+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.474043+0000) 2019-10-13T13:51:33.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:33.469+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.475231+0000) 2019-10-13T13:51:33.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:33.473+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.475899+0000) 2019-10-13T13:51:33.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:33.473+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.476435+0000) 2019-10-13T13:51:33.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:33.473+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.476913+0000) 2019-10-13T13:51:33.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:33.475+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.478518+0000) 2019-10-13T13:51:33.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:33.475+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.481410+0000) 2019-10-13T13:51:33.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:33.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:33.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:33.911+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:33.911609+0000) 2019-10-13T13:51:35.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:35.609+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.611943+0000) 2019-10-13T13:51:35.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:35.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.898550+0000) 2019-10-13T13:51:36.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:36.577+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:36.578219+0000) 2019-10-13T13:51:38.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:38.481+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.486315+0000) 2019-10-13T13:51:38.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:38.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.486761+0000) 2019-10-13T13:51:38.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:38.485+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.487684+0000) 2019-10-13T13:51:38.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:38.483+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.488338+0000) 2019-10-13T13:51:38.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:38.485+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.489059+0000) 2019-10-13T13:51:38.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:38.485+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.489353+0000) 2019-10-13T13:51:38.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:38.485+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.489727+0000) 2019-10-13T13:51:38.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:38.487+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.493980+0000) 2019-10-13T13:51:38.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:38.493+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:38.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:38.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.845653+0000) 2019-10-13T13:51:38.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:38.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.845653+0000) 2019-10-13T13:51:40.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:40.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.079031+0000) 2019-10-13T13:51:42.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:42.585+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:42.586624+0000) 2019-10-13T13:51:43.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:43.493+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.498949+0000) 2019-10-13T13:51:43.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:43.497+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.499901+0000) 2019-10-13T13:51:43.503 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:43.497+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.500410+0000) 2019-10-13T13:51:43.504 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:43.495+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.501710+0000) 2019-10-13T13:51:43.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:43.497+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.501167+0000) 2019-10-13T13:51:43.505 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:43.497+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.502446+0000) 2019-10-13T13:51:43.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:43.497+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.503003+0000) 2019-10-13T13:51:43.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:43.503+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.507260+0000) 2019-10-13T13:51:43.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:43.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:43.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:43.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:43.918239+0000) 2019-10-13T13:51:45.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:45.609+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.612180+0000) 2019-10-13T13:51:45.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:45.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.898702+0000) 2019-10-13T13:51:46.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:46.573+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:46.578421+0000) 2019-10-13T13:51:47.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:51:47.264 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:47.267 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:51:47.267 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:48.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:48.509+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.512063+0000) 2019-10-13T13:51:48.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:48.509+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.512729+0000) 2019-10-13T13:51:48.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:48.511+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.514840+0000) 2019-10-13T13:51:48.520 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:48.509+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.513249+0000) 2019-10-13T13:51:48.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:48.509+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.513687+0000) 2019-10-13T13:51:48.521 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:48.509+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.514301+0000) 2019-10-13T13:51:48.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:48.509+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.514967+0000) 2019-10-13T13:51:48.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:48.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:48.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:48.519+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.522207+0000) 2019-10-13T13:51:48.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:48.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.845855+0000) 2019-10-13T13:51:48.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:48.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.845855+0000) 2019-10-13T13:51:50.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:50.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.079226+0000) 2019-10-13T13:51:52.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:52.581+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:52.586832+0000) 2019-10-13T13:51:53.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:53.521+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.524412+0000) 2019-10-13T13:51:53.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:53.521+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.525332+0000) 2019-10-13T13:51:53.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:53.521+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.525749+0000) 2019-10-13T13:51:53.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:53.521+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.526558+0000) 2019-10-13T13:51:53.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:53.521+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.527142+0000) 2019-10-13T13:51:53.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:53.523+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.528031+0000) 2019-10-13T13:51:53.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:53.525+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.528243+0000) 2019-10-13T13:51:53.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:53.527+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.531901+0000) 2019-10-13T13:51:53.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:53.529+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:53.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:53.919+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:53.924905+0000) 2019-10-13T13:51:55.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:55.609+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.612377+0000) 2019-10-13T13:51:55.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:55.893+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.898895+0000) 2019-10-13T13:51:56.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:56.573+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:56.578614+0000) 2019-10-13T13:51:58.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:51:58.533+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.537231+0000) 2019-10-13T13:51:58.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:51:58.531+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.537960+0000) 2019-10-13T13:51:58.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:58.533+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.538852+0000) 2019-10-13T13:51:58.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:51:58.533+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.539168+0000) 2019-10-13T13:51:58.542 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:51:58.537+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.539858+0000) 2019-10-13T13:51:58.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:51:58.537+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.540065+0000) 2019-10-13T13:51:58.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:58.537+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.540784+0000) 2019-10-13T13:51:58.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:51:58.539+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.545183+0000) 2019-10-13T13:51:58.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:51:58.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:51:58.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:51:58.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.846049+0000) 2019-10-13T13:51:58.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:51:58.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.846049+0000) 2019-10-13T13:52:00.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:00.075+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.079444+0000) 2019-10-13T13:52:02.589 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:02.580+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:02.587077+0000) 2019-10-13T13:52:03.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:03.544+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.550312+0000) 2019-10-13T13:52:03.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:03.544+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.550692+0000) 2019-10-13T13:52:03.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:03.544+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.551083+0000) 2019-10-13T13:52:03.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:03.548+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.551921+0000) 2019-10-13T13:52:03.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:03.548+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.552287+0000) 2019-10-13T13:52:03.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:03.546+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.552927+0000) 2019-10-13T13:52:03.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:03.548+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.553184+0000) 2019-10-13T13:52:03.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:03.550+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.557961+0000) 2019-10-13T13:52:03.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:03.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:03.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:03.930+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:03.931378+0000) 2019-10-13T13:52:05.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:05.608+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.612582+0000) 2019-10-13T13:52:05.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:05.892+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.899104+0000) 2019-10-13T13:52:06.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:06.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:06.578801+0000) 2019-10-13T13:52:08.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:08.558+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.564406+0000) 2019-10-13T13:52:08.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:08.560+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.564479+0000) 2019-10-13T13:52:08.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:08.560+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.565281+0000) 2019-10-13T13:52:08.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:08.560+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.565655+0000) 2019-10-13T13:52:08.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:08.560+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.566101+0000) 2019-10-13T13:52:08.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:08.560+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.566966+0000) 2019-10-13T13:52:08.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:08.560+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.567205+0000) 2019-10-13T13:52:08.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:08.566+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.572271+0000) 2019-10-13T13:52:08.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:08.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:08.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:08.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.846241+0000) 2019-10-13T13:52:08.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:08.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.846241+0000) 2019-10-13T13:52:10.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:10.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.079644+0000) 2019-10-13T13:52:12.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:12.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:12.587281+0000) 2019-10-13T13:52:13.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:13.572+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.575744+0000) 2019-10-13T13:52:13.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:13.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.576019+0000) 2019-10-13T13:52:13.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:13.572+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.576706+0000) 2019-10-13T13:52:13.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:13.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.577667+0000) 2019-10-13T13:52:13.580 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:13.572+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.578048+0000) 2019-10-13T13:52:13.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:13.572+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.578698+0000) 2019-10-13T13:52:13.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:13.576+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.579960+0000) 2019-10-13T13:52:13.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:13.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:13.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:13.578+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.586067+0000) 2019-10-13T13:52:13.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:13.934+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:13.937897+0000) 2019-10-13T13:52:15.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:15.608+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.612800+0000) 2019-10-13T13:52:15.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:15.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.899326+0000) 2019-10-13T13:52:16.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:16.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:16.579032+0000) 2019-10-13T13:52:17.336 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:52:17.337 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:17.340 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:52:17.341 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:18.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:18.584+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.588091+0000) 2019-10-13T13:52:18.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:18.584+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.588324+0000) 2019-10-13T13:52:18.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:18.584+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.589416+0000) 2019-10-13T13:52:18.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:18.584+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.590059+0000) 2019-10-13T13:52:18.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:18.584+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.590619+0000) 2019-10-13T13:52:18.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:18.586+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.591188+0000) 2019-10-13T13:52:18.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:18.588+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.591812+0000) 2019-10-13T13:52:18.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:18.590+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.595251+0000) 2019-10-13T13:52:18.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:18.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:18.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:18.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.846438+0000) 2019-10-13T13:52:18.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:18.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.846438+0000) 2019-10-13T13:52:20.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:20.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.079868+0000) 2019-10-13T13:52:22.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:22.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:22.587551+0000) 2019-10-13T13:52:23.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:23.596+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.600460+0000) 2019-10-13T13:52:23.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:23.596+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.601282+0000) 2019-10-13T13:52:23.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:23.596+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.601599+0000) 2019-10-13T13:52:23.604 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:23.598+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.602256+0000) 2019-10-13T13:52:23.605 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:23.596+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.602516+0000) 2019-10-13T13:52:23.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:23.596+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.602696+0000) 2019-10-13T13:52:23.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:23.600+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.603522+0000) 2019-10-13T13:52:23.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:23.602+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.609151+0000) 2019-10-13T13:52:23.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:23.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:23.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:23.942+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:23.944421+0000) 2019-10-13T13:52:25.615 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:25.612+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.613016+0000) 2019-10-13T13:52:25.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:25.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.899558+0000) 2019-10-13T13:52:26.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:26.572+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:26.579259+0000) 2019-10-13T13:52:28.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:28.608+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.613048+0000) 2019-10-13T13:52:28.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:28.608+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.613396+0000) 2019-10-13T13:52:28.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:28.608+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.613669+0000) 2019-10-13T13:52:28.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:28.608+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.614397+0000) 2019-10-13T13:52:28.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:28.608+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.615033+0000) 2019-10-13T13:52:28.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:28.608+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.615208+0000) 2019-10-13T13:52:28.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:28.610+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.615783+0000) 2019-10-13T13:52:28.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:28.616+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182415 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:28.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:28.618+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.622798+0000) 2019-10-13T13:52:28.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:28.844+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.846639+0000) 2019-10-13T13:52:28.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:28.844+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.846639+0000) 2019-10-13T13:52:30.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:30.074+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.080063+0000) 2019-10-13T13:52:32.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:32.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:32.587752+0000) 2019-10-13T13:52:33.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:33.620+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.625204+0000) 2019-10-13T13:52:33.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:33.620+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.625575+0000) 2019-10-13T13:52:33.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:33.620+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.626090+0000) 2019-10-13T13:52:33.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:33.624+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.627919+0000) 2019-10-13T13:52:33.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:33.624+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.628478+0000) 2019-10-13T13:52:33.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:33.624+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.628700+0000) 2019-10-13T13:52:33.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:33.622+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.629550+0000) 2019-10-13T13:52:33.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:33.626+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.633963+0000) 2019-10-13T13:52:33.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:33.628+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182449 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:33.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:33.950+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:33.950686+0000) 2019-10-13T13:52:35.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:35.612+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.613250+0000) 2019-10-13T13:52:35.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:35.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.899764+0000) 2019-10-13T13:52:36.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:36.576+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:36.579491+0000) 2019-10-13T13:52:38.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:38.632+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.637870+0000) 2019-10-13T13:52:38.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:38.632+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.638512+0000) 2019-10-13T13:52:38.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:38.632+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.639055+0000) 2019-10-13T13:52:38.642 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:38.636+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.639584+0000) 2019-10-13T13:52:38.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:38.634+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.640616+0000) 2019-10-13T13:52:38.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:38.636+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.640755+0000) 2019-10-13T13:52:38.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:38.636+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.641162+0000) 2019-10-13T13:52:38.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:38.642+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.646723+0000) 2019-10-13T13:52:38.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:38.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:38.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:38.840+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.846835+0000) 2019-10-13T13:52:38.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:38.840+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.846835+0000) 2019-10-13T13:52:40.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:40.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.080261+0000) 2019-10-13T13:52:42.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:42.583+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:42.587988+0000) 2019-10-13T13:52:43.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:43.643+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.649469+0000) 2019-10-13T13:52:43.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:43.643+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.650364+0000) 2019-10-13T13:52:43.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:43.643+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.650861+0000) 2019-10-13T13:52:43.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:43.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.652171+0000) 2019-10-13T13:52:43.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:43.647+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.652620+0000) 2019-10-13T13:52:43.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:43.647+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.652792+0000) 2019-10-13T13:52:43.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:43.647+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.653032+0000) 2019-10-13T13:52:43.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:43.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.658294+0000) 2019-10-13T13:52:43.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:43.651+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:43.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:43.954+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:43.957487+0000) 2019-10-13T13:52:45.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:45.607+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.613474+0000) 2019-10-13T13:52:45.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:45.895+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.899963+0000) 2019-10-13T13:52:46.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:46.575+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:46.579704+0000) 2019-10-13T13:52:47.394 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:52:47.394 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:47.398 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:52:47.398 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:48.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:48.655+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.661626+0000) 2019-10-13T13:52:48.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:48.655+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.662195+0000) 2019-10-13T13:52:48.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:48.655+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.662926+0000) 2019-10-13T13:52:48.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:48.659+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.663568+0000) 2019-10-13T13:52:48.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:48.659+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.663899+0000) 2019-10-13T13:52:48.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:48.659+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.664686+0000) 2019-10-13T13:52:48.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:48.658+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.665244+0000) 2019-10-13T13:52:48.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:48.662+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.669181+0000) 2019-10-13T13:52:48.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:48.663+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:48.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:48.843+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.847033+0000) 2019-10-13T13:52:48.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:48.843+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.847033+0000) 2019-10-13T13:52:50.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:50.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.080454+0000) 2019-10-13T13:52:52.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:52.587+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:52.588218+0000) 2019-10-13T13:52:53.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:53.667+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.673526+0000) 2019-10-13T13:52:53.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:53.667+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.674205+0000) 2019-10-13T13:52:53.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:53.667+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.674795+0000) 2019-10-13T13:52:53.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:53.667+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.674996+0000) 2019-10-13T13:52:53.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:53.667+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.675252+0000) 2019-10-13T13:52:53.678 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:53.671+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.675861+0000) 2019-10-13T13:52:53.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:53.674+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.677881+0000) 2019-10-13T13:52:53.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:53.682+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.683195+0000) 2019-10-13T13:52:53.686 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:53.679+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:53.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:53.962+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:53.964492+0000) 2019-10-13T13:52:55.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:55.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.613621+0000) 2019-10-13T13:52:55.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:55.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.900183+0000) 2019-10-13T13:52:56.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:56.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:56.579895+0000) 2019-10-13T13:52:58.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:52:58.683+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.687401+0000) 2019-10-13T13:52:58.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:52:58.683+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.688364+0000) 2019-10-13T13:52:58.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:52:58.683+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.689348+0000) 2019-10-13T13:52:58.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:58.683+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.690103+0000) 2019-10-13T13:52:58.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:58.683+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.690557+0000) 2019-10-13T13:52:58.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:52:58.689+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.691970+0000) 2019-10-13T13:52:58.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:52:58.687+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.692036+0000) 2019-10-13T13:52:58.697 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:52:58.693+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.695374+0000) 2019-10-13T13:52:58.699 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:52:58.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182614 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:52:58.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:52:58.839+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.847230+0000) 2019-10-13T13:52:58.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:52:58.839+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.847230+0000) 2019-10-13T13:53:00.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:00.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.080680+0000) 2019-10-13T13:53:02.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:02.587+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:02.588423+0000) 2019-10-13T13:53:03.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:03.695+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.699710+0000) 2019-10-13T13:53:03.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:03.697+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.700502+0000) 2019-10-13T13:53:03.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:03.695+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.700604+0000) 2019-10-13T13:53:03.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:03.695+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.700959+0000) 2019-10-13T13:53:03.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:03.695+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.701844+0000) 2019-10-13T13:53:03.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:03.695+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.702377+0000) 2019-10-13T13:53:03.706 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:03.699+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.703923+0000) 2019-10-13T13:53:03.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:03.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:03.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:03.705+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.710038+0000) 2019-10-13T13:53:03.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:03.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:03.971003+0000) 2019-10-13T13:53:05.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:05.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.613855+0000) 2019-10-13T13:53:05.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:05.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.900378+0000) 2019-10-13T13:53:06.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:06.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:06.580092+0000) 2019-10-13T13:53:08.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:08.707+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.712504+0000) 2019-10-13T13:53:08.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:08.707+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.713671+0000) 2019-10-13T13:53:08.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:08.707+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.713895+0000) 2019-10-13T13:53:08.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:08.707+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.714661+0000) 2019-10-13T13:53:08.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:08.713+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.715359+0000) 2019-10-13T13:53:08.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:08.711+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.715602+0000) 2019-10-13T13:53:08.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:08.711+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.716273+0000) 2019-10-13T13:53:08.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:08.717+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.720887+0000) 2019-10-13T13:53:08.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:08.715+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:08.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:08.843+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.847384+0000) 2019-10-13T13:53:08.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:08.843+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.847384+0000) 2019-10-13T13:53:10.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:10.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.080880+0000) 2019-10-13T13:53:12.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:12.587+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:12.588666+0000) 2019-10-13T13:53:13.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:13.723+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.724725+0000) 2019-10-13T13:53:13.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:13.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.725338+0000) 2019-10-13T13:53:13.729 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:13.723+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.725658+0000) 2019-10-13T13:53:13.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:13.723+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.725968+0000) 2019-10-13T13:53:13.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:13.725+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.727827+0000) 2019-10-13T13:53:13.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:13.727+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.728056+0000) 2019-10-13T13:53:13.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:13.727+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.728782+0000) 2019-10-13T13:53:13.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:13.729+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.733470+0000) 2019-10-13T13:53:13.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:13.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:13.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:13.973+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:13.977028+0000) 2019-10-13T13:53:15.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:15.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.614058+0000) 2019-10-13T13:53:15.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:15.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.900617+0000) 2019-10-13T13:53:16.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:16.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:16.580318+0000) 2019-10-13T13:53:17.533 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:53:17.533 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:17.537 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:53:17.537 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:18.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:18.735+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.737390+0000) 2019-10-13T13:53:18.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:18.733+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.737911+0000) 2019-10-13T13:53:18.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:18.735+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.738034+0000) 2019-10-13T13:53:18.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:18.735+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.738513+0000) 2019-10-13T13:53:18.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:18.735+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.739056+0000) 2019-10-13T13:53:18.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:18.739+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.739842+0000) 2019-10-13T13:53:18.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:18.739+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.740266+0000) 2019-10-13T13:53:18.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:18.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:18.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:18.745+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.747725+0000) 2019-10-13T13:53:18.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:18.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.847596+0000) 2019-10-13T13:53:18.850 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:18.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.847596+0000) 2019-10-13T13:53:20.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:20.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.081088+0000) 2019-10-13T13:53:22.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:22.587+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:22.588885+0000) 2019-10-13T13:53:23.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:23.747+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.750730+0000) 2019-10-13T13:53:23.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:23.747+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.750981+0000) 2019-10-13T13:53:23.754 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:23.751+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.752250+0000) 2019-10-13T13:53:23.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:23.749+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.752931+0000) 2019-10-13T13:53:23.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:23.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.753026+0000) 2019-10-13T13:53:23.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:23.751+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.753845+0000) 2019-10-13T13:53:23.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:23.751+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.754641+0000) 2019-10-13T13:53:23.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:23.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:23.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:23.761+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.762345+0000) 2019-10-13T13:53:23.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:23.981+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:23.983460+0000) 2019-10-13T13:53:25.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:25.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.614283+0000) 2019-10-13T13:53:25.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:25.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.900849+0000) 2019-10-13T13:53:26.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:26.578+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:26.580533+0000) 2019-10-13T13:53:28.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:28.762+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.764362+0000) 2019-10-13T13:53:28.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:28.762+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.765378+0000) 2019-10-13T13:53:28.768 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:28.765+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.766486+0000) 2019-10-13T13:53:28.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:28.762+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.766563+0000) 2019-10-13T13:53:28.769 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:28.762+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.766964+0000) 2019-10-13T13:53:28.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:28.766+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.767849+0000) 2019-10-13T13:53:28.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:28.766+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.768235+0000) 2019-10-13T13:53:28.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:28.769+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.771315+0000) 2019-10-13T13:53:28.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:28.770+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:28.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:28.846+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.847797+0000) 2019-10-13T13:53:28.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:28.846+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.847797+0000) 2019-10-13T13:53:30.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:30.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.081301+0000) 2019-10-13T13:53:32.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:32.586+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:32.589091+0000) 2019-10-13T13:53:33.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:33.773+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.777411+0000) 2019-10-13T13:53:33.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:33.774+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.777845+0000) 2019-10-13T13:53:33.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:33.774+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.777985+0000) 2019-10-13T13:53:33.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:33.774+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.778379+0000) 2019-10-13T13:53:33.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:33.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.778799+0000) 2019-10-13T13:53:33.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:33.778+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.780644+0000) 2019-10-13T13:53:33.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:33.778+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.781074+0000) 2019-10-13T13:53:33.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:33.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182843 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:33.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:33.785+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.786870+0000) 2019-10-13T13:53:33.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:33.993+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:33.995115+0000) 2019-10-13T13:53:35.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:35.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.614500+0000) 2019-10-13T13:53:35.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:35.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.901052+0000) 2019-10-13T13:53:36.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:36.578+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:36.580728+0000) 2019-10-13T13:53:38.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:38.786+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.788587+0000) 2019-10-13T13:53:38.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:38.786+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.789479+0000) 2019-10-13T13:53:38.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:38.785+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.790026+0000) 2019-10-13T13:53:38.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:38.786+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.790323+0000) 2019-10-13T13:53:38.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:38.786+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.791047+0000) 2019-10-13T13:53:38.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:38.790+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.791902+0000) 2019-10-13T13:53:38.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:38.790+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.792530+0000) 2019-10-13T13:53:38.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:38.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182878 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:38.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:38.797+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.799591+0000) 2019-10-13T13:53:38.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:38.846+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.847996+0000) 2019-10-13T13:53:38.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:38.846+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.847996+0000) 2019-10-13T13:53:40.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:40.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.081498+0000) 2019-10-13T13:53:42.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:42.586+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:42.589286+0000) 2019-10-13T13:53:43.803 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:43.798+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.800567+0000) 2019-10-13T13:53:43.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:43.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.801929+0000) 2019-10-13T13:53:43.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:43.798+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.802188+0000) 2019-10-13T13:53:43.805 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:43.801+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.802853+0000) 2019-10-13T13:53:43.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:43.802+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.803419+0000) 2019-10-13T13:53:43.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:43.802+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.803853+0000) 2019-10-13T13:53:43.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:43.802+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.804096+0000) 2019-10-13T13:53:43.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:43.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:43.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:43.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:43.809866+0000) 2019-10-13T13:53:44.004 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:43.997+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:44.001717+0000) 2019-10-13T13:53:45.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:45.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.614697+0000) 2019-10-13T13:53:45.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:45.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.901260+0000) 2019-10-13T13:53:46.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:46.578+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:46.580924+0000) 2019-10-13T13:53:47.639 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:53:47.640 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:47.643 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:53:47.643 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:48.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:48.810+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.811902+0000) 2019-10-13T13:53:48.815 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:48.809+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.812984+0000) 2019-10-13T13:53:48.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:48.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.812500+0000) 2019-10-13T13:53:48.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:48.810+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.813554+0000) 2019-10-13T13:53:48.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:48.810+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.813604+0000) 2019-10-13T13:53:48.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:48.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.814239+0000) 2019-10-13T13:53:48.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:48.810+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.814721+0000) 2019-10-13T13:53:48.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:48.817+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.819687+0000) 2019-10-13T13:53:48.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:48.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182944 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:48.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:48.846+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.848225+0000) 2019-10-13T13:53:48.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:48.846+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.848225+0000) 2019-10-13T13:53:50.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:50.077+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.081692+0000) 2019-10-13T13:53:52.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:52.586+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:52.589486+0000) 2019-10-13T13:53:53.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:53.822+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.824527+0000) 2019-10-13T13:53:53.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:53.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.825242+0000) 2019-10-13T13:53:53.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:53.822+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.825795+0000) 2019-10-13T13:53:53.829 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:53.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.826385+0000) 2019-10-13T13:53:53.829 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:53.824+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.827132+0000) 2019-10-13T13:53:53.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:53.826+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.827344+0000) 2019-10-13T13:53:53.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:53.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.828309+0000) 2019-10-13T13:53:53.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:53.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 182975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:53.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:53.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:53.834524+0000) 2019-10-13T13:53:54.010 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:54.004+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:54.007835+0000) 2019-10-13T13:53:55.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:55.610+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.614928+0000) 2019-10-13T13:53:55.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:55.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.901488+0000) 2019-10-13T13:53:56.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:56.578+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:56.581123+0000) 2019-10-13T13:53:58.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:58.834+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.837321+0000) 2019-10-13T13:53:58.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:53:58.834+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.838056+0000) 2019-10-13T13:53:58.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:53:58.834+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.838438+0000) 2019-10-13T13:53:58.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:53:58.834+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.839080+0000) 2019-10-13T13:53:58.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:53:58.836+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.839740+0000) 2019-10-13T13:53:58.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:58.838+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.839922+0000) 2019-10-13T13:53:58.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:53:58.838+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.840261+0000) 2019-10-13T13:53:58.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:53:58.840+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.844128+0000) 2019-10-13T13:53:58.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:53:58.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:53:58.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:53:58.846+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.848366+0000) 2019-10-13T13:53:58.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:53:58.846+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.848441+0000) 2019-10-13T13:54:00.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:00.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.081883+0000) 2019-10-13T13:54:02.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:02.586+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:02.589708+0000) 2019-10-13T13:54:03.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:03.846+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.848903+0000) 2019-10-13T13:54:03.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:03.846+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.850214+0000) 2019-10-13T13:54:03.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:03.846+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.850842+0000) 2019-10-13T13:54:03.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:03.848+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.851396+0000) 2019-10-13T13:54:03.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:03.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.851724+0000) 2019-10-13T13:54:03.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:03.850+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.852018+0000) 2019-10-13T13:54:03.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:03.850+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.852770+0000) 2019-10-13T13:54:03.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:03.852+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:03.856507+0000) 2019-10-13T13:54:03.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:03.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:04.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:04.012+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:04.014373+0000) 2019-10-13T13:54:05.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:05.614+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.615128+0000) 2019-10-13T13:54:05.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:05.898+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.901726+0000) 2019-10-13T13:54:06.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:06.578+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:06.581316+0000) 2019-10-13T13:54:08.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:08.846+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.848541+0000) 2019-10-13T13:54:08.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:08.846+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.848720+0000) 2019-10-13T13:54:08.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:08.857+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.862282+0000) 2019-10-13T13:54:08.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:08.857+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.862915+0000) 2019-10-13T13:54:08.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:08.860+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.863629+0000) 2019-10-13T13:54:08.866 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:08.861+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.863869+0000) 2019-10-13T13:54:08.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:08.861+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.864006+0000) 2019-10-13T13:54:08.867 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:08.861+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.864799+0000) 2019-10-13T13:54:08.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:08.861+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.866426+0000) 2019-10-13T13:54:08.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:08.868+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.871100+0000) 2019-10-13T13:54:08.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:08.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:10.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:10.076+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.082075+0000) 2019-10-13T13:54:12.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:12.585+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:12.589907+0000) 2019-10-13T13:54:13.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:13.869+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.874993+0000) 2019-10-13T13:54:13.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:13.873+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.876271+0000) 2019-10-13T13:54:13.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:13.872+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.876944+0000) 2019-10-13T13:54:13.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:13.873+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.877077+0000) 2019-10-13T13:54:13.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:13.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.877638+0000) 2019-10-13T13:54:13.880 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:13.873+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.877983+0000) 2019-10-13T13:54:13.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:13.873+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.878301+0000) 2019-10-13T13:54:13.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:13.880+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:13.882418+0000) 2019-10-13T13:54:13.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:13.881+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:14.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:14.020+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:14.023837+0000) 2019-10-13T13:54:15.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:15.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.615355+0000) 2019-10-13T13:54:15.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:15.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.901964+0000) 2019-10-13T13:54:16.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:16.577+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:16.581561+0000) 2019-10-13T13:54:17.778 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:54:17.779 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:17.782 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:54:17.782 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:18.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:18.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.848740+0000) 2019-10-13T13:54:18.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:18.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.848920+0000) 2019-10-13T13:54:18.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:18.885+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.887989+0000) 2019-10-13T13:54:18.891 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:18.885+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.888439+0000) 2019-10-13T13:54:18.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:18.885+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.888695+0000) 2019-10-13T13:54:18.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:18.884+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.889478+0000) 2019-10-13T13:54:18.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:18.885+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.889532+0000) 2019-10-13T13:54:18.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:18.885+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.889964+0000) 2019-10-13T13:54:18.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:18.885+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.890553+0000) 2019-10-13T13:54:18.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:18.892+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.895533+0000) 2019-10-13T13:54:18.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:18.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:20.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:20.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.082261+0000) 2019-10-13T13:54:22.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:22.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:22.590114+0000) 2019-10-13T13:54:23.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:23.897+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.900640+0000) 2019-10-13T13:54:23.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:23.897+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.901144+0000) 2019-10-13T13:54:23.904 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:23.897+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.901725+0000) 2019-10-13T13:54:23.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:23.897+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.902152+0000) 2019-10-13T13:54:23.905 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:23.897+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.902409+0000) 2019-10-13T13:54:23.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:23.900+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.903014+0000) 2019-10-13T13:54:23.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:23.901+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.904049+0000) 2019-10-13T13:54:23.911 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:23.904+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:23.909383+0000) 2019-10-13T13:54:23.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:23.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:24.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:24.032+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:24.035554+0000) 2019-10-13T13:54:25.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:25.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.615572+0000) 2019-10-13T13:54:25.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:25.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.902159+0000) 2019-10-13T13:54:26.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:26.581+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:26.581764+0000) 2019-10-13T13:54:28.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:28.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.848937+0000) 2019-10-13T13:54:28.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:28.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.849146+0000) 2019-10-13T13:54:28.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:28.909+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.913211+0000) 2019-10-13T13:54:28.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:28.909+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.913810+0000) 2019-10-13T13:54:28.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:28.909+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.914461+0000) 2019-10-13T13:54:28.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:28.912+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.915125+0000) 2019-10-13T13:54:28.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:28.909+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.915154+0000) 2019-10-13T13:54:28.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:28.913+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.916502+0000) 2019-10-13T13:54:28.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:28.913+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.916282+0000) 2019-10-13T13:54:28.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:28.916+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.922108+0000) 2019-10-13T13:54:28.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:28.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:30.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:30.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.082455+0000) 2019-10-13T13:54:32.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:32.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:32.590381+0000) 2019-10-13T13:54:33.929 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:33.921+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.925997+0000) 2019-10-13T13:54:33.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:33.921+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.926327+0000) 2019-10-13T13:54:33.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:33.921+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.926749+0000) 2019-10-13T13:54:33.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:33.925+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.927379+0000) 2019-10-13T13:54:33.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:33.925+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.928739+0000) 2019-10-13T13:54:33.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:33.925+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.928880+0000) 2019-10-13T13:54:33.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:33.924+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.929454+0000) 2019-10-13T13:54:33.938 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:33.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:33.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:33.936+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:33.939703+0000) 2019-10-13T13:54:34.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:34.040+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:34.041980+0000) 2019-10-13T13:54:35.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:35.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.615760+0000) 2019-10-13T13:54:35.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:35.897+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.902372+0000) 2019-10-13T13:54:36.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:36.577+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:36.581936+0000) 2019-10-13T13:54:38.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:38.845+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.849120+0000) 2019-10-13T13:54:38.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:38.845+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.849378+0000) 2019-10-13T13:54:38.942 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:38.933+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.939149+0000) 2019-10-13T13:54:38.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:38.937+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.940576+0000) 2019-10-13T13:54:38.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:38.937+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.941014+0000) 2019-10-13T13:54:38.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:38.936+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.941618+0000) 2019-10-13T13:54:38.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:38.937+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.942020+0000) 2019-10-13T13:54:38.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:38.937+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.942391+0000) 2019-10-13T13:54:38.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:38.937+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.942993+0000) 2019-10-13T13:54:38.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:38.944+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.949815+0000) 2019-10-13T13:54:38.953 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:38.945+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:40.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:40.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.082652+0000) 2019-10-13T13:54:42.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:42.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:42.590581+0000) 2019-10-13T13:54:43.955 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:43.949+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.952922+0000) 2019-10-13T13:54:43.956 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:43.949+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.953319+0000) 2019-10-13T13:54:43.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:43.949+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.953991+0000) 2019-10-13T13:54:43.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:43.949+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.954066+0000) 2019-10-13T13:54:43.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:43.953+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.955997+0000) 2019-10-13T13:54:43.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:43.952+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.956689+0000) 2019-10-13T13:54:43.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:43.953+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.956593+0000) 2019-10-13T13:54:43.964 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:43.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:43.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:43.960+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:43.963475+0000) 2019-10-13T13:54:44.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:44.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:44.048474+0000) 2019-10-13T13:54:45.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:45.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.615967+0000) 2019-10-13T13:54:45.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:45.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.902642+0000) 2019-10-13T13:54:46.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:46.577+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:46.582124+0000) 2019-10-13T13:54:47.889 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:54:47.889 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:47.893 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:54:47.893 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:48.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:48.849+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.849350+0000) 2019-10-13T13:54:48.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:48.849+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.849560+0000) 2019-10-13T13:54:48.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:48.961+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.965664+0000) 2019-10-13T13:54:48.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:48.961+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.966000+0000) 2019-10-13T13:54:48.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:48.963+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.966710+0000) 2019-10-13T13:54:48.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:48.965+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.967476+0000) 2019-10-13T13:54:48.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:48.965+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.967472+0000) 2019-10-13T13:54:48.970 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:48.965+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.968069+0000) 2019-10-13T13:54:48.972 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:48.965+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.968876+0000) 2019-10-13T13:54:48.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:48.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:48.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:48.971+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.975355+0000) 2019-10-13T13:54:50.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:50.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.082860+0000) 2019-10-13T13:54:52.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:52.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:52.590823+0000) 2019-10-13T13:54:53.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:53.972+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.978246+0000) 2019-10-13T13:54:53.982 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:53.972+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.978905+0000) 2019-10-13T13:54:53.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:53.975+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.980425+0000) 2019-10-13T13:54:53.983 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:53.976+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.980509+0000) 2019-10-13T13:54:53.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:53.976+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.980815+0000) 2019-10-13T13:54:53.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:53.976+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.981195+0000) 2019-10-13T13:54:53.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:53.976+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.981691+0000) 2019-10-13T13:54:53.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:53.983+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:53.986594+0000) 2019-10-13T13:54:53.990 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:53.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:54.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:54.051+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:54.055085+0000) 2019-10-13T13:54:55.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:55.612+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.616168+0000) 2019-10-13T13:54:55.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:55.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.902847+0000) 2019-10-13T13:54:56.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:56.580+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:56.582335+0000) 2019-10-13T13:54:58.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:58.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.849542+0000) 2019-10-13T13:54:58.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:58.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.849742+0000) 2019-10-13T13:54:58.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:54:58.984+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.990756+0000) 2019-10-13T13:54:58.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:54:58.988+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.991640+0000) 2019-10-13T13:54:58.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:54:58.988+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.992062+0000) 2019-10-13T13:54:58.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:54:58.987+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.992672+0000) 2019-10-13T13:54:58.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:54:58.988+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.992714+0000) 2019-10-13T13:54:58.996 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:54:58.988+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.993387+0000) 2019-10-13T13:54:58.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:54:58.988+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.994081+0000) 2019-10-13T13:54:59.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:54:58.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:54:59.004 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:54:58.995+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:59.000189+0000) 2019-10-13T13:55:00.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:00.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.083048+0000) 2019-10-13T13:55:02.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:02.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:02.591046+0000) 2019-10-13T13:55:04.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:04.000+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.003923+0000) 2019-10-13T13:55:04.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:04.000+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.004661+0000) 2019-10-13T13:55:04.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:04.000+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.005403+0000) 2019-10-13T13:55:04.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:04.000+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.006158+0000) 2019-10-13T13:55:04.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:04.000+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.006443+0000) 2019-10-13T13:55:04.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:04.003+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.007054+0000) 2019-10-13T13:55:04.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:04.000+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.007018+0000) 2019-10-13T13:55:04.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:04.007+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.013447+0000) 2019-10-13T13:55:04.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:04.008+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:04.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:04.055+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.061862+0000) 2019-10-13T13:55:05.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:05.612+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.616366+0000) 2019-10-13T13:55:05.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:05.896+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.903065+0000) 2019-10-13T13:55:06.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:06.580+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:06.582530+0000) 2019-10-13T13:55:08.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:08.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:08.849777+0000) 2019-10-13T13:55:08.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:08.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:08.849914+0000) 2019-10-13T13:55:09.020 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:09.012+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.017228+0000) 2019-10-13T13:55:09.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:09.012+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.017714+0000) 2019-10-13T13:55:09.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:09.012+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.018441+0000) 2019-10-13T13:55:09.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:09.012+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.019270+0000) 2019-10-13T13:55:09.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:09.016+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.020085+0000) 2019-10-13T13:55:09.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:09.015+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.020483+0000) 2019-10-13T13:55:09.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:09.016+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.020510+0000) 2019-10-13T13:55:09.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:09.019+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:09.025435+0000) 2019-10-13T13:55:09.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:09.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:10.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:10.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.083198+0000) 2019-10-13T13:55:12.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:12.584+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:12.591257+0000) 2019-10-13T13:55:14.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:14.024+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.029031+0000) 2019-10-13T13:55:14.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:14.024+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.029743+0000) 2019-10-13T13:55:14.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:14.027+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.030898+0000) 2019-10-13T13:55:14.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:14.024+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.030953+0000) 2019-10-13T13:55:14.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:14.028+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.031400+0000) 2019-10-13T13:55:14.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:14.028+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.032241+0000) 2019-10-13T13:55:14.035 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:14.028+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.032547+0000) 2019-10-13T13:55:14.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:14.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:14.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:14.035+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.040206+0000) 2019-10-13T13:55:14.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:14.063+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.067844+0000) 2019-10-13T13:55:15.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:15.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.616554+0000) 2019-10-13T13:55:15.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:15.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.903263+0000) 2019-10-13T13:55:16.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:16.580+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:16.582724+0000) 2019-10-13T13:55:18.049 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:55:18.050 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:18.053 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:55:18.053 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:18.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:18.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:18.850013+0000) 2019-10-13T13:55:18.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:18.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:18.850077+0000) 2019-10-13T13:55:19.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:19.036+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.042223+0000) 2019-10-13T13:55:19.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:19.036+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.042285+0000) 2019-10-13T13:55:19.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:19.036+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.042609+0000) 2019-10-13T13:55:19.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:19.036+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.042973+0000) 2019-10-13T13:55:19.046 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:19.040+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.043969+0000) 2019-10-13T13:55:19.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:19.040+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.044278+0000) 2019-10-13T13:55:19.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:19.039+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.045226+0000) 2019-10-13T13:55:19.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:19.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183537 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:19.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:19.047+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:19.051740+0000) 2019-10-13T13:55:20.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:20.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.083346+0000) 2019-10-13T13:55:22.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:22.588+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:22.591486+0000) 2019-10-13T13:55:24.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:24.048+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.054722+0000) 2019-10-13T13:55:24.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:24.048+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.054592+0000) 2019-10-13T13:55:24.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:24.052+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.055387+0000) 2019-10-13T13:55:24.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:24.052+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.055821+0000) 2019-10-13T13:55:24.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:24.052+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.056732+0000) 2019-10-13T13:55:24.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:24.051+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.057562+0000) 2019-10-13T13:55:24.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:24.052+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.057748+0000) 2019-10-13T13:55:24.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:24.059+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.063086+0000) 2019-10-13T13:55:24.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:24.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:24.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:24.067+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.072961+0000) 2019-10-13T13:55:25.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:25.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.616761+0000) 2019-10-13T13:55:25.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:25.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.903517+0000) 2019-10-13T13:55:26.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:26.580+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:26.582933+0000) 2019-10-13T13:55:28.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:28.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:28.850242+0000) 2019-10-13T13:55:28.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:28.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:28.850231+0000) 2019-10-13T13:55:29.070 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:29.064+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.067399+0000) 2019-10-13T13:55:29.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:29.064+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.067942+0000) 2019-10-13T13:55:29.071 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:29.064+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.068626+0000) 2019-10-13T13:55:29.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:29.064+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.068699+0000) 2019-10-13T13:55:29.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:29.064+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.069067+0000) 2019-10-13T13:55:29.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:29.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.070392+0000) 2019-10-13T13:55:29.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:29.064+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.070451+0000) 2019-10-13T13:55:29.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:29.071+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:29.075929+0000) 2019-10-13T13:55:29.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:29.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:30.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:30.079+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.083519+0000) 2019-10-13T13:55:32.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:32.588+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:32.591681+0000) 2019-10-13T13:55:34.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:34.076+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.078917+0000) 2019-10-13T13:55:34.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:34.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.079611+0000) 2019-10-13T13:55:34.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:34.076+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.080737+0000) 2019-10-13T13:55:34.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:34.075+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.081900+0000) 2019-10-13T13:55:34.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:34.076+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.081246+0000) 2019-10-13T13:55:34.085 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:34.076+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.082017+0000) 2019-10-13T13:55:34.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:34.076+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.082492+0000) 2019-10-13T13:55:34.091 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:34.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:34.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:34.083+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.089998+0000) 2019-10-13T13:55:35.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:35.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.616990+0000) 2019-10-13T13:55:35.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:35.899+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.903727+0000) 2019-10-13T13:55:36.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:36.575+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:36.583123+0000) 2019-10-13T13:55:38.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:38.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:38.850461+0000) 2019-10-13T13:55:38.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:38.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:38.850461+0000) 2019-10-13T13:55:39.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:39.087+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.091271+0000) 2019-10-13T13:55:39.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:39.087+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.092084+0000) 2019-10-13T13:55:39.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:39.087+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.093023+0000) 2019-10-13T13:55:39.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:39.087+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.093596+0000) 2019-10-13T13:55:39.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:39.087+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.094089+0000) 2019-10-13T13:55:39.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:39.087+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.094588+0000) 2019-10-13T13:55:39.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:39.091+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.095385+0000) 2019-10-13T13:55:39.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:39.095+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:39.101485+0000) 2019-10-13T13:55:39.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:39.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:40.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:40.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.083669+0000) 2019-10-13T13:55:40.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:40.827+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.829279+0000) 2019-10-13T13:55:42.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:42.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:42.591884+0000) 2019-10-13T13:55:44.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:44.099+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.104346+0000) 2019-10-13T13:55:44.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:44.099+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.104522+0000) 2019-10-13T13:55:44.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:44.098+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.105378+0000) 2019-10-13T13:55:44.108 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:44.099+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.105838+0000) 2019-10-13T13:55:44.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:44.099+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.106262+0000) 2019-10-13T13:55:44.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:44.099+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.106887+0000) 2019-10-13T13:55:44.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:44.103+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.107586+0000) 2019-10-13T13:55:44.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:44.106+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.113897+0000) 2019-10-13T13:55:44.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:44.107+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:45.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:45.615+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:45.617193+0000) 2019-10-13T13:55:45.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:45.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:45.903924+0000) 2019-10-13T13:55:46.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:46.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.583342+0000) 2019-10-13T13:55:48.283 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:55:48.283 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:48.287 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:55:48.287 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:48.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:48.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:48.850665+0000) 2019-10-13T13:55:48.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:48.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:48.850663+0000) 2019-10-13T13:55:49.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:49.111+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.117571+0000) 2019-10-13T13:55:49.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:49.114+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.118374+0000) 2019-10-13T13:55:49.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:49.111+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.118503+0000) 2019-10-13T13:55:49.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:49.111+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.119148+0000) 2019-10-13T13:55:49.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:49.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.119525+0000) 2019-10-13T13:55:49.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:49.115+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.120603+0000) 2019-10-13T13:55:49.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:49.115+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.121540+0000) 2019-10-13T13:55:49.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:49.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:49.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:49.122+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:49.127146+0000) 2019-10-13T13:55:50.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:50.078+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:50.083873+0000) 2019-10-13T13:55:50.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:50.834+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:50.835759+0000) 2019-10-13T13:55:52.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:52.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:52.592100+0000) 2019-10-13T13:55:54.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:54.123+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.129797+0000) 2019-10-13T13:55:54.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:54.123+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.130471+0000) 2019-10-13T13:55:54.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:54.123+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.130910+0000) 2019-10-13T13:55:54.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:54.127+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.132062+0000) 2019-10-13T13:55:54.135 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:54.127+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.133050+0000) 2019-10-13T13:55:54.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:54.126+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.133542+0000) 2019-10-13T13:55:54.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:54.127+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.133730+0000) 2019-10-13T13:55:54.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:54.134+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.139666+0000) 2019-10-13T13:55:54.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:54.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:55:55.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:55.611+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:55.617411+0000) 2019-10-13T13:55:55.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:55.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:55.904135+0000) 2019-10-13T13:55:56.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:56.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.583538+0000) 2019-10-13T13:55:58.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:58.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:58.850901+0000) 2019-10-13T13:55:58.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:58.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:58.850901+0000) 2019-10-13T13:55:59.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:55:59.139+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.142684+0000) 2019-10-13T13:55:59.146 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:55:59.139+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.142180+0000) 2019-10-13T13:55:59.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:55:59.139+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.143248+0000) 2019-10-13T13:55:59.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:55:59.138+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.144035+0000) 2019-10-13T13:55:59.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:55:59.139+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.144098+0000) 2019-10-13T13:55:59.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:55:59.139+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.144880+0000) 2019-10-13T13:55:59.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:55:59.139+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.145992+0000) 2019-10-13T13:55:59.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:55:59.146+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:59.151924+0000) 2019-10-13T13:55:59.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:55:59.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:00.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:00.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:00.084039+0000) 2019-10-13T13:56:00.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:00.842+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:00.846201+0000) 2019-10-13T13:56:02.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:02.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:02.592300+0000) 2019-10-13T13:56:04.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:04.151+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.155338+0000) 2019-10-13T13:56:04.158 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:04.151+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.155987+0000) 2019-10-13T13:56:04.159 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:04.151+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.156284+0000) 2019-10-13T13:56:04.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:04.151+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.157345+0000) 2019-10-13T13:56:04.161 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:04.154+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.158689+0000) 2019-10-13T13:56:04.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:04.151+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.159111+0000) 2019-10-13T13:56:04.162 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:04.155+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.159447+0000) 2019-10-13T13:56:04.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:04.159+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:04.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:04.162+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.168310+0000) 2019-10-13T13:56:05.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:05.615+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:05.617648+0000) 2019-10-13T13:56:05.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:05.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:05.904349+0000) 2019-10-13T13:56:06.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:06.583+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.583758+0000) 2019-10-13T13:56:08.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:08.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:08.851141+0000) 2019-10-13T13:56:08.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:08.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:08.851141+0000) 2019-10-13T13:56:09.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:09.167+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.168200+0000) 2019-10-13T13:56:09.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:09.162+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.168836+0000) 2019-10-13T13:56:09.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:09.167+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.170061+0000) 2019-10-13T13:56:09.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:09.167+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.170369+0000) 2019-10-13T13:56:09.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:09.167+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.170690+0000) 2019-10-13T13:56:09.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:09.167+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.171070+0000) 2019-10-13T13:56:09.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:09.167+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.171210+0000) 2019-10-13T13:56:09.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:09.170+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:09.176431+0000) 2019-10-13T13:56:09.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:09.175+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:10.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:10.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:10.084243+0000) 2019-10-13T13:56:10.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:10.854+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:10.857916+0000) 2019-10-13T13:56:12.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:12.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:12.592522+0000) 2019-10-13T13:56:14.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:14.179+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.182187+0000) 2019-10-13T13:56:14.185 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:14.179+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.182842+0000) 2019-10-13T13:56:14.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:14.179+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.182981+0000) 2019-10-13T13:56:14.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:14.183+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.183909+0000) 2019-10-13T13:56:14.187 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:14.178+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.184487+0000) 2019-10-13T13:56:14.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:14.183+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.184688+0000) 2019-10-13T13:56:14.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:14.183+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.185110+0000) 2019-10-13T13:56:14.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:14.182+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.188713+0000) 2019-10-13T13:56:14.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:14.187+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:15.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:15.615+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:15.617844+0000) 2019-10-13T13:56:15.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:15.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:15.904559+0000) 2019-10-13T13:56:16.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:16.583+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.583953+0000) 2019-10-13T13:56:18.389 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:56:18.389 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:18.393 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:56:18.393 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:18.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:18.850+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:18.851368+0000) 2019-10-13T13:56:18.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:18.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:18.851368+0000) 2019-10-13T13:56:19.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:19.190+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.193496+0000) 2019-10-13T13:56:19.197 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:19.190+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.193919+0000) 2019-10-13T13:56:19.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:19.190+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.195013+0000) 2019-10-13T13:56:19.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:19.190+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.195275+0000) 2019-10-13T13:56:19.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:19.190+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.196029+0000) 2019-10-13T13:56:19.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:19.194+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.196541+0000) 2019-10-13T13:56:19.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:19.194+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.196771+0000) 2019-10-13T13:56:19.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:19.194+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:19.200414+0000) 2019-10-13T13:56:19.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:19.198+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:20.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:20.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:20.084431+0000) 2019-10-13T13:56:20.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:20.862+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:20.864415+0000) 2019-10-13T13:56:22.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:22.590+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:22.592722+0000) 2019-10-13T13:56:24.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:24.202+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.206773+0000) 2019-10-13T13:56:24.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:24.202+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.207070+0000) 2019-10-13T13:56:24.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:24.206+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.207523+0000) 2019-10-13T13:56:24.211 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:24.206+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.207760+0000) 2019-10-13T13:56:24.211 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:24.206+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.208498+0000) 2019-10-13T13:56:24.211 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:24.202+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.209137+0000) 2019-10-13T13:56:24.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:24.206+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.209996+0000) 2019-10-13T13:56:24.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:24.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 183970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:24.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:24.214+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.218253+0000) 2019-10-13T13:56:25.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:25.614+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:25.618049+0000) 2019-10-13T13:56:25.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:25.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:25.904806+0000) 2019-10-13T13:56:26.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:26.582+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.584146+0000) 2019-10-13T13:56:28.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:28.850+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:28.851601+0000) 2019-10-13T13:56:28.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:28.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:28.851602+0000) 2019-10-13T13:56:29.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:29.218+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.219620+0000) 2019-10-13T13:56:29.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:29.218+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.219884+0000) 2019-10-13T13:56:29.223 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:29.218+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.220507+0000) 2019-10-13T13:56:29.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:29.218+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.221058+0000) 2019-10-13T13:56:29.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:29.218+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.221989+0000) 2019-10-13T13:56:29.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:29.218+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.222170+0000) 2019-10-13T13:56:29.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:29.218+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.222849+0000) 2019-10-13T13:56:29.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:29.226+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:29.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:29.226+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:29.229808+0000) 2019-10-13T13:56:30.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:30.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:30.084602+0000) 2019-10-13T13:56:30.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:30.870+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:30.870869+0000) 2019-10-13T13:56:32.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:32.590+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:32.592927+0000) 2019-10-13T13:56:34.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:34.226+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.230952+0000) 2019-10-13T13:56:34.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:34.230+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.231771+0000) 2019-10-13T13:56:34.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:34.230+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.232542+0000) 2019-10-13T13:56:34.235 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:34.230+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.232768+0000) 2019-10-13T13:56:34.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:34.230+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.233829+0000) 2019-10-13T13:56:34.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:34.230+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.234299+0000) 2019-10-13T13:56:34.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:34.234+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.234762+0000) 2019-10-13T13:56:34.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:34.238+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:34.244 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:34.242+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.242389+0000) 2019-10-13T13:56:35.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:35.614+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:35.618266+0000) 2019-10-13T13:56:35.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:35.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:35.905006+0000) 2019-10-13T13:56:36.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:36.582+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.584356+0000) 2019-10-13T13:56:38.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:38.850+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:38.851825+0000) 2019-10-13T13:56:38.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:38.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:38.851825+0000) 2019-10-13T13:56:39.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:39.242+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.244600+0000) 2019-10-13T13:56:39.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:39.242+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.245329+0000) 2019-10-13T13:56:39.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:39.242+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.245747+0000) 2019-10-13T13:56:39.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:39.242+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.246398+0000) 2019-10-13T13:56:39.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:39.242+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.246797+0000) 2019-10-13T13:56:39.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:39.246+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.247648+0000) 2019-10-13T13:56:39.250 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:39.245+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.248209+0000) 2019-10-13T13:56:39.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:39.249+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:39.252709+0000) 2019-10-13T13:56:39.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:39.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:40.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:40.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:40.084775+0000) 2019-10-13T13:56:40.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:40.877+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:40.882110+0000) 2019-10-13T13:56:42.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:42.590+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:42.593126+0000) 2019-10-13T13:56:44.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:44.254+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.257508+0000) 2019-10-13T13:56:44.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:44.254+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.258010+0000) 2019-10-13T13:56:44.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:44.254+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.258599+0000) 2019-10-13T13:56:44.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:44.258+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.259512+0000) 2019-10-13T13:56:44.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:44.258+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.259843+0000) 2019-10-13T13:56:44.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:44.258+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.260126+0000) 2019-10-13T13:56:44.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:44.257+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.261270+0000) 2019-10-13T13:56:44.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:44.265+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.266860+0000) 2019-10-13T13:56:44.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:44.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:45.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:45.614+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:45.618462+0000) 2019-10-13T13:56:45.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:45.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:45.905202+0000) 2019-10-13T13:56:46.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:46.582+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.584554+0000) 2019-10-13T13:56:48.501 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:56:48.501 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:48.504 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:56:48.505 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:48.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:48.850+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:48.852047+0000) 2019-10-13T13:56:48.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:48.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:48.852048+0000) 2019-10-13T13:56:49.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:49.266+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.269672+0000) 2019-10-13T13:56:49.273 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:49.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.270777+0000) 2019-10-13T13:56:49.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:49.270+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.271550+0000) 2019-10-13T13:56:49.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:49.270+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.271969+0000) 2019-10-13T13:56:49.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:49.270+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.272743+0000) 2019-10-13T13:56:49.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:49.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.273172+0000) 2019-10-13T13:56:49.276 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:49.270+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.273647+0000) 2019-10-13T13:56:49.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:49.273+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:49.277652+0000) 2019-10-13T13:56:49.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:49.278+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:50.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:50.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:50.084937+0000) 2019-10-13T13:56:50.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:50.885+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:50.888547+0000) 2019-10-13T13:56:52.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:52.590+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:52.593334+0000) 2019-10-13T13:56:54.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:54.278+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.282945+0000) 2019-10-13T13:56:54.286 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:54.281+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.284477+0000) 2019-10-13T13:56:54.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:54.282+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.285220+0000) 2019-10-13T13:56:54.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:54.282+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.285369+0000) 2019-10-13T13:56:54.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:54.282+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.286205+0000) 2019-10-13T13:56:54.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:54.282+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.286669+0000) 2019-10-13T13:56:54.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:54.282+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.287093+0000) 2019-10-13T13:56:54.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:54.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:56:54.299 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:54.293+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.296461+0000) 2019-10-13T13:56:55.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:55.614+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:55.618665+0000) 2019-10-13T13:56:55.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:55.902+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:55.905398+0000) 2019-10-13T13:56:56.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:56.582+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.584745+0000) 2019-10-13T13:56:58.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:58.850+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:58.852291+0000) 2019-10-13T13:56:58.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:58.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:58.852292+0000) 2019-10-13T13:56:59.300 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:56:59.294+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.297755+0000) 2019-10-13T13:56:59.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:56:59.294+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.298219+0000) 2019-10-13T13:56:59.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:56:59.294+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.298720+0000) 2019-10-13T13:56:59.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:56:59.297+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.299179+0000) 2019-10-13T13:56:59.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:56:59.294+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.299187+0000) 2019-10-13T13:56:59.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:56:59.298+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.299832+0000) 2019-10-13T13:56:59.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:56:59.298+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.300267+0000) 2019-10-13T13:56:59.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:56:59.301+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:59.303534+0000) 2019-10-13T13:56:59.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:56:59.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:00.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:00.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:00.085111+0000) 2019-10-13T13:57:00.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:00.893+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:00.894909+0000) 2019-10-13T13:57:02.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:02.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:02.593542+0000) 2019-10-13T13:57:04.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:04.305+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.309275+0000) 2019-10-13T13:57:04.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:04.305+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.309921+0000) 2019-10-13T13:57:04.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:04.305+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.310249+0000) 2019-10-13T13:57:04.313 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:04.309+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.311186+0000) 2019-10-13T13:57:04.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:04.305+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.311236+0000) 2019-10-13T13:57:04.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:04.309+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.312205+0000) 2019-10-13T13:57:04.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:04.309+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.313233+0000) 2019-10-13T13:57:04.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:04.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:04.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:04.317+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.319427+0000) 2019-10-13T13:57:05.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:05.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:05.618855+0000) 2019-10-13T13:57:05.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:05.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:05.905600+0000) 2019-10-13T13:57:06.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:06.581+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.584939+0000) 2019-10-13T13:57:08.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:08.849+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:08.852524+0000) 2019-10-13T13:57:08.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:08.849+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:08.852524+0000) 2019-10-13T13:57:09.324 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:09.317+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.321962+0000) 2019-10-13T13:57:09.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:09.321+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.323348+0000) 2019-10-13T13:57:09.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:09.321+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.323812+0000) 2019-10-13T13:57:09.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:09.321+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.324076+0000) 2019-10-13T13:57:09.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:09.321+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.324717+0000) 2019-10-13T13:57:09.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:09.321+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.324907+0000) 2019-10-13T13:57:09.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:09.321+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.325168+0000) 2019-10-13T13:57:09.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:09.325+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:09.329589+0000) 2019-10-13T13:57:09.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:09.329+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:10.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:10.081+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:10.085278+0000) 2019-10-13T13:57:10.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:10.901+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:10.901420+0000) 2019-10-13T13:57:12.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:12.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:12.593758+0000) 2019-10-13T13:57:14.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:14.333+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.336729+0000) 2019-10-13T13:57:14.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:14.333+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.337126+0000) 2019-10-13T13:57:14.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:14.333+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.337607+0000) 2019-10-13T13:57:14.341 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:14.333+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.337878+0000) 2019-10-13T13:57:14.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:14.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.338441+0000) 2019-10-13T13:57:14.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:14.333+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.338605+0000) 2019-10-13T13:57:14.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:14.333+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.339084+0000) 2019-10-13T13:57:14.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:14.341+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:14.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:14.345+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.346543+0000) 2019-10-13T13:57:15.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:15.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:15.619058+0000) 2019-10-13T13:57:15.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:15.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:15.905797+0000) 2019-10-13T13:57:16.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:16.581+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.585123+0000) 2019-10-13T13:57:18.604 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:57:18.604 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:18.607 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:57:18.608 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:18.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:18.849+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:18.852737+0000) 2019-10-13T13:57:18.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:18.849+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:18.852720+0000) 2019-10-13T13:57:19.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:19.345+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.348460+0000) 2019-10-13T13:57:19.352 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:19.345+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.349863+0000) 2019-10-13T13:57:19.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:19.345+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.350137+0000) 2019-10-13T13:57:19.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:19.345+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.350668+0000) 2019-10-13T13:57:19.353 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:19.345+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.350870+0000) 2019-10-13T13:57:19.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:19.349+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.351781+0000) 2019-10-13T13:57:19.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:19.349+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.352473+0000) 2019-10-13T13:57:19.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:19.353+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:19.356214+0000) 2019-10-13T13:57:19.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:19.353+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:20.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:20.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:20.085474+0000) 2019-10-13T13:57:20.914 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:20.909+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:20.911117+0000) 2019-10-13T13:57:22.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:22.589+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:22.593974+0000) 2019-10-13T13:57:24.364 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:24.357+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.361613+0000) 2019-10-13T13:57:24.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:24.361+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.363391+0000) 2019-10-13T13:57:24.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:24.361+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.363548+0000) 2019-10-13T13:57:24.366 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:24.361+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.364239+0000) 2019-10-13T13:57:24.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:24.361+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.364741+0000) 2019-10-13T13:57:24.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:24.361+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.365270+0000) 2019-10-13T13:57:24.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:24.361+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.365672+0000) 2019-10-13T13:57:24.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:24.365+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.368593+0000) 2019-10-13T13:57:24.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:24.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:25.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:25.613+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:25.619259+0000) 2019-10-13T13:57:25.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:25.901+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:25.905996+0000) 2019-10-13T13:57:26.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:26.581+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.585342+0000) 2019-10-13T13:57:28.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:28.849+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:28.852954+0000) 2019-10-13T13:57:28.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:28.849+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:28.852959+0000) 2019-10-13T13:57:29.376 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:29.369+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.374125+0000) 2019-10-13T13:57:29.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:29.369+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.374396+0000) 2019-10-13T13:57:29.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:29.373+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.375496+0000) 2019-10-13T13:57:29.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:29.373+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.376278+0000) 2019-10-13T13:57:29.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:29.373+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.376635+0000) 2019-10-13T13:57:29.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:29.373+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.377228+0000) 2019-10-13T13:57:29.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:29.373+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.377444+0000) 2019-10-13T13:57:29.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:29.381+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:29.382683+0000) 2019-10-13T13:57:29.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:29.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:30.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:30.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:30.085644+0000) 2019-10-13T13:57:30.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:30.912+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:30.917743+0000) 2019-10-13T13:57:32.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:32.593+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:32.594207+0000) 2019-10-13T13:57:34.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:34.381+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.387011+0000) 2019-10-13T13:57:34.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:34.385+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.387706+0000) 2019-10-13T13:57:34.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:34.385+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.388005+0000) 2019-10-13T13:57:34.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:34.385+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.389626+0000) 2019-10-13T13:57:34.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:34.385+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.390375+0000) 2019-10-13T13:57:34.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:34.388+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.390923+0000) 2019-10-13T13:57:34.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:34.385+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.390902+0000) 2019-10-13T13:57:34.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:34.392+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.394947+0000) 2019-10-13T13:57:34.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:34.393+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:35.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:35.617+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:35.619445+0000) 2019-10-13T13:57:35.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:35.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:35.906230+0000) 2019-10-13T13:57:36.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:36.581+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.585533+0000) 2019-10-13T13:57:38.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:38.849+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:38.853159+0000) 2019-10-13T13:57:38.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:38.849+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:38.853159+0000) 2019-10-13T13:57:39.403 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:39.396+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.400354+0000) 2019-10-13T13:57:39.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:39.397+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.401109+0000) 2019-10-13T13:57:39.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:39.397+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.401896+0000) 2019-10-13T13:57:39.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:39.397+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.402445+0000) 2019-10-13T13:57:39.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:39.397+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.402479+0000) 2019-10-13T13:57:39.406 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:39.397+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.403158+0000) 2019-10-13T13:57:39.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:39.401+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.403580+0000) 2019-10-13T13:57:39.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:39.405+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:39.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:39.404+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:39.409553+0000) 2019-10-13T13:57:40.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:40.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:40.085815+0000) 2019-10-13T13:57:40.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:40.920+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:40.924228+0000) 2019-10-13T13:57:42.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:42.593+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:42.594410+0000) 2019-10-13T13:57:44.414 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:44.409+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.411918+0000) 2019-10-13T13:57:44.415 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:44.408+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.412857+0000) 2019-10-13T13:57:44.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:44.409+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.413026+0000) 2019-10-13T13:57:44.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:44.409+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.413378+0000) 2019-10-13T13:57:44.416 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:44.409+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.413942+0000) 2019-10-13T13:57:44.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:44.409+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.414775+0000) 2019-10-13T13:57:44.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:44.409+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.415115+0000) 2019-10-13T13:57:44.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:44.417+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:44.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:44.416+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.422128+0000) 2019-10-13T13:57:45.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:45.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:45.619652+0000) 2019-10-13T13:57:45.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:45.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:45.906463+0000) 2019-10-13T13:57:46.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:46.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.585746+0000) 2019-10-13T13:57:48.840 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:57:48.840 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:48.844 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:57:48.844 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:48.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:48.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:48.853369+0000) 2019-10-13T13:57:48.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:48.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:48.853372+0000) 2019-10-13T13:57:49.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:49.420+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.424682+0000) 2019-10-13T13:57:49.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:49.420+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.425716+0000) 2019-10-13T13:57:49.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:49.420+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.425964+0000) 2019-10-13T13:57:49.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:49.420+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.426775+0000) 2019-10-13T13:57:49.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:49.424+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.427441+0000) 2019-10-13T13:57:49.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:49.424+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.427549+0000) 2019-10-13T13:57:49.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:49.424+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.428065+0000) 2019-10-13T13:57:49.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:49.428+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:49.430933+0000) 2019-10-13T13:57:49.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:49.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:50.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:50.080+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:50.086012+0000) 2019-10-13T13:57:50.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:50.928+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:50.932879+0000) 2019-10-13T13:57:52.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:52.588+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:52.594612+0000) 2019-10-13T13:57:54.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:54.432+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.438177+0000) 2019-10-13T13:57:54.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:54.436+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.438918+0000) 2019-10-13T13:57:54.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:54.436+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.439422+0000) 2019-10-13T13:57:54.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:54.436+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.440011+0000) 2019-10-13T13:57:54.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:54.436+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.440260+0000) 2019-10-13T13:57:54.443 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:54.436+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.440597+0000) 2019-10-13T13:57:54.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:54.436+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.441883+0000) 2019-10-13T13:57:54.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:54.440+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.443934+0000) 2019-10-13T13:57:54.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:54.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:55.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:55.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:55.619854+0000) 2019-10-13T13:57:55.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:55.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:55.906670+0000) 2019-10-13T13:57:56.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:56.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.585938+0000) 2019-10-13T13:57:58.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:58.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:58.853556+0000) 2019-10-13T13:57:58.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:58.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:58.853583+0000) 2019-10-13T13:57:59.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:57:59.444+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.450725+0000) 2019-10-13T13:57:59.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:57:59.444+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.450979+0000) 2019-10-13T13:57:59.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:57:59.448+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.451675+0000) 2019-10-13T13:57:59.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:57:59.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.451977+0000) 2019-10-13T13:57:59.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:57:59.448+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.452688+0000) 2019-10-13T13:57:59.455 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:57:59.448+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.453051+0000) 2019-10-13T13:57:59.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:57:59.448+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.453418+0000) 2019-10-13T13:57:59.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:57:59.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:57:59.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:57:59.456+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:59.460429+0000) 2019-10-13T13:58:00.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:00.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:00.086221+0000) 2019-10-13T13:58:00.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:00.936+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:00.939295+0000) 2019-10-13T13:58:02.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:02.592+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:02.594806+0000) 2019-10-13T13:58:04.466 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:04.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.463446+0000) 2019-10-13T13:58:04.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:04.460+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.463893+0000) 2019-10-13T13:58:04.467 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:04.460+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.464770+0000) 2019-10-13T13:58:04.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:04.460+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.465561+0000) 2019-10-13T13:58:04.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:04.460+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.465754+0000) 2019-10-13T13:58:04.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:04.460+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.466234+0000) 2019-10-13T13:58:04.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:04.460+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.467126+0000) 2019-10-13T13:58:04.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:04.468+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.471758+0000) 2019-10-13T13:58:04.475 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:04.468+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184647 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:05.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:05.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:05.620005+0000) 2019-10-13T13:58:05.909 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:05.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:05.906879+0000) 2019-10-13T13:58:06.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:06.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.586135+0000) 2019-10-13T13:58:08.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:08.852+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:08.853767+0000) 2019-10-13T13:58:08.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:08.852+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:08.853797+0000) 2019-10-13T13:58:09.478 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:09.472+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.475842+0000) 2019-10-13T13:58:09.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:09.472+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.476538+0000) 2019-10-13T13:58:09.480 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:09.472+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.477424+0000) 2019-10-13T13:58:09.480 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:09.472+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.477701+0000) 2019-10-13T13:58:09.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:09.472+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.477914+0000) 2019-10-13T13:58:09.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:09.472+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.478197+0000) 2019-10-13T13:58:09.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:09.472+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.479150+0000) 2019-10-13T13:58:09.486 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:09.480+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:09.483690+0000) 2019-10-13T13:58:09.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:09.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:10.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:10.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:10.086393+0000) 2019-10-13T13:58:10.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:10.944+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:10.948286+0000) 2019-10-13T13:58:12.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:12.592+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:12.595008+0000) 2019-10-13T13:58:14.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:14.484+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.488793+0000) 2019-10-13T13:58:14.492 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:14.484+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.489173+0000) 2019-10-13T13:58:14.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:14.484+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.489664+0000) 2019-10-13T13:58:14.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:14.484+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.490075+0000) 2019-10-13T13:58:14.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:14.484+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.490396+0000) 2019-10-13T13:58:14.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:14.488+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.491432+0000) 2019-10-13T13:58:14.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:14.488+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.491918+0000) 2019-10-13T13:58:14.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:14.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:14.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:14.496+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.498736+0000) 2019-10-13T13:58:15.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:15.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:15.620208+0000) 2019-10-13T13:58:15.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:15.900+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:15.907106+0000) 2019-10-13T13:58:16.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:16.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.586324+0000) 2019-10-13T13:58:18.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:18.848+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:18.853966+0000) 2019-10-13T13:58:18.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:18.848+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:18.854022+0000) 2019-10-13T13:58:18.884 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:58:18.884 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:18.888 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:58:18.888 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:19.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:19.496+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.501009+0000) 2019-10-13T13:58:19.504 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:19.496+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.501638+0000) 2019-10-13T13:58:19.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:19.496+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.502242+0000) 2019-10-13T13:58:19.506 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:19.496+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.503199+0000) 2019-10-13T13:58:19.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:19.496+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.503102+0000) 2019-10-13T13:58:19.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:19.500+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.504223+0000) 2019-10-13T13:58:19.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:19.500+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.504946+0000) 2019-10-13T13:58:19.513 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:19.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:19.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:19.512+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:19.514246+0000) 2019-10-13T13:58:20.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:20.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:20.086566+0000) 2019-10-13T13:58:20.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:20.956+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:20.959628+0000) 2019-10-13T13:58:22.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:22.592+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:22.595210+0000) 2019-10-13T13:58:24.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:24.512+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.514587+0000) 2019-10-13T13:58:24.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:24.508+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.514924+0000) 2019-10-13T13:58:24.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:24.508+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.515232+0000) 2019-10-13T13:58:24.518 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:24.512+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.515692+0000) 2019-10-13T13:58:24.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:24.512+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.516166+0000) 2019-10-13T13:58:24.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:24.512+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.516533+0000) 2019-10-13T13:58:24.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:24.512+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.517294+0000) 2019-10-13T13:58:24.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:24.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:24.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:24.520+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.525379+0000) 2019-10-13T13:58:25.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:25.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:25.620409+0000) 2019-10-13T13:58:25.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:25.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:25.907355+0000) 2019-10-13T13:58:26.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:26.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.586532+0000) 2019-10-13T13:58:28.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:28.851+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:28.854228+0000) 2019-10-13T13:58:28.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:28.851+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:28.854228+0000) 2019-10-13T13:58:29.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:29.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.526681+0000) 2019-10-13T13:58:29.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:29.523+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.527426+0000) 2019-10-13T13:58:29.531 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:29.523+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.527607+0000) 2019-10-13T13:58:29.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:29.523+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.528448+0000) 2019-10-13T13:58:29.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:29.523+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.528810+0000) 2019-10-13T13:58:29.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:29.523+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.529502+0000) 2019-10-13T13:58:29.533 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:29.523+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.529923+0000) 2019-10-13T13:58:29.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:29.531+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:29.535653+0000) 2019-10-13T13:58:29.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:29.531+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:30.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:30.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:30.086753+0000) 2019-10-13T13:58:30.968 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:30.959+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:30.966053+0000) 2019-10-13T13:58:32.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:32.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:32.595413+0000) 2019-10-13T13:58:34.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:34.535+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.539724+0000) 2019-10-13T13:58:34.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:34.535+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.539950+0000) 2019-10-13T13:58:34.543 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:34.535+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.540260+0000) 2019-10-13T13:58:34.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:34.535+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.541588+0000) 2019-10-13T13:58:34.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:34.535+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.541861+0000) 2019-10-13T13:58:34.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:34.535+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.542097+0000) 2019-10-13T13:58:34.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:34.535+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.542465+0000) 2019-10-13T13:58:34.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:34.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:34.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:34.543+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.547996+0000) 2019-10-13T13:58:35.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:35.615+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:35.620644+0000) 2019-10-13T13:58:35.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:35.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:35.907525+0000) 2019-10-13T13:58:36.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:36.583+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.586762+0000) 2019-10-13T13:58:38.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:38.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:38.854448+0000) 2019-10-13T13:58:38.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:38.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:38.854442+0000) 2019-10-13T13:58:39.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:39.547+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.552118+0000) 2019-10-13T13:58:39.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:39.547+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.552705+0000) 2019-10-13T13:58:39.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:39.547+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.553377+0000) 2019-10-13T13:58:39.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:39.547+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.553370+0000) 2019-10-13T13:58:39.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:39.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.554077+0000) 2019-10-13T13:58:39.557 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:39.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.555088+0000) 2019-10-13T13:58:39.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:39.551+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.555984+0000) 2019-10-13T13:58:39.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:39.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:39.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:39.555+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:39.561953+0000) 2019-10-13T13:58:40.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:40.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:40.086925+0000) 2019-10-13T13:58:40.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:40.967+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:40.972326+0000) 2019-10-13T13:58:42.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:42.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:42.595614+0000) 2019-10-13T13:58:44.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:44.559+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.564607+0000) 2019-10-13T13:58:44.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:44.559+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.565555+0000) 2019-10-13T13:58:44.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:44.563+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.566321+0000) 2019-10-13T13:58:44.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:44.559+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.566578+0000) 2019-10-13T13:58:44.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:44.559+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.566956+0000) 2019-10-13T13:58:44.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:44.559+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.567252+0000) 2019-10-13T13:58:44.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:44.563+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.567960+0000) 2019-10-13T13:58:44.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:44.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:44.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:44.571+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.574822+0000) 2019-10-13T13:58:45.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:45.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:45.620876+0000) 2019-10-13T13:58:45.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:45.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:45.907737+0000) 2019-10-13T13:58:46.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:46.579+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.586957+0000) 2019-10-13T13:58:48.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:48.847+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:48.854651+0000) 2019-10-13T13:58:48.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:48.847+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:48.854650+0000) 2019-10-13T13:58:49.012 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:58:49.012 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:49.016 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:58:49.016 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:49.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:49.571+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.576336+0000) 2019-10-13T13:58:49.580 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:49.571+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.577885+0000) 2019-10-13T13:58:49.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:49.571+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.578153+0000) 2019-10-13T13:58:49.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:49.571+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.578708+0000) 2019-10-13T13:58:49.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:49.571+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.579243+0000) 2019-10-13T13:58:49.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:49.575+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.579851+0000) 2019-10-13T13:58:49.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:49.575+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.580874+0000) 2019-10-13T13:58:49.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:49.579+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:49.586030+0000) 2019-10-13T13:58:49.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:49.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:50.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:50.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:50.087111+0000) 2019-10-13T13:58:50.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:50.979+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:50.981563+0000) 2019-10-13T13:58:52.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:52.595+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:52.595808+0000) 2019-10-13T13:58:54.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:54.587+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.591290+0000) 2019-10-13T13:58:54.595 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:54.587+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.592061+0000) 2019-10-13T13:58:54.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:54.587+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.592204+0000) 2019-10-13T13:58:54.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:54.587+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.592615+0000) 2019-10-13T13:58:54.596 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:54.587+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.593163+0000) 2019-10-13T13:58:54.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:54.587+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.593579+0000) 2019-10-13T13:58:54.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:54.587+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.594443+0000) 2019-10-13T13:58:54.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:54.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 184988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:54.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:54.595+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.601490+0000) 2019-10-13T13:58:55.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:55.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:55.621075+0000) 2019-10-13T13:58:55.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:55.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:55.907955+0000) 2019-10-13T13:58:56.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:56.583+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.587181+0000) 2019-10-13T13:58:58.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:58.851+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:58.854834+0000) 2019-10-13T13:58:58.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:58.851+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:58.854859+0000) 2019-10-13T13:58:59.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:58:59.603+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.604047+0000) 2019-10-13T13:58:59.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:58:59.603+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.604748+0000) 2019-10-13T13:58:59.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:58:59.603+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.604389+0000) 2019-10-13T13:58:59.608 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:58:59.599+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.605377+0000) 2019-10-13T13:58:59.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:58:59.603+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.605912+0000) 2019-10-13T13:58:59.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:58:59.603+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.606655+0000) 2019-10-13T13:58:59.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:58:59.603+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.606710+0000) 2019-10-13T13:58:59.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:58:59.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:58:59.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:58:59.607+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:59.613997+0000) 2019-10-13T13:59:00.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:00.083+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:00.087282+0000) 2019-10-13T13:59:00.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:00.987+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:00.987965+0000) 2019-10-13T13:59:02.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:02.595+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:02.596014+0000) 2019-10-13T13:59:04.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:04.615+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.616563+0000) 2019-10-13T13:59:04.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:04.615+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.616799+0000) 2019-10-13T13:59:04.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:04.615+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.618399+0000) 2019-10-13T13:59:04.621 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:04.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.617733+0000) 2019-10-13T13:59:04.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:04.615+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.617887+0000) 2019-10-13T13:59:04.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:04.615+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.619045+0000) 2019-10-13T13:59:04.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:04.619+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.620028+0000) 2019-10-13T13:59:04.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:04.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:04.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:04.623+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.626611+0000) 2019-10-13T13:59:05.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:05.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:05.621272+0000) 2019-10-13T13:59:05.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:05.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:05.908199+0000) 2019-10-13T13:59:06.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:06.587+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.587383+0000) 2019-10-13T13:59:08.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:08.851+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:08.855025+0000) 2019-10-13T13:59:08.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:08.851+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:08.855073+0000) 2019-10-13T13:59:09.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:09.627+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.629985+0000) 2019-10-13T13:59:09.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:09.627+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.630518+0000) 2019-10-13T13:59:09.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:09.627+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.631002+0000) 2019-10-13T13:59:09.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:09.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.631606+0000) 2019-10-13T13:59:09.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:09.631+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.632194+0000) 2019-10-13T13:59:09.636 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:09.627+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.632929+0000) 2019-10-13T13:59:09.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:09.631+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.634072+0000) 2019-10-13T13:59:09.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:09.635+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:09.638185+0000) 2019-10-13T13:59:09.641 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:09.635+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:10.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:10.087+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:10.087454+0000) 2019-10-13T13:59:10.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:10.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:10.994421+0000) 2019-10-13T13:59:12.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:12.594+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:12.596220+0000) 2019-10-13T13:59:14.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:14.638+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.642241+0000) 2019-10-13T13:59:14.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:14.638+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.642675+0000) 2019-10-13T13:59:14.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:14.642+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.643748+0000) 2019-10-13T13:59:14.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:14.639+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.644750+0000) 2019-10-13T13:59:14.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:14.642+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.645034+0000) 2019-10-13T13:59:14.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:14.642+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.645459+0000) 2019-10-13T13:59:14.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:14.642+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.646300+0000) 2019-10-13T13:59:14.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:14.647+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.651097+0000) 2019-10-13T13:59:14.654 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:14.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:15.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:15.618+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:15.621469+0000) 2019-10-13T13:59:15.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:15.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:15.908430+0000) 2019-10-13T13:59:16.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:16.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.587579+0000) 2019-10-13T13:59:18.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:18.850+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:18.855220+0000) 2019-10-13T13:59:18.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:18.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:18.855283+0000) 2019-10-13T13:59:19.205 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:59:19.205 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:19.209 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:59:19.209 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:19.659 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:19.654+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.655877+0000) 2019-10-13T13:59:19.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:19.654+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.656187+0000) 2019-10-13T13:59:19.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:19.654+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.656714+0000) 2019-10-13T13:59:19.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:19.654+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.657078+0000) 2019-10-13T13:59:19.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:19.654+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.658091+0000) 2019-10-13T13:59:19.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:19.654+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.658917+0000) 2019-10-13T13:59:19.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:19.654+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.659008+0000) 2019-10-13T13:59:19.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:19.662+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:19.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:19.662+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:19.666263+0000) 2019-10-13T13:59:20.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:20.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:20.087660+0000) 2019-10-13T13:59:21.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:20.998+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.000857+0000) 2019-10-13T13:59:22.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:22.594+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:22.596421+0000) 2019-10-13T13:59:24.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:24.666+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.667947+0000) 2019-10-13T13:59:24.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:24.662+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.668867+0000) 2019-10-13T13:59:24.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:24.666+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.668950+0000) 2019-10-13T13:59:24.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:24.666+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.670391+0000) 2019-10-13T13:59:24.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:24.666+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.670896+0000) 2019-10-13T13:59:24.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:24.666+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.670947+0000) 2019-10-13T13:59:24.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:24.670+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.671634+0000) 2019-10-13T13:59:24.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:24.670+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.676526+0000) 2019-10-13T13:59:24.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:24.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185191 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:25.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:25.618+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:25.621645+0000) 2019-10-13T13:59:25.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:25.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:25.908628+0000) 2019-10-13T13:59:26.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:26.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.587778+0000) 2019-10-13T13:59:28.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:28.854+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:28.855414+0000) 2019-10-13T13:59:28.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:28.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:28.855544+0000) 2019-10-13T13:59:29.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:29.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.680983+0000) 2019-10-13T13:59:29.685 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:29.678+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.682099+0000) 2019-10-13T13:59:29.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:29.678+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.682424+0000) 2019-10-13T13:59:29.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:29.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.683256+0000) 2019-10-13T13:59:29.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:29.682+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.683845+0000) 2019-10-13T13:59:29.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:29.682+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.684227+0000) 2019-10-13T13:59:29.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:29.682+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.684421+0000) 2019-10-13T13:59:29.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:29.682+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:29.689611+0000) 2019-10-13T13:59:29.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:29.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:30.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:30.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:30.087848+0000) 2019-10-13T13:59:31.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:31.006+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.012620+0000) 2019-10-13T13:59:32.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:32.594+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:32.596635+0000) 2019-10-13T13:59:34.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:34.690+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.694319+0000) 2019-10-13T13:59:34.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:34.690+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.695397+0000) 2019-10-13T13:59:34.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:34.690+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.694774+0000) 2019-10-13T13:59:34.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:34.694+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.695846+0000) 2019-10-13T13:59:34.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:34.694+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.696124+0000) 2019-10-13T13:59:34.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:34.694+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.696866+0000) 2019-10-13T13:59:34.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:34.694+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.697871+0000) 2019-10-13T13:59:34.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:34.698+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.702939+0000) 2019-10-13T13:59:34.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:34.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:35.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:35.618+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:35.621798+0000) 2019-10-13T13:59:35.911 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:35.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:35.908831+0000) 2019-10-13T13:59:36.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:36.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.587971+0000) 2019-10-13T13:59:38.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:38.854+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:38.855645+0000) 2019-10-13T13:59:38.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:38.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:38.855800+0000) 2019-10-13T13:59:39.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:39.706+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.707364+0000) 2019-10-13T13:59:39.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:39.706+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.708879+0000) 2019-10-13T13:59:39.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:39.706+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.709638+0000) 2019-10-13T13:59:39.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:39.706+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.710470+0000) 2019-10-13T13:59:39.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:39.706+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.710352+0000) 2019-10-13T13:59:39.713 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:39.706+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.710683+0000) 2019-10-13T13:59:39.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:39.706+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.711172+0000) 2019-10-13T13:59:39.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:39.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:39.716160+0000) 2019-10-13T13:59:39.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:39.714+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:40.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:40.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:40.088020+0000) 2019-10-13T13:59:41.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:41.018+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.019232+0000) 2019-10-13T13:59:42.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:42.594+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:42.596840+0000) 2019-10-13T13:59:44.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:44.718+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.720520+0000) 2019-10-13T13:59:44.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:44.714+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.721224+0000) 2019-10-13T13:59:44.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:44.718+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.721782+0000) 2019-10-13T13:59:44.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:44.718+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.722099+0000) 2019-10-13T13:59:44.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:44.718+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.722628+0000) 2019-10-13T13:59:44.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:44.718+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.723033+0000) 2019-10-13T13:59:44.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:44.722+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.724030+0000) 2019-10-13T13:59:44.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:44.722+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.727965+0000) 2019-10-13T13:59:44.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:44.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:45.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:45.618+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:45.621966+0000) 2019-10-13T13:59:45.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:45.906+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:45.909098+0000) 2019-10-13T13:59:46.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:46.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.588174+0000) 2019-10-13T13:59:48.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:48.854+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:48.855796+0000) 2019-10-13T13:59:48.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:48.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:48.855967+0000) 2019-10-13T13:59:49.335 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T13:59:49.335 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:49.338 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T13:59:49.339 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:49.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:49.730+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.732265+0000) 2019-10-13T13:59:49.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:49.730+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.733311+0000) 2019-10-13T13:59:49.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:49.726+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.733929+0000) 2019-10-13T13:59:49.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:49.730+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.734258+0000) 2019-10-13T13:59:49.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:49.730+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.734630+0000) 2019-10-13T13:59:49.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:49.730+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.735015+0000) 2019-10-13T13:59:49.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:49.734+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.736034+0000) 2019-10-13T13:59:49.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:49.734+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:49.741822+0000) 2019-10-13T13:59:49.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:49.738+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:50.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:50.082+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:50.088221+0000) 2019-10-13T13:59:51.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:51.018+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.025627+0000) 2019-10-13T13:59:52.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:52.594+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:52.597072+0000) 2019-10-13T13:59:54.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:54.741+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.744995+0000) 2019-10-13T13:59:54.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:54.741+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.746202+0000) 2019-10-13T13:59:54.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:54.742+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.746954+0000) 2019-10-13T13:59:54.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:54.745+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.747500+0000) 2019-10-13T13:59:54.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:54.745+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.747928+0000) 2019-10-13T13:59:54.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:54.745+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.748205+0000) 2019-10-13T13:59:54.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:54.745+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.749056+0000) 2019-10-13T13:59:54.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:54.746+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.752999+0000) 2019-10-13T13:59:54.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:54.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T13:59:55.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:55.617+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:55.622136+0000) 2019-10-13T13:59:55.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:55.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:55.909296+0000) 2019-10-13T13:59:56.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:56.585+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.588388+0000) 2019-10-13T13:59:58.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:58.853+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:58.856009+0000) 2019-10-13T13:59:58.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:58.853+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:58.856166+0000) 2019-10-13T13:59:59.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T13:59:59.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.758991+0000) 2019-10-13T13:59:59.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T13:59:59.753+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.759013+0000) 2019-10-13T13:59:59.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T13:59:59.757+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.760127+0000) 2019-10-13T13:59:59.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T13:59:59.757+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.760461+0000) 2019-10-13T13:59:59.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T13:59:59.757+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.760874+0000) 2019-10-13T13:59:59.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T13:59:59.754+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.761481+0000) 2019-10-13T13:59:59.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T13:59:59.757+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.761613+0000) 2019-10-13T13:59:59.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T13:59:59.762+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:59.767083+0000) 2019-10-13T13:59:59.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T13:59:59.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:00.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:00.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:00.088401+0000) 2019-10-13T14:00:01.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:01.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.032099+0000) 2019-10-13T14:00:02.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:02.593+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:02.597274+0000) 2019-10-13T14:00:04.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:04.769+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.771843+0000) 2019-10-13T14:00:04.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:04.769+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.772507+0000) 2019-10-13T14:00:04.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:04.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.772943+0000) 2019-10-13T14:00:04.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:04.766+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.773721+0000) 2019-10-13T14:00:04.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:04.769+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.773407+0000) 2019-10-13T14:00:04.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:04.770+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.774327+0000) 2019-10-13T14:00:04.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:04.769+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.774111+0000) 2019-10-13T14:00:04.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:04.769+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.774336+0000) 2019-10-13T14:00:04.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:04.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:05.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:05.621+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:05.622260+0000) 2019-10-13T14:00:05.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:05.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:05.909509+0000) 2019-10-13T14:00:06.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:06.585+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.588582+0000) 2019-10-13T14:00:08.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:08.853+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:08.856225+0000) 2019-10-13T14:00:08.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:08.853+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:08.856366+0000) 2019-10-13T14:00:09.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:09.781+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.784369+0000) 2019-10-13T14:00:09.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:09.781+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.785124+0000) 2019-10-13T14:00:09.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:09.781+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.785819+0000) 2019-10-13T14:00:09.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:09.781+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.786019+0000) 2019-10-13T14:00:09.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:09.786+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.786553+0000) 2019-10-13T14:00:09.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:09.781+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.786714+0000) 2019-10-13T14:00:09.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:09.785+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.787361+0000) 2019-10-13T14:00:09.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:09.786+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:09.789485+0000) 2019-10-13T14:00:09.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:09.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:10.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:10.086+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:10.088608+0000) 2019-10-13T14:00:11.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:11.038+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.038455+0000) 2019-10-13T14:00:12.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:12.593+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:12.597499+0000) 2019-10-13T14:00:14.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:14.793+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.798178+0000) 2019-10-13T14:00:14.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:14.793+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.797927+0000) 2019-10-13T14:00:14.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:14.793+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.798936+0000) 2019-10-13T14:00:14.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:14.797+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.800263+0000) 2019-10-13T14:00:14.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:14.797+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.799917+0000) 2019-10-13T14:00:14.804 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:14.797+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.800426+0000) 2019-10-13T14:00:14.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:14.797+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.800731+0000) 2019-10-13T14:00:14.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:14.801+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.805744+0000) 2019-10-13T14:00:14.808 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:14.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:15.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:15.617+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:15.622473+0000) 2019-10-13T14:00:15.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:15.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:15.909767+0000) 2019-10-13T14:00:16.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:16.585+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.588773+0000) 2019-10-13T14:00:18.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:18.853+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:18.856452+0000) 2019-10-13T14:00:18.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:18.853+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:18.856587+0000) 2019-10-13T14:00:19.438 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:00:19.438 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:19.441 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:00:19.442 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:19.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:19.805+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.809179+0000) 2019-10-13T14:00:19.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:19.805+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.810089+0000) 2019-10-13T14:00:19.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:19.809+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.810787+0000) 2019-10-13T14:00:19.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:19.805+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.810828+0000) 2019-10-13T14:00:19.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:19.809+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.811403+0000) 2019-10-13T14:00:19.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:19.809+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.812145+0000) 2019-10-13T14:00:19.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:19.809+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.812639+0000) 2019-10-13T14:00:19.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:19.817+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:19.818591+0000) 2019-10-13T14:00:19.821 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:19.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:20.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:20.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:20.088782+0000) 2019-10-13T14:00:21.047 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:21.041+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.045201+0000) 2019-10-13T14:00:22.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:22.593+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:22.597720+0000) 2019-10-13T14:00:24.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:24.817+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.821922+0000) 2019-10-13T14:00:24.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:24.821+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.822947+0000) 2019-10-13T14:00:24.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:24.817+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.822381+0000) 2019-10-13T14:00:24.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:24.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.823462+0000) 2019-10-13T14:00:24.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:24.821+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.823882+0000) 2019-10-13T14:00:24.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:24.821+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.824338+0000) 2019-10-13T14:00:24.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:24.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.825127+0000) 2019-10-13T14:00:24.835 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:24.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:24.836 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:24.833+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.834209+0000) 2019-10-13T14:00:25.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:25.617+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:25.622626+0000) 2019-10-13T14:00:25.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:25.905+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:25.909994+0000) 2019-10-13T14:00:26.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:26.585+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.588967+0000) 2019-10-13T14:00:28.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:28.853+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:28.856651+0000) 2019-10-13T14:00:28.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:28.853+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:28.856786+0000) 2019-10-13T14:00:29.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:29.833+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.836256+0000) 2019-10-13T14:00:29.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:29.833+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.836553+0000) 2019-10-13T14:00:29.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:29.833+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.837406+0000) 2019-10-13T14:00:29.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:29.833+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.837723+0000) 2019-10-13T14:00:29.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:29.833+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.838563+0000) 2019-10-13T14:00:29.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:29.833+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.839077+0000) 2019-10-13T14:00:29.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:29.837+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.839433+0000) 2019-10-13T14:00:29.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:29.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:29.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:29.841+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:29.845420+0000) 2019-10-13T14:00:30.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:30.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:30.088971+0000) 2019-10-13T14:00:31.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:31.049+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.051629+0000) 2019-10-13T14:00:32.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:32.597+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:32.597921+0000) 2019-10-13T14:00:34.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:34.845+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.848781+0000) 2019-10-13T14:00:34.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:34.845+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.848409+0000) 2019-10-13T14:00:34.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:34.845+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.850024+0000) 2019-10-13T14:00:34.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:34.845+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.850653+0000) 2019-10-13T14:00:34.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:34.845+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.850959+0000) 2019-10-13T14:00:34.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:34.849+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.851354+0000) 2019-10-13T14:00:34.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:34.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.851614+0000) 2019-10-13T14:00:34.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:34.853+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.856853+0000) 2019-10-13T14:00:34.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:34.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:35.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:35.617+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:35.622793+0000) 2019-10-13T14:00:35.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:35.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:35.910191+0000) 2019-10-13T14:00:36.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:36.589+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.589177+0000) 2019-10-13T14:00:38.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:38.852+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:38.856882+0000) 2019-10-13T14:00:38.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:38.852+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:38.856980+0000) 2019-10-13T14:00:39.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:39.857+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.860880+0000) 2019-10-13T14:00:39.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:39.856+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.861499+0000) 2019-10-13T14:00:39.864 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:39.856+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.861144+0000) 2019-10-13T14:00:39.865 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:39.856+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.862078+0000) 2019-10-13T14:00:39.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:39.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.863022+0000) 2019-10-13T14:00:39.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:39.860+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.864398+0000) 2019-10-13T14:00:39.867 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:39.860+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.864741+0000) 2019-10-13T14:00:39.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:39.865+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:39.869491+0000) 2019-10-13T14:00:39.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:39.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:40.091 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:40.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:40.089147+0000) 2019-10-13T14:00:41.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:41.053+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.058042+0000) 2019-10-13T14:00:42.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:42.592+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:42.598111+0000) 2019-10-13T14:00:44.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:44.868+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.874107+0000) 2019-10-13T14:00:44.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:44.868+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.874467+0000) 2019-10-13T14:00:44.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:44.873+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.875468+0000) 2019-10-13T14:00:44.878 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:44.872+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.875503+0000) 2019-10-13T14:00:44.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:44.872+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.875881+0000) 2019-10-13T14:00:44.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:44.872+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.876637+0000) 2019-10-13T14:00:44.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:44.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.877126+0000) 2019-10-13T14:00:44.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:44.881+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.883300+0000) 2019-10-13T14:00:44.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:44.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:45.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:45.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:45.622930+0000) 2019-10-13T14:00:45.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:45.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:45.910398+0000) 2019-10-13T14:00:46.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:46.588+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.589370+0000) 2019-10-13T14:00:48.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:48.856+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:48.857085+0000) 2019-10-13T14:00:48.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:48.856+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:48.857204+0000) 2019-10-13T14:00:49.575 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:00:49.575 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:49.579 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:00:49.579 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:49.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:49.880+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.886748+0000) 2019-10-13T14:00:49.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:49.884+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.887294+0000) 2019-10-13T14:00:49.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:49.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.887975+0000) 2019-10-13T14:00:49.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:49.884+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.888567+0000) 2019-10-13T14:00:49.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:49.884+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.888889+0000) 2019-10-13T14:00:49.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:49.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.889189+0000) 2019-10-13T14:00:49.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:49.889+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.890528+0000) 2019-10-13T14:00:49.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:49.892+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:49.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:49.893+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:49.897878+0000) 2019-10-13T14:00:50.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:50.085+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:50.089316+0000) 2019-10-13T14:00:51.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:51.061+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.064466+0000) 2019-10-13T14:00:52.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:52.596+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:52.598322+0000) 2019-10-13T14:00:54.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:54.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.900321+0000) 2019-10-13T14:00:54.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:54.896+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.900928+0000) 2019-10-13T14:00:54.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:54.896+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.901353+0000) 2019-10-13T14:00:54.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:54.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.902356+0000) 2019-10-13T14:00:54.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:54.896+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.902620+0000) 2019-10-13T14:00:54.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:54.896+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.903222+0000) 2019-10-13T14:00:54.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:54.901+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.904052+0000) 2019-10-13T14:00:54.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:54.905+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.907216+0000) 2019-10-13T14:00:54.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:54.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:00:55.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:55.616+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:55.623108+0000) 2019-10-13T14:00:55.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:55.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:55.910592+0000) 2019-10-13T14:00:56.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:56.588+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.589520+0000) 2019-10-13T14:00:58.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:58.852+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:58.857280+0000) 2019-10-13T14:00:58.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:58.852+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:58.857374+0000) 2019-10-13T14:00:59.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:00:59.908+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.913813+0000) 2019-10-13T14:00:59.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:00:59.908+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.914283+0000) 2019-10-13T14:00:59.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:00:59.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.914583+0000) 2019-10-13T14:00:59.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:00:59.908+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.914931+0000) 2019-10-13T14:00:59.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:00:59.913+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.915958+0000) 2019-10-13T14:00:59.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:00:59.912+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.915964+0000) 2019-10-13T14:00:59.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:00:59.912+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.916812+0000) 2019-10-13T14:00:59.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:00:59.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:59.919854+0000) 2019-10-13T14:00:59.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:00:59.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:00.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:00.089+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:00.089505+0000) 2019-10-13T14:01:01.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:01.069+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.070844+0000) 2019-10-13T14:01:02.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:02.592+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:02.598523+0000) 2019-10-13T14:01:04.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:04.920+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.926321+0000) 2019-10-13T14:01:04.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:04.920+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.926842+0000) 2019-10-13T14:01:04.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:04.925+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.927532+0000) 2019-10-13T14:01:04.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:04.924+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.927597+0000) 2019-10-13T14:01:04.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:04.924+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.927850+0000) 2019-10-13T14:01:04.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:04.924+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.928646+0000) 2019-10-13T14:01:04.932 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:04.924+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.929253+0000) 2019-10-13T14:01:04.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:04.933+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.935158+0000) 2019-10-13T14:01:04.937 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:04.928+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:05.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:05.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:05.623287+0000) 2019-10-13T14:01:05.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:05.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:05.910789+0000) 2019-10-13T14:01:06.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:06.588+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.589711+0000) 2019-10-13T14:01:08.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:08.856+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:08.857499+0000) 2019-10-13T14:01:08.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:08.856+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:08.857596+0000) 2019-10-13T14:01:09.942 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:09.932+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.938808+0000) 2019-10-13T14:01:09.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:09.936+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.939733+0000) 2019-10-13T14:01:09.943 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:09.936+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.939864+0000) 2019-10-13T14:01:09.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:09.936+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.939288+0000) 2019-10-13T14:01:09.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:09.936+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.941255+0000) 2019-10-13T14:01:09.944 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:09.936+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.941359+0000) 2019-10-13T14:01:09.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:09.936+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.941706+0000) 2019-10-13T14:01:09.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:09.940+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:09.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:09.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:09.951029+0000) 2019-10-13T14:01:10.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:10.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:10.089676+0000) 2019-10-13T14:01:11.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:11.072+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.077184+0000) 2019-10-13T14:01:12.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:12.596+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:12.598754+0000) 2019-10-13T14:01:14.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:14.948+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.951208+0000) 2019-10-13T14:01:14.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:14.948+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.951356+0000) 2019-10-13T14:01:14.954 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:14.948+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.951701+0000) 2019-10-13T14:01:14.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:14.948+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.952122+0000) 2019-10-13T14:01:14.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:14.948+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.952391+0000) 2019-10-13T14:01:14.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:14.948+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.952989+0000) 2019-10-13T14:01:14.956 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:14.948+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.953312+0000) 2019-10-13T14:01:14.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:14.956+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:14.963 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:14.956+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.961206+0000) 2019-10-13T14:01:15.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:15.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:15.623507+0000) 2019-10-13T14:01:15.913 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:15.904+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:15.910984+0000) 2019-10-13T14:01:16.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:16.584+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.589911+0000) 2019-10-13T14:01:18.860 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:18.856+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:18.857729+0000) 2019-10-13T14:01:18.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:18.856+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:18.857827+0000) 2019-10-13T14:01:19.693 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:01:19.693 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:19.697 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:01:19.697 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:19.965 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:19.956+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.962816+0000) 2019-10-13T14:01:19.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:19.960+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.963644+0000) 2019-10-13T14:01:19.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:19.960+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.963914+0000) 2019-10-13T14:01:19.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:19.960+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.964476+0000) 2019-10-13T14:01:19.967 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:19.960+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.964815+0000) 2019-10-13T14:01:19.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:19.960+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.965216+0000) 2019-10-13T14:01:19.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:19.960+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.966899+0000) 2019-10-13T14:01:19.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:19.968+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:19.970941+0000) 2019-10-13T14:01:19.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:19.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 185972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:20.092 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:20.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:20.089847+0000) 2019-10-13T14:01:21.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:21.080+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.083533+0000) 2019-10-13T14:01:22.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:22.591+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:22.598954+0000) 2019-10-13T14:01:24.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:24.971+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.975652+0000) 2019-10-13T14:01:24.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:24.971+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.976037+0000) 2019-10-13T14:01:24.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:24.971+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.977122+0000) 2019-10-13T14:01:24.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:24.972+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.977622+0000) 2019-10-13T14:01:24.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:24.971+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.977917+0000) 2019-10-13T14:01:24.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:24.971+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.978196+0000) 2019-10-13T14:01:24.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:24.971+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.978911+0000) 2019-10-13T14:01:24.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:24.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:24.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:24.984+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.986461+0000) 2019-10-13T14:01:25.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:25.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:25.623690+0000) 2019-10-13T14:01:25.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:25.903+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:25.911166+0000) 2019-10-13T14:01:26.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:26.587+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.590108+0000) 2019-10-13T14:01:28.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:28.855+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:28.857931+0000) 2019-10-13T14:01:28.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:28.855+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:28.858043+0000) 2019-10-13T14:01:29.991 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:29.983+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.988529+0000) 2019-10-13T14:01:29.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:29.983+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.989027+0000) 2019-10-13T14:01:29.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:29.983+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.989829+0000) 2019-10-13T14:01:29.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:29.983+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.990487+0000) 2019-10-13T14:01:29.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:29.988+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.991054+0000) 2019-10-13T14:01:29.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:29.983+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.991244+0000) 2019-10-13T14:01:29.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:29.987+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.991716+0000) 2019-10-13T14:01:29.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:29.992+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:29.995928+0000) 2019-10-13T14:01:29.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:29.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186040 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:30.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:30.084+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:30.090034+0000) 2019-10-13T14:01:31.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:31.088+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.089893+0000) 2019-10-13T14:01:32.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:32.595+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:32.599152+0000) 2019-10-13T14:01:35.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:34.995+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:34.999913+0000) 2019-10-13T14:01:35.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:34.995+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.000441+0000) 2019-10-13T14:01:35.003 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:34.995+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.001218+0000) 2019-10-13T14:01:35.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:34.995+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.001617+0000) 2019-10-13T14:01:35.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:34.995+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.001285+0000) 2019-10-13T14:01:35.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:34.995+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.002871+0000) 2019-10-13T14:01:35.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:35.000+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.004246+0000) 2019-10-13T14:01:35.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:35.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:35.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:35.004+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.010048+0000) 2019-10-13T14:01:35.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:35.623+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.623897+0000) 2019-10-13T14:01:35.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:35.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:35.911350+0000) 2019-10-13T14:01:36.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:36.583+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.590299+0000) 2019-10-13T14:01:38.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:38.855+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:38.858132+0000) 2019-10-13T14:01:38.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:38.855+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:38.858246+0000) 2019-10-13T14:01:40.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:40.007+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.012124+0000) 2019-10-13T14:01:40.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:40.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.012859+0000) 2019-10-13T14:01:40.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:40.007+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.012784+0000) 2019-10-13T14:01:40.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:40.007+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.013092+0000) 2019-10-13T14:01:40.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:40.007+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.013435+0000) 2019-10-13T14:01:40.017 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:40.007+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.013979+0000) 2019-10-13T14:01:40.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:40.007+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.014381+0000) 2019-10-13T14:01:40.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:40.016+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.020272+0000) 2019-10-13T14:01:40.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:40.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:40.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:40.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:40.090224+0000) 2019-10-13T14:01:41.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:41.092+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.096444+0000) 2019-10-13T14:01:42.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:42.595+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:42.599354+0000) 2019-10-13T14:01:45.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:45.019+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.024925+0000) 2019-10-13T14:01:45.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:45.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.026477+0000) 2019-10-13T14:01:45.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:45.019+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.026559+0000) 2019-10-13T14:01:45.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:45.019+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.026848+0000) 2019-10-13T14:01:45.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:45.023+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.027878+0000) 2019-10-13T14:01:45.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:45.023+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.027523+0000) 2019-10-13T14:01:45.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:45.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.028183+0000) 2019-10-13T14:01:45.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:45.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.033487+0000) 2019-10-13T14:01:45.038 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:45.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:45.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:45.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.624069+0000) 2019-10-13T14:01:45.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:45.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:45.911493+0000) 2019-10-13T14:01:46.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:46.587+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.590478+0000) 2019-10-13T14:01:48.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:48.855+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:48.858335+0000) 2019-10-13T14:01:48.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:48.855+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:48.858445+0000) 2019-10-13T14:01:49.819 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:01:49.819 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:49.822 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:01:49.823 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:50.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:50.031+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.037787+0000) 2019-10-13T14:01:50.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:50.031+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.038756+0000) 2019-10-13T14:01:50.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:50.035+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.039666+0000) 2019-10-13T14:01:50.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:50.035+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.040624+0000) 2019-10-13T14:01:50.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:50.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.041236+0000) 2019-10-13T14:01:50.044 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:50.035+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.040860+0000) 2019-10-13T14:01:50.044 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:50.035+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.041361+0000) 2019-10-13T14:01:50.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:50.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:50.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:50.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.048095+0000) 2019-10-13T14:01:50.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:50.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:50.090423+0000) 2019-10-13T14:01:51.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:51.100+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.102938+0000) 2019-10-13T14:01:52.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:52.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:52.599560+0000) 2019-10-13T14:01:55.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:55.047+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.050534+0000) 2019-10-13T14:01:55.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:55.047+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.051258+0000) 2019-10-13T14:01:55.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:55.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.051913+0000) 2019-10-13T14:01:55.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:01:55.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.052462+0000) 2019-10-13T14:01:55.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:55.051+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.052703+0000) 2019-10-13T14:01:55.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:01:55.051+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.053840+0000) 2019-10-13T14:01:55.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:55.051+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.054356+0000) 2019-10-13T14:01:55.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:01:55.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:01:55.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:01:55.060+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.062362+0000) 2019-10-13T14:01:55.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:01:55.623+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.624247+0000) 2019-10-13T14:01:55.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:01:55.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:55.911746+0000) 2019-10-13T14:01:56.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:01:56.587+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.590680+0000) 2019-10-13T14:01:58.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:01:58.855+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:58.858544+0000) 2019-10-13T14:01:58.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:01:58.855+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:58.858662+0000) 2019-10-13T14:02:00.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:00.063+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.063517+0000) 2019-10-13T14:02:00.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:00.063+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.064320+0000) 2019-10-13T14:02:00.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:00.063+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.065131+0000) 2019-10-13T14:02:00.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:00.063+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.065599+0000) 2019-10-13T14:02:00.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:00.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.067202+0000) 2019-10-13T14:02:00.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:00.067+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.067477+0000) 2019-10-13T14:02:00.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:00.067+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.067907+0000) 2019-10-13T14:02:00.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:00.068+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.071096+0000) 2019-10-13T14:02:00.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:00.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:00.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:00.088+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:00.090552+0000) 2019-10-13T14:02:01.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:01.104+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:01.109368+0000) 2019-10-13T14:02:02.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:02.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.599757+0000) 2019-10-13T14:02:05.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:05.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.077188+0000) 2019-10-13T14:02:05.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:05.071+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.077599+0000) 2019-10-13T14:02:05.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:05.074+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.077741+0000) 2019-10-13T14:02:05.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:05.074+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.078399+0000) 2019-10-13T14:02:05.081 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:05.074+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.078841+0000) 2019-10-13T14:02:05.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:05.078+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.079701+0000) 2019-10-13T14:02:05.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:05.078+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.079738+0000) 2019-10-13T14:02:05.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:05.079+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.084609+0000) 2019-10-13T14:02:05.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:05.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:05.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:05.622+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.624425+0000) 2019-10-13T14:02:05.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:05.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:05.911993+0000) 2019-10-13T14:02:06.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:06.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:06.590926+0000) 2019-10-13T14:02:08.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:08.854+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:08.858745+0000) 2019-10-13T14:02:08.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:08.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:08.858887+0000) 2019-10-13T14:02:10.091 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:10.086+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.088782+0000) 2019-10-13T14:02:10.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:10.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.089251+0000) 2019-10-13T14:02:10.092 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:10.086+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.089624+0000) 2019-10-13T14:02:10.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:10.090+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.092064+0000) 2019-10-13T14:02:10.095 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:10.090+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.092648+0000) 2019-10-13T14:02:10.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:10.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.093183+0000) 2019-10-13T14:02:10.096 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:10.087+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.094072+0000) 2019-10-13T14:02:10.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:10.091+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:10.098121+0000) 2019-10-13T14:02:10.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:10.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:11.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:11.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:11.120699+0000) 2019-10-13T14:02:12.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:12.598+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.599971+0000) 2019-10-13T14:02:15.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:15.095+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.101903+0000) 2019-10-13T14:02:15.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:15.098+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.102173+0000) 2019-10-13T14:02:15.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:15.098+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.102600+0000) 2019-10-13T14:02:15.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:15.098+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.102868+0000) 2019-10-13T14:02:15.106 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:15.102+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.103348+0000) 2019-10-13T14:02:15.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:15.102+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.104570+0000) 2019-10-13T14:02:15.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:15.102+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.105188+0000) 2019-10-13T14:02:15.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:15.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:15.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:15.111+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.114266+0000) 2019-10-13T14:02:15.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:15.622+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.624605+0000) 2019-10-13T14:02:15.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:15.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:15.912210+0000) 2019-10-13T14:02:16.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:16.586+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:16.591122+0000) 2019-10-13T14:02:16.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:16.835+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:16.841084+0000) 2019-10-13T14:02:18.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:18.858+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:18.858941+0000) 2019-10-13T14:02:18.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:18.858+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:18.859055+0000) 2019-10-13T14:02:20.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:20.110+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.114854+0000) 2019-10-13T14:02:20.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:20.110+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.115038+0000) 2019-10-13T14:02:20.118 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:20.114+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.115806+0000) 2019-10-13T14:02:20.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:20.114+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.116233+0000) 2019-10-13T14:02:20.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:20.114+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.116641+0000) 2019-10-13T14:02:20.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:20.111+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.117150+0000) 2019-10-13T14:02:20.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:20.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.117567+0000) 2019-10-13T14:02:20.125 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:20.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:20.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:20.119+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:20.122715+0000) 2019-10-13T14:02:20.144 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:02:20.145 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:20.148 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:02:20.148 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:21.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:21.123+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:21.127186+0000) 2019-10-13T14:02:22.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:22.598+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.600189+0000) 2019-10-13T14:02:25.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:25.122+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.125964+0000) 2019-10-13T14:02:25.129 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:25.122+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.125673+0000) 2019-10-13T14:02:25.129 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:25.122+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.126272+0000) 2019-10-13T14:02:25.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:25.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.126881+0000) 2019-10-13T14:02:25.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:25.126+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.127933+0000) 2019-10-13T14:02:25.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:25.126+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.128496+0000) 2019-10-13T14:02:25.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:25.126+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.128791+0000) 2019-10-13T14:02:25.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:25.131+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.134527+0000) 2019-10-13T14:02:25.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:25.134+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:25.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:25.622+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.624787+0000) 2019-10-13T14:02:25.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:25.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:25.912402+0000) 2019-10-13T14:02:26.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:26.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:26.591343+0000) 2019-10-13T14:02:26.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:26.835+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:26.841278+0000) 2019-10-13T14:02:28.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:28.854+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:28.859189+0000) 2019-10-13T14:02:28.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:28.854+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:28.859253+0000) 2019-10-13T14:02:30.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:30.134+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.137871+0000) 2019-10-13T14:02:30.141 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:30.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.139057+0000) 2019-10-13T14:02:30.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:30.138+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.139682+0000) 2019-10-13T14:02:30.142 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:30.138+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.140135+0000) 2019-10-13T14:02:30.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:30.138+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.140679+0000) 2019-10-13T14:02:30.144 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:30.138+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.140692+0000) 2019-10-13T14:02:30.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:30.135+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.142047+0000) 2019-10-13T14:02:30.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:30.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186442 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:30.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:30.143+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:30.149204+0000) 2019-10-13T14:02:31.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:31.127+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:31.133533+0000) 2019-10-13T14:02:32.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:32.598+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.600426+0000) 2019-10-13T14:02:35.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:35.146+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.151213+0000) 2019-10-13T14:02:35.154 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:35.150+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.151791+0000) 2019-10-13T14:02:35.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:35.150+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.152463+0000) 2019-10-13T14:02:35.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:35.147+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.153329+0000) 2019-10-13T14:02:35.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:35.150+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.152883+0000) 2019-10-13T14:02:35.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:35.150+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.154323+0000) 2019-10-13T14:02:35.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:35.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.154666+0000) 2019-10-13T14:02:35.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:35.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:35.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:35.159+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.164101+0000) 2019-10-13T14:02:35.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:35.622+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.625000+0000) 2019-10-13T14:02:35.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:35.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:35.912588+0000) 2019-10-13T14:02:36.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:36.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:36.591538+0000) 2019-10-13T14:02:36.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:36.835+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:36.841470+0000) 2019-10-13T14:02:38.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:38.858+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:38.859419+0000) 2019-10-13T14:02:38.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:38.858+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:38.859521+0000) 2019-10-13T14:02:40.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:40.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.163800+0000) 2019-10-13T14:02:40.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:40.162+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.163297+0000) 2019-10-13T14:02:40.167 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:40.162+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.164163+0000) 2019-10-13T14:02:40.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:40.162+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.165030+0000) 2019-10-13T14:02:40.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:40.162+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.165368+0000) 2019-10-13T14:02:40.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:40.162+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.166235+0000) 2019-10-13T14:02:40.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:40.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.167419+0000) 2019-10-13T14:02:40.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:40.170+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186508 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:40.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:40.167+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:40.173732+0000) 2019-10-13T14:02:41.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:41.139+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:41.139912+0000) 2019-10-13T14:02:42.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:42.598+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.600625+0000) 2019-10-13T14:02:45.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:45.174+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.176930+0000) 2019-10-13T14:02:45.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:45.174+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.177287+0000) 2019-10-13T14:02:45.180 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:45.174+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.177725+0000) 2019-10-13T14:02:45.181 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:45.174+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.178191+0000) 2019-10-13T14:02:45.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:45.174+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.178778+0000) 2019-10-13T14:02:45.181 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:45.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.179028+0000) 2019-10-13T14:02:45.182 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:45.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.179229+0000) 2019-10-13T14:02:45.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:45.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:45.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:45.183+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.186901+0000) 2019-10-13T14:02:45.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:45.622+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.625179+0000) 2019-10-13T14:02:45.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:45.910+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:45.912778+0000) 2019-10-13T14:02:46.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:46.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:46.591732+0000) 2019-10-13T14:02:46.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:46.839+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:46.841660+0000) 2019-10-13T14:02:48.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:48.857+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:48.859621+0000) 2019-10-13T14:02:48.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:48.857+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:48.859758+0000) 2019-10-13T14:02:50.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:50.185+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.188393+0000) 2019-10-13T14:02:50.191 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:50.185+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.188841+0000) 2019-10-13T14:02:50.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:50.183+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.189473+0000) 2019-10-13T14:02:50.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:50.185+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.189585+0000) 2019-10-13T14:02:50.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:50.185+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.190424+0000) 2019-10-13T14:02:50.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:50.185+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.190766+0000) 2019-10-13T14:02:50.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:50.185+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.191235+0000) 2019-10-13T14:02:50.200 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:50.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186576 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:50.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:50.195+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:50.198298+0000) 2019-10-13T14:02:50.276 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:02:50.276 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:50.280 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:02:50.280 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:51.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:51.143+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:51.146190+0000) 2019-10-13T14:02:52.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:52.597+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.600834+0000) 2019-10-13T14:02:55.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:55.197+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.201285+0000) 2019-10-13T14:02:55.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:55.197+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.201846+0000) 2019-10-13T14:02:55.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:55.197+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.202029+0000) 2019-10-13T14:02:55.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:55.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.202871+0000) 2019-10-13T14:02:55.206 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:55.197+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.202971+0000) 2019-10-13T14:02:55.206 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:55.201+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.203840+0000) 2019-10-13T14:02:55.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:02:55.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.204445+0000) 2019-10-13T14:02:55.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:02:55.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.210647+0000) 2019-10-13T14:02:55.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:02:55.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:02:55.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:02:55.621+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.625358+0000) 2019-10-13T14:02:55.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:02:55.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:55.912996+0000) 2019-10-13T14:02:56.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:02:56.589+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:56.591930+0000) 2019-10-13T14:02:56.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:02:56.838+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:56.841857+0000) 2019-10-13T14:02:58.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:02:58.857+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:58.859851+0000) 2019-10-13T14:02:58.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:02:58.857+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:58.859979+0000) 2019-10-13T14:03:00.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:00.209+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.214050+0000) 2019-10-13T14:03:00.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:00.210+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.214682+0000) 2019-10-13T14:03:00.218 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:00.209+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.215085+0000) 2019-10-13T14:03:00.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:00.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.216048+0000) 2019-10-13T14:03:00.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:00.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.216355+0000) 2019-10-13T14:03:00.219 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:00.213+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.217152+0000) 2019-10-13T14:03:00.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:00.213+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.217474+0000) 2019-10-13T14:03:00.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:00.218+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:00.223274+0000) 2019-10-13T14:03:00.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:00.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:01.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:01.150+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:01.152775+0000) 2019-10-13T14:03:02.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:02.597+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.601034+0000) 2019-10-13T14:03:05.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:05.225+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.227284+0000) 2019-10-13T14:03:05.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:05.225+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.227691+0000) 2019-10-13T14:03:05.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:05.225+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.227993+0000) 2019-10-13T14:03:05.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:05.222+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.228515+0000) 2019-10-13T14:03:05.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:05.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.229100+0000) 2019-10-13T14:03:05.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:05.225+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.229339+0000) 2019-10-13T14:03:05.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:05.225+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.230306+0000) 2019-10-13T14:03:05.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:05.230+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.236677+0000) 2019-10-13T14:03:05.239 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:05.233+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:05.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:05.621+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.625560+0000) 2019-10-13T14:03:05.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:05.909+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:05.913183+0000) 2019-10-13T14:03:06.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:06.589+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:06.592132+0000) 2019-10-13T14:03:06.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:06.838+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:06.842051+0000) 2019-10-13T14:03:08.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:08.857+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:08.860064+0000) 2019-10-13T14:03:08.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:08.857+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:08.860178+0000) 2019-10-13T14:03:10.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:10.233+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.239113+0000) 2019-10-13T14:03:10.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:10.237+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.239849+0000) 2019-10-13T14:03:10.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:10.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.240513+0000) 2019-10-13T14:03:10.244 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:10.234+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.241908+0000) 2019-10-13T14:03:10.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:10.237+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.242528+0000) 2019-10-13T14:03:10.245 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:10.237+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.243119+0000) 2019-10-13T14:03:10.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:10.241+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.243986+0000) 2019-10-13T14:03:10.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:10.242+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:10.247428+0000) 2019-10-13T14:03:10.252 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:10.245+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:11.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:11.162+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:11.163947+0000) 2019-10-13T14:03:12.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:12.597+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.601234+0000) 2019-10-13T14:03:15.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:15.249+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.252741+0000) 2019-10-13T14:03:15.257 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:15.249+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.254064+0000) 2019-10-13T14:03:15.257 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:15.249+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.254188+0000) 2019-10-13T14:03:15.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:15.249+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.254932+0000) 2019-10-13T14:03:15.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:15.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.255493+0000) 2019-10-13T14:03:15.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:15.253+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.255851+0000) 2019-10-13T14:03:15.259 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:15.253+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.256874+0000) 2019-10-13T14:03:15.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:15.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.259526+0000) 2019-10-13T14:03:15.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:15.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:15.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:15.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.625757+0000) 2019-10-13T14:03:15.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:15.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:15.913390+0000) 2019-10-13T14:03:16.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:16.589+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:16.592332+0000) 2019-10-13T14:03:16.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:16.838+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:16.842265+0000) 2019-10-13T14:03:18.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:18.857+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:18.860296+0000) 2019-10-13T14:03:18.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:18.857+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:18.860426+0000) 2019-10-13T14:03:20.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:20.261+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.264584+0000) 2019-10-13T14:03:20.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:20.261+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.265103+0000) 2019-10-13T14:03:20.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:20.258+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.265747+0000) 2019-10-13T14:03:20.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:20.261+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.266042+0000) 2019-10-13T14:03:20.270 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:20.261+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.267229+0000) 2019-10-13T14:03:20.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:20.265+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.267595+0000) 2019-10-13T14:03:20.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:20.265+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.267853+0000) 2019-10-13T14:03:20.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:20.266+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:20.273022+0000) 2019-10-13T14:03:20.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:20.269+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186775 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:20.388 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:03:20.388 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:20.392 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:03:20.392 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:21.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:21.170+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:21.171255+0000) 2019-10-13T14:03:22.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:22.597+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.601464+0000) 2019-10-13T14:03:25.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:25.273+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.277734+0000) 2019-10-13T14:03:25.281 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:25.273+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.278703+0000) 2019-10-13T14:03:25.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:25.273+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.278653+0000) 2019-10-13T14:03:25.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:25.273+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.279058+0000) 2019-10-13T14:03:25.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:25.274+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.279704+0000) 2019-10-13T14:03:25.283 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:25.277+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.280608+0000) 2019-10-13T14:03:25.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:25.277+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.280924+0000) 2019-10-13T14:03:25.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:25.282+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.287368+0000) 2019-10-13T14:03:25.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:25.285+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:25.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:25.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.625932+0000) 2019-10-13T14:03:25.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:25.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:25.913603+0000) 2019-10-13T14:03:26.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:26.589+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:26.592524+0000) 2019-10-13T14:03:26.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:26.838+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:26.842464+0000) 2019-10-13T14:03:28.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:28.857+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:28.860492+0000) 2019-10-13T14:03:28.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:28.857+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:28.860667+0000) 2019-10-13T14:03:30.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:30.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.291141+0000) 2019-10-13T14:03:30.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:30.288+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.291413+0000) 2019-10-13T14:03:30.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:30.288+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.292476+0000) 2019-10-13T14:03:30.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:30.288+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.292902+0000) 2019-10-13T14:03:30.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:30.288+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.293211+0000) 2019-10-13T14:03:30.296 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:30.288+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.293632+0000) 2019-10-13T14:03:30.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:30.288+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.294532+0000) 2019-10-13T14:03:30.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:30.294+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:30.299379+0000) 2019-10-13T14:03:30.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:30.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:31.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:31.182+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:31.182668+0000) 2019-10-13T14:03:32.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:32.600+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.601697+0000) 2019-10-13T14:03:35.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:35.300+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.304656+0000) 2019-10-13T14:03:35.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:35.300+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.305057+0000) 2019-10-13T14:03:35.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:35.300+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.305978+0000) 2019-10-13T14:03:35.309 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:35.300+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.306680+0000) 2019-10-13T14:03:35.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:35.300+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.306423+0000) 2019-10-13T14:03:35.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:35.302+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.307367+0000) 2019-10-13T14:03:35.310 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:35.304+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.307733+0000) 2019-10-13T14:03:35.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:35.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:35.319 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:35.310+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.316909+0000) 2019-10-13T14:03:35.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:35.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.626111+0000) 2019-10-13T14:03:35.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:35.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:35.913788+0000) 2019-10-13T14:03:36.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:36.588+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:36.592720+0000) 2019-10-13T14:03:36.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:36.842+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:36.842686+0000) 2019-10-13T14:03:38.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:38.856+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:38.860733+0000) 2019-10-13T14:03:38.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:38.856+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:38.860891+0000) 2019-10-13T14:03:40.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:40.312+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.317321+0000) 2019-10-13T14:03:40.321 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:40.312+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.318518+0000) 2019-10-13T14:03:40.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:40.314+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.319231+0000) 2019-10-13T14:03:40.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:40.316+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.319510+0000) 2019-10-13T14:03:40.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:40.316+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.320116+0000) 2019-10-13T14:03:40.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:40.316+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.320557+0000) 2019-10-13T14:03:40.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:40.316+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.320877+0000) 2019-10-13T14:03:40.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:40.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:40.332 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:40.326+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:40.330247+0000) 2019-10-13T14:03:41.194 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:41.190+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:41.191580+0000) 2019-10-13T14:03:42.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:42.600+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.601910+0000) 2019-10-13T14:03:45.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:45.326+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.329721+0000) 2019-10-13T14:03:45.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:45.324+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.330511+0000) 2019-10-13T14:03:45.333 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:45.324+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.330733+0000) 2019-10-13T14:03:45.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:45.324+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.331116+0000) 2019-10-13T14:03:45.334 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:45.328+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.331740+0000) 2019-10-13T14:03:45.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:45.328+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.333016+0000) 2019-10-13T14:03:45.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:45.328+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.333246+0000) 2019-10-13T14:03:45.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:45.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:45.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:45.338+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.340742+0000) 2019-10-13T14:03:45.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:45.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.626310+0000) 2019-10-13T14:03:45.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:45.908+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:45.913969+0000) 2019-10-13T14:03:46.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:46.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:46.592957+0000) 2019-10-13T14:03:46.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:46.842+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:46.842877+0000) 2019-10-13T14:03:48.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:48.860+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:48.860960+0000) 2019-10-13T14:03:48.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:48.860+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:48.861115+0000) 2019-10-13T14:03:50.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:50.340+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.343823+0000) 2019-10-13T14:03:50.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:50.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.344190+0000) 2019-10-13T14:03:50.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:50.342+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.345161+0000) 2019-10-13T14:03:50.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:50.340+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.345018+0000) 2019-10-13T14:03:50.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:50.340+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.345398+0000) 2019-10-13T14:03:50.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:50.340+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.345943+0000) 2019-10-13T14:03:50.350 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:50.344+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.347551+0000) 2019-10-13T14:03:50.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:50.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 186974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:50.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:50.350+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:50.354039+0000) 2019-10-13T14:03:50.438 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:03:50.438 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:50.441 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:03:50.442 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:51.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:51.201+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:51.202888+0000) 2019-10-13T14:03:52.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:52.600+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.602107+0000) 2019-10-13T14:03:55.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:55.352+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.357277+0000) 2019-10-13T14:03:55.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:55.353+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.357786+0000) 2019-10-13T14:03:55.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:55.352+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.358106+0000) 2019-10-13T14:03:55.361 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:03:55.352+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.358322+0000) 2019-10-13T14:03:55.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:55.352+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.358944+0000) 2019-10-13T14:03:55.362 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:55.352+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.359084+0000) 2019-10-13T14:03:55.362 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:55.356+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.359737+0000) 2019-10-13T14:03:55.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:03:55.361+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.363160+0000) 2019-10-13T14:03:55.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:03:55.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:03:55.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:03:55.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.626509+0000) 2019-10-13T14:03:55.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:03:55.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:55.914150+0000) 2019-10-13T14:03:56.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:03:56.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:56.593155+0000) 2019-10-13T14:03:56.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:03:56.841+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:56.843082+0000) 2019-10-13T14:03:58.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:03:58.860+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:58.861160+0000) 2019-10-13T14:03:58.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:03:58.860+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:58.861287+0000) 2019-10-13T14:04:00.372 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:00.364+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.368845+0000) 2019-10-13T14:04:00.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:00.364+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.369124+0000) 2019-10-13T14:04:00.373 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:00.364+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.369858+0000) 2019-10-13T14:04:00.373 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:00.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.371368+0000) 2019-10-13T14:04:00.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:00.368+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.371797+0000) 2019-10-13T14:04:00.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:00.368+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.371996+0000) 2019-10-13T14:04:00.375 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:00.368+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.372744+0000) 2019-10-13T14:04:00.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:00.373+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:00.375958+0000) 2019-10-13T14:04:00.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:00.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:01.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:01.205+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:01.209254+0000) 2019-10-13T14:04:02.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:02.600+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.602306+0000) 2019-10-13T14:04:05.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:05.376+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.382334+0000) 2019-10-13T14:04:05.385 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:05.376+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.382655+0000) 2019-10-13T14:04:05.386 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:05.380+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.383857+0000) 2019-10-13T14:04:05.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:05.381+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.384510+0000) 2019-10-13T14:04:05.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:05.380+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.384777+0000) 2019-10-13T14:04:05.387 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:05.380+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.385006+0000) 2019-10-13T14:04:05.389 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:05.380+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.386451+0000) 2019-10-13T14:04:05.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:05.389+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.391636+0000) 2019-10-13T14:04:05.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:05.388+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:05.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:05.620+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.626704+0000) 2019-10-13T14:04:05.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:05.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:05.914361+0000) 2019-10-13T14:04:06.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:06.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:06.593376+0000) 2019-10-13T14:04:06.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:06.841+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:06.843271+0000) 2019-10-13T14:04:08.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:08.856+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:08.861388+0000) 2019-10-13T14:04:08.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:08.856+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:08.861510+0000) 2019-10-13T14:04:10.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:10.388+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.395160+0000) 2019-10-13T14:04:10.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:10.392+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.395560+0000) 2019-10-13T14:04:10.399 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:10.392+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.396000+0000) 2019-10-13T14:04:10.399 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:10.392+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.396873+0000) 2019-10-13T14:04:10.400 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:10.392+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.397307+0000) 2019-10-13T14:04:10.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:10.392+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.397705+0000) 2019-10-13T14:04:10.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:10.397+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.398633+0000) 2019-10-13T14:04:10.406 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:10.401+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:10.403553+0000) 2019-10-13T14:04:10.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:10.400+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:11.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:11.213+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:11.215721+0000) 2019-10-13T14:04:12.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:12.600+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.602505+0000) 2019-10-13T14:04:15.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:15.403+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.407725+0000) 2019-10-13T14:04:15.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:15.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.408438+0000) 2019-10-13T14:04:15.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:15.403+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.408697+0000) 2019-10-13T14:04:15.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:15.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.409527+0000) 2019-10-13T14:04:15.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:15.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.410332+0000) 2019-10-13T14:04:15.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:15.403+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.410724+0000) 2019-10-13T14:04:15.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:15.409+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.411415+0000) 2019-10-13T14:04:15.419 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:15.413+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.416818+0000) 2019-10-13T14:04:15.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:15.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:15.629 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:15.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.626916+0000) 2019-10-13T14:04:15.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:15.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:15.914574+0000) 2019-10-13T14:04:16.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:16.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:16.593566+0000) 2019-10-13T14:04:16.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:16.841+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:16.843475+0000) 2019-10-13T14:04:18.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:18.855+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:18.861586+0000) 2019-10-13T14:04:18.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:18.855+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:18.861717+0000) 2019-10-13T14:04:20.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:20.415+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.420296+0000) 2019-10-13T14:04:20.423 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:20.415+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.420616+0000) 2019-10-13T14:04:20.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:20.415+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.421789+0000) 2019-10-13T14:04:20.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:20.415+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.422831+0000) 2019-10-13T14:04:20.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:20.415+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.423119+0000) 2019-10-13T14:04:20.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:20.419+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.423709+0000) 2019-10-13T14:04:20.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:20.421+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.424079+0000) 2019-10-13T14:04:20.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:20.425+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:20.429092+0000) 2019-10-13T14:04:20.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:20.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:20.503 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:04:20.503 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:20.506 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:04:20.507 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:21.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:21.225+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:21.226890+0000) 2019-10-13T14:04:22.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:22.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.602743+0000) 2019-10-13T14:04:25.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:25.427+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.433872+0000) 2019-10-13T14:04:25.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:25.427+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.434845+0000) 2019-10-13T14:04:25.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:25.427+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.435061+0000) 2019-10-13T14:04:25.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:25.431+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.435366+0000) 2019-10-13T14:04:25.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:25.431+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.435809+0000) 2019-10-13T14:04:25.439 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:25.431+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.436655+0000) 2019-10-13T14:04:25.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:25.433+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.437342+0000) 2019-10-13T14:04:25.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:25.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.440588+0000) 2019-10-13T14:04:25.445 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:25.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:25.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:25.619+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.627094+0000) 2019-10-13T14:04:25.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:25.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:25.914759+0000) 2019-10-13T14:04:26.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:26.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:26.593760+0000) 2019-10-13T14:04:26.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:26.841+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:26.843665+0000) 2019-10-13T14:04:28.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:28.859+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:28.861781+0000) 2019-10-13T14:04:28.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:28.859+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:28.861924+0000) 2019-10-13T14:04:30.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:30.439+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.445572+0000) 2019-10-13T14:04:30.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:30.439+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.446234+0000) 2019-10-13T14:04:30.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:30.439+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.446621+0000) 2019-10-13T14:04:30.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:30.443+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.447996+0000) 2019-10-13T14:04:30.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:30.445+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.448597+0000) 2019-10-13T14:04:30.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:30.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.448530+0000) 2019-10-13T14:04:30.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:30.443+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.449241+0000) 2019-10-13T14:04:30.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:30.453+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:30.456188+0000) 2019-10-13T14:04:30.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:30.451+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:31.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:31.229+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:31.233353+0000) 2019-10-13T14:04:32.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:32.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.602943+0000) 2019-10-13T14:04:35.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:35.455+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.458624+0000) 2019-10-13T14:04:35.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:35.455+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.459385+0000) 2019-10-13T14:04:35.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:35.455+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.459723+0000) 2019-10-13T14:04:35.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:35.455+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.460542+0000) 2019-10-13T14:04:35.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:35.455+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.461676+0000) 2019-10-13T14:04:35.465 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:35.455+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.462180+0000) 2019-10-13T14:04:35.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:35.461+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.462761+0000) 2019-10-13T14:04:35.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:35.465+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.466961+0000) 2019-10-13T14:04:35.471 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:35.463+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187270 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:35.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:35.623+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.627225+0000) 2019-10-13T14:04:35.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:35.907+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:35.914988+0000) 2019-10-13T14:04:36.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:36.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:36.593956+0000) 2019-10-13T14:04:36.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:36.841+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:36.843878+0000) 2019-10-13T14:04:38.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:38.859+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:38.862017+0000) 2019-10-13T14:04:38.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:38.859+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:38.862162+0000) 2019-10-13T14:04:40.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:40.467+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.472021+0000) 2019-10-13T14:04:40.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:40.469+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.472575+0000) 2019-10-13T14:04:40.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:40.467+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.472841+0000) 2019-10-13T14:04:40.476 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:40.467+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.473520+0000) 2019-10-13T14:04:40.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:40.467+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.473869+0000) 2019-10-13T14:04:40.477 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:40.467+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.474767+0000) 2019-10-13T14:04:40.478 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:40.467+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.475001+0000) 2019-10-13T14:04:40.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:40.475+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:40.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:40.481+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:40.485171+0000) 2019-10-13T14:04:41.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:41.237+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:41.239765+0000) 2019-10-13T14:04:42.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:42.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.603193+0000) 2019-10-13T14:04:45.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:45.483+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.485227+0000) 2019-10-13T14:04:45.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:45.483+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.485656+0000) 2019-10-13T14:04:45.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:45.485+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.486302+0000) 2019-10-13T14:04:45.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:45.483+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.486254+0000) 2019-10-13T14:04:45.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:45.483+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.486813+0000) 2019-10-13T14:04:45.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:45.483+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.487109+0000) 2019-10-13T14:04:45.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:45.487+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.487837+0000) 2019-10-13T14:04:45.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:45.493+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.494966+0000) 2019-10-13T14:04:45.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:45.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:45.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:45.627+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.627353+0000) 2019-10-13T14:04:45.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:45.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:45.915169+0000) 2019-10-13T14:04:46.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:46.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:46.594186+0000) 2019-10-13T14:04:46.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:46.840+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:46.844085+0000) 2019-10-13T14:04:48.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:48.859+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:48.862222+0000) 2019-10-13T14:04:48.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:48.859+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:48.862357+0000) 2019-10-13T14:04:50.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:50.495+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.497728+0000) 2019-10-13T14:04:50.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:50.495+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.498591+0000) 2019-10-13T14:04:50.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:50.495+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.499175+0000) 2019-10-13T14:04:50.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:50.499+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.499534+0000) 2019-10-13T14:04:50.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:50.496+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.500298+0000) 2019-10-13T14:04:50.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:50.499+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.500705+0000) 2019-10-13T14:04:50.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:50.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.501656+0000) 2019-10-13T14:04:50.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:50.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187370 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:50.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:50.508+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:50.511304+0000) 2019-10-13T14:04:50.547 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:04:50.547 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:50.551 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:04:50.551 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:51.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:51.244+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:51.246188+0000) 2019-10-13T14:04:52.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:52.603+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.603425+0000) 2019-10-13T14:04:55.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:55.511+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.512016+0000) 2019-10-13T14:04:55.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:55.511+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.512503+0000) 2019-10-13T14:04:55.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:55.508+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.513833+0000) 2019-10-13T14:04:55.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:04:55.511+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.514265+0000) 2019-10-13T14:04:55.517 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:55.511+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.514636+0000) 2019-10-13T14:04:55.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:55.511+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.515052+0000) 2019-10-13T14:04:55.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:55.511+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.515218+0000) 2019-10-13T14:04:55.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:04:55.516+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.519961+0000) 2019-10-13T14:04:55.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:04:55.519+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:04:55.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:04:55.627+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.627561+0000) 2019-10-13T14:04:55.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:04:55.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:55.915390+0000) 2019-10-13T14:04:56.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:04:56.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:56.594401+0000) 2019-10-13T14:04:56.846 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:04:56.840+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:56.844280+0000) 2019-10-13T14:04:58.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:04:58.858+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:58.862421+0000) 2019-10-13T14:04:58.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:04:58.858+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:58.862572+0000) 2019-10-13T14:05:00.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:00.522+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.524383+0000) 2019-10-13T14:05:00.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:00.522+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.524666+0000) 2019-10-13T14:05:00.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:00.520+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.525799+0000) 2019-10-13T14:05:00.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:00.522+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.525391+0000) 2019-10-13T14:05:00.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:00.522+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.525975+0000) 2019-10-13T14:05:00.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:00.522+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.526778+0000) 2019-10-13T14:05:00.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:00.526+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.527760+0000) 2019-10-13T14:05:00.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:00.532+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:00.534293+0000) 2019-10-13T14:05:00.537 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:00.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:01.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:01.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:01.252524+0000) 2019-10-13T14:05:02.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:02.602+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.603629+0000) 2019-10-13T14:05:05.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:05.534+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.537120+0000) 2019-10-13T14:05:05.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:05.534+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.537961+0000) 2019-10-13T14:05:05.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:05.534+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.538677+0000) 2019-10-13T14:05:05.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:05.538+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.539577+0000) 2019-10-13T14:05:05.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:05.538+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.539618+0000) 2019-10-13T14:05:05.543 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:05.534+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.539115+0000) 2019-10-13T14:05:05.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:05.536+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.540501+0000) 2019-10-13T14:05:05.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:05.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:05.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:05.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.547466+0000) 2019-10-13T14:05:05.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:05.626+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.627779+0000) 2019-10-13T14:05:05.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:05.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:05.915529+0000) 2019-10-13T14:05:06.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:06.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:06.594594+0000) 2019-10-13T14:05:06.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:06.840+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:06.844496+0000) 2019-10-13T14:05:08.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:08.858+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:08.862637+0000) 2019-10-13T14:05:08.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:08.858+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:08.862796+0000) 2019-10-13T14:05:10.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:10.546+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.549305+0000) 2019-10-13T14:05:10.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:10.546+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.550333+0000) 2019-10-13T14:05:10.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:10.546+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.550659+0000) 2019-10-13T14:05:10.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:10.546+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.551276+0000) 2019-10-13T14:05:10.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:10.550+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.551852+0000) 2019-10-13T14:05:10.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:10.550+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.552196+0000) 2019-10-13T14:05:10.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:10.548+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.553621+0000) 2019-10-13T14:05:10.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:10.556+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:10.558987+0000) 2019-10-13T14:05:10.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:10.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:11.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:11.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:11.258854+0000) 2019-10-13T14:05:12.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:12.602+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.603804+0000) 2019-10-13T14:05:15.565 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:15.558+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.562357+0000) 2019-10-13T14:05:15.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:15.558+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.563064+0000) 2019-10-13T14:05:15.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:15.562+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.563673+0000) 2019-10-13T14:05:15.567 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:15.562+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.563916+0000) 2019-10-13T14:05:15.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:15.562+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.564554+0000) 2019-10-13T14:05:15.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:15.562+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.564848+0000) 2019-10-13T14:05:15.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:15.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.565898+0000) 2019-10-13T14:05:15.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:15.568+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.571986+0000) 2019-10-13T14:05:15.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:15.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187534 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:15.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:15.626+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.627988+0000) 2019-10-13T14:05:15.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:15.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:15.915752+0000) 2019-10-13T14:05:16.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:16.594+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:16.594814+0000) 2019-10-13T14:05:16.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:16.840+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:16.844710+0000) 2019-10-13T14:05:18.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:18.858+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:18.862856+0000) 2019-10-13T14:05:18.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:18.858+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:18.862973+0000) 2019-10-13T14:05:20.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:20.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.575727+0000) 2019-10-13T14:05:20.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:20.574+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.576620+0000) 2019-10-13T14:05:20.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:20.574+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.575448+0000) 2019-10-13T14:05:20.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:20.574+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.577932+0000) 2019-10-13T14:05:20.582 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:20.576+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.578910+0000) 2019-10-13T14:05:20.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:20.574+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.578683+0000) 2019-10-13T14:05:20.582 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:20.574+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.578956+0000) 2019-10-13T14:05:20.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:20.580+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:20.584127+0000) 2019-10-13T14:05:20.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:20.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:20.666 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:05:20.666 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:20.670 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:05:20.670 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:21.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:21.264+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:21.265097+0000) 2019-10-13T14:05:22.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:22.602+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.604037+0000) 2019-10-13T14:05:25.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:25.586+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.587913+0000) 2019-10-13T14:05:25.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:25.586+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.588605+0000) 2019-10-13T14:05:25.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:25.586+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.589279+0000) 2019-10-13T14:05:25.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:25.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.589957+0000) 2019-10-13T14:05:25.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:25.586+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.590621+0000) 2019-10-13T14:05:25.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:25.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.591350+0000) 2019-10-13T14:05:25.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:25.590+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.591766+0000) 2019-10-13T14:05:25.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:25.592+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.596844+0000) 2019-10-13T14:05:25.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:25.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187601 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:25.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:25.626+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.628201+0000) 2019-10-13T14:05:25.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:25.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:25.915916+0000) 2019-10-13T14:05:26.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:26.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:26.595003+0000) 2019-10-13T14:05:26.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:26.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:26.844905+0000) 2019-10-13T14:05:28.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:28.862+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:28.863058+0000) 2019-10-13T14:05:28.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:28.862+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:28.863154+0000) 2019-10-13T14:05:30.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:30.598+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.601318+0000) 2019-10-13T14:05:30.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:30.598+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.601433+0000) 2019-10-13T14:05:30.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:30.598+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.602168+0000) 2019-10-13T14:05:30.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:30.598+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.602607+0000) 2019-10-13T14:05:30.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:30.598+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.602862+0000) 2019-10-13T14:05:30.608 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:30.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.603846+0000) 2019-10-13T14:05:30.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:30.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.604244+0000) 2019-10-13T14:05:30.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:30.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187634 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:30.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:30.608+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:30.611405+0000) 2019-10-13T14:05:31.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:31.268+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:31.271511+0000) 2019-10-13T14:05:32.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:32.602+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.604256+0000) 2019-10-13T14:05:35.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:35.610+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.614438+0000) 2019-10-13T14:05:35.617 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:35.612+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.615053+0000) 2019-10-13T14:05:35.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:35.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.615977+0000) 2019-10-13T14:05:35.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:35.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.616885+0000) 2019-10-13T14:05:35.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:35.614+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.617010+0000) 2019-10-13T14:05:35.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:35.614+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.617523+0000) 2019-10-13T14:05:35.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:35.614+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.618277+0000) 2019-10-13T14:05:35.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:35.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:35.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:35.620+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.625936+0000) 2019-10-13T14:05:35.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:35.626+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.628329+0000) 2019-10-13T14:05:35.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:35.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:35.916111+0000) 2019-10-13T14:05:36.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:36.590+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:36.595201+0000) 2019-10-13T14:05:36.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:36.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:36.845113+0000) 2019-10-13T14:05:38.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:38.862+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:38.863266+0000) 2019-10-13T14:05:38.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:38.862+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:38.863359+0000) 2019-10-13T14:05:40.630 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:40.621+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.627118+0000) 2019-10-13T14:05:40.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:40.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.627610+0000) 2019-10-13T14:05:40.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:40.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.628397+0000) 2019-10-13T14:05:40.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:40.623+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.629359+0000) 2019-10-13T14:05:40.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:40.625+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.629464+0000) 2019-10-13T14:05:40.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:40.625+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.630445+0000) 2019-10-13T14:05:40.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:40.625+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.630920+0000) 2019-10-13T14:05:40.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:40.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:40.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:40.635+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:40.638772+0000) 2019-10-13T14:05:41.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:41.275+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:41.277942+0000) 2019-10-13T14:05:42.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:42.601+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.604455+0000) 2019-10-13T14:05:45.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:45.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.628492+0000) 2019-10-13T14:05:45.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:45.637+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.640816+0000) 2019-10-13T14:05:45.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:45.637+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.641950+0000) 2019-10-13T14:05:45.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:45.637+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.642403+0000) 2019-10-13T14:05:45.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:45.637+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.643151+0000) 2019-10-13T14:05:45.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:45.639+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.643924+0000) 2019-10-13T14:05:45.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:45.641+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.644060+0000) 2019-10-13T14:05:45.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:45.641+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.645119+0000) 2019-10-13T14:05:45.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:45.647+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.650914+0000) 2019-10-13T14:05:45.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:45.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:45.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:45.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:45.916333+0000) 2019-10-13T14:05:46.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:46.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:46.595377+0000) 2019-10-13T14:05:46.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:46.839+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:46.845312+0000) 2019-10-13T14:05:48.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:48.861+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:48.863531+0000) 2019-10-13T14:05:48.866 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:48.861+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:48.863521+0000) 2019-10-13T14:05:50.656 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:50.649+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.653823+0000) 2019-10-13T14:05:50.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:50.649+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.654744+0000) 2019-10-13T14:05:50.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:50.653+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.655386+0000) 2019-10-13T14:05:50.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:50.653+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.655792+0000) 2019-10-13T14:05:50.659 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:50.653+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.656610+0000) 2019-10-13T14:05:50.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:50.651+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.657679+0000) 2019-10-13T14:05:50.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:50.653+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.656938+0000) 2019-10-13T14:05:50.666 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:50.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:50.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:50.659+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:50.664301+0000) 2019-10-13T14:05:50.786 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:05:50.786 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:50.789 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:05:50.790 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:51.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:51.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:51.284455+0000) 2019-10-13T14:05:52.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:52.601+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.604652+0000) 2019-10-13T14:05:55.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:55.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.628686+0000) 2019-10-13T14:05:55.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:55.665+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.667757+0000) 2019-10-13T14:05:55.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:05:55.665+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.668139+0000) 2019-10-13T14:05:55.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:55.665+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.668586+0000) 2019-10-13T14:05:55.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:05:55.665+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.668785+0000) 2019-10-13T14:05:55.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:55.665+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.669798+0000) 2019-10-13T14:05:55.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:55.667+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.670235+0000) 2019-10-13T14:05:55.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:55.665+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.670338+0000) 2019-10-13T14:05:55.679 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:05:55.673+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187798 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:05:55.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:05:55.675+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.678555+0000) 2019-10-13T14:05:55.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:05:55.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:55.916497+0000) 2019-10-13T14:05:56.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:05:56.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:56.595555+0000) 2019-10-13T14:05:56.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:05:56.839+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:56.845508+0000) 2019-10-13T14:05:58.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:05:58.861+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:58.863753+0000) 2019-10-13T14:05:58.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:05:58.861+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:58.863753+0000) 2019-10-13T14:06:00.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:00.677+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.680009+0000) 2019-10-13T14:06:00.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:00.677+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.679563+0000) 2019-10-13T14:06:00.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:00.677+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.680695+0000) 2019-10-13T14:06:00.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:00.677+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.681408+0000) 2019-10-13T14:06:00.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:00.677+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.681880+0000) 2019-10-13T14:06:00.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:00.677+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.682975+0000) 2019-10-13T14:06:00.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:00.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.683441+0000) 2019-10-13T14:06:00.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:00.683+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:00.690095+0000) 2019-10-13T14:06:00.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:00.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:01.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:01.291+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:01.295192+0000) 2019-10-13T14:06:02.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:02.601+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.604855+0000) 2019-10-13T14:06:05.631 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:05.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.628890+0000) 2019-10-13T14:06:05.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:05.689+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.693382+0000) 2019-10-13T14:06:05.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:05.689+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.693981+0000) 2019-10-13T14:06:05.697 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:05.689+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.695230+0000) 2019-10-13T14:06:05.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:05.693+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.695721+0000) 2019-10-13T14:06:05.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:05.693+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.696320+0000) 2019-10-13T14:06:05.699 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:05.693+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.696681+0000) 2019-10-13T14:06:05.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:05.691+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.698062+0000) 2019-10-13T14:06:05.707 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:05.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:05.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:05.699+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.704688+0000) 2019-10-13T14:06:05.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:05.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:05.916683+0000) 2019-10-13T14:06:06.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:06.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:06.595725+0000) 2019-10-13T14:06:06.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:06.839+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:06.845708+0000) 2019-10-13T14:06:08.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:08.861+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:08.864001+0000) 2019-10-13T14:06:08.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:08.861+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:08.864001+0000) 2019-10-13T14:06:10.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:10.701+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.707200+0000) 2019-10-13T14:06:10.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:10.705+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.707639+0000) 2019-10-13T14:06:10.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:10.705+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.708070+0000) 2019-10-13T14:06:10.711 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:10.705+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.708693+0000) 2019-10-13T14:06:10.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:10.705+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.709858+0000) 2019-10-13T14:06:10.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:10.705+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.710363+0000) 2019-10-13T14:06:10.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:10.707+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.711380+0000) 2019-10-13T14:06:10.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:10.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:10.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:10.715+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:10.719637+0000) 2019-10-13T14:06:11.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:11.295+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:11.301505+0000) 2019-10-13T14:06:12.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:12.601+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.605087+0000) 2019-10-13T14:06:15.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:15.625+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.629091+0000) 2019-10-13T14:06:15.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:15.715+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.721337+0000) 2019-10-13T14:06:15.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:15.717+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.721639+0000) 2019-10-13T14:06:15.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:15.717+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.721932+0000) 2019-10-13T14:06:15.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:15.717+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.722270+0000) 2019-10-13T14:06:15.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:15.717+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.722663+0000) 2019-10-13T14:06:15.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:15.721+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.723860+0000) 2019-10-13T14:06:15.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:15.721+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.724512+0000) 2019-10-13T14:06:15.733 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:15.725+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:15.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:15.731+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.734739+0000) 2019-10-13T14:06:15.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:15.913+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:15.916894+0000) 2019-10-13T14:06:16.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:16.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:16.595898+0000) 2019-10-13T14:06:16.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:16.843+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:16.845923+0000) 2019-10-13T14:06:18.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:18.861+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:18.864231+0000) 2019-10-13T14:06:18.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:18.861+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:18.864231+0000) 2019-10-13T14:06:20.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:20.729+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.733741+0000) 2019-10-13T14:06:20.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:20.729+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.734018+0000) 2019-10-13T14:06:20.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:20.729+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.734591+0000) 2019-10-13T14:06:20.738 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:20.729+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.735176+0000) 2019-10-13T14:06:20.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:20.733+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.735428+0000) 2019-10-13T14:06:20.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:20.731+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.736775+0000) 2019-10-13T14:06:20.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:20.733+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.737109+0000) 2019-10-13T14:06:20.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:20.741+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187963 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:20.746 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:20.739+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:20.743894+0000) 2019-10-13T14:06:20.905 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:06:20.906 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:20.909 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:06:20.910 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:21.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:21.307+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:21.307941+0000) 2019-10-13T14:06:22.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:22.601+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.605286+0000) 2019-10-13T14:06:25.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:25.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.629304+0000) 2019-10-13T14:06:25.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:25.740+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.746523+0000) 2019-10-13T14:06:25.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:25.744+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.747895+0000) 2019-10-13T14:06:25.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:25.744+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.748251+0000) 2019-10-13T14:06:25.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:25.743+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.748989+0000) 2019-10-13T14:06:25.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:25.744+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.748944+0000) 2019-10-13T14:06:25.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:25.744+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.749991+0000) 2019-10-13T14:06:25.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:25.744+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.750400+0000) 2019-10-13T14:06:25.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:25.751+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.756019+0000) 2019-10-13T14:06:25.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:25.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 187996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:25.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:25.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:25.917059+0000) 2019-10-13T14:06:26.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:26.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:26.596116+0000) 2019-10-13T14:06:26.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:26.843+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:26.846097+0000) 2019-10-13T14:06:28.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:28.860+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:28.864461+0000) 2019-10-13T14:06:28.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:28.860+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:28.864461+0000) 2019-10-13T14:06:30.762 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:30.755+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.759758+0000) 2019-10-13T14:06:30.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:30.756+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.760566+0000) 2019-10-13T14:06:30.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:30.756+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.760952+0000) 2019-10-13T14:06:30.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:30.756+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.761035+0000) 2019-10-13T14:06:30.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:30.756+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.761616+0000) 2019-10-13T14:06:30.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:30.756+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.762557+0000) 2019-10-13T14:06:30.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:30.756+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.763057+0000) 2019-10-13T14:06:30.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:30.763+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:30.766816+0000) 2019-10-13T14:06:30.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:30.764+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:31.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:31.311+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:31.314306+0000) 2019-10-13T14:06:32.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:32.604+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.605510+0000) 2019-10-13T14:06:35.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:35.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.629514+0000) 2019-10-13T14:06:35.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:35.768+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.772926+0000) 2019-10-13T14:06:35.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:35.768+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.774333+0000) 2019-10-13T14:06:35.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:35.770+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.775170+0000) 2019-10-13T14:06:35.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:35.772+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.775518+0000) 2019-10-13T14:06:35.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:35.772+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.776188+0000) 2019-10-13T14:06:35.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:35.772+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.776725+0000) 2019-10-13T14:06:35.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:35.772+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.777260+0000) 2019-10-13T14:06:35.782 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:35.774+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.780243+0000) 2019-10-13T14:06:35.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:35.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:35.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:35.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:35.917257+0000) 2019-10-13T14:06:36.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:36.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:36.596305+0000) 2019-10-13T14:06:36.849 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:36.842+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:36.846340+0000) 2019-10-13T14:06:38.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:38.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:38.864687+0000) 2019-10-13T14:06:38.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:38.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:38.864688+0000) 2019-10-13T14:06:40.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:40.780+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.786340+0000) 2019-10-13T14:06:40.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:40.780+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.787178+0000) 2019-10-13T14:06:40.790 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:40.784+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.787539+0000) 2019-10-13T14:06:40.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:40.784+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.788223+0000) 2019-10-13T14:06:40.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:40.782+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.789484+0000) 2019-10-13T14:06:40.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:40.784+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.790348+0000) 2019-10-13T14:06:40.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:40.784+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.790639+0000) 2019-10-13T14:06:40.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:40.790+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:40.796337+0000) 2019-10-13T14:06:40.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:40.792+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:41.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:41.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:41.321336+0000) 2019-10-13T14:06:42.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:42.604+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.605742+0000) 2019-10-13T14:06:45.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:45.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.629717+0000) 2019-10-13T14:06:45.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:45.796+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.800505+0000) 2019-10-13T14:06:45.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:45.796+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.800778+0000) 2019-10-13T14:06:45.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:45.796+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.801421+0000) 2019-10-13T14:06:45.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:45.794+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.801946+0000) 2019-10-13T14:06:45.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:45.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.802025+0000) 2019-10-13T14:06:45.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:45.796+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.802353+0000) 2019-10-13T14:06:45.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:45.796+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.802751+0000) 2019-10-13T14:06:45.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:45.802+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.806625+0000) 2019-10-13T14:06:45.811 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:45.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188128 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:45.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:45.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:45.917462+0000) 2019-10-13T14:06:46.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:46.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:46.596479+0000) 2019-10-13T14:06:46.849 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:46.842+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:46.846490+0000) 2019-10-13T14:06:48.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:48.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:48.864866+0000) 2019-10-13T14:06:48.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:48.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:48.864868+0000) 2019-10-13T14:06:50.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:50.808+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.811443+0000) 2019-10-13T14:06:50.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:50.808+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.812066+0000) 2019-10-13T14:06:50.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:50.808+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.813113+0000) 2019-10-13T14:06:50.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:50.806+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.813890+0000) 2019-10-13T14:06:50.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:50.808+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.814586+0000) 2019-10-13T14:06:50.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:50.808+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.814913+0000) 2019-10-13T14:06:50.818 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:50.812+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.815407+0000) 2019-10-13T14:06:50.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:50.814+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:50.820442+0000) 2019-10-13T14:06:50.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:50.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188162 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:51.016 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:06:51.016 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:51.020 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:06:51.020 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:51.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:51.326+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:51.327606+0000) 2019-10-13T14:06:52.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:52.604+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.605979+0000) 2019-10-13T14:06:55.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:55.624+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.629944+0000) 2019-10-13T14:06:55.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:55.818+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.825077+0000) 2019-10-13T14:06:55.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:06:55.820+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.825651+0000) 2019-10-13T14:06:55.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:55.820+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.825910+0000) 2019-10-13T14:06:55.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:55.820+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.827006+0000) 2019-10-13T14:06:55.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:06:55.824+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.827610+0000) 2019-10-13T14:06:55.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:55.824+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.828156+0000) 2019-10-13T14:06:55.831 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:55.824+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.828962+0000) 2019-10-13T14:06:55.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:06:55.830+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.834682+0000) 2019-10-13T14:06:55.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:06:55.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:06:55.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:06:55.912+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:55.917623+0000) 2019-10-13T14:06:56.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:06:56.592+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:56.596670+0000) 2019-10-13T14:06:56.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:06:56.846+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:56.846690+0000) 2019-10-13T14:06:58.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:06:58.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:58.865123+0000) 2019-10-13T14:06:58.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:06:58.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:58.865123+0000) 2019-10-13T14:07:00.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:00.832+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.838344+0000) 2019-10-13T14:07:00.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:00.836+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.839632+0000) 2019-10-13T14:07:00.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:00.836+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.839875+0000) 2019-10-13T14:07:00.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:00.836+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.840105+0000) 2019-10-13T14:07:00.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:00.836+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.841416+0000) 2019-10-13T14:07:00.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:00.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.842094+0000) 2019-10-13T14:07:00.845 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:00.836+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.842256+0000) 2019-10-13T14:07:00.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:00.838+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:00.845652+0000) 2019-10-13T14:07:00.851 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:00.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188228 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:01.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:01.330+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:01.333978+0000) 2019-10-13T14:07:02.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:02.604+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.606207+0000) 2019-10-13T14:07:05.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:05.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.630178+0000) 2019-10-13T14:07:05.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:05.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.852060+0000) 2019-10-13T14:07:05.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:05.848+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.852288+0000) 2019-10-13T14:07:05.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:05.848+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.852803+0000) 2019-10-13T14:07:05.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:05.848+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.854049+0000) 2019-10-13T14:07:05.857 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:05.848+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.854415+0000) 2019-10-13T14:07:05.857 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:05.848+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.854867+0000) 2019-10-13T14:07:05.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:05.850+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.855873+0000) 2019-10-13T14:07:05.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:05.854+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.859593+0000) 2019-10-13T14:07:05.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:05.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188259 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:05.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:05.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:05.917832+0000) 2019-10-13T14:07:06.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:06.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:06.596846+0000) 2019-10-13T14:07:06.849 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:06.846+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:06.846914+0000) 2019-10-13T14:07:08.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:08.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:08.865346+0000) 2019-10-13T14:07:08.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:08.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:08.865346+0000) 2019-10-13T14:07:10.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:10.859+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.866140+0000) 2019-10-13T14:07:10.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:10.859+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.866475+0000) 2019-10-13T14:07:10.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:10.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.867154+0000) 2019-10-13T14:07:10.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:10.863+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.867371+0000) 2019-10-13T14:07:10.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:10.862+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.867982+0000) 2019-10-13T14:07:10.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:10.863+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.868665+0000) 2019-10-13T14:07:10.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:10.863+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.869104+0000) 2019-10-13T14:07:10.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:10.866+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:10.871320+0000) 2019-10-13T14:07:10.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:10.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:11.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:11.338+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:11.340479+0000) 2019-10-13T14:07:12.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:12.599+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.606386+0000) 2019-10-13T14:07:15.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:15.627+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.630378+0000) 2019-10-13T14:07:15.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:15.875+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.878453+0000) 2019-10-13T14:07:15.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:15.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.879418+0000) 2019-10-13T14:07:15.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:15.875+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.880249+0000) 2019-10-13T14:07:15.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:15.875+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.880819+0000) 2019-10-13T14:07:15.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:15.875+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.881129+0000) 2019-10-13T14:07:15.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:15.875+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.881964+0000) 2019-10-13T14:07:15.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:15.878+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.882531+0000) 2019-10-13T14:07:15.889 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:15.882+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.887182+0000) 2019-10-13T14:07:15.891 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:15.883+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:15.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:15.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:15.918015+0000) 2019-10-13T14:07:16.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:16.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:16.597020+0000) 2019-10-13T14:07:16.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:16.842+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:16.847104+0000) 2019-10-13T14:07:18.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:18.859+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:18.865590+0000) 2019-10-13T14:07:18.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:18.859+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:18.865589+0000) 2019-10-13T14:07:20.895 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:20.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.892053+0000) 2019-10-13T14:07:20.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:20.887+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.892365+0000) 2019-10-13T14:07:20.896 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:20.886+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.893176+0000) 2019-10-13T14:07:20.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:20.887+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.893423+0000) 2019-10-13T14:07:20.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:20.887+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.893564+0000) 2019-10-13T14:07:20.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:20.891+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.895383+0000) 2019-10-13T14:07:20.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:20.891+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.895761+0000) 2019-10-13T14:07:20.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:20.894+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:20.900650+0000) 2019-10-13T14:07:20.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:20.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188360 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:21.144 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:07:21.145 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:21.148 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:07:21.148 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:21.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:21.346+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:21.346653+0000) 2019-10-13T14:07:22.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:22.603+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.606616+0000) 2019-10-13T14:07:25.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:25.623+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.630599+0000) 2019-10-13T14:07:25.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:25.899+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.904183+0000) 2019-10-13T14:07:25.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:25.899+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.905125+0000) 2019-10-13T14:07:25.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:25.899+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.905366+0000) 2019-10-13T14:07:25.909 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:25.899+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.906511+0000) 2019-10-13T14:07:25.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:25.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.907420+0000) 2019-10-13T14:07:25.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:25.903+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.907859+0000) 2019-10-13T14:07:25.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:25.903+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.908321+0000) 2019-10-13T14:07:25.917 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:25.914+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.914874+0000) 2019-10-13T14:07:25.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:25.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:25.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:25.911+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:25.918148+0000) 2019-10-13T14:07:26.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:26.591+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:26.597223+0000) 2019-10-13T14:07:26.849 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:26.846+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:26.847312+0000) 2019-10-13T14:07:28.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:28.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:28.865813+0000) 2019-10-13T14:07:28.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:28.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:28.865813+0000) 2019-10-13T14:07:30.920 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:30.911+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.917870+0000) 2019-10-13T14:07:30.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:30.911+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.918572+0000) 2019-10-13T14:07:30.922 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:30.917+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.919602+0000) 2019-10-13T14:07:30.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:30.915+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.920798+0000) 2019-10-13T14:07:30.924 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:30.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.921398+0000) 2019-10-13T14:07:30.924 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:30.915+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.921693+0000) 2019-10-13T14:07:30.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:30.915+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.922269+0000) 2019-10-13T14:07:30.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:30.925+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:30.926382+0000) 2019-10-13T14:07:30.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:30.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:31.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:31.349+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:31.353056+0000) 2019-10-13T14:07:32.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:32.603+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.606873+0000) 2019-10-13T14:07:35.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:35.627+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.630799+0000) 2019-10-13T14:07:35.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:35.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.918337+0000) 2019-10-13T14:07:35.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:35.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.930641+0000) 2019-10-13T14:07:35.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:35.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.931633+0000) 2019-10-13T14:07:35.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:35.927+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.932145+0000) 2019-10-13T14:07:35.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:35.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.932740+0000) 2019-10-13T14:07:35.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:35.929+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.933425+0000) 2019-10-13T14:07:35.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:35.927+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.934037+0000) 2019-10-13T14:07:35.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:35.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.934841+0000) 2019-10-13T14:07:35.942 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:35.937+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:35.940375+0000) 2019-10-13T14:07:35.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:35.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:36.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:36.595+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:36.597396+0000) 2019-10-13T14:07:36.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:36.845+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:36.847546+0000) 2019-10-13T14:07:38.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:38.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:38.866037+0000) 2019-10-13T14:07:38.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:38.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:38.866037+0000) 2019-10-13T14:07:40.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:40.943+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.943433+0000) 2019-10-13T14:07:40.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:40.943+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.943775+0000) 2019-10-13T14:07:40.948 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:40.943+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.945046+0000) 2019-10-13T14:07:40.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:40.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.945441+0000) 2019-10-13T14:07:40.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:40.943+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.945899+0000) 2019-10-13T14:07:40.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:40.945+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.946382+0000) 2019-10-13T14:07:40.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:40.943+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.946507+0000) 2019-10-13T14:07:40.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:40.949+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:40.951274+0000) 2019-10-13T14:07:40.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:40.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:41.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:41.357+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:41.359405+0000) 2019-10-13T14:07:42.609 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:42.603+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.607041+0000) 2019-10-13T14:07:45.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:45.627+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.631000+0000) 2019-10-13T14:07:45.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:45.915+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.918546+0000) 2019-10-13T14:07:45.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:45.953+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.956824+0000) 2019-10-13T14:07:45.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:45.955+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.957030+0000) 2019-10-13T14:07:45.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:45.955+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.957662+0000) 2019-10-13T14:07:45.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:45.955+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.958962+0000) 2019-10-13T14:07:45.962 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:45.959+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.959581+0000) 2019-10-13T14:07:45.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:45.959+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.959720+0000) 2019-10-13T14:07:45.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:45.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.960307+0000) 2019-10-13T14:07:45.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:45.965+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:45.966675+0000) 2019-10-13T14:07:45.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:45.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:46.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:46.595+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:46.597570+0000) 2019-10-13T14:07:46.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:46.845+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:46.847747+0000) 2019-10-13T14:07:48.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:48.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:48.866293+0000) 2019-10-13T14:07:48.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:48.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:48.866293+0000) 2019-10-13T14:07:50.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:50.966+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.970616+0000) 2019-10-13T14:07:50.974 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:50.966+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.971006+0000) 2019-10-13T14:07:50.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:50.969+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.972317+0000) 2019-10-13T14:07:50.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:50.970+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.972503+0000) 2019-10-13T14:07:50.975 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:50.970+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.972617+0000) 2019-10-13T14:07:50.976 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:50.970+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.973464+0000) 2019-10-13T14:07:50.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:50.970+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.974089+0000) 2019-10-13T14:07:50.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:50.977+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:50.979337+0000) 2019-10-13T14:07:50.982 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:50.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:51.214 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:07:51.214 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:51.218 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:07:51.218 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:51.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:51.365+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:51.365865+0000) 2019-10-13T14:07:52.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:52.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.607248+0000) 2019-10-13T14:07:55.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:55.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.631216+0000) 2019-10-13T14:07:55.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:55.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.918783+0000) 2019-10-13T14:07:55.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:07:55.982+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.983660+0000) 2019-10-13T14:07:55.986 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:55.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.984330+0000) 2019-10-13T14:07:55.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:07:55.982+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.984427+0000) 2019-10-13T14:07:55.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:55.982+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.984732+0000) 2019-10-13T14:07:55.988 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:55.982+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.985074+0000) 2019-10-13T14:07:55.988 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:55.982+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.985431+0000) 2019-10-13T14:07:55.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:07:55.982+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.986810+0000) 2019-10-13T14:07:55.994 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:07:55.989+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:55.992397+0000) 2019-10-13T14:07:55.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:07:55.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:07:56.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:07:56.594+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:56.597776+0000) 2019-10-13T14:07:56.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:07:56.845+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:56.847940+0000) 2019-10-13T14:07:58.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:07:58.862+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:58.866518+0000) 2019-10-13T14:07:58.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:07:58.862+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:58.866517+0000) 2019-10-13T14:08:00.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:00.994+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.996085+0000) 2019-10-13T14:08:00.999 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:00.994+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.996525+0000) 2019-10-13T14:08:01.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:00.993+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.997355+0000) 2019-10-13T14:08:01.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:00.994+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.997773+0000) 2019-10-13T14:08:01.000 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:00.994+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.997475+0000) 2019-10-13T14:08:01.001 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:00.994+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.998625+0000) 2019-10-13T14:08:01.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:00.994+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:00.999122+0000) 2019-10-13T14:08:01.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:01.001+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:01.004665+0000) 2019-10-13T14:08:01.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:01.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:01.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:01.369+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:01.372431+0000) 2019-10-13T14:08:02.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:02.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.607495+0000) 2019-10-13T14:08:05.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:05.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:05.631426+0000) 2019-10-13T14:08:05.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:05.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:05.919004+0000) 2019-10-13T14:08:06.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:06.006+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.008677+0000) 2019-10-13T14:08:06.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:06.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.009723+0000) 2019-10-13T14:08:06.013 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:06.006+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.010547+0000) 2019-10-13T14:08:06.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:06.006+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.011033+0000) 2019-10-13T14:08:06.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:06.006+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.011226+0000) 2019-10-13T14:08:06.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:06.009+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.012218+0000) 2019-10-13T14:08:06.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:06.010+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.012510+0000) 2019-10-13T14:08:06.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:06.017+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.018440+0000) 2019-10-13T14:08:06.021 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:06.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:06.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:06.594+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.597982+0000) 2019-10-13T14:08:06.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:06.845+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:06.848118+0000) 2019-10-13T14:08:08.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:08.862+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:08.866743+0000) 2019-10-13T14:08:08.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:08.862+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:08.866743+0000) 2019-10-13T14:08:11.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:11.021+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.022646+0000) 2019-10-13T14:08:11.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:11.018+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.023268+0000) 2019-10-13T14:08:11.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:11.022+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.024577+0000) 2019-10-13T14:08:11.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:11.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.025515+0000) 2019-10-13T14:08:11.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:11.022+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.025618+0000) 2019-10-13T14:08:11.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:11.022+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.026074+0000) 2019-10-13T14:08:11.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:11.022+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.026438+0000) 2019-10-13T14:08:11.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:11.029+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.030349+0000) 2019-10-13T14:08:11.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:11.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:11.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:11.377+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.378945+0000) 2019-10-13T14:08:12.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:12.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.607686+0000) 2019-10-13T14:08:15.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:15.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:15.631631+0000) 2019-10-13T14:08:15.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:15.914+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:15.919214+0000) 2019-10-13T14:08:16.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:16.034+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.035473+0000) 2019-10-13T14:08:16.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:16.034+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.036706+0000) 2019-10-13T14:08:16.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:16.034+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.036841+0000) 2019-10-13T14:08:16.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:16.033+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.037425+0000) 2019-10-13T14:08:16.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:16.034+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.037959+0000) 2019-10-13T14:08:16.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:16.034+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.038405+0000) 2019-10-13T14:08:16.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:16.034+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.038997+0000) 2019-10-13T14:08:16.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:16.041+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.042495+0000) 2019-10-13T14:08:16.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:16.042+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:16.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:16.594+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.598209+0000) 2019-10-13T14:08:16.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:16.845+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:16.848305+0000) 2019-10-13T14:08:18.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:18.862+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.866984+0000) 2019-10-13T14:08:18.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:18.862+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.866984+0000) 2019-10-13T14:08:21.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:21.046+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.049119+0000) 2019-10-13T14:08:21.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:21.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.049888+0000) 2019-10-13T14:08:21.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:21.049+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.050431+0000) 2019-10-13T14:08:21.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:21.046+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.050379+0000) 2019-10-13T14:08:21.054 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:21.046+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.050760+0000) 2019-10-13T14:08:21.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:21.046+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.051114+0000) 2019-10-13T14:08:21.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:21.050+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.052052+0000) 2019-10-13T14:08:21.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:21.057+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.058525+0000) 2019-10-13T14:08:21.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:21.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:21.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:21.381+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.385460+0000) 2019-10-13T14:08:21.425 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:08:21.426 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:21.429 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:08:21.429 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:22.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:22.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:22.607895+0000) 2019-10-13T14:08:25.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:25.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:25.631841+0000) 2019-10-13T14:08:25.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:25.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:25.919426+0000) 2019-10-13T14:08:26.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:26.058+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.062932+0000) 2019-10-13T14:08:26.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:26.062+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.063459+0000) 2019-10-13T14:08:26.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:26.062+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.063775+0000) 2019-10-13T14:08:26.067 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:26.062+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.064286+0000) 2019-10-13T14:08:26.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:26.062+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.064613+0000) 2019-10-13T14:08:26.067 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:26.060+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.065302+0000) 2019-10-13T14:08:26.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:26.062+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.065280+0000) 2019-10-13T14:08:26.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:26.068+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.071998+0000) 2019-10-13T14:08:26.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:26.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188787 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:26.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:26.598+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.598388+0000) 2019-10-13T14:08:26.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:26.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:26.848494+0000) 2019-10-13T14:08:28.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:28.866+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.867214+0000) 2019-10-13T14:08:28.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:28.866+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.867214+0000) 2019-10-13T14:08:31.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:31.070+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.075273+0000) 2019-10-13T14:08:31.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:31.074+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.076539+0000) 2019-10-13T14:08:31.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:31.072+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.077570+0000) 2019-10-13T14:08:31.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:31.074+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.077192+0000) 2019-10-13T14:08:31.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:31.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.077613+0000) 2019-10-13T14:08:31.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:31.074+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.078040+0000) 2019-10-13T14:08:31.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:31.074+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.079026+0000) 2019-10-13T14:08:31.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:31.080+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.082240+0000) 2019-10-13T14:08:31.087 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:31.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:31.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:31.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.391885+0000) 2019-10-13T14:08:32.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:32.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:32.608069+0000) 2019-10-13T14:08:35.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:35.629+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:35.632062+0000) 2019-10-13T14:08:35.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:35.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:35.919590+0000) 2019-10-13T14:08:36.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:36.085+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.088865+0000) 2019-10-13T14:08:36.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:36.085+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.089348+0000) 2019-10-13T14:08:36.092 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:36.085+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.089632+0000) 2019-10-13T14:08:36.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:36.085+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.090414+0000) 2019-10-13T14:08:36.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:36.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.090988+0000) 2019-10-13T14:08:36.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:36.085+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.091022+0000) 2019-10-13T14:08:36.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:36.092+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.096462+0000) 2019-10-13T14:08:36.099 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:36.089+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.091829+0000) 2019-10-13T14:08:36.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:36.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:36.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:36.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.598567+0000) 2019-10-13T14:08:36.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:36.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:36.848644+0000) 2019-10-13T14:08:38.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:38.865+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.867511+0000) 2019-10-13T14:08:38.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:38.865+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.867511+0000) 2019-10-13T14:08:41.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:41.097+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.101251+0000) 2019-10-13T14:08:41.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:41.097+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.101479+0000) 2019-10-13T14:08:41.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:41.097+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.102961+0000) 2019-10-13T14:08:41.105 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:41.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.103140+0000) 2019-10-13T14:08:41.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:41.100+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.104230+0000) 2019-10-13T14:08:41.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:41.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.104477+0000) 2019-10-13T14:08:41.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:41.101+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.104901+0000) 2019-10-13T14:08:41.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:41.104+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.108482+0000) 2019-10-13T14:08:41.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:41.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188886 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:41.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:41.396+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.398188+0000) 2019-10-13T14:08:42.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:42.605+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:42.608268+0000) 2019-10-13T14:08:45.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:45.629+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:45.632294+0000) 2019-10-13T14:08:45.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:45.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:45.919805+0000) 2019-10-13T14:08:46.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:46.112+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.115204+0000) 2019-10-13T14:08:46.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:46.113+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.115356+0000) 2019-10-13T14:08:46.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:46.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.115760+0000) 2019-10-13T14:08:46.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:46.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.116374+0000) 2019-10-13T14:08:46.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:46.113+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.116643+0000) 2019-10-13T14:08:46.120 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:46.113+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.117346+0000) 2019-10-13T14:08:46.121 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:46.113+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.119277+0000) 2019-10-13T14:08:46.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:46.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:46.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:46.120+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.125572+0000) 2019-10-13T14:08:46.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:46.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.598762+0000) 2019-10-13T14:08:46.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:46.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:46.848825+0000) 2019-10-13T14:08:48.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:48.865+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.867737+0000) 2019-10-13T14:08:48.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:48.865+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.867736+0000) 2019-10-13T14:08:51.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:51.124+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.127251+0000) 2019-10-13T14:08:51.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:51.125+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.129548+0000) 2019-10-13T14:08:51.132 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:51.125+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.130013+0000) 2019-10-13T14:08:51.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:51.125+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.130287+0000) 2019-10-13T14:08:51.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:51.125+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.130966+0000) 2019-10-13T14:08:51.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:51.125+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.130605+0000) 2019-10-13T14:08:51.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:51.129+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.131380+0000) 2019-10-13T14:08:51.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:51.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:51.141 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:51.136+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.138951+0000) 2019-10-13T14:08:51.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:51.404+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.404621+0000) 2019-10-13T14:08:51.479 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:08:51.479 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:51.483 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:08:51.484 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:52.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:52.605+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:52.608470+0000) 2019-10-13T14:08:55.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:55.629+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:55.632495+0000) 2019-10-13T14:08:55.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:55.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:55.920008+0000) 2019-10-13T14:08:56.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:08:56.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.140752+0000) 2019-10-13T14:08:56.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:56.136+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.141762+0000) 2019-10-13T14:08:56.145 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:08:56.137+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.142212+0000) 2019-10-13T14:08:56.145 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:56.137+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.142551+0000) 2019-10-13T14:08:56.146 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:56.137+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.141913+0000) 2019-10-13T14:08:56.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:08:56.137+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.143121+0000) 2019-10-13T14:08:56.146 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:56.141+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.144188+0000) 2019-10-13T14:08:56.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:08:56.144+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.147271+0000) 2019-10-13T14:08:56.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:08:56.145+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 188984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:08:56.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:08:56.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.598946+0000) 2019-10-13T14:08:56.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:08:56.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:56.849013+0000) 2019-10-13T14:08:58.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:08:58.865+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.867972+0000) 2019-10-13T14:08:58.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:08:58.865+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.867972+0000) 2019-10-13T14:09:01.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:01.149+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.154863+0000) 2019-10-13T14:09:01.158 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:01.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.155359+0000) 2019-10-13T14:09:01.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:01.153+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.156387+0000) 2019-10-13T14:09:01.159 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:01.152+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.156953+0000) 2019-10-13T14:09:01.160 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:01.153+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.157214+0000) 2019-10-13T14:09:01.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:01.153+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.157350+0000) 2019-10-13T14:09:01.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:01.153+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.158069+0000) 2019-10-13T14:09:01.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:01.160+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.162807+0000) 2019-10-13T14:09:01.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:01.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:01.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:01.408+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.411421+0000) 2019-10-13T14:09:02.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:02.605+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:02.608702+0000) 2019-10-13T14:09:05.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:05.629+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:05.632710+0000) 2019-10-13T14:09:05.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:05.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:05.920243+0000) 2019-10-13T14:09:06.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:06.165+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.168354+0000) 2019-10-13T14:09:06.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:06.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.169008+0000) 2019-10-13T14:09:06.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:06.165+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.169401+0000) 2019-10-13T14:09:06.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:06.165+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.169870+0000) 2019-10-13T14:09:06.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:06.168+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.170566+0000) 2019-10-13T14:09:06.173 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:06.165+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.170750+0000) 2019-10-13T14:09:06.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:06.169+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.171622+0000) 2019-10-13T14:09:06.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:06.172+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.176572+0000) 2019-10-13T14:09:06.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:06.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:06.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:06.593+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.599144+0000) 2019-10-13T14:09:06.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:06.844+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:06.849224+0000) 2019-10-13T14:09:08.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:08.865+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.868214+0000) 2019-10-13T14:09:08.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:08.865+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.868214+0000) 2019-10-13T14:09:11.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:11.177+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.181060+0000) 2019-10-13T14:09:11.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:11.177+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.181905+0000) 2019-10-13T14:09:11.185 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:11.177+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.182369+0000) 2019-10-13T14:09:11.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:11.180+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.183003+0000) 2019-10-13T14:09:11.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:11.177+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.183093+0000) 2019-10-13T14:09:11.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:11.181+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.183889+0000) 2019-10-13T14:09:11.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:11.181+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.184388+0000) 2019-10-13T14:09:11.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:11.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:11.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:11.188+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.193256+0000) 2019-10-13T14:09:11.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:11.412+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.417622+0000) 2019-10-13T14:09:12.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:12.605+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:12.608910+0000) 2019-10-13T14:09:15.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:15.629+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:15.632912+0000) 2019-10-13T14:09:15.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:15.917+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:15.920455+0000) 2019-10-13T14:09:16.196 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:16.188+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.193744+0000) 2019-10-13T14:09:16.197 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:16.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.194684+0000) 2019-10-13T14:09:16.197 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:16.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.195402+0000) 2019-10-13T14:09:16.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:16.192+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.196342+0000) 2019-10-13T14:09:16.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:16.192+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.196737+0000) 2019-10-13T14:09:16.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:16.192+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.196772+0000) 2019-10-13T14:09:16.199 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:16.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.197101+0000) 2019-10-13T14:09:16.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:16.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:16.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:16.200+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.204263+0000) 2019-10-13T14:09:16.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:16.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.599264+0000) 2019-10-13T14:09:16.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:16.848+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:16.849400+0000) 2019-10-13T14:09:18.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:18.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.868446+0000) 2019-10-13T14:09:18.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:18.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.868446+0000) 2019-10-13T14:09:21.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:21.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.206729+0000) 2019-10-13T14:09:21.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:21.200+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.206947+0000) 2019-10-13T14:09:21.210 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:21.204+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.207297+0000) 2019-10-13T14:09:21.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:21.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.207546+0000) 2019-10-13T14:09:21.211 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:21.203+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.208303+0000) 2019-10-13T14:09:21.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:21.204+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.208895+0000) 2019-10-13T14:09:21.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:21.204+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.209274+0000) 2019-10-13T14:09:21.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:21.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.214914+0000) 2019-10-13T14:09:21.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:21.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:21.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:21.419+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.424094+0000) 2019-10-13T14:09:21.572 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:09:21.573 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:21.576 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:09:21.577 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:22.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:22.604+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:22.609120+0000) 2019-10-13T14:09:25.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:25.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:25.633113+0000) 2019-10-13T14:09:25.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:25.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:25.920689+0000) 2019-10-13T14:09:26.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:26.216+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.219442+0000) 2019-10-13T14:09:26.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:26.216+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.219794+0000) 2019-10-13T14:09:26.223 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:26.216+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.220212+0000) 2019-10-13T14:09:26.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:26.216+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.220922+0000) 2019-10-13T14:09:26.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:26.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.221678+0000) 2019-10-13T14:09:26.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:26.216+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.221185+0000) 2019-10-13T14:09:26.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:26.216+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.222408+0000) 2019-10-13T14:09:26.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:26.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:26.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:26.227+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.231331+0000) 2019-10-13T14:09:26.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:26.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.599418+0000) 2019-10-13T14:09:26.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:26.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:26.849601+0000) 2019-10-13T14:09:28.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:28.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.868674+0000) 2019-10-13T14:09:28.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:28.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.868674+0000) 2019-10-13T14:09:31.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:31.228+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.231589+0000) 2019-10-13T14:09:31.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:31.227+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.232716+0000) 2019-10-13T14:09:31.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:31.228+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.232855+0000) 2019-10-13T14:09:31.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:31.228+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.233578+0000) 2019-10-13T14:09:31.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:31.228+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.233822+0000) 2019-10-13T14:09:31.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:31.228+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.234184+0000) 2019-10-13T14:09:31.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:31.228+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.234879+0000) 2019-10-13T14:09:31.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:31.235+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.241148+0000) 2019-10-13T14:09:31.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:31.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:31.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:31.427+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.430447+0000) 2019-10-13T14:09:32.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:32.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:32.609343+0000) 2019-10-13T14:09:35.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:35.628+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:35.633307+0000) 2019-10-13T14:09:35.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:35.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:35.920898+0000) 2019-10-13T14:09:36.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:36.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.244870+0000) 2019-10-13T14:09:36.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:36.240+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.245317+0000) 2019-10-13T14:09:36.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:36.240+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.245971+0000) 2019-10-13T14:09:36.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:36.243+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.246524+0000) 2019-10-13T14:09:36.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:36.240+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.246653+0000) 2019-10-13T14:09:36.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:36.244+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.247813+0000) 2019-10-13T14:09:36.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:36.244+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.248087+0000) 2019-10-13T14:09:36.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:36.251+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.254917+0000) 2019-10-13T14:09:36.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:36.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189246 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:36.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:36.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.599596+0000) 2019-10-13T14:09:36.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:36.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:36.849805+0000) 2019-10-13T14:09:38.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:38.864+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.868918+0000) 2019-10-13T14:09:38.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:38.864+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.868918+0000) 2019-10-13T14:09:41.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:41.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.258412+0000) 2019-10-13T14:09:41.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:41.256+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.259428+0000) 2019-10-13T14:09:41.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:41.256+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.259803+0000) 2019-10-13T14:09:41.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:41.256+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.260422+0000) 2019-10-13T14:09:41.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:41.255+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.261184+0000) 2019-10-13T14:09:41.264 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:41.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.261758+0000) 2019-10-13T14:09:41.264 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:41.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.261910+0000) 2019-10-13T14:09:41.271 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:41.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:41.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:41.267+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.271550+0000) 2019-10-13T14:09:41.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:41.431+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.436741+0000) 2019-10-13T14:09:42.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:42.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:42.609551+0000) 2019-10-13T14:09:45.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:45.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:45.633544+0000) 2019-10-13T14:09:45.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:45.916+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:45.921093+0000) 2019-10-13T14:09:46.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:46.268+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.271779+0000) 2019-10-13T14:09:46.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:46.268+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.272177+0000) 2019-10-13T14:09:46.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:46.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.272737+0000) 2019-10-13T14:09:46.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:46.268+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.273502+0000) 2019-10-13T14:09:46.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:46.268+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.274378+0000) 2019-10-13T14:09:46.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:46.268+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.274827+0000) 2019-10-13T14:09:46.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:46.271+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.275488+0000) 2019-10-13T14:09:46.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:46.275+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.280719+0000) 2019-10-13T14:09:46.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:46.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:46.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:46.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.599776+0000) 2019-10-13T14:09:46.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:46.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:46.849987+0000) 2019-10-13T14:09:48.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:48.868+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.869147+0000) 2019-10-13T14:09:48.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:48.868+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.869147+0000) 2019-10-13T14:09:51.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:51.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.285489+0000) 2019-10-13T14:09:51.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:51.280+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.285848+0000) 2019-10-13T14:09:51.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:51.280+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.286383+0000) 2019-10-13T14:09:51.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:51.280+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.286552+0000) 2019-10-13T14:09:51.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:51.284+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.287423+0000) 2019-10-13T14:09:51.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:51.284+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.288179+0000) 2019-10-13T14:09:51.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:51.283+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.288827+0000) 2019-10-13T14:09:51.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:51.287+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.293958+0000) 2019-10-13T14:09:51.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:51.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:51.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:51.439+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.442961+0000) 2019-10-13T14:09:51.683 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:09:51.684 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:51.688 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:09:51.688 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:52.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:52.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:52.609790+0000) 2019-10-13T14:09:55.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:55.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:55.633747+0000) 2019-10-13T14:09:55.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:55.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:55.921328+0000) 2019-10-13T14:09:56.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:56.292+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.297766+0000) 2019-10-13T14:09:56.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:56.295+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.298369+0000) 2019-10-13T14:09:56.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:09:56.292+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.298898+0000) 2019-10-13T14:09:56.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:56.292+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.299145+0000) 2019-10-13T14:09:56.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:09:56.296+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.299467+0000) 2019-10-13T14:09:56.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:09:56.296+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.300613+0000) 2019-10-13T14:09:56.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:56.296+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.301203+0000) 2019-10-13T14:09:56.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:09:56.299+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.305140+0000) 2019-10-13T14:09:56.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:09:56.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:09:56.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:09:56.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.599954+0000) 2019-10-13T14:09:56.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:09:56.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:56.850194+0000) 2019-10-13T14:09:58.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:09:58.868+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.869375+0000) 2019-10-13T14:09:58.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:09:58.868+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.869374+0000) 2019-10-13T14:10:01.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:01.303+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.310847+0000) 2019-10-13T14:10:01.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:01.307+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.311293+0000) 2019-10-13T14:10:01.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:01.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.312061+0000) 2019-10-13T14:10:01.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:01.307+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.312251+0000) 2019-10-13T14:10:01.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:01.307+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.313675+0000) 2019-10-13T14:10:01.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:01.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.313897+0000) 2019-10-13T14:10:01.317 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:01.307+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.314030+0000) 2019-10-13T14:10:01.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:01.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:01.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:01.315+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.321773+0000) 2019-10-13T14:10:01.451 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:01.443+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.449267+0000) 2019-10-13T14:10:02.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:02.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:02.609993+0000) 2019-10-13T14:10:05.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:05.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:05.633968+0000) 2019-10-13T14:10:05.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:05.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:05.921543+0000) 2019-10-13T14:10:06.326 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:06.319+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.323456+0000) 2019-10-13T14:10:06.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:06.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.324490+0000) 2019-10-13T14:10:06.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:06.319+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.325061+0000) 2019-10-13T14:10:06.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:06.319+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.325384+0000) 2019-10-13T14:10:06.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:06.319+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.325587+0000) 2019-10-13T14:10:06.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:06.319+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.326484+0000) 2019-10-13T14:10:06.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:06.319+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.326739+0000) 2019-10-13T14:10:06.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:06.327+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.332820+0000) 2019-10-13T14:10:06.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:06.327+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:06.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:06.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.600132+0000) 2019-10-13T14:10:06.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:06.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:06.850361+0000) 2019-10-13T14:10:08.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:08.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.869603+0000) 2019-10-13T14:10:08.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:08.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.869602+0000) 2019-10-13T14:10:11.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:11.331+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.336997+0000) 2019-10-13T14:10:11.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:11.331+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.337383+0000) 2019-10-13T14:10:11.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:11.335+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.338119+0000) 2019-10-13T14:10:11.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:11.331+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.338261+0000) 2019-10-13T14:10:11.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:11.331+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.338856+0000) 2019-10-13T14:10:11.342 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:11.335+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.340210+0000) 2019-10-13T14:10:11.343 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:11.335+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.340604+0000) 2019-10-13T14:10:11.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:11.339+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.345717+0000) 2019-10-13T14:10:11.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:11.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:11.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:11.451+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.455758+0000) 2019-10-13T14:10:12.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:12.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:12.610232+0000) 2019-10-13T14:10:15.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:15.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:15.634173+0000) 2019-10-13T14:10:15.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:15.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:15.921785+0000) 2019-10-13T14:10:16.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:16.342+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.350043+0000) 2019-10-13T14:10:16.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:16.343+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.350243+0000) 2019-10-13T14:10:16.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:16.343+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.350707+0000) 2019-10-13T14:10:16.353 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:16.343+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.350859+0000) 2019-10-13T14:10:16.354 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:16.347+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.351432+0000) 2019-10-13T14:10:16.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:16.347+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.351758+0000) 2019-10-13T14:10:16.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:16.347+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.353353+0000) 2019-10-13T14:10:16.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:16.354+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.358993+0000) 2019-10-13T14:10:16.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:16.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:16.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:16.595+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.600333+0000) 2019-10-13T14:10:16.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:16.846+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:16.850501+0000) 2019-10-13T14:10:18.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:18.867+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.869838+0000) 2019-10-13T14:10:18.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:18.867+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.869838+0000) 2019-10-13T14:10:21.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:21.355+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.362250+0000) 2019-10-13T14:10:21.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:21.358+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.362891+0000) 2019-10-13T14:10:21.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:21.355+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.363016+0000) 2019-10-13T14:10:21.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:21.359+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.363446+0000) 2019-10-13T14:10:21.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:21.359+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.363791+0000) 2019-10-13T14:10:21.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:21.359+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.364893+0000) 2019-10-13T14:10:21.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:21.359+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.365147+0000) 2019-10-13T14:10:21.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:21.366+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.370751+0000) 2019-10-13T14:10:21.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:21.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:21.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:21.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.464508+0000) 2019-10-13T14:10:21.811 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:10:21.811 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:21.815 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:10:21.815 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:22.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:22.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:22.610434+0000) 2019-10-13T14:10:25.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:25.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:25.634401+0000) 2019-10-13T14:10:25.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:25.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:25.922054+0000) 2019-10-13T14:10:26.378 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:26.370+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.375753+0000) 2019-10-13T14:10:26.378 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:26.371+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.376205+0000) 2019-10-13T14:10:26.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:26.371+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.377089+0000) 2019-10-13T14:10:26.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:26.371+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.376831+0000) 2019-10-13T14:10:26.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:26.371+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.377706+0000) 2019-10-13T14:10:26.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:26.371+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.378350+0000) 2019-10-13T14:10:26.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:26.371+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.379164+0000) 2019-10-13T14:10:26.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:26.378+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.385753+0000) 2019-10-13T14:10:26.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:26.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:26.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:26.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.600516+0000) 2019-10-13T14:10:26.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:26.846+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:26.850671+0000) 2019-10-13T14:10:28.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:28.863+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.870057+0000) 2019-10-13T14:10:28.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:28.863+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.870057+0000) 2019-10-13T14:10:31.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:31.387+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.390092+0000) 2019-10-13T14:10:31.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:31.387+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.390330+0000) 2019-10-13T14:10:31.394 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:31.387+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.390706+0000) 2019-10-13T14:10:31.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:31.386+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.391618+0000) 2019-10-13T14:10:31.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:31.391+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.391869+0000) 2019-10-13T14:10:31.395 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:31.391+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.392492+0000) 2019-10-13T14:10:31.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:31.391+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.393144+0000) 2019-10-13T14:10:31.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:31.390+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.395865+0000) 2019-10-13T14:10:31.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:31.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:31.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:31.466+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.470579+0000) 2019-10-13T14:10:32.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:32.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:32.610645+0000) 2019-10-13T14:10:35.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:35.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:35.634621+0000) 2019-10-13T14:10:35.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:35.919+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:35.922310+0000) 2019-10-13T14:10:36.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:36.399+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.402806+0000) 2019-10-13T14:10:36.406 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:36.398+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.403484+0000) 2019-10-13T14:10:36.406 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:36.403+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.404311+0000) 2019-10-13T14:10:36.407 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:36.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.404921+0000) 2019-10-13T14:10:36.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:36.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.405091+0000) 2019-10-13T14:10:36.408 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:36.403+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.405673+0000) 2019-10-13T14:10:36.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:36.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.406495+0000) 2019-10-13T14:10:36.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:36.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189642 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:36.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:36.406+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.412363+0000) 2019-10-13T14:10:36.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:36.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.600693+0000) 2019-10-13T14:10:36.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:36.850+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:36.850876+0000) 2019-10-13T14:10:38.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:38.867+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.870290+0000) 2019-10-13T14:10:38.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:38.867+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.870290+0000) 2019-10-13T14:10:41.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:41.415+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.415392+0000) 2019-10-13T14:10:41.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:41.415+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.416278+0000) 2019-10-13T14:10:41.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:41.415+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.417579+0000) 2019-10-13T14:10:41.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:41.415+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.417969+0000) 2019-10-13T14:10:41.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:41.415+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.418506+0000) 2019-10-13T14:10:41.421 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:41.414+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.419078+0000) 2019-10-13T14:10:41.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:41.415+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.419226+0000) 2019-10-13T14:10:41.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:41.418+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.423735+0000) 2019-10-13T14:10:41.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:41.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:41.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:41.478+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.481234+0000) 2019-10-13T14:10:42.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:42.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:42.610848+0000) 2019-10-13T14:10:45.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:45.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:45.634822+0000) 2019-10-13T14:10:45.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:45.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:45.922548+0000) 2019-10-13T14:10:46.431 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:46.426+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.428628+0000) 2019-10-13T14:10:46.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:46.422+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.429563+0000) 2019-10-13T14:10:46.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:46.426+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.429195+0000) 2019-10-13T14:10:46.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:46.426+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.429622+0000) 2019-10-13T14:10:46.433 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:46.426+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.429946+0000) 2019-10-13T14:10:46.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:46.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.430472+0000) 2019-10-13T14:10:46.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:46.426+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.431013+0000) 2019-10-13T14:10:46.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:46.430+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.434493+0000) 2019-10-13T14:10:46.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:46.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:46.603 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:46.598+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.600872+0000) 2019-10-13T14:10:46.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:46.850+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:46.851045+0000) 2019-10-13T14:10:48.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:48.866+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.870527+0000) 2019-10-13T14:10:48.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:48.866+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.870528+0000) 2019-10-13T14:10:51.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:51.438+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.440109+0000) 2019-10-13T14:10:51.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:51.434+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.440664+0000) 2019-10-13T14:10:51.444 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:51.438+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.441433+0000) 2019-10-13T14:10:51.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:51.438+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.441965+0000) 2019-10-13T14:10:51.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:51.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.442375+0000) 2019-10-13T14:10:51.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:51.438+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.442802+0000) 2019-10-13T14:10:51.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:51.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.444038+0000) 2019-10-13T14:10:51.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:51.446+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.450337+0000) 2019-10-13T14:10:51.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:51.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:51.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:51.486+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.487493+0000) 2019-10-13T14:10:51.923 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:10:51.923 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:51.927 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:10:51.927 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:52.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:52.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:52.611016+0000) 2019-10-13T14:10:55.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:55.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:55.635026+0000) 2019-10-13T14:10:55.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:55.918+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:55.922717+0000) 2019-10-13T14:10:56.456 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:10:56.450+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.453352+0000) 2019-10-13T14:10:56.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:10:56.450+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.454257+0000) 2019-10-13T14:10:56.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:56.450+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.454784+0000) 2019-10-13T14:10:56.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:10:56.454+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.455566+0000) 2019-10-13T14:10:56.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:56.454+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.455905+0000) 2019-10-13T14:10:56.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:56.450+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.456765+0000) 2019-10-13T14:10:56.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:56.454+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.456159+0000) 2019-10-13T14:10:56.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:10:56.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:10:56.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:10:56.458+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.463746+0000) 2019-10-13T14:10:56.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:10:56.598+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.601052+0000) 2019-10-13T14:10:56.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:10:56.850+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:56.851220+0000) 2019-10-13T14:10:58.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:10:58.866+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.870752+0000) 2019-10-13T14:10:58.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:10:58.866+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.870752+0000) 2019-10-13T14:11:01.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:01.462+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.466594+0000) 2019-10-13T14:11:01.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:01.466+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.467888+0000) 2019-10-13T14:11:01.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:01.466+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.468294+0000) 2019-10-13T14:11:01.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:01.466+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.468938+0000) 2019-10-13T14:11:01.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:01.466+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.468951+0000) 2019-10-13T14:11:01.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:01.466+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.469494+0000) 2019-10-13T14:11:01.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:01.466+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.469851+0000) 2019-10-13T14:11:01.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:01.470+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.473374+0000) 2019-10-13T14:11:01.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:01.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189806 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:01.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:01.490+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.493935+0000) 2019-10-13T14:11:02.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:02.606+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:02.611232+0000) 2019-10-13T14:11:05.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:05.630+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:05.635230+0000) 2019-10-13T14:11:05.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:05.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:05.922917+0000) 2019-10-13T14:11:06.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:06.478+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.480055+0000) 2019-10-13T14:11:06.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:06.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.481007+0000) 2019-10-13T14:11:06.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:06.478+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.481439+0000) 2019-10-13T14:11:06.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:06.478+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.482323+0000) 2019-10-13T14:11:06.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:06.478+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.483049+0000) 2019-10-13T14:11:06.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:06.482+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.483466+0000) 2019-10-13T14:11:06.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:06.482+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.483776+0000) 2019-10-13T14:11:06.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:06.485+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.488107+0000) 2019-10-13T14:11:06.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:06.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:06.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:06.598+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.601259+0000) 2019-10-13T14:11:06.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:06.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:06.851390+0000) 2019-10-13T14:11:08.873 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:08.870+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.870980+0000) 2019-10-13T14:11:08.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:08.870+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.870980+0000) 2019-10-13T14:11:11.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:11.490+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.494406+0000) 2019-10-13T14:11:11.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:11.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.494896+0000) 2019-10-13T14:11:11.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:11.493+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.495443+0000) 2019-10-13T14:11:11.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:11.494+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.495481+0000) 2019-10-13T14:11:11.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:11.494+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.495958+0000) 2019-10-13T14:11:11.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:11.494+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.496806+0000) 2019-10-13T14:11:11.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:11.494+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.496712+0000) 2019-10-13T14:11:11.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:11.501+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.504049+0000) 2019-10-13T14:11:11.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:11.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:11.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:11.501+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.505586+0000) 2019-10-13T14:11:12.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:12.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:12.611456+0000) 2019-10-13T14:11:15.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:15.634+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:15.635478+0000) 2019-10-13T14:11:15.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:15.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:15.923119+0000) 2019-10-13T14:11:16.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:16.506+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.508099+0000) 2019-10-13T14:11:16.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:16.506+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.508720+0000) 2019-10-13T14:11:16.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:16.505+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.509216+0000) 2019-10-13T14:11:16.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:16.506+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.509513+0000) 2019-10-13T14:11:16.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:16.506+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.509764+0000) 2019-10-13T14:11:16.513 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:16.506+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.510072+0000) 2019-10-13T14:11:16.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:16.506+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.511006+0000) 2019-10-13T14:11:16.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:16.509+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.513998+0000) 2019-10-13T14:11:16.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:16.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189905 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:16.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:16.598+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.601435+0000) 2019-10-13T14:11:16.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:16.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:16.851569+0000) 2019-10-13T14:11:18.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:18.866+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.871227+0000) 2019-10-13T14:11:18.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:18.866+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.871227+0000) 2019-10-13T14:11:21.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:21.509+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.511980+0000) 2019-10-13T14:11:21.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:21.514+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.519057+0000) 2019-10-13T14:11:21.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:21.518+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.520868+0000) 2019-10-13T14:11:21.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:21.518+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.521132+0000) 2019-10-13T14:11:21.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:21.517+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.521555+0000) 2019-10-13T14:11:21.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:21.518+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.522109+0000) 2019-10-13T14:11:21.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:21.518+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.522413+0000) 2019-10-13T14:11:21.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:21.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.522670+0000) 2019-10-13T14:11:21.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:21.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:21.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:21.529+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.530491+0000) 2019-10-13T14:11:22.033 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:11:22.033 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:22.037 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:11:22.037 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:22.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:22.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:22.611659+0000) 2019-10-13T14:11:25.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:25.634+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:25.635645+0000) 2019-10-13T14:11:25.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:25.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:25.923361+0000) 2019-10-13T14:11:26.536 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:26.529+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.533641+0000) 2019-10-13T14:11:26.537 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:26.529+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.533948+0000) 2019-10-13T14:11:26.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:26.533+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.534738+0000) 2019-10-13T14:11:26.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:26.529+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.534780+0000) 2019-10-13T14:11:26.538 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:26.533+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.535408+0000) 2019-10-13T14:11:26.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:26.533+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.535466+0000) 2019-10-13T14:11:26.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:26.533+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.536644+0000) 2019-10-13T14:11:26.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:26.537+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.541038+0000) 2019-10-13T14:11:26.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:26.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 189972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:26.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:26.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.601611+0000) 2019-10-13T14:11:26.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:26.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:26.851741+0000) 2019-10-13T14:11:28.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:28.869+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.871496+0000) 2019-10-13T14:11:28.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:28.869+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.871496+0000) 2019-10-13T14:11:31.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:31.517+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.518869+0000) 2019-10-13T14:11:31.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:31.545+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.546477+0000) 2019-10-13T14:11:31.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:31.541+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.547016+0000) 2019-10-13T14:11:31.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:31.545+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.547370+0000) 2019-10-13T14:11:31.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:31.545+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.547798+0000) 2019-10-13T14:11:31.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:31.545+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.549097+0000) 2019-10-13T14:11:31.552 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:31.545+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.549751+0000) 2019-10-13T14:11:31.552 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:31.545+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.550277+0000) 2019-10-13T14:11:31.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:31.553+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.555102+0000) 2019-10-13T14:11:31.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:31.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:32.614 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:32.609+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:32.611881+0000) 2019-10-13T14:11:35.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:35.633+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:35.635876+0000) 2019-10-13T14:11:35.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:35.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:35.923561+0000) 2019-10-13T14:11:36.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:36.557+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.559987+0000) 2019-10-13T14:11:36.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:36.557+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.560856+0000) 2019-10-13T14:11:36.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:36.557+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.561555+0000) 2019-10-13T14:11:36.564 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:36.557+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.561812+0000) 2019-10-13T14:11:36.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:36.557+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.562033+0000) 2019-10-13T14:11:36.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:36.557+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.562684+0000) 2019-10-13T14:11:36.566 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:36.561+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.563472+0000) 2019-10-13T14:11:36.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:36.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:36.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:36.565+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.569255+0000) 2019-10-13T14:11:36.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:36.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.601779+0000) 2019-10-13T14:11:36.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:36.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:36.851908+0000) 2019-10-13T14:11:38.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:38.869+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.871723+0000) 2019-10-13T14:11:38.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:38.869+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.871723+0000) 2019-10-13T14:11:41.527 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:41.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.525333+0000) 2019-10-13T14:11:41.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:41.569+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.573021+0000) 2019-10-13T14:11:41.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:41.569+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.573298+0000) 2019-10-13T14:11:41.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:41.569+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.573597+0000) 2019-10-13T14:11:41.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:41.573+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.574195+0000) 2019-10-13T14:11:41.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:41.569+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.574413+0000) 2019-10-13T14:11:41.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:41.569+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.574845+0000) 2019-10-13T14:11:41.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:41.569+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.575249+0000) 2019-10-13T14:11:41.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:41.577+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.578944+0000) 2019-10-13T14:11:41.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:41.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:42.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:42.609+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:42.612083+0000) 2019-10-13T14:11:45.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:45.633+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:45.636092+0000) 2019-10-13T14:11:45.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:45.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:45.923759+0000) 2019-10-13T14:11:46.588 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:46.581+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.585490+0000) 2019-10-13T14:11:46.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:46.581+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.586055+0000) 2019-10-13T14:11:46.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:46.581+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.586461+0000) 2019-10-13T14:11:46.589 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:46.585+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.587024+0000) 2019-10-13T14:11:46.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:46.581+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.587170+0000) 2019-10-13T14:11:46.590 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:46.585+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.588366+0000) 2019-10-13T14:11:46.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:46.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.588707+0000) 2019-10-13T14:11:46.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:46.593+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.595402+0000) 2019-10-13T14:11:46.598 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:46.593+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:46.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:46.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.601881+0000) 2019-10-13T14:11:46.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:46.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:46.852078+0000) 2019-10-13T14:11:48.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:48.869+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.871946+0000) 2019-10-13T14:11:48.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:48.869+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.871946+0000) 2019-10-13T14:11:51.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:51.529+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.532135+0000) 2019-10-13T14:11:51.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:51.593+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.598172+0000) 2019-10-13T14:11:51.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:51.593+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.598887+0000) 2019-10-13T14:11:51.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:51.597+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.599551+0000) 2019-10-13T14:11:51.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:51.597+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.600777+0000) 2019-10-13T14:11:51.604 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:51.597+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.601909+0000) 2019-10-13T14:11:51.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:51.597+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.602267+0000) 2019-10-13T14:11:51.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:51.597+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.602582+0000) 2019-10-13T14:11:51.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:51.605+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.607386+0000) 2019-10-13T14:11:51.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:51.601+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:52.145 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:11:52.145 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:52.149 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:11:52.149 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:52.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:52.609+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:52.612308+0000) 2019-10-13T14:11:55.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:55.633+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:55.636292+0000) 2019-10-13T14:11:55.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:55.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:55.923955+0000) 2019-10-13T14:11:56.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:56.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.602039+0000) 2019-10-13T14:11:56.612 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:11:56.605+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.610025+0000) 2019-10-13T14:11:56.613 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:56.605+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.610804+0000) 2019-10-13T14:11:56.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:56.609+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.611400+0000) 2019-10-13T14:11:56.614 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:11:56.609+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.611904+0000) 2019-10-13T14:11:56.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:56.609+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.613754+0000) 2019-10-13T14:11:56.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:11:56.609+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.613870+0000) 2019-10-13T14:11:56.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:11:56.609+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.614079+0000) 2019-10-13T14:11:56.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:11:56.617+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.620005+0000) 2019-10-13T14:11:56.623 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:11:56.617+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190168 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:11:56.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:11:56.849+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:56.852277+0000) 2019-10-13T14:11:58.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:11:58.869+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.872173+0000) 2019-10-13T14:11:58.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:11:58.869+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.872173+0000) 2019-10-13T14:12:01.541 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:01.536+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.538523+0000) 2019-10-13T14:12:01.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:01.621+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.624544+0000) 2019-10-13T14:12:01.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:01.621+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.624851+0000) 2019-10-13T14:12:01.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:01.621+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.625143+0000) 2019-10-13T14:12:01.628 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:01.621+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.625505+0000) 2019-10-13T14:12:01.629 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:01.624+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.626450+0000) 2019-10-13T14:12:01.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:01.621+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.626923+0000) 2019-10-13T14:12:01.630 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:01.625+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.627520+0000) 2019-10-13T14:12:01.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:01.628+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.633193+0000) 2019-10-13T14:12:01.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:01.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:02.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:02.609+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:02.612518+0000) 2019-10-13T14:12:05.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:05.633+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:05.636493+0000) 2019-10-13T14:12:05.926 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:05.921+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:05.924160+0000) 2019-10-13T14:12:06.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:06.597+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.602254+0000) 2019-10-13T14:12:06.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:06.633+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.635921+0000) 2019-10-13T14:12:06.639 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:06.632+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.637249+0000) 2019-10-13T14:12:06.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:06.633+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.637341+0000) 2019-10-13T14:12:06.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:06.633+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.637917+0000) 2019-10-13T14:12:06.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:06.633+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.638332+0000) 2019-10-13T14:12:06.641 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:06.633+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.639030+0000) 2019-10-13T14:12:06.642 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:06.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.639535+0000) 2019-10-13T14:12:06.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:06.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:06.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:06.644+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.646299+0000) 2019-10-13T14:12:06.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:06.848+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:06.852478+0000) 2019-10-13T14:12:08.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:08.869+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.872399+0000) 2019-10-13T14:12:08.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:08.869+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.872398+0000) 2019-10-13T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:11.544+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.545031+0000) 2019-10-13T14:12:11.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:11.644+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.649155+0000) 2019-10-13T14:12:11.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:11.644+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.649714+0000) 2019-10-13T14:12:11.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:11.644+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.649942+0000) 2019-10-13T14:12:11.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:11.644+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.650316+0000) 2019-10-13T14:12:11.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:11.644+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.650846+0000) 2019-10-13T14:12:11.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:11.648+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.651397+0000) 2019-10-13T14:12:11.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:11.648+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.652332+0000) 2019-10-13T14:12:11.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:11.656+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.658564+0000) 2019-10-13T14:12:11.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:11.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:12.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:12.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:12.612718+0000) 2019-10-13T14:12:15.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:15.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:15.636726+0000) 2019-10-13T14:12:15.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:15.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:15.924362+0000) 2019-10-13T14:12:16.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:16.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.602459+0000) 2019-10-13T14:12:16.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:16.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.661852+0000) 2019-10-13T14:12:16.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:16.656+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.662429+0000) 2019-10-13T14:12:16.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:16.660+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.663152+0000) 2019-10-13T14:12:16.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:16.660+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.663657+0000) 2019-10-13T14:12:16.667 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:16.660+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.664500+0000) 2019-10-13T14:12:16.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:16.660+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.664630+0000) 2019-10-13T14:12:16.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:16.660+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.665464+0000) 2019-10-13T14:12:16.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:16.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:16.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:16.672+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.674690+0000) 2019-10-13T14:12:16.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:16.848+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:16.852652+0000) 2019-10-13T14:12:18.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:18.868+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.872659+0000) 2019-10-13T14:12:18.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:18.868+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.872659+0000) 2019-10-13T14:12:21.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:21.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.551701+0000) 2019-10-13T14:12:21.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:21.668+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.675136+0000) 2019-10-13T14:12:21.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:21.672+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.676819+0000) 2019-10-13T14:12:21.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:21.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.676834+0000) 2019-10-13T14:12:21.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:21.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.677380+0000) 2019-10-13T14:12:21.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:21.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.677490+0000) 2019-10-13T14:12:21.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:21.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.678132+0000) 2019-10-13T14:12:21.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:21.672+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.678385+0000) 2019-10-13T14:12:21.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:21.680+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.682883+0000) 2019-10-13T14:12:21.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:21.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:22.239 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:12:22.239 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:22.243 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:12:22.243 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:22.615 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:22.612+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:22.612897+0000) 2019-10-13T14:12:25.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:25.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:25.636924+0000) 2019-10-13T14:12:25.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:25.920+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:25.924589+0000) 2019-10-13T14:12:26.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:26.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.602650+0000) 2019-10-13T14:12:26.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:26.684+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.688561+0000) 2019-10-13T14:12:26.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:26.684+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.688545+0000) 2019-10-13T14:12:26.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:26.684+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.689476+0000) 2019-10-13T14:12:26.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:26.684+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.689871+0000) 2019-10-13T14:12:26.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:26.688+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.691191+0000) 2019-10-13T14:12:26.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:26.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.691276+0000) 2019-10-13T14:12:26.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:26.688+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.691697+0000) 2019-10-13T14:12:26.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:26.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190367 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:26.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:26.696+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.698827+0000) 2019-10-13T14:12:26.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:26.852+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:26.852807+0000) 2019-10-13T14:12:28.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:28.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.872890+0000) 2019-10-13T14:12:28.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:28.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.872890+0000) 2019-10-13T14:12:31.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:31.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.558108+0000) 2019-10-13T14:12:31.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:31.696+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.701987+0000) 2019-10-13T14:12:31.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:31.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.702355+0000) 2019-10-13T14:12:31.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:31.696+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.702632+0000) 2019-10-13T14:12:31.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:31.700+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.704228+0000) 2019-10-13T14:12:31.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:31.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.704802+0000) 2019-10-13T14:12:31.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:31.700+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.704649+0000) 2019-10-13T14:12:31.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:31.700+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.705329+0000) 2019-10-13T14:12:31.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:31.704+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.710093+0000) 2019-10-13T14:12:31.713 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:31.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:32.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:32.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:32.613103+0000) 2019-10-13T14:12:35.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:35.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:35.637140+0000) 2019-10-13T14:12:35.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:35.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:35.924790+0000) 2019-10-13T14:12:36.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:36.600+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.602849+0000) 2019-10-13T14:12:36.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:36.708+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.713819+0000) 2019-10-13T14:12:36.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:36.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.714968+0000) 2019-10-13T14:12:36.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:36.712+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.715511+0000) 2019-10-13T14:12:36.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:36.712+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.715667+0000) 2019-10-13T14:12:36.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:36.712+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.715873+0000) 2019-10-13T14:12:36.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:36.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.716669+0000) 2019-10-13T14:12:36.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:36.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.717047+0000) 2019-10-13T14:12:36.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:36.720+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.722718+0000) 2019-10-13T14:12:36.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:36.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190432 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:36.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:36.852+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:36.852936+0000) 2019-10-13T14:12:38.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:38.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.873117+0000) 2019-10-13T14:12:38.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:38.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.873117+0000) 2019-10-13T14:12:41.567 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:41.560+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.564487+0000) 2019-10-13T14:12:41.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:41.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.727481+0000) 2019-10-13T14:12:41.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:41.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.728705+0000) 2019-10-13T14:12:41.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:41.724+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.729297+0000) 2019-10-13T14:12:41.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:41.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.729368+0000) 2019-10-13T14:12:41.732 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:41.724+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.729882+0000) 2019-10-13T14:12:41.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:41.724+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.730308+0000) 2019-10-13T14:12:41.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:41.724+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.730927+0000) 2019-10-13T14:12:41.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:41.732+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.735959+0000) 2019-10-13T14:12:41.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:41.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190467 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:42.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:42.612+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:42.613302+0000) 2019-10-13T14:12:45.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:45.632+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:45.637355+0000) 2019-10-13T14:12:45.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:45.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:45.925012+0000) 2019-10-13T14:12:46.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:46.596+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.603040+0000) 2019-10-13T14:12:46.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:46.736+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.739182+0000) 2019-10-13T14:12:46.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:46.736+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.739567+0000) 2019-10-13T14:12:46.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:46.736+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.740872+0000) 2019-10-13T14:12:46.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:46.736+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.741137+0000) 2019-10-13T14:12:46.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:46.736+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.741821+0000) 2019-10-13T14:12:46.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:46.736+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.742491+0000) 2019-10-13T14:12:46.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:46.736+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.743046+0000) 2019-10-13T14:12:46.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:46.744+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.746685+0000) 2019-10-13T14:12:46.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:46.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:46.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:46.852+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:46.853134+0000) 2019-10-13T14:12:48.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:48.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.873339+0000) 2019-10-13T14:12:48.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:48.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.873339+0000) 2019-10-13T14:12:51.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:51.568+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.570874+0000) 2019-10-13T14:12:51.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:51.748+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.754003+0000) 2019-10-13T14:12:51.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:51.748+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.754829+0000) 2019-10-13T14:12:51.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:51.748+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.755237+0000) 2019-10-13T14:12:51.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:51.752+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.756182+0000) 2019-10-13T14:12:51.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:51.752+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.755697+0000) 2019-10-13T14:12:51.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:51.752+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.756318+0000) 2019-10-13T14:12:51.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:51.752+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.757031+0000) 2019-10-13T14:12:51.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:51.756+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.761334+0000) 2019-10-13T14:12:51.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:51.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:52.296 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:12:52.296 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:52.300 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:12:52.300 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:52.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:52.608+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:52.613510+0000) 2019-10-13T14:12:55.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:55.635+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:55.637614+0000) 2019-10-13T14:12:55.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:55.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:55.925278+0000) 2019-10-13T14:12:56.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:56.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.603266+0000) 2019-10-13T14:12:56.768 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:12:56.759+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.765773+0000) 2019-10-13T14:12:56.769 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:56.763+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.767105+0000) 2019-10-13T14:12:56.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:12:56.763+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.767331+0000) 2019-10-13T14:12:56.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:56.763+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.767835+0000) 2019-10-13T14:12:56.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:12:56.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.768232+0000) 2019-10-13T14:12:56.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:12:56.763+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.768575+0000) 2019-10-13T14:12:56.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:56.763+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.769205+0000) 2019-10-13T14:12:56.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:12:56.767+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.773882+0000) 2019-10-13T14:12:56.777 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:12:56.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:12:56.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:12:56.851+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:56.853301+0000) 2019-10-13T14:12:58.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:12:58.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.873580+0000) 2019-10-13T14:12:58.876 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:12:58.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.873580+0000) 2019-10-13T14:13:01.580 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:01.575+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.577208+0000) 2019-10-13T14:13:01.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:01.771+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.778729+0000) 2019-10-13T14:13:01.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:01.775+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.779473+0000) 2019-10-13T14:13:01.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:01.775+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.779694+0000) 2019-10-13T14:13:01.782 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:01.775+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.780409+0000) 2019-10-13T14:13:01.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:01.775+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.781419+0000) 2019-10-13T14:13:01.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:01.775+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.781361+0000) 2019-10-13T14:13:01.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:01.775+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.781749+0000) 2019-10-13T14:13:01.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:01.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:01.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:01.783+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.789225+0000) 2019-10-13T14:13:02.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:02.611+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:02.613691+0000) 2019-10-13T14:13:05.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:05.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:05.637802+0000) 2019-10-13T14:13:05.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:05.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:05.925434+0000) 2019-10-13T14:13:06.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:06.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.603496+0000) 2019-10-13T14:13:06.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:06.787+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.790603+0000) 2019-10-13T14:13:06.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:06.787+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.791002+0000) 2019-10-13T14:13:06.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:06.787+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.792059+0000) 2019-10-13T14:13:06.795 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:06.787+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.792719+0000) 2019-10-13T14:13:06.796 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:06.787+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.793730+0000) 2019-10-13T14:13:06.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:06.787+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.794010+0000) 2019-10-13T14:13:06.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:06.787+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.794789+0000) 2019-10-13T14:13:06.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:06.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:06.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:06.795+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.800871+0000) 2019-10-13T14:13:06.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:06.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:06.853512+0000) 2019-10-13T14:13:08.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:08.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.873809+0000) 2019-10-13T14:13:08.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:08.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.873809+0000) 2019-10-13T14:13:11.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:11.579+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.583814+0000) 2019-10-13T14:13:11.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:11.799+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.803439+0000) 2019-10-13T14:13:11.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:11.799+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.804058+0000) 2019-10-13T14:13:11.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:11.799+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.804402+0000) 2019-10-13T14:13:11.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:11.799+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.804773+0000) 2019-10-13T14:13:11.808 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:11.799+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.805388+0000) 2019-10-13T14:13:11.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:11.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.806722+0000) 2019-10-13T14:13:11.809 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:11.803+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.807290+0000) 2019-10-13T14:13:11.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:11.807+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.812298+0000) 2019-10-13T14:13:11.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:11.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:12.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:12.607+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:12.613867+0000) 2019-10-13T14:13:15.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:15.631+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:15.638006+0000) 2019-10-13T14:13:15.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:15.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:15.925623+0000) 2019-10-13T14:13:16.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:16.603+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.603689+0000) 2019-10-13T14:13:16.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:16.811+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.817356+0000) 2019-10-13T14:13:16.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:16.811+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.817682+0000) 2019-10-13T14:13:16.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:16.811+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.818314+0000) 2019-10-13T14:13:16.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:16.811+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.819278+0000) 2019-10-13T14:13:16.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:16.815+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.819701+0000) 2019-10-13T14:13:16.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:16.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.820022+0000) 2019-10-13T14:13:16.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:16.815+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.820830+0000) 2019-10-13T14:13:16.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:16.819+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.826094+0000) 2019-10-13T14:13:16.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:16.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190696 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:16.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:16.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:16.853711+0000) 2019-10-13T14:13:18.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:18.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.874067+0000) 2019-10-13T14:13:18.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:18.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.874068+0000) 2019-10-13T14:13:21.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:21.587+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.590181+0000) 2019-10-13T14:13:21.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:21.827+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.830555+0000) 2019-10-13T14:13:21.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:21.823+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.830772+0000) 2019-10-13T14:13:21.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:21.823+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.831128+0000) 2019-10-13T14:13:21.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:21.827+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.831526+0000) 2019-10-13T14:13:21.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:21.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.832188+0000) 2019-10-13T14:13:21.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:21.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.832365+0000) 2019-10-13T14:13:21.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:21.827+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.832885+0000) 2019-10-13T14:13:21.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:21.831+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.835370+0000) 2019-10-13T14:13:21.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:21.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:22.344 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:13:22.344 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:22.348 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:13:22.348 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:22.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:22.611+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:22.614044+0000) 2019-10-13T14:13:25.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:25.635+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:25.638198+0000) 2019-10-13T14:13:25.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:25.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:25.925830+0000) 2019-10-13T14:13:26.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:26.603+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.603883+0000) 2019-10-13T14:13:26.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:26.843+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.843751+0000) 2019-10-13T14:13:26.847 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:26.843+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.844410+0000) 2019-10-13T14:13:26.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:26.843+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.845479+0000) 2019-10-13T14:13:26.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:26.843+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.846179+0000) 2019-10-13T14:13:26.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:26.843+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.846249+0000) 2019-10-13T14:13:26.849 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:26.843+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.846581+0000) 2019-10-13T14:13:26.849 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:26.843+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.847269+0000) 2019-10-13T14:13:26.855 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:26.851+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:26.856 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:26.847+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.853620+0000) 2019-10-13T14:13:26.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:26.847+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:26.853829+0000) 2019-10-13T14:13:28.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:28.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.874300+0000) 2019-10-13T14:13:28.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:28.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.874300+0000) 2019-10-13T14:13:31.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:31.591+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.596698+0000) 2019-10-13T14:13:31.859 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:31.855+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.856418+0000) 2019-10-13T14:13:31.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:31.855+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.857426+0000) 2019-10-13T14:13:31.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:31.855+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.858016+0000) 2019-10-13T14:13:31.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:31.855+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.858268+0000) 2019-10-13T14:13:31.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:31.855+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.859136+0000) 2019-10-13T14:13:31.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:31.859+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.859588+0000) 2019-10-13T14:13:31.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:31.855+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.860598+0000) 2019-10-13T14:13:31.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:31.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:31.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:31.863+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.866659+0000) 2019-10-13T14:13:32.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:32.611+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:32.614269+0000) 2019-10-13T14:13:35.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:35.635+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:35.638424+0000) 2019-10-13T14:13:35.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:35.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:35.926036+0000) 2019-10-13T14:13:36.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:36.602+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.604092+0000) 2019-10-13T14:13:36.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:36.851+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.853988+0000) 2019-10-13T14:13:36.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:36.866+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.868636+0000) 2019-10-13T14:13:36.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:36.866+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.869189+0000) 2019-10-13T14:13:36.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:36.863+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.869689+0000) 2019-10-13T14:13:36.872 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:36.866+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.870032+0000) 2019-10-13T14:13:36.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:36.866+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.870732+0000) 2019-10-13T14:13:36.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:36.870+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.871651+0000) 2019-10-13T14:13:36.874 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:36.870+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.871915+0000) 2019-10-13T14:13:36.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:36.871+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:36.874812+0000) 2019-10-13T14:13:36.880 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:36.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:38.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:38.870+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.874528+0000) 2019-10-13T14:13:38.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:38.870+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.874528+0000) 2019-10-13T14:13:41.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:41.599+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.603119+0000) 2019-10-13T14:13:41.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:41.878+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.881775+0000) 2019-10-13T14:13:41.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:41.878+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.882156+0000) 2019-10-13T14:13:41.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:41.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.882698+0000) 2019-10-13T14:13:41.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:41.879+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.883806+0000) 2019-10-13T14:13:41.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:41.882+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.883985+0000) 2019-10-13T14:13:41.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:41.882+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.884377+0000) 2019-10-13T14:13:41.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:41.882+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.884889+0000) 2019-10-13T14:13:41.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:41.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.891326+0000) 2019-10-13T14:13:41.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:41.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190864 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:42.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:42.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:42.614468+0000) 2019-10-13T14:13:45.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:45.634+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:45.638623+0000) 2019-10-13T14:13:45.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:45.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:45.926267+0000) 2019-10-13T14:13:46.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:46.602+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.604304+0000) 2019-10-13T14:13:46.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:46.851+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.854188+0000) 2019-10-13T14:13:46.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:46.894+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.895822+0000) 2019-10-13T14:13:46.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:46.894+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.896432+0000) 2019-10-13T14:13:46.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:46.894+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.896697+0000) 2019-10-13T14:13:46.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:46.894+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.897474+0000) 2019-10-13T14:13:46.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:46.894+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.898173+0000) 2019-10-13T14:13:46.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:46.895+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.898742+0000) 2019-10-13T14:13:46.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:46.894+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.898858+0000) 2019-10-13T14:13:46.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:46.899+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:46.903576+0000) 2019-10-13T14:13:46.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:46.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:48.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:48.870+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.874752+0000) 2019-10-13T14:13:48.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:48.870+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.874752+0000) 2019-10-13T14:13:51.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:51.606+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.609772+0000) 2019-10-13T14:13:51.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:51.906+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.908959+0000) 2019-10-13T14:13:51.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:51.906+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.910526+0000) 2019-10-13T14:13:51.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:51.906+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.910808+0000) 2019-10-13T14:13:51.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:51.906+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.911074+0000) 2019-10-13T14:13:51.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:51.910+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.911523+0000) 2019-10-13T14:13:51.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:51.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.912097+0000) 2019-10-13T14:13:51.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:51.910+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.912419+0000) 2019-10-13T14:13:51.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:51.910+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.916207+0000) 2019-10-13T14:13:51.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:51.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:52.476 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:13:52.477 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:52.480 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:13:52.480 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:52.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:52.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:52.614652+0000) 2019-10-13T14:13:55.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:55.634+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:55.638857+0000) 2019-10-13T14:13:55.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:55.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:55.926478+0000) 2019-10-13T14:13:56.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:56.602+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.604509+0000) 2019-10-13T14:13:56.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:56.850+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.854408+0000) 2019-10-13T14:13:56.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:56.918+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.922198+0000) 2019-10-13T14:13:56.925 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:56.918+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.923279+0000) 2019-10-13T14:13:56.926 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:13:56.922+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.923867+0000) 2019-10-13T14:13:56.926 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:13:56.922+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.924187+0000) 2019-10-13T14:13:56.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:13:56.918+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.924702+0000) 2019-10-13T14:13:56.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:13:56.922+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.925600+0000) 2019-10-13T14:13:56.928 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:13:56.922+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.926127+0000) 2019-10-13T14:13:56.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:13:56.922+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:56.929102+0000) 2019-10-13T14:13:56.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:13:56.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:13:58.877 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:13:58.870+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.874978+0000) 2019-10-13T14:13:58.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:13:58.870+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.874978+0000) 2019-10-13T14:14:01.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:01.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.616498+0000) 2019-10-13T14:14:01.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:01.934+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.936737+0000) 2019-10-13T14:14:01.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:01.934+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.937306+0000) 2019-10-13T14:14:01.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:01.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.937753+0000) 2019-10-13T14:14:01.941 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:01.934+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.938979+0000) 2019-10-13T14:14:01.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:01.934+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.939654+0000) 2019-10-13T14:14:01.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:01.938+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.939869+0000) 2019-10-13T14:14:01.942 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:01.938+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.940213+0000) 2019-10-13T14:14:01.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:01.938+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.944876+0000) 2019-10-13T14:14:01.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:01.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 190998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:02.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:02.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:02.614834+0000) 2019-10-13T14:14:05.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:05.634+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:05.639081+0000) 2019-10-13T14:14:05.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:05.922+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:05.926679+0000) 2019-10-13T14:14:06.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:06.602+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.604708+0000) 2019-10-13T14:14:06.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:06.850+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.854600+0000) 2019-10-13T14:14:06.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:06.946+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.948675+0000) 2019-10-13T14:14:06.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:06.946+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.949014+0000) 2019-10-13T14:14:06.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:06.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.949844+0000) 2019-10-13T14:14:06.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:06.946+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.950124+0000) 2019-10-13T14:14:06.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:06.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.950500+0000) 2019-10-13T14:14:06.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:06.950+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.952177+0000) 2019-10-13T14:14:06.955 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:06.946+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.952931+0000) 2019-10-13T14:14:06.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:06.954+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:06.958190+0000) 2019-10-13T14:14:06.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:06.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:08.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:08.870+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.875221+0000) 2019-10-13T14:14:08.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:08.870+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.875222+0000) 2019-10-13T14:14:11.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:11.618+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.623126+0000) 2019-10-13T14:14:11.965 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:11.958+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.962814+0000) 2019-10-13T14:14:11.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:11.958+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.963224+0000) 2019-10-13T14:14:11.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:11.962+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.964141+0000) 2019-10-13T14:14:11.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:11.962+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.964338+0000) 2019-10-13T14:14:11.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:11.958+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.964965+0000) 2019-10-13T14:14:11.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:11.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.965175+0000) 2019-10-13T14:14:11.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:11.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.965593+0000) 2019-10-13T14:14:11.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:11.966+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.970344+0000) 2019-10-13T14:14:11.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:11.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:12.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:12.610+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:12.615051+0000) 2019-10-13T14:14:15.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:15.638+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:15.639286+0000) 2019-10-13T14:14:15.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:15.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:15.926875+0000) 2019-10-13T14:14:16.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:16.602+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.604918+0000) 2019-10-13T14:14:16.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:16.854+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.854792+0000) 2019-10-13T14:14:16.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:16.974+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.975446+0000) 2019-10-13T14:14:16.978 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:16.970+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.976342+0000) 2019-10-13T14:14:16.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:16.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.976894+0000) 2019-10-13T14:14:16.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:16.974+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.977154+0000) 2019-10-13T14:14:16.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:16.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.977436+0000) 2019-10-13T14:14:16.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:16.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.977838+0000) 2019-10-13T14:14:16.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:16.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.978300+0000) 2019-10-13T14:14:16.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:16.978+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:16.982409+0000) 2019-10-13T14:14:16.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:16.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:18.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:18.874+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.875487+0000) 2019-10-13T14:14:18.878 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:18.874+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.875487+0000) 2019-10-13T14:14:21.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:21.626+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.629385+0000) 2019-10-13T14:14:21.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:21.985+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.988149+0000) 2019-10-13T14:14:21.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:21.985+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.988661+0000) 2019-10-13T14:14:21.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:21.985+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.988964+0000) 2019-10-13T14:14:21.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:21.985+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.989340+0000) 2019-10-13T14:14:21.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:21.982+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.990024+0000) 2019-10-13T14:14:21.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:21.985+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.990420+0000) 2019-10-13T14:14:21.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:21.985+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.991165+0000) 2019-10-13T14:14:21.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:21.993+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191132 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:21.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:21.990+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.997501+0000) 2019-10-13T14:14:22.573 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:14:22.574 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:22.577 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:14:22.577 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:22.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:22.609+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:22.615220+0000) 2019-10-13T14:14:25.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:25.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:25.639551+0000) 2019-10-13T14:14:25.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:25.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:25.927075+0000) 2019-10-13T14:14:26.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:26.601+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:26.605113+0000) 2019-10-13T14:14:26.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:26.854+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:26.855036+0000) 2019-10-13T14:14:27.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:26.997+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.000014+0000) 2019-10-13T14:14:27.003 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:26.997+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.001136+0000) 2019-10-13T14:14:27.004 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:26.997+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.001322+0000) 2019-10-13T14:14:27.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:26.997+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.001726+0000) 2019-10-13T14:14:27.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:26.997+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.002023+0000) 2019-10-13T14:14:27.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:26.998+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.002507+0000) 2019-10-13T14:14:27.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:26.997+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.002638+0000) 2019-10-13T14:14:27.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:27.002+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:27.009775+0000) 2019-10-13T14:14:27.012 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:27.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:28.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:28.873+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.875710+0000) 2019-10-13T14:14:28.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:28.873+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.875710+0000) 2019-10-13T14:14:31.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:31.634+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:31.635887+0000) 2019-10-13T14:14:32.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:32.009+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.012482+0000) 2019-10-13T14:14:32.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:32.009+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.013628+0000) 2019-10-13T14:14:32.017 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:32.009+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.014307+0000) 2019-10-13T14:14:32.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:32.009+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.014861+0000) 2019-10-13T14:14:32.017 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:32.013+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.015400+0000) 2019-10-13T14:14:32.018 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:32.010+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.015979+0000) 2019-10-13T14:14:32.018 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:32.013+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.016193+0000) 2019-10-13T14:14:32.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:32.014+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.020437+0000) 2019-10-13T14:14:32.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:32.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:32.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:32.613+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.615401+0000) 2019-10-13T14:14:35.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:35.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:35.639745+0000) 2019-10-13T14:14:35.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:35.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:35.927286+0000) 2019-10-13T14:14:36.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:36.601+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:36.605302+0000) 2019-10-13T14:14:36.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:36.854+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:36.855184+0000) 2019-10-13T14:14:37.029 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:37.021+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.026641+0000) 2019-10-13T14:14:37.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:37.021+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.027023+0000) 2019-10-13T14:14:37.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:37.025+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.027633+0000) 2019-10-13T14:14:37.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:37.025+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.027890+0000) 2019-10-13T14:14:37.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:37.025+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.028406+0000) 2019-10-13T14:14:37.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:37.025+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.029157+0000) 2019-10-13T14:14:37.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:37.026+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.029763+0000) 2019-10-13T14:14:37.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:37.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:37.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:37.034+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:37.035317+0000) 2019-10-13T14:14:38.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:38.873+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:38.875952+0000) 2019-10-13T14:14:38.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:38.873+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:38.875952+0000) 2019-10-13T14:14:41.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:41.642+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:41.642244+0000) 2019-10-13T14:14:42.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:42.033+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.037894+0000) 2019-10-13T14:14:42.041 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:42.033+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.038196+0000) 2019-10-13T14:14:42.041 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:42.038+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.039272+0000) 2019-10-13T14:14:42.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:42.037+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.039721+0000) 2019-10-13T14:14:42.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:42.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.040352+0000) 2019-10-13T14:14:42.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:42.037+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.040284+0000) 2019-10-13T14:14:42.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:42.037+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.041468+0000) 2019-10-13T14:14:42.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:42.046+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.047279+0000) 2019-10-13T14:14:42.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:42.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191266 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:42.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:42.613+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.615598+0000) 2019-10-13T14:14:45.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:45.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:45.639972+0000) 2019-10-13T14:14:45.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:45.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:45.927551+0000) 2019-10-13T14:14:46.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:46.601+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:46.605549+0000) 2019-10-13T14:14:46.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:46.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:46.855379+0000) 2019-10-13T14:14:47.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:47.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.050599+0000) 2019-10-13T14:14:47.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:47.045+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.051162+0000) 2019-10-13T14:14:47.054 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:47.049+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.052367+0000) 2019-10-13T14:14:47.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:47.049+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.052943+0000) 2019-10-13T14:14:47.055 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:47.049+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.053326+0000) 2019-10-13T14:14:47.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:47.049+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.053670+0000) 2019-10-13T14:14:47.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:47.049+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.054215+0000) 2019-10-13T14:14:47.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:47.057+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:47.058472+0000) 2019-10-13T14:14:47.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:47.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:48.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:48.873+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:48.876212+0000) 2019-10-13T14:14:48.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:48.873+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:48.876211+0000) 2019-10-13T14:14:51.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:51.645+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:51.648659+0000) 2019-10-13T14:14:52.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:52.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.063389+0000) 2019-10-13T14:14:52.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:52.061+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.063779+0000) 2019-10-13T14:14:52.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:52.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.064034+0000) 2019-10-13T14:14:52.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:52.061+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.064653+0000) 2019-10-13T14:14:52.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:52.061+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.065695+0000) 2019-10-13T14:14:52.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:52.061+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.065068+0000) 2019-10-13T14:14:52.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:52.061+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.066542+0000) 2019-10-13T14:14:52.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:52.069+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.072325+0000) 2019-10-13T14:14:52.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:52.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:52.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:52.613+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.615802+0000) 2019-10-13T14:14:52.689 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:14:52.689 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:52.693 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:14:52.693 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:55.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:55.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:55.640199+0000) 2019-10-13T14:14:55.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:55.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:55.927791+0000) 2019-10-13T14:14:56.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:56.605+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:56.605777+0000) 2019-10-13T14:14:56.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:56.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:56.855596+0000) 2019-10-13T14:14:57.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:14:57.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.076029+0000) 2019-10-13T14:14:57.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:14:57.073+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.077144+0000) 2019-10-13T14:14:57.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:14:57.073+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.077769+0000) 2019-10-13T14:14:57.080 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:57.073+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.078293+0000) 2019-10-13T14:14:57.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:14:57.073+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.078595+0000) 2019-10-13T14:14:57.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:14:57.073+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.079250+0000) 2019-10-13T14:14:57.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:57.077+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.079781+0000) 2019-10-13T14:14:57.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:14:57.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191367 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:14:57.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:14:57.085+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:57.089578+0000) 2019-10-13T14:14:58.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:14:58.873+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:58.876402+0000) 2019-10-13T14:14:58.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:14:58.873+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:58.876407+0000) 2019-10-13T14:15:01.657 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:01.653+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:01.655174+0000) 2019-10-13T14:15:02.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:02.085+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.089684+0000) 2019-10-13T14:15:02.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:02.085+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.090914+0000) 2019-10-13T14:15:02.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:02.089+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.091504+0000) 2019-10-13T14:15:02.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:02.089+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.091594+0000) 2019-10-13T14:15:02.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:02.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.092098+0000) 2019-10-13T14:15:02.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:02.089+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.092536+0000) 2019-10-13T14:15:02.096 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:02.089+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.093668+0000) 2019-10-13T14:15:02.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:02.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:02.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:02.097+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.099763+0000) 2019-10-13T14:15:02.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:02.612+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.615996+0000) 2019-10-13T14:15:05.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:05.636+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:05.640394+0000) 2019-10-13T14:15:05.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:05.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:05.927980+0000) 2019-10-13T14:15:06.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:06.604+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:06.605924+0000) 2019-10-13T14:15:06.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:06.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:06.855821+0000) 2019-10-13T14:15:07.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:07.096+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.102096+0000) 2019-10-13T14:15:07.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:07.096+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.102821+0000) 2019-10-13T14:15:07.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:07.100+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.103472+0000) 2019-10-13T14:15:07.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:07.101+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.104148+0000) 2019-10-13T14:15:07.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:07.100+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.104283+0000) 2019-10-13T14:15:07.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:07.100+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.104742+0000) 2019-10-13T14:15:07.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:07.100+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.105516+0000) 2019-10-13T14:15:07.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:07.105+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:07.110108+0000) 2019-10-13T14:15:07.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:07.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:08.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:08.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:08.876627+0000) 2019-10-13T14:15:08.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:08.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:08.876627+0000) 2019-10-13T14:15:11.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:11.657+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:11.661601+0000) 2019-10-13T14:15:12.118 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:12.112+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.115690+0000) 2019-10-13T14:15:12.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:12.112+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.116967+0000) 2019-10-13T14:15:12.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:12.113+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.117263+0000) 2019-10-13T14:15:12.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:12.112+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.117386+0000) 2019-10-13T14:15:12.120 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:12.112+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.117773+0000) 2019-10-13T14:15:12.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:12.112+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.118793+0000) 2019-10-13T14:15:12.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:12.116+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.120108+0000) 2019-10-13T14:15:12.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:12.120+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:12.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:12.125+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.126516+0000) 2019-10-13T14:15:12.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:12.612+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.616204+0000) 2019-10-13T14:15:15.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:15.636+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:15.640625+0000) 2019-10-13T14:15:15.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:15.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:15.928184+0000) 2019-10-13T14:15:16.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:16.600+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:16.606136+0000) 2019-10-13T14:15:16.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:16.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:16.856035+0000) 2019-10-13T14:15:17.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:17.124+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.127802+0000) 2019-10-13T14:15:17.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:17.124+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.128688+0000) 2019-10-13T14:15:17.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:17.129+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.130797+0000) 2019-10-13T14:15:17.133 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:17.124+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.131088+0000) 2019-10-13T14:15:17.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:17.128+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.131787+0000) 2019-10-13T14:15:17.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:17.128+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.132204+0000) 2019-10-13T14:15:17.135 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:17.128+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.132999+0000) 2019-10-13T14:15:17.140 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:17.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:17.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:17.137+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:17.140255+0000) 2019-10-13T14:15:18.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:18.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:18.876879+0000) 2019-10-13T14:15:18.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:18.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:18.876879+0000) 2019-10-13T14:15:21.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:21.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:21.668051+0000) 2019-10-13T14:15:22.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:22.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.140251+0000) 2019-10-13T14:15:22.143 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:22.136+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.141169+0000) 2019-10-13T14:15:22.144 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:22.136+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.141868+0000) 2019-10-13T14:15:22.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:22.136+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.142508+0000) 2019-10-13T14:15:22.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:22.141+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.143511+0000) 2019-10-13T14:15:22.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:22.140+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.144348+0000) 2019-10-13T14:15:22.147 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:22.140+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.144690+0000) 2019-10-13T14:15:22.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:22.149+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.150394+0000) 2019-10-13T14:15:22.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:22.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191534 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:22.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:22.612+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.616381+0000) 2019-10-13T14:15:22.798 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:15:22.799 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:22.802 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:15:22.802 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:25.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:25.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:25.640829+0000) 2019-10-13T14:15:25.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:25.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:25.928383+0000) 2019-10-13T14:15:26.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:26.604+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:26.606369+0000) 2019-10-13T14:15:26.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:26.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:26.856233+0000) 2019-10-13T14:15:27.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:27.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.154011+0000) 2019-10-13T14:15:27.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:27.148+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.154372+0000) 2019-10-13T14:15:27.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:27.153+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.155399+0000) 2019-10-13T14:15:27.158 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:27.152+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.155782+0000) 2019-10-13T14:15:27.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:27.152+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.156222+0000) 2019-10-13T14:15:27.159 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:27.152+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.156724+0000) 2019-10-13T14:15:27.160 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:27.152+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.157612+0000) 2019-10-13T14:15:27.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:27.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:27.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:27.161+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:27.165095+0000) 2019-10-13T14:15:28.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:28.872+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:28.877140+0000) 2019-10-13T14:15:28.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:28.872+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:28.877141+0000) 2019-10-13T14:15:31.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:31.669+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:31.670616+0000) 2019-10-13T14:15:32.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:32.164+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.168075+0000) 2019-10-13T14:15:32.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:32.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.168398+0000) 2019-10-13T14:15:32.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:32.169+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.170319+0000) 2019-10-13T14:15:32.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:32.164+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.170335+0000) 2019-10-13T14:15:32.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:32.164+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.170681+0000) 2019-10-13T14:15:32.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:32.168+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.171507+0000) 2019-10-13T14:15:32.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:32.168+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.171918+0000) 2019-10-13T14:15:32.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:32.173+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.175932+0000) 2019-10-13T14:15:32.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:32.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:32.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:32.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.616564+0000) 2019-10-13T14:15:35.643 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:35.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:35.641032+0000) 2019-10-13T14:15:35.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:35.924+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:35.928586+0000) 2019-10-13T14:15:36.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:36.604+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:36.606564+0000) 2019-10-13T14:15:36.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:36.853+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:36.856456+0000) 2019-10-13T14:15:37.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:37.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.180057+0000) 2019-10-13T14:15:37.183 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:37.176+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.180411+0000) 2019-10-13T14:15:37.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:37.177+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.181416+0000) 2019-10-13T14:15:37.184 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:37.176+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.180979+0000) 2019-10-13T14:15:37.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:37.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.181628+0000) 2019-10-13T14:15:37.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:37.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.182385+0000) 2019-10-13T14:15:37.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:37.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.182834+0000) 2019-10-13T14:15:37.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:37.181+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:37.186117+0000) 2019-10-13T14:15:37.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:37.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:38.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:38.876+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:38.877365+0000) 2019-10-13T14:15:38.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:38.876+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:38.877365+0000) 2019-10-13T14:15:41.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:41.676+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:41.677046+0000) 2019-10-13T14:15:42.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:42.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.193419+0000) 2019-10-13T14:15:42.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:42.188+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.193860+0000) 2019-10-13T14:15:42.197 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:42.188+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.194686+0000) 2019-10-13T14:15:42.197 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:42.188+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.195139+0000) 2019-10-13T14:15:42.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:42.192+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.196160+0000) 2019-10-13T14:15:42.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:42.188+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.194540+0000) 2019-10-13T14:15:42.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:42.192+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.196787+0000) 2019-10-13T14:15:42.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:42.200+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.202987+0000) 2019-10-13T14:15:42.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:42.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:42.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:42.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.616744+0000) 2019-10-13T14:15:45.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:45.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:45.641254+0000) 2019-10-13T14:15:45.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:45.923+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:45.928741+0000) 2019-10-13T14:15:46.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:46.599+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:46.606762+0000) 2019-10-13T14:15:46.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:46.852+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:46.856677+0000) 2019-10-13T14:15:47.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:47.199+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.206487+0000) 2019-10-13T14:15:47.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:47.199+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.207086+0000) 2019-10-13T14:15:47.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:47.204+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.208043+0000) 2019-10-13T14:15:47.211 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:47.203+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.208060+0000) 2019-10-13T14:15:47.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:47.203+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.208432+0000) 2019-10-13T14:15:47.212 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:47.203+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.209187+0000) 2019-10-13T14:15:47.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:47.203+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.209756+0000) 2019-10-13T14:15:47.219 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:47.212+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:47.216705+0000) 2019-10-13T14:15:47.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:47.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:48.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:48.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:48.877609+0000) 2019-10-13T14:15:48.880 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:48.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:48.877610+0000) 2019-10-13T14:15:51.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:51.680+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:51.683807+0000) 2019-10-13T14:15:52.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:52.215+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.221116+0000) 2019-10-13T14:15:52.224 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:52.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.221308+0000) 2019-10-13T14:15:52.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:52.215+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.221615+0000) 2019-10-13T14:15:52.225 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:52.215+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.221977+0000) 2019-10-13T14:15:52.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:52.220+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.222451+0000) 2019-10-13T14:15:52.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:52.215+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.222632+0000) 2019-10-13T14:15:52.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:52.215+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.223175+0000) 2019-10-13T14:15:52.232 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:52.224+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.229672+0000) 2019-10-13T14:15:52.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:52.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191736 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:52.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:52.611+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.616939+0000) 2019-10-13T14:15:53.001 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:15:53.001 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:53.005 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:15:53.005 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:55.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:55.639+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:55.641458+0000) 2019-10-13T14:15:55.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:55.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:55.928946+0000) 2019-10-13T14:15:56.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:56.603+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:56.606959+0000) 2019-10-13T14:15:56.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:56.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:56.856878+0000) 2019-10-13T14:15:57.235 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:15:57.227+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.233194+0000) 2019-10-13T14:15:57.237 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:15:57.227+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.234629+0000) 2019-10-13T14:15:57.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:15:57.227+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.235163+0000) 2019-10-13T14:15:57.238 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:15:57.231+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.235439+0000) 2019-10-13T14:15:57.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:15:57.232+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.235977+0000) 2019-10-13T14:15:57.239 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:57.231+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.236262+0000) 2019-10-13T14:15:57.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:57.231+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.236697+0000) 2019-10-13T14:15:57.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:15:57.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191771 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:15:57.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:15:57.240+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:57.243814+0000) 2019-10-13T14:15:58.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:15:58.875+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:58.877853+0000) 2019-10-13T14:15:58.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:15:58.875+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:58.877853+0000) 2019-10-13T14:16:01.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:01.688+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:01.690178+0000) 2019-10-13T14:16:02.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:02.240+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.245740+0000) 2019-10-13T14:16:02.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:02.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.245884+0000) 2019-10-13T14:16:02.249 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:02.239+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.246389+0000) 2019-10-13T14:16:02.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:02.239+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.246987+0000) 2019-10-13T14:16:02.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:02.239+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.247028+0000) 2019-10-13T14:16:02.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:02.243+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.248815+0000) 2019-10-13T14:16:02.252 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:02.243+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.249543+0000) 2019-10-13T14:16:02.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:02.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:02.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:02.256+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.259172+0000) 2019-10-13T14:16:02.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:02.615+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.617152+0000) 2019-10-13T14:16:05.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:05.639+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:05.641657+0000) 2019-10-13T14:16:05.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:05.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:05.929140+0000) 2019-10-13T14:16:06.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:06.603+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:06.607161+0000) 2019-10-13T14:16:06.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:06.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:06.857116+0000) 2019-10-13T14:16:07.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:07.251+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.258842+0000) 2019-10-13T14:16:07.262 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:07.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.258925+0000) 2019-10-13T14:16:07.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:07.255+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.259343+0000) 2019-10-13T14:16:07.263 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:07.255+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.260549+0000) 2019-10-13T14:16:07.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:07.256+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.261021+0000) 2019-10-13T14:16:07.264 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:07.255+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.261777+0000) 2019-10-13T14:16:07.264 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:07.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.262108+0000) 2019-10-13T14:16:07.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:07.264+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:07.268110+0000) 2019-10-13T14:16:07.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:07.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:08.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:08.871+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:08.878080+0000) 2019-10-13T14:16:08.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:08.871+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:08.878080+0000) 2019-10-13T14:16:11.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:11.696+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:11.696658+0000) 2019-10-13T14:16:12.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:12.267+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.271063+0000) 2019-10-13T14:16:12.274 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:12.267+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.271439+0000) 2019-10-13T14:16:12.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:12.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.272862+0000) 2019-10-13T14:16:12.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:12.268+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.273330+0000) 2019-10-13T14:16:12.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:12.267+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.273519+0000) 2019-10-13T14:16:12.276 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:12.267+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.274059+0000) 2019-10-13T14:16:12.277 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:12.267+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.275044+0000) 2019-10-13T14:16:12.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:12.276+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.280133+0000) 2019-10-13T14:16:12.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:12.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:12.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:12.615+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.617332+0000) 2019-10-13T14:16:15.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:15.639+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:15.641861+0000) 2019-10-13T14:16:15.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:15.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:15.929356+0000) 2019-10-13T14:16:16.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:16.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:16.607357+0000) 2019-10-13T14:16:16.859 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:16.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:16.857320+0000) 2019-10-13T14:16:17.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:17.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.285034+0000) 2019-10-13T14:16:17.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:17.283+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.285585+0000) 2019-10-13T14:16:17.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:17.283+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.285908+0000) 2019-10-13T14:16:17.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:17.283+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.286337+0000) 2019-10-13T14:16:17.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:17.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.286676+0000) 2019-10-13T14:16:17.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:17.284+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.287821+0000) 2019-10-13T14:16:17.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:17.287+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.288358+0000) 2019-10-13T14:16:17.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:17.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:17.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:17.292+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:17.296257+0000) 2019-10-13T14:16:18.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:18.875+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:18.878307+0000) 2019-10-13T14:16:18.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:18.875+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:18.878306+0000) 2019-10-13T14:16:21.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:21.700+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:21.703411+0000) 2019-10-13T14:16:22.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:22.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.297604+0000) 2019-10-13T14:16:22.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:22.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.297636+0000) 2019-10-13T14:16:22.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:22.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.297950+0000) 2019-10-13T14:16:22.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:22.296+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.298792+0000) 2019-10-13T14:16:22.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:22.299+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.299588+0000) 2019-10-13T14:16:22.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:22.299+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.299735+0000) 2019-10-13T14:16:22.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:22.299+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.300122+0000) 2019-10-13T14:16:22.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:22.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:22.311 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:22.304+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.309257+0000) 2019-10-13T14:16:22.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:22.615+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.617513+0000) 2019-10-13T14:16:23.208 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:16:23.208 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:23.212 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:16:23.212 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:25.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:25.639+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:25.642057+0000) 2019-10-13T14:16:25.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:25.927+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:25.929554+0000) 2019-10-13T14:16:26.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:26.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:26.607552+0000) 2019-10-13T14:16:26.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:26.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:26.857532+0000) 2019-10-13T14:16:27.312 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:27.307+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.310084+0000) 2019-10-13T14:16:27.313 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:27.307+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.310614+0000) 2019-10-13T14:16:27.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:27.311+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.311711+0000) 2019-10-13T14:16:27.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:27.311+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.311956+0000) 2019-10-13T14:16:27.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:27.311+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.312872+0000) 2019-10-13T14:16:27.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:27.308+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.313371+0000) 2019-10-13T14:16:27.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:27.311+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.314249+0000) 2019-10-13T14:16:27.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:27.319+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 191974 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:27.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:27.320+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:27.322340+0000) 2019-10-13T14:16:28.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:28.875+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:28.878533+0000) 2019-10-13T14:16:28.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:28.875+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:28.878533+0000) 2019-10-13T14:16:31.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:31.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:31.709689+0000) 2019-10-13T14:16:32.325 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:32.318+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.322989+0000) 2019-10-13T14:16:32.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:32.322+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.324017+0000) 2019-10-13T14:16:32.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:32.322+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.323741+0000) 2019-10-13T14:16:32.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:32.322+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.324846+0000) 2019-10-13T14:16:32.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:32.322+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.325420+0000) 2019-10-13T14:16:32.328 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:32.322+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.325903+0000) 2019-10-13T14:16:32.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:32.324+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.327518+0000) 2019-10-13T14:16:32.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:32.328+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.332655+0000) 2019-10-13T14:16:32.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:32.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:32.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:32.614+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.617726+0000) 2019-10-13T14:16:35.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:35.638+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:35.642250+0000) 2019-10-13T14:16:35.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:35.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:35.929749+0000) 2019-10-13T14:16:36.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:36.606+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:36.607765+0000) 2019-10-13T14:16:36.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:36.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:36.857758+0000) 2019-10-13T14:16:37.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:37.334+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.336027+0000) 2019-10-13T14:16:37.339 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:37.334+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.336566+0000) 2019-10-13T14:16:37.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:37.334+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.336945+0000) 2019-10-13T14:16:37.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:37.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.337591+0000) 2019-10-13T14:16:37.340 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:37.335+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.338206+0000) 2019-10-13T14:16:37.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:37.334+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.338808+0000) 2019-10-13T14:16:37.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:37.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.338936+0000) 2019-10-13T14:16:37.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:37.339+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:37.342418+0000) 2019-10-13T14:16:37.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:37.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:38.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:38.874+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:38.878780+0000) 2019-10-13T14:16:38.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:38.874+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:38.878780+0000) 2019-10-13T14:16:41.719 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:41.711+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:41.716110+0000) 2019-10-13T14:16:42.351 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:42.346+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.349082+0000) 2019-10-13T14:16:42.352 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:42.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.349470+0000) 2019-10-13T14:16:42.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:42.346+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.350252+0000) 2019-10-13T14:16:42.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:42.347+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.350801+0000) 2019-10-13T14:16:42.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:42.346+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.351100+0000) 2019-10-13T14:16:42.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:42.346+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.351214+0000) 2019-10-13T14:16:42.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:42.350+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.351701+0000) 2019-10-13T14:16:42.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:42.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192077 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:42.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:42.355+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.359166+0000) 2019-10-13T14:16:42.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:42.614+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.617933+0000) 2019-10-13T14:16:45.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:45.638+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:45.642472+0000) 2019-10-13T14:16:45.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:45.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:45.929945+0000) 2019-10-13T14:16:46.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:46.606+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:46.607959+0000) 2019-10-13T14:16:46.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:46.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:46.857953+0000) 2019-10-13T14:16:47.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:47.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.362290+0000) 2019-10-13T14:16:47.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:47.358+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.362488+0000) 2019-10-13T14:16:47.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:47.358+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.362841+0000) 2019-10-13T14:16:47.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:47.362+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.364477+0000) 2019-10-13T14:16:47.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:47.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.364778+0000) 2019-10-13T14:16:47.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:47.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.366026+0000) 2019-10-13T14:16:47.368 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:47.362+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.366292+0000) 2019-10-13T14:16:47.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:47.363+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:47.369840+0000) 2019-10-13T14:16:47.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:47.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:48.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:48.874+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:48.879033+0000) 2019-10-13T14:16:48.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:48.874+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:48.879033+0000) 2019-10-13T14:16:51.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:51.719+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:51.722528+0000) 2019-10-13T14:16:52.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:52.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.375951+0000) 2019-10-13T14:16:52.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:52.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.376153+0000) 2019-10-13T14:16:52.379 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:52.371+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.376563+0000) 2019-10-13T14:16:52.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:52.374+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.376778+0000) 2019-10-13T14:16:52.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:52.374+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.377212+0000) 2019-10-13T14:16:52.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:52.374+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.378386+0000) 2019-10-13T14:16:52.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:52.374+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.378609+0000) 2019-10-13T14:16:52.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:52.379+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.384550+0000) 2019-10-13T14:16:52.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:52.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:52.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:52.614+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.618132+0000) 2019-10-13T14:16:53.273 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:16:53.273 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:53.277 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:16:53.277 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:55.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:55.638+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:55.642668+0000) 2019-10-13T14:16:55.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:55.926+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:55.930142+0000) 2019-10-13T14:16:56.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:56.606+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:56.608150+0000) 2019-10-13T14:16:56.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:56.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:56.858201+0000) 2019-10-13T14:16:57.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:16:57.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.389250+0000) 2019-10-13T14:16:57.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:57.386+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.389573+0000) 2019-10-13T14:16:57.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:16:57.387+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.390185+0000) 2019-10-13T14:16:57.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:57.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.390259+0000) 2019-10-13T14:16:57.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:16:57.386+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.390558+0000) 2019-10-13T14:16:57.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:16:57.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.391018+0000) 2019-10-13T14:16:57.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:16:57.390+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.392514+0000) 2019-10-13T14:16:57.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:16:57.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:57.398233+0000) 2019-10-13T14:16:57.401 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:16:57.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:16:58.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:16:58.874+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:58.879251+0000) 2019-10-13T14:16:58.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:16:58.874+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:58.879251+0000) 2019-10-13T14:17:01.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:01.727+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:01.729044+0000) 2019-10-13T14:17:02.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:02.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.401784+0000) 2019-10-13T14:17:02.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:02.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.402686+0000) 2019-10-13T14:17:02.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:02.398+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.403131+0000) 2019-10-13T14:17:02.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:02.399+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.404838+0000) 2019-10-13T14:17:02.407 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:02.402+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.404823+0000) 2019-10-13T14:17:02.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:02.402+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.405455+0000) 2019-10-13T14:17:02.408 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:02.402+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.405868+0000) 2019-10-13T14:17:02.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:02.407+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.410763+0000) 2019-10-13T14:17:02.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:02.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:02.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:02.614+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.618333+0000) 2019-10-13T14:17:05.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:05.642+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:05.642870+0000) 2019-10-13T14:17:05.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:05.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:05.930293+0000) 2019-10-13T14:17:06.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:06.606+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:06.608344+0000) 2019-10-13T14:17:06.860 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:06.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:06.858418+0000) 2019-10-13T14:17:07.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:07.414+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.415298+0000) 2019-10-13T14:17:07.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:07.414+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.416532+0000) 2019-10-13T14:17:07.419 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:07.414+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.416922+0000) 2019-10-13T14:17:07.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:07.414+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.417858+0000) 2019-10-13T14:17:07.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:07.414+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.418269+0000) 2019-10-13T14:17:07.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:07.414+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.418715+0000) 2019-10-13T14:17:07.421 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:07.415+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.419225+0000) 2019-10-13T14:17:07.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:07.419+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:07.422342+0000) 2019-10-13T14:17:07.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:07.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192246 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:08.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:08.878+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:08.879478+0000) 2019-10-13T14:17:08.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:08.878+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:08.879478+0000) 2019-10-13T14:17:11.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:11.731+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:11.735079+0000) 2019-10-13T14:17:12.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:12.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.429233+0000) 2019-10-13T14:17:12.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:12.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.429659+0000) 2019-10-13T14:17:12.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:12.426+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.430390+0000) 2019-10-13T14:17:12.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:12.426+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.430585+0000) 2019-10-13T14:17:12.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:12.430+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.431295+0000) 2019-10-13T14:17:12.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:12.430+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.431646+0000) 2019-10-13T14:17:12.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:12.427+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.433140+0000) 2019-10-13T14:17:12.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:12.431+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.434698+0000) 2019-10-13T14:17:12.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:12.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:12.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:12.613+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.618537+0000) 2019-10-13T14:17:15.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:15.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:15.643064+0000) 2019-10-13T14:17:15.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:15.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:15.930503+0000) 2019-10-13T14:17:16.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:16.605+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:16.608570+0000) 2019-10-13T14:17:16.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:16.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:16.858613+0000) 2019-10-13T14:17:17.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:17.437+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.441909+0000) 2019-10-13T14:17:17.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:17.437+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.442706+0000) 2019-10-13T14:17:17.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:17.437+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.443254+0000) 2019-10-13T14:17:17.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:17.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.443902+0000) 2019-10-13T14:17:17.446 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:17.441+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.444084+0000) 2019-10-13T14:17:17.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:17.439+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.444896+0000) 2019-10-13T14:17:17.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:17.441+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.445730+0000) 2019-10-13T14:17:17.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:17.449+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:17.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:17.451+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:17.455125+0000) 2019-10-13T14:17:18.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:18.877+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:18.879739+0000) 2019-10-13T14:17:18.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:18.877+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:18.879739+0000) 2019-10-13T14:17:21.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:21.739+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:21.741353+0000) 2019-10-13T14:17:22.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:22.449+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.454839+0000) 2019-10-13T14:17:22.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:22.451+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.455495+0000) 2019-10-13T14:17:22.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:22.453+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.455775+0000) 2019-10-13T14:17:22.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:22.453+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.456457+0000) 2019-10-13T14:17:22.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:22.453+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.457250+0000) 2019-10-13T14:17:22.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:22.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.457856+0000) 2019-10-13T14:17:22.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:22.453+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.457949+0000) 2019-10-13T14:17:22.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:22.459+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.462388+0000) 2019-10-13T14:17:22.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:22.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:22.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:22.617+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.618684+0000) 2019-10-13T14:17:23.376 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:17:23.376 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:23.380 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:17:23.380 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:25.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:25.637+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:25.643262+0000) 2019-10-13T14:17:25.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:25.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:25.930750+0000) 2019-10-13T14:17:26.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:26.605+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:26.608762+0000) 2019-10-13T14:17:26.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:26.855+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:26.858785+0000) 2019-10-13T14:17:27.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:27.465+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.468816+0000) 2019-10-13T14:17:27.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:27.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.469563+0000) 2019-10-13T14:17:27.472 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:27.465+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.470069+0000) 2019-10-13T14:17:27.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:27.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.470742+0000) 2019-10-13T14:17:27.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:27.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.470912+0000) 2019-10-13T14:17:27.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:27.469+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.471718+0000) 2019-10-13T14:17:27.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:27.469+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.472380+0000) 2019-10-13T14:17:27.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:27.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:27.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:27.474+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:27.479469+0000) 2019-10-13T14:17:28.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:28.877+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:28.879994+0000) 2019-10-13T14:17:28.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:28.877+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:28.879994+0000) 2019-10-13T14:17:31.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:31.742+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:31.747777+0000) 2019-10-13T14:17:32.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:32.477+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.482806+0000) 2019-10-13T14:17:32.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:32.481+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.483294+0000) 2019-10-13T14:17:32.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:32.478+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.484012+0000) 2019-10-13T14:17:32.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:32.481+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.484216+0000) 2019-10-13T14:17:32.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:32.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.484726+0000) 2019-10-13T14:17:32.488 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:32.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.484806+0000) 2019-10-13T14:17:32.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:32.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.485995+0000) 2019-10-13T14:17:32.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:32.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:32.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:32.486+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.493609+0000) 2019-10-13T14:17:32.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:32.617+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.618900+0000) 2019-10-13T14:17:35.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:35.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:35.643496+0000) 2019-10-13T14:17:35.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:35.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:35.930984+0000) 2019-10-13T14:17:36.611 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:36.605+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:36.608970+0000) 2019-10-13T14:17:36.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:36.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:36.858992+0000) 2019-10-13T14:17:37.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:37.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.495688+0000) 2019-10-13T14:17:37.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:37.493+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.496036+0000) 2019-10-13T14:17:37.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:37.490+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.496758+0000) 2019-10-13T14:17:37.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:37.493+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.496950+0000) 2019-10-13T14:17:37.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:37.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.498216+0000) 2019-10-13T14:17:37.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:37.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.498573+0000) 2019-10-13T14:17:37.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:37.497+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.499755+0000) 2019-10-13T14:17:37.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:37.498+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:37.504676+0000) 2019-10-13T14:17:37.507 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:37.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192450 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:38.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:38.877+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:38.880219+0000) 2019-10-13T14:17:38.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:38.877+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:38.880218+0000) 2019-10-13T14:17:41.756 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:41.750+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:41.754102+0000) 2019-10-13T14:17:42.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:42.505+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.508670+0000) 2019-10-13T14:17:42.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:42.505+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.509392+0000) 2019-10-13T14:17:42.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:42.505+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.509887+0000) 2019-10-13T14:17:42.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:42.506+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.510398+0000) 2019-10-13T14:17:42.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:42.505+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.510792+0000) 2019-10-13T14:17:42.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:42.505+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.510848+0000) 2019-10-13T14:17:42.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:42.509+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.511736+0000) 2019-10-13T14:17:42.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:42.510+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.515651+0000) 2019-10-13T14:17:42.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:42.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:42.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:42.617+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.619080+0000) 2019-10-13T14:17:45.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:45.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:45.643742+0000) 2019-10-13T14:17:45.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:45.925+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:45.931211+0000) 2019-10-13T14:17:46.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:46.605+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:46.609164+0000) 2019-10-13T14:17:46.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:46.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:46.859217+0000) 2019-10-13T14:17:47.524 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:47.517+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.521492+0000) 2019-10-13T14:17:47.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:47.517+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.521925+0000) 2019-10-13T14:17:47.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:47.517+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.522543+0000) 2019-10-13T14:17:47.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:47.518+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.523185+0000) 2019-10-13T14:17:47.526 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:47.517+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.523262+0000) 2019-10-13T14:17:47.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:47.521+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.523907+0000) 2019-10-13T14:17:47.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:47.521+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.524892+0000) 2019-10-13T14:17:47.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:47.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192518 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:47.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:47.526+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:47.532955+0000) 2019-10-13T14:17:48.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:48.877+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:48.880442+0000) 2019-10-13T14:17:48.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:48.877+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:48.880441+0000) 2019-10-13T14:17:51.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:51.758+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:51.760490+0000) 2019-10-13T14:17:52.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:52.529+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.534386+0000) 2019-10-13T14:17:52.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:52.529+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.535039+0000) 2019-10-13T14:17:52.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:52.529+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.535282+0000) 2019-10-13T14:17:52.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:52.533+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.536353+0000) 2019-10-13T14:17:52.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:52.533+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.537706+0000) 2019-10-13T14:17:52.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:52.534+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.538527+0000) 2019-10-13T14:17:52.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:52.533+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.538693+0000) 2019-10-13T14:17:52.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:52.538+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.543563+0000) 2019-10-13T14:17:52.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:52.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:52.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:52.617+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.619264+0000) 2019-10-13T14:17:53.438 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:17:53.438 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:53.442 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:17:53.442 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:55.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:55.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:55.643971+0000) 2019-10-13T14:17:55.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:55.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:55.931422+0000) 2019-10-13T14:17:56.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:56.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:56.609397+0000) 2019-10-13T14:17:56.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:56.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:56.859416+0000) 2019-10-13T14:17:57.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:17:57.544+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.549283+0000) 2019-10-13T14:17:57.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:17:57.542+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.550039+0000) 2019-10-13T14:17:57.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:57.544+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.549548+0000) 2019-10-13T14:17:57.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:17:57.544+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.550484+0000) 2019-10-13T14:17:57.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:17:57.544+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.550787+0000) 2019-10-13T14:17:57.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:57.544+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.551063+0000) 2019-10-13T14:17:57.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:17:57.548+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.552460+0000) 2019-10-13T14:17:57.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:17:57.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:17:57.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:17:57.550+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:57.558002+0000) 2019-10-13T14:17:58.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:17:58.876+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:58.880672+0000) 2019-10-13T14:17:58.883 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:17:58.876+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:58.880671+0000) 2019-10-13T14:18:01.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:01.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:01.766730+0000) 2019-10-13T14:18:02.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:02.556+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.560582+0000) 2019-10-13T14:18:02.564 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:02.556+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.562140+0000) 2019-10-13T14:18:02.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:02.556+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.562392+0000) 2019-10-13T14:18:02.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:02.556+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.563051+0000) 2019-10-13T14:18:02.566 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:02.558+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.563602+0000) 2019-10-13T14:18:02.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:02.560+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.563634+0000) 2019-10-13T14:18:02.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:02.560+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.564576+0000) 2019-10-13T14:18:02.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:02.562+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.567852+0000) 2019-10-13T14:18:02.573 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:02.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:02.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:02.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.619446+0000) 2019-10-13T14:18:05.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:05.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:05.644205+0000) 2019-10-13T14:18:05.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:05.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:05.931615+0000) 2019-10-13T14:18:06.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:06.604+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:06.609583+0000) 2019-10-13T14:18:06.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:06.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:06.859673+0000) 2019-10-13T14:18:07.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:07.572+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.575923+0000) 2019-10-13T14:18:07.579 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:07.572+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.576672+0000) 2019-10-13T14:18:07.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:07.572+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.576969+0000) 2019-10-13T14:18:07.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:07.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.577500+0000) 2019-10-13T14:18:07.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:07.572+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.577650+0000) 2019-10-13T14:18:07.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:07.572+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.577983+0000) 2019-10-13T14:18:07.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:07.572+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.578385+0000) 2019-10-13T14:18:07.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:07.574+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:07.581662+0000) 2019-10-13T14:18:07.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:07.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:08.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:08.880+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:08.880896+0000) 2019-10-13T14:18:08.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:08.880+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:08.880895+0000) 2019-10-13T14:18:11.775 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:11.770+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:11.772955+0000) 2019-10-13T14:18:12.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:12.584+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.588282+0000) 2019-10-13T14:18:12.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:12.584+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.588796+0000) 2019-10-13T14:18:12.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:12.584+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.589856+0000) 2019-10-13T14:18:12.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:12.584+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.590359+0000) 2019-10-13T14:18:12.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:12.584+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.591126+0000) 2019-10-13T14:18:12.594 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:12.586+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.591742+0000) 2019-10-13T14:18:12.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:12.588+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.591985+0000) 2019-10-13T14:18:12.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:12.590+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.597836+0000) 2019-10-13T14:18:12.600 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:12.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192687 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:12.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:12.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.619598+0000) 2019-10-13T14:18:15.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:15.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:15.644422+0000) 2019-10-13T14:18:15.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:15.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:15.931857+0000) 2019-10-13T14:18:16.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:16.604+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:16.609779+0000) 2019-10-13T14:18:16.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:16.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:16.859892+0000) 2019-10-13T14:18:17.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:17.596+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.601433+0000) 2019-10-13T14:18:17.604 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:17.598+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.602104+0000) 2019-10-13T14:18:17.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:17.596+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.602529+0000) 2019-10-13T14:18:17.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:17.596+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.603270+0000) 2019-10-13T14:18:17.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:17.600+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.603735+0000) 2019-10-13T14:18:17.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:17.600+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.604052+0000) 2019-10-13T14:18:17.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:17.600+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.604447+0000) 2019-10-13T14:18:17.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:17.606+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:17.609331+0000) 2019-10-13T14:18:17.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:17.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:18.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:18.880+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:18.881120+0000) 2019-10-13T14:18:18.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:18.880+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:18.881120+0000) 2019-10-13T14:18:21.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:21.777+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:21.779259+0000) 2019-10-13T14:18:22.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:22.612+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.615953+0000) 2019-10-13T14:18:22.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:22.613+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.616977+0000) 2019-10-13T14:18:22.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:22.612+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.616352+0000) 2019-10-13T14:18:22.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:22.612+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.617435+0000) 2019-10-13T14:18:22.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:22.612+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.617839+0000) 2019-10-13T14:18:22.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:22.612+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.618171+0000) 2019-10-13T14:18:22.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:22.612+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.618502+0000) 2019-10-13T14:18:22.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:22.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.619720+0000) 2019-10-13T14:18:22.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:22.621+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.622207+0000) 2019-10-13T14:18:22.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:22.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:23.549 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:18:23.549 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:23.553 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:18:23.553 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:25.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:25.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:25.644654+0000) 2019-10-13T14:18:25.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:25.928+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:25.932077+0000) 2019-10-13T14:18:26.612 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:26.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:26.610009+0000) 2019-10-13T14:18:26.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:26.857+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:26.860044+0000) 2019-10-13T14:18:27.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:27.624+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.629178+0000) 2019-10-13T14:18:27.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:27.624+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.629723+0000) 2019-10-13T14:18:27.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:27.624+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.630275+0000) 2019-10-13T14:18:27.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:27.624+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.630766+0000) 2019-10-13T14:18:27.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:27.624+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.631290+0000) 2019-10-13T14:18:27.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:27.629+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.631997+0000) 2019-10-13T14:18:27.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:27.628+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.632416+0000) 2019-10-13T14:18:27.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:27.633+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:27.637731+0000) 2019-10-13T14:18:27.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:27.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:28.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:28.880+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:28.881363+0000) 2019-10-13T14:18:28.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:28.880+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:28.881363+0000) 2019-10-13T14:18:31.788 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:31.781+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:31.785740+0000) 2019-10-13T14:18:32.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:32.616+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.619903+0000) 2019-10-13T14:18:32.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:32.637+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.642079+0000) 2019-10-13T14:18:32.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:32.636+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.642724+0000) 2019-10-13T14:18:32.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:32.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.642986+0000) 2019-10-13T14:18:32.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:32.640+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.643621+0000) 2019-10-13T14:18:32.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:32.640+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.644114+0000) 2019-10-13T14:18:32.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:32.640+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.644521+0000) 2019-10-13T14:18:32.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:32.640+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.645498+0000) 2019-10-13T14:18:32.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:32.649+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.650638+0000) 2019-10-13T14:18:32.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:32.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:35.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:35.640+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:35.644856+0000) 2019-10-13T14:18:35.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:35.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:35.932285+0000) 2019-10-13T14:18:36.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:36.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:36.610203+0000) 2019-10-13T14:18:36.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:36.857+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:36.860251+0000) 2019-10-13T14:18:37.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:37.648+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.654670+0000) 2019-10-13T14:18:37.657 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:37.648+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.654977+0000) 2019-10-13T14:18:37.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:37.653+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.656416+0000) 2019-10-13T14:18:37.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:37.652+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.656597+0000) 2019-10-13T14:18:37.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:37.652+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.657227+0000) 2019-10-13T14:18:37.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:37.652+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.657613+0000) 2019-10-13T14:18:37.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:37.652+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.658526+0000) 2019-10-13T14:18:37.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:37.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:37.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:37.661+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:37.664837+0000) 2019-10-13T14:18:38.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:38.880+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:38.881593+0000) 2019-10-13T14:18:38.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:38.880+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:38.881592+0000) 2019-10-13T14:18:41.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:41.789+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:41.792179+0000) 2019-10-13T14:18:42.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:42.619+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.620102+0000) 2019-10-13T14:18:42.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:42.663+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.667477+0000) 2019-10-13T14:18:42.670 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:42.663+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.668262+0000) 2019-10-13T14:18:42.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:42.665+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.669466+0000) 2019-10-13T14:18:42.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:42.663+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.669167+0000) 2019-10-13T14:18:42.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:42.663+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.669650+0000) 2019-10-13T14:18:42.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:42.663+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.670471+0000) 2019-10-13T14:18:42.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:42.663+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.670965+0000) 2019-10-13T14:18:42.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:42.671+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:42.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:42.677+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.678327+0000) 2019-10-13T14:18:45.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:45.639+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:45.645049+0000) 2019-10-13T14:18:45.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:45.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:45.932481+0000) 2019-10-13T14:18:46.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:46.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:46.610405+0000) 2019-10-13T14:18:46.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:46.857+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:46.860446+0000) 2019-10-13T14:18:47.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:47.675+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.681083+0000) 2019-10-13T14:18:47.684 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:47.675+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.681547+0000) 2019-10-13T14:18:47.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:47.675+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.681956+0000) 2019-10-13T14:18:47.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:47.681+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.682451+0000) 2019-10-13T14:18:47.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:47.675+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.682486+0000) 2019-10-13T14:18:47.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:47.679+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.683717+0000) 2019-10-13T14:18:47.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:47.679+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.684812+0000) 2019-10-13T14:18:47.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:47.685+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:47.689788+0000) 2019-10-13T14:18:47.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:47.683+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:48.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:48.875+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:48.881819+0000) 2019-10-13T14:18:48.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:48.875+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:48.881819+0000) 2019-10-13T14:18:51.801 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:51.797+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:51.798537+0000) 2019-10-13T14:18:52.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:52.615+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.620316+0000) 2019-10-13T14:18:52.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:52.687+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.693966+0000) 2019-10-13T14:18:52.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:52.687+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.694842+0000) 2019-10-13T14:18:52.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:52.693+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.695226+0000) 2019-10-13T14:18:52.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:52.691+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.695781+0000) 2019-10-13T14:18:52.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:52.691+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.696567+0000) 2019-10-13T14:18:52.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:52.691+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.697202+0000) 2019-10-13T14:18:52.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:52.691+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.698082+0000) 2019-10-13T14:18:52.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:52.697+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.701770+0000) 2019-10-13T14:18:52.706 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:52.699+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:53.594 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:18:53.594 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:53.598 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:18:53.598 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:55.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:55.643+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:55.645269+0000) 2019-10-13T14:18:55.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:55.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:55.932694+0000) 2019-10-13T14:18:56.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:56.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:56.610621+0000) 2019-10-13T14:18:56.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:56.857+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:56.860642+0000) 2019-10-13T14:18:57.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:18:57.703+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.706982+0000) 2019-10-13T14:18:57.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:18:57.703+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.707364+0000) 2019-10-13T14:18:57.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:18:57.705+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.707878+0000) 2019-10-13T14:18:57.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:57.703+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.708356+0000) 2019-10-13T14:18:57.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:18:57.703+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.708679+0000) 2019-10-13T14:18:57.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:18:57.703+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.709444+0000) 2019-10-13T14:18:57.712 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:57.703+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.709479+0000) 2019-10-13T14:18:57.716 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:18:57.713+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:57.714584+0000) 2019-10-13T14:18:57.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:18:57.711+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 192995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:18:58.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:18:58.879+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:58.882064+0000) 2019-10-13T14:18:58.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:18:58.879+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:58.882065+0000) 2019-10-13T14:19:01.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:01.801+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:01.804868+0000) 2019-10-13T14:19:02.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:02.615+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.620510+0000) 2019-10-13T14:19:02.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:02.715+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.719672+0000) 2019-10-13T14:19:02.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:02.717+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.720208+0000) 2019-10-13T14:19:02.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:02.715+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.720931+0000) 2019-10-13T14:19:02.724 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:02.715+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.721574+0000) 2019-10-13T14:19:02.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:02.715+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.722024+0000) 2019-10-13T14:19:02.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:02.715+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.722459+0000) 2019-10-13T14:19:02.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:02.715+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.723107+0000) 2019-10-13T14:19:02.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:02.725+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.727201+0000) 2019-10-13T14:19:02.732 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:02.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:05.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:05.643+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:05.645487+0000) 2019-10-13T14:19:05.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:05.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:05.932918+0000) 2019-10-13T14:19:06.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:06.603+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:06.610819+0000) 2019-10-13T14:19:06.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:06.857+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:06.860839+0000) 2019-10-13T14:19:07.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:07.727+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.733481+0000) 2019-10-13T14:19:07.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:07.727+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.733808+0000) 2019-10-13T14:19:07.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:07.727+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.734698+0000) 2019-10-13T14:19:07.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:07.733+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.735410+0000) 2019-10-13T14:19:07.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:07.731+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.735675+0000) 2019-10-13T14:19:07.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:07.731+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.736642+0000) 2019-10-13T14:19:07.739 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:07.731+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.737258+0000) 2019-10-13T14:19:07.745 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:07.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:07.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:07.741+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:07.744761+0000) 2019-10-13T14:19:08.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:08.879+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:08.882327+0000) 2019-10-13T14:19:08.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:08.879+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:08.882327+0000) 2019-10-13T14:19:11.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:11.809+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:11.811483+0000) 2019-10-13T14:19:12.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:12.619+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.620730+0000) 2019-10-13T14:19:12.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:12.743+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.746182+0000) 2019-10-13T14:19:12.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:12.747+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.747547+0000) 2019-10-13T14:19:12.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:12.745+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.748188+0000) 2019-10-13T14:19:12.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:12.747+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.748837+0000) 2019-10-13T14:19:12.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:12.747+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.749000+0000) 2019-10-13T14:19:12.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:12.747+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.749443+0000) 2019-10-13T14:19:12.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:12.747+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.750092+0000) 2019-10-13T14:19:12.756 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:12.749+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.753857+0000) 2019-10-13T14:19:12.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:12.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193095 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:15.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:15.643+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:15.645726+0000) 2019-10-13T14:19:15.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:15.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:15.933116+0000) 2019-10-13T14:19:16.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:16.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:16.611039+0000) 2019-10-13T14:19:16.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:16.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:16.860990+0000) 2019-10-13T14:19:17.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:17.759+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.760299+0000) 2019-10-13T14:19:17.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:17.759+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.760747+0000) 2019-10-13T14:19:17.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:17.759+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.761130+0000) 2019-10-13T14:19:17.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:17.759+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.761672+0000) 2019-10-13T14:19:17.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:17.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.762838+0000) 2019-10-13T14:19:17.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:17.760+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.763430+0000) 2019-10-13T14:19:17.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:17.763+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.763578+0000) 2019-10-13T14:19:17.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:17.764+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:17.768804+0000) 2019-10-13T14:19:17.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:17.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:18.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:18.879+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:18.882585+0000) 2019-10-13T14:19:18.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:18.879+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:18.882585+0000) 2019-10-13T14:19:21.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:21.816+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:21.817716+0000) 2019-10-13T14:19:22.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:22.618+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.620931+0000) 2019-10-13T14:19:22.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:22.770+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.772771+0000) 2019-10-13T14:19:22.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:22.770+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.773038+0000) 2019-10-13T14:19:22.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:22.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.773459+0000) 2019-10-13T14:19:22.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:22.770+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.773717+0000) 2019-10-13T14:19:22.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:22.770+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.774760+0000) 2019-10-13T14:19:22.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:22.770+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.775011+0000) 2019-10-13T14:19:22.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:22.774+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.775903+0000) 2019-10-13T14:19:22.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:22.778+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:22.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:22.780+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.782647+0000) 2019-10-13T14:19:23.715 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:19:23.715 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:23.719 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:19:23.719 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:25.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:25.642+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:25.645965+0000) 2019-10-13T14:19:25.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:25.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:25.933297+0000) 2019-10-13T14:19:26.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:26.606+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:26.611238+0000) 2019-10-13T14:19:26.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:26.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:26.861189+0000) 2019-10-13T14:19:27.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:27.782+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.785229+0000) 2019-10-13T14:19:27.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:27.782+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.786031+0000) 2019-10-13T14:19:27.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:27.782+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.786616+0000) 2019-10-13T14:19:27.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:27.784+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.787687+0000) 2019-10-13T14:19:27.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:27.786+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.787899+0000) 2019-10-13T14:19:27.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:27.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.788468+0000) 2019-10-13T14:19:27.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:27.786+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.788774+0000) 2019-10-13T14:19:27.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:27.794+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:27.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:27.792+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:27.797474+0000) 2019-10-13T14:19:28.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:28.878+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:28.882848+0000) 2019-10-13T14:19:28.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:28.878+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:28.882847+0000) 2019-10-13T14:19:31.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:31.820+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:31.824444+0000) 2019-10-13T14:19:32.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:32.618+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.621134+0000) 2019-10-13T14:19:32.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:32.798+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.799992+0000) 2019-10-13T14:19:32.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:32.798+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.800130+0000) 2019-10-13T14:19:32.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:32.798+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.800860+0000) 2019-10-13T14:19:32.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:32.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.801600+0000) 2019-10-13T14:19:32.805 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:32.800+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.802714+0000) 2019-10-13T14:19:32.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:32.798+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.802769+0000) 2019-10-13T14:19:32.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:32.802+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.803444+0000) 2019-10-13T14:19:32.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:32.806+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:32.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:32.808+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.810797+0000) 2019-10-13T14:19:35.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:35.642+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:35.646119+0000) 2019-10-13T14:19:35.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:35.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:35.933495+0000) 2019-10-13T14:19:36.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:36.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:36.611477+0000) 2019-10-13T14:19:36.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:36.856+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:36.861419+0000) 2019-10-13T14:19:37.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:37.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.813372+0000) 2019-10-13T14:19:37.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:37.808+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.813915+0000) 2019-10-13T14:19:37.816 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:37.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.814107+0000) 2019-10-13T14:19:37.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:37.810+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.814823+0000) 2019-10-13T14:19:37.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:37.814+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.815731+0000) 2019-10-13T14:19:37.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:37.814+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.816791+0000) 2019-10-13T14:19:37.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:37.814+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.817401+0000) 2019-10-13T14:19:37.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:37.816+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:37.821766+0000) 2019-10-13T14:19:37.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:37.822+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193267 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:38.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:38.878+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:38.883073+0000) 2019-10-13T14:19:38.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:38.878+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:38.883073+0000) 2019-10-13T14:19:41.833 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:41.828+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:41.830916+0000) 2019-10-13T14:19:42.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:42.618+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.621337+0000) 2019-10-13T14:19:42.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:42.826+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.828316+0000) 2019-10-13T14:19:42.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:42.826+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.828510+0000) 2019-10-13T14:19:42.832 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:42.826+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.829032+0000) 2019-10-13T14:19:42.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:42.826+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.829594+0000) 2019-10-13T14:19:42.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:42.826+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.829923+0000) 2019-10-13T14:19:42.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:42.826+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.830423+0000) 2019-10-13T14:19:42.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:42.828+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.831242+0000) 2019-10-13T14:19:42.841 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:42.834+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:42.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:42.836+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.840051+0000) 2019-10-13T14:19:45.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:45.642+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:45.646318+0000) 2019-10-13T14:19:45.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:45.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:45.933696+0000) 2019-10-13T14:19:46.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:46.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:46.611692+0000) 2019-10-13T14:19:46.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:46.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:46.861616+0000) 2019-10-13T14:19:47.845 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:47.840+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.842948+0000) 2019-10-13T14:19:47.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:47.842+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.843468+0000) 2019-10-13T14:19:47.846 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:47.842+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.843699+0000) 2019-10-13T14:19:47.846 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:47.842+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.844066+0000) 2019-10-13T14:19:47.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:47.842+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.845230+0000) 2019-10-13T14:19:47.848 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:47.842+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.845541+0000) 2019-10-13T14:19:47.848 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:47.842+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.846225+0000) 2019-10-13T14:19:47.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:47.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193334 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:47.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:47.848+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:47.852728+0000) 2019-10-13T14:19:48.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:48.882+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:48.883336+0000) 2019-10-13T14:19:48.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:48.882+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:48.883337+0000) 2019-10-13T14:19:51.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:51.832+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:51.836957+0000) 2019-10-13T14:19:52.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:52.618+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.621558+0000) 2019-10-13T14:19:52.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:52.852+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.855833+0000) 2019-10-13T14:19:52.859 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:52.854+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.856280+0000) 2019-10-13T14:19:52.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:52.854+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.856656+0000) 2019-10-13T14:19:52.860 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:52.854+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.857603+0000) 2019-10-13T14:19:52.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:52.854+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.857794+0000) 2019-10-13T14:19:52.861 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:52.854+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.858174+0000) 2019-10-13T14:19:52.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:52.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.858977+0000) 2019-10-13T14:19:52.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:52.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:52.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:52.864+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.866385+0000) 2019-10-13T14:19:53.755 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:19:53.755 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:53.759 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:19:53.759 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:55.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:55.642+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:55.646549+0000) 2019-10-13T14:19:55.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:55.930+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:55.933894+0000) 2019-10-13T14:19:56.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:56.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:56.611900+0000) 2019-10-13T14:19:56.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:56.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:56.861765+0000) 2019-10-13T14:19:57.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:19:57.866+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.868348+0000) 2019-10-13T14:19:57.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:19:57.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.868846+0000) 2019-10-13T14:19:57.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:19:57.864+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.869643+0000) 2019-10-13T14:19:57.872 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:19:57.866+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.869857+0000) 2019-10-13T14:19:57.873 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:57.866+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.871281+0000) 2019-10-13T14:19:57.874 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:57.870+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.871796+0000) 2019-10-13T14:19:57.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:19:57.870+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.871606+0000) 2019-10-13T14:19:57.879 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:19:57.872+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:57.877282+0000) 2019-10-13T14:19:57.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:19:57.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:19:58.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:19:58.882+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:58.883601+0000) 2019-10-13T14:19:58.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:19:58.882+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:58.883601+0000) 2019-10-13T14:20:01.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:01.840+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:01.843553+0000) 2019-10-13T14:20:02.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:02.618+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.621791+0000) 2019-10-13T14:20:02.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:02.880+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.882919+0000) 2019-10-13T14:20:02.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:02.878+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.882949+0000) 2019-10-13T14:20:02.886 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:02.882+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.883387+0000) 2019-10-13T14:20:02.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:02.882+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.884455+0000) 2019-10-13T14:20:02.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:02.882+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.885312+0000) 2019-10-13T14:20:02.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:02.882+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.885668+0000) 2019-10-13T14:20:02.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:02.882+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.886292+0000) 2019-10-13T14:20:02.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:02.888+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.891581+0000) 2019-10-13T14:20:02.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:02.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:05.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:05.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:05.646748+0000) 2019-10-13T14:20:05.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:05.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:05.934100+0000) 2019-10-13T14:20:06.614 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:06.609+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:06.612100+0000) 2019-10-13T14:20:06.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:06.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:06.861981+0000) 2019-10-13T14:20:07.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:07.892+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.896561+0000) 2019-10-13T14:20:07.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:07.893+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.897586+0000) 2019-10-13T14:20:07.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:07.893+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.898102+0000) 2019-10-13T14:20:07.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:07.893+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.898650+0000) 2019-10-13T14:20:07.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:07.893+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.898712+0000) 2019-10-13T14:20:07.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:07.897+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.899346+0000) 2019-10-13T14:20:07.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:07.897+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.899803+0000) 2019-10-13T14:20:07.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:07.900+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:07.904050+0000) 2019-10-13T14:20:07.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:07.901+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:08.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:08.881+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:08.883842+0000) 2019-10-13T14:20:08.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:08.881+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:08.883842+0000) 2019-10-13T14:20:11.852 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:11.847+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:11.850236+0000) 2019-10-13T14:20:12.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:12.621+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.621990+0000) 2019-10-13T14:20:12.912 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:12.905+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.909727+0000) 2019-10-13T14:20:12.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:12.905+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.910009+0000) 2019-10-13T14:20:12.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:12.905+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.910511+0000) 2019-10-13T14:20:12.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:12.905+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.910653+0000) 2019-10-13T14:20:12.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:12.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.911243+0000) 2019-10-13T14:20:12.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:12.909+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.911507+0000) 2019-10-13T14:20:12.914 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:12.909+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.911597+0000) 2019-10-13T14:20:12.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:12.915+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.918408+0000) 2019-10-13T14:20:12.921 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:12.913+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193504 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:15.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:15.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:15.646943+0000) 2019-10-13T14:20:15.936 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:15.929+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:15.934296+0000) 2019-10-13T14:20:16.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:16.609+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:16.612315+0000) 2019-10-13T14:20:16.864 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:16.859+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:16.862182+0000) 2019-10-13T14:20:17.924 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:17.917+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.921617+0000) 2019-10-13T14:20:17.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:17.917+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.921774+0000) 2019-10-13T14:20:17.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:17.917+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.922322+0000) 2019-10-13T14:20:17.925 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:17.917+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.922769+0000) 2019-10-13T14:20:17.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:17.921+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.924699+0000) 2019-10-13T14:20:17.927 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:17.921+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.925048+0000) 2019-10-13T14:20:17.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:17.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.925471+0000) 2019-10-13T14:20:17.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:17.927+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:17.930443+0000) 2019-10-13T14:20:17.934 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:17.929+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193538 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:18.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:18.881+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:18.884012+0000) 2019-10-13T14:20:18.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:18.881+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:18.884012+0000) 2019-10-13T14:20:21.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:21.851+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:21.856864+0000) 2019-10-13T14:20:22.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:22.621+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.622190+0000) 2019-10-13T14:20:22.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:22.933+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.935341+0000) 2019-10-13T14:20:22.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:22.933+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.936135+0000) 2019-10-13T14:20:22.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:22.933+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.936723+0000) 2019-10-13T14:20:22.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:22.931+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.937529+0000) 2019-10-13T14:20:22.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:22.933+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.937214+0000) 2019-10-13T14:20:22.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:22.933+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.937897+0000) 2019-10-13T14:20:22.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:22.933+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.938676+0000) 2019-10-13T14:20:22.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:22.941+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:22.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:22.939+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.945576+0000) 2019-10-13T14:20:23.911 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:20:23.912 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:23.915 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:20:23.915 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:25.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:25.641+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:25.647176+0000) 2019-10-13T14:20:25.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:25.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:25.934490+0000) 2019-10-13T14:20:26.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:26.609+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:26.612549+0000) 2019-10-13T14:20:26.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:26.859+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:26.862405+0000) 2019-10-13T14:20:27.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:27.945+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.948271+0000) 2019-10-13T14:20:27.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:27.945+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.949110+0000) 2019-10-13T14:20:27.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:27.945+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.949812+0000) 2019-10-13T14:20:27.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:27.945+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.950215+0000) 2019-10-13T14:20:27.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:27.945+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.950553+0000) 2019-10-13T14:20:27.953 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:27.945+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.951234+0000) 2019-10-13T14:20:27.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:27.947+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.952619+0000) 2019-10-13T14:20:27.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:27.951+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:27.956054+0000) 2019-10-13T14:20:27.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:27.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:28.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:28.881+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:28.884177+0000) 2019-10-13T14:20:28.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:28.881+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:28.884177+0000) 2019-10-13T14:20:31.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:31.859+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:31.863219+0000) 2019-10-13T14:20:32.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:32.621+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.622417+0000) 2019-10-13T14:20:32.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:32.957+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.961462+0000) 2019-10-13T14:20:32.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:32.957+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.962242+0000) 2019-10-13T14:20:32.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:32.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.962673+0000) 2019-10-13T14:20:32.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:32.957+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.962890+0000) 2019-10-13T14:20:32.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:32.961+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.963828+0000) 2019-10-13T14:20:32.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:32.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.964114+0000) 2019-10-13T14:20:32.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:32.961+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.964808+0000) 2019-10-13T14:20:32.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:32.963+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.969927+0000) 2019-10-13T14:20:32.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:32.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:35.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:35.645+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:35.647406+0000) 2019-10-13T14:20:35.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:35.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:35.934689+0000) 2019-10-13T14:20:36.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:36.609+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:36.612740+0000) 2019-10-13T14:20:36.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:36.859+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:36.862579+0000) 2019-10-13T14:20:37.977 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:37.969+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.974299+0000) 2019-10-13T14:20:37.978 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:37.973+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.975575+0000) 2019-10-13T14:20:37.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:37.973+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.975861+0000) 2019-10-13T14:20:37.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:37.971+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.976970+0000) 2019-10-13T14:20:37.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:37.973+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.977204+0000) 2019-10-13T14:20:37.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:37.973+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.977882+0000) 2019-10-13T14:20:37.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:37.973+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.978311+0000) 2019-10-13T14:20:37.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:37.979+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:37.984298+0000) 2019-10-13T14:20:37.988 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:37.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:38.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:38.881+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:38.884371+0000) 2019-10-13T14:20:38.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:38.881+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:38.884371+0000) 2019-10-13T14:20:41.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:41.867+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:41.869662+0000) 2019-10-13T14:20:42.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:42.617+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.622619+0000) 2019-10-13T14:20:42.992 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:42.985+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.989358+0000) 2019-10-13T14:20:42.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:42.985+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.989832+0000) 2019-10-13T14:20:42.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:42.985+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.990338+0000) 2019-10-13T14:20:42.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:42.987+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.990963+0000) 2019-10-13T14:20:42.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:42.989+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.991444+0000) 2019-10-13T14:20:42.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:42.989+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.992007+0000) 2019-10-13T14:20:42.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:42.989+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.992418+0000) 2019-10-13T14:20:43.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:42.995+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.999233+0000) 2019-10-13T14:20:43.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:42.997+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193710 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:45.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:45.645+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.647612+0000) 2019-10-13T14:20:45.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:45.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.934896+0000) 2019-10-13T14:20:46.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:46.609+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:46.612917+0000) 2019-10-13T14:20:46.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:46.859+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:46.862773+0000) 2019-10-13T14:20:48.006 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:48.001+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.003575+0000) 2019-10-13T14:20:48.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:47.999+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.004158+0000) 2019-10-13T14:20:48.007 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:48.001+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.004415+0000) 2019-10-13T14:20:48.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:48.001+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.004833+0000) 2019-10-13T14:20:48.008 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:48.001+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.005799+0000) 2019-10-13T14:20:48.009 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:48.001+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.006838+0000) 2019-10-13T14:20:48.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:48.001+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.007070+0000) 2019-10-13T14:20:48.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:48.007+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.010901+0000) 2019-10-13T14:20:48.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:48.009+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:48.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:48.881+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.884570+0000) 2019-10-13T14:20:48.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:48.881+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.884570+0000) 2019-10-13T14:20:51.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:51.875+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:51.876145+0000) 2019-10-13T14:20:52.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:52.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:52.622840+0000) 2019-10-13T14:20:53.018 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:53.012+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.015656+0000) 2019-10-13T14:20:53.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:53.012+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.016601+0000) 2019-10-13T14:20:53.019 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:53.012+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.017113+0000) 2019-10-13T14:20:53.021 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:53.012+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.018380+0000) 2019-10-13T14:20:53.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:53.015+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.019015+0000) 2019-10-13T14:20:53.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:53.016+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.019444+0000) 2019-10-13T14:20:53.022 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:53.016+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.019731+0000) 2019-10-13T14:20:53.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:53.020+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:53.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:53.019+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:53.026118+0000) 2019-10-13T14:20:53.960 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:20:53.960 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:53.964 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:20:53.964 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:55.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:55.644+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.647833+0000) 2019-10-13T14:20:55.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:55.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.935149+0000) 2019-10-13T14:20:56.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:56.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:56.613082+0000) 2019-10-13T14:20:56.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:56.859+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:56.862969+0000) 2019-10-13T14:20:58.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:20:58.024+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.028787+0000) 2019-10-13T14:20:58.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:20:58.023+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.029894+0000) 2019-10-13T14:20:58.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:20:58.024+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.031061+0000) 2019-10-13T14:20:58.034 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:58.024+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.031260+0000) 2019-10-13T14:20:58.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:20:58.028+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.031633+0000) 2019-10-13T14:20:58.035 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:58.028+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.032442+0000) 2019-10-13T14:20:58.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:20:58.028+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.033023+0000) 2019-10-13T14:20:58.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:20:58.031+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.036948+0000) 2019-10-13T14:20:58.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:20:58.032+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:20:58.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:20:58.880+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.884800+0000) 2019-10-13T14:20:58.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:20:58.880+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.884800+0000) 2019-10-13T14:21:01.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:01.879+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:01.882565+0000) 2019-10-13T14:21:02.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:02.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:02.623045+0000) 2019-10-13T14:21:03.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:03.036+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.041365+0000) 2019-10-13T14:21:03.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:03.036+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.041824+0000) 2019-10-13T14:21:03.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:03.036+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.042262+0000) 2019-10-13T14:21:03.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:03.036+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.043041+0000) 2019-10-13T14:21:03.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:03.040+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.043775+0000) 2019-10-13T14:21:03.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:03.039+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.044246+0000) 2019-10-13T14:21:03.047 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:03.040+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.045140+0000) 2019-10-13T14:21:03.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:03.047+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:03.051008+0000) 2019-10-13T14:21:03.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:03.048+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193848 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:05.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:05.644+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.648032+0000) 2019-10-13T14:21:05.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:05.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.935342+0000) 2019-10-13T14:21:06.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:06.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:06.613306+0000) 2019-10-13T14:21:06.865 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:06.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:06.863199+0000) 2019-10-13T14:21:08.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:08.052+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.056239+0000) 2019-10-13T14:21:08.059 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:08.052+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.056756+0000) 2019-10-13T14:21:08.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:08.052+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.056937+0000) 2019-10-13T14:21:08.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:08.052+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.057931+0000) 2019-10-13T14:21:08.060 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:08.052+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.058193+0000) 2019-10-13T14:21:08.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:08.052+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.058442+0000) 2019-10-13T14:21:08.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:08.054+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.058975+0000) 2019-10-13T14:21:08.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:08.062+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.066220+0000) 2019-10-13T14:21:08.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:08.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:08.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:08.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.884996+0000) 2019-10-13T14:21:08.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:08.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.884996+0000) 2019-10-13T14:21:11.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:11.886+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:11.889185+0000) 2019-10-13T14:21:12.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:12.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:12.623274+0000) 2019-10-13T14:21:13.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:13.064+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.069531+0000) 2019-10-13T14:21:13.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:13.064+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.070123+0000) 2019-10-13T14:21:13.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:13.066+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.071514+0000) 2019-10-13T14:21:13.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:13.068+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.072173+0000) 2019-10-13T14:21:13.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:13.068+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.072579+0000) 2019-10-13T14:21:13.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:13.068+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.073124+0000) 2019-10-13T14:21:13.076 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:13.068+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.073443+0000) 2019-10-13T14:21:13.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:13.070+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:13.076982+0000) 2019-10-13T14:21:13.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:13.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:15.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:15.644+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.648261+0000) 2019-10-13T14:21:15.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:15.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.935573+0000) 2019-10-13T14:21:16.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:16.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:16.613455+0000) 2019-10-13T14:21:16.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:16.858+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:16.863416+0000) 2019-10-13T14:21:18.086 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:18.080+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.083797+0000) 2019-10-13T14:21:18.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:18.080+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.084665+0000) 2019-10-13T14:21:18.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:18.080+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.085476+0000) 2019-10-13T14:21:18.088 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:18.080+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.085696+0000) 2019-10-13T14:21:18.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:18.082+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.086400+0000) 2019-10-13T14:21:18.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:18.080+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.086926+0000) 2019-10-13T14:21:18.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:18.080+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.087182+0000) 2019-10-13T14:21:18.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:18.088+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:18.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:18.090+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.095057+0000) 2019-10-13T14:21:18.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:18.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.885195+0000) 2019-10-13T14:21:18.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:18.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.885195+0000) 2019-10-13T14:21:21.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:21.894+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:21.895632+0000) 2019-10-13T14:21:22.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:22.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:22.623523+0000) 2019-10-13T14:21:23.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:23.092+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.095464+0000) 2019-10-13T14:21:23.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:23.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.096832+0000) 2019-10-13T14:21:23.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:23.092+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.096584+0000) 2019-10-13T14:21:23.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:23.092+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.097473+0000) 2019-10-13T14:21:23.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:23.090+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.098066+0000) 2019-10-13T14:21:23.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:23.092+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.098087+0000) 2019-10-13T14:21:23.101 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:23.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.098820+0000) 2019-10-13T14:21:23.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:23.098+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:23.102542+0000) 2019-10-13T14:21:23.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:23.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 193986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:24.084 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:21:24.084 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:24.088 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:21:24.088 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:25.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:25.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.648462+0000) 2019-10-13T14:21:25.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:25.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.935741+0000) 2019-10-13T14:21:26.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:26.608+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:26.613652+0000) 2019-10-13T14:21:26.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:26.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:26.863607+0000) 2019-10-13T14:21:28.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:28.104+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.108847+0000) 2019-10-13T14:21:28.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:28.104+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.109224+0000) 2019-10-13T14:21:28.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:28.102+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.109770+0000) 2019-10-13T14:21:28.112 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:28.104+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.109956+0000) 2019-10-13T14:21:28.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:28.104+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.110505+0000) 2019-10-13T14:21:28.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:28.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.111349+0000) 2019-10-13T14:21:28.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:28.108+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.111960+0000) 2019-10-13T14:21:28.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:28.112+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:28.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:28.114+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.120207+0000) 2019-10-13T14:21:28.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:28.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.885438+0000) 2019-10-13T14:21:28.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:28.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.885438+0000) 2019-10-13T14:21:31.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:31.902+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:31.904809+0000) 2019-10-13T14:21:32.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:32.619+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:32.623722+0000) 2019-10-13T14:21:33.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:33.115+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.121288+0000) 2019-10-13T14:21:33.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:33.115+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.121541+0000) 2019-10-13T14:21:33.124 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:33.115+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.121934+0000) 2019-10-13T14:21:33.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:33.118+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.122865+0000) 2019-10-13T14:21:33.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:33.119+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.123801+0000) 2019-10-13T14:21:33.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:33.119+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.123823+0000) 2019-10-13T14:21:33.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:33.119+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.125469+0000) 2019-10-13T14:21:33.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:33.126+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:33.131622+0000) 2019-10-13T14:21:33.135 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:33.127+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:35.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:35.643+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.648688+0000) 2019-10-13T14:21:35.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:35.931+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.935995+0000) 2019-10-13T14:21:36.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:36.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:36.613861+0000) 2019-10-13T14:21:36.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:36.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:36.863833+0000) 2019-10-13T14:21:38.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:38.131+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.136153+0000) 2019-10-13T14:21:38.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:38.131+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.135853+0000) 2019-10-13T14:21:38.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:38.131+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.136916+0000) 2019-10-13T14:21:38.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:38.131+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.137669+0000) 2019-10-13T14:21:38.140 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:38.131+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.138394+0000) 2019-10-13T14:21:38.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:38.134+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.139067+0000) 2019-10-13T14:21:38.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:38.131+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.139207+0000) 2019-10-13T14:21:38.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:38.138+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.143128+0000) 2019-10-13T14:21:38.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:38.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:38.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:38.883+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.885669+0000) 2019-10-13T14:21:38.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:38.883+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.885669+0000) 2019-10-13T14:21:41.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:41.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:41.911186+0000) 2019-10-13T14:21:42.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:42.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:42.623927+0000) 2019-10-13T14:21:43.152 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:43.143+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.149722+0000) 2019-10-13T14:21:43.153 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:43.143+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.150530+0000) 2019-10-13T14:21:43.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:43.143+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.151158+0000) 2019-10-13T14:21:43.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:43.147+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.151771+0000) 2019-10-13T14:21:43.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:43.146+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.152402+0000) 2019-10-13T14:21:43.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:43.147+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.152368+0000) 2019-10-13T14:21:43.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:43.147+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.152665+0000) 2019-10-13T14:21:43.160 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:43.154+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:43.158245+0000) 2019-10-13T14:21:43.161 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:43.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:45.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:45.647+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.648891+0000) 2019-10-13T14:21:45.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:45.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.936205+0000) 2019-10-13T14:21:46.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:46.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:46.613971+0000) 2019-10-13T14:21:46.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:46.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:46.864038+0000) 2019-10-13T14:21:48.165 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:48.155+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.162550+0000) 2019-10-13T14:21:48.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:48.159+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.163414+0000) 2019-10-13T14:21:48.166 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:48.159+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.163690+0000) 2019-10-13T14:21:48.166 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:48.159+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.163990+0000) 2019-10-13T14:21:48.167 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:48.159+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.164397+0000) 2019-10-13T14:21:48.167 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:48.158+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.165235+0000) 2019-10-13T14:21:48.168 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:48.159+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.166256+0000) 2019-10-13T14:21:48.175 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:48.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194156 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:48.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:48.170+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.175207+0000) 2019-10-13T14:21:48.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:48.883+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.885915+0000) 2019-10-13T14:21:48.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:48.883+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.885915+0000) 2019-10-13T14:21:51.920 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:51.914+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:51.917547+0000) 2019-10-13T14:21:52.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:52.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:52.624156+0000) 2019-10-13T14:21:53.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:53.171+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.176817+0000) 2019-10-13T14:21:53.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:53.171+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.177102+0000) 2019-10-13T14:21:53.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:53.171+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.178023+0000) 2019-10-13T14:21:53.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:53.171+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.178532+0000) 2019-10-13T14:21:53.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:53.171+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.179094+0000) 2019-10-13T14:21:53.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:53.178+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.179958+0000) 2019-10-13T14:21:53.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:53.175+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.179989+0000) 2019-10-13T14:21:53.186 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:53.182+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:53.184003+0000) 2019-10-13T14:21:53.189 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:53.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:54.194 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:21:54.194 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:54.198 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:21:54.198 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:55.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:55.647+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.649108+0000) 2019-10-13T14:21:55.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:55.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.936447+0000) 2019-10-13T14:21:56.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:56.607+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:56.614165+0000) 2019-10-13T14:21:56.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:56.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:56.864231+0000) 2019-10-13T14:21:58.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:21:58.187+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.190232+0000) 2019-10-13T14:21:58.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:21:58.187+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.190751+0000) 2019-10-13T14:21:58.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:21:58.187+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.191649+0000) 2019-10-13T14:21:58.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:21:58.187+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.192699+0000) 2019-10-13T14:21:58.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:58.187+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.193299+0000) 2019-10-13T14:21:58.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:58.187+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.193575+0000) 2019-10-13T14:21:58.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:21:58.194+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.194112+0000) 2019-10-13T14:21:58.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:21:58.198+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.199602+0000) 2019-10-13T14:21:58.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:21:58.195+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:21:58.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:21:58.883+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.886062+0000) 2019-10-13T14:21:58.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:21:58.883+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.886062+0000) 2019-10-13T14:22:01.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:01.921+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:01.923900+0000) 2019-10-13T14:22:02.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:02.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:02.624357+0000) 2019-10-13T14:22:03.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:03.203+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.204861+0000) 2019-10-13T14:22:03.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:03.203+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.205231+0000) 2019-10-13T14:22:03.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:03.203+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.206014+0000) 2019-10-13T14:22:03.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:03.203+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.206104+0000) 2019-10-13T14:22:03.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:03.203+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.206429+0000) 2019-10-13T14:22:03.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:03.203+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.206817+0000) 2019-10-13T14:22:03.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:03.205+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.208081+0000) 2019-10-13T14:22:03.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:03.209+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:03.211125+0000) 2019-10-13T14:22:03.216 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:03.211+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:05.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:05.647+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.649322+0000) 2019-10-13T14:22:05.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:05.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.936675+0000) 2019-10-13T14:22:06.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:06.611+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:06.614360+0000) 2019-10-13T14:22:06.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:06.861+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:06.864433+0000) 2019-10-13T14:22:08.220 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:08.215+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.217217+0000) 2019-10-13T14:22:08.221 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:08.215+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.218394+0000) 2019-10-13T14:22:08.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:08.215+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.218703+0000) 2019-10-13T14:22:08.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:08.219+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.219352+0000) 2019-10-13T14:22:08.222 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:08.217+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.220004+0000) 2019-10-13T14:22:08.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:08.219+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.220144+0000) 2019-10-13T14:22:08.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:08.219+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.220600+0000) 2019-10-13T14:22:08.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:08.225+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.227183+0000) 2019-10-13T14:22:08.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:08.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194294 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:08.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:08.883+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.886285+0000) 2019-10-13T14:22:08.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:08.883+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.886285+0000) 2019-10-13T14:22:11.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:11.929+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:11.930278+0000) 2019-10-13T14:22:12.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:12.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:12.624576+0000) 2019-10-13T14:22:13.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:13.227+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.231170+0000) 2019-10-13T14:22:13.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:13.231+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.231697+0000) 2019-10-13T14:22:13.235 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:13.229+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.232193+0000) 2019-10-13T14:22:13.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:13.231+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.232281+0000) 2019-10-13T14:22:13.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:13.231+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.233212+0000) 2019-10-13T14:22:13.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:13.231+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.234185+0000) 2019-10-13T14:22:13.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:13.231+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.234597+0000) 2019-10-13T14:22:13.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:13.233+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:13.237321+0000) 2019-10-13T14:22:13.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:13.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194329 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:15.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:15.646+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.649539+0000) 2019-10-13T14:22:15.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:15.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.936895+0000) 2019-10-13T14:22:16.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:16.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:16.614567+0000) 2019-10-13T14:22:16.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:16.861+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:16.864626+0000) 2019-10-13T14:22:18.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:18.242+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.243743+0000) 2019-10-13T14:22:18.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:18.242+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.245135+0000) 2019-10-13T14:22:18.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:18.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.245682+0000) 2019-10-13T14:22:18.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:18.242+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.245913+0000) 2019-10-13T14:22:18.249 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:18.242+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.246706+0000) 2019-10-13T14:22:18.250 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:18.246+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.247878+0000) 2019-10-13T14:22:18.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:18.246+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.248449+0000) 2019-10-13T14:22:18.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:18.249+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.253757+0000) 2019-10-13T14:22:18.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:18.250+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194362 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:18.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:18.882+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.886496+0000) 2019-10-13T14:22:18.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:18.882+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.886496+0000) 2019-10-13T14:22:21.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:21.933+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:21.936586+0000) 2019-10-13T14:22:22.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:22.622+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:22.624799+0000) 2019-10-13T14:22:23.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:23.254+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.257883+0000) 2019-10-13T14:22:23.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:23.257+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.258713+0000) 2019-10-13T14:22:23.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:23.254+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.258968+0000) 2019-10-13T14:22:23.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:23.254+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.259246+0000) 2019-10-13T14:22:23.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:23.258+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.259810+0000) 2019-10-13T14:22:23.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:23.258+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.260680+0000) 2019-10-13T14:22:23.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:23.258+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.261027+0000) 2019-10-13T14:22:23.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:23.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:23.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:23.269+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:23.271110+0000) 2019-10-13T14:22:24.305 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:22:24.305 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:24.309 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:22:24.309 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:25.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:25.646+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.649737+0000) 2019-10-13T14:22:25.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:25.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.937051+0000) 2019-10-13T14:22:26.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:26.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:26.614763+0000) 2019-10-13T14:22:26.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:26.861+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:26.864814+0000) 2019-10-13T14:22:28.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:28.266+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.270568+0000) 2019-10-13T14:22:28.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:28.266+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.271047+0000) 2019-10-13T14:22:28.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:28.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.271941+0000) 2019-10-13T14:22:28.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:28.270+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.272111+0000) 2019-10-13T14:22:28.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:28.270+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.272477+0000) 2019-10-13T14:22:28.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:28.270+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.272819+0000) 2019-10-13T14:22:28.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:28.270+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.273164+0000) 2019-10-13T14:22:28.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:28.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:28.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:28.277+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.280287+0000) 2019-10-13T14:22:28.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:28.882+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.886697+0000) 2019-10-13T14:22:28.889 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:28.882+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.886697+0000) 2019-10-13T14:22:31.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:31.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:31.942938+0000) 2019-10-13T14:22:32.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:32.622+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:32.625011+0000) 2019-10-13T14:22:33.285 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:33.278+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.282781+0000) 2019-10-13T14:22:33.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:33.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.283193+0000) 2019-10-13T14:22:33.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:33.282+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.284019+0000) 2019-10-13T14:22:33.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:33.282+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.284418+0000) 2019-10-13T14:22:33.287 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:33.282+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.284657+0000) 2019-10-13T14:22:33.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:33.281+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.285368+0000) 2019-10-13T14:22:33.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:33.282+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.285776+0000) 2019-10-13T14:22:33.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:33.285+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:33.289329+0000) 2019-10-13T14:22:33.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:33.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:35.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:35.646+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.649940+0000) 2019-10-13T14:22:35.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:35.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.937263+0000) 2019-10-13T14:22:36.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:36.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:36.614960+0000) 2019-10-13T14:22:36.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:36.861+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:36.865015+0000) 2019-10-13T14:22:38.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:38.290+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.295262+0000) 2019-10-13T14:22:38.298 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:38.294+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.296438+0000) 2019-10-13T14:22:38.299 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:38.294+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.297016+0000) 2019-10-13T14:22:38.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:38.293+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.297570+0000) 2019-10-13T14:22:38.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:38.294+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.297660+0000) 2019-10-13T14:22:38.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:38.294+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.298148+0000) 2019-10-13T14:22:38.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:38.294+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.298200+0000) 2019-10-13T14:22:38.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:38.301+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.303714+0000) 2019-10-13T14:22:38.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:38.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194496 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:38.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:38.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.886892+0000) 2019-10-13T14:22:38.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:38.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.886892+0000) 2019-10-13T14:22:41.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:41.945+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:41.949260+0000) 2019-10-13T14:22:42.627 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:42.622+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:42.625229+0000) 2019-10-13T14:22:43.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:43.306+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.308988+0000) 2019-10-13T14:22:43.314 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:43.306+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.309566+0000) 2019-10-13T14:22:43.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:43.306+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.309951+0000) 2019-10-13T14:22:43.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:43.306+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.311050+0000) 2019-10-13T14:22:43.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:43.310+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.311661+0000) 2019-10-13T14:22:43.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:43.309+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.312342+0000) 2019-10-13T14:22:43.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:43.310+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.313071+0000) 2019-10-13T14:22:43.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:43.318+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:43.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:43.317+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:43.321053+0000) 2019-10-13T14:22:45.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:45.646+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.650137+0000) 2019-10-13T14:22:45.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:45.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.937501+0000) 2019-10-13T14:22:46.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:46.610+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:46.615151+0000) 2019-10-13T14:22:46.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:46.861+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:46.865224+0000) 2019-10-13T14:22:48.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:48.322+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.323368+0000) 2019-10-13T14:22:48.326 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:48.321+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.323983+0000) 2019-10-13T14:22:48.327 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:48.322+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.324466+0000) 2019-10-13T14:22:48.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:48.322+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.325137+0000) 2019-10-13T14:22:48.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:48.322+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.325757+0000) 2019-10-13T14:22:48.328 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:48.322+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.326342+0000) 2019-10-13T14:22:48.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:48.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.326868+0000) 2019-10-13T14:22:48.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:48.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:48.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:48.329+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.333810+0000) 2019-10-13T14:22:48.889 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:48.882+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.887088+0000) 2019-10-13T14:22:48.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:48.882+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.887087+0000) 2019-10-13T14:22:51.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:51.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:51.955336+0000) 2019-10-13T14:22:52.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:52.622+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:52.625425+0000) 2019-10-13T14:22:53.339 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:53.334+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.336753+0000) 2019-10-13T14:22:53.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:53.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.337598+0000) 2019-10-13T14:22:53.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:53.334+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.337899+0000) 2019-10-13T14:22:53.341 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:53.334+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.338459+0000) 2019-10-13T14:22:53.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:53.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.339565+0000) 2019-10-13T14:22:53.342 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:53.338+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.340497+0000) 2019-10-13T14:22:53.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:53.338+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.341318+0000) 2019-10-13T14:22:53.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:53.345+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:53.346806+0000) 2019-10-13T14:22:53.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:53.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:54.437 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:22:54.438 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:54.441 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:22:54.441 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:55.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:55.650+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.650337+0000) 2019-10-13T14:22:55.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:55.934+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.937748+0000) 2019-10-13T14:22:56.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:56.614+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:56.615372+0000) 2019-10-13T14:22:56.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:56.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:56.865411+0000) 2019-10-13T14:22:58.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:22:58.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.350544+0000) 2019-10-13T14:22:58.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:22:58.346+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.351012+0000) 2019-10-13T14:22:58.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:22:58.350+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.351343+0000) 2019-10-13T14:22:58.354 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:58.350+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.351680+0000) 2019-10-13T14:22:58.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:22:58.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.352556+0000) 2019-10-13T14:22:58.355 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:22:58.348+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.353164+0000) 2019-10-13T14:22:58.356 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:58.350+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.353685+0000) 2019-10-13T14:22:58.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:22:58.356+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.359888+0000) 2019-10-13T14:22:58.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:22:58.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:22:58.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:22:58.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.887281+0000) 2019-10-13T14:22:58.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:22:58.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.887281+0000) 2019-10-13T14:23:01.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:01.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:01.961744+0000) 2019-10-13T14:23:02.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:02.621+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:02.625644+0000) 2019-10-13T14:23:03.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:03.357+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.362879+0000) 2019-10-13T14:23:03.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:03.361+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.364526+0000) 2019-10-13T14:23:03.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:03.361+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.365072+0000) 2019-10-13T14:23:03.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:03.361+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.365425+0000) 2019-10-13T14:23:03.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:03.361+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.366000+0000) 2019-10-13T14:23:03.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:03.364+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.366363+0000) 2019-10-13T14:23:03.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:03.361+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.366661+0000) 2019-10-13T14:23:03.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:03.368+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:03.370482+0000) 2019-10-13T14:23:03.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:03.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:05.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:05.645+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.650536+0000) 2019-10-13T14:23:05.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:05.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.937952+0000) 2019-10-13T14:23:06.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:06.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:06.615594+0000) 2019-10-13T14:23:06.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:06.864+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:06.865613+0000) 2019-10-13T14:23:08.380 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:08.373+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.377460+0000) 2019-10-13T14:23:08.381 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:08.373+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.378450+0000) 2019-10-13T14:23:08.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:08.373+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.379034+0000) 2019-10-13T14:23:08.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:08.377+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.379484+0000) 2019-10-13T14:23:08.382 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:08.377+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.380185+0000) 2019-10-13T14:23:08.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:08.376+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.381502+0000) 2019-10-13T14:23:08.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:08.377+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.381743+0000) 2019-10-13T14:23:08.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:08.384+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.386694+0000) 2019-10-13T14:23:08.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:08.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194696 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:08.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:08.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.887539+0000) 2019-10-13T14:23:08.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:08.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.887539+0000) 2019-10-13T14:23:11.971 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:11.964+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:11.968417+0000) 2019-10-13T14:23:12.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:12.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:12.625875+0000) 2019-10-13T14:23:13.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:13.389+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.391369+0000) 2019-10-13T14:23:13.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:13.389+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.392092+0000) 2019-10-13T14:23:13.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:13.389+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.392537+0000) 2019-10-13T14:23:13.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:13.389+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.392866+0000) 2019-10-13T14:23:13.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:13.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.393553+0000) 2019-10-13T14:23:13.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:13.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.394002+0000) 2019-10-13T14:23:13.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:13.392+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.395484+0000) 2019-10-13T14:23:13.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:13.396+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:13.401103+0000) 2019-10-13T14:23:13.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:13.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194729 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:15.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:15.649+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.650737+0000) 2019-10-13T14:23:15.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:15.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.938154+0000) 2019-10-13T14:23:16.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:16.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:16.615785+0000) 2019-10-13T14:23:16.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:16.860+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:16.865762+0000) 2019-10-13T14:23:18.408 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:18.401+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.405640+0000) 2019-10-13T14:23:18.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:18.401+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.406085+0000) 2019-10-13T14:23:18.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:18.404+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.407105+0000) 2019-10-13T14:23:18.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:18.405+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.407568+0000) 2019-10-13T14:23:18.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:18.405+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.408034+0000) 2019-10-13T14:23:18.410 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:18.405+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.408216+0000) 2019-10-13T14:23:18.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:18.405+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.409318+0000) 2019-10-13T14:23:18.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:18.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.415177+0000) 2019-10-13T14:23:18.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:18.409+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194764 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:18.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:18.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.887730+0000) 2019-10-13T14:23:18.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:18.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.887730+0000) 2019-10-13T14:23:21.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:21.972+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:21.974871+0000) 2019-10-13T14:23:22.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:22.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:22.626102+0000) 2019-10-13T14:23:23.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:23.413+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.417984+0000) 2019-10-13T14:23:23.421 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:23.413+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.418655+0000) 2019-10-13T14:23:23.421 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:23.413+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.419197+0000) 2019-10-13T14:23:23.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:23.416+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.420307+0000) 2019-10-13T14:23:23.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:23.417+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.420928+0000) 2019-10-13T14:23:23.424 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:23.417+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.421423+0000) 2019-10-13T14:23:23.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:23.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.421928+0000) 2019-10-13T14:23:23.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:23.424+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:23.428248+0000) 2019-10-13T14:23:23.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:23.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:24.576 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:23:24.576 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:24.580 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:23:24.580 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:25.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:25.649+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.650932+0000) 2019-10-13T14:23:25.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:25.933+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.938355+0000) 2019-10-13T14:23:26.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:26.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:26.615973+0000) 2019-10-13T14:23:26.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:26.864+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:26.865914+0000) 2019-10-13T14:23:28.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:28.429+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.431899+0000) 2019-10-13T14:23:28.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:28.429+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.432194+0000) 2019-10-13T14:23:28.436 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:28.429+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.433524+0000) 2019-10-13T14:23:28.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:28.429+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.434450+0000) 2019-10-13T14:23:28.437 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:28.429+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.434565+0000) 2019-10-13T14:23:28.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:28.432+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.435271+0000) 2019-10-13T14:23:28.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:28.433+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.435686+0000) 2019-10-13T14:23:28.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:28.436+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.439440+0000) 2019-10-13T14:23:28.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:28.437+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:28.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:28.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.887928+0000) 2019-10-13T14:23:28.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:28.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.887927+0000) 2019-10-13T14:23:31.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:31.980+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:31.981092+0000) 2019-10-13T14:23:32.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:32.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:32.626307+0000) 2019-10-13T14:23:33.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:33.441+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.445535+0000) 2019-10-13T14:23:33.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:33.441+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.446365+0000) 2019-10-13T14:23:33.449 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:33.441+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.446744+0000) 2019-10-13T14:23:33.449 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:33.441+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.447220+0000) 2019-10-13T14:23:33.450 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:33.444+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.447729+0000) 2019-10-13T14:23:33.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:33.445+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.448264+0000) 2019-10-13T14:23:33.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:33.445+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.448724+0000) 2019-10-13T14:23:33.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:33.452+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:33.454426+0000) 2019-10-13T14:23:33.458 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:33.453+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:35.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:35.645+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.651144+0000) 2019-10-13T14:23:35.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:35.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.938554+0000) 2019-10-13T14:23:36.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:36.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:36.616166+0000) 2019-10-13T14:23:36.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:36.864+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:36.866123+0000) 2019-10-13T14:23:38.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:38.453+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.459178+0000) 2019-10-13T14:23:38.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:38.457+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.460276+0000) 2019-10-13T14:23:38.463 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:38.457+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.460875+0000) 2019-10-13T14:23:38.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:38.457+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.460869+0000) 2019-10-13T14:23:38.464 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:38.457+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.461316+0000) 2019-10-13T14:23:38.464 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:38.457+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.461687+0000) 2019-10-13T14:23:38.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:38.460+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.462611+0000) 2019-10-13T14:23:38.468 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:38.464+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.466421+0000) 2019-10-13T14:23:38.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:38.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:38.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:38.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.888124+0000) 2019-10-13T14:23:38.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:38.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.888124+0000) 2019-10-13T14:23:41.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:41.984+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:41.987431+0000) 2019-10-13T14:23:42.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:42.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:42.626506+0000) 2019-10-13T14:23:43.473 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:43.464+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.470825+0000) 2019-10-13T14:23:43.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:43.468+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.472328+0000) 2019-10-13T14:23:43.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:43.468+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.472547+0000) 2019-10-13T14:23:43.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:43.468+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.472696+0000) 2019-10-13T14:23:43.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:43.468+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.473337+0000) 2019-10-13T14:23:43.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:43.468+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.473072+0000) 2019-10-13T14:23:43.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:43.468+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.473637+0000) 2019-10-13T14:23:43.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:43.476+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:43.480475+0000) 2019-10-13T14:23:43.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:43.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:45.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:45.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.651383+0000) 2019-10-13T14:23:45.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:45.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.938751+0000) 2019-10-13T14:23:46.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:46.612+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:46.616397+0000) 2019-10-13T14:23:46.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:46.864+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:46.866400+0000) 2019-10-13T14:23:48.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:48.479+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.484875+0000) 2019-10-13T14:23:48.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:48.480+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.485240+0000) 2019-10-13T14:23:48.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:48.480+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.485645+0000) 2019-10-13T14:23:48.489 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:48.480+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.486033+0000) 2019-10-13T14:23:48.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:48.480+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.487096+0000) 2019-10-13T14:23:48.490 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:48.484+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.487525+0000) 2019-10-13T14:23:48.490 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:48.484+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.488414+0000) 2019-10-13T14:23:48.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:48.491+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.494866+0000) 2019-10-13T14:23:48.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:48.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:48.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:48.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.888295+0000) 2019-10-13T14:23:48.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:48.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.888295+0000) 2019-10-13T14:23:51.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:51.987+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:51.993876+0000) 2019-10-13T14:23:52.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:52.624+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:52.626706+0000) 2019-10-13T14:23:53.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:53.492+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.498104+0000) 2019-10-13T14:23:53.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:53.492+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.498545+0000) 2019-10-13T14:23:53.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:53.492+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.498941+0000) 2019-10-13T14:23:53.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:53.495+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.499626+0000) 2019-10-13T14:23:53.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:53.496+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.499934+0000) 2019-10-13T14:23:53.503 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:53.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.500688+0000) 2019-10-13T14:23:53.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:53.496+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.501647+0000) 2019-10-13T14:23:53.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:53.503+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:53.508688+0000) 2019-10-13T14:23:53.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:53.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 194997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:54.706 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:23:54.706 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:54.709 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:23:54.709 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:55.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:55.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.651625+0000) 2019-10-13T14:23:55.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:55.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.938951+0000) 2019-10-13T14:23:56.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:56.612+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:56.616592+0000) 2019-10-13T14:23:56.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:56.863+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:56.866557+0000) 2019-10-13T14:23:58.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:58.504+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.511134+0000) 2019-10-13T14:23:58.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:23:58.508+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.511941+0000) 2019-10-13T14:23:58.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:23:58.508+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.512838+0000) 2019-10-13T14:23:58.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:23:58.508+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.513101+0000) 2019-10-13T14:23:58.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:58.508+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.513465+0000) 2019-10-13T14:23:58.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:23:58.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.513988+0000) 2019-10-13T14:23:58.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:23:58.511+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.514373+0000) 2019-10-13T14:23:58.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:23:58.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:23:58.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:23:58.515+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.521662+0000) 2019-10-13T14:23:58.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:23:58.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.888490+0000) 2019-10-13T14:23:58.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:23:58.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.888490+0000) 2019-10-13T14:24:02.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:01.999+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:02.000294+0000) 2019-10-13T14:24:02.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:02.624+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:02.626903+0000) 2019-10-13T14:24:03.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:03.520+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.524508+0000) 2019-10-13T14:24:03.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:03.520+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.524860+0000) 2019-10-13T14:24:03.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:03.519+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.525655+0000) 2019-10-13T14:24:03.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:03.520+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.525169+0000) 2019-10-13T14:24:03.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:03.520+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.526109+0000) 2019-10-13T14:24:03.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:03.524+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.527533+0000) 2019-10-13T14:24:03.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:03.524+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.528043+0000) 2019-10-13T14:24:03.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:03.531+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:03.534994+0000) 2019-10-13T14:24:03.538 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:03.528+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:05.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:05.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.651778+0000) 2019-10-13T14:24:05.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:05.932+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.939155+0000) 2019-10-13T14:24:06.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:06.612+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:06.616815+0000) 2019-10-13T14:24:06.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:06.863+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:06.866745+0000) 2019-10-13T14:24:08.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:08.532+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.538760+0000) 2019-10-13T14:24:08.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:08.532+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.539183+0000) 2019-10-13T14:24:08.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:08.535+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.540848+0000) 2019-10-13T14:24:08.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:08.536+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.541242+0000) 2019-10-13T14:24:08.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:08.536+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.541312+0000) 2019-10-13T14:24:08.544 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:08.536+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.541794+0000) 2019-10-13T14:24:08.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:08.536+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.542201+0000) 2019-10-13T14:24:08.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:08.543+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.548762+0000) 2019-10-13T14:24:08.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:08.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:08.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:08.884+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.888706+0000) 2019-10-13T14:24:08.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:08.884+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.888706+0000) 2019-10-13T14:24:12.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:12.003+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:12.006838+0000) 2019-10-13T14:24:12.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:12.620+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:12.627134+0000) 2019-10-13T14:24:13.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:13.547+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.552984+0000) 2019-10-13T14:24:13.556 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:13.548+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.553510+0000) 2019-10-13T14:24:13.556 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:13.548+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.553701+0000) 2019-10-13T14:24:13.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:13.548+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.554141+0000) 2019-10-13T14:24:13.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:13.548+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.555102+0000) 2019-10-13T14:24:13.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:13.548+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.554744+0000) 2019-10-13T14:24:13.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:13.552+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.555402+0000) 2019-10-13T14:24:13.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:13.559+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:13.562626+0000) 2019-10-13T14:24:13.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:13.556+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195132 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:15.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:15.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.651979+0000) 2019-10-13T14:24:15.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:15.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.939354+0000) 2019-10-13T14:24:16.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:16.612+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:16.616998+0000) 2019-10-13T14:24:16.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:16.863+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:16.866934+0000) 2019-10-13T14:24:18.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:18.560+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.566075+0000) 2019-10-13T14:24:18.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:18.560+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.566899+0000) 2019-10-13T14:24:18.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:18.564+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.568062+0000) 2019-10-13T14:24:18.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:18.563+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.568074+0000) 2019-10-13T14:24:18.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:18.564+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.568592+0000) 2019-10-13T14:24:18.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:18.564+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.568379+0000) 2019-10-13T14:24:18.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:18.564+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.569536+0000) 2019-10-13T14:24:18.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:18.571+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.576212+0000) 2019-10-13T14:24:18.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:18.572+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:18.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:18.888+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.888912+0000) 2019-10-13T14:24:18.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:18.888+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.888912+0000) 2019-10-13T14:24:22.016 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:22.011+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:22.013242+0000) 2019-10-13T14:24:22.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:22.624+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:22.627349+0000) 2019-10-13T14:24:23.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:23.575+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.581603+0000) 2019-10-13T14:24:23.585 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:23.576+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.582250+0000) 2019-10-13T14:24:23.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:23.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.582653+0000) 2019-10-13T14:24:23.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:23.576+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.582806+0000) 2019-10-13T14:24:23.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:23.580+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.583408+0000) 2019-10-13T14:24:23.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:23.580+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.583628+0000) 2019-10-13T14:24:23.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:23.580+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.584439+0000) 2019-10-13T14:24:23.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:23.587+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:23.590534+0000) 2019-10-13T14:24:23.593 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:23.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:24.842 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:24:24.842 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:24.845 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:24:24.845 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:25.654 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:25.647+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.652200+0000) 2019-10-13T14:24:25.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:25.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.939545+0000) 2019-10-13T14:24:26.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:26.615+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:26.617195+0000) 2019-10-13T14:24:26.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:26.863+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:26.867171+0000) 2019-10-13T14:24:28.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:28.587+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.594542+0000) 2019-10-13T14:24:28.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:28.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.594434+0000) 2019-10-13T14:24:28.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:28.587+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.595087+0000) 2019-10-13T14:24:28.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:28.591+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.595498+0000) 2019-10-13T14:24:28.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:28.591+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.595871+0000) 2019-10-13T14:24:28.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:28.591+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.596813+0000) 2019-10-13T14:24:28.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:28.591+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.597498+0000) 2019-10-13T14:24:28.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:28.599+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.603861+0000) 2019-10-13T14:24:28.607 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:28.599+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195232 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:28.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:28.887+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.889109+0000) 2019-10-13T14:24:28.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:28.887+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.889108+0000) 2019-10-13T14:24:32.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:32.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:32.019737+0000) 2019-10-13T14:24:32.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:32.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:32.627639+0000) 2019-10-13T14:24:33.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:33.603+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.608411+0000) 2019-10-13T14:24:33.611 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:33.603+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.609256+0000) 2019-10-13T14:24:33.612 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:33.603+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.609617+0000) 2019-10-13T14:24:33.612 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:33.603+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.609941+0000) 2019-10-13T14:24:33.613 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:33.603+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.610260+0000) 2019-10-13T14:24:33.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:33.603+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.610437+0000) 2019-10-13T14:24:33.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:33.603+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.610794+0000) 2019-10-13T14:24:33.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:33.611+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:33.617498+0000) 2019-10-13T14:24:33.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:33.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195266 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:35.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:35.651+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.652425+0000) 2019-10-13T14:24:35.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:35.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.939759+0000) 2019-10-13T14:24:36.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:36.611+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:36.617390+0000) 2019-10-13T14:24:36.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:36.863+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:36.867365+0000) 2019-10-13T14:24:38.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:38.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.621977+0000) 2019-10-13T14:24:38.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:38.615+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.622574+0000) 2019-10-13T14:24:38.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:38.615+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.622941+0000) 2019-10-13T14:24:38.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:38.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.623824+0000) 2019-10-13T14:24:38.627 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:38.619+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.624643+0000) 2019-10-13T14:24:38.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:38.619+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.625115+0000) 2019-10-13T14:24:38.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:38.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.625943+0000) 2019-10-13T14:24:38.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:38.627+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.631080+0000) 2019-10-13T14:24:38.634 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:38.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195299 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:38.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:38.887+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.889306+0000) 2019-10-13T14:24:38.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:38.887+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.889306+0000) 2019-10-13T14:24:42.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:42.023+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:42.026085+0000) 2019-10-13T14:24:42.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:42.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:42.627856+0000) 2019-10-13T14:24:43.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:43.631+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.635195+0000) 2019-10-13T14:24:43.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:43.631+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.636439+0000) 2019-10-13T14:24:43.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:43.631+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.636853+0000) 2019-10-13T14:24:43.640 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:43.631+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.637453+0000) 2019-10-13T14:24:43.640 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:43.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.638055+0000) 2019-10-13T14:24:43.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:43.631+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.638228+0000) 2019-10-13T14:24:43.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:43.631+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.638675+0000) 2019-10-13T14:24:43.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:43.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195332 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:43.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:43.642+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:43.647512+0000) 2019-10-13T14:24:45.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:45.651+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.652632+0000) 2019-10-13T14:24:45.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:45.935+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.939983+0000) 2019-10-13T14:24:46.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:46.611+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:46.617580+0000) 2019-10-13T14:24:46.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:46.862+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:46.867550+0000) 2019-10-13T14:24:48.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:48.643+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.648163+0000) 2019-10-13T14:24:48.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:48.643+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.648852+0000) 2019-10-13T14:24:48.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:48.643+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.649288+0000) 2019-10-13T14:24:48.652 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:48.643+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.650237+0000) 2019-10-13T14:24:48.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:48.643+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.650984+0000) 2019-10-13T14:24:48.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:48.646+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.651574+0000) 2019-10-13T14:24:48.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:48.647+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.651885+0000) 2019-10-13T14:24:48.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:48.651+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:48.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:48.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.658969+0000) 2019-10-13T14:24:48.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:48.887+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.889503+0000) 2019-10-13T14:24:48.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:48.887+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.889503+0000) 2019-10-13T14:24:52.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:52.030+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:52.032579+0000) 2019-10-13T14:24:52.630 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:52.627+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:52.628058+0000) 2019-10-13T14:24:53.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:53.655+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.661970+0000) 2019-10-13T14:24:53.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:53.655+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.662619+0000) 2019-10-13T14:24:53.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:53.658+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.663623+0000) 2019-10-13T14:24:53.666 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:53.659+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.663795+0000) 2019-10-13T14:24:53.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:53.659+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.664142+0000) 2019-10-13T14:24:53.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:53.659+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.664568+0000) 2019-10-13T14:24:53.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:53.659+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.664932+0000) 2019-10-13T14:24:53.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:53.666+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:53.671112+0000) 2019-10-13T14:24:53.674 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:53.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195400 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:54.896 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:24:54.896 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:54.900 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:24:54.900 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:55.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:55.651+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.652836+0000) 2019-10-13T14:24:55.942 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:55.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.940183+0000) 2019-10-13T14:24:56.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:56.615+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:56.617786+0000) 2019-10-13T14:24:56.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:56.866+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:56.867753+0000) 2019-10-13T14:24:58.678 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:24:58.675+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.676254+0000) 2019-10-13T14:24:58.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:58.675+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.676776+0000) 2019-10-13T14:24:58.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:24:58.675+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.677433+0000) 2019-10-13T14:24:58.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:24:58.670+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.677867+0000) 2019-10-13T14:24:58.680 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:24:58.675+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.678000+0000) 2019-10-13T14:24:58.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:24:58.675+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.678431+0000) 2019-10-13T14:24:58.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:58.675+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.678230+0000) 2019-10-13T14:24:58.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:24:58.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.683746+0000) 2019-10-13T14:24:58.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:24:58.683+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195433 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:24:58.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:24:58.887+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.889697+0000) 2019-10-13T14:24:58.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:24:58.887+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.889697+0000) 2019-10-13T14:25:02.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:02.042+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:02.043871+0000) 2019-10-13T14:25:02.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:02.627+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:02.628260+0000) 2019-10-13T14:25:03.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:03.687+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.689562+0000) 2019-10-13T14:25:03.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:03.686+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.690052+0000) 2019-10-13T14:25:03.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:03.687+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.690257+0000) 2019-10-13T14:25:03.693 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:03.687+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.691020+0000) 2019-10-13T14:25:03.694 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:03.691+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.691762+0000) 2019-10-13T14:25:03.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:03.691+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.691744+0000) 2019-10-13T14:25:03.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:03.691+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.692821+0000) 2019-10-13T14:25:03.703 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:03.699+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:03.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:03.698+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:03.703070+0000) 2019-10-13T14:25:05.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:05.651+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.653053+0000) 2019-10-13T14:25:05.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:05.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.940392+0000) 2019-10-13T14:25:06.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:06.615+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:06.617981+0000) 2019-10-13T14:25:06.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:06.866+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:06.867951+0000) 2019-10-13T14:25:08.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:08.698+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.704080+0000) 2019-10-13T14:25:08.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:08.702+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.704294+0000) 2019-10-13T14:25:08.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:08.702+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.704601+0000) 2019-10-13T14:25:08.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:08.702+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.705419+0000) 2019-10-13T14:25:08.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:08.702+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.706095+0000) 2019-10-13T14:25:08.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:08.702+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.706561+0000) 2019-10-13T14:25:08.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:08.702+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.707110+0000) 2019-10-13T14:25:08.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:08.706+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.712161+0000) 2019-10-13T14:25:08.716 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:08.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:08.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:08.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.889894+0000) 2019-10-13T14:25:08.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:08.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.889894+0000) 2019-10-13T14:25:12.053 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:12.046+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:12.050480+0000) 2019-10-13T14:25:12.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:12.626+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:12.628464+0000) 2019-10-13T14:25:13.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:13.714+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.717928+0000) 2019-10-13T14:25:13.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:13.714+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.718471+0000) 2019-10-13T14:25:13.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:13.714+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.719132+0000) 2019-10-13T14:25:13.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:13.718+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.719869+0000) 2019-10-13T14:25:13.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:13.718+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.720184+0000) 2019-10-13T14:25:13.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:13.714+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.721326+0000) 2019-10-13T14:25:13.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:13.718+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.721486+0000) 2019-10-13T14:25:13.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:13.722+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:13.726401+0000) 2019-10-13T14:25:13.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:13.722+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195534 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:15.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:15.650+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.653254+0000) 2019-10-13T14:25:15.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:15.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.940626+0000) 2019-10-13T14:25:16.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:16.614+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:16.618177+0000) 2019-10-13T14:25:16.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:16.866+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:16.868147+0000) 2019-10-13T14:25:18.733 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:18.726+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.730316+0000) 2019-10-13T14:25:18.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:18.726+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.730638+0000) 2019-10-13T14:25:18.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:18.726+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.731090+0000) 2019-10-13T14:25:18.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:18.730+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.731903+0000) 2019-10-13T14:25:18.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:18.730+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.733261+0000) 2019-10-13T14:25:18.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:18.730+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.733334+0000) 2019-10-13T14:25:18.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:18.730+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.733729+0000) 2019-10-13T14:25:18.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:18.738+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195567 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:18.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:18.734+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.741998+0000) 2019-10-13T14:25:18.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:18.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.890091+0000) 2019-10-13T14:25:18.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:18.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.890091+0000) 2019-10-13T14:25:22.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:22.050+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:22.056961+0000) 2019-10-13T14:25:22.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:22.626+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:22.628664+0000) 2019-10-13T14:25:23.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:23.738+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.743945+0000) 2019-10-13T14:25:23.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:23.742+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.744625+0000) 2019-10-13T14:25:23.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:23.742+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.744708+0000) 2019-10-13T14:25:23.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:23.742+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.745384+0000) 2019-10-13T14:25:23.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:23.742+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.745993+0000) 2019-10-13T14:25:23.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:23.742+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.746872+0000) 2019-10-13T14:25:23.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:23.742+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.747203+0000) 2019-10-13T14:25:23.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:23.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195600 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:23.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:23.750+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:23.754639+0000) 2019-10-13T14:25:25.014 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:25:25.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:25.018 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:25:25.018 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:25.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:25.650+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.653454+0000) 2019-10-13T14:25:25.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:25.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.940869+0000) 2019-10-13T14:25:26.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:26.614+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:26.618367+0000) 2019-10-13T14:25:26.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:26.866+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:26.868383+0000) 2019-10-13T14:25:28.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:28.754+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.756275+0000) 2019-10-13T14:25:28.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:28.754+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.756838+0000) 2019-10-13T14:25:28.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:28.750+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.757559+0000) 2019-10-13T14:25:28.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:28.754+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.757709+0000) 2019-10-13T14:25:28.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:28.754+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.757856+0000) 2019-10-13T14:25:28.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:28.754+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.759129+0000) 2019-10-13T14:25:28.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:28.758+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.759875+0000) 2019-10-13T14:25:28.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:28.758+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.765507+0000) 2019-10-13T14:25:28.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:28.762+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:28.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:28.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.890290+0000) 2019-10-13T14:25:28.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:28.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.890290+0000) 2019-10-13T14:25:32.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:32.062+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:32.063226+0000) 2019-10-13T14:25:32.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:32.626+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:32.628870+0000) 2019-10-13T14:25:33.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:33.766+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.770209+0000) 2019-10-13T14:25:33.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:33.766+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.771088+0000) 2019-10-13T14:25:33.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:33.770+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.771558+0000) 2019-10-13T14:25:33.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:33.770+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.772056+0000) 2019-10-13T14:25:33.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:33.770+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.772698+0000) 2019-10-13T14:25:33.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:33.770+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.773084+0000) 2019-10-13T14:25:33.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:33.770+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.773658+0000) 2019-10-13T14:25:33.782 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:33.778+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195668 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:33.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:33.782+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:33.783556+0000) 2019-10-13T14:25:35.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:35.650+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.653686+0000) 2019-10-13T14:25:35.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:35.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.941077+0000) 2019-10-13T14:25:36.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:36.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:36.618582+0000) 2019-10-13T14:25:36.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:36.866+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:36.868595+0000) 2019-10-13T14:25:38.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:38.782+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.783886+0000) 2019-10-13T14:25:38.787 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:38.782+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.784362+0000) 2019-10-13T14:25:38.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:38.782+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.785768+0000) 2019-10-13T14:25:38.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:38.785+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.786661+0000) 2019-10-13T14:25:38.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:38.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.787357+0000) 2019-10-13T14:25:38.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:38.786+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.787653+0000) 2019-10-13T14:25:38.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:38.786+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.788052+0000) 2019-10-13T14:25:38.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:38.789+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.792035+0000) 2019-10-13T14:25:38.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:38.790+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195700 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:38.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:38.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.890464+0000) 2019-10-13T14:25:38.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:38.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.890464+0000) 2019-10-13T14:25:42.072 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:42.065+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:42.069719+0000) 2019-10-13T14:25:42.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:42.626+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:42.629071+0000) 2019-10-13T14:25:43.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:43.794+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.796797+0000) 2019-10-13T14:25:43.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:43.794+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.797983+0000) 2019-10-13T14:25:43.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:43.794+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.798967+0000) 2019-10-13T14:25:43.801 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:43.798+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.799465+0000) 2019-10-13T14:25:43.803 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:43.797+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.801160+0000) 2019-10-13T14:25:43.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:43.798+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.801538+0000) 2019-10-13T14:25:43.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:43.798+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.802222+0000) 2019-10-13T14:25:43.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:43.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:43.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:43.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:43.808322+0000) 2019-10-13T14:25:45.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:45.650+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.653888+0000) 2019-10-13T14:25:45.943 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:45.938+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.941270+0000) 2019-10-13T14:25:46.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:46.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:46.618818+0000) 2019-10-13T14:25:46.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:46.865+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:46.868753+0000) 2019-10-13T14:25:48.812 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:48.809+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.810364+0000) 2019-10-13T14:25:48.814 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:48.810+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.811814+0000) 2019-10-13T14:25:48.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:48.810+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.811664+0000) 2019-10-13T14:25:48.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:48.810+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.812169+0000) 2019-10-13T14:25:48.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:48.810+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.812685+0000) 2019-10-13T14:25:48.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:48.810+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.813063+0000) 2019-10-13T14:25:48.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:48.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.813417+0000) 2019-10-13T14:25:48.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:48.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:48.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:48.817+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.820542+0000) 2019-10-13T14:25:48.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:48.886+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.890692+0000) 2019-10-13T14:25:48.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:48.886+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.890692+0000) 2019-10-13T14:25:52.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:52.073+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:52.076030+0000) 2019-10-13T14:25:52.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:52.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:52.629301+0000) 2019-10-13T14:25:53.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:53.821+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.823476+0000) 2019-10-13T14:25:53.826 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:53.821+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.824240+0000) 2019-10-13T14:25:53.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:53.821+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.825193+0000) 2019-10-13T14:25:53.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:53.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.824579+0000) 2019-10-13T14:25:53.828 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:53.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.825386+0000) 2019-10-13T14:25:53.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:53.821+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.825835+0000) 2019-10-13T14:25:53.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:53.821+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.826665+0000) 2019-10-13T14:25:53.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:53.829+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:53.831892+0000) 2019-10-13T14:25:53.835 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:53.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:55.151 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:25:55.151 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:55.155 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:25:55.155 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:55.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:55.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.654113+0000) 2019-10-13T14:25:55.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:55.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.941470+0000) 2019-10-13T14:25:56.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:56.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:56.619015+0000) 2019-10-13T14:25:56.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:56.865+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:56.868945+0000) 2019-10-13T14:25:58.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:25:58.833+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.835517+0000) 2019-10-13T14:25:58.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:58.833+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.836232+0000) 2019-10-13T14:25:58.839 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:25:58.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.837123+0000) 2019-10-13T14:25:58.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:25:58.837+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.838919+0000) 2019-10-13T14:25:58.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:25:58.833+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.839197+0000) 2019-10-13T14:25:58.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:25:58.837+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.839710+0000) 2019-10-13T14:25:58.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:58.837+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.840287+0000) 2019-10-13T14:25:58.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:25:58.841+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.842992+0000) 2019-10-13T14:25:58.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:25:58.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:25:58.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:25:58.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.890831+0000) 2019-10-13T14:25:58.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:25:58.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.890831+0000) 2019-10-13T14:26:02.085 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:02.081+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:02.082428+0000) 2019-10-13T14:26:02.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:02.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:02.629536+0000) 2019-10-13T14:26:03.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:03.845+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.849351+0000) 2019-10-13T14:26:03.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:03.845+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.850042+0000) 2019-10-13T14:26:03.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:03.845+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.850602+0000) 2019-10-13T14:26:03.853 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:03.845+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.850306+0000) 2019-10-13T14:26:03.853 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:03.845+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.850971+0000) 2019-10-13T14:26:03.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:03.849+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.851814+0000) 2019-10-13T14:26:03.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:03.849+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.853096+0000) 2019-10-13T14:26:03.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:03.857+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:03.858421+0000) 2019-10-13T14:26:03.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:03.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:05.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:05.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.654316+0000) 2019-10-13T14:26:05.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:05.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.941690+0000) 2019-10-13T14:26:06.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:06.613+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:06.619209+0000) 2019-10-13T14:26:06.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:06.865+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:06.869151+0000) 2019-10-13T14:26:08.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:08.857+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.861953+0000) 2019-10-13T14:26:08.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:08.857+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.862290+0000) 2019-10-13T14:26:08.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:08.857+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.862658+0000) 2019-10-13T14:26:08.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:08.861+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.863298+0000) 2019-10-13T14:26:08.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:08.861+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.864596+0000) 2019-10-13T14:26:08.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:08.861+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.865295+0000) 2019-10-13T14:26:08.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:08.861+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.865782+0000) 2019-10-13T14:26:08.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:08.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:08.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:08.873+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.875870+0000) 2019-10-13T14:26:08.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:08.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.891009+0000) 2019-10-13T14:26:08.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:08.885+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.891009+0000) 2019-10-13T14:26:12.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:12.085+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:12.088826+0000) 2019-10-13T14:26:12.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:12.629+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:12.629727+0000) 2019-10-13T14:26:13.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:13.873+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.875637+0000) 2019-10-13T14:26:13.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:13.873+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.876236+0000) 2019-10-13T14:26:13.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:13.873+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.876721+0000) 2019-10-13T14:26:13.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:13.877+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.878259+0000) 2019-10-13T14:26:13.881 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:13.873+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.878709+0000) 2019-10-13T14:26:13.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:13.873+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.879005+0000) 2019-10-13T14:26:13.882 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:13.877+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.880144+0000) 2019-10-13T14:26:13.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:13.881+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:13.883489+0000) 2019-10-13T14:26:13.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:13.881+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:15.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:15.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.654529+0000) 2019-10-13T14:26:15.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:15.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.941843+0000) 2019-10-13T14:26:16.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:16.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:16.619413+0000) 2019-10-13T14:26:16.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:16.869+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:16.869374+0000) 2019-10-13T14:26:18.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:18.885+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.890104+0000) 2019-10-13T14:26:18.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:18.885+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.890772+0000) 2019-10-13T14:26:18.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:18.889+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.891283+0000) 2019-10-13T14:26:18.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:18.889+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.891629+0000) 2019-10-13T14:26:18.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:18.885+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.891164+0000) 2019-10-13T14:26:18.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:18.889+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.892833+0000) 2019-10-13T14:26:18.895 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:18.889+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.892264+0000) 2019-10-13T14:26:18.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:18.889+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.893049+0000) 2019-10-13T14:26:18.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:18.893+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.895259+0000) 2019-10-13T14:26:18.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:18.897+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 195969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:22.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:22.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:22.099924+0000) 2019-10-13T14:26:22.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:22.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:22.629947+0000) 2019-10-13T14:26:23.906 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:23.897+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.903237+0000) 2019-10-13T14:26:23.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:23.897+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.903238+0000) 2019-10-13T14:26:23.906 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:23.897+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.903248+0000) 2019-10-13T14:26:23.907 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:23.901+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.904893+0000) 2019-10-13T14:26:23.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:23.901+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.905502+0000) 2019-10-13T14:26:23.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:23.901+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.906112+0000) 2019-10-13T14:26:23.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:23.905+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.906267+0000) 2019-10-13T14:26:23.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:23.909+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:23.911601+0000) 2019-10-13T14:26:23.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:23.909+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:25.274 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:26:25.274 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:25.278 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:26:25.278 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:25.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:25.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.641744+0000) 2019-10-13T14:26:25.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:25.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.654729+0000) 2019-10-13T14:26:25.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:25.937+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.942049+0000) 2019-10-13T14:26:26.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:26.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:26.619607+0000) 2019-10-13T14:26:26.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:26.865+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:26.869588+0000) 2019-10-13T14:26:28.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:28.889+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.891370+0000) 2019-10-13T14:26:28.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:28.913+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.916525+0000) 2019-10-13T14:26:28.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:28.913+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.917605+0000) 2019-10-13T14:26:28.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:28.913+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.918073+0000) 2019-10-13T14:26:28.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:28.913+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.918590+0000) 2019-10-13T14:26:28.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:28.913+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.919083+0000) 2019-10-13T14:26:28.922 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:28.913+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.919269+0000) 2019-10-13T14:26:28.922 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:28.917+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.920482+0000) 2019-10-13T14:26:28.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:28.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:28.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:28.925+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.928798+0000) 2019-10-13T14:26:32.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:32.104+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:32.106483+0000) 2019-10-13T14:26:32.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:32.625+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:32.630152+0000) 2019-10-13T14:26:33.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:33.925+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.928908+0000) 2019-10-13T14:26:33.932 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:33.925+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.929916+0000) 2019-10-13T14:26:33.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:33.925+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.930508+0000) 2019-10-13T14:26:33.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:33.925+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.930976+0000) 2019-10-13T14:26:33.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:33.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.931880+0000) 2019-10-13T14:26:33.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:33.929+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.931968+0000) 2019-10-13T14:26:33.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:33.929+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.932865+0000) 2019-10-13T14:26:33.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:33.936+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:33.939180+0000) 2019-10-13T14:26:33.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:33.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196070 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:35.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:35.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.641975+0000) 2019-10-13T14:26:35.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:35.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.654923+0000) 2019-10-13T14:26:35.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:35.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.942261+0000) 2019-10-13T14:26:36.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:36.616+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:36.619820+0000) 2019-10-13T14:26:36.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:36.868+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:36.869799+0000) 2019-10-13T14:26:38.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:38.888+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.891593+0000) 2019-10-13T14:26:38.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:38.936+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.942371+0000) 2019-10-13T14:26:38.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:38.940+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.943561+0000) 2019-10-13T14:26:38.946 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:38.940+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.943811+0000) 2019-10-13T14:26:38.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:38.940+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.944160+0000) 2019-10-13T14:26:38.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:38.940+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.944507+0000) 2019-10-13T14:26:38.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:38.940+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.945086+0000) 2019-10-13T14:26:38.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:38.944+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.946414+0000) 2019-10-13T14:26:38.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:38.948+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.951579+0000) 2019-10-13T14:26:38.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:38.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:42.115 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:42.112+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:42.112872+0000) 2019-10-13T14:26:42.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:42.624+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:42.630358+0000) 2019-10-13T14:26:43.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:43.948+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.954541+0000) 2019-10-13T14:26:43.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:43.952+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.956128+0000) 2019-10-13T14:26:43.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:43.952+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.957182+0000) 2019-10-13T14:26:43.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:43.952+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.957679+0000) 2019-10-13T14:26:43.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:43.952+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.957792+0000) 2019-10-13T14:26:43.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:43.952+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.958213+0000) 2019-10-13T14:26:43.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:43.952+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.958654+0000) 2019-10-13T14:26:43.965 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:43.960+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:43.963062+0000) 2019-10-13T14:26:43.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:43.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196136 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:45.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:45.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.642179+0000) 2019-10-13T14:26:45.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:45.648+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.655119+0000) 2019-10-13T14:26:45.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:45.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.942469+0000) 2019-10-13T14:26:46.622 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:46.616+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:46.620011+0000) 2019-10-13T14:26:46.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:46.864+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:46.870011+0000) 2019-10-13T14:26:48.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:48.888+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.891793+0000) 2019-10-13T14:26:48.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:48.964+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.969247+0000) 2019-10-13T14:26:48.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:48.964+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.969530+0000) 2019-10-13T14:26:48.972 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:48.964+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.969992+0000) 2019-10-13T14:26:48.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:48.964+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.970708+0000) 2019-10-13T14:26:48.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:48.968+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.971667+0000) 2019-10-13T14:26:48.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:48.968+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.972001+0000) 2019-10-13T14:26:48.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:48.968+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.972617+0000) 2019-10-13T14:26:48.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:48.972+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.976754+0000) 2019-10-13T14:26:48.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:48.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:52.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:52.116+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:52.119355+0000) 2019-10-13T14:26:52.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:52.628+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:52.630569+0000) 2019-10-13T14:26:53.986 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:53.980+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.983796+0000) 2019-10-13T14:26:53.987 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:53.980+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.984595+0000) 2019-10-13T14:26:53.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:53.980+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.984854+0000) 2019-10-13T14:26:53.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:53.980+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.985367+0000) 2019-10-13T14:26:53.988 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:53.980+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.986120+0000) 2019-10-13T14:26:53.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:53.984+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.986576+0000) 2019-10-13T14:26:53.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:53.980+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.987016+0000) 2019-10-13T14:26:53.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:53.988+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:53.993876+0000) 2019-10-13T14:26:53.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:53.988+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:55.365 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:26:55.365 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:55.369 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:26:55.369 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:55.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:55.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:55.642425+0000) 2019-10-13T14:26:55.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:55.652+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:55.655345+0000) 2019-10-13T14:26:55.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:55.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:55.942676+0000) 2019-10-13T14:26:56.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:56.616+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.620202+0000) 2019-10-13T14:26:56.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:56.868+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.870218+0000) 2019-10-13T14:26:58.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:58.888+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.891987+0000) 2019-10-13T14:26:58.999 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:26:58.992+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.997170+0000) 2019-10-13T14:26:59.000 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:26:58.992+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.997922+0000) 2019-10-13T14:26:59.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:26:58.992+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.998279+0000) 2019-10-13T14:26:59.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:26:58.992+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.998770+0000) 2019-10-13T14:26:59.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:26:58.996+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.999586+0000) 2019-10-13T14:26:59.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:26:58.996+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:59.000103+0000) 2019-10-13T14:26:59.003 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:26:58.996+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:59.000702+0000) 2019-10-13T14:26:59.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:26:59.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:26:59.010 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:26:59.004+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:59.008399+0000) 2019-10-13T14:27:02.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:02.120+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:02.125745+0000) 2019-10-13T14:27:02.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:02.624+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:02.630776+0000) 2019-10-13T14:27:04.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:04.004+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.010532+0000) 2019-10-13T14:27:04.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:04.004+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.011189+0000) 2019-10-13T14:27:04.015 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:04.008+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.012456+0000) 2019-10-13T14:27:04.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:04.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.013253+0000) 2019-10-13T14:27:04.016 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:04.008+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.013441+0000) 2019-10-13T14:27:04.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:04.008+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.013933+0000) 2019-10-13T14:27:04.016 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:04.008+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.014283+0000) 2019-10-13T14:27:04.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:04.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.018094+0000) 2019-10-13T14:27:04.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:04.016+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:05.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:05.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:05.642626+0000) 2019-10-13T14:27:05.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:05.652+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:05.655613+0000) 2019-10-13T14:27:05.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:05.936+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:05.942911+0000) 2019-10-13T14:27:06.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:06.616+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.620355+0000) 2019-10-13T14:27:06.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:06.868+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.870459+0000) 2019-10-13T14:27:08.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:08.888+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:08.892186+0000) 2019-10-13T14:27:09.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:09.020+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.023497+0000) 2019-10-13T14:27:09.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:09.020+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.023714+0000) 2019-10-13T14:27:09.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:09.020+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.024385+0000) 2019-10-13T14:27:09.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:09.020+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.025035+0000) 2019-10-13T14:27:09.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:09.020+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.025844+0000) 2019-10-13T14:27:09.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:09.020+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.025402+0000) 2019-10-13T14:27:09.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:09.024+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.027001+0000) 2019-10-13T14:27:09.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:09.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:09.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:09.032+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:09.035790+0000) 2019-10-13T14:27:12.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:12.128+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:12.132110+0000) 2019-10-13T14:27:12.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:12.628+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:12.630981+0000) 2019-10-13T14:27:14.039 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:14.032+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.036307+0000) 2019-10-13T14:27:14.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:14.032+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.037110+0000) 2019-10-13T14:27:14.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:14.032+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.038306+0000) 2019-10-13T14:27:14.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:14.032+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.039117+0000) 2019-10-13T14:27:14.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:14.036+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.039680+0000) 2019-10-13T14:27:14.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:14.036+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.039940+0000) 2019-10-13T14:27:14.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:14.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.041655+0000) 2019-10-13T14:27:14.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:14.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:14.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:14.044+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.047059+0000) 2019-10-13T14:27:15.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:15.636+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:15.642858+0000) 2019-10-13T14:27:15.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:15.652+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:15.655810+0000) 2019-10-13T14:27:15.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:15.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:15.943132+0000) 2019-10-13T14:27:16.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:16.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.620546+0000) 2019-10-13T14:27:16.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:16.868+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.870651+0000) 2019-10-13T14:27:18.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:18.891+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:18.892380+0000) 2019-10-13T14:27:19.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:19.043+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.050074+0000) 2019-10-13T14:27:19.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:19.048+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.050837+0000) 2019-10-13T14:27:19.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:19.047+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.051456+0000) 2019-10-13T14:27:19.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:19.047+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.052101+0000) 2019-10-13T14:27:19.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:19.047+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.052551+0000) 2019-10-13T14:27:19.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:19.047+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.052952+0000) 2019-10-13T14:27:19.056 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:19.047+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.053590+0000) 2019-10-13T14:27:19.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:19.056+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:19.059820+0000) 2019-10-13T14:27:19.062 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:19.055+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:22.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:22.140+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:22.142744+0000) 2019-10-13T14:27:22.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:22.623+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:22.631207+0000) 2019-10-13T14:27:24.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:24.055+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.063085+0000) 2019-10-13T14:27:24.066 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:24.059+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.064329+0000) 2019-10-13T14:27:24.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:24.059+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.065034+0000) 2019-10-13T14:27:24.068 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:24.059+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.065811+0000) 2019-10-13T14:27:24.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:24.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.066441+0000) 2019-10-13T14:27:24.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:24.059+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.066575+0000) 2019-10-13T14:27:24.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:24.059+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.067022+0000) 2019-10-13T14:27:24.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:24.068+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.070622+0000) 2019-10-13T14:27:24.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:24.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196403 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:25.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:25.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:25.643088+0000) 2019-10-13T14:27:25.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:25.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:25.656009+0000) 2019-10-13T14:27:25.738 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:27:25.738 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:25.742 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:27:25.742 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:25.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:25.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:25.943358+0000) 2019-10-13T14:27:26.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:26.619+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.620761+0000) 2019-10-13T14:27:26.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:26.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.870862+0000) 2019-10-13T14:27:28.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:28.891+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:28.892625+0000) 2019-10-13T14:27:29.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:29.071+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.078011+0000) 2019-10-13T14:27:29.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:29.071+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.078605+0000) 2019-10-13T14:27:29.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:29.071+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.079064+0000) 2019-10-13T14:27:29.082 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:29.075+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.079624+0000) 2019-10-13T14:27:29.082 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:29.075+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.079946+0000) 2019-10-13T14:27:29.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:29.075+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.080401+0000) 2019-10-13T14:27:29.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:29.075+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.081147+0000) 2019-10-13T14:27:29.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:29.083+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:29.086819+0000) 2019-10-13T14:27:29.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:29.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:32.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:32.147+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:32.149104+0000) 2019-10-13T14:27:32.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:32.627+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:32.631423+0000) 2019-10-13T14:27:34.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:34.087+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.091073+0000) 2019-10-13T14:27:34.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:34.087+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.091522+0000) 2019-10-13T14:27:34.095 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:34.087+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.092535+0000) 2019-10-13T14:27:34.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:34.087+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.093215+0000) 2019-10-13T14:27:34.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:34.087+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.093809+0000) 2019-10-13T14:27:34.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:34.091+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.095470+0000) 2019-10-13T14:27:34.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:34.091+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.095715+0000) 2019-10-13T14:27:34.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:34.095+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.098967+0000) 2019-10-13T14:27:34.104 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:34.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:35.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:35.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:35.643346+0000) 2019-10-13T14:27:35.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:35.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:35.656203+0000) 2019-10-13T14:27:35.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:35.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:35.943577+0000) 2019-10-13T14:27:36.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:36.619+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.620957+0000) 2019-10-13T14:27:36.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:36.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.871090+0000) 2019-10-13T14:27:38.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:38.887+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:38.892831+0000) 2019-10-13T14:27:39.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:39.099+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.104933+0000) 2019-10-13T14:27:39.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:39.099+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.105476+0000) 2019-10-13T14:27:39.108 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:39.099+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.105844+0000) 2019-10-13T14:27:39.109 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:39.099+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.106421+0000) 2019-10-13T14:27:39.109 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:39.099+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.106702+0000) 2019-10-13T14:27:39.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:39.103+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.108425+0000) 2019-10-13T14:27:39.111 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:39.103+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.108629+0000) 2019-10-13T14:27:39.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:39.111+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:39.115131+0000) 2019-10-13T14:27:39.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:39.111+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:42.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:42.151+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:42.155411+0000) 2019-10-13T14:27:42.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:42.631+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:42.631671+0000) 2019-10-13T14:27:44.120 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:44.115+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.118115+0000) 2019-10-13T14:27:44.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:44.115+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.119649+0000) 2019-10-13T14:27:44.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:44.115+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.120582+0000) 2019-10-13T14:27:44.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:44.115+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.121127+0000) 2019-10-13T14:27:44.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:44.115+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.121538+0000) 2019-10-13T14:27:44.124 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:44.119+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.122235+0000) 2019-10-13T14:27:44.125 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:44.115+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.122806+0000) 2019-10-13T14:27:44.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:44.123+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:44.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:44.123+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.130032+0000) 2019-10-13T14:27:45.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:45.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:45.643578+0000) 2019-10-13T14:27:45.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:45.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:45.656451+0000) 2019-10-13T14:27:45.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:45.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:45.943795+0000) 2019-10-13T14:27:46.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:46.619+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.621156+0000) 2019-10-13T14:27:46.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:46.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.871279+0000) 2019-10-13T14:27:48.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:48.891+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:48.893037+0000) 2019-10-13T14:27:49.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:49.131+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.132743+0000) 2019-10-13T14:27:49.136 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:49.131+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.133852+0000) 2019-10-13T14:27:49.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:49.131+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.134261+0000) 2019-10-13T14:27:49.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:49.135+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.135530+0000) 2019-10-13T14:27:49.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:49.131+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.136274+0000) 2019-10-13T14:27:49.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:49.135+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.136481+0000) 2019-10-13T14:27:49.139 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:49.135+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.137110+0000) 2019-10-13T14:27:49.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:49.139+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:49.142853+0000) 2019-10-13T14:27:49.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:49.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:52.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:52.159+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:52.161963+0000) 2019-10-13T14:27:52.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:52.631+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:52.631907+0000) 2019-10-13T14:27:54.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:54.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.146349+0000) 2019-10-13T14:27:54.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:54.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.147291+0000) 2019-10-13T14:27:54.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:54.143+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.146916+0000) 2019-10-13T14:27:54.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:54.147+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.148263+0000) 2019-10-13T14:27:54.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:54.147+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.148780+0000) 2019-10-13T14:27:54.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:54.147+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.149578+0000) 2019-10-13T14:27:54.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:54.147+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.149840+0000) 2019-10-13T14:27:54.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:54.151+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.156457+0000) 2019-10-13T14:27:54.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:54.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196604 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:55.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:55.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:55.643781+0000) 2019-10-13T14:27:55.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:55.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:55.656652+0000) 2019-10-13T14:27:55.852 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:27:55.852 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:55.856 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:27:55.857 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:55.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:55.943+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:55.943996+0000) 2019-10-13T14:27:56.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:56.619+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.621390+0000) 2019-10-13T14:27:56.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:56.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.871486+0000) 2019-10-13T14:27:58.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:58.891+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:58.893287+0000) 2019-10-13T14:27:59.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:27:59.159+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.160966+0000) 2019-10-13T14:27:59.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:27:59.159+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.161975+0000) 2019-10-13T14:27:59.164 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:27:59.159+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.162344+0000) 2019-10-13T14:27:59.165 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:27:59.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.163229+0000) 2019-10-13T14:27:59.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:27:59.163+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.163368+0000) 2019-10-13T14:27:59.166 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:27:59.163+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.164273+0000) 2019-10-13T14:27:59.167 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:27:59.163+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.165344+0000) 2019-10-13T14:27:59.172 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:27:59.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:27:59.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:27:59.167+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:59.171366+0000) 2019-10-13T14:28:02.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:02.163+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:02.168431+0000) 2019-10-13T14:28:02.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:02.630+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:02.632140+0000) 2019-10-13T14:28:04.175 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:04.167+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.173307+0000) 2019-10-13T14:28:04.176 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:04.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.173595+0000) 2019-10-13T14:28:04.176 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:04.170+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.173866+0000) 2019-10-13T14:28:04.178 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:04.174+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.175905+0000) 2019-10-13T14:28:04.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:04.174+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.176254+0000) 2019-10-13T14:28:04.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:04.174+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.176560+0000) 2019-10-13T14:28:04.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:04.174+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.177193+0000) 2019-10-13T14:28:04.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:04.175+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.180877+0000) 2019-10-13T14:28:04.185 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:04.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:05.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:05.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:05.643976+0000) 2019-10-13T14:28:05.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:05.654+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:05.656862+0000) 2019-10-13T14:28:05.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:05.942+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:05.944221+0000) 2019-10-13T14:28:06.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:06.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.621583+0000) 2019-10-13T14:28:06.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:06.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.871704+0000) 2019-10-13T14:28:08.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:08.890+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:08.893484+0000) 2019-10-13T14:28:09.189 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:09.182+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.186654+0000) 2019-10-13T14:28:09.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:09.182+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.187083+0000) 2019-10-13T14:28:09.191 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:09.186+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.187857+0000) 2019-10-13T14:28:09.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:09.186+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.187984+0000) 2019-10-13T14:28:09.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:09.186+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.188672+0000) 2019-10-13T14:28:09.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:09.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.189352+0000) 2019-10-13T14:28:09.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:09.186+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.189395+0000) 2019-10-13T14:28:09.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:09.187+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:09.193941+0000) 2019-10-13T14:28:09.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:09.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196706 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:12.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:12.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:12.174790+0000) 2019-10-13T14:28:12.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:12.630+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:12.632343+0000) 2019-10-13T14:28:14.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:14.198+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.200743+0000) 2019-10-13T14:28:14.204 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:14.198+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.201070+0000) 2019-10-13T14:28:14.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:14.198+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.201362+0000) 2019-10-13T14:28:14.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:14.198+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.202406+0000) 2019-10-13T14:28:14.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:14.198+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.202880+0000) 2019-10-13T14:28:14.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:14.202+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.203425+0000) 2019-10-13T14:28:14.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:14.199+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.204117+0000) 2019-10-13T14:28:14.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:14.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:14.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:14.207+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.210695+0000) 2019-10-13T14:28:15.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:15.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:15.644175+0000) 2019-10-13T14:28:15.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:15.654+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:15.657054+0000) 2019-10-13T14:28:15.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:15.942+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:15.944456+0000) 2019-10-13T14:28:16.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:16.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.621777+0000) 2019-10-13T14:28:16.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:16.867+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.871892+0000) 2019-10-13T14:28:18.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:18.890+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:18.893687+0000) 2019-10-13T14:28:19.216 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:19.210+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.213874+0000) 2019-10-13T14:28:19.217 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:19.210+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.214255+0000) 2019-10-13T14:28:19.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:19.210+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.214001+0000) 2019-10-13T14:28:19.217 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:19.210+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.215029+0000) 2019-10-13T14:28:19.218 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:19.211+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.215500+0000) 2019-10-13T14:28:19.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:19.214+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.216431+0000) 2019-10-13T14:28:19.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:19.214+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.216802+0000) 2019-10-13T14:28:19.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:19.219+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:19.222236+0000) 2019-10-13T14:28:19.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:19.222+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:22.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:22.174+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:22.181297+0000) 2019-10-13T14:28:22.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:22.630+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:22.632571+0000) 2019-10-13T14:28:24.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:24.222+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.226789+0000) 2019-10-13T14:28:24.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:24.226+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.227285+0000) 2019-10-13T14:28:24.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:24.226+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.228321+0000) 2019-10-13T14:28:24.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:24.226+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.228717+0000) 2019-10-13T14:28:24.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:24.222+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.229203+0000) 2019-10-13T14:28:24.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:24.226+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.229800+0000) 2019-10-13T14:28:24.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:24.226+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.230383+0000) 2019-10-13T14:28:24.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:24.230+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.235672+0000) 2019-10-13T14:28:24.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:24.234+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:25.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:25.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:25.644371+0000) 2019-10-13T14:28:25.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:25.654+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:25.657256+0000) 2019-10-13T14:28:25.900 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:28:25.901 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:25.904 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:28:25.905 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:25.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:25.942+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:25.944673+0000) 2019-10-13T14:28:26.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:26.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.621972+0000) 2019-10-13T14:28:26.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:26.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.872087+0000) 2019-10-13T14:28:28.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:28.890+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:28.893880+0000) 2019-10-13T14:28:29.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:29.238+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.241227+0000) 2019-10-13T14:28:29.244 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:29.238+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.241919+0000) 2019-10-13T14:28:29.244 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:29.238+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.242308+0000) 2019-10-13T14:28:29.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:29.238+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.242843+0000) 2019-10-13T14:28:29.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:29.238+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.243842+0000) 2019-10-13T14:28:29.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:29.242+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.244005+0000) 2019-10-13T14:28:29.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:29.242+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.244296+0000) 2019-10-13T14:28:29.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:29.242+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:29.249685+0000) 2019-10-13T14:28:29.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:29.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:32.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:32.182+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:32.187705+0000) 2019-10-13T14:28:32.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:32.630+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:32.632810+0000) 2019-10-13T14:28:34.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:34.250+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.254423+0000) 2019-10-13T14:28:34.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:34.250+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.254631+0000) 2019-10-13T14:28:34.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:34.254+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.255374+0000) 2019-10-13T14:28:34.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:34.250+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.255061+0000) 2019-10-13T14:28:34.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:34.254+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.255895+0000) 2019-10-13T14:28:34.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:34.254+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.257311+0000) 2019-10-13T14:28:34.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:34.254+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.257977+0000) 2019-10-13T14:28:34.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:34.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:34.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:34.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.265534+0000) 2019-10-13T14:28:35.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:35.642+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:35.644571+0000) 2019-10-13T14:28:35.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:35.654+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:35.657457+0000) 2019-10-13T14:28:35.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:35.942+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:35.944904+0000) 2019-10-13T14:28:36.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:36.618+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.622199+0000) 2019-10-13T14:28:36.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:36.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.872275+0000) 2019-10-13T14:28:38.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:38.890+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:38.894082+0000) 2019-10-13T14:28:39.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:39.262+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.266523+0000) 2019-10-13T14:28:39.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:39.262+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.266949+0000) 2019-10-13T14:28:39.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:39.266+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.267417+0000) 2019-10-13T14:28:39.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:39.266+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.268081+0000) 2019-10-13T14:28:39.271 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:39.262+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.268445+0000) 2019-10-13T14:28:39.271 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:39.266+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.268570+0000) 2019-10-13T14:28:39.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:39.266+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.269446+0000) 2019-10-13T14:28:39.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:39.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:39.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:39.274+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:39.278519+0000) 2019-10-13T14:28:42.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:42.198+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:42.198758+0000) 2019-10-13T14:28:42.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:42.630+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:42.633010+0000) 2019-10-13T14:28:44.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:44.278+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.279573+0000) 2019-10-13T14:28:44.282 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:44.278+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.280128+0000) 2019-10-13T14:28:44.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:44.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.281262+0000) 2019-10-13T14:28:44.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:44.278+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.281373+0000) 2019-10-13T14:28:44.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:44.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.281980+0000) 2019-10-13T14:28:44.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:44.278+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.281913+0000) 2019-10-13T14:28:44.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:44.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.283035+0000) 2019-10-13T14:28:44.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:44.282+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.288905+0000) 2019-10-13T14:28:44.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:44.286+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:45.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:45.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:45.644774+0000) 2019-10-13T14:28:45.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:45.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:45.657645+0000) 2019-10-13T14:28:45.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:45.941+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:45.945136+0000) 2019-10-13T14:28:46.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:46.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.622395+0000) 2019-10-13T14:28:46.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:46.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.872472+0000) 2019-10-13T14:28:48.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:48.893+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:48.894282+0000) 2019-10-13T14:28:49.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:49.289+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.292648+0000) 2019-10-13T14:28:49.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:49.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.293052+0000) 2019-10-13T14:28:49.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:49.289+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.294302+0000) 2019-10-13T14:28:49.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:49.289+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.293976+0000) 2019-10-13T14:28:49.297 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:49.289+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.294598+0000) 2019-10-13T14:28:49.298 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:49.293+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.295334+0000) 2019-10-13T14:28:49.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:49.293+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.295954+0000) 2019-10-13T14:28:49.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:49.294+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:49.300629+0000) 2019-10-13T14:28:49.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:49.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 196980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:52.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:52.202+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:52.205145+0000) 2019-10-13T14:28:52.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:52.629+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:52.633212+0000) 2019-10-13T14:28:54.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:54.301+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.305506+0000) 2019-10-13T14:28:54.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:54.301+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.306458+0000) 2019-10-13T14:28:54.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:54.301+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.307153+0000) 2019-10-13T14:28:54.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:54.302+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.308203+0000) 2019-10-13T14:28:54.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:54.305+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.309027+0000) 2019-10-13T14:28:54.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:54.305+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.309435+0000) 2019-10-13T14:28:54.312 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:54.305+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.309159+0000) 2019-10-13T14:28:54.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:54.310+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.315021+0000) 2019-10-13T14:28:54.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:54.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197012 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:28:55.647 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:55.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:55.644977+0000) 2019-10-13T14:28:55.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:55.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:55.657865+0000) 2019-10-13T14:28:55.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:55.941+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:55.945380+0000) 2019-10-13T14:28:56.015 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:28:56.015 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:56.019 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:28:56.019 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:56.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:56.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.622588+0000) 2019-10-13T14:28:56.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:56.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.872687+0000) 2019-10-13T14:28:58.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:58.893+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:58.894431+0000) 2019-10-13T14:28:59.322 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:28:59.317+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.319781+0000) 2019-10-13T14:28:59.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:28:59.314+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.320630+0000) 2019-10-13T14:28:59.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:28:59.317+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.321222+0000) 2019-10-13T14:28:59.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:28:59.317+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.321823+0000) 2019-10-13T14:28:59.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:28:59.317+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.322212+0000) 2019-10-13T14:28:59.325 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:28:59.317+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.322495+0000) 2019-10-13T14:28:59.325 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:28:59.317+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.323158+0000) 2019-10-13T14:28:59.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:28:59.322+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:59.329509+0000) 2019-10-13T14:28:59.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:28:59.329+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197049 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:02.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:02.210+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:02.211535+0000) 2019-10-13T14:29:02.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:02.629+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:02.633443+0000) 2019-10-13T14:29:04.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:04.329+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.334639+0000) 2019-10-13T14:29:04.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:04.333+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.335462+0000) 2019-10-13T14:29:04.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:04.333+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.336087+0000) 2019-10-13T14:29:04.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:04.330+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.336788+0000) 2019-10-13T14:29:04.339 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:04.333+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.337192+0000) 2019-10-13T14:29:04.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:04.333+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.337471+0000) 2019-10-13T14:29:04.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:04.333+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.338501+0000) 2019-10-13T14:29:04.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:04.338+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.344324+0000) 2019-10-13T14:29:04.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:04.341+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:05.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:05.641+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:05.645176+0000) 2019-10-13T14:29:05.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:05.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:05.658063+0000) 2019-10-13T14:29:05.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:05.941+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:05.945607+0000) 2019-10-13T14:29:06.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:06.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.622813+0000) 2019-10-13T14:29:06.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:06.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.872914+0000) 2019-10-13T14:29:08.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:08.893+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:08.894658+0000) 2019-10-13T14:29:09.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:09.350+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.351277+0000) 2019-10-13T14:29:09.359 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:09.345+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.349454+0000) 2019-10-13T14:29:09.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:09.345+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.349743+0000) 2019-10-13T14:29:09.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:09.345+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.350127+0000) 2019-10-13T14:29:09.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:09.345+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.350676+0000) 2019-10-13T14:29:09.361 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:09.349+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.351493+0000) 2019-10-13T14:29:09.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:09.349+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.352533+0000) 2019-10-13T14:29:09.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:09.354+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:09.358118+0000) 2019-10-13T14:29:09.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:09.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:12.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:12.214+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:12.217765+0000) 2019-10-13T14:29:12.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:12.629+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:12.633642+0000) 2019-10-13T14:29:14.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:14.357+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.362676+0000) 2019-10-13T14:29:14.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:14.362+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.363082+0000) 2019-10-13T14:29:14.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:14.361+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.364405+0000) 2019-10-13T14:29:14.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:14.361+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.364728+0000) 2019-10-13T14:29:14.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:14.361+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.365120+0000) 2019-10-13T14:29:14.368 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:14.361+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.365387+0000) 2019-10-13T14:29:14.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:14.361+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.366552+0000) 2019-10-13T14:29:14.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:14.366+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.369723+0000) 2019-10-13T14:29:14.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:14.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197150 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:15.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:15.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:15.645376+0000) 2019-10-13T14:29:15.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:15.653+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:15.658214+0000) 2019-10-13T14:29:15.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:15.945+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:15.945832+0000) 2019-10-13T14:29:16.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:16.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.623009+0000) 2019-10-13T14:29:16.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:16.869+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.873110+0000) 2019-10-13T14:29:18.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:18.889+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:18.894865+0000) 2019-10-13T14:29:19.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:19.373+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.377741+0000) 2019-10-13T14:29:19.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:19.373+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.378484+0000) 2019-10-13T14:29:19.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:19.373+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.379152+0000) 2019-10-13T14:29:19.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:19.377+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.379560+0000) 2019-10-13T14:29:19.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:19.377+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.379971+0000) 2019-10-13T14:29:19.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:19.377+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.380479+0000) 2019-10-13T14:29:19.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:19.377+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.380874+0000) 2019-10-13T14:29:19.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:19.381+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:19.384005+0000) 2019-10-13T14:29:19.389 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:19.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:22.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:22.221+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:22.224223+0000) 2019-10-13T14:29:22.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:22.633+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:22.633875+0000) 2019-10-13T14:29:24.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:24.389+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.390377+0000) 2019-10-13T14:29:24.393 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:24.385+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.389882+0000) 2019-10-13T14:29:24.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:24.389+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.391374+0000) 2019-10-13T14:29:24.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:24.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.392528+0000) 2019-10-13T14:29:24.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:24.389+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.392968+0000) 2019-10-13T14:29:24.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:24.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.392976+0000) 2019-10-13T14:29:24.396 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:24.389+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.393709+0000) 2019-10-13T14:29:24.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:24.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.398344+0000) 2019-10-13T14:29:24.403 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:24.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:25.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:25.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:25.645576+0000) 2019-10-13T14:29:25.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:25.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:25.658413+0000) 2019-10-13T14:29:25.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:25.945+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:25.946033+0000) 2019-10-13T14:29:26.130 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:29:26.130 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:26.134 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:29:26.134 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:26.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:26.617+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.623198+0000) 2019-10-13T14:29:26.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:26.869+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.873303+0000) 2019-10-13T14:29:28.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:28.892+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:28.895074+0000) 2019-10-13T14:29:29.407 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:29.400+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.404380+0000) 2019-10-13T14:29:29.407 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:29.400+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.404835+0000) 2019-10-13T14:29:29.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:29.400+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.405265+0000) 2019-10-13T14:29:29.408 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:29.400+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.405933+0000) 2019-10-13T14:29:29.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:29.400+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.406297+0000) 2019-10-13T14:29:29.409 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:29.400+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.405998+0000) 2019-10-13T14:29:29.409 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:29.405+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.406760+0000) 2019-10-13T14:29:29.416 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:29.408+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197255 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:29.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:29.413+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:29.414217+0000) 2019-10-13T14:29:32.233 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:32.229+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:32.230472+0000) 2019-10-13T14:29:32.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:32.632+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:32.634081+0000) 2019-10-13T14:29:34.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:34.412+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.416906+0000) 2019-10-13T14:29:34.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:34.412+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.417276+0000) 2019-10-13T14:29:34.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:34.412+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.418135+0000) 2019-10-13T14:29:34.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:34.412+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.418607+0000) 2019-10-13T14:29:34.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:34.416+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.419790+0000) 2019-10-13T14:29:34.422 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:34.416+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.419604+0000) 2019-10-13T14:29:34.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:34.417+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.421165+0000) 2019-10-13T14:29:34.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:34.425+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.426670+0000) 2019-10-13T14:29:34.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:34.424+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197288 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:35.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:35.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:35.645784+0000) 2019-10-13T14:29:35.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:35.656+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:35.658645+0000) 2019-10-13T14:29:35.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:35.940+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:35.946236+0000) 2019-10-13T14:29:36.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:36.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.623391+0000) 2019-10-13T14:29:36.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:36.869+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.873507+0000) 2019-10-13T14:29:38.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:38.892+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:38.895274+0000) 2019-10-13T14:29:39.434 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:39.424+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.431025+0000) 2019-10-13T14:29:39.434 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:39.424+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.431232+0000) 2019-10-13T14:29:39.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:39.428+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.432000+0000) 2019-10-13T14:29:39.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:39.428+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.432226+0000) 2019-10-13T14:29:39.435 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:39.428+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.432928+0000) 2019-10-13T14:29:39.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:39.428+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.433619+0000) 2019-10-13T14:29:39.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:39.433+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.434380+0000) 2019-10-13T14:29:39.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:39.436+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:39.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:39.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:39.441205+0000) 2019-10-13T14:29:42.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:42.233+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:42.236711+0000) 2019-10-13T14:29:42.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:42.628+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:42.634284+0000) 2019-10-13T14:29:44.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:44.436+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.443041+0000) 2019-10-13T14:29:44.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:44.440+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.444119+0000) 2019-10-13T14:29:44.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:44.440+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.444377+0000) 2019-10-13T14:29:44.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:44.441+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.445423+0000) 2019-10-13T14:29:44.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:44.440+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.445805+0000) 2019-10-13T14:29:44.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:44.440+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.446113+0000) 2019-10-13T14:29:44.449 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:44.440+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.447123+0000) 2019-10-13T14:29:44.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:44.449+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.452905+0000) 2019-10-13T14:29:44.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:44.448+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:45.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:45.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:45.645984+0000) 2019-10-13T14:29:45.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:45.656+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:45.658846+0000) 2019-10-13T14:29:45.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:45.944+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:45.946392+0000) 2019-10-13T14:29:46.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:46.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.623556+0000) 2019-10-13T14:29:46.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:46.873+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.873734+0000) 2019-10-13T14:29:48.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:48.892+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:48.895534+0000) 2019-10-13T14:29:49.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:49.452+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.456053+0000) 2019-10-13T14:29:49.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:49.452+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.456599+0000) 2019-10-13T14:29:49.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:49.452+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.458603+0000) 2019-10-13T14:29:49.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:49.457+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.459009+0000) 2019-10-13T14:29:49.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:49.456+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.459379+0000) 2019-10-13T14:29:49.462 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:49.452+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.459149+0000) 2019-10-13T14:29:49.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:49.456+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.459766+0000) 2019-10-13T14:29:49.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:49.461+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:49.464794+0000) 2019-10-13T14:29:49.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:49.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:52.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:52.241+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:52.243199+0000) 2019-10-13T14:29:52.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:52.632+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:52.634518+0000) 2019-10-13T14:29:54.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:54.464+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.469635+0000) 2019-10-13T14:29:54.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:54.469+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.470252+0000) 2019-10-13T14:29:54.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:54.464+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.470400+0000) 2019-10-13T14:29:54.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:54.464+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.471084+0000) 2019-10-13T14:29:54.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:54.468+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.471383+0000) 2019-10-13T14:29:54.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:54.468+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.471859+0000) 2019-10-13T14:29:54.475 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:54.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.472810+0000) 2019-10-13T14:29:54.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:54.477+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.478817+0000) 2019-10-13T14:29:54.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:54.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:29:55.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:55.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:55.646183+0000) 2019-10-13T14:29:55.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:55.652+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:55.659086+0000) 2019-10-13T14:29:55.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:55.944+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:55.946605+0000) 2019-10-13T14:29:56.252 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:29:56.252 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:56.256 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:29:56.256 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:56.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:56.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.623770+0000) 2019-10-13T14:29:56.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:56.869+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.873888+0000) 2019-10-13T14:29:58.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:58.892+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:58.895728+0000) 2019-10-13T14:29:59.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:29:59.480+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.483721+0000) 2019-10-13T14:29:59.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:29:59.480+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.484263+0000) 2019-10-13T14:29:59.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:29:59.480+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.484635+0000) 2019-10-13T14:29:59.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:29:59.481+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.486002+0000) 2019-10-13T14:29:59.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:29:59.480+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.486089+0000) 2019-10-13T14:29:59.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:29:59.480+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.486507+0000) 2019-10-13T14:29:59.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:29:59.484+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.487361+0000) 2019-10-13T14:29:59.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:29:59.489+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:59.493592+0000) 2019-10-13T14:29:59.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:29:59.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:02.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:02.249+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:02.253287+0000) 2019-10-13T14:30:02.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:02.628+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:02.634747+0000) 2019-10-13T14:30:04.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:04.492+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.496652+0000) 2019-10-13T14:30:04.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:04.492+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.497945+0000) 2019-10-13T14:30:04.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:04.497+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.498931+0000) 2019-10-13T14:30:04.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:04.492+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.499148+0000) 2019-10-13T14:30:04.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:04.496+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.499443+0000) 2019-10-13T14:30:04.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:04.496+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.499730+0000) 2019-10-13T14:30:04.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:04.496+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.500307+0000) 2019-10-13T14:30:04.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:04.501+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.504964+0000) 2019-10-13T14:30:04.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:04.504+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:05.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:05.640+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:05.646414+0000) 2019-10-13T14:30:05.661 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:05.652+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:05.659279+0000) 2019-10-13T14:30:05.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:05.944+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:05.946821+0000) 2019-10-13T14:30:06.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:06.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.623999+0000) 2019-10-13T14:30:06.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:06.873+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.874094+0000) 2019-10-13T14:30:08.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:08.892+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:08.895934+0000) 2019-10-13T14:30:09.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:09.508+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.511516+0000) 2019-10-13T14:30:09.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:09.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.512191+0000) 2019-10-13T14:30:09.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:09.509+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.512867+0000) 2019-10-13T14:30:09.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:09.508+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.513006+0000) 2019-10-13T14:30:09.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:09.508+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.513518+0000) 2019-10-13T14:30:09.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:09.508+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.514115+0000) 2019-10-13T14:30:09.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:09.508+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.514470+0000) 2019-10-13T14:30:09.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:09.513+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:09.517650+0000) 2019-10-13T14:30:09.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:09.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:12.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:12.260+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:12.262607+0000) 2019-10-13T14:30:12.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:12.631+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:12.634951+0000) 2019-10-13T14:30:14.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:14.519+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.524654+0000) 2019-10-13T14:30:14.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:14.519+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.525524+0000) 2019-10-13T14:30:14.528 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:14.519+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.526008+0000) 2019-10-13T14:30:14.529 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:14.519+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.526553+0000) 2019-10-13T14:30:14.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:14.519+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.527036+0000) 2019-10-13T14:30:14.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:14.523+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.527514+0000) 2019-10-13T14:30:14.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:14.524+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.528286+0000) 2019-10-13T14:30:14.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:14.528+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.531350+0000) 2019-10-13T14:30:14.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:14.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:15.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:15.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:15.646635+0000) 2019-10-13T14:30:15.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:15.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:15.659509+0000) 2019-10-13T14:30:15.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:15.939+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:15.947035+0000) 2019-10-13T14:30:16.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:16.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.624220+0000) 2019-10-13T14:30:16.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:16.872+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.874323+0000) 2019-10-13T14:30:18.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:18.891+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:18.896150+0000) 2019-10-13T14:30:19.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:19.531+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.536889+0000) 2019-10-13T14:30:19.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:19.531+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.537152+0000) 2019-10-13T14:30:19.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:19.531+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.537569+0000) 2019-10-13T14:30:19.541 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:19.531+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.538115+0000) 2019-10-13T14:30:19.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:19.536+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.539502+0000) 2019-10-13T14:30:19.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:19.535+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.539796+0000) 2019-10-13T14:30:19.542 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:19.535+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.540320+0000) 2019-10-13T14:30:19.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:19.544+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:19.546700+0000) 2019-10-13T14:30:19.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:19.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:22.276 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:22.272+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:22.273907+0000) 2019-10-13T14:30:22.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:22.631+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:22.635215+0000) 2019-10-13T14:30:24.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:24.543+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.550829+0000) 2019-10-13T14:30:24.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:24.547+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.551461+0000) 2019-10-13T14:30:24.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:24.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.552261+0000) 2019-10-13T14:30:24.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:24.547+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.552559+0000) 2019-10-13T14:30:24.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:24.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.553284+0000) 2019-10-13T14:30:24.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:24.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.553321+0000) 2019-10-13T14:30:24.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:24.547+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.554503+0000) 2019-10-13T14:30:24.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:24.556+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.560314+0000) 2019-10-13T14:30:24.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:24.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:25.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:25.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:25.646835+0000) 2019-10-13T14:30:25.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:25.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:25.659733+0000) 2019-10-13T14:30:25.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:25.943+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:25.947232+0000) 2019-10-13T14:30:26.308 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:30:26.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:26.312 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:30:26.312 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:26.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:26.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.624424+0000) 2019-10-13T14:30:26.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:26.872+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.874523+0000) 2019-10-13T14:30:28.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:28.895+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:28.896351+0000) 2019-10-13T14:30:29.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:29.559+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.565011+0000) 2019-10-13T14:30:29.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:29.559+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.565835+0000) 2019-10-13T14:30:29.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:29.559+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.566148+0000) 2019-10-13T14:30:29.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:29.559+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.566645+0000) 2019-10-13T14:30:29.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:29.563+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.567616+0000) 2019-10-13T14:30:29.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:29.564+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.568682+0000) 2019-10-13T14:30:29.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:29.563+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.568834+0000) 2019-10-13T14:30:29.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:29.567+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:29.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:29.572+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:29.575306+0000) 2019-10-13T14:30:32.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:32.276+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:32.278268+0000) 2019-10-13T14:30:32.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:32.631+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:32.635437+0000) 2019-10-13T14:30:34.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:34.571+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.578787+0000) 2019-10-13T14:30:34.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:34.571+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.579165+0000) 2019-10-13T14:30:34.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:34.575+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.579652+0000) 2019-10-13T14:30:34.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:34.575+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.580538+0000) 2019-10-13T14:30:34.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:34.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.580611+0000) 2019-10-13T14:30:34.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:34.576+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.581167+0000) 2019-10-13T14:30:34.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:34.575+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.581980+0000) 2019-10-13T14:30:34.590 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:34.583+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:34.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:34.588+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.590479+0000) 2019-10-13T14:30:35.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:35.639+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:35.647036+0000) 2019-10-13T14:30:35.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:35.655+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:35.659945+0000) 2019-10-13T14:30:35.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:35.943+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:35.947436+0000) 2019-10-13T14:30:36.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:36.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.624622+0000) 2019-10-13T14:30:36.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:36.872+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.874735+0000) 2019-10-13T14:30:38.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:38.895+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:38.896586+0000) 2019-10-13T14:30:39.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:39.583+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.591247+0000) 2019-10-13T14:30:39.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:39.587+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.591831+0000) 2019-10-13T14:30:39.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:39.587+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.592676+0000) 2019-10-13T14:30:39.595 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:39.587+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.593056+0000) 2019-10-13T14:30:39.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:39.587+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.593591+0000) 2019-10-13T14:30:39.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:39.588+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.593766+0000) 2019-10-13T14:30:39.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:39.587+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.594827+0000) 2019-10-13T14:30:39.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:39.596+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:39.601103+0000) 2019-10-13T14:30:39.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:39.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:42.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:42.280+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:42.284619+0000) 2019-10-13T14:30:42.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:42.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:42.635638+0000) 2019-10-13T14:30:44.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:44.603+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.604879+0000) 2019-10-13T14:30:44.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:44.603+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.605716+0000) 2019-10-13T14:30:44.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:44.603+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.606011+0000) 2019-10-13T14:30:44.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:44.603+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.606363+0000) 2019-10-13T14:30:44.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:44.604+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.607841+0000) 2019-10-13T14:30:44.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:44.607+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.608222+0000) 2019-10-13T14:30:44.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:44.607+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.608923+0000) 2019-10-13T14:30:44.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:44.608+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.612016+0000) 2019-10-13T14:30:44.617 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:44.611+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:45.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:45.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:45.647265+0000) 2019-10-13T14:30:45.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:45.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:45.660151+0000) 2019-10-13T14:30:45.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:45.947+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:45.947707+0000) 2019-10-13T14:30:46.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:46.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.624846+0000) 2019-10-13T14:30:46.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:46.872+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.874950+0000) 2019-10-13T14:30:48.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:48.895+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:48.896821+0000) 2019-10-13T14:30:49.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:49.615+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.618536+0000) 2019-10-13T14:30:49.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:49.619+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.619502+0000) 2019-10-13T14:30:49.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:49.616+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.619956+0000) 2019-10-13T14:30:49.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:49.619+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.620312+0000) 2019-10-13T14:30:49.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:49.619+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.620929+0000) 2019-10-13T14:30:49.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:49.619+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.621305+0000) 2019-10-13T14:30:49.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:49.619+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.621614+0000) 2019-10-13T14:30:49.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:49.624+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:49.627105+0000) 2019-10-13T14:30:49.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:49.627+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:52.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:52.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:52.294546+0000) 2019-10-13T14:30:52.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:52.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:52.635872+0000) 2019-10-13T14:30:54.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:54.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.631744+0000) 2019-10-13T14:30:54.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:54.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.631943+0000) 2019-10-13T14:30:54.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:54.631+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.633673+0000) 2019-10-13T14:30:54.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:54.631+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.633963+0000) 2019-10-13T14:30:54.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:54.632+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.634502+0000) 2019-10-13T14:30:54.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:54.631+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.634596+0000) 2019-10-13T14:30:54.638 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:54.635+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.635960+0000) 2019-10-13T14:30:54.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:54.640+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.642229+0000) 2019-10-13T14:30:54.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:54.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:30:55.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:55.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:55.647508+0000) 2019-10-13T14:30:55.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:55.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:55.660350+0000) 2019-10-13T14:30:55.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:55.946+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:55.947906+0000) 2019-10-13T14:30:56.357 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:30:56.357 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:56.360 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:30:56.361 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:56.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:56.622+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.625037+0000) 2019-10-13T14:30:56.877 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:56.872+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.875109+0000) 2019-10-13T14:30:58.899 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:58.894+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:58.897047+0000) 2019-10-13T14:30:59.648 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:30:59.642+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.645695+0000) 2019-10-13T14:30:59.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:30:59.642+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.646172+0000) 2019-10-13T14:30:59.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:30:59.642+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.646563+0000) 2019-10-13T14:30:59.650 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:30:59.642+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.647224+0000) 2019-10-13T14:30:59.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:30:59.644+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.647782+0000) 2019-10-13T14:30:59.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:30:59.646+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.648227+0000) 2019-10-13T14:30:59.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:30:59.646+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.648561+0000) 2019-10-13T14:30:59.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:30:59.648+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:59.652363+0000) 2019-10-13T14:30:59.657 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:30:59.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:02.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:02.300+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:02.305149+0000) 2019-10-13T14:31:02.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:02.634+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:02.636109+0000) 2019-10-13T14:31:04.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:04.655+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.658819+0000) 2019-10-13T14:31:04.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:04.654+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.659014+0000) 2019-10-13T14:31:04.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:04.658+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.659803+0000) 2019-10-13T14:31:04.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:04.658+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.660381+0000) 2019-10-13T14:31:04.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:04.658+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.660918+0000) 2019-10-13T14:31:04.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:04.658+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.660421+0000) 2019-10-13T14:31:04.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:04.658+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.661661+0000) 2019-10-13T14:31:04.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:04.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:04.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:04.663+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.669162+0000) 2019-10-13T14:31:05.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:05.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:05.647695+0000) 2019-10-13T14:31:05.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:05.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:05.660557+0000) 2019-10-13T14:31:05.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:05.946+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:05.948106+0000) 2019-10-13T14:31:06.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:06.622+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.625265+0000) 2019-10-13T14:31:06.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:06.871+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.875306+0000) 2019-10-13T14:31:08.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:08.894+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:08.897248+0000) 2019-10-13T14:31:09.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:09.670+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.672166+0000) 2019-10-13T14:31:09.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:09.670+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.672709+0000) 2019-10-13T14:31:09.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:09.670+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.673131+0000) 2019-10-13T14:31:09.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:09.667+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.674019+0000) 2019-10-13T14:31:09.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:09.670+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.674304+0000) 2019-10-13T14:31:09.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:09.670+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.674547+0000) 2019-10-13T14:31:09.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:09.670+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.675151+0000) 2019-10-13T14:31:09.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:09.675+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:09.678249+0000) 2019-10-13T14:31:09.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:09.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:12.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:12.307+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:12.311523+0000) 2019-10-13T14:31:12.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:12.634+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:12.636348+0000) 2019-10-13T14:31:14.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:14.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.686036+0000) 2019-10-13T14:31:14.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:14.682+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.686238+0000) 2019-10-13T14:31:14.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:14.682+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.686737+0000) 2019-10-13T14:31:14.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:14.686+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.687455+0000) 2019-10-13T14:31:14.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:14.686+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.688011+0000) 2019-10-13T14:31:14.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:14.686+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.688114+0000) 2019-10-13T14:31:14.691 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:14.686+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.688486+0000) 2019-10-13T14:31:14.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:14.687+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.693654+0000) 2019-10-13T14:31:14.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:14.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 197978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:15.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:15.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:15.647879+0000) 2019-10-13T14:31:15.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:15.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:15.660772+0000) 2019-10-13T14:31:15.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:15.946+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:15.948313+0000) 2019-10-13T14:31:16.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:16.622+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.625461+0000) 2019-10-13T14:31:16.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:16.871+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.875476+0000) 2019-10-13T14:31:18.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:18.894+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:18.897448+0000) 2019-10-13T14:31:19.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:19.698+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.699355+0000) 2019-10-13T14:31:19.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:19.698+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.699974+0000) 2019-10-13T14:31:19.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:19.698+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.700323+0000) 2019-10-13T14:31:19.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:19.695+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.700890+0000) 2019-10-13T14:31:19.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:19.698+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.701389+0000) 2019-10-13T14:31:19.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:19.698+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.701724+0000) 2019-10-13T14:31:19.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:19.698+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.702143+0000) 2019-10-13T14:31:19.707 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:19.699+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:19.705423+0000) 2019-10-13T14:31:19.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:19.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:22.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:22.319+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:22.322680+0000) 2019-10-13T14:31:22.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:22.634+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:22.636599+0000) 2019-10-13T14:31:24.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:24.710+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.712354+0000) 2019-10-13T14:31:24.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:24.710+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.712778+0000) 2019-10-13T14:31:24.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:24.710+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.713326+0000) 2019-10-13T14:31:24.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:24.710+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.714060+0000) 2019-10-13T14:31:24.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:24.710+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.715114+0000) 2019-10-13T14:31:24.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:24.714+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.715571+0000) 2019-10-13T14:31:24.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:24.711+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.716205+0000) 2019-10-13T14:31:24.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:24.719+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.722663+0000) 2019-10-13T14:31:24.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:24.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:25.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:25.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:25.648076+0000) 2019-10-13T14:31:25.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:25.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:25.660946+0000) 2019-10-13T14:31:25.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:25.946+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:25.948528+0000) 2019-10-13T14:31:26.405 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:31:26.405 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:26.409 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:31:26.409 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:26.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:26.622+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.625712+0000) 2019-10-13T14:31:26.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:26.871+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.875671+0000) 2019-10-13T14:31:28.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:28.894+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:28.897605+0000) 2019-10-13T14:31:29.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:29.722+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.725528+0000) 2019-10-13T14:31:29.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:29.722+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.726854+0000) 2019-10-13T14:31:29.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:29.722+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.727039+0000) 2019-10-13T14:31:29.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:29.722+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.727266+0000) 2019-10-13T14:31:29.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:29.726+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.728005+0000) 2019-10-13T14:31:29.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:29.723+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.728476+0000) 2019-10-13T14:31:29.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:29.726+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.729275+0000) 2019-10-13T14:31:29.736 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:29.727+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:29.734014+0000) 2019-10-13T14:31:29.738 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:29.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:32.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:32.323+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:32.329006+0000) 2019-10-13T14:31:32.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:32.634+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:32.636828+0000) 2019-10-13T14:31:34.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:34.734+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.738944+0000) 2019-10-13T14:31:34.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:34.735+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.739526+0000) 2019-10-13T14:31:34.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:34.738+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.739917+0000) 2019-10-13T14:31:34.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:34.738+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.740908+0000) 2019-10-13T14:31:34.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:34.738+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.740732+0000) 2019-10-13T14:31:34.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:34.738+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.741499+0000) 2019-10-13T14:31:34.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:34.738+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.741508+0000) 2019-10-13T14:31:34.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:34.746+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:34.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:34.747+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.752064+0000) 2019-10-13T14:31:35.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:35.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:35.648249+0000) 2019-10-13T14:31:35.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:35.658+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:35.661069+0000) 2019-10-13T14:31:35.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:35.946+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:35.948727+0000) 2019-10-13T14:31:36.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:36.622+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.625905+0000) 2019-10-13T14:31:36.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:36.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.875897+0000) 2019-10-13T14:31:38.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:38.893+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:38.897834+0000) 2019-10-13T14:31:39.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:39.749+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.751823+0000) 2019-10-13T14:31:39.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:39.747+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.752792+0000) 2019-10-13T14:31:39.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:39.749+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.753752+0000) 2019-10-13T14:31:39.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:39.749+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.753534+0000) 2019-10-13T14:31:39.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:39.749+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.754279+0000) 2019-10-13T14:31:39.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:39.749+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.754589+0000) 2019-10-13T14:31:39.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:39.753+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.755477+0000) 2019-10-13T14:31:39.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:39.755+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:39.759503+0000) 2019-10-13T14:31:39.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:39.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:42.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:42.335+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:42.335430+0000) 2019-10-13T14:31:42.639 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:42.633+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:42.637038+0000) 2019-10-13T14:31:44.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:44.761+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.765748+0000) 2019-10-13T14:31:44.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:44.765+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.767496+0000) 2019-10-13T14:31:44.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:44.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.767979+0000) 2019-10-13T14:31:44.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:44.763+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.768389+0000) 2019-10-13T14:31:44.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:44.765+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.768633+0000) 2019-10-13T14:31:44.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:44.765+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.768895+0000) 2019-10-13T14:31:44.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:44.765+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.769306+0000) 2019-10-13T14:31:44.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:44.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.775276+0000) 2019-10-13T14:31:44.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:44.773+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:45.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:45.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:45.648428+0000) 2019-10-13T14:31:45.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:45.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:45.661234+0000) 2019-10-13T14:31:45.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:45.945+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:45.948891+0000) 2019-10-13T14:31:46.628 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:46.621+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.626095+0000) 2019-10-13T14:31:46.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:46.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.876091+0000) 2019-10-13T14:31:48.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:48.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:48.898035+0000) 2019-10-13T14:31:49.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:49.777+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.779626+0000) 2019-10-13T14:31:49.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:49.777+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.780254+0000) 2019-10-13T14:31:49.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:49.777+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.781069+0000) 2019-10-13T14:31:49.783 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:49.775+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.781514+0000) 2019-10-13T14:31:49.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:49.777+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.781734+0000) 2019-10-13T14:31:49.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:49.777+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.782095+0000) 2019-10-13T14:31:49.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:49.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.782946+0000) 2019-10-13T14:31:49.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:49.783+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:49.789492+0000) 2019-10-13T14:31:49.792 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:49.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198221 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:52.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:52.339+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:52.341755+0000) 2019-10-13T14:31:52.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:52.633+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:52.637283+0000) 2019-10-13T14:31:54.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:54.789+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.792763+0000) 2019-10-13T14:31:54.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:54.789+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.793319+0000) 2019-10-13T14:31:54.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:54.789+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.793872+0000) 2019-10-13T14:31:54.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:54.789+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.794536+0000) 2019-10-13T14:31:54.797 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:54.791+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.795287+0000) 2019-10-13T14:31:54.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:54.793+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.795734+0000) 2019-10-13T14:31:54.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:54.793+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.796303+0000) 2019-10-13T14:31:54.804 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:54.795+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.802044+0000) 2019-10-13T14:31:54.805 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:54.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:55.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:55.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:55.648631+0000) 2019-10-13T14:31:55.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:55.657+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:55.661409+0000) 2019-10-13T14:31:55.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:55.945+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:55.949129+0000) 2019-10-13T14:31:56.528 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:31:56.529 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:56.532 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:31:56.532 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:56.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:56.621+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.626291+0000) 2019-10-13T14:31:56.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:56.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.876318+0000) 2019-10-13T14:31:58.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:58.893+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:58.898232+0000) 2019-10-13T14:31:59.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:31:59.805+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.807579+0000) 2019-10-13T14:31:59.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:31:59.805+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.807902+0000) 2019-10-13T14:31:59.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:31:59.805+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.808391+0000) 2019-10-13T14:31:59.811 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:31:59.805+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.808732+0000) 2019-10-13T14:31:59.812 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:31:59.802+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.809772+0000) 2019-10-13T14:31:59.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:31:59.805+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.810530+0000) 2019-10-13T14:31:59.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:31:59.805+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.810708+0000) 2019-10-13T14:31:59.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:31:59.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:31:59.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:31:59.810+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:59.817292+0000) 2019-10-13T14:32:02.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:02.346+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:02.348361+0000) 2019-10-13T14:32:02.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:02.633+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:02.637485+0000) 2019-10-13T14:32:04.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:04.817+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.820314+0000) 2019-10-13T14:32:04.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:04.817+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.821184+0000) 2019-10-13T14:32:04.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:04.817+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.820902+0000) 2019-10-13T14:32:04.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:04.817+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.821862+0000) 2019-10-13T14:32:04.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:04.817+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.822362+0000) 2019-10-13T14:32:04.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:04.818+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.823012+0000) 2019-10-13T14:32:04.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:04.821+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.823837+0000) 2019-10-13T14:32:04.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:04.822+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.829646+0000) 2019-10-13T14:32:04.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:04.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:05.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:05.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:05.648810+0000) 2019-10-13T14:32:05.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:05.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:05.661584+0000) 2019-10-13T14:32:05.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:05.945+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:05.949377+0000) 2019-10-13T14:32:06.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:06.621+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.626517+0000) 2019-10-13T14:32:06.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:06.874+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.876521+0000) 2019-10-13T14:32:08.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:08.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:08.898464+0000) 2019-10-13T14:32:09.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:09.829+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.833697+0000) 2019-10-13T14:32:09.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:09.829+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.834272+0000) 2019-10-13T14:32:09.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:09.829+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.834824+0000) 2019-10-13T14:32:09.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:09.829+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.835100+0000) 2019-10-13T14:32:09.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:09.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.835535+0000) 2019-10-13T14:32:09.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:09.833+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.835979+0000) 2019-10-13T14:32:09.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:09.830+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.836927+0000) 2019-10-13T14:32:09.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:09.838+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:09.842850+0000) 2019-10-13T14:32:09.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:09.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:12.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:12.350+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:12.354709+0000) 2019-10-13T14:32:12.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:12.633+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:12.637680+0000) 2019-10-13T14:32:14.850 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:14.845+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.847672+0000) 2019-10-13T14:32:14.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:14.845+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.848221+0000) 2019-10-13T14:32:14.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:14.845+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.848505+0000) 2019-10-13T14:32:14.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:14.845+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.849088+0000) 2019-10-13T14:32:14.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:14.845+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.849607+0000) 2019-10-13T14:32:14.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:14.842+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.850102+0000) 2019-10-13T14:32:14.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:14.845+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.851110+0000) 2019-10-13T14:32:14.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:14.854+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.858267+0000) 2019-10-13T14:32:14.861 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:14.853+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:15.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:15.645+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:15.648986+0000) 2019-10-13T14:32:15.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:15.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:15.661768+0000) 2019-10-13T14:32:15.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:15.949+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:15.949581+0000) 2019-10-13T14:32:16.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:16.621+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.626710+0000) 2019-10-13T14:32:16.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:16.874+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.876731+0000) 2019-10-13T14:32:18.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:18.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:18.898717+0000) 2019-10-13T14:32:19.864 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:19.857+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.861089+0000) 2019-10-13T14:32:19.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:19.857+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.861623+0000) 2019-10-13T14:32:19.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:19.857+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.862378+0000) 2019-10-13T14:32:19.865 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:19.857+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.862672+0000) 2019-10-13T14:32:19.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:19.857+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.863276+0000) 2019-10-13T14:32:19.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:19.858+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.865019+0000) 2019-10-13T14:32:19.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:19.861+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.865507+0000) 2019-10-13T14:32:19.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:19.866+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:19.871212+0000) 2019-10-13T14:32:19.874 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:19.869+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:22.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:22.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:22.361072+0000) 2019-10-13T14:32:22.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:22.636+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:22.637884+0000) 2019-10-13T14:32:24.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:24.868+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.874576+0000) 2019-10-13T14:32:24.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:24.872+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.876167+0000) 2019-10-13T14:32:24.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:24.872+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.876561+0000) 2019-10-13T14:32:24.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:24.872+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.877336+0000) 2019-10-13T14:32:24.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:24.872+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.877563+0000) 2019-10-13T14:32:24.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:24.870+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.878027+0000) 2019-10-13T14:32:24.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:24.872+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.878071+0000) 2019-10-13T14:32:24.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:24.878+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.883794+0000) 2019-10-13T14:32:24.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:24.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:25.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:25.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:25.649165+0000) 2019-10-13T14:32:25.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:25.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:25.661983+0000) 2019-10-13T14:32:25.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:25.944+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:25.949784+0000) 2019-10-13T14:32:26.577 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:32:26.577 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:26.583 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:32:26.583 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:26.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:26.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.626914+0000) 2019-10-13T14:32:26.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:26.874+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.876947+0000) 2019-10-13T14:32:28.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:28.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:28.898911+0000) 2019-10-13T14:32:29.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:29.884+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.887777+0000) 2019-10-13T14:32:29.891 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:29.884+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.888374+0000) 2019-10-13T14:32:29.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:29.884+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.888930+0000) 2019-10-13T14:32:29.892 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:29.884+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.890186+0000) 2019-10-13T14:32:29.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:29.886+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.890541+0000) 2019-10-13T14:32:29.893 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:29.884+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.890762+0000) 2019-10-13T14:32:29.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:29.888+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.891519+0000) 2019-10-13T14:32:29.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:29.892+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:29.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:29.890+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:29.897826+0000) 2019-10-13T14:32:32.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:32.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:32.367382+0000) 2019-10-13T14:32:32.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:32.632+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:32.638086+0000) 2019-10-13T14:32:34.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:34.896+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.900654+0000) 2019-10-13T14:32:34.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:34.896+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.901277+0000) 2019-10-13T14:32:34.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:34.896+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.901681+0000) 2019-10-13T14:32:34.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:34.898+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.902211+0000) 2019-10-13T14:32:34.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:34.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.902538+0000) 2019-10-13T14:32:34.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:34.896+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.903157+0000) 2019-10-13T14:32:34.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:34.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.904090+0000) 2019-10-13T14:32:34.911 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:34.902+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.908674+0000) 2019-10-13T14:32:34.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:34.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:35.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:35.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:35.649373+0000) 2019-10-13T14:32:35.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:35.656+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:35.662159+0000) 2019-10-13T14:32:35.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:35.948+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:35.950002+0000) 2019-10-13T14:32:36.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:36.620+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.627107+0000) 2019-10-13T14:32:36.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:36.870+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.877174+0000) 2019-10-13T14:32:38.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:38.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:38.899131+0000) 2019-10-13T14:32:39.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:39.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.914757+0000) 2019-10-13T14:32:39.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:39.908+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.915127+0000) 2019-10-13T14:32:39.918 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:39.912+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.915592+0000) 2019-10-13T14:32:39.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:39.912+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.916240+0000) 2019-10-13T14:32:39.919 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:39.910+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.916897+0000) 2019-10-13T14:32:39.919 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:39.912+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.917191+0000) 2019-10-13T14:32:39.920 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:39.912+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.917640+0000) 2019-10-13T14:32:39.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:39.918+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:39.924308+0000) 2019-10-13T14:32:39.927 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:39.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:42.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:42.370+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:42.373812+0000) 2019-10-13T14:32:42.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:42.632+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:42.638313+0000) 2019-10-13T14:32:44.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:44.924+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.928648+0000) 2019-10-13T14:32:44.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:44.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.929099+0000) 2019-10-13T14:32:44.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:44.924+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.929535+0000) 2019-10-13T14:32:44.933 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:44.924+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.929893+0000) 2019-10-13T14:32:44.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:44.924+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.930228+0000) 2019-10-13T14:32:44.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:44.924+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.930498+0000) 2019-10-13T14:32:44.935 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:44.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.931204+0000) 2019-10-13T14:32:44.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:44.930+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.938078+0000) 2019-10-13T14:32:44.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:44.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:45.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:45.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:45.649555+0000) 2019-10-13T14:32:45.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:45.656+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:45.662339+0000) 2019-10-13T14:32:45.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:45.948+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:45.950215+0000) 2019-10-13T14:32:46.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:46.624+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.627296+0000) 2019-10-13T14:32:46.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:46.874+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.877370+0000) 2019-10-13T14:32:48.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:48.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:48.899362+0000) 2019-10-13T14:32:49.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:49.936+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.941369+0000) 2019-10-13T14:32:49.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:49.936+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.941889+0000) 2019-10-13T14:32:49.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:49.936+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.942409+0000) 2019-10-13T14:32:49.945 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:49.942+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.943037+0000) 2019-10-13T14:32:49.946 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:49.940+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.943543+0000) 2019-10-13T14:32:49.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:49.940+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.944186+0000) 2019-10-13T14:32:49.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:49.940+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.944737+0000) 2019-10-13T14:32:49.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:49.950+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:49.950677+0000) 2019-10-13T14:32:49.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:49.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:52.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:52.378+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:52.380346+0000) 2019-10-13T14:32:52.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:52.636+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:52.638530+0000) 2019-10-13T14:32:54.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:54.952+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.955891+0000) 2019-10-13T14:32:54.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:54.952+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.956180+0000) 2019-10-13T14:32:54.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:54.953+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.957192+0000) 2019-10-13T14:32:54.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:54.952+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.956635+0000) 2019-10-13T14:32:54.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:54.952+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.957676+0000) 2019-10-13T14:32:54.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:54.952+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.958042+0000) 2019-10-13T14:32:54.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:54.952+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.958336+0000) 2019-10-13T14:32:54.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:54.961+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.963965+0000) 2019-10-13T14:32:54.968 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:54.960+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:32:55.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:55.644+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:55.649753+0000) 2019-10-13T14:32:55.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:55.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:55.662511+0000) 2019-10-13T14:32:55.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:55.944+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:55.950415+0000) 2019-10-13T14:32:56.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:56.624+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.627543+0000) 2019-10-13T14:32:56.636 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:32:56.636 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:56.640 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:32:56.640 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:56.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:56.873+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.877568+0000) 2019-10-13T14:32:58.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:58.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:58.899523+0000) 2019-10-13T14:32:59.971 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:32:59.964+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.968582+0000) 2019-10-13T14:32:59.972 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:32:59.964+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.969629+0000) 2019-10-13T14:32:59.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:32:59.964+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.969962+0000) 2019-10-13T14:32:59.973 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:32:59.964+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.970343+0000) 2019-10-13T14:32:59.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:32:59.964+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.971210+0000) 2019-10-13T14:32:59.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:32:59.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.971753+0000) 2019-10-13T14:32:59.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:32:59.968+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.972094+0000) 2019-10-13T14:32:59.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:32:59.977+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:59.978643+0000) 2019-10-13T14:32:59.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:32:59.972+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:02.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:02.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.386768+0000) 2019-10-13T14:33:02.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:02.632+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.638733+0000) 2019-10-13T14:33:04.985 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:04.975+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.982667+0000) 2019-10-13T14:33:04.985 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:04.975+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.982913+0000) 2019-10-13T14:33:04.986 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:04.979+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.983746+0000) 2019-10-13T14:33:04.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:04.979+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.984502+0000) 2019-10-13T14:33:04.987 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:04.979+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.984820+0000) 2019-10-13T14:33:04.988 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:04.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.985911+0000) 2019-10-13T14:33:04.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:04.979+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.986059+0000) 2019-10-13T14:33:04.994 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:04.989+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.991863+0000) 2019-10-13T14:33:04.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:04.987+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:05.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:05.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:05.649928+0000) 2019-10-13T14:33:05.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:05.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:05.662684+0000) 2019-10-13T14:33:05.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:05.947+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:05.950600+0000) 2019-10-13T14:33:06.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:06.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:06.627737+0000) 2019-10-13T14:33:06.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:06.873+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:06.877778+0000) 2019-10-13T14:33:08.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:08.895+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:08.899687+0000) 2019-10-13T14:33:09.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:09.991+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.997178+0000) 2019-10-13T14:33:10.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:09.991+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.997629+0000) 2019-10-13T14:33:10.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:09.991+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.998014+0000) 2019-10-13T14:33:10.001 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:09.991+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.998434+0000) 2019-10-13T14:33:10.001 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:09.991+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.998668+0000) 2019-10-13T14:33:10.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:09.991+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.999156+0000) 2019-10-13T14:33:10.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:09.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:09.999709+0000) 2019-10-13T14:33:10.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:10.001+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:10.005550+0000) 2019-10-13T14:33:10.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:09.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:12.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:12.389+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.393265+0000) 2019-10-13T14:33:12.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:12.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.638941+0000) 2019-10-13T14:33:15.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:15.003+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.009619+0000) 2019-10-13T14:33:15.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:15.003+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.010463+0000) 2019-10-13T14:33:15.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:15.003+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.011262+0000) 2019-10-13T14:33:15.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:15.007+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.011776+0000) 2019-10-13T14:33:15.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:15.007+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.012042+0000) 2019-10-13T14:33:15.015 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:15.007+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.012381+0000) 2019-10-13T14:33:15.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:15.009+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.013156+0000) 2019-10-13T14:33:15.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:15.013+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.017255+0000) 2019-10-13T14:33:15.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:15.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:15.652 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:15.643+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.650105+0000) 2019-10-13T14:33:15.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:15.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.662855+0000) 2019-10-13T14:33:15.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:15.947+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:15.950824+0000) 2019-10-13T14:33:16.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:16.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:16.627931+0000) 2019-10-13T14:33:16.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:16.873+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:16.877986+0000) 2019-10-13T14:33:18.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:18.899+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:18.899910+0000) 2019-10-13T14:33:20.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:20.015+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.022630+0000) 2019-10-13T14:33:20.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:20.021+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.023013+0000) 2019-10-13T14:33:20.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:20.019+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.023752+0000) 2019-10-13T14:33:20.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:20.019+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.023974+0000) 2019-10-13T14:33:20.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:20.019+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.024248+0000) 2019-10-13T14:33:20.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:20.019+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.024853+0000) 2019-10-13T14:33:20.028 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:20.019+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.025780+0000) 2019-10-13T14:33:20.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:20.029+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.031967+0000) 2019-10-13T14:33:20.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:20.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:22.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:22.397+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.399688+0000) 2019-10-13T14:33:22.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:22.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.639178+0000) 2019-10-13T14:33:25.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:25.031+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.035815+0000) 2019-10-13T14:33:25.039 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:25.031+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.036596+0000) 2019-10-13T14:33:25.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:25.031+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.036800+0000) 2019-10-13T14:33:25.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:25.031+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.037146+0000) 2019-10-13T14:33:25.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:25.037+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.038340+0000) 2019-10-13T14:33:25.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:25.031+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.038652+0000) 2019-10-13T14:33:25.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:25.031+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.038900+0000) 2019-10-13T14:33:25.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:25.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:25.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:25.045+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.048032+0000) 2019-10-13T14:33:25.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:25.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.650277+0000) 2019-10-13T14:33:25.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:25.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.663031+0000) 2019-10-13T14:33:25.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:25.943+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:25.951038+0000) 2019-10-13T14:33:26.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:26.623+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:26.628162+0000) 2019-10-13T14:33:26.740 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:33:26.740 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:26.744 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:33:26.744 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:26.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:26.877+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:26.878171+0000) 2019-10-13T14:33:28.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:28.899+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:28.900113+0000) 2019-10-13T14:33:30.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:30.043+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.048730+0000) 2019-10-13T14:33:30.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:30.043+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.049334+0000) 2019-10-13T14:33:30.052 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:30.043+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.049841+0000) 2019-10-13T14:33:30.052 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:30.049+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.050371+0000) 2019-10-13T14:33:30.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:30.043+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.050600+0000) 2019-10-13T14:33:30.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:30.047+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.051376+0000) 2019-10-13T14:33:30.054 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:30.043+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.051271+0000) 2019-10-13T14:33:30.061 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:30.051+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:30.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:30.053+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.058108+0000) 2019-10-13T14:33:32.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:32.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.406081+0000) 2019-10-13T14:33:32.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:32.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.639377+0000) 2019-10-13T14:33:35.064 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:35.059+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.061311+0000) 2019-10-13T14:33:35.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:35.059+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.061818+0000) 2019-10-13T14:33:35.065 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:35.059+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.062443+0000) 2019-10-13T14:33:35.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:35.061+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.063928+0000) 2019-10-13T14:33:35.066 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:35.063+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.064012+0000) 2019-10-13T14:33:35.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:35.063+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.064617+0000) 2019-10-13T14:33:35.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:35.063+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.064997+0000) 2019-10-13T14:33:35.072 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:35.065+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.069849+0000) 2019-10-13T14:33:35.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:35.071+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:35.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:35.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.650488+0000) 2019-10-13T14:33:35.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:35.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.663207+0000) 2019-10-13T14:33:35.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:35.947+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:35.951213+0000) 2019-10-13T14:33:36.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:36.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:36.628347+0000) 2019-10-13T14:33:36.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:36.877+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:36.878371+0000) 2019-10-13T14:33:38.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:38.899+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:38.900337+0000) 2019-10-13T14:33:40.078 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:40.075+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.075350+0000) 2019-10-13T14:33:40.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:40.075+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.075960+0000) 2019-10-13T14:33:40.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:40.075+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.076840+0000) 2019-10-13T14:33:40.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:40.075+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.077405+0000) 2019-10-13T14:33:40.080 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:40.073+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.078066+0000) 2019-10-13T14:33:40.080 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:40.075+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.078201+0000) 2019-10-13T14:33:40.081 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:40.075+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.078546+0000) 2019-10-13T14:33:40.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:40.083+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 198972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:40.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:40.085+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.086904+0000) 2019-10-13T14:33:42.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:42.409+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.412357+0000) 2019-10-13T14:33:42.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:42.639+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.639544+0000) 2019-10-13T14:33:45.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:45.087+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.088908+0000) 2019-10-13T14:33:45.092 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:45.087+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.089216+0000) 2019-10-13T14:33:45.093 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:45.087+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.090231+0000) 2019-10-13T14:33:45.093 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:45.087+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.090457+0000) 2019-10-13T14:33:45.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:45.089+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.091052+0000) 2019-10-13T14:33:45.094 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:45.091+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.091416+0000) 2019-10-13T14:33:45.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:45.091+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.092188+0000) 2019-10-13T14:33:45.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:45.097+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.098693+0000) 2019-10-13T14:33:45.102 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:45.099+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199005 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:45.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:45.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.650720+0000) 2019-10-13T14:33:45.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:45.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.663409+0000) 2019-10-13T14:33:45.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:45.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:45.951348+0000) 2019-10-13T14:33:46.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:46.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:46.628539+0000) 2019-10-13T14:33:46.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:46.877+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:46.878560+0000) 2019-10-13T14:33:48.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:48.898+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:48.900533+0000) 2019-10-13T14:33:50.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:50.100+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.102965+0000) 2019-10-13T14:33:50.106 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:50.102+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.103564+0000) 2019-10-13T14:33:50.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:50.102+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.103564+0000) 2019-10-13T14:33:50.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:50.102+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.104470+0000) 2019-10-13T14:33:50.108 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:50.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.105044+0000) 2019-10-13T14:33:50.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:50.102+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.105510+0000) 2019-10-13T14:33:50.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:50.102+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.105838+0000) 2019-10-13T14:33:50.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:50.108+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.110488+0000) 2019-10-13T14:33:50.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:50.106+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:52.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:52.416+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.418719+0000) 2019-10-13T14:33:52.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:52.638+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.639789+0000) 2019-10-13T14:33:55.117 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:33:55.110+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.114437+0000) 2019-10-13T14:33:55.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:55.114+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.115417+0000) 2019-10-13T14:33:55.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:55.112+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.115863+0000) 2019-10-13T14:33:55.118 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:55.114+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.115948+0000) 2019-10-13T14:33:55.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:55.114+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.116558+0000) 2019-10-13T14:33:55.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:55.114+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.117054+0000) 2019-10-13T14:33:55.120 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:55.114+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.117400+0000) 2019-10-13T14:33:55.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:33:55.120+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.124365+0000) 2019-10-13T14:33:55.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:33:55.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:33:55.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:33:55.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.650903+0000) 2019-10-13T14:33:55.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:33:55.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.663581+0000) 2019-10-13T14:33:55.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:33:55.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:55.951518+0000) 2019-10-13T14:33:56.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:33:56.626+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:56.628733+0000) 2019-10-13T14:33:56.788 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:33:56.788 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:56.792 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:33:56.792 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:56.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:33:56.876+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:56.878754+0000) 2019-10-13T14:33:58.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:33:58.898+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:58.900731+0000) 2019-10-13T14:34:00.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:00.126+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.129094+0000) 2019-10-13T14:34:00.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:00.126+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.129420+0000) 2019-10-13T14:34:00.132 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:00.126+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.130022+0000) 2019-10-13T14:34:00.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:00.126+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.130785+0000) 2019-10-13T14:34:00.134 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:00.130+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.131689+0000) 2019-10-13T14:34:00.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:00.128+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.132877+0000) 2019-10-13T14:34:00.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:00.130+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.132887+0000) 2019-10-13T14:34:00.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:00.132+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.136867+0000) 2019-10-13T14:34:00.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:00.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:02.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:02.420+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.425037+0000) 2019-10-13T14:34:02.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:02.638+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.639988+0000) 2019-10-13T14:34:05.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:05.138+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.142542+0000) 2019-10-13T14:34:05.145 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:05.140+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.143382+0000) 2019-10-13T14:34:05.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:05.142+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.143764+0000) 2019-10-13T14:34:05.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:05.142+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.144434+0000) 2019-10-13T14:34:05.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:05.142+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.144243+0000) 2019-10-13T14:34:05.147 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:05.142+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.145430+0000) 2019-10-13T14:34:05.148 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:05.142+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.145952+0000) 2019-10-13T14:34:05.155 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:05.150+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199139 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:05.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:05.152+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.154305+0000) 2019-10-13T14:34:05.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:05.646+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.651083+0000) 2019-10-13T14:34:05.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:05.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.663791+0000) 2019-10-13T14:34:05.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:05.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:05.951700+0000) 2019-10-13T14:34:06.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:06.626+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:06.628929+0000) 2019-10-13T14:34:06.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:06.876+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:06.878953+0000) 2019-10-13T14:34:08.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:08.898+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:08.900930+0000) 2019-10-13T14:34:10.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:10.152+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.155902+0000) 2019-10-13T14:34:10.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:10.154+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.156431+0000) 2019-10-13T14:34:10.159 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:10.154+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.156537+0000) 2019-10-13T14:34:10.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:10.154+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.156923+0000) 2019-10-13T14:34:10.160 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:10.154+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.157439+0000) 2019-10-13T14:34:10.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:10.154+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.157710+0000) 2019-10-13T14:34:10.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:10.154+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.158053+0000) 2019-10-13T14:34:10.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:10.160+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.164917+0000) 2019-10-13T14:34:10.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:10.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:12.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:12.428+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.430755+0000) 2019-10-13T14:34:12.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:12.638+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.640221+0000) 2019-10-13T14:34:15.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:15.166+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.169519+0000) 2019-10-13T14:34:15.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:15.166+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.170093+0000) 2019-10-13T14:34:15.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:15.166+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.170504+0000) 2019-10-13T14:34:15.173 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:15.168+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.171368+0000) 2019-10-13T14:34:15.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:15.170+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.171614+0000) 2019-10-13T14:34:15.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:15.170+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.171768+0000) 2019-10-13T14:34:15.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:15.170+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.172264+0000) 2019-10-13T14:34:15.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:15.172+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.177474+0000) 2019-10-13T14:34:15.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:15.174+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:15.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:15.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.651262+0000) 2019-10-13T14:34:15.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:15.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.663971+0000) 2019-10-13T14:34:15.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:15.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:15.951872+0000) 2019-10-13T14:34:16.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:16.626+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:16.629122+0000) 2019-10-13T14:34:16.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:16.876+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:16.879160+0000) 2019-10-13T14:34:18.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:18.898+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:18.901126+0000) 2019-10-13T14:34:20.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:20.178+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.181936+0000) 2019-10-13T14:34:20.185 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:20.178+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.182363+0000) 2019-10-13T14:34:20.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:20.178+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.182626+0000) 2019-10-13T14:34:20.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:20.182+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.183436+0000) 2019-10-13T14:34:20.186 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:20.180+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.184269+0000) 2019-10-13T14:34:20.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:20.182+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.184402+0000) 2019-10-13T14:34:20.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:20.182+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.184895+0000) 2019-10-13T14:34:20.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:20.188+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.191258+0000) 2019-10-13T14:34:20.194 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:20.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:22.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:22.436+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.439872+0000) 2019-10-13T14:34:22.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:22.638+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.640417+0000) 2019-10-13T14:34:25.199 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:25.194+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.196328+0000) 2019-10-13T14:34:25.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:25.194+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.197259+0000) 2019-10-13T14:34:25.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:25.194+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.196970+0000) 2019-10-13T14:34:25.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:25.194+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.197649+0000) 2019-10-13T14:34:25.200 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:25.194+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.197922+0000) 2019-10-13T14:34:25.201 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:25.194+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.198246+0000) 2019-10-13T14:34:25.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:25.196+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.198897+0000) 2019-10-13T14:34:25.208 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:25.202+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:25.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:25.204+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.207383+0000) 2019-10-13T14:34:25.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:25.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.651467+0000) 2019-10-13T14:34:25.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:25.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.664116+0000) 2019-10-13T14:34:25.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:25.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:25.952046+0000) 2019-10-13T14:34:26.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:26.626+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:26.629329+0000) 2019-10-13T14:34:26.881 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:26.876+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:26.879358+0000) 2019-10-13T14:34:26.895 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:34:26.895 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:26.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:34:26.899 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:28.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:28.898+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:28.901327+0000) 2019-10-13T14:34:30.212 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:30.206+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.209257+0000) 2019-10-13T14:34:30.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:30.206+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.209950+0000) 2019-10-13T14:34:30.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:30.206+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.210330+0000) 2019-10-13T14:34:30.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:30.208+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.211093+0000) 2019-10-13T14:34:30.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:30.210+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.211534+0000) 2019-10-13T14:34:30.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:30.210+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.211699+0000) 2019-10-13T14:34:30.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:30.210+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.212035+0000) 2019-10-13T14:34:30.222 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:30.218+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:30.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:30.216+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.220291+0000) 2019-10-13T14:34:32.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:32.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.446429+0000) 2019-10-13T14:34:32.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:32.637+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.640630+0000) 2019-10-13T14:34:35.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:35.217+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.222321+0000) 2019-10-13T14:34:35.226 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:35.221+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.223497+0000) 2019-10-13T14:34:35.226 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:35.221+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.223908+0000) 2019-10-13T14:34:35.227 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:35.220+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.224883+0000) 2019-10-13T14:34:35.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:35.221+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.225141+0000) 2019-10-13T14:34:35.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:35.221+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.225791+0000) 2019-10-13T14:34:35.228 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:35.221+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.226474+0000) 2019-10-13T14:34:35.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:35.228+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.232544+0000) 2019-10-13T14:34:35.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:35.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:35.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:35.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.651611+0000) 2019-10-13T14:34:35.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:35.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.664321+0000) 2019-10-13T14:34:35.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:35.949+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:35.952256+0000) 2019-10-13T14:34:36.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:36.625+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:36.629525+0000) 2019-10-13T14:34:36.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:36.876+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:36.879552+0000) 2019-10-13T14:34:38.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:38.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:38.901587+0000) 2019-10-13T14:34:40.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:40.232+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.235853+0000) 2019-10-13T14:34:40.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:40.233+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.236113+0000) 2019-10-13T14:34:40.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:40.233+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.237763+0000) 2019-10-13T14:34:40.240 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:40.233+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.238235+0000) 2019-10-13T14:34:40.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:40.233+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.238756+0000) 2019-10-13T14:34:40.241 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:40.233+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.239219+0000) 2019-10-13T14:34:40.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:40.237+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.240098+0000) 2019-10-13T14:34:40.248 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:40.241+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199372 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:40.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:40.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.246750+0000) 2019-10-13T14:34:42.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:42.451+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.452830+0000) 2019-10-13T14:34:42.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:42.637+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.640832+0000) 2019-10-13T14:34:45.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:45.245+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.249226+0000) 2019-10-13T14:34:45.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:45.245+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.249991+0000) 2019-10-13T14:34:45.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:45.245+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.250204+0000) 2019-10-13T14:34:45.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:45.245+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.250602+0000) 2019-10-13T14:34:45.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:45.247+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.251235+0000) 2019-10-13T14:34:45.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:45.249+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.252182+0000) 2019-10-13T14:34:45.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:45.249+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.252456+0000) 2019-10-13T14:34:45.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:45.251+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.257757+0000) 2019-10-13T14:34:45.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:45.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:45.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:45.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.651820+0000) 2019-10-13T14:34:45.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:45.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.664456+0000) 2019-10-13T14:34:45.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:45.949+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:45.952433+0000) 2019-10-13T14:34:46.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:46.625+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:46.629716+0000) 2019-10-13T14:34:46.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:46.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:46.879749+0000) 2019-10-13T14:34:48.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:48.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:48.901834+0000) 2019-10-13T14:34:50.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:50.257+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.262403+0000) 2019-10-13T14:34:50.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:50.257+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.263033+0000) 2019-10-13T14:34:50.266 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:50.261+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.263893+0000) 2019-10-13T14:34:50.267 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:50.259+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.264882+0000) 2019-10-13T14:34:50.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:50.261+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.264934+0000) 2019-10-13T14:34:50.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:50.261+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.265868+0000) 2019-10-13T14:34:50.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:50.261+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.266375+0000) 2019-10-13T14:34:50.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:50.267+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.271581+0000) 2019-10-13T14:34:50.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:50.269+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199440 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:52.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:52.455+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.459336+0000) 2019-10-13T14:34:52.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:52.637+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.640992+0000) 2019-10-13T14:34:55.279 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:55.271+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.276789+0000) 2019-10-13T14:34:55.280 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:55.273+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.277063+0000) 2019-10-13T14:34:55.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:55.273+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.277408+0000) 2019-10-13T14:34:55.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:55.273+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.278167+0000) 2019-10-13T14:34:55.281 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:55.273+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.278257+0000) 2019-10-13T14:34:55.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:55.277+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.279642+0000) 2019-10-13T14:34:55.283 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:34:55.277+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.280582+0000) 2019-10-13T14:34:55.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:34:55.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:34:55.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:34:55.283+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.287197+0000) 2019-10-13T14:34:55.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:34:55.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.651996+0000) 2019-10-13T14:34:55.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:34:55.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.664624+0000) 2019-10-13T14:34:55.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:34:55.949+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:55.952606+0000) 2019-10-13T14:34:56.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:34:56.625+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:56.629909+0000) 2019-10-13T14:34:56.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:34:56.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:56.879941+0000) 2019-10-13T14:34:57.007 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:34:57.008 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:57.011 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:34:57.011 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:58.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:34:58.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:58.902080+0000) 2019-10-13T14:35:00.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:00.287+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.290786+0000) 2019-10-13T14:35:00.294 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:00.289+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.291520+0000) 2019-10-13T14:35:00.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:00.289+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.292017+0000) 2019-10-13T14:35:00.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:00.289+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.292830+0000) 2019-10-13T14:35:00.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:00.289+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.293158+0000) 2019-10-13T14:35:00.296 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:00.289+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.293763+0000) 2019-10-13T14:35:00.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:00.289+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.294251+0000) 2019-10-13T14:35:00.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:00.295+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.299560+0000) 2019-10-13T14:35:00.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:00.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:02.468 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:02.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.465938+0000) 2019-10-13T14:35:02.643 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:02.637+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.641141+0000) 2019-10-13T14:35:05.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:05.301+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.304569+0000) 2019-10-13T14:35:05.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:05.301+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.304879+0000) 2019-10-13T14:35:05.308 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:05.301+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.305559+0000) 2019-10-13T14:35:05.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:05.301+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.306396+0000) 2019-10-13T14:35:05.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:05.303+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.306717+0000) 2019-10-13T14:35:05.309 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:05.301+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.306709+0000) 2019-10-13T14:35:05.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:05.301+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.307199+0000) 2019-10-13T14:35:05.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:05.307+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.313456+0000) 2019-10-13T14:35:05.317 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:05.309+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:05.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:05.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.652173+0000) 2019-10-13T14:35:05.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:05.661+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.664809+0000) 2019-10-13T14:35:05.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:05.949+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:05.952787+0000) 2019-10-13T14:35:06.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:06.625+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:06.630115+0000) 2019-10-13T14:35:06.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:06.879+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:06.880135+0000) 2019-10-13T14:35:08.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:08.897+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:08.902309+0000) 2019-10-13T14:35:10.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:10.313+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.318192+0000) 2019-10-13T14:35:10.321 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:10.313+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.318280+0000) 2019-10-13T14:35:10.321 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:10.315+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.319413+0000) 2019-10-13T14:35:10.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:10.317+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.319578+0000) 2019-10-13T14:35:10.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:10.317+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.319826+0000) 2019-10-13T14:35:10.322 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:10.317+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.320134+0000) 2019-10-13T14:35:10.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:10.317+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.320406+0000) 2019-10-13T14:35:10.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:10.323+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.326681+0000) 2019-10-13T14:35:10.330 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:10.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:12.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:12.471+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.472282+0000) 2019-10-13T14:35:12.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:12.641+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.641345+0000) 2019-10-13T14:35:15.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:15.324+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.330351+0000) 2019-10-13T14:35:15.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:15.328+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.332224+0000) 2019-10-13T14:35:15.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:15.328+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.332541+0000) 2019-10-13T14:35:15.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:15.328+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.332724+0000) 2019-10-13T14:35:15.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:15.327+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.333943+0000) 2019-10-13T14:35:15.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:15.328+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.333516+0000) 2019-10-13T14:35:15.337 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:15.328+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.334494+0000) 2019-10-13T14:35:15.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:15.335+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.341291+0000) 2019-10-13T14:35:15.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:15.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199606 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:15.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:15.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.652409+0000) 2019-10-13T14:35:15.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:15.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.664980+0000) 2019-10-13T14:35:15.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:15.948+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:15.952967+0000) 2019-10-13T14:35:16.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:16.628+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:16.630316+0000) 2019-10-13T14:35:16.882 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:16.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:16.880329+0000) 2019-10-13T14:35:18.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:18.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:18.902523+0000) 2019-10-13T14:35:20.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:20.340+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.344579+0000) 2019-10-13T14:35:20.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:20.340+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.345391+0000) 2019-10-13T14:35:20.348 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:20.340+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.345462+0000) 2019-10-13T14:35:20.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:20.340+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.345777+0000) 2019-10-13T14:35:20.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:20.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.346294+0000) 2019-10-13T14:35:20.349 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:20.343+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.347042+0000) 2019-10-13T14:35:20.350 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:20.344+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.347660+0000) 2019-10-13T14:35:20.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:20.351+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.354668+0000) 2019-10-13T14:35:20.357 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:20.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:22.479 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:22.475+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.476896+0000) 2019-10-13T14:35:22.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:22.640+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.641582+0000) 2019-10-13T14:35:25.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:25.352+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.358712+0000) 2019-10-13T14:35:25.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:25.355+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.360028+0000) 2019-10-13T14:35:25.362 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:25.356+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.360330+0000) 2019-10-13T14:35:25.363 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:25.356+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.360609+0000) 2019-10-13T14:35:25.363 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:25.356+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.360901+0000) 2019-10-13T14:35:25.364 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:25.356+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.361233+0000) 2019-10-13T14:35:25.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:25.356+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.361908+0000) 2019-10-13T14:35:25.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:25.363+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.369109+0000) 2019-10-13T14:35:25.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:25.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:25.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:25.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.652585+0000) 2019-10-13T14:35:25.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:25.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.665156+0000) 2019-10-13T14:35:25.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:25.948+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:25.953167+0000) 2019-10-13T14:35:26.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:26.624+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:26.630539+0000) 2019-10-13T14:35:26.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:26.879+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:26.880520+0000) 2019-10-13T14:35:27.134 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:35:27.134 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:27.138 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:35:27.138 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:28.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:28.900+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:28.902749+0000) 2019-10-13T14:35:30.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:30.368+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.372023+0000) 2019-10-13T14:35:30.375 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:30.368+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.373015+0000) 2019-10-13T14:35:30.375 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:30.368+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.373428+0000) 2019-10-13T14:35:30.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:30.367+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.373966+0000) 2019-10-13T14:35:30.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:30.368+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.374251+0000) 2019-10-13T14:35:30.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:30.368+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.374723+0000) 2019-10-13T14:35:30.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:30.368+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.375032+0000) 2019-10-13T14:35:30.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:30.375+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.381249+0000) 2019-10-13T14:35:30.385 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:30.376+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:32.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:32.483+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.483326+0000) 2019-10-13T14:35:32.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:32.640+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.641785+0000) 2019-10-13T14:35:35.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:35.380+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.385463+0000) 2019-10-13T14:35:35.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:35.380+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.386158+0000) 2019-10-13T14:35:35.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:35.380+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.386917+0000) 2019-10-13T14:35:35.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:35.383+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.387857+0000) 2019-10-13T14:35:35.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:35.384+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.388073+0000) 2019-10-13T14:35:35.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:35.384+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.388520+0000) 2019-10-13T14:35:35.391 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:35.384+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.388930+0000) 2019-10-13T14:35:35.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:35.391+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.396268+0000) 2019-10-13T14:35:35.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:35.392+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199738 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:35.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:35.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.652795+0000) 2019-10-13T14:35:35.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:35.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.665350+0000) 2019-10-13T14:35:35.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:35.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:35.953350+0000) 2019-10-13T14:35:36.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:36.628+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:36.630726+0000) 2019-10-13T14:35:36.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:36.875+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:36.880687+0000) 2019-10-13T14:35:38.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:38.900+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:38.902977+0000) 2019-10-13T14:35:40.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:40.396+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.399603+0000) 2019-10-13T14:35:40.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:40.396+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.400086+0000) 2019-10-13T14:35:40.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:40.396+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.400958+0000) 2019-10-13T14:35:40.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:40.396+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.401478+0000) 2019-10-13T14:35:40.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:40.398+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.402287+0000) 2019-10-13T14:35:40.405 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:40.396+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.402614+0000) 2019-10-13T14:35:40.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:40.396+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.403174+0000) 2019-10-13T14:35:40.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:40.404+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:40.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:40.406+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.413114+0000) 2019-10-13T14:35:42.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:42.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.489623+0000) 2019-10-13T14:35:42.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:42.636+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.642006+0000) 2019-10-13T14:35:45.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:45.408+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.412486+0000) 2019-10-13T14:35:45.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:45.408+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.412928+0000) 2019-10-13T14:35:45.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:45.410+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.414580+0000) 2019-10-13T14:35:45.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:45.408+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.414601+0000) 2019-10-13T14:35:45.417 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:45.408+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.414697+0000) 2019-10-13T14:35:45.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:45.408+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.415239+0000) 2019-10-13T14:35:45.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:45.412+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.416486+0000) 2019-10-13T14:35:45.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:45.414+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.421982+0000) 2019-10-13T14:35:45.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:45.416+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:45.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:45.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.652999+0000) 2019-10-13T14:35:45.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:45.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.665530+0000) 2019-10-13T14:35:45.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:45.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:45.953537+0000) 2019-10-13T14:35:46.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:46.624+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:46.630932+0000) 2019-10-13T14:35:46.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:46.874+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:46.880878+0000) 2019-10-13T14:35:48.905 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:48.896+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:48.903175+0000) 2019-10-13T14:35:50.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:50.420+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.426244+0000) 2019-10-13T14:35:50.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:50.420+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.426618+0000) 2019-10-13T14:35:50.429 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:50.424+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.427332+0000) 2019-10-13T14:35:50.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:50.422+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.427758+0000) 2019-10-13T14:35:50.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:50.424+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.428240+0000) 2019-10-13T14:35:50.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:50.424+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.428585+0000) 2019-10-13T14:35:50.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:50.424+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.429932+0000) 2019-10-13T14:35:50.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:50.430+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.436655+0000) 2019-10-13T14:35:50.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:50.432+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:52.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:52.490+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.494596+0000) 2019-10-13T14:35:52.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:52.636+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.642209+0000) 2019-10-13T14:35:55.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:55.434+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.440011+0000) 2019-10-13T14:35:55.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:35:55.436+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.440813+0000) 2019-10-13T14:35:55.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:55.436+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.441367+0000) 2019-10-13T14:35:55.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:55.436+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.441781+0000) 2019-10-13T14:35:55.444 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:55.436+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.441982+0000) 2019-10-13T14:35:55.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:55.436+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.442398+0000) 2019-10-13T14:35:55.445 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:55.436+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.443001+0000) 2019-10-13T14:35:55.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:35:55.446+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.450384+0000) 2019-10-13T14:35:55.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:35:55.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:35:55.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:35:55.648+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.653208+0000) 2019-10-13T14:35:55.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:35:55.660+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.665705+0000) 2019-10-13T14:35:55.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:35:55.948+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:55.953744+0000) 2019-10-13T14:35:56.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:35:56.628+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:56.631128+0000) 2019-10-13T14:35:56.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:35:56.878+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:56.881126+0000) 2019-10-13T14:35:57.263 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:35:57.264 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:57.267 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:35:57.267 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:58.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:35:58.899+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:58.903373+0000) 2019-10-13T14:36:00.457 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:00.450+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.454347+0000) 2019-10-13T14:36:00.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:00.447+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.455080+0000) 2019-10-13T14:36:00.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:00.451+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.455597+0000) 2019-10-13T14:36:00.458 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:00.451+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.456108+0000) 2019-10-13T14:36:00.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:00.451+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.456273+0000) 2019-10-13T14:36:00.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:00.451+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.456563+0000) 2019-10-13T14:36:00.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:00.451+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.457100+0000) 2019-10-13T14:36:00.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:00.458+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.464445+0000) 2019-10-13T14:36:00.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:00.459+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:02.503 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:02.494+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.500867+0000) 2019-10-13T14:36:02.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:02.635+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.642407+0000) 2019-10-13T14:36:05.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:05.463+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.468232+0000) 2019-10-13T14:36:05.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:05.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.468460+0000) 2019-10-13T14:36:05.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:05.462+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.469596+0000) 2019-10-13T14:36:05.472 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:05.463+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.469075+0000) 2019-10-13T14:36:05.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:05.463+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.470123+0000) 2019-10-13T14:36:05.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:05.463+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.470397+0000) 2019-10-13T14:36:05.473 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:05.463+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.471244+0000) 2019-10-13T14:36:05.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:05.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:05.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:05.474+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.479540+0000) 2019-10-13T14:36:05.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:05.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.653421+0000) 2019-10-13T14:36:05.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:05.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.665912+0000) 2019-10-13T14:36:05.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:05.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:05.953933+0000) 2019-10-13T14:36:06.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:06.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:06.631344+0000) 2019-10-13T14:36:06.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:06.878+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:06.881340+0000) 2019-10-13T14:36:08.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:08.899+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:08.903543+0000) 2019-10-13T14:36:10.486 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:10.475+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.482709+0000) 2019-10-13T14:36:10.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:10.475+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.482596+0000) 2019-10-13T14:36:10.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:10.479+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.483545+0000) 2019-10-13T14:36:10.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:10.478+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.483494+0000) 2019-10-13T14:36:10.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:10.479+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.483881+0000) 2019-10-13T14:36:10.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:10.479+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.484091+0000) 2019-10-13T14:36:10.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:10.479+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.484377+0000) 2019-10-13T14:36:10.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:10.486+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.492336+0000) 2019-10-13T14:36:10.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:10.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 199968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:12.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:12.506+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.507299+0000) 2019-10-13T14:36:12.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:12.639+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.642613+0000) 2019-10-13T14:36:15.498 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:15.491+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.495798+0000) 2019-10-13T14:36:15.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:15.491+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.496379+0000) 2019-10-13T14:36:15.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:15.491+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.496683+0000) 2019-10-13T14:36:15.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:15.491+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.496985+0000) 2019-10-13T14:36:15.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:15.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.497943+0000) 2019-10-13T14:36:15.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:15.494+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.498347+0000) 2019-10-13T14:36:15.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:15.491+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.498497+0000) 2019-10-13T14:36:15.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:15.499+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:15.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:15.502+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.506588+0000) 2019-10-13T14:36:15.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:15.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.653600+0000) 2019-10-13T14:36:15.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:15.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.666117+0000) 2019-10-13T14:36:15.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:15.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:15.954116+0000) 2019-10-13T14:36:16.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:16.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:16.631541+0000) 2019-10-13T14:36:16.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:16.878+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:16.881564+0000) 2019-10-13T14:36:18.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:18.903+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:18.903737+0000) 2019-10-13T14:36:20.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:20.503+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.508925+0000) 2019-10-13T14:36:20.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:20.503+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.509343+0000) 2019-10-13T14:36:20.512 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:20.503+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.510208+0000) 2019-10-13T14:36:20.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:20.506+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.511607+0000) 2019-10-13T14:36:20.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:20.507+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.511492+0000) 2019-10-13T14:36:20.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:20.507+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.511814+0000) 2019-10-13T14:36:20.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:20.507+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.512401+0000) 2019-10-13T14:36:20.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:20.514+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.518582+0000) 2019-10-13T14:36:20.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:20.515+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:22.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:22.510+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.513733+0000) 2019-10-13T14:36:22.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:22.639+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.642815+0000) 2019-10-13T14:36:25.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:25.515+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.523031+0000) 2019-10-13T14:36:25.526 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:25.519+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.523617+0000) 2019-10-13T14:36:25.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:25.522+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.524523+0000) 2019-10-13T14:36:25.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:25.519+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.524731+0000) 2019-10-13T14:36:25.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:25.519+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.525175+0000) 2019-10-13T14:36:25.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:25.519+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.525569+0000) 2019-10-13T14:36:25.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:25.519+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.525905+0000) 2019-10-13T14:36:25.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:25.530+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.530227+0000) 2019-10-13T14:36:25.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:25.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:25.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:25.647+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.653761+0000) 2019-10-13T14:36:25.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:25.659+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.666294+0000) 2019-10-13T14:36:25.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:25.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:25.954301+0000) 2019-10-13T14:36:26.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:26.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:26.631738+0000) 2019-10-13T14:36:26.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:26.878+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:26.881773+0000) 2019-10-13T14:36:27.375 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:36:27.375 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:27.379 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:36:27.379 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:28.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:28.903+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:28.903922+0000) 2019-10-13T14:36:30.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:30.535+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.535954+0000) 2019-10-13T14:36:30.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:30.535+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.536404+0000) 2019-10-13T14:36:30.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:30.535+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.537273+0000) 2019-10-13T14:36:30.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:30.538+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.538236+0000) 2019-10-13T14:36:30.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:30.535+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.538261+0000) 2019-10-13T14:36:30.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:30.535+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.538607+0000) 2019-10-13T14:36:30.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:30.535+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.539173+0000) 2019-10-13T14:36:30.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:30.542+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.545123+0000) 2019-10-13T14:36:30.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:30.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:32.528 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:32.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.525288+0000) 2019-10-13T14:36:32.645 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:32.639+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.643021+0000) 2019-10-13T14:36:35.552 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:35.547+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.550123+0000) 2019-10-13T14:36:35.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:35.547+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.550858+0000) 2019-10-13T14:36:35.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:35.547+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.551009+0000) 2019-10-13T14:36:35.554 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:35.549+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.551493+0000) 2019-10-13T14:36:35.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:35.551+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.552216+0000) 2019-10-13T14:36:35.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:35.551+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.552778+0000) 2019-10-13T14:36:35.555 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:35.551+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.553366+0000) 2019-10-13T14:36:35.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:35.557+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.558890+0000) 2019-10-13T14:36:35.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:35.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:35.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:35.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.653960+0000) 2019-10-13T14:36:35.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:35.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.666468+0000) 2019-10-13T14:36:35.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:35.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:35.954487+0000) 2019-10-13T14:36:36.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:36.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:36.631967+0000) 2019-10-13T14:36:36.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:36.881+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:36.881988+0000) 2019-10-13T14:36:38.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:38.903+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:38.904110+0000) 2019-10-13T14:36:40.567 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:40.563+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.564290+0000) 2019-10-13T14:36:40.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:40.563+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.564763+0000) 2019-10-13T14:36:40.568 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:40.561+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.565367+0000) 2019-10-13T14:36:40.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:40.563+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.566412+0000) 2019-10-13T14:36:40.570 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:40.563+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.567030+0000) 2019-10-13T14:36:40.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:40.563+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.567048+0000) 2019-10-13T14:36:40.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:40.563+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.567282+0000) 2019-10-13T14:36:40.580 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:40.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:40.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:40.569+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.571995+0000) 2019-10-13T14:36:42.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:42.533+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.536155+0000) 2019-10-13T14:36:42.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:42.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.643250+0000) 2019-10-13T14:36:45.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:45.577+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.578243+0000) 2019-10-13T14:36:45.581 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:45.574+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.578599+0000) 2019-10-13T14:36:45.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:45.574+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.579248+0000) 2019-10-13T14:36:45.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:45.578+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.579791+0000) 2019-10-13T14:36:45.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:45.578+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.580457+0000) 2019-10-13T14:36:45.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:45.578+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.581087+0000) 2019-10-13T14:36:45.584 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:45.578+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.582495+0000) 2019-10-13T14:36:45.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:45.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:45.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:45.589+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.591230+0000) 2019-10-13T14:36:45.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:45.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.654156+0000) 2019-10-13T14:36:45.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:45.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.666667+0000) 2019-10-13T14:36:45.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:45.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:45.954695+0000) 2019-10-13T14:36:46.634 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:46.630+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:46.632141+0000) 2019-10-13T14:36:46.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:46.881+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:46.882228+0000) 2019-10-13T14:36:48.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:48.902+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:48.904308+0000) 2019-10-13T14:36:50.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:50.590+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.592228+0000) 2019-10-13T14:36:50.595 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:50.589+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.593145+0000) 2019-10-13T14:36:50.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:50.590+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.593701+0000) 2019-10-13T14:36:50.596 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:50.590+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.594125+0000) 2019-10-13T14:36:50.597 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:50.590+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.594483+0000) 2019-10-13T14:36:50.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:50.594+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.595754+0000) 2019-10-13T14:36:50.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:50.594+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.596091+0000) 2019-10-13T14:36:50.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:50.597+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.601566+0000) 2019-10-13T14:36:50.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:50.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200232 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:52.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:52.541+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.542525+0000) 2019-10-13T14:36:52.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:52.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.643477+0000) 2019-10-13T14:36:55.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:55.602+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.605592+0000) 2019-10-13T14:36:55.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:55.606+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.607374+0000) 2019-10-13T14:36:55.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:55.606+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.607900+0000) 2019-10-13T14:36:55.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:55.606+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.608349+0000) 2019-10-13T14:36:55.611 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:55.605+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.608828+0000) 2019-10-13T14:36:55.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:36:55.606+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.608916+0000) 2019-10-13T14:36:55.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:55.606+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.609224+0000) 2019-10-13T14:36:55.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:36:55.609+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.613957+0000) 2019-10-13T14:36:55.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:36:55.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200265 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:36:55.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:36:55.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.654332+0000) 2019-10-13T14:36:55.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:36:55.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.666841+0000) 2019-10-13T14:36:55.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:36:55.954+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:55.954879+0000) 2019-10-13T14:36:56.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:36:56.630+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:56.632329+0000) 2019-10-13T14:36:56.884 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:36:56.881+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:56.882422+0000) 2019-10-13T14:36:57.454 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:36:57.454 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:57.458 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:36:57.458 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:58.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:36:58.902+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:58.904512+0000) 2019-10-13T14:37:00.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:00.618+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.620350+0000) 2019-10-13T14:37:00.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:00.618+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.621063+0000) 2019-10-13T14:37:00.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:00.618+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.621559+0000) 2019-10-13T14:37:00.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:00.618+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.622035+0000) 2019-10-13T14:37:00.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:00.621+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.622530+0000) 2019-10-13T14:37:00.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:00.618+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.622571+0000) 2019-10-13T14:37:00.625 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:00.618+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.622893+0000) 2019-10-13T14:37:00.630 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:00.625+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.627669+0000) 2019-10-13T14:37:00.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:00.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200297 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:02.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:02.545+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.549184+0000) 2019-10-13T14:37:02.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:02.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.643707+0000) 2019-10-13T14:37:05.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:05.630+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.632901+0000) 2019-10-13T14:37:05.636 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:05.630+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.633550+0000) 2019-10-13T14:37:05.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:05.630+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.633653+0000) 2019-10-13T14:37:05.636 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:05.630+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.634076+0000) 2019-10-13T14:37:05.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:05.630+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.634981+0000) 2019-10-13T14:37:05.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:05.633+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.635974+0000) 2019-10-13T14:37:05.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:05.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.636117+0000) 2019-10-13T14:37:05.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:05.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:05.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:05.641+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.643561+0000) 2019-10-13T14:37:05.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:05.650+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.654480+0000) 2019-10-13T14:37:05.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:05.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.666961+0000) 2019-10-13T14:37:05.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:05.950+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:05.955017+0000) 2019-10-13T14:37:06.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:06.630+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:06.632555+0000) 2019-10-13T14:37:06.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:06.881+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:06.882630+0000) 2019-10-13T14:37:08.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:08.902+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:08.904709+0000) 2019-10-13T14:37:10.648 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:10.642+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.645689+0000) 2019-10-13T14:37:10.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:10.642+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.646615+0000) 2019-10-13T14:37:10.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:10.646+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.647443+0000) 2019-10-13T14:37:10.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:10.646+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.647631+0000) 2019-10-13T14:37:10.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:10.646+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.648070+0000) 2019-10-13T14:37:10.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:10.646+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.648668+0000) 2019-10-13T14:37:10.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:10.645+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.649086+0000) 2019-10-13T14:37:10.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:10.653+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.655810+0000) 2019-10-13T14:37:10.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:10.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200364 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:12.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:12.553+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.555506+0000) 2019-10-13T14:37:12.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:12.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.643899+0000) 2019-10-13T14:37:15.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:15.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.654635+0000) 2019-10-13T14:37:15.663 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:15.658+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.661133+0000) 2019-10-13T14:37:15.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:15.658+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.661538+0000) 2019-10-13T14:37:15.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:15.658+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.662030+0000) 2019-10-13T14:37:15.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:15.658+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.662417+0000) 2019-10-13T14:37:15.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:15.658+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.662637+0000) 2019-10-13T14:37:15.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:15.658+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.663051+0000) 2019-10-13T14:37:15.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:15.661+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.664144+0000) 2019-10-13T14:37:15.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:15.662+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.667086+0000) 2019-10-13T14:37:15.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:15.665+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.669764+0000) 2019-10-13T14:37:15.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:15.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:15.957 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:15.954+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:15.955202+0000) 2019-10-13T14:37:16.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:16.630+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:16.632753+0000) 2019-10-13T14:37:16.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:16.881+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:16.882843+0000) 2019-10-13T14:37:18.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:18.902+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:18.904910+0000) 2019-10-13T14:37:20.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:20.670+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.673867+0000) 2019-10-13T14:37:20.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:20.670+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.674841+0000) 2019-10-13T14:37:20.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:20.670+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.675174+0000) 2019-10-13T14:37:20.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:20.674+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.676038+0000) 2019-10-13T14:37:20.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:20.674+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.676406+0000) 2019-10-13T14:37:20.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:20.673+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.677192+0000) 2019-10-13T14:37:20.680 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:20.674+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.677292+0000) 2019-10-13T14:37:20.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:20.681+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.682862+0000) 2019-10-13T14:37:20.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:20.682+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200429 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:22.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:22.557+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.562009+0000) 2019-10-13T14:37:22.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:22.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.644078+0000) 2019-10-13T14:37:25.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:25.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.654860+0000) 2019-10-13T14:37:25.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:25.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.667338+0000) 2019-10-13T14:37:25.689 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:25.681+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.687178+0000) 2019-10-13T14:37:25.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:25.685+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.687718+0000) 2019-10-13T14:37:25.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:25.685+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.688348+0000) 2019-10-13T14:37:25.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:25.685+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.689225+0000) 2019-10-13T14:37:25.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:25.685+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.689952+0000) 2019-10-13T14:37:25.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:25.685+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.690220+0000) 2019-10-13T14:37:25.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:25.685+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.690614+0000) 2019-10-13T14:37:25.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:25.693+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.696130+0000) 2019-10-13T14:37:25.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:25.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:25.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:25.953+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:25.955426+0000) 2019-10-13T14:37:26.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:26.629+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:26.632967+0000) 2019-10-13T14:37:26.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:26.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:26.883040+0000) 2019-10-13T14:37:27.556 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:37:27.557 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:27.560 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:37:27.561 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:28.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:28.901+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:28.905064+0000) 2019-10-13T14:37:30.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:30.697+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.700723+0000) 2019-10-13T14:37:30.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:30.696+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.701264+0000) 2019-10-13T14:37:30.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:30.697+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.701956+0000) 2019-10-13T14:37:30.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:30.697+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.701990+0000) 2019-10-13T14:37:30.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:30.697+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.702636+0000) 2019-10-13T14:37:30.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:30.697+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.702932+0000) 2019-10-13T14:37:30.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:30.701+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.704077+0000) 2019-10-13T14:37:30.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:30.708+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.710592+0000) 2019-10-13T14:37:30.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:30.709+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200496 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:32.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:32.564+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.568287+0000) 2019-10-13T14:37:32.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:32.641+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.644308+0000) 2019-10-13T14:37:35.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:35.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.655054+0000) 2019-10-13T14:37:35.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:35.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.667542+0000) 2019-10-13T14:37:35.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:35.709+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.714982+0000) 2019-10-13T14:37:35.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:35.713+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.715389+0000) 2019-10-13T14:37:35.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:35.713+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.715951+0000) 2019-10-13T14:37:35.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:35.712+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.716597+0000) 2019-10-13T14:37:35.719 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:35.713+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.716764+0000) 2019-10-13T14:37:35.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:35.713+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.717244+0000) 2019-10-13T14:37:35.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:35.713+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.718068+0000) 2019-10-13T14:37:35.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:35.720+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.724420+0000) 2019-10-13T14:37:35.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:35.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:35.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:35.953+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:35.955646+0000) 2019-10-13T14:37:36.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:36.629+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:36.633093+0000) 2019-10-13T14:37:36.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:36.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:36.883230+0000) 2019-10-13T14:37:38.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:38.901+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:38.905291+0000) 2019-10-13T14:37:40.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:40.725+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.728455+0000) 2019-10-13T14:37:40.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:40.725+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.729132+0000) 2019-10-13T14:37:40.732 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:40.725+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.729618+0000) 2019-10-13T14:37:40.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:40.725+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.730096+0000) 2019-10-13T14:37:40.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:40.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.730827+0000) 2019-10-13T14:37:40.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:40.725+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.731127+0000) 2019-10-13T14:37:40.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:40.729+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.731569+0000) 2019-10-13T14:37:40.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:40.732+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.735642+0000) 2019-10-13T14:37:40.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:40.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200561 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:42.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:42.572+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.574658+0000) 2019-10-13T14:37:42.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:42.641+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.644536+0000) 2019-10-13T14:37:45.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:45.649+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.655254+0000) 2019-10-13T14:37:45.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:45.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.667738+0000) 2019-10-13T14:37:45.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:45.737+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.742703+0000) 2019-10-13T14:37:45.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:45.737+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.743213+0000) 2019-10-13T14:37:45.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:45.741+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.743685+0000) 2019-10-13T14:37:45.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:45.740+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.744397+0000) 2019-10-13T14:37:45.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:45.741+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.744300+0000) 2019-10-13T14:37:45.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:45.741+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.744652+0000) 2019-10-13T14:37:45.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:45.741+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.745449+0000) 2019-10-13T14:37:45.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:45.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200595 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:45.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:45.748+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.751786+0000) 2019-10-13T14:37:45.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:45.953+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:45.955833+0000) 2019-10-13T14:37:46.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:46.629+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:46.633225+0000) 2019-10-13T14:37:46.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:46.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:46.883467+0000) 2019-10-13T14:37:48.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:48.901+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:48.905511+0000) 2019-10-13T14:37:50.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:50.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.755738+0000) 2019-10-13T14:37:50.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:50.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.756349+0000) 2019-10-13T14:37:50.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:50.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.756573+0000) 2019-10-13T14:37:50.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:50.753+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.757046+0000) 2019-10-13T14:37:50.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:50.753+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.757485+0000) 2019-10-13T14:37:50.761 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:50.756+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.758659+0000) 2019-10-13T14:37:50.761 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:50.753+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.758932+0000) 2019-10-13T14:37:50.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:50.760+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.764822+0000) 2019-10-13T14:37:50.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:50.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200628 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:52.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:52.580+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.580964+0000) 2019-10-13T14:37:52.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:52.641+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.644737+0000) 2019-10-13T14:37:55.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:55.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.655518+0000) 2019-10-13T14:37:55.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:55.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.667953+0000) 2019-10-13T14:37:55.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:37:55.765+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.769330+0000) 2019-10-13T14:37:55.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:55.765+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.769876+0000) 2019-10-13T14:37:55.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:37:55.765+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.770122+0000) 2019-10-13T14:37:55.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:37:55.765+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.770481+0000) 2019-10-13T14:37:55.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:55.765+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.770779+0000) 2019-10-13T14:37:55.775 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:55.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.771476+0000) 2019-10-13T14:37:55.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:55.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.772062+0000) 2019-10-13T14:37:55.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:37:55.776+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.778373+0000) 2019-10-13T14:37:55.782 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:37:55.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200660 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:37:55.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:37:55.953+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:55.956035+0000) 2019-10-13T14:37:56.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:37:56.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:56.633398+0000) 2019-10-13T14:37:56.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:37:56.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:56.883683+0000) 2019-10-13T14:37:57.626 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:37:57.626 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:57.630 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:37:57.630 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:58.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:37:58.901+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:58.905730+0000) 2019-10-13T14:38:00.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:00.777+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.782845+0000) 2019-10-13T14:38:00.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:00.781+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.783491+0000) 2019-10-13T14:38:00.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:00.781+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.784227+0000) 2019-10-13T14:38:00.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:00.781+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.785085+0000) 2019-10-13T14:38:00.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:00.781+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.785673+0000) 2019-10-13T14:38:00.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:00.781+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.785934+0000) 2019-10-13T14:38:00.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:00.784+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.786819+0000) 2019-10-13T14:38:00.796 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:00.789+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:00.797 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:00.792+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.795122+0000) 2019-10-13T14:38:02.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:02.584+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.587368+0000) 2019-10-13T14:38:02.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:02.641+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.644928+0000) 2019-10-13T14:38:05.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:05.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.655771+0000) 2019-10-13T14:38:05.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:05.665+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.668153+0000) 2019-10-13T14:38:05.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:05.793+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.797624+0000) 2019-10-13T14:38:05.801 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:05.796+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.798241+0000) 2019-10-13T14:38:05.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:05.793+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.798506+0000) 2019-10-13T14:38:05.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:05.793+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.799149+0000) 2019-10-13T14:38:05.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:05.797+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.799468+0000) 2019-10-13T14:38:05.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:05.797+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.799814+0000) 2019-10-13T14:38:05.803 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:05.797+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.800854+0000) 2019-10-13T14:38:05.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:05.805+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:05.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:05.804+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.809563+0000) 2019-10-13T14:38:05.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:05.953+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:05.956257+0000) 2019-10-13T14:38:06.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:06.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:06.633596+0000) 2019-10-13T14:38:06.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:06.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:06.883878+0000) 2019-10-13T14:38:08.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:08.900+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:08.905927+0000) 2019-10-13T14:38:10.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:10.804+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.811233+0000) 2019-10-13T14:38:10.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:10.808+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.811862+0000) 2019-10-13T14:38:10.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:10.808+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.812683+0000) 2019-10-13T14:38:10.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:10.808+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.813023+0000) 2019-10-13T14:38:10.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:10.808+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.813773+0000) 2019-10-13T14:38:10.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:10.808+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.813789+0000) 2019-10-13T14:38:10.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:10.808+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.814783+0000) 2019-10-13T14:38:10.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:10.816+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.821594+0000) 2019-10-13T14:38:10.824 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:10.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:12.596 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:12.592+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.593895+0000) 2019-10-13T14:38:12.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:12.644+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.645131+0000) 2019-10-13T14:38:15.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:15.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.655980+0000) 2019-10-13T14:38:15.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:15.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.668351+0000) 2019-10-13T14:38:15.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:15.820+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.824027+0000) 2019-10-13T14:38:15.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:15.820+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.824811+0000) 2019-10-13T14:38:15.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:15.820+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.826103+0000) 2019-10-13T14:38:15.829 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:15.820+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.826408+0000) 2019-10-13T14:38:15.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:15.820+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.826974+0000) 2019-10-13T14:38:15.830 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:15.824+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.827565+0000) 2019-10-13T14:38:15.830 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:15.824+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.828139+0000) 2019-10-13T14:38:15.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:15.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:15.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:15.832+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.835993+0000) 2019-10-13T14:38:15.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:15.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:15.956479+0000) 2019-10-13T14:38:16.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:16.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:16.633772+0000) 2019-10-13T14:38:16.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:16.880+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:16.884085+0000) 2019-10-13T14:38:18.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:18.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:18.906157+0000) 2019-10-13T14:38:20.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:20.832+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.838288+0000) 2019-10-13T14:38:20.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:20.832+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.839005+0000) 2019-10-13T14:38:20.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:20.836+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.839346+0000) 2019-10-13T14:38:20.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:20.836+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.839666+0000) 2019-10-13T14:38:20.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:20.836+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.840064+0000) 2019-10-13T14:38:20.843 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:20.835+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.841193+0000) 2019-10-13T14:38:20.844 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:20.836+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.841782+0000) 2019-10-13T14:38:20.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:20.839+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.845279+0000) 2019-10-13T14:38:20.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:20.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:22.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:22.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.600335+0000) 2019-10-13T14:38:22.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:22.640+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.645333+0000) 2019-10-13T14:38:25.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:25.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.656186+0000) 2019-10-13T14:38:25.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:25.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.668581+0000) 2019-10-13T14:38:25.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:25.844+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.850742+0000) 2019-10-13T14:38:25.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:25.848+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.851837+0000) 2019-10-13T14:38:25.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:25.848+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.851469+0000) 2019-10-13T14:38:25.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:25.847+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.852700+0000) 2019-10-13T14:38:25.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:25.848+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.852886+0000) 2019-10-13T14:38:25.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:25.848+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.853412+0000) 2019-10-13T14:38:25.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:25.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.854903+0000) 2019-10-13T14:38:25.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:25.855+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.860671+0000) 2019-10-13T14:38:25.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:25.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:25.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:25.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:25.956617+0000) 2019-10-13T14:38:26.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:26.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:26.633977+0000) 2019-10-13T14:38:26.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:26.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:26.884275+0000) 2019-10-13T14:38:27.756 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:38:27.757 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:27.760 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:38:27.760 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:28.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:28.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:28.906352+0000) 2019-10-13T14:38:30.866 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:30.860+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.863870+0000) 2019-10-13T14:38:30.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:30.860+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.866021+0000) 2019-10-13T14:38:30.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:30.860+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.866296+0000) 2019-10-13T14:38:30.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:30.860+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.866639+0000) 2019-10-13T14:38:30.869 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:30.860+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.867217+0000) 2019-10-13T14:38:30.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:30.864+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.867394+0000) 2019-10-13T14:38:30.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:30.863+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.868024+0000) 2019-10-13T14:38:30.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:30.867+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.872093+0000) 2019-10-13T14:38:30.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:30.868+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:32.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:32.603+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.606637+0000) 2019-10-13T14:38:32.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:32.640+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.645569+0000) 2019-10-13T14:38:35.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:35.652+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.656392+0000) 2019-10-13T14:38:35.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:35.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.668766+0000) 2019-10-13T14:38:35.880 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:35.872+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.877846+0000) 2019-10-13T14:38:35.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:35.872+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.878334+0000) 2019-10-13T14:38:35.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:35.876+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.879400+0000) 2019-10-13T14:38:35.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:35.876+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.879616+0000) 2019-10-13T14:38:35.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:35.876+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.880221+0000) 2019-10-13T14:38:35.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:35.875+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.880736+0000) 2019-10-13T14:38:35.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:35.876+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.881070+0000) 2019-10-13T14:38:35.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:35.883+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.886568+0000) 2019-10-13T14:38:35.890 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:35.884+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:35.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:35.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:35.956853+0000) 2019-10-13T14:38:36.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:36.628+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:36.634143+0000) 2019-10-13T14:38:36.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:36.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:36.884466+0000) 2019-10-13T14:38:38.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:38.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:38.906550+0000) 2019-10-13T14:38:40.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:40.888+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.891667+0000) 2019-10-13T14:38:40.895 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:40.888+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.892184+0000) 2019-10-13T14:38:40.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:40.887+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.893068+0000) 2019-10-13T14:38:40.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:40.888+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.892554+0000) 2019-10-13T14:38:40.896 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:40.888+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.893283+0000) 2019-10-13T14:38:40.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:40.888+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.893881+0000) 2019-10-13T14:38:40.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:40.888+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.894480+0000) 2019-10-13T14:38:40.902 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:40.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.900482+0000) 2019-10-13T14:38:40.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:40.896+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200956 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:42.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:42.611+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.612979+0000) 2019-10-13T14:38:42.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:42.644+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.645765+0000) 2019-10-13T14:38:45.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:45.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.656589+0000) 2019-10-13T14:38:45.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:45.664+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.668918+0000) 2019-10-13T14:38:45.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:45.900+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.904808+0000) 2019-10-13T14:38:45.908 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:45.899+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.905361+0000) 2019-10-13T14:38:45.908 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:45.900+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.905701+0000) 2019-10-13T14:38:45.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:45.900+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.905917+0000) 2019-10-13T14:38:45.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:45.900+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.906118+0000) 2019-10-13T14:38:45.909 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:45.900+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.907284+0000) 2019-10-13T14:38:45.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:45.904+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.908141+0000) 2019-10-13T14:38:45.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:45.907+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.913486+0000) 2019-10-13T14:38:45.916 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:45.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 200991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:45.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:45.952+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:45.957041+0000) 2019-10-13T14:38:46.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:46.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:46.634318+0000) 2019-10-13T14:38:46.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:46.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:46.884655+0000) 2019-10-13T14:38:48.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:48.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:48.906795+0000) 2019-10-13T14:38:50.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:50.912+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.917261+0000) 2019-10-13T14:38:50.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:50.912+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.918012+0000) 2019-10-13T14:38:50.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:50.912+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.918281+0000) 2019-10-13T14:38:50.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:50.915+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.918749+0000) 2019-10-13T14:38:50.921 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:50.912+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.918784+0000) 2019-10-13T14:38:50.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:50.916+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.919847+0000) 2019-10-13T14:38:50.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:50.916+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.920700+0000) 2019-10-13T14:38:50.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:50.923+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.926251+0000) 2019-10-13T14:38:50.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:50.924+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201023 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:52.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:52.615+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.619338+0000) 2019-10-13T14:38:52.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:52.639+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.645984+0000) 2019-10-13T14:38:55.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:55.651+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.656739+0000) 2019-10-13T14:38:55.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:55.663+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.669146+0000) 2019-10-13T14:38:55.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:55.923+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.931071+0000) 2019-10-13T14:38:55.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:55.927+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.932171+0000) 2019-10-13T14:38:55.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:55.927+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.932839+0000) 2019-10-13T14:38:55.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:38:55.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.932880+0000) 2019-10-13T14:38:55.935 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:38:55.927+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.933216+0000) 2019-10-13T14:38:55.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:38:55.927+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.933840+0000) 2019-10-13T14:38:55.936 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:55.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.934208+0000) 2019-10-13T14:38:55.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:38:55.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:38:55.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:38:55.935+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.941687+0000) 2019-10-13T14:38:55.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:38:55.951+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:55.957261+0000) 2019-10-13T14:38:56.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:38:56.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:56.634518+0000) 2019-10-13T14:38:56.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:38:56.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:56.884823+0000) 2019-10-13T14:38:57.879 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:38:57.880 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:57.883 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:38:57.883 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:58.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:38:58.903+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:58.907020+0000) 2019-10-13T14:39:00.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:00.939+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.943087+0000) 2019-10-13T14:39:00.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:00.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.943613+0000) 2019-10-13T14:39:00.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:00.939+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.944070+0000) 2019-10-13T14:39:00.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:00.939+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.944705+0000) 2019-10-13T14:39:00.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:00.939+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.945913+0000) 2019-10-13T14:39:00.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:00.943+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.946441+0000) 2019-10-13T14:39:00.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:00.943+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.947247+0000) 2019-10-13T14:39:00.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:00.947+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.951712+0000) 2019-10-13T14:39:00.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:00.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201089 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:02.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:02.623+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:02.625731+0000) 2019-10-13T14:39:02.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:02.643+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:02.646180+0000) 2019-10-13T14:39:05.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:05.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.656960+0000) 2019-10-13T14:39:05.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:05.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.669347+0000) 2019-10-13T14:39:05.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:05.951+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.956748+0000) 2019-10-13T14:39:05.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:05.951+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.959031+0000) 2019-10-13T14:39:05.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:05.955+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.959342+0000) 2019-10-13T14:39:05.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:05.955+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.960188+0000) 2019-10-13T14:39:05.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:05.955+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.960260+0000) 2019-10-13T14:39:05.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:05.955+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.960609+0000) 2019-10-13T14:39:05.963 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:05.955+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.960873+0000) 2019-10-13T14:39:05.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:05.959+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:05.964262+0000) 2019-10-13T14:39:05.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:05.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:06.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:06.627+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:06.634709+0000) 2019-10-13T14:39:06.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:06.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:06.885027+0000) 2019-10-13T14:39:08.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:08.903+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.907223+0000) 2019-10-13T14:39:10.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:10.967+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.971996+0000) 2019-10-13T14:39:10.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:10.967+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.972308+0000) 2019-10-13T14:39:10.975 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:10.967+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.972747+0000) 2019-10-13T14:39:10.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:10.967+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.973054+0000) 2019-10-13T14:39:10.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:10.967+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.973912+0000) 2019-10-13T14:39:10.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:10.967+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.974463+0000) 2019-10-13T14:39:10.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:10.967+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.974849+0000) 2019-10-13T14:39:10.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:10.975+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.980595+0000) 2019-10-13T14:39:10.984 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:10.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201154 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:12.634 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:12.631+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:12.632158+0000) 2019-10-13T14:39:12.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:12.643+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:12.646371+0000) 2019-10-13T14:39:12.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:12.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:12.707880+0000) 2019-10-13T14:39:15.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:15.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.657174+0000) 2019-10-13T14:39:15.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:15.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.669575+0000) 2019-10-13T14:39:15.988 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:15.979+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.985870+0000) 2019-10-13T14:39:15.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:15.979+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.986261+0000) 2019-10-13T14:39:15.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:15.979+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.986717+0000) 2019-10-13T14:39:15.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:15.979+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.987149+0000) 2019-10-13T14:39:15.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:15.983+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.988245+0000) 2019-10-13T14:39:15.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:15.982+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.988468+0000) 2019-10-13T14:39:15.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:15.983+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.988931+0000) 2019-10-13T14:39:15.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:15.990+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:15.994608+0000) 2019-10-13T14:39:15.998 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:15.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:16.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:16.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:16.634922+0000) 2019-10-13T14:39:16.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:16.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:16.885201+0000) 2019-10-13T14:39:18.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:18.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.907422+0000) 2019-10-13T14:39:21.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:20.995+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:20.999043+0000) 2019-10-13T14:39:21.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:20.995+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:20.999561+0000) 2019-10-13T14:39:21.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:20.995+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:20.999919+0000) 2019-10-13T14:39:21.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:20.994+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.000918+0000) 2019-10-13T14:39:21.003 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:20.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.001022+0000) 2019-10-13T14:39:21.004 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:20.995+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.001894+0000) 2019-10-13T14:39:21.004 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:20.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.002109+0000) 2019-10-13T14:39:21.010 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:21.002+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.008047+0000) 2019-10-13T14:39:21.011 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:21.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:22.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:22.634+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:22.638494+0000) 2019-10-13T14:39:22.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:22.643+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:22.646582+0000) 2019-10-13T14:39:22.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:22.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:22.708076+0000) 2019-10-13T14:39:25.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:25.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:25.657350+0000) 2019-10-13T14:39:25.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:25.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:25.669771+0000) 2019-10-13T14:39:26.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:26.011+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.011903+0000) 2019-10-13T14:39:26.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:26.011+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.013326+0000) 2019-10-13T14:39:26.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:26.006+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.013905+0000) 2019-10-13T14:39:26.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:26.011+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.014358+0000) 2019-10-13T14:39:26.017 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:26.011+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.014510+0000) 2019-10-13T14:39:26.017 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:26.011+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.015027+0000) 2019-10-13T14:39:26.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:26.015+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.016281+0000) 2019-10-13T14:39:26.023 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:26.014+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.021396+0000) 2019-10-13T14:39:26.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:26.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:26.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:26.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.635086+0000) 2019-10-13T14:39:26.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:26.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:26.885368+0000) 2019-10-13T14:39:27.985 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:39:27.985 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:27.989 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:39:27.989 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:28.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:28.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.907654+0000) 2019-10-13T14:39:31.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:31.023+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.026159+0000) 2019-10-13T14:39:31.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:31.023+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.026750+0000) 2019-10-13T14:39:31.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:31.022+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.027297+0000) 2019-10-13T14:39:31.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:31.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.027741+0000) 2019-10-13T14:39:31.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:31.027+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.028530+0000) 2019-10-13T14:39:31.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:31.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.028901+0000) 2019-10-13T14:39:31.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:31.027+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.029449+0000) 2019-10-13T14:39:31.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:31.035+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:31.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:31.034+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.039732+0000) 2019-10-13T14:39:32.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:32.642+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:32.644810+0000) 2019-10-13T14:39:32.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:32.643+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:32.646786+0000) 2019-10-13T14:39:32.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:32.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:32.708277+0000) 2019-10-13T14:39:35.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:35.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:35.657573+0000) 2019-10-13T14:39:35.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:35.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:35.669961+0000) 2019-10-13T14:39:36.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:36.038+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.040156+0000) 2019-10-13T14:39:36.043 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:36.038+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.040521+0000) 2019-10-13T14:39:36.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:36.038+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.040947+0000) 2019-10-13T14:39:36.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:36.038+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.041568+0000) 2019-10-13T14:39:36.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:36.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.041742+0000) 2019-10-13T14:39:36.045 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:36.038+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.043143+0000) 2019-10-13T14:39:36.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:36.042+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.043341+0000) 2019-10-13T14:39:36.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:36.042+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.048565+0000) 2019-10-13T14:39:36.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:36.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:36.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:36.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.635256+0000) 2019-10-13T14:39:36.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:36.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:36.885541+0000) 2019-10-13T14:39:38.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:38.906+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.907855+0000) 2019-10-13T14:39:41.056 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:41.050+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.053699+0000) 2019-10-13T14:39:41.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:41.050+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.054625+0000) 2019-10-13T14:39:41.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:41.050+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.054941+0000) 2019-10-13T14:39:41.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:41.054+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.055625+0000) 2019-10-13T14:39:41.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:41.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.056745+0000) 2019-10-13T14:39:41.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:41.054+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.056963+0000) 2019-10-13T14:39:41.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:41.054+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.057218+0000) 2019-10-13T14:39:41.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:41.058+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.062909+0000) 2019-10-13T14:39:41.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:41.062+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:42.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:42.642+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:42.646985+0000) 2019-10-13T14:39:42.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:42.650+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:42.651198+0000) 2019-10-13T14:39:42.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:42.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:42.708548+0000) 2019-10-13T14:39:45.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:45.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:45.657773+0000) 2019-10-13T14:39:45.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:45.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:45.670175+0000) 2019-10-13T14:39:46.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:46.062+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.067114+0000) 2019-10-13T14:39:46.070 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:46.066+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.067799+0000) 2019-10-13T14:39:46.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:46.066+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.068544+0000) 2019-10-13T14:39:46.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:46.062+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.070052+0000) 2019-10-13T14:39:46.072 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:46.066+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.070255+0000) 2019-10-13T14:39:46.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:46.066+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.070687+0000) 2019-10-13T14:39:46.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:46.066+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.070701+0000) 2019-10-13T14:39:46.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:46.070+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.076419+0000) 2019-10-13T14:39:46.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:46.074+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:46.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:46.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.635454+0000) 2019-10-13T14:39:46.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:46.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:46.885770+0000) 2019-10-13T14:39:48.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:48.906+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.908076+0000) 2019-10-13T14:39:51.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:51.078+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.081702+0000) 2019-10-13T14:39:51.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:51.078+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.082415+0000) 2019-10-13T14:39:51.085 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:51.078+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.083140+0000) 2019-10-13T14:39:51.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:51.082+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.084253+0000) 2019-10-13T14:39:51.087 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:51.078+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.084766+0000) 2019-10-13T14:39:51.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:51.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.084851+0000) 2019-10-13T14:39:51.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:51.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.085673+0000) 2019-10-13T14:39:51.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:51.086+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.092913+0000) 2019-10-13T14:39:51.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:51.090+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201420 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:52.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:52.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:52.647187+0000) 2019-10-13T14:39:52.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:52.658+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:52.660388+0000) 2019-10-13T14:39:52.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:52.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:52.708790+0000) 2019-10-13T14:39:55.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:55.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:55.657970+0000) 2019-10-13T14:39:55.672 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:55.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:55.670329+0000) 2019-10-13T14:39:56.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:39:56.094+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.096000+0000) 2019-10-13T14:39:56.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:39:56.094+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.096911+0000) 2019-10-13T14:39:56.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:56.094+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.097094+0000) 2019-10-13T14:39:56.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:39:56.094+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.097939+0000) 2019-10-13T14:39:56.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:56.094+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.098312+0000) 2019-10-13T14:39:56.101 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:56.094+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.099074+0000) 2019-10-13T14:39:56.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:39:56.098+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.099443+0000) 2019-10-13T14:39:56.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:39:56.098+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.105505+0000) 2019-10-13T14:39:56.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:39:56.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:39:56.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:39:56.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.635585+0000) 2019-10-13T14:39:56.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:39:56.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:56.885958+0000) 2019-10-13T14:39:58.095 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:39:58.096 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:58.099 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:39:58.099 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:58.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:39:58.906+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.908277+0000) 2019-10-13T14:40:01.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:01.106+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.110414+0000) 2019-10-13T14:40:01.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:01.106+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.110736+0000) 2019-10-13T14:40:01.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:01.110+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.111546+0000) 2019-10-13T14:40:01.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:01.110+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.112067+0000) 2019-10-13T14:40:01.115 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:01.110+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.112586+0000) 2019-10-13T14:40:01.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:01.110+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.112980+0000) 2019-10-13T14:40:01.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:01.110+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.114313+0000) 2019-10-13T14:40:01.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:01.114+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.120369+0000) 2019-10-13T14:40:01.123 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:01.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:02.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:02.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:02.647413+0000) 2019-10-13T14:40:02.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:02.670+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:02.671467+0000) 2019-10-13T14:40:02.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:02.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:02.708959+0000) 2019-10-13T14:40:05.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:05.654+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:05.658168+0000) 2019-10-13T14:40:05.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:05.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:05.670526+0000) 2019-10-13T14:40:06.126 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:06.122+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.123453+0000) 2019-10-13T14:40:06.126 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:06.122+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.123810+0000) 2019-10-13T14:40:06.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:06.122+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.124118+0000) 2019-10-13T14:40:06.127 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:06.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.124843+0000) 2019-10-13T14:40:06.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:06.122+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.126276+0000) 2019-10-13T14:40:06.129 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:06.122+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.127046+0000) 2019-10-13T14:40:06.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:06.126+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.127067+0000) 2019-10-13T14:40:06.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:06.130+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.132380+0000) 2019-10-13T14:40:06.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:06.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201524 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:06.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:06.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.635759+0000) 2019-10-13T14:40:06.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:06.886+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:06.886130+0000) 2019-10-13T14:40:08.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:08.906+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.908498+0000) 2019-10-13T14:40:11.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:11.134+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.137409+0000) 2019-10-13T14:40:11.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:11.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.137806+0000) 2019-10-13T14:40:11.140 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:11.134+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.138122+0000) 2019-10-13T14:40:11.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:11.134+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.138566+0000) 2019-10-13T14:40:11.141 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:11.134+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.138886+0000) 2019-10-13T14:40:11.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:11.134+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.139211+0000) 2019-10-13T14:40:11.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:11.137+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.140150+0000) 2019-10-13T14:40:11.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:11.145+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.146805+0000) 2019-10-13T14:40:11.150 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:11.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:12.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:12.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:12.647566+0000) 2019-10-13T14:40:12.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:12.681+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:12.682457+0000) 2019-10-13T14:40:12.711 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:12.706+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:12.709235+0000) 2019-10-13T14:40:15.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:15.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:15.658385+0000) 2019-10-13T14:40:15.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:15.666+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:15.670724+0000) 2019-10-13T14:40:16.153 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:16.146+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.151026+0000) 2019-10-13T14:40:16.154 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:16.150+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.151373+0000) 2019-10-13T14:40:16.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:16.150+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.151796+0000) 2019-10-13T14:40:16.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:16.150+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.152343+0000) 2019-10-13T14:40:16.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:16.150+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.152609+0000) 2019-10-13T14:40:16.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:16.149+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.153171+0000) 2019-10-13T14:40:16.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:16.150+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.154185+0000) 2019-10-13T14:40:16.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:16.157+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.159364+0000) 2019-10-13T14:40:16.163 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:16.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201590 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:16.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:16.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.635962+0000) 2019-10-13T14:40:16.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:16.885+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:16.886342+0000) 2019-10-13T14:40:18.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:18.905+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.908706+0000) 2019-10-13T14:40:21.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:21.161+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.164087+0000) 2019-10-13T14:40:21.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:21.161+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.164289+0000) 2019-10-13T14:40:21.168 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:21.161+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.165899+0000) 2019-10-13T14:40:21.169 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:21.161+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.166283+0000) 2019-10-13T14:40:21.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:21.161+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.166275+0000) 2019-10-13T14:40:21.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:21.161+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.166866+0000) 2019-10-13T14:40:21.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:21.165+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.167629+0000) 2019-10-13T14:40:21.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:21.169+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.171394+0000) 2019-10-13T14:40:21.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:21.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:22.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:22.645+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:22.647762+0000) 2019-10-13T14:40:22.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:22.689+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:22.690466+0000) 2019-10-13T14:40:22.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:22.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:22.709456+0000) 2019-10-13T14:40:25.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:25.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:25.658602+0000) 2019-10-13T14:40:25.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:25.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:25.670932+0000) 2019-10-13T14:40:26.179 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:26.173+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.177310+0000) 2019-10-13T14:40:26.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:26.173+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.178239+0000) 2019-10-13T14:40:26.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:26.173+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.178632+0000) 2019-10-13T14:40:26.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:26.173+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.179150+0000) 2019-10-13T14:40:26.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:26.177+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.179502+0000) 2019-10-13T14:40:26.182 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:26.177+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.179880+0000) 2019-10-13T14:40:26.183 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:26.177+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.180597+0000) 2019-10-13T14:40:26.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:26.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:26.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:26.185+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.189372+0000) 2019-10-13T14:40:26.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:26.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.636135+0000) 2019-10-13T14:40:26.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:26.885+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:26.886522+0000) 2019-10-13T14:40:28.215 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:40:28.215 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:28.219 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:40:28.219 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:28.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:28.905+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.908901+0000) 2019-10-13T14:40:31.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:31.185+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.189560+0000) 2019-10-13T14:40:31.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:31.185+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.190007+0000) 2019-10-13T14:40:31.193 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:31.185+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.190827+0000) 2019-10-13T14:40:31.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:31.189+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.191488+0000) 2019-10-13T14:40:31.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:31.189+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.191848+0000) 2019-10-13T14:40:31.196 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:31.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.192278+0000) 2019-10-13T14:40:31.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:31.189+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.193070+0000) 2019-10-13T14:40:31.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:31.193+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.197056+0000) 2019-10-13T14:40:31.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:31.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:32.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:32.645+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:32.647965+0000) 2019-10-13T14:40:32.699 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:32.693+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:32.696784+0000) 2019-10-13T14:40:32.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:32.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:32.709657+0000) 2019-10-13T14:40:35.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:35.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:35.658835+0000) 2019-10-13T14:40:35.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:35.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:35.671127+0000) 2019-10-13T14:40:36.206 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:36.197+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.203216+0000) 2019-10-13T14:40:36.206 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:36.201+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.203551+0000) 2019-10-13T14:40:36.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:36.201+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.204119+0000) 2019-10-13T14:40:36.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:36.201+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.204451+0000) 2019-10-13T14:40:36.207 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:36.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.205105+0000) 2019-10-13T14:40:36.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:36.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.206336+0000) 2019-10-13T14:40:36.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:36.201+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.206635+0000) 2019-10-13T14:40:36.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:36.209+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.211584+0000) 2019-10-13T14:40:36.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:36.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:36.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:36.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.636329+0000) 2019-10-13T14:40:36.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:36.885+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:36.886718+0000) 2019-10-13T14:40:38.911 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:38.905+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.909133+0000) 2019-10-13T14:40:41.220 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:41.213+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.217031+0000) 2019-10-13T14:40:41.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:41.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.217372+0000) 2019-10-13T14:40:41.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:41.217+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.218396+0000) 2019-10-13T14:40:41.221 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:41.213+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.218700+0000) 2019-10-13T14:40:41.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:41.213+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.218987+0000) 2019-10-13T14:40:41.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:41.213+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.219226+0000) 2019-10-13T14:40:41.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:41.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.219594+0000) 2019-10-13T14:40:41.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:41.225+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.226497+0000) 2019-10-13T14:40:41.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:41.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:42.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:42.645+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:42.648200+0000) 2019-10-13T14:40:42.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:42.701+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:42.703142+0000) 2019-10-13T14:40:42.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:42.705+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:42.709924+0000) 2019-10-13T14:40:45.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:45.657+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:45.658991+0000) 2019-10-13T14:40:45.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:45.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:45.671348+0000) 2019-10-13T14:40:46.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:46.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.229949+0000) 2019-10-13T14:40:46.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:46.225+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.231089+0000) 2019-10-13T14:40:46.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:46.229+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.231733+0000) 2019-10-13T14:40:46.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:46.229+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.232410+0000) 2019-10-13T14:40:46.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:46.229+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.232862+0000) 2019-10-13T14:40:46.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:46.229+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.233242+0000) 2019-10-13T14:40:46.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:46.229+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.233577+0000) 2019-10-13T14:40:46.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:46.233+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.237470+0000) 2019-10-13T14:40:46.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:46.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:46.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:46.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.636506+0000) 2019-10-13T14:40:46.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:46.885+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:46.886930+0000) 2019-10-13T14:40:48.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:48.905+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.909364+0000) 2019-10-13T14:40:51.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:51.241+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.244319+0000) 2019-10-13T14:40:51.248 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:51.241+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.245308+0000) 2019-10-13T14:40:51.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:51.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.245777+0000) 2019-10-13T14:40:51.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:51.241+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.245942+0000) 2019-10-13T14:40:51.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:51.241+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.246364+0000) 2019-10-13T14:40:51.250 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:51.241+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.246771+0000) 2019-10-13T14:40:51.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:51.245+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.248578+0000) 2019-10-13T14:40:51.251 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:51.245+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.247930+0000) 2019-10-13T14:40:51.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:51.249+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:52.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:52.645+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:52.648423+0000) 2019-10-13T14:40:52.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:52.709+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:52.709525+0000) 2019-10-13T14:40:52.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:52.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:52.710076+0000) 2019-10-13T14:40:55.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:55.653+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:55.659215+0000) 2019-10-13T14:40:55.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:55.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:55.671535+0000) 2019-10-13T14:40:56.259 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:40:56.253+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.256414+0000) 2019-10-13T14:40:56.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:56.253+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.257817+0000) 2019-10-13T14:40:56.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:40:56.253+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.258099+0000) 2019-10-13T14:40:56.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:40:56.253+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.258547+0000) 2019-10-13T14:40:56.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:56.253+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.258929+0000) 2019-10-13T14:40:56.261 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:40:56.253+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.259183+0000) 2019-10-13T14:40:56.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:56.257+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.259562+0000) 2019-10-13T14:40:56.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:40:56.261+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.263334+0000) 2019-10-13T14:40:56.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:40:56.261+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:40:56.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:40:56.633+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.636710+0000) 2019-10-13T14:40:56.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:40:56.885+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:56.887099+0000) 2019-10-13T14:40:58.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:40:58.535 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:58.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:40:58.539 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:58.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:40:58.905+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.909565+0000) 2019-10-13T14:41:01.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:01.264+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.269418+0000) 2019-10-13T14:41:01.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:01.264+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.270432+0000) 2019-10-13T14:41:01.273 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:01.269+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.271476+0000) 2019-10-13T14:41:01.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:01.268+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.271728+0000) 2019-10-13T14:41:01.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:01.268+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.272088+0000) 2019-10-13T14:41:01.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:01.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.272498+0000) 2019-10-13T14:41:01.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:01.268+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.274050+0000) 2019-10-13T14:41:01.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:01.277+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.278982+0000) 2019-10-13T14:41:01.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:01.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:02.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:02.644+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:02.648658+0000) 2019-10-13T14:41:02.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:02.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:02.710274+0000) 2019-10-13T14:41:02.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:02.713+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:02.716051+0000) 2019-10-13T14:41:05.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:05.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:05.659411+0000) 2019-10-13T14:41:05.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:05.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:05.671740+0000) 2019-10-13T14:41:06.285 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:06.276+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.282772+0000) 2019-10-13T14:41:06.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:06.280+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.283462+0000) 2019-10-13T14:41:06.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:06.280+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.283898+0000) 2019-10-13T14:41:06.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:06.280+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.285160+0000) 2019-10-13T14:41:06.288 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:06.280+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.285165+0000) 2019-10-13T14:41:06.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:06.280+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.285644+0000) 2019-10-13T14:41:06.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:06.280+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.286064+0000) 2019-10-13T14:41:06.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:06.288+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.291129+0000) 2019-10-13T14:41:06.295 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:06.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:06.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:06.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.636884+0000) 2019-10-13T14:41:06.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:06.884+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:06.887298+0000) 2019-10-13T14:41:08.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:08.908+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.909724+0000) 2019-10-13T14:41:11.299 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:11.292+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.296421+0000) 2019-10-13T14:41:11.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:11.292+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.296990+0000) 2019-10-13T14:41:11.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:11.292+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.297216+0000) 2019-10-13T14:41:11.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:11.292+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.298186+0000) 2019-10-13T14:41:11.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:11.292+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.298586+0000) 2019-10-13T14:41:11.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:11.292+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.299245+0000) 2019-10-13T14:41:11.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:11.296+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.300060+0000) 2019-10-13T14:41:11.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:11.300+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.305370+0000) 2019-10-13T14:41:11.308 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:11.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201954 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:12.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:12.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:12.648886+0000) 2019-10-13T14:41:12.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:12.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:12.710475+0000) 2019-10-13T14:41:12.724 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:12.720+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:12.722441+0000) 2019-10-13T14:41:15.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:15.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:15.659636+0000) 2019-10-13T14:41:15.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:15.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:15.671940+0000) 2019-10-13T14:41:16.312 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:16.304+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.309977+0000) 2019-10-13T14:41:16.313 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:16.308+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.310567+0000) 2019-10-13T14:41:16.313 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:16.304+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.310940+0000) 2019-10-13T14:41:16.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:16.308+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.311297+0000) 2019-10-13T14:41:16.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:16.308+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.311699+0000) 2019-10-13T14:41:16.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:16.308+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.312036+0000) 2019-10-13T14:41:16.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:16.308+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.312568+0000) 2019-10-13T14:41:16.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:16.316+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.319954+0000) 2019-10-13T14:41:16.323 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:16.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 201989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:16.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:16.636+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.637090+0000) 2019-10-13T14:41:16.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:16.884+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:16.887466+0000) 2019-10-13T14:41:18.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:18.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.909940+0000) 2019-10-13T14:41:21.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:21.320+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.323480+0000) 2019-10-13T14:41:21.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:21.320+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.324013+0000) 2019-10-13T14:41:21.327 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:21.320+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.324269+0000) 2019-10-13T14:41:21.327 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:21.320+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.325091+0000) 2019-10-13T14:41:21.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:21.320+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.326219+0000) 2019-10-13T14:41:21.329 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:21.320+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.326490+0000) 2019-10-13T14:41:21.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:21.320+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.327099+0000) 2019-10-13T14:41:21.337 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:21.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202021 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:21.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:21.332+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.335194+0000) 2019-10-13T14:41:22.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:22.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:22.649099+0000) 2019-10-13T14:41:22.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:22.704+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:22.710657+0000) 2019-10-13T14:41:22.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:22.728+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:22.728992+0000) 2019-10-13T14:41:25.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:25.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:25.659837+0000) 2019-10-13T14:41:25.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:25.668+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:25.672131+0000) 2019-10-13T14:41:26.340 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:26.332+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.337241+0000) 2019-10-13T14:41:26.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:26.332+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.337465+0000) 2019-10-13T14:41:26.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:26.332+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.338649+0000) 2019-10-13T14:41:26.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:26.332+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.338762+0000) 2019-10-13T14:41:26.342 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:26.332+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.339164+0000) 2019-10-13T14:41:26.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:26.336+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.340157+0000) 2019-10-13T14:41:26.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:26.336+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.340344+0000) 2019-10-13T14:41:26.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:26.344+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.347323+0000) 2019-10-13T14:41:26.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:26.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202057 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:26.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:26.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.637266+0000) 2019-10-13T14:41:26.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:26.884+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:26.887649+0000) 2019-10-13T14:41:28.586 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:41:28.587 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:28.590 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:41:28.590 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:28.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:28.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.910142+0000) 2019-10-13T14:41:31.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:31.344+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.351185+0000) 2019-10-13T14:41:31.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:31.348+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.351705+0000) 2019-10-13T14:41:31.355 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:31.348+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.352418+0000) 2019-10-13T14:41:31.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:31.348+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.352929+0000) 2019-10-13T14:41:31.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:31.348+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.353334+0000) 2019-10-13T14:41:31.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:31.348+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.354011+0000) 2019-10-13T14:41:31.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:31.348+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.354569+0000) 2019-10-13T14:41:31.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:31.356+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.360696+0000) 2019-10-13T14:41:31.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:31.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202088 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:32.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:32.644+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:32.649253+0000) 2019-10-13T14:41:32.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:32.708+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:32.710855+0000) 2019-10-13T14:41:32.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:32.732+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:32.735549+0000) 2019-10-13T14:41:35.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:35.656+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:35.660040+0000) 2019-10-13T14:41:35.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:35.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:35.672325+0000) 2019-10-13T14:41:36.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:36.360+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.364694+0000) 2019-10-13T14:41:36.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:36.360+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.365241+0000) 2019-10-13T14:41:36.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:36.360+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.365778+0000) 2019-10-13T14:41:36.369 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:36.360+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.366178+0000) 2019-10-13T14:41:36.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:36.360+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.366772+0000) 2019-10-13T14:41:36.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:36.364+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.367843+0000) 2019-10-13T14:41:36.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:36.364+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.368110+0000) 2019-10-13T14:41:36.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:36.372+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.375004+0000) 2019-10-13T14:41:36.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:36.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:36.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:36.632+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.637439+0000) 2019-10-13T14:41:36.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:36.884+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:36.887815+0000) 2019-10-13T14:41:38.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:38.904+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.910394+0000) 2019-10-13T14:41:41.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:41.372+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.378812+0000) 2019-10-13T14:41:41.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:41.376+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.379212+0000) 2019-10-13T14:41:41.382 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:41.376+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.379686+0000) 2019-10-13T14:41:41.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:41.376+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.380407+0000) 2019-10-13T14:41:41.383 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:41.376+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.381126+0000) 2019-10-13T14:41:41.384 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:41.376+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.381486+0000) 2019-10-13T14:41:41.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:41.376+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.382627+0000) 2019-10-13T14:41:41.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:41.384+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.387112+0000) 2019-10-13T14:41:41.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:41.384+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:42.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:42.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:42.649457+0000) 2019-10-13T14:41:42.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:42.708+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:42.711072+0000) 2019-10-13T14:41:42.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:42.736+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:42.741954+0000) 2019-10-13T14:41:45.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:45.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:45.660235+0000) 2019-10-13T14:41:45.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:45.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:45.672528+0000) 2019-10-13T14:41:46.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:46.387+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.392304+0000) 2019-10-13T14:41:46.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:46.387+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.392690+0000) 2019-10-13T14:41:46.396 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:46.388+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.393702+0000) 2019-10-13T14:41:46.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:46.387+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.394388+0000) 2019-10-13T14:41:46.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:46.387+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.394779+0000) 2019-10-13T14:41:46.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:46.391+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.395935+0000) 2019-10-13T14:41:46.399 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:46.391+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.396390+0000) 2019-10-13T14:41:46.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:46.396+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.400548+0000) 2019-10-13T14:41:46.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:46.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:46.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:46.635+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.637634+0000) 2019-10-13T14:41:46.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:46.884+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:46.887992+0000) 2019-10-13T14:41:48.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:48.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.910592+0000) 2019-10-13T14:41:51.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:51.399+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.406294+0000) 2019-10-13T14:41:51.409 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:51.399+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.406979+0000) 2019-10-13T14:41:51.410 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:51.403+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.407261+0000) 2019-10-13T14:41:51.410 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:51.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.408104+0000) 2019-10-13T14:41:51.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:51.403+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.408488+0000) 2019-10-13T14:41:51.411 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:51.404+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.409250+0000) 2019-10-13T14:41:51.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:51.403+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.409360+0000) 2019-10-13T14:41:51.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:51.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.416552+0000) 2019-10-13T14:41:51.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:51.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:52.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:52.647+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:52.649684+0000) 2019-10-13T14:41:52.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:52.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:52.711269+0000) 2019-10-13T14:41:52.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:52.744+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:52.748285+0000) 2019-10-13T14:41:55.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:55.655+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:55.660459+0000) 2019-10-13T14:41:55.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:55.667+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:55.672726+0000) 2019-10-13T14:41:56.423 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:56.416+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.420655+0000) 2019-10-13T14:41:56.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:41:56.415+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.421065+0000) 2019-10-13T14:41:56.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:56.415+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.421875+0000) 2019-10-13T14:41:56.424 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:56.415+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.422162+0000) 2019-10-13T14:41:56.425 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:41:56.415+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.422344+0000) 2019-10-13T14:41:56.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:41:56.415+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.422889+0000) 2019-10-13T14:41:56.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:41:56.419+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.423578+0000) 2019-10-13T14:41:56.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:41:56.424+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.429098+0000) 2019-10-13T14:41:56.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:41:56.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:41:56.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:41:56.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.637809+0000) 2019-10-13T14:41:56.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:41:56.888+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:56.888162+0000) 2019-10-13T14:41:58.766 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:41:58.766 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:58.770 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:41:58.770 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:58.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:41:58.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.910808+0000) 2019-10-13T14:42:01.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:01.431+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.434818+0000) 2019-10-13T14:42:01.438 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:01.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.435481+0000) 2019-10-13T14:42:01.439 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:01.431+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.436275+0000) 2019-10-13T14:42:01.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:01.431+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.436306+0000) 2019-10-13T14:42:01.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:01.431+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.436631+0000) 2019-10-13T14:42:01.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:01.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.437100+0000) 2019-10-13T14:42:01.440 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:01.431+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.438156+0000) 2019-10-13T14:42:01.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:01.439+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.443380+0000) 2019-10-13T14:42:01.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:01.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:02.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:02.647+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:02.649864+0000) 2019-10-13T14:42:02.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:02.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:02.711547+0000) 2019-10-13T14:42:02.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:02.751+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:02.754725+0000) 2019-10-13T14:42:05.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:05.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:05.660662+0000) 2019-10-13T14:42:05.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:05.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:05.672924+0000) 2019-10-13T14:42:06.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:06.443+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.447972+0000) 2019-10-13T14:42:06.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:06.443+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.448541+0000) 2019-10-13T14:42:06.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:06.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.448911+0000) 2019-10-13T14:42:06.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:06.443+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.449311+0000) 2019-10-13T14:42:06.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:06.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.450619+0000) 2019-10-13T14:42:06.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:06.443+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.451014+0000) 2019-10-13T14:42:06.453 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:06.443+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.450986+0000) 2019-10-13T14:42:06.460 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:06.451+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:06.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:06.455+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.459892+0000) 2019-10-13T14:42:06.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:06.631+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.637983+0000) 2019-10-13T14:42:06.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:06.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:06.888341+0000) 2019-10-13T14:42:08.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:08.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.911013+0000) 2019-10-13T14:42:11.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:11.455+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.462187+0000) 2019-10-13T14:42:11.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:11.459+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.462625+0000) 2019-10-13T14:42:11.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:11.455+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.462911+0000) 2019-10-13T14:42:11.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:11.459+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.463713+0000) 2019-10-13T14:42:11.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:11.455+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.463136+0000) 2019-10-13T14:42:11.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:11.459+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.464470+0000) 2019-10-13T14:42:11.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:11.459+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.464706+0000) 2019-10-13T14:42:11.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:11.467+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.470615+0000) 2019-10-13T14:42:11.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:11.467+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:12.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:12.647+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:12.650079+0000) 2019-10-13T14:42:12.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:12.711+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:12.711782+0000) 2019-10-13T14:42:12.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:12.759+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:12.761101+0000) 2019-10-13T14:42:15.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:15.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:15.660852+0000) 2019-10-13T14:42:15.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:15.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:15.673121+0000) 2019-10-13T14:42:16.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:16.471+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.476564+0000) 2019-10-13T14:42:16.479 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:16.475+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.476808+0000) 2019-10-13T14:42:16.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:16.475+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.477248+0000) 2019-10-13T14:42:16.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:16.475+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.477319+0000) 2019-10-13T14:42:16.480 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:16.475+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.477782+0000) 2019-10-13T14:42:16.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:16.475+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.478532+0000) 2019-10-13T14:42:16.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:16.475+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.479119+0000) 2019-10-13T14:42:16.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:16.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.485008+0000) 2019-10-13T14:42:16.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:16.483+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:16.640 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:16.635+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.638180+0000) 2019-10-13T14:42:16.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:16.883+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:16.888540+0000) 2019-10-13T14:42:18.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:18.911+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.911243+0000) 2019-10-13T14:42:21.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:21.487+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.488546+0000) 2019-10-13T14:42:21.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:21.487+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.488976+0000) 2019-10-13T14:42:21.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:21.487+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.490150+0000) 2019-10-13T14:42:21.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:21.487+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.490773+0000) 2019-10-13T14:42:21.494 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:21.487+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.491592+0000) 2019-10-13T14:42:21.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:21.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.491436+0000) 2019-10-13T14:42:21.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:21.491+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.491885+0000) 2019-10-13T14:42:21.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:21.495+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.498580+0000) 2019-10-13T14:42:21.502 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:21.499+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:22.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:22.647+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:22.650308+0000) 2019-10-13T14:42:22.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:22.711+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:22.712005+0000) 2019-10-13T14:42:22.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:22.763+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:22.767393+0000) 2019-10-13T14:42:25.663 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:25.659+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:25.661092+0000) 2019-10-13T14:42:25.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:25.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:25.673343+0000) 2019-10-13T14:42:26.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:26.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.503112+0000) 2019-10-13T14:42:26.506 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:26.499+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.503073+0000) 2019-10-13T14:42:26.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:26.503+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.504252+0000) 2019-10-13T14:42:26.507 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:26.503+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.504540+0000) 2019-10-13T14:42:26.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:26.503+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.505131+0000) 2019-10-13T14:42:26.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:26.499+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.505943+0000) 2019-10-13T14:42:26.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:26.503+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.506425+0000) 2019-10-13T14:42:26.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:26.507+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.512612+0000) 2019-10-13T14:42:26.516 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:26.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:26.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:26.635+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.638387+0000) 2019-10-13T14:42:26.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:26.887+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:26.888720+0000) 2019-10-13T14:42:28.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:28.910+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.911476+0000) 2019-10-13T14:42:28.974 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:42:28.975 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:28.980 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:42:28.980 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:31.520 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:31.514+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.517268+0000) 2019-10-13T14:42:31.521 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:31.514+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.518056+0000) 2019-10-13T14:42:31.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:31.514+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.518900+0000) 2019-10-13T14:42:31.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:31.518+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.519296+0000) 2019-10-13T14:42:31.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:31.518+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.519886+0000) 2019-10-13T14:42:31.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:31.515+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.520084+0000) 2019-10-13T14:42:31.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:31.518+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.520456+0000) 2019-10-13T14:42:31.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:31.526+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:31.530 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:31.523+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.528104+0000) 2019-10-13T14:42:32.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:32.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:32.650510+0000) 2019-10-13T14:42:32.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:32.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:32.712239+0000) 2019-10-13T14:42:32.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:32.771+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:32.773940+0000) 2019-10-13T14:42:35.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:35.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:35.661287+0000) 2019-10-13T14:42:35.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:35.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:35.673556+0000) 2019-10-13T14:42:36.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:36.526+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.531173+0000) 2019-10-13T14:42:36.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:36.530+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.531607+0000) 2019-10-13T14:42:36.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:36.530+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.532724+0000) 2019-10-13T14:42:36.536 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:36.530+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.533621+0000) 2019-10-13T14:42:36.536 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:36.530+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.533818+0000) 2019-10-13T14:42:36.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:36.531+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.534656+0000) 2019-10-13T14:42:36.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:36.530+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.535016+0000) 2019-10-13T14:42:36.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:36.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202524 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:36.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:36.539+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.542358+0000) 2019-10-13T14:42:36.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:36.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.638563+0000) 2019-10-13T14:42:36.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:36.887+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:36.888895+0000) 2019-10-13T14:42:38.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:38.910+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.911635+0000) 2019-10-13T14:42:41.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:41.542+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.545705+0000) 2019-10-13T14:42:41.549 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:41.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.547245+0000) 2019-10-13T14:42:41.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:41.546+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.548069+0000) 2019-10-13T14:42:41.551 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:41.546+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.548377+0000) 2019-10-13T14:42:41.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:41.546+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.548721+0000) 2019-10-13T14:42:41.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:41.543+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.549360+0000) 2019-10-13T14:42:41.552 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:41.546+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.549641+0000) 2019-10-13T14:42:41.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:41.547+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.552567+0000) 2019-10-13T14:42:41.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:41.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:42.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:42.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:42.650661+0000) 2019-10-13T14:42:42.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:42.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:42.712437+0000) 2019-10-13T14:42:42.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:42.779+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:42.780389+0000) 2019-10-13T14:42:45.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:45.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:45.661521+0000) 2019-10-13T14:42:45.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:45.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:45.673749+0000) 2019-10-13T14:42:46.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:46.554+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.559239+0000) 2019-10-13T14:42:46.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:46.558+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.559842+0000) 2019-10-13T14:42:46.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:46.558+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.561117+0000) 2019-10-13T14:42:46.564 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:46.558+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.561926+0000) 2019-10-13T14:42:46.564 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:46.558+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.562167+0000) 2019-10-13T14:42:46.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:46.558+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.562450+0000) 2019-10-13T14:42:46.565 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:46.559+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.563472+0000) 2019-10-13T14:42:46.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:46.563+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.566740+0000) 2019-10-13T14:42:46.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:46.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202592 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:46.642 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:46.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.638773+0000) 2019-10-13T14:42:46.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:46.887+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:46.889108+0000) 2019-10-13T14:42:48.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:48.910+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.911851+0000) 2019-10-13T14:42:51.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:51.570+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.573761+0000) 2019-10-13T14:42:51.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:51.570+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.574668+0000) 2019-10-13T14:42:51.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:51.574+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.575570+0000) 2019-10-13T14:42:51.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:51.574+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.575841+0000) 2019-10-13T14:42:51.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:51.574+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.576312+0000) 2019-10-13T14:42:51.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:51.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.576727+0000) 2019-10-13T14:42:51.579 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:51.571+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.577559+0000) 2019-10-13T14:42:51.586 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:51.579+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.583798+0000) 2019-10-13T14:42:51.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:51.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:52.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:52.646+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:52.650896+0000) 2019-10-13T14:42:52.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:52.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:52.712664+0000) 2019-10-13T14:42:52.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:52.783+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:52.786707+0000) 2019-10-13T14:42:55.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:55.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:55.661737+0000) 2019-10-13T14:42:55.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:55.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:55.673960+0000) 2019-10-13T14:42:56.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:42:56.586+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.587484+0000) 2019-10-13T14:42:56.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:42:56.586+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.588234+0000) 2019-10-13T14:42:56.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:42:56.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.589371+0000) 2019-10-13T14:42:56.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:56.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.590143+0000) 2019-10-13T14:42:56.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:56.586+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.590486+0000) 2019-10-13T14:42:56.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:42:56.586+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.590833+0000) 2019-10-13T14:42:56.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:56.586+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.591463+0000) 2019-10-13T14:42:56.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:42:56.590+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.595086+0000) 2019-10-13T14:42:56.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:42:56.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:42:56.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:42:56.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.638944+0000) 2019-10-13T14:42:56.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:42:56.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:56.889281+0000) 2019-10-13T14:42:58.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:42:58.910+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.912078+0000) 2019-10-13T14:42:59.106 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:42:59.106 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:59.110 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:42:59.110 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:01.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:01.598+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.600790+0000) 2019-10-13T14:43:01.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:01.598+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.601242+0000) 2019-10-13T14:43:01.604 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:01.598+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.602467+0000) 2019-10-13T14:43:01.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:01.598+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.602412+0000) 2019-10-13T14:43:01.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:01.598+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.602685+0000) 2019-10-13T14:43:01.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:01.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.603374+0000) 2019-10-13T14:43:01.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:01.602+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.604455+0000) 2019-10-13T14:43:01.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:01.602+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.608235+0000) 2019-10-13T14:43:01.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:01.606+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:02.653 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:02.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:02.651106+0000) 2019-10-13T14:43:02.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:02.710+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:02.712864+0000) 2019-10-13T14:43:02.796 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:02.790+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:02.793202+0000) 2019-10-13T14:43:05.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:05.658+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:05.661956+0000) 2019-10-13T14:43:05.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:05.670+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:05.674175+0000) 2019-10-13T14:43:06.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:06.610+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.613260+0000) 2019-10-13T14:43:06.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:06.610+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.613828+0000) 2019-10-13T14:43:06.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:06.610+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.614041+0000) 2019-10-13T14:43:06.617 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:06.610+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.615436+0000) 2019-10-13T14:43:06.618 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:06.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.615516+0000) 2019-10-13T14:43:06.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:06.614+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.616003+0000) 2019-10-13T14:43:06.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:06.614+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.617326+0000) 2019-10-13T14:43:06.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:06.618+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.622744+0000) 2019-10-13T14:43:06.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:06.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202726 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:06.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:06.634+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.639118+0000) 2019-10-13T14:43:06.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:06.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:06.889450+0000) 2019-10-13T14:43:08.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:08.910+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.912280+0000) 2019-10-13T14:43:11.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:11.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.627421+0000) 2019-10-13T14:43:11.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:11.625+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.628240+0000) 2019-10-13T14:43:11.631 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:11.625+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.629099+0000) 2019-10-13T14:43:11.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:11.625+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.629507+0000) 2019-10-13T14:43:11.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:11.622+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.629898+0000) 2019-10-13T14:43:11.633 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:11.625+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.630064+0000) 2019-10-13T14:43:11.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:11.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.630709+0000) 2019-10-13T14:43:11.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:11.630+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.637748+0000) 2019-10-13T14:43:11.641 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:11.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:12.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:12.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:12.651278+0000) 2019-10-13T14:43:12.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:12.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:12.713069+0000) 2019-10-13T14:43:12.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:12.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:12.799532+0000) 2019-10-13T14:43:15.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:15.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:15.662189+0000) 2019-10-13T14:43:15.676 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:15.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:15.674377+0000) 2019-10-13T14:43:16.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:16.637+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.642828+0000) 2019-10-13T14:43:16.646 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:16.637+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.643254+0000) 2019-10-13T14:43:16.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:16.641+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.643613+0000) 2019-10-13T14:43:16.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:16.638+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.644044+0000) 2019-10-13T14:43:16.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:16.641+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.644201+0000) 2019-10-13T14:43:16.647 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:16.641+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.644798+0000) 2019-10-13T14:43:16.647 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:16.641+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.645352+0000) 2019-10-13T14:43:16.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:16.642+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.649971+0000) 2019-10-13T14:43:16.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:16.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202792 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:16.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:16.886+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:16.889619+0000) 2019-10-13T14:43:18.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:18.909+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.912487+0000) 2019-10-13T14:43:21.659 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:21.653+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.656727+0000) 2019-10-13T14:43:21.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:21.653+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.657338+0000) 2019-10-13T14:43:21.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:21.653+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.657685+0000) 2019-10-13T14:43:21.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:21.654+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.658620+0000) 2019-10-13T14:43:21.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:21.653+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.658984+0000) 2019-10-13T14:43:21.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:21.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.659989+0000) 2019-10-13T14:43:21.663 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:21.657+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.660654+0000) 2019-10-13T14:43:21.668 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:21.658+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.665949+0000) 2019-10-13T14:43:21.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:21.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:22.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:22.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:22.651518+0000) 2019-10-13T14:43:22.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:22.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:22.713269+0000) 2019-10-13T14:43:22.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:22.802+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:22.805842+0000) 2019-10-13T14:43:23.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:23.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.390272+0000) 2019-10-13T14:43:25.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:25.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:25.662415+0000) 2019-10-13T14:43:25.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:25.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:25.674598+0000) 2019-10-13T14:43:26.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:26.665+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.670636+0000) 2019-10-13T14:43:26.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:26.669+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.671373+0000) 2019-10-13T14:43:26.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:26.666+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.671995+0000) 2019-10-13T14:43:26.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:26.665+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.671065+0000) 2019-10-13T14:43:26.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:26.669+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.673290+0000) 2019-10-13T14:43:26.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:26.669+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.673812+0000) 2019-10-13T14:43:26.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:26.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.673991+0000) 2019-10-13T14:43:26.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:26.674+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.678830+0000) 2019-10-13T14:43:26.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:26.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202860 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:26.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:26.882+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:26.889740+0000) 2019-10-13T14:43:28.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:28.909+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.912678+0000) 2019-10-13T14:43:29.341 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:43:29.341 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:29.346 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:43:29.346 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:31.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:31.677+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.683007+0000) 2019-10-13T14:43:31.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:31.681+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.683507+0000) 2019-10-13T14:43:31.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:31.681+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.684040+0000) 2019-10-13T14:43:31.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:31.681+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.684874+0000) 2019-10-13T14:43:31.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:31.681+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.684628+0000) 2019-10-13T14:43:31.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:31.678+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.685428+0000) 2019-10-13T14:43:31.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:31.681+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.686375+0000) 2019-10-13T14:43:31.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:31.686+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.691931+0000) 2019-10-13T14:43:31.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:31.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202891 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:32.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:32.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:32.651697+0000) 2019-10-13T14:43:32.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:32.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:32.713518+0000) 2019-10-13T14:43:32.817 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:32.810+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:32.812204+0000) 2019-10-13T14:43:33.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:33.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.390463+0000) 2019-10-13T14:43:35.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:35.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:35.662612+0000) 2019-10-13T14:43:35.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:35.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:35.674786+0000) 2019-10-13T14:43:36.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:36.693+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.697294+0000) 2019-10-13T14:43:36.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:36.693+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.697756+0000) 2019-10-13T14:43:36.701 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:36.693+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.698501+0000) 2019-10-13T14:43:36.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:36.693+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.698770+0000) 2019-10-13T14:43:36.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:36.693+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.699081+0000) 2019-10-13T14:43:36.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:36.694+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.699952+0000) 2019-10-13T14:43:36.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:36.697+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.699949+0000) 2019-10-13T14:43:36.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:36.698+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.705891+0000) 2019-10-13T14:43:36.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:36.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:36.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:36.886+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:36.889911+0000) 2019-10-13T14:43:38.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:38.909+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.912878+0000) 2019-10-13T14:43:41.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:41.705+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.711056+0000) 2019-10-13T14:43:41.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:41.709+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.711452+0000) 2019-10-13T14:43:41.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:41.709+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.711783+0000) 2019-10-13T14:43:41.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:41.709+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.712539+0000) 2019-10-13T14:43:41.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:41.710+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.712987+0000) 2019-10-13T14:43:41.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:41.709+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.714045+0000) 2019-10-13T14:43:41.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:41.709+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.714333+0000) 2019-10-13T14:43:41.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:41.718+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.720056+0000) 2019-10-13T14:43:41.722 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:41.717+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202959 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:42.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:42.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:42.651855+0000) 2019-10-13T14:43:42.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:42.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:42.713716+0000) 2019-10-13T14:43:42.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:42.822+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:42.823558+0000) 2019-10-13T14:43:43.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:43.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.390654+0000) 2019-10-13T14:43:45.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:45.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:45.662838+0000) 2019-10-13T14:43:45.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:45.669+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:45.674979+0000) 2019-10-13T14:43:46.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:46.721+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.723733+0000) 2019-10-13T14:43:46.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:46.721+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.724497+0000) 2019-10-13T14:43:46.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:46.721+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.724898+0000) 2019-10-13T14:43:46.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:46.721+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.725358+0000) 2019-10-13T14:43:46.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:46.721+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.725860+0000) 2019-10-13T14:43:46.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:46.726+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.726471+0000) 2019-10-13T14:43:46.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:46.721+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.726708+0000) 2019-10-13T14:43:46.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:46.730+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.733287+0000) 2019-10-13T14:43:46.736 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:46.729+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 202994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:46.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:46.886+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:46.890098+0000) 2019-10-13T14:43:48.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:48.909+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.913082+0000) 2019-10-13T14:43:51.739 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:51.733+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.737280+0000) 2019-10-13T14:43:51.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:51.733+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.737803+0000) 2019-10-13T14:43:51.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:51.733+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.738396+0000) 2019-10-13T14:43:51.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:51.737+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.740071+0000) 2019-10-13T14:43:51.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:51.737+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.740237+0000) 2019-10-13T14:43:51.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:51.737+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.740797+0000) 2019-10-13T14:43:51.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:51.737+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.741636+0000) 2019-10-13T14:43:51.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:51.741+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.745476+0000) 2019-10-13T14:43:51.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:51.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203025 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:52.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:52.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:52.651977+0000) 2019-10-13T14:43:52.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:52.709+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:52.713921+0000) 2019-10-13T14:43:52.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:52.825+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:52.829899+0000) 2019-10-13T14:43:53.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:53.385+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.390877+0000) 2019-10-13T14:43:55.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:55.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:55.663034+0000) 2019-10-13T14:43:55.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:55.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:55.675175+0000) 2019-10-13T14:43:56.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:56.749+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.751656+0000) 2019-10-13T14:43:56.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:43:56.744+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.751184+0000) 2019-10-13T14:43:56.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:43:56.748+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.752488+0000) 2019-10-13T14:43:56.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:43:56.748+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.752983+0000) 2019-10-13T14:43:56.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:43:56.748+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.754291+0000) 2019-10-13T14:43:56.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:43:56.748+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.754688+0000) 2019-10-13T14:43:56.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:56.752+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.755437+0000) 2019-10-13T14:43:56.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:43:56.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:43:56.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:43:56.761+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.763370+0000) 2019-10-13T14:43:56.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:43:56.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:56.890299+0000) 2019-10-13T14:43:58.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:43:58.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.913281+0000) 2019-10-13T14:43:59.452 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:43:59.452 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:59.455 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:43:59.456 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:01.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:01.760+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.765821+0000) 2019-10-13T14:44:01.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:01.760+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.766523+0000) 2019-10-13T14:44:01.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:01.760+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.767147+0000) 2019-10-13T14:44:01.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:01.765+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.767505+0000) 2019-10-13T14:44:01.770 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:01.764+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.768072+0000) 2019-10-13T14:44:01.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:01.764+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.768465+0000) 2019-10-13T14:44:01.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:01.764+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.768902+0000) 2019-10-13T14:44:01.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:01.769+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.771987+0000) 2019-10-13T14:44:01.778 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:01.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:02.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:02.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:02.652150+0000) 2019-10-13T14:44:02.716 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:02.708+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:02.714126+0000) 2019-10-13T14:44:02.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:02.833+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:02.836429+0000) 2019-10-13T14:44:03.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:03.384+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.391062+0000) 2019-10-13T14:44:05.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:05.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:05.663257+0000) 2019-10-13T14:44:05.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:05.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:05.675367+0000) 2019-10-13T14:44:06.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:06.772+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.778497+0000) 2019-10-13T14:44:06.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:06.776+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.779794+0000) 2019-10-13T14:44:06.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:06.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.780292+0000) 2019-10-13T14:44:06.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:06.776+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.780683+0000) 2019-10-13T14:44:06.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:06.776+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.781535+0000) 2019-10-13T14:44:06.784 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:06.781+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.782218+0000) 2019-10-13T14:44:06.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:06.776+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.782420+0000) 2019-10-13T14:44:06.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:06.781+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.785533+0000) 2019-10-13T14:44:06.791 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:06.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203128 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:06.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:06.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:06.890502+0000) 2019-10-13T14:44:08.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:08.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.913496+0000) 2019-10-13T14:44:11.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:11.788+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.791745+0000) 2019-10-13T14:44:11.795 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:11.789+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.792023+0000) 2019-10-13T14:44:11.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:11.788+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.792401+0000) 2019-10-13T14:44:11.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:11.788+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.793212+0000) 2019-10-13T14:44:11.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:11.788+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.793815+0000) 2019-10-13T14:44:11.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:11.788+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.795180+0000) 2019-10-13T14:44:11.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:11.792+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.795493+0000) 2019-10-13T14:44:11.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:11.797+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.799869+0000) 2019-10-13T14:44:11.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:11.796+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:12.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:12.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:12.652334+0000) 2019-10-13T14:44:12.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:12.712+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:12.714326+0000) 2019-10-13T14:44:12.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:12.841+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:12.842838+0000) 2019-10-13T14:44:13.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:13.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.391269+0000) 2019-10-13T14:44:15.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:15.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:15.663527+0000) 2019-10-13T14:44:15.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:15.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:15.675581+0000) 2019-10-13T14:44:16.809 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:16.800+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.805929+0000) 2019-10-13T14:44:16.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:16.800+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.805625+0000) 2019-10-13T14:44:16.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:16.800+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.806421+0000) 2019-10-13T14:44:16.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:16.800+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.806687+0000) 2019-10-13T14:44:16.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:16.804+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.807558+0000) 2019-10-13T14:44:16.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:16.805+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.808524+0000) 2019-10-13T14:44:16.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:16.804+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.807947+0000) 2019-10-13T14:44:16.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:16.809+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.813453+0000) 2019-10-13T14:44:16.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:16.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:16.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:16.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:16.890704+0000) 2019-10-13T14:44:18.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:18.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.913691+0000) 2019-10-13T14:44:21.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:21.812+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.818401+0000) 2019-10-13T14:44:21.822 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:21.817+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.819405+0000) 2019-10-13T14:44:21.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:21.812+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.818992+0000) 2019-10-13T14:44:21.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:21.816+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.820180+0000) 2019-10-13T14:44:21.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:21.816+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.820637+0000) 2019-10-13T14:44:21.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:21.816+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.821273+0000) 2019-10-13T14:44:21.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:21.816+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.821558+0000) 2019-10-13T14:44:21.831 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:21.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:21.833 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:21.829+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.831178+0000) 2019-10-13T14:44:22.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:22.648+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:22.652552+0000) 2019-10-13T14:44:22.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:22.708+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:22.714518+0000) 2019-10-13T14:44:22.851 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:22.845+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:22.849151+0000) 2019-10-13T14:44:23.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:23.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.391487+0000) 2019-10-13T14:44:25.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:25.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:25.663756+0000) 2019-10-13T14:44:25.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:25.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:25.675804+0000) 2019-10-13T14:44:26.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:26.828+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.832584+0000) 2019-10-13T14:44:26.835 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:26.828+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.832779+0000) 2019-10-13T14:44:26.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:26.829+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.833711+0000) 2019-10-13T14:44:26.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:26.828+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.834667+0000) 2019-10-13T14:44:26.837 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:26.828+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.835107+0000) 2019-10-13T14:44:26.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:26.832+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.835540+0000) 2019-10-13T14:44:26.838 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:26.832+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.836002+0000) 2019-10-13T14:44:26.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:26.837+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.840362+0000) 2019-10-13T14:44:26.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:26.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:26.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:26.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:26.890884+0000) 2019-10-13T14:44:28.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:28.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.913887+0000) 2019-10-13T14:44:29.575 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:44:29.575 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:29.579 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:44:29.579 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:31.849 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:31.840+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.846269+0000) 2019-10-13T14:44:31.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:31.844+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.847293+0000) 2019-10-13T14:44:31.850 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:31.844+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.847516+0000) 2019-10-13T14:44:31.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:31.844+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.848135+0000) 2019-10-13T14:44:31.851 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:31.844+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.848692+0000) 2019-10-13T14:44:31.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:31.845+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.849203+0000) 2019-10-13T14:44:31.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:31.844+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.850124+0000) 2019-10-13T14:44:31.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:31.849+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.852696+0000) 2019-10-13T14:44:31.859 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:31.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:32.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:32.652+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:32.652769+0000) 2019-10-13T14:44:32.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:32.712+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:32.714722+0000) 2019-10-13T14:44:32.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:32.853+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:32.855612+0000) 2019-10-13T14:44:33.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:33.388+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.391729+0000) 2019-10-13T14:44:35.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:35.660+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:35.663959+0000) 2019-10-13T14:44:35.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:35.672+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:35.676033+0000) 2019-10-13T14:44:36.862 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:36.856+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.859623+0000) 2019-10-13T14:44:36.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:36.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.860510+0000) 2019-10-13T14:44:36.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:36.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.860754+0000) 2019-10-13T14:44:36.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:36.857+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.861452+0000) 2019-10-13T14:44:36.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:36.856+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.862389+0000) 2019-10-13T14:44:36.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:36.856+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.863102+0000) 2019-10-13T14:44:36.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:36.856+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.862547+0000) 2019-10-13T14:44:36.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:36.865+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.869799+0000) 2019-10-13T14:44:36.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:36.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:36.893 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:36.889+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:36.891036+0000) 2019-10-13T14:44:38.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:38.911+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.914084+0000) 2019-10-13T14:44:41.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:41.867+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.874585+0000) 2019-10-13T14:44:41.878 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:41.867+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.874988+0000) 2019-10-13T14:44:41.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:41.871+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.875478+0000) 2019-10-13T14:44:41.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:41.873+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.876267+0000) 2019-10-13T14:44:41.879 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:41.871+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.875913+0000) 2019-10-13T14:44:41.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:41.871+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.876689+0000) 2019-10-13T14:44:41.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:41.871+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.876918+0000) 2019-10-13T14:44:41.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:41.881+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.883930+0000) 2019-10-13T14:44:41.887 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:41.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203360 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:42.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:42.651+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:42.652948+0000) 2019-10-13T14:44:42.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:42.711+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:42.714911+0000) 2019-10-13T14:44:42.869 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:42.864+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:42.866381+0000) 2019-10-13T14:44:43.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:43.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.391923+0000) 2019-10-13T14:44:45.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:45.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:45.664159+0000) 2019-10-13T14:44:45.678 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:45.671+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:45.676263+0000) 2019-10-13T14:44:46.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:46.883+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.888296+0000) 2019-10-13T14:44:46.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:46.883+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.889427+0000) 2019-10-13T14:44:46.892 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:46.883+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.889805+0000) 2019-10-13T14:44:46.893 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:46.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.890293+0000) 2019-10-13T14:44:46.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:46.883+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.890530+0000) 2019-10-13T14:44:46.895 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:46.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.892822+0000) 2019-10-13T14:44:46.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:46.888+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.892997+0000) 2019-10-13T14:44:46.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:46.892+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:46.897787+0000) 2019-10-13T14:44:46.901 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:46.891+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203396 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:48.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:48.911+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.914312+0000) 2019-10-13T14:44:51.905 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:51.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.903015+0000) 2019-10-13T14:44:51.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:51.895+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.903264+0000) 2019-10-13T14:44:51.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:51.900+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.903846+0000) 2019-10-13T14:44:51.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:51.899+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.904384+0000) 2019-10-13T14:44:51.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:51.899+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.904735+0000) 2019-10-13T14:44:51.907 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:51.899+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.905035+0000) 2019-10-13T14:44:51.908 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:51.899+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.905421+0000) 2019-10-13T14:44:51.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:51.908+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.912990+0000) 2019-10-13T14:44:51.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:51.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203426 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:52.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:52.647+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:52.653130+0000) 2019-10-13T14:44:52.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:52.707+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:52.715111+0000) 2019-10-13T14:44:52.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:52.868+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:52.872830+0000) 2019-10-13T14:44:53.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:53.387+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.392133+0000) 2019-10-13T14:44:53.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:53.636+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.641678+0000) 2019-10-13T14:44:55.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:55.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:55.664313+0000) 2019-10-13T14:44:55.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:55.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:55.676522+0000) 2019-10-13T14:44:56.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:44:56.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.915700+0000) 2019-10-13T14:44:56.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:44:56.911+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.917163+0000) 2019-10-13T14:44:56.920 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:44:56.911+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.917503+0000) 2019-10-13T14:44:56.921 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:44:56.916+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.918755+0000) 2019-10-13T14:44:56.922 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:44:56.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.918338+0000) 2019-10-13T14:44:56.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:44:56.911+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.919139+0000) 2019-10-13T14:44:56.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:56.915+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.919853+0000) 2019-10-13T14:44:56.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:44:56.919+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203462 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:44:56.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:44:56.924+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:56.926543+0000) 2019-10-13T14:44:58.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:44:58.907+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:58.914514+0000) 2019-10-13T14:44:59.699 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:44:59.700 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:59.703 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:44:59.703 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:01.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:01.923+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.929699+0000) 2019-10-13T14:45:01.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:01.923+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.930490+0000) 2019-10-13T14:45:01.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:01.923+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.930709+0000) 2019-10-13T14:45:01.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:01.928+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.931499+0000) 2019-10-13T14:45:01.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:01.923+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.930942+0000) 2019-10-13T14:45:01.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:01.927+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.931793+0000) 2019-10-13T14:45:01.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:01.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.931991+0000) 2019-10-13T14:45:01.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:01.936+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.938859+0000) 2019-10-13T14:45:01.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:01.935+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:02.655 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:02.651+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:02.653308+0000) 2019-10-13T14:45:02.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:02.711+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:02.715264+0000) 2019-10-13T14:45:02.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:02.876+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:02.879401+0000) 2019-10-13T14:45:03.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:03.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:03.392323+0000) 2019-10-13T14:45:03.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:03.636+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:03.641899+0000) 2019-10-13T14:45:05.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:05.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:05.664585+0000) 2019-10-13T14:45:05.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:05.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:05.676715+0000) 2019-10-13T14:45:06.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:06.939+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.943091+0000) 2019-10-13T14:45:06.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:06.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.943438+0000) 2019-10-13T14:45:06.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:06.939+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.943774+0000) 2019-10-13T14:45:06.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:06.939+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.944539+0000) 2019-10-13T14:45:06.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:06.939+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.944915+0000) 2019-10-13T14:45:06.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:06.940+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.945660+0000) 2019-10-13T14:45:06.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:06.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.946089+0000) 2019-10-13T14:45:06.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:06.947+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203530 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:06.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:06.952+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:06.954868+0000) 2019-10-13T14:45:08.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:08.911+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:08.914728+0000) 2019-10-13T14:45:11.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:11.955+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.957854+0000) 2019-10-13T14:45:11.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:11.955+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.958280+0000) 2019-10-13T14:45:11.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:11.955+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.959271+0000) 2019-10-13T14:45:11.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:11.959+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.959541+0000) 2019-10-13T14:45:11.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:11.959+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.960280+0000) 2019-10-13T14:45:11.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:11.959+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.960592+0000) 2019-10-13T14:45:11.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:11.956+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.961922+0000) 2019-10-13T14:45:11.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:11.967+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:11.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:11.964+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.968065+0000) 2019-10-13T14:45:12.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:12.651+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:12.653494+0000) 2019-10-13T14:45:12.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:12.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:12.715447+0000) 2019-10-13T14:45:12.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:12.884+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:12.885663+0000) 2019-10-13T14:45:13.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:13.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:13.392546+0000) 2019-10-13T14:45:13.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:13.640+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:13.642095+0000) 2019-10-13T14:45:15.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:15.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:15.664741+0000) 2019-10-13T14:45:15.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:15.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:15.676925+0000) 2019-10-13T14:45:16.974 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:16.968+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.972150+0000) 2019-10-13T14:45:16.975 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:16.971+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.972456+0000) 2019-10-13T14:45:16.975 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:16.971+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.972767+0000) 2019-10-13T14:45:16.976 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:16.971+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.973100+0000) 2019-10-13T14:45:16.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:16.971+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.973417+0000) 2019-10-13T14:45:16.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:16.971+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.974026+0000) 2019-10-13T14:45:16.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:16.971+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.973910+0000) 2019-10-13T14:45:16.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:16.972+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:16.977788+0000) 2019-10-13T14:45:16.983 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:16.979+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:18.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:18.911+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:18.914927+0000) 2019-10-13T14:45:21.987 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:21.982+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.984431+0000) 2019-10-13T14:45:21.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:21.982+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.984890+0000) 2019-10-13T14:45:21.988 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:21.982+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.986217+0000) 2019-10-13T14:45:21.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:21.982+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.986572+0000) 2019-10-13T14:45:21.990 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:21.984+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.987137+0000) 2019-10-13T14:45:21.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:21.986+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.987338+0000) 2019-10-13T14:45:21.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:21.986+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.987641+0000) 2019-10-13T14:45:21.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:21.990+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203628 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:21.997 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:21.992+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.994941+0000) 2019-10-13T14:45:22.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:22.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:22.653671+0000) 2019-10-13T14:45:22.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:22.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:22.715622+0000) 2019-10-13T14:45:22.899 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:22.896+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:22.896766+0000) 2019-10-13T14:45:23.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:23.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:23.392758+0000) 2019-10-13T14:45:23.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:23.640+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:23.642312+0000) 2019-10-13T14:45:25.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:25.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:25.664936+0000) 2019-10-13T14:45:25.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:25.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:25.677123+0000) 2019-10-13T14:45:27.000 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:26.994+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:26.997514+0000) 2019-10-13T14:45:27.001 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:26.994+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:26.998449+0000) 2019-10-13T14:45:27.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:26.994+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:26.998952+0000) 2019-10-13T14:45:27.002 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:26.994+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:26.999195+0000) 2019-10-13T14:45:27.002 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:26.998+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:27.000106+0000) 2019-10-13T14:45:27.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:26.998+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:27.000503+0000) 2019-10-13T14:45:27.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:26.996+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:27.000954+0000) 2019-10-13T14:45:27.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:27.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:27.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:27.004+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:27.009913+0000) 2019-10-13T14:45:28.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:28.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:28.915130+0000) 2019-10-13T14:45:29.838 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:45:29.838 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:29.842 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:45:29.843 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:32.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:32.006+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.010430+0000) 2019-10-13T14:45:32.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:32.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.011182+0000) 2019-10-13T14:45:32.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:32.010+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.011929+0000) 2019-10-13T14:45:32.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:32.010+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.012477+0000) 2019-10-13T14:45:32.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:32.010+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.012712+0000) 2019-10-13T14:45:32.016 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:32.010+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.013408+0000) 2019-10-13T14:45:32.016 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:32.008+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.014092+0000) 2019-10-13T14:45:32.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:32.016+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.020240+0000) 2019-10-13T14:45:32.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:32.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:32.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:32.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.653843+0000) 2019-10-13T14:45:32.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:32.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.715771+0000) 2019-10-13T14:45:32.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:32.900+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:32.903179+0000) 2019-10-13T14:45:33.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:33.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:33.392954+0000) 2019-10-13T14:45:33.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:33.640+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:33.642510+0000) 2019-10-13T14:45:35.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:35.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:35.665135+0000) 2019-10-13T14:45:35.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:35.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:35.677313+0000) 2019-10-13T14:45:37.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:37.022+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.025243+0000) 2019-10-13T14:45:37.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:37.022+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.025708+0000) 2019-10-13T14:45:37.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:37.023+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.026255+0000) 2019-10-13T14:45:37.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:37.022+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.026365+0000) 2019-10-13T14:45:37.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:37.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.026965+0000) 2019-10-13T14:45:37.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:37.026+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.027835+0000) 2019-10-13T14:45:37.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:37.026+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.028592+0000) 2019-10-13T14:45:37.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:37.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203730 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:37.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:37.031+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:37.036439+0000) 2019-10-13T14:45:38.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:38.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:38.915354+0000) 2019-10-13T14:45:42.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:42.034+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.037383+0000) 2019-10-13T14:45:42.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:42.034+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.038380+0000) 2019-10-13T14:45:42.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:42.034+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.039225+0000) 2019-10-13T14:45:42.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:42.034+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.038776+0000) 2019-10-13T14:45:42.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:42.038+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.039622+0000) 2019-10-13T14:45:42.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:42.035+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.040668+0000) 2019-10-13T14:45:42.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:42.038+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.040194+0000) 2019-10-13T14:45:42.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:42.043+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.047587+0000) 2019-10-13T14:45:42.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:42.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203762 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:42.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:42.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.654022+0000) 2019-10-13T14:45:42.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:42.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.715971+0000) 2019-10-13T14:45:42.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:42.903+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:42.909422+0000) 2019-10-13T14:45:43.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:43.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:43.393145+0000) 2019-10-13T14:45:43.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:43.639+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:43.642702+0000) 2019-10-13T14:45:45.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:45.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:45.665332+0000) 2019-10-13T14:45:45.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:45.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:45.677514+0000) 2019-10-13T14:45:47.053 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:47.046+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.050526+0000) 2019-10-13T14:45:47.053 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:47.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.051112+0000) 2019-10-13T14:45:47.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:47.050+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.051988+0000) 2019-10-13T14:45:47.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:47.050+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.052415+0000) 2019-10-13T14:45:47.055 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:47.050+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.053178+0000) 2019-10-13T14:45:47.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:47.050+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.053534+0000) 2019-10-13T14:45:47.056 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:47.051+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.054282+0000) 2019-10-13T14:45:47.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:47.055+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:47.059460+0000) 2019-10-13T14:45:47.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:47.058+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203797 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:48.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:48.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:48.915578+0000) 2019-10-13T14:45:52.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:52.062+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.064488+0000) 2019-10-13T14:45:52.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:52.062+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.065391+0000) 2019-10-13T14:45:52.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:52.062+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.065953+0000) 2019-10-13T14:45:52.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:52.062+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.066783+0000) 2019-10-13T14:45:52.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:52.062+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.066254+0000) 2019-10-13T14:45:52.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:52.063+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.067593+0000) 2019-10-13T14:45:52.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:52.066+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.067929+0000) 2019-10-13T14:45:52.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:52.067+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.071530+0000) 2019-10-13T14:45:52.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:52.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203828 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:52.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:52.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.654199+0000) 2019-10-13T14:45:52.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:52.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.716148+0000) 2019-10-13T14:45:52.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:52.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:52.915926+0000) 2019-10-13T14:45:53.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:53.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:53.393366+0000) 2019-10-13T14:45:53.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:53.639+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:53.642902+0000) 2019-10-13T14:45:55.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:55.662+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:55.665545+0000) 2019-10-13T14:45:55.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:55.674+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:55.677713+0000) 2019-10-13T14:45:57.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:57.074+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.078549+0000) 2019-10-13T14:45:57.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:45:57.074+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.079199+0000) 2019-10-13T14:45:57.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:45:57.078+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.079806+0000) 2019-10-13T14:45:57.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:45:57.078+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.080727+0000) 2019-10-13T14:45:57.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:45:57.078+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.080852+0000) 2019-10-13T14:45:57.084 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:45:57.078+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.081865+0000) 2019-10-13T14:45:57.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:45:57.079+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.082444+0000) 2019-10-13T14:45:57.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:45:57.079+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:57.084995+0000) 2019-10-13T14:45:57.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:45:57.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:45:58.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:45:58.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:58.915778+0000) 2019-10-13T14:45:59.903 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:45:59.903 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:59.907 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:45:59.907 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:02.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:02.087+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.091544+0000) 2019-10-13T14:46:02.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:02.090+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.092134+0000) 2019-10-13T14:46:02.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:02.090+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.093162+0000) 2019-10-13T14:46:02.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:02.090+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.093764+0000) 2019-10-13T14:46:02.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:02.090+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.094358+0000) 2019-10-13T14:46:02.097 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:02.090+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.094964+0000) 2019-10-13T14:46:02.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:02.094+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.095695+0000) 2019-10-13T14:46:02.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:02.095+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.099078+0000) 2019-10-13T14:46:02.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:02.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:02.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:02.650+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.654410+0000) 2019-10-13T14:46:02.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:02.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.716329+0000) 2019-10-13T14:46:02.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:02.923+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:02.927015+0000) 2019-10-13T14:46:03.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:03.390+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:03.393576+0000) 2019-10-13T14:46:03.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:03.639+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:03.643115+0000) 2019-10-13T14:46:05.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:05.661+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:05.665746+0000) 2019-10-13T14:46:05.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:05.673+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:05.677907+0000) 2019-10-13T14:46:07.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:07.101+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.105203+0000) 2019-10-13T14:46:07.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:07.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.105771+0000) 2019-10-13T14:46:07.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:07.101+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.106040+0000) 2019-10-13T14:46:07.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:07.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.106416+0000) 2019-10-13T14:46:07.109 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:07.103+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.107118+0000) 2019-10-13T14:46:07.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:07.105+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.107368+0000) 2019-10-13T14:46:07.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:07.105+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.108689+0000) 2019-10-13T14:46:07.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:07.107+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:07.110878+0000) 2019-10-13T14:46:07.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:07.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:08.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:08.913+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:08.915979+0000) 2019-10-13T14:46:12.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:12.113+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.118717+0000) 2019-10-13T14:46:12.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:12.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.119415+0000) 2019-10-13T14:46:12.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:12.117+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.119681+0000) 2019-10-13T14:46:12.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:12.117+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.120800+0000) 2019-10-13T14:46:12.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:12.117+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.121077+0000) 2019-10-13T14:46:12.124 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:12.117+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.121744+0000) 2019-10-13T14:46:12.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:12.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.121784+0000) 2019-10-13T14:46:12.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:12.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203960 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:12.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:12.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.131049+0000) 2019-10-13T14:46:12.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:12.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.654591+0000) 2019-10-13T14:46:12.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:12.713+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.716508+0000) 2019-10-13T14:46:12.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:12.931+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:12.936035+0000) 2019-10-13T14:46:13.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:13.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:13.393775+0000) 2019-10-13T14:46:13.645 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:13.639+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:13.643313+0000) 2019-10-13T14:46:15.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:15.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:15.665950+0000) 2019-10-13T14:46:15.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:15.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:15.678100+0000) 2019-10-13T14:46:17.134 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:17.129+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.131758+0000) 2019-10-13T14:46:17.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:17.129+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.132496+0000) 2019-10-13T14:46:17.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:17.127+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.132983+0000) 2019-10-13T14:46:17.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:17.129+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.133698+0000) 2019-10-13T14:46:17.137 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:17.129+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.134110+0000) 2019-10-13T14:46:17.137 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:17.129+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.134708+0000) 2019-10-13T14:46:17.137 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:17.133+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.135370+0000) 2019-10-13T14:46:17.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:17.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 203994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:17.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:17.139+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:17.142616+0000) 2019-10-13T14:46:18.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:18.913+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:18.916183+0000) 2019-10-13T14:46:22.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:22.139+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.145229+0000) 2019-10-13T14:46:22.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:22.141+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.145894+0000) 2019-10-13T14:46:22.149 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:22.141+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.146230+0000) 2019-10-13T14:46:22.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:22.141+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.146867+0000) 2019-10-13T14:46:22.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:22.141+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.147243+0000) 2019-10-13T14:46:22.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:22.145+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.147531+0000) 2019-10-13T14:46:22.151 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:22.145+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.149210+0000) 2019-10-13T14:46:22.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:22.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204027 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:22.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:22.151+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.156611+0000) 2019-10-13T14:46:22.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:22.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.654762+0000) 2019-10-13T14:46:22.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:22.713+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.716708+0000) 2019-10-13T14:46:22.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:22.943+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:22.947199+0000) 2019-10-13T14:46:23.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:23.389+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:23.393998+0000) 2019-10-13T14:46:23.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:23.643+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:23.643507+0000) 2019-10-13T14:46:25.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:25.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:25.666149+0000) 2019-10-13T14:46:25.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:25.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:25.678327+0000) 2019-10-13T14:46:27.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:27.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.159017+0000) 2019-10-13T14:46:27.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:27.155+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.159462+0000) 2019-10-13T14:46:27.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:27.157+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.160007+0000) 2019-10-13T14:46:27.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:27.157+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.160735+0000) 2019-10-13T14:46:27.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:27.157+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.161155+0000) 2019-10-13T14:46:27.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:27.157+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.161977+0000) 2019-10-13T14:46:27.164 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:27.157+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.162354+0000) 2019-10-13T14:46:27.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:27.163+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:27.167052+0000) 2019-10-13T14:46:27.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:27.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204060 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:28.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:28.913+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:28.916405+0000) 2019-10-13T14:46:30.043 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:46:30.043 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:30.047 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:46:30.047 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:32.175 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:32.169+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.172278+0000) 2019-10-13T14:46:32.175 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:32.169+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.173155+0000) 2019-10-13T14:46:32.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:32.166+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.173675+0000) 2019-10-13T14:46:32.176 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:32.169+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.174007+0000) 2019-10-13T14:46:32.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:32.169+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.174380+0000) 2019-10-13T14:46:32.177 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:32.169+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.175060+0000) 2019-10-13T14:46:32.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:32.173+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.176203+0000) 2019-10-13T14:46:32.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:32.178+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.182598+0000) 2019-10-13T14:46:32.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:32.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:32.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:32.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.654883+0000) 2019-10-13T14:46:32.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:32.713+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.716889+0000) 2019-10-13T14:46:32.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:32.954+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:32.958341+0000) 2019-10-13T14:46:33.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:33.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:33.394194+0000) 2019-10-13T14:46:33.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:33.638+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:33.643717+0000) 2019-10-13T14:46:35.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:35.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:35.666364+0000) 2019-10-13T14:46:35.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:35.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:35.678514+0000) 2019-10-13T14:46:37.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:37.185+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.187683+0000) 2019-10-13T14:46:37.191 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:37.185+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.188229+0000) 2019-10-13T14:46:37.191 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:37.182+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.189123+0000) 2019-10-13T14:46:37.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:37.185+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.189394+0000) 2019-10-13T14:46:37.192 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:37.185+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.189574+0000) 2019-10-13T14:46:37.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:37.185+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.190230+0000) 2019-10-13T14:46:37.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:37.185+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.190943+0000) 2019-10-13T14:46:37.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:37.190+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:37.195397+0000) 2019-10-13T14:46:37.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:37.193+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:38.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:38.913+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:38.916628+0000) 2019-10-13T14:46:42.205 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:42.197+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.202241+0000) 2019-10-13T14:46:42.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:42.197+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.203112+0000) 2019-10-13T14:46:42.205 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:42.198+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.203547+0000) 2019-10-13T14:46:42.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:42.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.203871+0000) 2019-10-13T14:46:42.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:42.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.204167+0000) 2019-10-13T14:46:42.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:42.201+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.204574+0000) 2019-10-13T14:46:42.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:42.201+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.204941+0000) 2019-10-13T14:46:42.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:42.206+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.211327+0000) 2019-10-13T14:46:42.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:42.209+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204159 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:42.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:42.649+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.655064+0000) 2019-10-13T14:46:42.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:42.713+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.717071+0000) 2019-10-13T14:46:42.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:42.962+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:42.964627+0000) 2019-10-13T14:46:43.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:43.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:43.394388+0000) 2019-10-13T14:46:43.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:43.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:43.643942+0000) 2019-10-13T14:46:45.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:45.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:45.666570+0000) 2019-10-13T14:46:45.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:45.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:45.678718+0000) 2019-10-13T14:46:47.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:47.210+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.214643+0000) 2019-10-13T14:46:47.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:47.209+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.214891+0000) 2019-10-13T14:46:47.218 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:47.213+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.215337+0000) 2019-10-13T14:46:47.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:47.213+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.216119+0000) 2019-10-13T14:46:47.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:47.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.216846+0000) 2019-10-13T14:46:47.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:47.213+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.217262+0000) 2019-10-13T14:46:47.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:47.213+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.218231+0000) 2019-10-13T14:46:47.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:47.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:47.224285+0000) 2019-10-13T14:46:47.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:47.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204191 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:48.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:48.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:48.916851+0000) 2019-10-13T14:46:52.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:52.224+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.228015+0000) 2019-10-13T14:46:52.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:52.224+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.228862+0000) 2019-10-13T14:46:52.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:52.224+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.229342+0000) 2019-10-13T14:46:52.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:52.224+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.229750+0000) 2019-10-13T14:46:52.233 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:52.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.230271+0000) 2019-10-13T14:46:52.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:52.224+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.230623+0000) 2019-10-13T14:46:52.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:52.224+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.231088+0000) 2019-10-13T14:46:52.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:52.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:52.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:52.234+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.239698+0000) 2019-10-13T14:46:52.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:52.652+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.655191+0000) 2019-10-13T14:46:52.719 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:52.716+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.717208+0000) 2019-10-13T14:46:52.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:52.970+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:52.970951+0000) 2019-10-13T14:46:53.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:53.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:53.394541+0000) 2019-10-13T14:46:53.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:53.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:53.644170+0000) 2019-10-13T14:46:55.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:55.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:55.666772+0000) 2019-10-13T14:46:55.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:55.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:55.678945+0000) 2019-10-13T14:46:57.244 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:46:57.238+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.242229+0000) 2019-10-13T14:46:57.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:46:57.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.242474+0000) 2019-10-13T14:46:57.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:57.240+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.243877+0000) 2019-10-13T14:46:57.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:46:57.240+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.244268+0000) 2019-10-13T14:46:57.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:46:57.240+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.244590+0000) 2019-10-13T14:46:57.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:46:57.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.245041+0000) 2019-10-13T14:46:57.248 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:46:57.240+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.245424+0000) 2019-10-13T14:46:57.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:46:57.242+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:57.248930+0000) 2019-10-13T14:46:57.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:46:57.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204257 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:46:58.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:46:58.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:58.917055+0000) 2019-10-13T14:47:00.169 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:47:00.169 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:00.173 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:47:00.173 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:02.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:02.252+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.256340+0000) 2019-10-13T14:47:02.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:02.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.257837+0000) 2019-10-13T14:47:02.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:02.252+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.258218+0000) 2019-10-13T14:47:02.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:02.254+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.259015+0000) 2019-10-13T14:47:02.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:02.252+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.259105+0000) 2019-10-13T14:47:02.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:02.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.259762+0000) 2019-10-13T14:47:02.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:02.256+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.260138+0000) 2019-10-13T14:47:02.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:02.258+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.265459+0000) 2019-10-13T14:47:02.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:02.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:02.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:02.652+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.655366+0000) 2019-10-13T14:47:02.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:02.716+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.717388+0000) 2019-10-13T14:47:02.984 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:02.978+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:02.981799+0000) 2019-10-13T14:47:03.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:03.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:03.394742+0000) 2019-10-13T14:47:03.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:03.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:03.644367+0000) 2019-10-13T14:47:05.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:05.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:05.667007+0000) 2019-10-13T14:47:05.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:05.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:05.679142+0000) 2019-10-13T14:47:07.274 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:07.268+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.271557+0000) 2019-10-13T14:47:07.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:07.266+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.272035+0000) 2019-10-13T14:47:07.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:07.268+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.272266+0000) 2019-10-13T14:47:07.275 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:07.268+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.272561+0000) 2019-10-13T14:47:07.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:07.268+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.273214+0000) 2019-10-13T14:47:07.276 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:07.268+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.273495+0000) 2019-10-13T14:47:07.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:07.268+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.274798+0000) 2019-10-13T14:47:07.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:07.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:07.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:07.278+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:07.283504+0000) 2019-10-13T14:47:08.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:08.916+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:08.917281+0000) 2019-10-13T14:47:12.287 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:12.280+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.284581+0000) 2019-10-13T14:47:12.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:12.280+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.284987+0000) 2019-10-13T14:47:12.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:12.280+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.285358+0000) 2019-10-13T14:47:12.289 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:12.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.287097+0000) 2019-10-13T14:47:12.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:12.284+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.287513+0000) 2019-10-13T14:47:12.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:12.284+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.287671+0000) 2019-10-13T14:47:12.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:12.284+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.288158+0000) 2019-10-13T14:47:12.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:12.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.292820+0000) 2019-10-13T14:47:12.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:12.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:12.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:12.652+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.655547+0000) 2019-10-13T14:47:12.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:12.716+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.717566+0000) 2019-10-13T14:47:12.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:12.986+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:12.988160+0000) 2019-10-13T14:47:13.397 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:13.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:13.394949+0000) 2019-10-13T14:47:13.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:13.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:13.644585+0000) 2019-10-13T14:47:15.669 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:15.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:15.667223+0000) 2019-10-13T14:47:15.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:15.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:15.679346+0000) 2019-10-13T14:47:17.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:17.292+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.298519+0000) 2019-10-13T14:47:17.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:17.292+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.298950+0000) 2019-10-13T14:47:17.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:17.294+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.299516+0000) 2019-10-13T14:47:17.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:17.296+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.299833+0000) 2019-10-13T14:47:17.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:17.296+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.300616+0000) 2019-10-13T14:47:17.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:17.296+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.301565+0000) 2019-10-13T14:47:17.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:17.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.302034+0000) 2019-10-13T14:47:17.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:17.302+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:17.307848+0000) 2019-10-13T14:47:17.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:17.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:18.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:18.916+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:18.917488+0000) 2019-10-13T14:47:22.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:22.308+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.311988+0000) 2019-10-13T14:47:22.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:22.308+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.312389+0000) 2019-10-13T14:47:22.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:22.308+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.313654+0000) 2019-10-13T14:47:22.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:22.308+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.314043+0000) 2019-10-13T14:47:22.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:22.314+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.314874+0000) 2019-10-13T14:47:22.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:22.308+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.315129+0000) 2019-10-13T14:47:22.318 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:22.312+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.316026+0000) 2019-10-13T14:47:22.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:22.316+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:22.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:22.322+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.324471+0000) 2019-10-13T14:47:22.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:22.652+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.655754+0000) 2019-10-13T14:47:22.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:22.712+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.717745+0000) 2019-10-13T14:47:23.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:22.998+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:22.999247+0000) 2019-10-13T14:47:23.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:23.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:23.395155+0000) 2019-10-13T14:47:23.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:23.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:23.644773+0000) 2019-10-13T14:47:25.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:25.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:25.667415+0000) 2019-10-13T14:47:25.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:25.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:25.679583+0000) 2019-10-13T14:47:27.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:27.320+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.326776+0000) 2019-10-13T14:47:27.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:27.324+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.327438+0000) 2019-10-13T14:47:27.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:27.324+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.327618+0000) 2019-10-13T14:47:27.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:27.324+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.328059+0000) 2019-10-13T14:47:27.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:27.324+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.328890+0000) 2019-10-13T14:47:27.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:27.324+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.329234+0000) 2019-10-13T14:47:27.332 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:27.329+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.330309+0000) 2019-10-13T14:47:27.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:27.333+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:27.334932+0000) 2019-10-13T14:47:27.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:27.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:28.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:28.912+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:28.917712+0000) 2019-10-13T14:47:30.473 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:47:30.473 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:30.477 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:47:30.477 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:32.344 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:32.335+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.341486+0000) 2019-10-13T14:47:32.344 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:32.341+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.342348+0000) 2019-10-13T14:47:32.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:32.335+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.342435+0000) 2019-10-13T14:47:32.345 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:32.335+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.343112+0000) 2019-10-13T14:47:32.346 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:32.339+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.343492+0000) 2019-10-13T14:47:32.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:32.339+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.345016+0000) 2019-10-13T14:47:32.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:32.339+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.345185+0000) 2019-10-13T14:47:32.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:32.345+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.348373+0000) 2019-10-13T14:47:32.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:32.347+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:32.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:32.651+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.655937+0000) 2019-10-13T14:47:32.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:32.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:32.717924+0000) 2019-10-13T14:47:33.008 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:33.001+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:33.005578+0000) 2019-10-13T14:47:33.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:33.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:33.395350+0000) 2019-10-13T14:47:33.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:33.641+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:33.644982+0000) 2019-10-13T14:47:35.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:35.663+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:35.667604+0000) 2019-10-13T14:47:35.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:35.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:35.679781+0000) 2019-10-13T14:47:37.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:37.351+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.357796+0000) 2019-10-13T14:47:37.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:37.351+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.357958+0000) 2019-10-13T14:47:37.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:37.357+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.358479+0000) 2019-10-13T14:47:37.361 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:37.351+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.358579+0000) 2019-10-13T14:47:37.362 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:37.351+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.358954+0000) 2019-10-13T14:47:37.362 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:37.355+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.359650+0000) 2019-10-13T14:47:37.362 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:37.351+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.359224+0000) 2019-10-13T14:47:37.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:37.361+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:37.362708+0000) 2019-10-13T14:47:37.369 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:37.359+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204521 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:38.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:38.915+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:38.917915+0000) 2019-10-13T14:47:42.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:42.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.370987+0000) 2019-10-13T14:47:42.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:42.363+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.371238+0000) 2019-10-13T14:47:42.374 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:42.367+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.371959+0000) 2019-10-13T14:47:42.375 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:42.367+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.372231+0000) 2019-10-13T14:47:42.375 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:42.367+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.373005+0000) 2019-10-13T14:47:42.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:42.369+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.373886+0000) 2019-10-13T14:47:42.376 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:42.367+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.374314+0000) 2019-10-13T14:47:42.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:42.373+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.376444+0000) 2019-10-13T14:47:42.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:42.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204553 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:42.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:42.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.656117+0000) 2019-10-13T14:47:42.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:42.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:42.718114+0000) 2019-10-13T14:47:43.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:43.009+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:43.011714+0000) 2019-10-13T14:47:43.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:43.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:43.395541+0000) 2019-10-13T14:47:43.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:43.641+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:43.645211+0000) 2019-10-13T14:47:45.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:45.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:45.667803+0000) 2019-10-13T14:47:45.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:45.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:45.679969+0000) 2019-10-13T14:47:47.387 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:47.379+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.385237+0000) 2019-10-13T14:47:47.388 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:47.381+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.385730+0000) 2019-10-13T14:47:47.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:47.379+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.386030+0000) 2019-10-13T14:47:47.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:47.379+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.386420+0000) 2019-10-13T14:47:47.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:47.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.387403+0000) 2019-10-13T14:47:47.390 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:47.383+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.387632+0000) 2019-10-13T14:47:47.390 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:47.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.387916+0000) 2019-10-13T14:47:47.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:47.389+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:47.393157+0000) 2019-10-13T14:47:47.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:47.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:48.920 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:48.915+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:48.918115+0000) 2019-10-13T14:47:52.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:52.391+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.398797+0000) 2019-10-13T14:47:52.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:52.397+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.399874+0000) 2019-10-13T14:47:52.402 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:52.395+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.400386+0000) 2019-10-13T14:47:52.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:52.395+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.400869+0000) 2019-10-13T14:47:52.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:52.395+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.401195+0000) 2019-10-13T14:47:52.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:52.395+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.401705+0000) 2019-10-13T14:47:52.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:52.395+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.402120+0000) 2019-10-13T14:47:52.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:52.401+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.406060+0000) 2019-10-13T14:47:52.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:52.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:52.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:52.651+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.656300+0000) 2019-10-13T14:47:52.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:52.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:52.718294+0000) 2019-10-13T14:47:53.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:53.017+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:53.017977+0000) 2019-10-13T14:47:53.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:53.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:53.395694+0000) 2019-10-13T14:47:53.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:53.641+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:53.645408+0000) 2019-10-13T14:47:55.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:55.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:55.667966+0000) 2019-10-13T14:47:55.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:55.675+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:55.680110+0000) 2019-10-13T14:47:57.414 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:47:57.407+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.411205+0000) 2019-10-13T14:47:57.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:47:57.407+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.412411+0000) 2019-10-13T14:47:57.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:47:57.407+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.412810+0000) 2019-10-13T14:47:57.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:57.407+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.413091+0000) 2019-10-13T14:47:57.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:47:57.407+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.413564+0000) 2019-10-13T14:47:57.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:47:57.407+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.413809+0000) 2019-10-13T14:47:57.416 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:47:57.413+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.414304+0000) 2019-10-13T14:47:57.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:47:57.415+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204653 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:47:57.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:47:57.417+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:57.421969+0000) 2019-10-13T14:47:58.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:47:58.915+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:58.918339+0000) 2019-10-13T14:48:00.637 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:48:00.637 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:00.641 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:48:00.641 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:02.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:02.423+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.424804+0000) 2019-10-13T14:48:02.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:02.423+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.425491+0000) 2019-10-13T14:48:02.428 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:02.423+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.425581+0000) 2019-10-13T14:48:02.429 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:02.423+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.426361+0000) 2019-10-13T14:48:02.429 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:02.425+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.426726+0000) 2019-10-13T14:48:02.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:02.423+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.427013+0000) 2019-10-13T14:48:02.430 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:02.427+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.427787+0000) 2019-10-13T14:48:02.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:02.433+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.434394+0000) 2019-10-13T14:48:02.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:02.431+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204685 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:02.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:02.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.656489+0000) 2019-10-13T14:48:02.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:02.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:02.718426+0000) 2019-10-13T14:48:03.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:03.021+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:03.022481+0000) 2019-10-13T14:48:03.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:03.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:03.395861+0000) 2019-10-13T14:48:03.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:03.641+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:03.645607+0000) 2019-10-13T14:48:05.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:05.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:05.668194+0000) 2019-10-13T14:48:05.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:05.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:05.680273+0000) 2019-10-13T14:48:07.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:07.435+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.437805+0000) 2019-10-13T14:48:07.440 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:07.435+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.438442+0000) 2019-10-13T14:48:07.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:07.437+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.438891+0000) 2019-10-13T14:48:07.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:07.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.439510+0000) 2019-10-13T14:48:07.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:07.439+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.439773+0000) 2019-10-13T14:48:07.443 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:07.439+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.440404+0000) 2019-10-13T14:48:07.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:07.439+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.440862+0000) 2019-10-13T14:48:07.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:07.441+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:07.446024+0000) 2019-10-13T14:48:07.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:07.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204719 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:08.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:08.915+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:08.918558+0000) 2019-10-13T14:48:12.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:12.451+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.451672+0000) 2019-10-13T14:48:12.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:12.451+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.452200+0000) 2019-10-13T14:48:12.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:12.449+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.452809+0000) 2019-10-13T14:48:12.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:12.451+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.453598+0000) 2019-10-13T14:48:12.456 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:12.451+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.453894+0000) 2019-10-13T14:48:12.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:12.451+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.454074+0000) 2019-10-13T14:48:12.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:12.451+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.454752+0000) 2019-10-13T14:48:12.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:12.457+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.459573+0000) 2019-10-13T14:48:12.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:12.459+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204750 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:12.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:12.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.656620+0000) 2019-10-13T14:48:12.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:12.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:12.718625+0000) 2019-10-13T14:48:13.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:13.025+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:13.028992+0000) 2019-10-13T14:48:13.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:13.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:13.396038+0000) 2019-10-13T14:48:13.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:13.641+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:13.645802+0000) 2019-10-13T14:48:15.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:15.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:15.668344+0000) 2019-10-13T14:48:15.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:15.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:15.680470+0000) 2019-10-13T14:48:17.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:17.462+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.465954+0000) 2019-10-13T14:48:17.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:17.462+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.466453+0000) 2019-10-13T14:48:17.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:17.462+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.466703+0000) 2019-10-13T14:48:17.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:17.465+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.467241+0000) 2019-10-13T14:48:17.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:17.466+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.467956+0000) 2019-10-13T14:48:17.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:17.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.468938+0000) 2019-10-13T14:48:17.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:17.466+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.469210+0000) 2019-10-13T14:48:17.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:17.469+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:17.474032+0000) 2019-10-13T14:48:17.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:17.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:18.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:18.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:18.918759+0000) 2019-10-13T14:48:22.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:22.478+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.480899+0000) 2019-10-13T14:48:22.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:22.478+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.481227+0000) 2019-10-13T14:48:22.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:22.478+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.481806+0000) 2019-10-13T14:48:22.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:22.478+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.482112+0000) 2019-10-13T14:48:22.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:22.480+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.482873+0000) 2019-10-13T14:48:22.486 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:22.482+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.483574+0000) 2019-10-13T14:48:22.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:22.482+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.484130+0000) 2019-10-13T14:48:22.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:22.484+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.489713+0000) 2019-10-13T14:48:22.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:22.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204817 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:22.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:22.654+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.656796+0000) 2019-10-13T14:48:22.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:22.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:22.718805+0000) 2019-10-13T14:48:23.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:23.032+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:23.035432+0000) 2019-10-13T14:48:23.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:23.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:23.396236+0000) 2019-10-13T14:48:23.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:23.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:23.645998+0000) 2019-10-13T14:48:25.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:25.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:25.668541+0000) 2019-10-13T14:48:25.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:25.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:25.680667+0000) 2019-10-13T14:48:27.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:27.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.495094+0000) 2019-10-13T14:48:27.498 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:27.494+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.495700+0000) 2019-10-13T14:48:27.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:27.494+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.496020+0000) 2019-10-13T14:48:27.498 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:27.494+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.496385+0000) 2019-10-13T14:48:27.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:27.494+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.496476+0000) 2019-10-13T14:48:27.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:27.494+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.497047+0000) 2019-10-13T14:48:27.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:27.492+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.497658+0000) 2019-10-13T14:48:27.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:27.500+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:27.503656+0000) 2019-10-13T14:48:27.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:27.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:28.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:28.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:28.918961+0000) 2019-10-13T14:48:30.748 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:48:30.748 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:30.752 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:48:30.752 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:32.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:32.506+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.507872+0000) 2019-10-13T14:48:32.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:32.506+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.508512+0000) 2019-10-13T14:48:32.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:32.506+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.508900+0000) 2019-10-13T14:48:32.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:32.506+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.509594+0000) 2019-10-13T14:48:32.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:32.506+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.510081+0000) 2019-10-13T14:48:32.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:32.506+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.510441+0000) 2019-10-13T14:48:32.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:32.508+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.511940+0000) 2019-10-13T14:48:32.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:32.512+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.517936+0000) 2019-10-13T14:48:32.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:32.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:32.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:32.654+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.656986+0000) 2019-10-13T14:48:32.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:32.714+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:32.718987+0000) 2019-10-13T14:48:33.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:33.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:33.046926+0000) 2019-10-13T14:48:33.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:33.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:33.396415+0000) 2019-10-13T14:48:33.648 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:33.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:33.646172+0000) 2019-10-13T14:48:35.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:35.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:35.668743+0000) 2019-10-13T14:48:35.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:35.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:35.680864+0000) 2019-10-13T14:48:37.524 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:37.518+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.521963+0000) 2019-10-13T14:48:37.525 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:37.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.522362+0000) 2019-10-13T14:48:37.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:37.518+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.522776+0000) 2019-10-13T14:48:37.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:37.520+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.523778+0000) 2019-10-13T14:48:37.526 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:37.522+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.523378+0000) 2019-10-13T14:48:37.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:37.522+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.524035+0000) 2019-10-13T14:48:37.527 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:37.522+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.525031+0000) 2019-10-13T14:48:37.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:37.528+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:37.531593+0000) 2019-10-13T14:48:37.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:37.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:38.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:38.914+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:38.919164+0000) 2019-10-13T14:48:42.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:42.534+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.536760+0000) 2019-10-13T14:48:42.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:42.532+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.537662+0000) 2019-10-13T14:48:42.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:42.534+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.537820+0000) 2019-10-13T14:48:42.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:42.534+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.538228+0000) 2019-10-13T14:48:42.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:42.534+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.538500+0000) 2019-10-13T14:48:42.541 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:42.534+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.539209+0000) 2019-10-13T14:48:42.542 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:42.538+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.539909+0000) 2019-10-13T14:48:42.548 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:42.542+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204948 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:42.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:42.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.546799+0000) 2019-10-13T14:48:42.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:42.654+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.657168+0000) 2019-10-13T14:48:42.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:42.718+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:42.719176+0000) 2019-10-13T14:48:43.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:43.056+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:43.058027+0000) 2019-10-13T14:48:43.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:43.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:43.396626+0000) 2019-10-13T14:48:43.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:43.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:43.646365+0000) 2019-10-13T14:48:45.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:45.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:45.668937+0000) 2019-10-13T14:48:45.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:45.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:45.681064+0000) 2019-10-13T14:48:47.552 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:47.546+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.549873+0000) 2019-10-13T14:48:47.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:47.546+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.550515+0000) 2019-10-13T14:48:47.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:47.548+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.551357+0000) 2019-10-13T14:48:47.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:47.550+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.551855+0000) 2019-10-13T14:48:47.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:47.550+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.552410+0000) 2019-10-13T14:48:47.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:47.550+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.552919+0000) 2019-10-13T14:48:47.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:47.550+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.553616+0000) 2019-10-13T14:48:47.560 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:47.552+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:47.557762+0000) 2019-10-13T14:48:47.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:47.558+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 204984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:48.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:48.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:48.919368+0000) 2019-10-13T14:48:52.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:52.562+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.563978+0000) 2019-10-13T14:48:52.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:52.562+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.564544+0000) 2019-10-13T14:48:52.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:52.562+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.566321+0000) 2019-10-13T14:48:52.569 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:52.562+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.566628+0000) 2019-10-13T14:48:52.569 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:52.564+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.567159+0000) 2019-10-13T14:48:52.570 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:52.566+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.567599+0000) 2019-10-13T14:48:52.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:52.566+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.568150+0000) 2019-10-13T14:48:52.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:52.568+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.571426+0000) 2019-10-13T14:48:52.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:52.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:52.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:52.654+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.657303+0000) 2019-10-13T14:48:52.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:52.718+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:52.719360+0000) 2019-10-13T14:48:53.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:53.060+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:53.064335+0000) 2019-10-13T14:48:53.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:53.394+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:53.396807+0000) 2019-10-13T14:48:53.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:53.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:53.646559+0000) 2019-10-13T14:48:55.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:55.666+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:55.669136+0000) 2019-10-13T14:48:55.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:55.678+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:55.681252+0000) 2019-10-13T14:48:57.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:48:57.574+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.577027+0000) 2019-10-13T14:48:57.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:48:57.574+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.577393+0000) 2019-10-13T14:48:57.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:48:57.574+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.577838+0000) 2019-10-13T14:48:57.580 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:48:57.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.578230+0000) 2019-10-13T14:48:57.581 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:48:57.576+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.578891+0000) 2019-10-13T14:48:57.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:48:57.578+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.579606+0000) 2019-10-13T14:48:57.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:57.578+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.579848+0000) 2019-10-13T14:48:57.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:48:57.580+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:57.585719+0000) 2019-10-13T14:48:57.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:48:57.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205050 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:48:58.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:48:58.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:58.919569+0000) 2019-10-13T14:49:00.866 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:49:00.867 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:00.870 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:49:00.870 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:02.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:02.584+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.589458+0000) 2019-10-13T14:49:02.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:02.585+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.590793+0000) 2019-10-13T14:49:02.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:02.585+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.590810+0000) 2019-10-13T14:49:02.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:02.585+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.591162+0000) 2019-10-13T14:49:02.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:02.589+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.592298+0000) 2019-10-13T14:49:02.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:02.589+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.593013+0000) 2019-10-13T14:49:02.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:02.589+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.593384+0000) 2019-10-13T14:49:02.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:02.592+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.596484+0000) 2019-10-13T14:49:02.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:02.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:02.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:02.653+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.657483+0000) 2019-10-13T14:49:02.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:02.717+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:02.719543+0000) 2019-10-13T14:49:03.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:03.072+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:03.073693+0000) 2019-10-13T14:49:03.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:03.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:03.397004+0000) 2019-10-13T14:49:03.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:03.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:03.646754+0000) 2019-10-13T14:49:05.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:05.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:05.669351+0000) 2019-10-13T14:49:05.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:05.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:05.681451+0000) 2019-10-13T14:49:07.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:07.601+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.604001+0000) 2019-10-13T14:49:07.607 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:07.601+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.604432+0000) 2019-10-13T14:49:07.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:07.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.604986+0000) 2019-10-13T14:49:07.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:07.601+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.605207+0000) 2019-10-13T14:49:07.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:07.601+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.605435+0000) 2019-10-13T14:49:07.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:07.601+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.606033+0000) 2019-10-13T14:49:07.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:07.601+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.606964+0000) 2019-10-13T14:49:07.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:07.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:07.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:07.612+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:07.614736+0000) 2019-10-13T14:49:08.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:08.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:08.919766+0000) 2019-10-13T14:49:12.621 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:12.613+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.618649+0000) 2019-10-13T14:49:12.621 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:12.613+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.619053+0000) 2019-10-13T14:49:12.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:12.613+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.619235+0000) 2019-10-13T14:49:12.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:12.617+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.619780+0000) 2019-10-13T14:49:12.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:12.617+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.619528+0000) 2019-10-13T14:49:12.623 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:12.616+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.620325+0000) 2019-10-13T14:49:12.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:12.617+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.620614+0000) 2019-10-13T14:49:12.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:12.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205145 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:12.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:12.624+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.629906+0000) 2019-10-13T14:49:12.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:12.653+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.657710+0000) 2019-10-13T14:49:12.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:12.717+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:12.719727+0000) 2019-10-13T14:49:13.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:13.076+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:13.079983+0000) 2019-10-13T14:49:13.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:13.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:13.397197+0000) 2019-10-13T14:49:13.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:13.644+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:13.646960+0000) 2019-10-13T14:49:15.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:15.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:15.669554+0000) 2019-10-13T14:49:15.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:15.677+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:15.681676+0000) 2019-10-13T14:49:17.633 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:17.627+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.630452+0000) 2019-10-13T14:49:17.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:17.625+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.631005+0000) 2019-10-13T14:49:17.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:17.629+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.631808+0000) 2019-10-13T14:49:17.634 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:17.629+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.632330+0000) 2019-10-13T14:49:17.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:17.629+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.632357+0000) 2019-10-13T14:49:17.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:17.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.632898+0000) 2019-10-13T14:49:17.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:17.629+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.633155+0000) 2019-10-13T14:49:17.643 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:17.637+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:17.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:17.639+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:17.643393+0000) 2019-10-13T14:49:18.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:18.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:18.920015+0000) 2019-10-13T14:49:22.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:22.639+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.644139+0000) 2019-10-13T14:49:22.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:22.641+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.644273+0000) 2019-10-13T14:49:22.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:22.641+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.644590+0000) 2019-10-13T14:49:22.648 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:22.641+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.645181+0000) 2019-10-13T14:49:22.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:22.641+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.646379+0000) 2019-10-13T14:49:22.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:22.641+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.646729+0000) 2019-10-13T14:49:22.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:22.645+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.647360+0000) 2019-10-13T14:49:22.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:22.647+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.653407+0000) 2019-10-13T14:49:22.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:22.649+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:22.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:22.653+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.657835+0000) 2019-10-13T14:49:22.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:22.717+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:22.719868+0000) 2019-10-13T14:49:23.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:23.083+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:23.086445+0000) 2019-10-13T14:49:23.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:23.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:23.397377+0000) 2019-10-13T14:49:23.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:23.643+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:23.647160+0000) 2019-10-13T14:49:25.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:25.665+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:25.669787+0000) 2019-10-13T14:49:25.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:25.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:25.681869+0000) 2019-10-13T14:49:27.660 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:27.653+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.657439+0000) 2019-10-13T14:49:27.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:27.653+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.658088+0000) 2019-10-13T14:49:27.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:27.653+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.658606+0000) 2019-10-13T14:49:27.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:27.655+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.659138+0000) 2019-10-13T14:49:27.662 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:27.657+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.659420+0000) 2019-10-13T14:49:27.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:27.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.659967+0000) 2019-10-13T14:49:27.663 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:27.657+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.660159+0000) 2019-10-13T14:49:27.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:27.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:27.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:27.667+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:27.671357+0000) 2019-10-13T14:49:28.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:28.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:28.920212+0000) 2019-10-13T14:49:31.004 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:49:31.005 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:31.008 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:49:31.008 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:32.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:32.653+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.658006+0000) 2019-10-13T14:49:32.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:32.665+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.670657+0000) 2019-10-13T14:49:32.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:32.665+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.671048+0000) 2019-10-13T14:49:32.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:32.669+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.672262+0000) 2019-10-13T14:49:32.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:32.669+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.672600+0000) 2019-10-13T14:49:32.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:32.669+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.673814+0000) 2019-10-13T14:49:32.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:32.671+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.674556+0000) 2019-10-13T14:49:32.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:32.669+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.674594+0000) 2019-10-13T14:49:32.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:32.675+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.679724+0000) 2019-10-13T14:49:32.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:32.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:32.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:32.717+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:32.720033+0000) 2019-10-13T14:49:33.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:33.095+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:33.097961+0000) 2019-10-13T14:49:33.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:33.393+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:33.397552+0000) 2019-10-13T14:49:33.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:33.643+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:33.647369+0000) 2019-10-13T14:49:35.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:35.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:35.669992+0000) 2019-10-13T14:49:35.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:35.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:35.682063+0000) 2019-10-13T14:49:37.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:37.681+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.684346+0000) 2019-10-13T14:49:37.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:37.681+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.685575+0000) 2019-10-13T14:49:37.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:37.681+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.685831+0000) 2019-10-13T14:49:37.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:37.681+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.686468+0000) 2019-10-13T14:49:37.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:37.681+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.686809+0000) 2019-10-13T14:49:37.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:37.685+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.687365+0000) 2019-10-13T14:49:37.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:37.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.688161+0000) 2019-10-13T14:49:37.693 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:37.687+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:37.691450+0000) 2019-10-13T14:49:37.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:37.689+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:38.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:38.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:38.920387+0000) 2019-10-13T14:49:42.660 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:42.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.658205+0000) 2019-10-13T14:49:42.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:42.692+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.698138+0000) 2019-10-13T14:49:42.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:42.692+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.698665+0000) 2019-10-13T14:49:42.701 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:42.692+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.699107+0000) 2019-10-13T14:49:42.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:42.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.699448+0000) 2019-10-13T14:49:42.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:42.695+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.700801+0000) 2019-10-13T14:49:42.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:42.696+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.701115+0000) 2019-10-13T14:49:42.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:42.696+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.702277+0000) 2019-10-13T14:49:42.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:42.703+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.708112+0000) 2019-10-13T14:49:42.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:42.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:42.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:42.716+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:42.720184+0000) 2019-10-13T14:49:43.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:43.103+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:43.104410+0000) 2019-10-13T14:49:43.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:43.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:43.397759+0000) 2019-10-13T14:49:43.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:43.643+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:43.647541+0000) 2019-10-13T14:49:45.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:45.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:45.670198+0000) 2019-10-13T14:49:45.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:45.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:45.682264+0000) 2019-10-13T14:49:47.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:47.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.711871+0000) 2019-10-13T14:49:47.717 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:47.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.713097+0000) 2019-10-13T14:49:47.717 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:47.708+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.713314+0000) 2019-10-13T14:49:47.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:47.708+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.713701+0000) 2019-10-13T14:49:47.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:47.711+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.714572+0000) 2019-10-13T14:49:47.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:47.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.714998+0000) 2019-10-13T14:49:47.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:47.712+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.715720+0000) 2019-10-13T14:49:47.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:47.715+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:47.721308+0000) 2019-10-13T14:49:47.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:47.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205380 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:48.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:48.916+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:48.920587+0000) 2019-10-13T14:49:52.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:52.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.658437+0000) 2019-10-13T14:49:52.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:52.716+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.720373+0000) 2019-10-13T14:49:52.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:52.720+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.725742+0000) 2019-10-13T14:49:52.728 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:52.720+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.725992+0000) 2019-10-13T14:49:52.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:52.720+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.726309+0000) 2019-10-13T14:49:52.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:52.724+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.727608+0000) 2019-10-13T14:49:52.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:52.724+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.728004+0000) 2019-10-13T14:49:52.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:52.724+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.728703+0000) 2019-10-13T14:49:52.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:52.723+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.729618+0000) 2019-10-13T14:49:52.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:52.731+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:52.734816+0000) 2019-10-13T14:49:52.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:52.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:53.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:53.107+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:53.110902+0000) 2019-10-13T14:49:53.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:53.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:53.397936+0000) 2019-10-13T14:49:53.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:53.647+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:53.647709+0000) 2019-10-13T14:49:55.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:55.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:55.670349+0000) 2019-10-13T14:49:55.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:55.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:55.682426+0000) 2019-10-13T14:49:57.742 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:49:57.736+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.740113+0000) 2019-10-13T14:49:57.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:49:57.736+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.740860+0000) 2019-10-13T14:49:57.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:49:57.736+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.741362+0000) 2019-10-13T14:49:57.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:49:57.736+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.742227+0000) 2019-10-13T14:49:57.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:49:57.736+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.742739+0000) 2019-10-13T14:49:57.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:57.736+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.742411+0000) 2019-10-13T14:49:57.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:49:57.739+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.743103+0000) 2019-10-13T14:49:57.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:49:57.743+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:57.748143+0000) 2019-10-13T14:49:57.752 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:49:57.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:49:58.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:49:58.916+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:58.920838+0000) 2019-10-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:02.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:02.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.658639+0000) 2019-10-13T14:50:02.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:02.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.720616+0000) 2019-10-13T14:50:02.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:02.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.754159+0000) 2019-10-13T14:50:02.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:02.748+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.754870+0000) 2019-10-13T14:50:02.757 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:02.751+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.755412+0000) 2019-10-13T14:50:02.758 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:02.752+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.755716+0000) 2019-10-13T14:50:02.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:02.752+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.756038+0000) 2019-10-13T14:50:02.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:02.752+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.757025+0000) 2019-10-13T14:50:02.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:02.752+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.756844+0000) 2019-10-13T14:50:02.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:02.759+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:02.762592+0000) 2019-10-13T14:50:02.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:02.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:03.119 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:03.115+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:03.117224+0000) 2019-10-13T14:50:03.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:03.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:03.398115+0000) 2019-10-13T14:50:03.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:03.643+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:03.647889+0000) 2019-10-13T14:50:05.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:05.668+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:05.670549+0000) 2019-10-13T14:50:05.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:05.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:05.682624+0000) 2019-10-13T14:50:07.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:07.764+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.767522+0000) 2019-10-13T14:50:07.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:07.764+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.768365+0000) 2019-10-13T14:50:07.771 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:07.763+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.769059+0000) 2019-10-13T14:50:07.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:07.764+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.769162+0000) 2019-10-13T14:50:07.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:07.764+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.769656+0000) 2019-10-13T14:50:07.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:07.764+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.770651+0000) 2019-10-13T14:50:07.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:07.768+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.771406+0000) 2019-10-13T14:50:07.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:07.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:07.776591+0000) 2019-10-13T14:50:07.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:07.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:08.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:08.916+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:08.921071+0000) 2019-10-13T14:50:12.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:12.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.658868+0000) 2019-10-13T14:50:12.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:12.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.720852+0000) 2019-10-13T14:50:12.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:12.776+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.781394+0000) 2019-10-13T14:50:12.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:12.776+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.781822+0000) 2019-10-13T14:50:12.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:12.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.782761+0000) 2019-10-13T14:50:12.785 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:12.778+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.783384+0000) 2019-10-13T14:50:12.786 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:12.780+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.784004+0000) 2019-10-13T14:50:12.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:12.780+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.784489+0000) 2019-10-13T14:50:12.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:12.780+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.785447+0000) 2019-10-13T14:50:12.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:12.786+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:12.790749+0000) 2019-10-13T14:50:12.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:12.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:13.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:13.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:13.128866+0000) 2019-10-13T14:50:13.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:13.392+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:13.398352+0000) 2019-10-13T14:50:13.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:13.646+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:13.648071+0000) 2019-10-13T14:50:15.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:15.664+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:15.670750+0000) 2019-10-13T14:50:15.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:15.676+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:15.682821+0000) 2019-10-13T14:50:17.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:17.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.795104+0000) 2019-10-13T14:50:17.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:17.792+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.795536+0000) 2019-10-13T14:50:17.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:17.792+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.796917+0000) 2019-10-13T14:50:17.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:17.792+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.797335+0000) 2019-10-13T14:50:17.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:17.792+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.797604+0000) 2019-10-13T14:50:17.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:17.792+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.798240+0000) 2019-10-13T14:50:17.801 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:17.794+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.798905+0000) 2019-10-13T14:50:17.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:17.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:17.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:17.802+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:17.806133+0000) 2019-10-13T14:50:18.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:18.920+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:18.921265+0000) 2019-10-13T14:50:22.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:22.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.659071+0000) 2019-10-13T14:50:22.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:22.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.721051+0000) 2019-10-13T14:50:22.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:22.804+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.808439+0000) 2019-10-13T14:50:22.811 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:22.804+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.808866+0000) 2019-10-13T14:50:22.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:22.804+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.809207+0000) 2019-10-13T14:50:22.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:22.804+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.809712+0000) 2019-10-13T14:50:22.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:22.804+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.810373+0000) 2019-10-13T14:50:22.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:22.804+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.810991+0000) 2019-10-13T14:50:22.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:22.806+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.812298+0000) 2019-10-13T14:50:22.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:22.810+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:22.817819+0000) 2019-10-13T14:50:22.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:22.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:23.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:23.134+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:23.135314+0000) 2019-10-13T14:50:23.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:23.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:23.398485+0000) 2019-10-13T14:50:23.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:23.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:23.648278+0000) 2019-10-13T14:50:25.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:25.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:25.670963+0000) 2019-10-13T14:50:25.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:25.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:25.683018+0000) 2019-10-13T14:50:27.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:27.819+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.823833+0000) 2019-10-13T14:50:27.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:27.819+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.823912+0000) 2019-10-13T14:50:27.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:27.819+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.824293+0000) 2019-10-13T14:50:27.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:27.819+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.824680+0000) 2019-10-13T14:50:27.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:27.819+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.824977+0000) 2019-10-13T14:50:27.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:27.819+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.825868+0000) 2019-10-13T14:50:27.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:27.822+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.826344+0000) 2019-10-13T14:50:27.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:27.826+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:27.833254+0000) 2019-10-13T14:50:27.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:27.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:28.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:28.919+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:28.921465+0000) 2019-10-13T14:50:31.177 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:50:31.177 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:31.181 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:50:31.181 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:32.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:32.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.659290+0000) 2019-10-13T14:50:32.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:32.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.721264+0000) 2019-10-13T14:50:32.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:32.831+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.837152+0000) 2019-10-13T14:50:32.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:32.831+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.837716+0000) 2019-10-13T14:50:32.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:32.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.838097+0000) 2019-10-13T14:50:32.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:32.831+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.838400+0000) 2019-10-13T14:50:32.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:32.831+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.839004+0000) 2019-10-13T14:50:32.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:32.834+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.839397+0000) 2019-10-13T14:50:32.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:32.835+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.839632+0000) 2019-10-13T14:50:32.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:32.838+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:32.843194+0000) 2019-10-13T14:50:32.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:32.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:33.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:33.138+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:33.141666+0000) 2019-10-13T14:50:33.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:33.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:33.398665+0000) 2019-10-13T14:50:33.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:33.646+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:33.648452+0000) 2019-10-13T14:50:35.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:35.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:35.671157+0000) 2019-10-13T14:50:35.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:35.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:35.683160+0000) 2019-10-13T14:50:37.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:37.847+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.850714+0000) 2019-10-13T14:50:37.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:37.847+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.851893+0000) 2019-10-13T14:50:37.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:37.847+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.852436+0000) 2019-10-13T14:50:37.855 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:37.847+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.853181+0000) 2019-10-13T14:50:37.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:37.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.853733+0000) 2019-10-13T14:50:37.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:37.847+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.854391+0000) 2019-10-13T14:50:37.857 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:37.847+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.853923+0000) 2019-10-13T14:50:37.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:37.854+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:37.859610+0000) 2019-10-13T14:50:37.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:37.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:38.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:38.919+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:38.921662+0000) 2019-10-13T14:50:42.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:42.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.659571+0000) 2019-10-13T14:50:42.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:42.715+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.721464+0000) 2019-10-13T14:50:42.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:42.859+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.865184+0000) 2019-10-13T14:50:42.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:42.859+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.865464+0000) 2019-10-13T14:50:42.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:42.859+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.866376+0000) 2019-10-13T14:50:42.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:42.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.866823+0000) 2019-10-13T14:50:42.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:42.862+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.867206+0000) 2019-10-13T14:50:42.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:42.863+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.867764+0000) 2019-10-13T14:50:42.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:42.863+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.868009+0000) 2019-10-13T14:50:42.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:42.870+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:42.875068+0000) 2019-10-13T14:50:42.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:42.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:43.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:43.146+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:43.147974+0000) 2019-10-13T14:50:43.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:43.391+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:43.398879+0000) 2019-10-13T14:50:43.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:43.642+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:43.648637+0000) 2019-10-13T14:50:45.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:45.667+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:45.671403+0000) 2019-10-13T14:50:45.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:45.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:45.683338+0000) 2019-10-13T14:50:47.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:47.875+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.878290+0000) 2019-10-13T14:50:47.882 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:47.875+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.879513+0000) 2019-10-13T14:50:47.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:47.875+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.879865+0000) 2019-10-13T14:50:47.883 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:47.875+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.880222+0000) 2019-10-13T14:50:47.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:47.874+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.880938+0000) 2019-10-13T14:50:47.883 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:47.875+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.881471+0000) 2019-10-13T14:50:47.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:47.875+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.881968+0000) 2019-10-13T14:50:47.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:47.882+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:47.888722+0000) 2019-10-13T14:50:47.891 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:47.883+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:48.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:48.919+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:48.921867+0000) 2019-10-13T14:50:52.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:52.659+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.659802+0000) 2019-10-13T14:50:52.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:52.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.721666+0000) 2019-10-13T14:50:52.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:52.887+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.892192+0000) 2019-10-13T14:50:52.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:52.886+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.892878+0000) 2019-10-13T14:50:52.895 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:52.887+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.893092+0000) 2019-10-13T14:50:52.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:52.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.894974+0000) 2019-10-13T14:50:52.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:52.887+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.894636+0000) 2019-10-13T14:50:52.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:52.891+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.895803+0000) 2019-10-13T14:50:52.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:52.891+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.896554+0000) 2019-10-13T14:50:52.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:52.898+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:52.902597+0000) 2019-10-13T14:50:52.905 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:52.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:53.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:53.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:53.154424+0000) 2019-10-13T14:50:53.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:53.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:53.399089+0000) 2019-10-13T14:50:53.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:53.646+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:53.648786+0000) 2019-10-13T14:50:55.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:55.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.671635+0000) 2019-10-13T14:50:55.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:55.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.683529+0000) 2019-10-13T14:50:57.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:50:57.903+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.906082+0000) 2019-10-13T14:50:57.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:57.903+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.906713+0000) 2019-10-13T14:50:57.909 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:50:57.907+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.907357+0000) 2019-10-13T14:50:57.911 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:50:57.907+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.908582+0000) 2019-10-13T14:50:57.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:50:57.906+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.908952+0000) 2019-10-13T14:50:57.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:50:57.907+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.909567+0000) 2019-10-13T14:50:57.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:50:57.907+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.910014+0000) 2019-10-13T14:50:57.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:50:57.914+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:57.914273+0000) 2019-10-13T14:50:57.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:50:57.915+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:50:58.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:50:58.919+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:58.922070+0000) 2019-10-13T14:51:01.296 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:51:01.296 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:01.300 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:51:01.300 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:02.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:02.659+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.660054+0000) 2019-10-13T14:51:02.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:02.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.721888+0000) 2019-10-13T14:51:02.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:02.919+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.920551+0000) 2019-10-13T14:51:02.923 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:02.919+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.920781+0000) 2019-10-13T14:51:02.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:02.919+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.921116+0000) 2019-10-13T14:51:02.924 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:02.918+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.921955+0000) 2019-10-13T14:51:02.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:02.919+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.921528+0000) 2019-10-13T14:51:02.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:02.919+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.922629+0000) 2019-10-13T14:51:02.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:02.919+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.923227+0000) 2019-10-13T14:51:02.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:02.926+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:02.928840+0000) 2019-10-13T14:51:02.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:02.931+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:03.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:03.158+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:03.160910+0000) 2019-10-13T14:51:03.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:03.395+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:03.399263+0000) 2019-10-13T14:51:03.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:03.645+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:03.648959+0000) 2019-10-13T14:51:05.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:05.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.671895+0000) 2019-10-13T14:51:05.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:05.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.683733+0000) 2019-10-13T14:51:07.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:07.933+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.934804+0000) 2019-10-13T14:51:07.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:07.931+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.935065+0000) 2019-10-13T14:51:07.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:07.935+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.936270+0000) 2019-10-13T14:51:07.939 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:07.935+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.936929+0000) 2019-10-13T14:51:07.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:07.935+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.937574+0000) 2019-10-13T14:51:07.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:07.935+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.937959+0000) 2019-10-13T14:51:07.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:07.935+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.938635+0000) 2019-10-13T14:51:07.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:07.941+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:07.942326+0000) 2019-10-13T14:51:07.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:07.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:08.924 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:08.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:08.922269+0000) 2019-10-13T14:51:12.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:12.658+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.660256+0000) 2019-10-13T14:51:12.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:12.718+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.722094+0000) 2019-10-13T14:51:12.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:12.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.949154+0000) 2019-10-13T14:51:12.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:12.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.950104+0000) 2019-10-13T14:51:12.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:12.946+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.950809+0000) 2019-10-13T14:51:12.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:12.949+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.951770+0000) 2019-10-13T14:51:12.954 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:12.950+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.952124+0000) 2019-10-13T14:51:12.955 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:12.950+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.952343+0000) 2019-10-13T14:51:12.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:12.950+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.952890+0000) 2019-10-13T14:51:12.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:12.953+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:12.956985+0000) 2019-10-13T14:51:12.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:12.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:13.170 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:13.165+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:13.167482+0000) 2019-10-13T14:51:13.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:13.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:13.399482+0000) 2019-10-13T14:51:13.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:13.645+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:13.649130+0000) 2019-10-13T14:51:15.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:15.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.672113+0000) 2019-10-13T14:51:15.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:15.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.683951+0000) 2019-10-13T14:51:17.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:17.962+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.964534+0000) 2019-10-13T14:51:17.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:17.961+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.964884+0000) 2019-10-13T14:51:17.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:17.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.965061+0000) 2019-10-13T14:51:17.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:17.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.965326+0000) 2019-10-13T14:51:17.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:17.962+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.965814+0000) 2019-10-13T14:51:17.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:17.962+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.966126+0000) 2019-10-13T14:51:17.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:17.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.966586+0000) 2019-10-13T14:51:17.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:17.969+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:17.970535+0000) 2019-10-13T14:51:17.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:17.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:18.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:18.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:18.922463+0000) 2019-10-13T14:51:22.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:22.658+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.660487+0000) 2019-10-13T14:51:22.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:22.718+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.722314+0000) 2019-10-13T14:51:22.980 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:22.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.977461+0000) 2019-10-13T14:51:22.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:22.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.977535+0000) 2019-10-13T14:51:22.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:22.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.978969+0000) 2019-10-13T14:51:22.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:22.978+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.979345+0000) 2019-10-13T14:51:22.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:22.977+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.980050+0000) 2019-10-13T14:51:22.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:22.978+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.980219+0000) 2019-10-13T14:51:22.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:22.978+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.981127+0000) 2019-10-13T14:51:22.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:22.985+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:22.987178+0000) 2019-10-13T14:51:22.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:22.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 205995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:23.176 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:23.173+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:23.173888+0000) 2019-10-13T14:51:23.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:23.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:23.399666+0000) 2019-10-13T14:51:23.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:23.645+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:23.649302+0000) 2019-10-13T14:51:25.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:25.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.672348+0000) 2019-10-13T14:51:25.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:25.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.684189+0000) 2019-10-13T14:51:27.995 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:27.990+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.992282+0000) 2019-10-13T14:51:27.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:27.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.992742+0000) 2019-10-13T14:51:27.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:27.993+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.994383+0000) 2019-10-13T14:51:27.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:27.990+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.994563+0000) 2019-10-13T14:51:27.997 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:27.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.995057+0000) 2019-10-13T14:51:27.998 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:27.994+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.995624+0000) 2019-10-13T14:51:27.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:27.994+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.996250+0000) 2019-10-13T14:51:28.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:27.997+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:27.999111+0000) 2019-10-13T14:51:28.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:27.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206028 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:28.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:28.922+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:28.922644+0000) 2019-10-13T14:51:31.410 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:51:31.411 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:31.414 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:51:31.414 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:32.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:32.658+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:32.660692+0000) 2019-10-13T14:51:32.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:32.722+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:32.722534+0000) 2019-10-13T14:51:33.009 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:33.002+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.006316+0000) 2019-10-13T14:51:33.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:33.002+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.006436+0000) 2019-10-13T14:51:33.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:33.002+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.007026+0000) 2019-10-13T14:51:33.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:33.006+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.007983+0000) 2019-10-13T14:51:33.010 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:33.006+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.008415+0000) 2019-10-13T14:51:33.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:33.005+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.009488+0000) 2019-10-13T14:51:33.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:33.006+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.009748+0000) 2019-10-13T14:51:33.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:33.013+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.015601+0000) 2019-10-13T14:51:33.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:33.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:33.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:33.177+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.180140+0000) 2019-10-13T14:51:33.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:33.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.399843+0000) 2019-10-13T14:51:33.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:33.645+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:33.649467+0000) 2019-10-13T14:51:35.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:35.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.672549+0000) 2019-10-13T14:51:35.686 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:35.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.684405+0000) 2019-10-13T14:51:38.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:38.014+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.018898+0000) 2019-10-13T14:51:38.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:38.018+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.020286+0000) 2019-10-13T14:51:38.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:38.018+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.020527+0000) 2019-10-13T14:51:38.023 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:38.018+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.020886+0000) 2019-10-13T14:51:38.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:38.017+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.021870+0000) 2019-10-13T14:51:38.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:38.018+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.021909+0000) 2019-10-13T14:51:38.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:38.018+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.022576+0000) 2019-10-13T14:51:38.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:38.026+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:38.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:38.029+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.030363+0000) 2019-10-13T14:51:38.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:38.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.922853+0000) 2019-10-13T14:51:42.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:42.658+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:42.660898+0000) 2019-10-13T14:51:42.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:42.718+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:42.722735+0000) 2019-10-13T14:51:43.034 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:43.030+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.031966+0000) 2019-10-13T14:51:43.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:43.029+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.033612+0000) 2019-10-13T14:51:43.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:43.030+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.034231+0000) 2019-10-13T14:51:43.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:43.030+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.034981+0000) 2019-10-13T14:51:43.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:43.030+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.035090+0000) 2019-10-13T14:51:43.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:43.034+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.035581+0000) 2019-10-13T14:51:43.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:43.034+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.035965+0000) 2019-10-13T14:51:43.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:43.037+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.039726+0000) 2019-10-13T14:51:43.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:43.038+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:43.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:43.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.190538+0000) 2019-10-13T14:51:43.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:43.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.400019+0000) 2019-10-13T14:51:43.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:43.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:43.649671+0000) 2019-10-13T14:51:45.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:45.670+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.672783+0000) 2019-10-13T14:51:45.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:45.682+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.684607+0000) 2019-10-13T14:51:48.049 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:48.042+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.046336+0000) 2019-10-13T14:51:48.049 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:48.042+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.046809+0000) 2019-10-13T14:51:48.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:48.046+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.047356+0000) 2019-10-13T14:51:48.050 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:48.045+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.048063+0000) 2019-10-13T14:51:48.051 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:48.046+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.048965+0000) 2019-10-13T14:51:48.051 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:48.046+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.049332+0000) 2019-10-13T14:51:48.052 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:48.046+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.050200+0000) 2019-10-13T14:51:48.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:48.053+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.055483+0000) 2019-10-13T14:51:48.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:48.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:48.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:48.918+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.923025+0000) 2019-10-13T14:51:52.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:52.657+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:52.661124+0000) 2019-10-13T14:51:52.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:52.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:52.722964+0000) 2019-10-13T14:51:53.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:53.057+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.060508+0000) 2019-10-13T14:51:53.064 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:53.057+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.061638+0000) 2019-10-13T14:51:53.064 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:53.057+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.062051+0000) 2019-10-13T14:51:53.065 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:53.057+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.062557+0000) 2019-10-13T14:51:53.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:53.057+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.062886+0000) 2019-10-13T14:51:53.066 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:53.061+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.063737+0000) 2019-10-13T14:51:53.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:53.057+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.063251+0000) 2019-10-13T14:51:53.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:53.065+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.067219+0000) 2019-10-13T14:51:53.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:53.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:53.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:53.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.201276+0000) 2019-10-13T14:51:53.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:53.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.400204+0000) 2019-10-13T14:51:53.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:53.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:53.649884+0000) 2019-10-13T14:51:55.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:55.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.672982+0000) 2019-10-13T14:51:55.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:55.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.684805+0000) 2019-10-13T14:51:58.076 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:58.069+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.073933+0000) 2019-10-13T14:51:58.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:51:58.069+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.074447+0000) 2019-10-13T14:51:58.077 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:51:58.069+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.074557+0000) 2019-10-13T14:51:58.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:51:58.073+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.075068+0000) 2019-10-13T14:51:58.078 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:51:58.073+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.076203+0000) 2019-10-13T14:51:58.079 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:51:58.073+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.076298+0000) 2019-10-13T14:51:58.079 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:51:58.073+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.076687+0000) 2019-10-13T14:51:58.085 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:51:58.081+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.083183+0000) 2019-10-13T14:51:58.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:51:58.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206225 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:51:58.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:51:58.917+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.923200+0000) 2019-10-13T14:52:01.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:52:01.535 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:01.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:52:01.539 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:02.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:02.657+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:02.661335+0000) 2019-10-13T14:52:02.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:02.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:02.723167+0000) 2019-10-13T14:52:03.090 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:03.081+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.087272+0000) 2019-10-13T14:52:03.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:03.084+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.087714+0000) 2019-10-13T14:52:03.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:03.085+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.088271+0000) 2019-10-13T14:52:03.091 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:03.085+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.088352+0000) 2019-10-13T14:52:03.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:03.085+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.089051+0000) 2019-10-13T14:52:03.092 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:03.085+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.089539+0000) 2019-10-13T14:52:03.092 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:03.085+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.090544+0000) 2019-10-13T14:52:03.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:03.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:03.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:03.092+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.097894+0000) 2019-10-13T14:52:03.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:03.204+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.207189+0000) 2019-10-13T14:52:03.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:03.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.400384+0000) 2019-10-13T14:52:03.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:03.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:03.650051+0000) 2019-10-13T14:52:05.675 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:05.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.673187+0000) 2019-10-13T14:52:05.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:05.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.685036+0000) 2019-10-13T14:52:08.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:08.097+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.100788+0000) 2019-10-13T14:52:08.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:08.097+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.100963+0000) 2019-10-13T14:52:08.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:08.097+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.101690+0000) 2019-10-13T14:52:08.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:08.097+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.102081+0000) 2019-10-13T14:52:08.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:08.097+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.102532+0000) 2019-10-13T14:52:08.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:08.100+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.102992+0000) 2019-10-13T14:52:08.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:08.101+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.103344+0000) 2019-10-13T14:52:08.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:08.108+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.110617+0000) 2019-10-13T14:52:08.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:08.105+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206290 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:08.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:08.921+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.923377+0000) 2019-10-13T14:52:12.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:12.657+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:12.661538+0000) 2019-10-13T14:52:12.725 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:12.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:12.723412+0000) 2019-10-13T14:52:13.116 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:13.108+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.113710+0000) 2019-10-13T14:52:13.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:13.109+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.114218+0000) 2019-10-13T14:52:13.117 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:13.109+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.114741+0000) 2019-10-13T14:52:13.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:13.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.115339+0000) 2019-10-13T14:52:13.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:13.113+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.115850+0000) 2019-10-13T14:52:13.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:13.113+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.116467+0000) 2019-10-13T14:52:13.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:13.113+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.116984+0000) 2019-10-13T14:52:13.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:13.116+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.121646+0000) 2019-10-13T14:52:13.126 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:13.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206324 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:13.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:13.212+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.213397+0000) 2019-10-13T14:52:13.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:13.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.400603+0000) 2019-10-13T14:52:13.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:13.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:13.650219+0000) 2019-10-13T14:52:15.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:15.669+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.673436+0000) 2019-10-13T14:52:15.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:15.681+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.685244+0000) 2019-10-13T14:52:18.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:18.125+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.127364+0000) 2019-10-13T14:52:18.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:18.125+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.128235+0000) 2019-10-13T14:52:18.131 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:18.125+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.128806+0000) 2019-10-13T14:52:18.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:18.125+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.128959+0000) 2019-10-13T14:52:18.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:18.125+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.130032+0000) 2019-10-13T14:52:18.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:18.128+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.130712+0000) 2019-10-13T14:52:18.133 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:18.125+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.130779+0000) 2019-10-13T14:52:18.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:18.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.136356+0000) 2019-10-13T14:52:18.140 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:18.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:18.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:18.921+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.923558+0000) 2019-10-13T14:52:22.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:22.657+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:22.661741+0000) 2019-10-13T14:52:22.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:22.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:22.723615+0000) 2019-10-13T14:52:23.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:23.136+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.142074+0000) 2019-10-13T14:52:23.145 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:23.137+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.142454+0000) 2019-10-13T14:52:23.145 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:23.137+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.142942+0000) 2019-10-13T14:52:23.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:23.141+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.143390+0000) 2019-10-13T14:52:23.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:23.141+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.144239+0000) 2019-10-13T14:52:23.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:23.141+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.144360+0000) 2019-10-13T14:52:23.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:23.141+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.144850+0000) 2019-10-13T14:52:23.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:23.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206389 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:23.154 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:23.148+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.152055+0000) 2019-10-13T14:52:23.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:23.216+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.219664+0000) 2019-10-13T14:52:23.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:23.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.400784+0000) 2019-10-13T14:52:23.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:23.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:23.650409+0000) 2019-10-13T14:52:25.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:25.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.673634+0000) 2019-10-13T14:52:25.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:25.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.685447+0000) 2019-10-13T14:52:28.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:28.153+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.156146+0000) 2019-10-13T14:52:28.159 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:28.152+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.156674+0000) 2019-10-13T14:52:28.159 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:28.153+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.156794+0000) 2019-10-13T14:52:28.160 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:28.153+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.157147+0000) 2019-10-13T14:52:28.160 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:28.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.157573+0000) 2019-10-13T14:52:28.160 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:28.153+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.157708+0000) 2019-10-13T14:52:28.161 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:28.153+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.158448+0000) 2019-10-13T14:52:28.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:28.160+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.162230+0000) 2019-10-13T14:52:28.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:28.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:28.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:28.921+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.923733+0000) 2019-10-13T14:52:31.707 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:52:31.707 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:31.711 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:52:31.711 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:32.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:32.661+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:32.661974+0000) 2019-10-13T14:52:32.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:32.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:32.723814+0000) 2019-10-13T14:52:33.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:33.165+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.169207+0000) 2019-10-13T14:52:33.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:33.164+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.169872+0000) 2019-10-13T14:52:33.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:33.165+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.170736+0000) 2019-10-13T14:52:33.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:33.165+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.171016+0000) 2019-10-13T14:52:33.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:33.169+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.171675+0000) 2019-10-13T14:52:33.175 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:33.169+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.172758+0000) 2019-10-13T14:52:33.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:33.169+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.173042+0000) 2019-10-13T14:52:33.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:33.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206456 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:33.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:33.180+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.182931+0000) 2019-10-13T14:52:33.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:33.224+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.228776+0000) 2019-10-13T14:52:33.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:33.397+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.400982+0000) 2019-10-13T14:52:33.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:33.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:33.650534+0000) 2019-10-13T14:52:35.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:35.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.673829+0000) 2019-10-13T14:52:35.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:35.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.685641+0000) 2019-10-13T14:52:38.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:38.180+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.182899+0000) 2019-10-13T14:52:38.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:38.180+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.184029+0000) 2019-10-13T14:52:38.187 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:38.180+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.185106+0000) 2019-10-13T14:52:38.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:38.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.185159+0000) 2019-10-13T14:52:38.188 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:38.180+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.185580+0000) 2019-10-13T14:52:38.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:38.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.187257+0000) 2019-10-13T14:52:38.190 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:38.184+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.187537+0000) 2019-10-13T14:52:38.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:38.188+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.191310+0000) 2019-10-13T14:52:38.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:38.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:38.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:38.920+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.923929+0000) 2019-10-13T14:52:42.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:42.660+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:42.662208+0000) 2019-10-13T14:52:42.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:42.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:42.724035+0000) 2019-10-13T14:52:43.200 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:43.192+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.197587+0000) 2019-10-13T14:52:43.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:43.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.198871+0000) 2019-10-13T14:52:43.202 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:43.196+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.199645+0000) 2019-10-13T14:52:43.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:43.196+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.199962+0000) 2019-10-13T14:52:43.202 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:43.196+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.200114+0000) 2019-10-13T14:52:43.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:43.196+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.200981+0000) 2019-10-13T14:52:43.203 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:43.196+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.201424+0000) 2019-10-13T14:52:43.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:43.204+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.206762+0000) 2019-10-13T14:52:43.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:43.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:43.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:43.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.235124+0000) 2019-10-13T14:52:43.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:43.396+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.401123+0000) 2019-10-13T14:52:43.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:43.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:43.650652+0000) 2019-10-13T14:52:45.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:45.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.674030+0000) 2019-10-13T14:52:45.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:45.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.685840+0000) 2019-10-13T14:52:48.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:48.208+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.212058+0000) 2019-10-13T14:52:48.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:48.208+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.212662+0000) 2019-10-13T14:52:48.215 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:48.208+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.213285+0000) 2019-10-13T14:52:48.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:48.208+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.213536+0000) 2019-10-13T14:52:48.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:48.208+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.214102+0000) 2019-10-13T14:52:48.217 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:48.208+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.215218+0000) 2019-10-13T14:52:48.218 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:48.212+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.216415+0000) 2019-10-13T14:52:48.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:48.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:48.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:48.220+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.222776+0000) 2019-10-13T14:52:48.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:48.920+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.924122+0000) 2019-10-13T14:52:52.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:52.656+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:52.662401+0000) 2019-10-13T14:52:52.726 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:52.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:52.724231+0000) 2019-10-13T14:52:53.228 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:53.220+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.225349+0000) 2019-10-13T14:52:53.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:53.220+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.226876+0000) 2019-10-13T14:52:53.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:53.223+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.227107+0000) 2019-10-13T14:52:53.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:53.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.227701+0000) 2019-10-13T14:52:53.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:53.224+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.228116+0000) 2019-10-13T14:52:53.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:53.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.228571+0000) 2019-10-13T14:52:53.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:53.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.229244+0000) 2019-10-13T14:52:53.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:53.231+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.235424+0000) 2019-10-13T14:52:53.238 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:53.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:53.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:53.235+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.240993+0000) 2019-10-13T14:52:53.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:53.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.401349+0000) 2019-10-13T14:52:53.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:53.647+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:53.650780+0000) 2019-10-13T14:52:55.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:55.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.674225+0000) 2019-10-13T14:52:55.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:55.680+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.686068+0000) 2019-10-13T14:52:58.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:58.236+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.240219+0000) 2019-10-13T14:52:58.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:52:58.236+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.240484+0000) 2019-10-13T14:52:58.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:52:58.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.241069+0000) 2019-10-13T14:52:58.244 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:52:58.235+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.241749+0000) 2019-10-13T14:52:58.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:52:58.236+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.242855+0000) 2019-10-13T14:52:58.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:52:58.236+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.242967+0000) 2019-10-13T14:52:58.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:52:58.240+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.243467+0000) 2019-10-13T14:52:58.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:52:58.243+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.249030+0000) 2019-10-13T14:52:58.252 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:52:58.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:52:58.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:52:58.920+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.924304+0000) 2019-10-13T14:53:01.972 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:53:01.973 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:01.976 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:53:01.976 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:02.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:02.660+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:02.662623+0000) 2019-10-13T14:53:02.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:02.720+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:02.724430+0000) 2019-10-13T14:53:03.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:03.239+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.244578+0000) 2019-10-13T14:53:03.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:03.248+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.253390+0000) 2019-10-13T14:53:03.256 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:03.248+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.253693+0000) 2019-10-13T14:53:03.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:03.248+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.254361+0000) 2019-10-13T14:53:03.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:03.248+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.254954+0000) 2019-10-13T14:53:03.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:03.251+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.255265+0000) 2019-10-13T14:53:03.258 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:03.252+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.255576+0000) 2019-10-13T14:53:03.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:03.252+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.255770+0000) 2019-10-13T14:53:03.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:03.251+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.257457+0000) 2019-10-13T14:53:03.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:03.256+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:03.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:03.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.401489+0000) 2019-10-13T14:53:03.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:03.647+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:03.650949+0000) 2019-10-13T14:53:05.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:05.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.674427+0000) 2019-10-13T14:53:05.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:05.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.686268+0000) 2019-10-13T14:53:08.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:08.260+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.264873+0000) 2019-10-13T14:53:08.268 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:08.260+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.265766+0000) 2019-10-13T14:53:08.268 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:08.260+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.266214+0000) 2019-10-13T14:53:08.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:08.263+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.266743+0000) 2019-10-13T14:53:08.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:08.260+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.267137+0000) 2019-10-13T14:53:08.270 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:08.264+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.267942+0000) 2019-10-13T14:53:08.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:08.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.268586+0000) 2019-10-13T14:53:08.274 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:08.267+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.272088+0000) 2019-10-13T14:53:08.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:08.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:08.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:08.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.924505+0000) 2019-10-13T14:53:12.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:12.660+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:12.662872+0000) 2019-10-13T14:53:12.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:12.724+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:12.724669+0000) 2019-10-13T14:53:13.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:13.247+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.251474+0000) 2019-10-13T14:53:13.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:13.276+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.279442+0000) 2019-10-13T14:53:13.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:13.275+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.279957+0000) 2019-10-13T14:53:13.283 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:13.276+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.280695+0000) 2019-10-13T14:53:13.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:13.276+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.281800+0000) 2019-10-13T14:53:13.284 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:13.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.282172+0000) 2019-10-13T14:53:13.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:13.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.282826+0000) 2019-10-13T14:53:13.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:13.276+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.282856+0000) 2019-10-13T14:53:13.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:13.283+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.288123+0000) 2019-10-13T14:53:13.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:13.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206718 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:13.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:13.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.401675+0000) 2019-10-13T14:53:13.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:13.647+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:13.651116+0000) 2019-10-13T14:53:15.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:15.672+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.674627+0000) 2019-10-13T14:53:15.688 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:15.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.686420+0000) 2019-10-13T14:53:18.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:18.287+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.293475+0000) 2019-10-13T14:53:18.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:18.288+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.294010+0000) 2019-10-13T14:53:18.297 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:18.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.294485+0000) 2019-10-13T14:53:18.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:18.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.294126+0000) 2019-10-13T14:53:18.297 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:18.288+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.295044+0000) 2019-10-13T14:53:18.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:18.292+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.295393+0000) 2019-10-13T14:53:18.298 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:18.292+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.296301+0000) 2019-10-13T14:53:18.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:18.299+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.302933+0000) 2019-10-13T14:53:18.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:18.300+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206752 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:18.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:18.923+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.924682+0000) 2019-10-13T14:53:22.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:22.655+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:22.663040+0000) 2019-10-13T14:53:22.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:22.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:22.724864+0000) 2019-10-13T14:53:23.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:23.259+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.262673+0000) 2019-10-13T14:53:23.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:23.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.306867+0000) 2019-10-13T14:53:23.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:23.303+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.307593+0000) 2019-10-13T14:53:23.310 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:23.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.308188+0000) 2019-10-13T14:53:23.311 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:23.303+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.308506+0000) 2019-10-13T14:53:23.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:23.303+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.309555+0000) 2019-10-13T14:53:23.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:23.303+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.309946+0000) 2019-10-13T14:53:23.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:23.303+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.310624+0000) 2019-10-13T14:53:23.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:23.311+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.315408+0000) 2019-10-13T14:53:23.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:23.311+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206783 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:23.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:23.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.401844+0000) 2019-10-13T14:53:23.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:23.647+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:23.651255+0000) 2019-10-13T14:53:25.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:25.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.674856+0000) 2019-10-13T14:53:25.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:25.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.686644+0000) 2019-10-13T14:53:28.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:28.315+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.320520+0000) 2019-10-13T14:53:28.324 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:28.315+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.321728+0000) 2019-10-13T14:53:28.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:28.315+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.322588+0000) 2019-10-13T14:53:28.325 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:28.315+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.322813+0000) 2019-10-13T14:53:28.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:28.315+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.323127+0000) 2019-10-13T14:53:28.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:28.319+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.324413+0000) 2019-10-13T14:53:28.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:28.319+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.324860+0000) 2019-10-13T14:53:28.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:28.323+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.327914+0000) 2019-10-13T14:53:28.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:28.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206816 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:28.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:28.919+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.924859+0000) 2019-10-13T14:53:32.077 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:53:32.077 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:32.081 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:53:32.081 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:32.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:32.659+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:32.663258+0000) 2019-10-13T14:53:32.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:32.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:32.725060+0000) 2019-10-13T14:53:33.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:33.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.269149+0000) 2019-10-13T14:53:33.336 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:33.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.333684+0000) 2019-10-13T14:53:33.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:33.327+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.334446+0000) 2019-10-13T14:53:33.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:33.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.334884+0000) 2019-10-13T14:53:33.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:33.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.335712+0000) 2019-10-13T14:53:33.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:33.331+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.335888+0000) 2019-10-13T14:53:33.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:33.331+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.336199+0000) 2019-10-13T14:53:33.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:33.331+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.336959+0000) 2019-10-13T14:53:33.345 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:33.339+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.343043+0000) 2019-10-13T14:53:33.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:33.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206848 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:33.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:33.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.402049+0000) 2019-10-13T14:53:33.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:33.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:33.651459+0000) 2019-10-13T14:53:35.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:35.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.675091+0000) 2019-10-13T14:53:35.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:35.679+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.686836+0000) 2019-10-13T14:53:38.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:38.343+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.348067+0000) 2019-10-13T14:53:38.351 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:38.343+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.348623+0000) 2019-10-13T14:53:38.352 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:38.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.349556+0000) 2019-10-13T14:53:38.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:38.343+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.349253+0000) 2019-10-13T14:53:38.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:38.347+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.350567+0000) 2019-10-13T14:53:38.353 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:38.343+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.350204+0000) 2019-10-13T14:53:38.354 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:38.343+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.349914+0000) 2019-10-13T14:53:38.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:38.351+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.357553+0000) 2019-10-13T14:53:38.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:38.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206882 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:38.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:38.923+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.925061+0000) 2019-10-13T14:53:42.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:42.663+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:42.663534+0000) 2019-10-13T14:53:42.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:42.719+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:42.725300+0000) 2019-10-13T14:53:43.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:43.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.280018+0000) 2019-10-13T14:53:43.363 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:43.355+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.361184+0000) 2019-10-13T14:53:43.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:43.355+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.362486+0000) 2019-10-13T14:53:43.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:43.355+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.362973+0000) 2019-10-13T14:53:43.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:43.355+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.362585+0000) 2019-10-13T14:53:43.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:43.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.363410+0000) 2019-10-13T14:53:43.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:43.359+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.363746+0000) 2019-10-13T14:53:43.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:43.359+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.363991+0000) 2019-10-13T14:53:43.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:43.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206913 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:43.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:43.367+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.373203+0000) 2019-10-13T14:53:43.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:43.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.402189+0000) 2019-10-13T14:53:43.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:43.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:43.651628+0000) 2019-10-13T14:53:45.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:45.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.675295+0000) 2019-10-13T14:53:45.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:45.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.687028+0000) 2019-10-13T14:53:48.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:48.367+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.374365+0000) 2019-10-13T14:53:48.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:48.367+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.374987+0000) 2019-10-13T14:53:48.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:48.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.375386+0000) 2019-10-13T14:53:48.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:48.375+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.375797+0000) 2019-10-13T14:53:48.378 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:48.375+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.376352+0000) 2019-10-13T14:53:48.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:48.375+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.377211+0000) 2019-10-13T14:53:48.381 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:48.374+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.378731+0000) 2019-10-13T14:53:48.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:48.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206946 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:48.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:48.378+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.384607+0000) 2019-10-13T14:53:48.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:48.923+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.925238+0000) 2019-10-13T14:53:52.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:52.663+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:52.663733+0000) 2019-10-13T14:53:52.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:52.723+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:52.725538+0000) 2019-10-13T14:53:53.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:53.282+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.286376+0000) 2019-10-13T14:53:53.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:53.387+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.388106+0000) 2019-10-13T14:53:53.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:53.387+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.388144+0000) 2019-10-13T14:53:53.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:53.387+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.388893+0000) 2019-10-13T14:53:53.392 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:53.387+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.389221+0000) 2019-10-13T14:53:53.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:53.387+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.389156+0000) 2019-10-13T14:53:53.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:53.387+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.390394+0000) 2019-10-13T14:53:53.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:53.386+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.391107+0000) 2019-10-13T14:53:53.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:53.390+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.395881+0000) 2019-10-13T14:53:53.399 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:53.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 206978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:53.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:53.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.402319+0000) 2019-10-13T14:53:53.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:53.650+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:53.651791+0000) 2019-10-13T14:53:55.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:55.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.675551+0000) 2019-10-13T14:53:55.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:55.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.687248+0000) 2019-10-13T14:53:58.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:53:58.399+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.400822+0000) 2019-10-13T14:53:58.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:53:58.399+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.401263+0000) 2019-10-13T14:53:58.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:53:58.399+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.401604+0000) 2019-10-13T14:53:58.404 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:53:58.399+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.402128+0000) 2019-10-13T14:53:58.405 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:53:58.399+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.402243+0000) 2019-10-13T14:53:58.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:53:58.398+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.403507+0000) 2019-10-13T14:53:58.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:58.403+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.403557+0000) 2019-10-13T14:53:58.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:53:58.406+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.410467+0000) 2019-10-13T14:53:58.413 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:53:58.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207012 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:53:58.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:53:58.923+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.925448+0000) 2019-10-13T14:54:02.210 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:54:02.210 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:02.214 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:54:02.214 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:02.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:02.662+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:02.663952+0000) 2019-10-13T14:54:02.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:02.722+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:02.725781+0000) 2019-10-13T14:54:03.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:03.290+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.292666+0000) 2019-10-13T14:54:03.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:03.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.402490+0000) 2019-10-13T14:54:03.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:03.410+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.414495+0000) 2019-10-13T14:54:03.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:03.410+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.414981+0000) 2019-10-13T14:54:03.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:03.414+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.415930+0000) 2019-10-13T14:54:03.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:03.414+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.415717+0000) 2019-10-13T14:54:03.419 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:03.410+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.417032+0000) 2019-10-13T14:54:03.419 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:03.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.416451+0000) 2019-10-13T14:54:03.420 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:03.414+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.416531+0000) 2019-10-13T14:54:03.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:03.418+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.422198+0000) 2019-10-13T14:54:03.427 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:03.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:03.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:03.650+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:03.651933+0000) 2019-10-13T14:54:05.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:05.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.675764+0000) 2019-10-13T14:54:05.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:05.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.687518+0000) 2019-10-13T14:54:08.431 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:08.422+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.429100+0000) 2019-10-13T14:54:08.432 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:08.426+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.429723+0000) 2019-10-13T14:54:08.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:08.426+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.429775+0000) 2019-10-13T14:54:08.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:08.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.430458+0000) 2019-10-13T14:54:08.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:08.426+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.430653+0000) 2019-10-13T14:54:08.434 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:08.430+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.431470+0000) 2019-10-13T14:54:08.434 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:08.430+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.432082+0000) 2019-10-13T14:54:08.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:08.430+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.436643+0000) 2019-10-13T14:54:08.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:08.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:08.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:08.922+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.925660+0000) 2019-10-13T14:54:12.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:12.662+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:12.664149+0000) 2019-10-13T14:54:12.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:12.722+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:12.726002+0000) 2019-10-13T14:54:13.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:13.298+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.299015+0000) 2019-10-13T14:54:13.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:13.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.402683+0000) 2019-10-13T14:54:13.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:13.442+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.443575+0000) 2019-10-13T14:54:13.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:13.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.444285+0000) 2019-10-13T14:54:13.447 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:13.442+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.444055+0000) 2019-10-13T14:54:13.448 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:13.438+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.444942+0000) 2019-10-13T14:54:13.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:13.442+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.445134+0000) 2019-10-13T14:54:13.448 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:13.442+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.445453+0000) 2019-10-13T14:54:13.449 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:13.442+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.446748+0000) 2019-10-13T14:54:13.456 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:13.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207108 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:13.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:13.450+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.455369+0000) 2019-10-13T14:54:13.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:13.650+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:13.652097+0000) 2019-10-13T14:54:15.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:15.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.675964+0000) 2019-10-13T14:54:15.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:15.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.687712+0000) 2019-10-13T14:54:18.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:18.454+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.457446+0000) 2019-10-13T14:54:18.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:18.454+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.457857+0000) 2019-10-13T14:54:18.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:18.454+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.458995+0000) 2019-10-13T14:54:18.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:18.458+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.459547+0000) 2019-10-13T14:54:18.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:18.458+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.459909+0000) 2019-10-13T14:54:18.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:18.458+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.460259+0000) 2019-10-13T14:54:18.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:18.458+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.460944+0000) 2019-10-13T14:54:18.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:18.462+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.467931+0000) 2019-10-13T14:54:18.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:18.466+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:18.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:18.922+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.925839+0000) 2019-10-13T14:54:22.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:22.662+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:22.664345+0000) 2019-10-13T14:54:22.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:22.722+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:22.726195+0000) 2019-10-13T14:54:23.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:23.298+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.305313+0000) 2019-10-13T14:54:23.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:23.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.402873+0000) 2019-10-13T14:54:23.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:23.466+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.470816+0000) 2019-10-13T14:54:23.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:23.470+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.471470+0000) 2019-10-13T14:54:23.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:23.470+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.471580+0000) 2019-10-13T14:54:23.474 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:23.470+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.472392+0000) 2019-10-13T14:54:23.476 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:23.470+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.473411+0000) 2019-10-13T14:54:23.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:23.466+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.473096+0000) 2019-10-13T14:54:23.476 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:23.470+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.474472+0000) 2019-10-13T14:54:23.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:23.474+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.480497+0000) 2019-10-13T14:54:23.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:23.478+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207173 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:23.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:23.650+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:23.652266+0000) 2019-10-13T14:54:25.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:25.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.676171+0000) 2019-10-13T14:54:25.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:25.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.687970+0000) 2019-10-13T14:54:28.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:28.482+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.485165+0000) 2019-10-13T14:54:28.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:28.482+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.486798+0000) 2019-10-13T14:54:28.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:28.482+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.487293+0000) 2019-10-13T14:54:28.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:28.486+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.487529+0000) 2019-10-13T14:54:28.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:28.486+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.487875+0000) 2019-10-13T14:54:28.491 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:28.486+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.488726+0000) 2019-10-13T14:54:28.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:28.486+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.488865+0000) 2019-10-13T14:54:28.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:28.486+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.492838+0000) 2019-10-13T14:54:28.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:28.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207206 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:28.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:28.922+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.926006+0000) 2019-10-13T14:54:32.267 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:54:32.267 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:32.271 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:54:32.271 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:32.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:32.662+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:32.664552+0000) 2019-10-13T14:54:32.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:32.726+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:32.726431+0000) 2019-10-13T14:54:33.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:33.310+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.311593+0000) 2019-10-13T14:54:33.405 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:33.398+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.403063+0000) 2019-10-13T14:54:33.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:33.494+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.498088+0000) 2019-10-13T14:54:33.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:33.494+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.499670+0000) 2019-10-13T14:54:33.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:33.498+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.499944+0000) 2019-10-13T14:54:33.504 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:33.498+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.500736+0000) 2019-10-13T14:54:33.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:33.498+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.501215+0000) 2019-10-13T14:54:33.504 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:33.498+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.502040+0000) 2019-10-13T14:54:33.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:33.498+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.502320+0000) 2019-10-13T14:54:33.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:33.498+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.503915+0000) 2019-10-13T14:54:33.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:33.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:33.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:33.650+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:33.652456+0000) 2019-10-13T14:54:35.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:35.674+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.676379+0000) 2019-10-13T14:54:35.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:35.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.688183+0000) 2019-10-13T14:54:38.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:38.510+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.512641+0000) 2019-10-13T14:54:38.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:38.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.513769+0000) 2019-10-13T14:54:38.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:38.510+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.514321+0000) 2019-10-13T14:54:38.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:38.510+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.514525+0000) 2019-10-13T14:54:38.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:38.510+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.514820+0000) 2019-10-13T14:54:38.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:38.510+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.515139+0000) 2019-10-13T14:54:38.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:38.514+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.515557+0000) 2019-10-13T14:54:38.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:38.518+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.518889+0000) 2019-10-13T14:54:38.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:38.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:38.928 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:38.926+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.926203+0000) 2019-10-13T14:54:42.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:42.662+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:42.664745+0000) 2019-10-13T14:54:42.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:42.722+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:42.726616+0000) 2019-10-13T14:54:43.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:43.313+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.317828+0000) 2019-10-13T14:54:43.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:43.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.403268+0000) 2019-10-13T14:54:43.528 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:43.522+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.526027+0000) 2019-10-13T14:54:43.529 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:43.522+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.526575+0000) 2019-10-13T14:54:43.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:43.522+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.527266+0000) 2019-10-13T14:54:43.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:43.526+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.527688+0000) 2019-10-13T14:54:43.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:43.526+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.528420+0000) 2019-10-13T14:54:43.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:43.526+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.529032+0000) 2019-10-13T14:54:43.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:43.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.529773+0000) 2019-10-13T14:54:43.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:43.533+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.536257+0000) 2019-10-13T14:54:43.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:43.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:43.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:43.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:43.652640+0000) 2019-10-13T14:54:45.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:45.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.676580+0000) 2019-10-13T14:54:45.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:45.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.688378+0000) 2019-10-13T14:54:48.542 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:48.537+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.540216+0000) 2019-10-13T14:54:48.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:48.537+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.541288+0000) 2019-10-13T14:54:48.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:48.537+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.542126+0000) 2019-10-13T14:54:48.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:48.537+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.542814+0000) 2019-10-13T14:54:48.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:48.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.542925+0000) 2019-10-13T14:54:48.545 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:48.541+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.543569+0000) 2019-10-13T14:54:48.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:48.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.543970+0000) 2019-10-13T14:54:48.549 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:48.545+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.547472+0000) 2019-10-13T14:54:48.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:48.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:48.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:48.925+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.926377+0000) 2019-10-13T14:54:52.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:52.661+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:52.664942+0000) 2019-10-13T14:54:52.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:52.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:52.726812+0000) 2019-10-13T14:54:53.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:53.321+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.324289+0000) 2019-10-13T14:54:53.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:53.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.403486+0000) 2019-10-13T14:54:53.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:53.549+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.554222+0000) 2019-10-13T14:54:53.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:53.549+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.554834+0000) 2019-10-13T14:54:53.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:53.553+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.555474+0000) 2019-10-13T14:54:53.558 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:53.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.555916+0000) 2019-10-13T14:54:53.558 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:53.553+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.556442+0000) 2019-10-13T14:54:53.560 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:53.553+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.557884+0000) 2019-10-13T14:54:53.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:53.553+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.558437+0000) 2019-10-13T14:54:53.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:53.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:53.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:53.565+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.566903+0000) 2019-10-13T14:54:53.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:53.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:53.652841+0000) 2019-10-13T14:54:55.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:55.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.676779+0000) 2019-10-13T14:54:55.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:55.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.688570+0000) 2019-10-13T14:54:58.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:54:58.565+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.568861+0000) 2019-10-13T14:54:58.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:54:58.565+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.569456+0000) 2019-10-13T14:54:58.573 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:54:58.565+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.570212+0000) 2019-10-13T14:54:58.573 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:54:58.569+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.571292+0000) 2019-10-13T14:54:58.574 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:54:58.565+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.570966+0000) 2019-10-13T14:54:58.574 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:58.569+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.571431+0000) 2019-10-13T14:54:58.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:54:58.569+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.571751+0000) 2019-10-13T14:54:58.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:54:58.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207401 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:54:58.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:54:58.577+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.580016+0000) 2019-10-13T14:54:58.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:54:58.921+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.926588+0000) 2019-10-13T14:55:02.390 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:55:02.390 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:02.394 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:55:02.394 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:02.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:02.661+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:02.665145+0000) 2019-10-13T14:55:02.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:02.721+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:02.727007+0000) 2019-10-13T14:55:03.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:03.329+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.330720+0000) 2019-10-13T14:55:03.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:03.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.403678+0000) 2019-10-13T14:55:03.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:03.577+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.581879+0000) 2019-10-13T14:55:03.585 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:03.577+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.582536+0000) 2019-10-13T14:55:03.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:03.577+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.582866+0000) 2019-10-13T14:55:03.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:03.581+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.583547+0000) 2019-10-13T14:55:03.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:03.581+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.584860+0000) 2019-10-13T14:55:03.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:03.581+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.584921+0000) 2019-10-13T14:55:03.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:03.581+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.585264+0000) 2019-10-13T14:55:03.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:03.589+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.590991+0000) 2019-10-13T14:55:03.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:03.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207433 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:03.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:03.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:03.653034+0000) 2019-10-13T14:55:05.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:05.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.676981+0000) 2019-10-13T14:55:05.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:05.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.688783+0000) 2019-10-13T14:55:08.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:08.593+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.595606+0000) 2019-10-13T14:55:08.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:08.593+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.596800+0000) 2019-10-13T14:55:08.599 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:08.593+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.597264+0000) 2019-10-13T14:55:08.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:08.593+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.597814+0000) 2019-10-13T14:55:08.600 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:08.593+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.598054+0000) 2019-10-13T14:55:08.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:08.593+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.598476+0000) 2019-10-13T14:55:08.601 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:08.593+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.599115+0000) 2019-10-13T14:55:08.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:08.601+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.605885+0000) 2019-10-13T14:55:08.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:08.605+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:08.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:08.925+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.926783+0000) 2019-10-13T14:55:12.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:12.661+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:12.665354+0000) 2019-10-13T14:55:12.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:12.725+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:12.727228+0000) 2019-10-13T14:55:13.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:13.333+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.337064+0000) 2019-10-13T14:55:13.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:13.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.403826+0000) 2019-10-13T14:55:13.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:13.609+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.611787+0000) 2019-10-13T14:55:13.615 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:13.609+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.612214+0000) 2019-10-13T14:55:13.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:13.609+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.613216+0000) 2019-10-13T14:55:13.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:13.609+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.613505+0000) 2019-10-13T14:55:13.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:13.609+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.613791+0000) 2019-10-13T14:55:13.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:13.609+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.614729+0000) 2019-10-13T14:55:13.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:13.613+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.615481+0000) 2019-10-13T14:55:13.621 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:13.617+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.619444+0000) 2019-10-13T14:55:13.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:13.617+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:13.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:13.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:13.653186+0000) 2019-10-13T14:55:15.679 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:15.673+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.677224+0000) 2019-10-13T14:55:15.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:15.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.689020+0000) 2019-10-13T14:55:18.627 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:18.621+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.624815+0000) 2019-10-13T14:55:18.628 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:18.625+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.626204+0000) 2019-10-13T14:55:18.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:18.621+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.626821+0000) 2019-10-13T14:55:18.629 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:18.621+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.627043+0000) 2019-10-13T14:55:18.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:18.625+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.627426+0000) 2019-10-13T14:55:18.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:18.625+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.628442+0000) 2019-10-13T14:55:18.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:18.625+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.628032+0000) 2019-10-13T14:55:18.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:18.633+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:18.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:18.637+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.638933+0000) 2019-10-13T14:55:18.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:18.921+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.926960+0000) 2019-10-13T14:55:22.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:22.661+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:22.665585+0000) 2019-10-13T14:55:22.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:22.725+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:22.727482+0000) 2019-10-13T14:55:23.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:23.345+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.348923+0000) 2019-10-13T14:55:23.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:23.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.404043+0000) 2019-10-13T14:55:23.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:23.637+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.640112+0000) 2019-10-13T14:55:23.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:23.637+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.641146+0000) 2019-10-13T14:55:23.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:23.637+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.641817+0000) 2019-10-13T14:55:23.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:23.637+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.642199+0000) 2019-10-13T14:55:23.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:23.637+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.642752+0000) 2019-10-13T14:55:23.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:23.637+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.643016+0000) 2019-10-13T14:55:23.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:23.641+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.643460+0000) 2019-10-13T14:55:23.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:23.645+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.646341+0000) 2019-10-13T14:55:23.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:23.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:23.655 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:23.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:23.653318+0000) 2019-10-13T14:55:25.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:25.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.677451+0000) 2019-10-13T14:55:25.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:25.685+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.689267+0000) 2019-10-13T14:55:28.655 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:28.648+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.653130+0000) 2019-10-13T14:55:28.656 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:28.648+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.653759+0000) 2019-10-13T14:55:28.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:28.648+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.653999+0000) 2019-10-13T14:55:28.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:28.648+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.654363+0000) 2019-10-13T14:55:28.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:28.648+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.655165+0000) 2019-10-13T14:55:28.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:28.652+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.656121+0000) 2019-10-13T14:55:28.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:28.653+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.656811+0000) 2019-10-13T14:55:28.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:28.657+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.661158+0000) 2019-10-13T14:55:28.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:28.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:28.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:28.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.927136+0000) 2019-10-13T14:55:32.514 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:55:32.514 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:32.518 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:55:32.518 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:32.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:32.664+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:32.665793+0000) 2019-10-13T14:55:32.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:32.724+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:32.727689+0000) 2019-10-13T14:55:33.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:33.357+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.360247+0000) 2019-10-13T14:55:33.406 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:33.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.404239+0000) 2019-10-13T14:55:33.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:33.649+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.653480+0000) 2019-10-13T14:55:33.670 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:33.664+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.668136+0000) 2019-10-13T14:55:33.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:33.664+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.668653+0000) 2019-10-13T14:55:33.671 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:33.664+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.669100+0000) 2019-10-13T14:55:33.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:33.665+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.670083+0000) 2019-10-13T14:55:33.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:33.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.669869+0000) 2019-10-13T14:55:33.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:33.664+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.670456+0000) 2019-10-13T14:55:33.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:33.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.671443+0000) 2019-10-13T14:55:33.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:33.673+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:33.677331+0000) 2019-10-13T14:55:33.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:33.672+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:35.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:35.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.677688+0000) 2019-10-13T14:55:35.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:35.684+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.689480+0000) 2019-10-13T14:55:38.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:38.676+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.682685+0000) 2019-10-13T14:55:38.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:38.680+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.683412+0000) 2019-10-13T14:55:38.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:38.680+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.683754+0000) 2019-10-13T14:55:38.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:38.680+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.684490+0000) 2019-10-13T14:55:38.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:38.680+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.685317+0000) 2019-10-13T14:55:38.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:38.680+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.685392+0000) 2019-10-13T14:55:38.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:38.680+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.685923+0000) 2019-10-13T14:55:38.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:38.688+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.692994+0000) 2019-10-13T14:55:38.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:38.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:38.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:38.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.927342+0000) 2019-10-13T14:55:42.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:42.664+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:42.665995+0000) 2019-10-13T14:55:42.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:42.724+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:42.727921+0000) 2019-10-13T14:55:43.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:43.364+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.370066+0000) 2019-10-13T14:55:43.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:43.400+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.404454+0000) 2019-10-13T14:55:43.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:43.648+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.653676+0000) 2019-10-13T14:55:43.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:43.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.696872+0000) 2019-10-13T14:55:43.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:43.692+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.697549+0000) 2019-10-13T14:55:43.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:43.692+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.698337+0000) 2019-10-13T14:55:43.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:43.692+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.698693+0000) 2019-10-13T14:55:43.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:43.692+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.699024+0000) 2019-10-13T14:55:43.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:43.696+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.700205+0000) 2019-10-13T14:55:43.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:43.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.699743+0000) 2019-10-13T14:55:43.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:43.704+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:43.706691+0000) 2019-10-13T14:55:43.710 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:43.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:45.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:45.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.677887+0000) 2019-10-13T14:55:45.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:45.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.689676+0000) 2019-10-13T14:55:48.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:48.708+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.711590+0000) 2019-10-13T14:55:48.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:48.708+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.712292+0000) 2019-10-13T14:55:48.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:48.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.712949+0000) 2019-10-13T14:55:48.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:48.708+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.713080+0000) 2019-10-13T14:55:48.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:48.708+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.714169+0000) 2019-10-13T14:55:48.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:48.712+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.714558+0000) 2019-10-13T14:55:48.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:48.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.714752+0000) 2019-10-13T14:55:48.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:48.716+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.718709+0000) 2019-10-13T14:55:48.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:48.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207728 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:48.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:48.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.927513+0000) 2019-10-13T14:55:52.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:52.664+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:52.666193+0000) 2019-10-13T14:55:52.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:52.724+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:52.728128+0000) 2019-10-13T14:55:53.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:53.372+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.376716+0000) 2019-10-13T14:55:53.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:53.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.404668+0000) 2019-10-13T14:55:53.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:53.652+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.653928+0000) 2019-10-13T14:55:53.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:53.720+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.725540+0000) 2019-10-13T14:55:53.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:53.720+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.725762+0000) 2019-10-13T14:55:53.729 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:53.720+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.726628+0000) 2019-10-13T14:55:53.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:53.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.727866+0000) 2019-10-13T14:55:53.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:53.724+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.727856+0000) 2019-10-13T14:55:53.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:53.724+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.728272+0000) 2019-10-13T14:55:53.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:53.724+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.729127+0000) 2019-10-13T14:55:53.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:53.728+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:53.733292+0000) 2019-10-13T14:55:53.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:53.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:55.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:55.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.678104+0000) 2019-10-13T14:55:55.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:55.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.689874+0000) 2019-10-13T14:55:58.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:55:58.736+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.739814+0000) 2019-10-13T14:55:58.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:55:58.736+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.740396+0000) 2019-10-13T14:55:58.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:55:58.736+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.741921+0000) 2019-10-13T14:55:58.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:55:58.736+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.742376+0000) 2019-10-13T14:55:58.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:55:58.736+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.742116+0000) 2019-10-13T14:55:58.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:55:58.736+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.742745+0000) 2019-10-13T14:55:58.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:58.740+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.743525+0000) 2019-10-13T14:55:58.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:55:58.744+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.747546+0000) 2019-10-13T14:55:58.751 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:55:58.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207795 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:55:58.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:55:58.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.927696+0000) 2019-10-13T14:56:02.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:02.664+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:02.666397+0000) 2019-10-13T14:56:02.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:02.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:02.728328+0000) 2019-10-13T14:56:02.737 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:56:02.737 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:02.741 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:56:02.741 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:03.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:03.384+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.387843+0000) 2019-10-13T14:56:03.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:03.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.404861+0000) 2019-10-13T14:56:03.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:03.652+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.654165+0000) 2019-10-13T14:56:03.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:03.748+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.752731+0000) 2019-10-13T14:56:03.756 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:03.748+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.753366+0000) 2019-10-13T14:56:03.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:03.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.753910+0000) 2019-10-13T14:56:03.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:03.748+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.754426+0000) 2019-10-13T14:56:03.757 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:03.752+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.754918+0000) 2019-10-13T14:56:03.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:03.748+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.755151+0000) 2019-10-13T14:56:03.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:03.752+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.756052+0000) 2019-10-13T14:56:03.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:03.760+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:03.762494+0000) 2019-10-13T14:56:03.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:03.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:05.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:05.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.678331+0000) 2019-10-13T14:56:05.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:05.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.690073+0000) 2019-10-13T14:56:08.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:08.760+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.766859+0000) 2019-10-13T14:56:08.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:08.764+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.767748+0000) 2019-10-13T14:56:08.770 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:08.764+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.767495+0000) 2019-10-13T14:56:08.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:08.764+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.768096+0000) 2019-10-13T14:56:08.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:08.764+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.768537+0000) 2019-10-13T14:56:08.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:08.764+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.769542+0000) 2019-10-13T14:56:08.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:08.764+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.770043+0000) 2019-10-13T14:56:08.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:08.772+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.775533+0000) 2019-10-13T14:56:08.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:08.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:08.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:08.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.927872+0000) 2019-10-13T14:56:12.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:12.659+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:12.666593+0000) 2019-10-13T14:56:12.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:12.723+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:12.728554+0000) 2019-10-13T14:56:13.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:13.392+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.394330+0000) 2019-10-13T14:56:13.407 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:13.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.405079+0000) 2019-10-13T14:56:13.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:13.652+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.654412+0000) 2019-10-13T14:56:13.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:13.775+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.781429+0000) 2019-10-13T14:56:13.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:13.775+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.782051+0000) 2019-10-13T14:56:13.787 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:13.780+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.783081+0000) 2019-10-13T14:56:13.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:13.775+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.782907+0000) 2019-10-13T14:56:13.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:13.779+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.783360+0000) 2019-10-13T14:56:13.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:13.779+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.783660+0000) 2019-10-13T14:56:13.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:13.779+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.785100+0000) 2019-10-13T14:56:13.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:13.788+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:13.790227+0000) 2019-10-13T14:56:13.794 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:13.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:15.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:15.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.678528+0000) 2019-10-13T14:56:15.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:15.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.690273+0000) 2019-10-13T14:56:18.799 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:18.792+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.796427+0000) 2019-10-13T14:56:18.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:18.791+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.796262+0000) 2019-10-13T14:56:18.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:18.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.796772+0000) 2019-10-13T14:56:18.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:18.791+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.797161+0000) 2019-10-13T14:56:18.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:18.791+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.797529+0000) 2019-10-13T14:56:18.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:18.791+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.797779+0000) 2019-10-13T14:56:18.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:18.791+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.799012+0000) 2019-10-13T14:56:18.804 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:18.800+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.802368+0000) 2019-10-13T14:56:18.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:18.799+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207929 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:18.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:18.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.928081+0000) 2019-10-13T14:56:22.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:22.663+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:22.666794+0000) 2019-10-13T14:56:22.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:22.723+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:22.728779+0000) 2019-10-13T14:56:23.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:23.399+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.405274+0000) 2019-10-13T14:56:23.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:23.404+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.406195+0000) 2019-10-13T14:56:23.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:23.652+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.654602+0000) 2019-10-13T14:56:23.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:23.803+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.809638+0000) 2019-10-13T14:56:23.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:23.803+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.810703+0000) 2019-10-13T14:56:23.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:23.808+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.810209+0000) 2019-10-13T14:56:23.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:23.803+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.811246+0000) 2019-10-13T14:56:23.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:23.807+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.811666+0000) 2019-10-13T14:56:23.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:23.807+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.812895+0000) 2019-10-13T14:56:23.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:23.807+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.813354+0000) 2019-10-13T14:56:23.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:23.815+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:23.825 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:23.820+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:23.823245+0000) 2019-10-13T14:56:25.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:25.671+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.678729+0000) 2019-10-13T14:56:25.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:25.683+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.690466+0000) 2019-10-13T14:56:28.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:28.819+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.823357+0000) 2019-10-13T14:56:28.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:28.819+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.823537+0000) 2019-10-13T14:56:28.826 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:28.820+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.824050+0000) 2019-10-13T14:56:28.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:28.819+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.824831+0000) 2019-10-13T14:56:28.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:28.819+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.825367+0000) 2019-10-13T14:56:28.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:28.819+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.826128+0000) 2019-10-13T14:56:28.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:28.819+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.826610+0000) 2019-10-13T14:56:28.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:28.828+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.831979+0000) 2019-10-13T14:56:28.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:28.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 207995 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:28.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:28.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.928264+0000) 2019-10-13T14:56:32.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:32.663+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:32.666991+0000) 2019-10-13T14:56:32.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:32.727+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:32.729006+0000) 2019-10-13T14:56:32.844 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:56:32.844 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:32.848 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:56:32.848 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:33.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:33.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.405467+0000) 2019-10-13T14:56:33.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:33.411+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.412807+0000) 2019-10-13T14:56:33.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:33.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.654834+0000) 2019-10-13T14:56:33.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:33.831+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.837041+0000) 2019-10-13T14:56:33.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:33.831+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.837264+0000) 2019-10-13T14:56:33.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:33.831+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.837642+0000) 2019-10-13T14:56:33.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:33.831+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.838592+0000) 2019-10-13T14:56:33.841 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:33.835+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.839296+0000) 2019-10-13T14:56:33.842 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:33.831+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.838919+0000) 2019-10-13T14:56:33.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:33.835+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.839910+0000) 2019-10-13T14:56:33.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:33.843+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:33.846972+0000) 2019-10-13T14:56:33.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:33.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:35.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:35.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.678960+0000) 2019-10-13T14:56:35.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:35.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.690632+0000) 2019-10-13T14:56:38.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:38.843+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.850340+0000) 2019-10-13T14:56:38.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:38.847+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.851472+0000) 2019-10-13T14:56:38.854 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:38.847+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.851702+0000) 2019-10-13T14:56:38.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:38.847+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.852693+0000) 2019-10-13T14:56:38.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:38.847+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.853044+0000) 2019-10-13T14:56:38.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:38.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.853666+0000) 2019-10-13T14:56:38.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:38.847+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.853876+0000) 2019-10-13T14:56:38.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:38.855+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.859920+0000) 2019-10-13T14:56:38.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:38.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208063 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:38.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:38.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.928472+0000) 2019-10-13T14:56:42.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:42.663+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:42.667194+0000) 2019-10-13T14:56:42.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:42.727+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:42.729213+0000) 2019-10-13T14:56:43.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:43.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.405658+0000) 2019-10-13T14:56:43.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:43.415+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.419207+0000) 2019-10-13T14:56:43.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:43.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.655025+0000) 2019-10-13T14:56:43.868 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:43.863+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.865860+0000) 2019-10-13T14:56:43.869 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:43.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.866700+0000) 2019-10-13T14:56:43.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:43.863+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.866539+0000) 2019-10-13T14:56:43.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:43.863+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.866996+0000) 2019-10-13T14:56:43.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:43.867+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.867999+0000) 2019-10-13T14:56:43.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:43.867+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.868448+0000) 2019-10-13T14:56:43.871 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:43.867+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.869277+0000) 2019-10-13T14:56:43.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:43.871+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:43.874928+0000) 2019-10-13T14:56:43.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:43.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:45.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:45.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.679160+0000) 2019-10-13T14:56:45.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:45.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.690832+0000) 2019-10-13T14:56:48.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:48.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.879095+0000) 2019-10-13T14:56:48.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:48.875+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.880877+0000) 2019-10-13T14:56:48.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:48.879+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.881057+0000) 2019-10-13T14:56:48.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:48.879+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.881814+0000) 2019-10-13T14:56:48.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:48.879+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.881954+0000) 2019-10-13T14:56:48.885 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:48.879+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.882399+0000) 2019-10-13T14:56:48.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:48.879+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.882729+0000) 2019-10-13T14:56:48.888 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:48.879+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.885887+0000) 2019-10-13T14:56:48.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:48.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:48.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:48.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.928696+0000) 2019-10-13T14:56:52.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:52.667+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:52.667426+0000) 2019-10-13T14:56:52.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:52.727+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:52.729410+0000) 2019-10-13T14:56:53.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:53.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.405880+0000) 2019-10-13T14:56:53.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:53.427+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.428858+0000) 2019-10-13T14:56:53.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:53.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.655223+0000) 2019-10-13T14:56:53.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:53.891+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.893778+0000) 2019-10-13T14:56:53.897 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:53.891+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.894169+0000) 2019-10-13T14:56:53.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:53.891+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.895120+0000) 2019-10-13T14:56:53.897 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:53.895+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.895415+0000) 2019-10-13T14:56:53.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:53.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.895850+0000) 2019-10-13T14:56:53.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:53.895+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.897194+0000) 2019-10-13T14:56:53.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:53.895+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.897556+0000) 2019-10-13T14:56:53.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:53.899+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:53.903257+0000) 2019-10-13T14:56:53.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:53.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:55.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:55.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:55.679357+0000) 2019-10-13T14:56:55.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:55.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:55.691048+0000) 2019-10-13T14:56:58.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:56:58.906+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.907953+0000) 2019-10-13T14:56:58.911 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:56:58.906+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.908338+0000) 2019-10-13T14:56:58.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:56:58.906+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.908843+0000) 2019-10-13T14:56:58.912 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:58.906+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.909491+0000) 2019-10-13T14:56:58.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:56:58.906+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.909877+0000) 2019-10-13T14:56:58.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:56:58.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.910777+0000) 2019-10-13T14:56:58.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:56:58.906+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.911130+0000) 2019-10-13T14:56:58.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:56:58.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208197 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:56:58.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:56:58.915+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.918722+0000) 2019-10-13T14:56:58.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:56:58.926+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.928867+0000) 2019-10-13T14:57:02.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:02.666+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:02.667630+0000) 2019-10-13T14:57:02.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:02.726+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:02.729650+0000) 2019-10-13T14:57:02.975 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:57:02.975 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:02.979 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:57:02.979 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:03.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:03.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.406073+0000) 2019-10-13T14:57:03.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:03.431+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.435300+0000) 2019-10-13T14:57:03.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:03.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.655411+0000) 2019-10-13T14:57:03.923 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:03.918+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.921257+0000) 2019-10-13T14:57:03.924 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:03.918+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.921801+0000) 2019-10-13T14:57:03.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:03.918+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.922687+0000) 2019-10-13T14:57:03.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:03.918+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.923153+0000) 2019-10-13T14:57:03.926 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:03.922+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.923433+0000) 2019-10-13T14:57:03.926 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:03.922+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.923796+0000) 2019-10-13T14:57:03.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:03.919+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.924562+0000) 2019-10-13T14:57:03.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:03.923+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:03.928712+0000) 2019-10-13T14:57:03.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:03.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208232 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:05.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:05.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:05.679556+0000) 2019-10-13T14:57:05.693 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:05.686+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:05.691279+0000) 2019-10-13T14:57:08.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:08.926+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.929029+0000) 2019-10-13T14:57:08.937 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:08.930+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.934918+0000) 2019-10-13T14:57:08.938 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:08.934+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.935566+0000) 2019-10-13T14:57:08.938 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:08.934+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.935967+0000) 2019-10-13T14:57:08.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:08.934+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.936332+0000) 2019-10-13T14:57:08.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:08.934+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.937290+0000) 2019-10-13T14:57:08.940 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:08.934+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.937590+0000) 2019-10-13T14:57:08.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:08.935+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.938593+0000) 2019-10-13T14:57:08.947 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:08.939+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.944671+0000) 2019-10-13T14:57:08.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:08.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:12.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:12.666+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:12.667866+0000) 2019-10-13T14:57:12.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:12.726+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:12.729883+0000) 2019-10-13T14:57:13.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:13.402+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.406313+0000) 2019-10-13T14:57:13.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:13.439+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.443193+0000) 2019-10-13T14:57:13.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:13.655+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.655628+0000) 2019-10-13T14:57:13.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:13.946+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.948501+0000) 2019-10-13T14:57:13.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:13.946+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.948945+0000) 2019-10-13T14:57:13.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:13.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.949185+0000) 2019-10-13T14:57:13.952 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:13.946+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.949875+0000) 2019-10-13T14:57:13.953 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:13.950+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.951291+0000) 2019-10-13T14:57:13.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:13.950+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.951372+0000) 2019-10-13T14:57:13.954 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:13.947+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.952535+0000) 2019-10-13T14:57:13.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:13.951+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:13.956212+0000) 2019-10-13T14:57:13.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:13.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:15.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:15.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:15.679785+0000) 2019-10-13T14:57:15.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:15.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:15.691511+0000) 2019-10-13T14:57:18.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:18.926+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.929230+0000) 2019-10-13T14:57:18.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:18.958+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.961824+0000) 2019-10-13T14:57:18.964 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:18.958+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.962435+0000) 2019-10-13T14:57:18.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:18.958+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.962788+0000) 2019-10-13T14:57:18.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:18.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.963376+0000) 2019-10-13T14:57:18.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:18.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.964165+0000) 2019-10-13T14:57:18.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:18.962+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.964642+0000) 2019-10-13T14:57:18.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:18.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.965015+0000) 2019-10-13T14:57:18.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:18.967+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.971838+0000) 2019-10-13T14:57:18.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:18.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:22.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:22.666+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:22.668067+0000) 2019-10-13T14:57:22.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:22.726+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:22.730087+0000) 2019-10-13T14:57:23.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:23.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.406508+0000) 2019-10-13T14:57:23.451 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:23.447+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.448708+0000) 2019-10-13T14:57:23.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:23.651+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.655823+0000) 2019-10-13T14:57:23.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:23.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.976012+0000) 2019-10-13T14:57:23.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:23.974+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.976392+0000) 2019-10-13T14:57:23.979 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:23.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.977356+0000) 2019-10-13T14:57:23.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:23.971+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.977946+0000) 2019-10-13T14:57:23.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:23.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.978691+0000) 2019-10-13T14:57:23.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:23.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.979007+0000) 2019-10-13T14:57:23.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:23.978+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.979386+0000) 2019-10-13T14:57:23.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:23.983+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:23.986575+0000) 2019-10-13T14:57:23.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:23.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:25.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:25.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:25.680015+0000) 2019-10-13T14:57:25.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:25.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:25.691714+0000) 2019-10-13T14:57:28.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:28.926+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.929430+0000) 2019-10-13T14:57:28.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:28.986+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.990328+0000) 2019-10-13T14:57:28.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:28.986+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.990471+0000) 2019-10-13T14:57:28.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:28.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.991430+0000) 2019-10-13T14:57:28.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:28.990+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.991771+0000) 2019-10-13T14:57:28.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:28.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.992520+0000) 2019-10-13T14:57:28.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:28.990+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.992867+0000) 2019-10-13T14:57:28.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:28.986+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.993736+0000) 2019-10-13T14:57:29.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:28.994+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.999841+0000) 2019-10-13T14:57:29.003 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:28.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208398 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:32.670 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:32.666+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:32.668264+0000) 2019-10-13T14:57:32.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:32.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:32.730305+0000) 2019-10-13T14:57:33.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:57:33.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:33.203 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:57:33.203 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:33.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:33.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:33.406705+0000) 2019-10-13T14:57:33.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:33.454+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:33.455167+0000) 2019-10-13T14:57:33.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:33.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:33.656018+0000) 2019-10-13T14:57:34.007 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:34.002+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.004809+0000) 2019-10-13T14:57:34.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:34.002+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.005276+0000) 2019-10-13T14:57:34.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:34.002+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.005497+0000) 2019-10-13T14:57:34.008 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:34.002+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.006047+0000) 2019-10-13T14:57:34.009 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:34.002+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.006724+0000) 2019-10-13T14:57:34.010 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:34.006+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.007361+0000) 2019-10-13T14:57:34.010 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:34.002+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.008115+0000) 2019-10-13T14:57:34.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:34.006+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.010777+0000) 2019-10-13T14:57:34.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:34.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208435 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:35.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:35.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:35.680214+0000) 2019-10-13T14:57:35.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:35.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:35.691947+0000) 2019-10-13T14:57:38.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:38.925+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:38.929626+0000) 2019-10-13T14:57:39.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:39.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.017966+0000) 2019-10-13T14:57:39.021 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:39.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.018722+0000) 2019-10-13T14:57:39.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:39.017+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.019525+0000) 2019-10-13T14:57:39.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:39.017+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.019930+0000) 2019-10-13T14:57:39.022 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:39.017+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.020478+0000) 2019-10-13T14:57:39.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:39.017+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.020944+0000) 2019-10-13T14:57:39.024 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:39.014+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.021812+0000) 2019-10-13T14:57:39.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:39.022+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:39.026685+0000) 2019-10-13T14:57:39.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:39.025+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208466 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:42.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:42.665+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:42.668466+0000) 2019-10-13T14:57:42.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:42.725+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:42.730505+0000) 2019-10-13T14:57:43.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:43.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:43.406907+0000) 2019-10-13T14:57:43.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:43.458+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:43.461522+0000) 2019-10-13T14:57:43.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:43.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:43.656267+0000) 2019-10-13T14:57:44.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:44.025+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.030879+0000) 2019-10-13T14:57:44.034 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:44.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.031741+0000) 2019-10-13T14:57:44.035 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:44.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.032204+0000) 2019-10-13T14:57:44.035 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:44.029+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.032288+0000) 2019-10-13T14:57:44.035 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:44.029+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.033035+0000) 2019-10-13T14:57:44.036 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:44.026+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.034051+0000) 2019-10-13T14:57:44.036 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:44.029+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.034574+0000) 2019-10-13T14:57:44.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:44.034+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.040346+0000) 2019-10-13T14:57:44.045 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:44.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:45.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:45.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:45.680418+0000) 2019-10-13T14:57:45.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:45.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:45.692157+0000) 2019-10-13T14:57:48.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:48.925+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:48.929783+0000) 2019-10-13T14:57:49.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:49.041+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.045758+0000) 2019-10-13T14:57:49.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:49.041+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.046063+0000) 2019-10-13T14:57:49.049 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:49.041+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.047246+0000) 2019-10-13T14:57:49.050 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:49.045+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.047652+0000) 2019-10-13T14:57:49.050 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:49.045+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.048126+0000) 2019-10-13T14:57:49.051 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:49.045+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.048635+0000) 2019-10-13T14:57:49.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:49.042+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.049086+0000) 2019-10-13T14:57:49.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:49.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208533 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:49.059 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:49.050+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:49.056877+0000) 2019-10-13T14:57:52.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:52.665+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:52.668665+0000) 2019-10-13T14:57:52.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:52.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:52.730708+0000) 2019-10-13T14:57:53.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:53.401+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:53.407097+0000) 2019-10-13T14:57:53.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:53.466+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:53.467909+0000) 2019-10-13T14:57:53.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:53.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:53.656460+0000) 2019-10-13T14:57:54.062 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:54.057+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.059739+0000) 2019-10-13T14:57:54.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:54.054+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.060219+0000) 2019-10-13T14:57:54.063 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:54.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.060508+0000) 2019-10-13T14:57:54.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:54.057+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.060850+0000) 2019-10-13T14:57:54.064 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:54.057+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.061995+0000) 2019-10-13T14:57:54.065 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:54.057+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.062590+0000) 2019-10-13T14:57:54.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:54.057+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.063176+0000) 2019-10-13T14:57:54.071 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:54.062+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.069080+0000) 2019-10-13T14:57:54.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:54.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:57:55.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:55.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:55.680596+0000) 2019-10-13T14:57:55.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:55.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:55.692381+0000) 2019-10-13T14:57:58.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:58.929+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:58.929995+0000) 2019-10-13T14:57:59.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:57:59.069+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.074762+0000) 2019-10-13T14:57:59.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:57:59.070+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.075316+0000) 2019-10-13T14:57:59.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:57:59.073+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.075563+0000) 2019-10-13T14:57:59.079 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:57:59.073+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.076857+0000) 2019-10-13T14:57:59.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:57:59.073+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.076807+0000) 2019-10-13T14:57:59.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:57:59.073+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.077566+0000) 2019-10-13T14:57:59.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:57:59.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.077932+0000) 2019-10-13T14:57:59.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:57:59.078+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:59.082396+0000) 2019-10-13T14:57:59.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:57:59.081+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208601 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:02.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:02.665+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:02.668899+0000) 2019-10-13T14:58:02.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:02.725+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:02.730904+0000) 2019-10-13T14:58:03.403 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:58:03.403 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:03.407 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:58:03.407 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:03.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:03.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:03.407346+0000) 2019-10-13T14:58:03.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:03.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:03.474324+0000) 2019-10-13T14:58:03.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:03.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:03.656659+0000) 2019-10-13T14:58:04.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:04.085+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.089068+0000) 2019-10-13T14:58:04.092 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:04.085+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.090004+0000) 2019-10-13T14:58:04.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:04.085+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.090423+0000) 2019-10-13T14:58:04.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:04.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.091635+0000) 2019-10-13T14:58:04.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:04.089+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.091852+0000) 2019-10-13T14:58:04.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:04.086+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.092670+0000) 2019-10-13T14:58:04.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:04.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.092232+0000) 2019-10-13T14:58:04.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:04.090+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.097310+0000) 2019-10-13T14:58:04.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:04.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208636 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:05.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:05.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:05.680812+0000) 2019-10-13T14:58:05.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:05.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:05.692603+0000) 2019-10-13T14:58:08.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:08.925+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:08.930206+0000) 2019-10-13T14:58:09.107 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:09.101+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.104301+0000) 2019-10-13T14:58:09.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:09.101+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.104607+0000) 2019-10-13T14:58:09.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:09.101+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.105040+0000) 2019-10-13T14:58:09.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:09.101+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.105561+0000) 2019-10-13T14:58:09.108 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:09.098+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.105906+0000) 2019-10-13T14:58:09.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:09.101+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.106922+0000) 2019-10-13T14:58:09.109 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:09.101+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.106052+0000) 2019-10-13T14:58:09.115 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:09.106+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:09.112593+0000) 2019-10-13T14:58:09.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:09.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208669 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:12.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:12.665+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:12.669098+0000) 2019-10-13T14:58:12.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:12.725+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:12.731159+0000) 2019-10-13T14:58:13.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:13.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:13.407545+0000) 2019-10-13T14:58:13.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:13.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:13.483022+0000) 2019-10-13T14:58:13.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:13.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:13.656857+0000) 2019-10-13T14:58:14.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:14.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.118179+0000) 2019-10-13T14:58:14.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:14.113+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.118589+0000) 2019-10-13T14:58:14.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:14.113+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.119176+0000) 2019-10-13T14:58:14.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:14.117+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.119658+0000) 2019-10-13T14:58:14.122 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:14.117+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.120197+0000) 2019-10-13T14:58:14.123 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:14.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.120935+0000) 2019-10-13T14:58:14.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:14.114+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.121177+0000) 2019-10-13T14:58:14.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:14.126+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.127770+0000) 2019-10-13T14:58:14.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:14.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208705 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:15.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:15.677+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:15.681011+0000) 2019-10-13T14:58:15.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:15.689+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:15.692803+0000) 2019-10-13T14:58:18.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:18.929+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:18.930402+0000) 2019-10-13T14:58:19.134 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:19.129+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.132006+0000) 2019-10-13T14:58:19.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:19.130+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.132592+0000) 2019-10-13T14:58:19.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:19.129+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.133798+0000) 2019-10-13T14:58:19.136 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:19.129+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.134279+0000) 2019-10-13T14:58:19.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:19.129+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.134593+0000) 2019-10-13T14:58:19.137 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:19.129+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.135064+0000) 2019-10-13T14:58:19.137 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:19.129+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.135219+0000) 2019-10-13T14:58:19.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:19.142+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:19.142598+0000) 2019-10-13T14:58:19.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:19.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:22.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:22.668+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:22.669305+0000) 2019-10-13T14:58:22.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:22.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:22.731355+0000) 2019-10-13T14:58:23.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:23.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:23.407737+0000) 2019-10-13T14:58:23.492 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:23.485+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:23.489478+0000) 2019-10-13T14:58:23.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:23.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:23.657065+0000) 2019-10-13T14:58:24.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:24.140+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.146362+0000) 2019-10-13T14:58:24.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:24.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.146940+0000) 2019-10-13T14:58:24.149 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:24.144+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.147269+0000) 2019-10-13T14:58:24.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:24.144+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.148199+0000) 2019-10-13T14:58:24.151 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:24.145+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.148590+0000) 2019-10-13T14:58:24.151 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:24.144+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.148763+0000) 2019-10-13T14:58:24.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:24.144+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.149251+0000) 2019-10-13T14:58:24.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:24.153+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.155730+0000) 2019-10-13T14:58:24.160 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:24.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:25.683 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:25.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:25.681229+0000) 2019-10-13T14:58:25.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:25.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:25.693030+0000) 2019-10-13T14:58:28.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:28.928+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:28.930597+0000) 2019-10-13T14:58:29.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:29.156+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.160672+0000) 2019-10-13T14:58:29.164 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:29.156+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.161967+0000) 2019-10-13T14:58:29.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:29.156+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.162684+0000) 2019-10-13T14:58:29.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:29.156+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.163039+0000) 2019-10-13T14:58:29.166 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:29.160+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.163716+0000) 2019-10-13T14:58:29.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:29.161+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.164291+0000) 2019-10-13T14:58:29.167 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:29.160+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.164631+0000) 2019-10-13T14:58:29.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:29.169+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:29.171175+0000) 2019-10-13T14:58:29.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:29.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:32.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:32.668+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:32.669531+0000) 2019-10-13T14:58:32.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:32.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:32.731555+0000) 2019-10-13T14:58:33.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:33.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:33.407930+0000) 2019-10-13T14:58:33.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:33.493+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:33.495861+0000) 2019-10-13T14:58:33.530 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:58:33.530 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:33.533 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:58:33.534 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:33.659 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:33.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:33.657265+0000) 2019-10-13T14:58:34.177 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:34.168+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.174692+0000) 2019-10-13T14:58:34.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:34.168+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.175224+0000) 2019-10-13T14:58:34.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:34.172+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.175915+0000) 2019-10-13T14:58:34.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:34.172+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.176146+0000) 2019-10-13T14:58:34.179 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:34.172+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.176927+0000) 2019-10-13T14:58:34.180 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:34.173+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.177917+0000) 2019-10-13T14:58:34.180 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:34.172+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.178216+0000) 2019-10-13T14:58:34.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:34.177+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.181829+0000) 2019-10-13T14:58:34.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:34.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:35.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:35.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:35.681427+0000) 2019-10-13T14:58:35.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:35.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:35.693250+0000) 2019-10-13T14:58:38.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:38.928+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:38.930796+0000) 2019-10-13T14:58:39.190 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:39.184+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.188234+0000) 2019-10-13T14:58:39.191 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:39.184+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.189331+0000) 2019-10-13T14:58:39.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:39.184+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.189687+0000) 2019-10-13T14:58:39.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:39.184+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.190269+0000) 2019-10-13T14:58:39.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:39.184+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.190488+0000) 2019-10-13T14:58:39.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:39.188+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.191442+0000) 2019-10-13T14:58:39.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:39.189+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.192010+0000) 2019-10-13T14:58:39.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:39.193+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:39.197621+0000) 2019-10-13T14:58:39.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:39.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:42.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:42.668+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:42.669728+0000) 2019-10-13T14:58:42.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:42.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:42.731802+0000) 2019-10-13T14:58:43.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:43.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:43.408117+0000) 2019-10-13T14:58:43.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:43.501+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:43.502479+0000) 2019-10-13T14:58:43.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:43.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:43.657470+0000) 2019-10-13T14:58:44.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:44.200+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.204617+0000) 2019-10-13T14:58:44.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:44.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.205053+0000) 2019-10-13T14:58:44.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:44.200+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.205382+0000) 2019-10-13T14:58:44.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:44.201+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.205753+0000) 2019-10-13T14:58:44.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:44.200+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.206014+0000) 2019-10-13T14:58:44.209 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:44.200+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.206392+0000) 2019-10-13T14:58:44.210 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:44.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.206972+0000) 2019-10-13T14:58:44.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:44.209+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.211018+0000) 2019-10-13T14:58:44.215 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:44.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:45.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:45.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:45.681621+0000) 2019-10-13T14:58:45.695 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:45.688+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:45.693451+0000) 2019-10-13T14:58:48.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:48.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:48.931008+0000) 2019-10-13T14:58:49.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:49.212+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.216477+0000) 2019-10-13T14:58:49.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:49.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.216990+0000) 2019-10-13T14:58:49.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:49.212+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.217697+0000) 2019-10-13T14:58:49.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:49.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.218083+0000) 2019-10-13T14:58:49.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:49.212+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.219199+0000) 2019-10-13T14:58:49.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:49.216+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.219876+0000) 2019-10-13T14:58:49.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:49.216+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.219716+0000) 2019-10-13T14:58:49.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:49.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208941 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:49.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:49.225+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:49.227607+0000) 2019-10-13T14:58:52.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:52.664+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:52.669927+0000) 2019-10-13T14:58:52.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:52.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:52.732015+0000) 2019-10-13T14:58:53.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:53.404+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:53.408309+0000) 2019-10-13T14:58:53.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:53.505+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:53.508834+0000) 2019-10-13T14:58:53.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:53.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:53.657697+0000) 2019-10-13T14:58:54.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:54.224+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.229768+0000) 2019-10-13T14:58:54.233 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:54.229+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.230573+0000) 2019-10-13T14:58:54.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:54.224+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.230868+0000) 2019-10-13T14:58:54.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:54.228+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.231293+0000) 2019-10-13T14:58:54.234 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:54.228+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.231776+0000) 2019-10-13T14:58:54.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:54.228+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.232173+0000) 2019-10-13T14:58:54.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:54.228+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.232407+0000) 2019-10-13T14:58:54.240 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:54.233+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.237945+0000) 2019-10-13T14:58:54.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:54.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 208977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:58:55.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:55.676+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:55.681843+0000) 2019-10-13T14:58:55.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:55.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:55.693650+0000) 2019-10-13T14:58:58.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:58.924+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:58.931206+0000) 2019-10-13T14:58:59.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:58:59.240+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.243438+0000) 2019-10-13T14:58:59.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:58:59.240+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.244332+0000) 2019-10-13T14:58:59.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:58:59.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.244641+0000) 2019-10-13T14:58:59.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:58:59.240+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.245157+0000) 2019-10-13T14:58:59.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:58:59.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.245930+0000) 2019-10-13T14:58:59.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:58:59.240+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.246224+0000) 2019-10-13T14:58:59.249 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:58:59.240+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.246825+0000) 2019-10-13T14:58:59.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:58:59.249+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:59.252358+0000) 2019-10-13T14:58:59.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:58:59.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209009 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:02.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:02.668+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:02.670126+0000) 2019-10-13T14:59:02.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:02.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:02.732254+0000) 2019-10-13T14:59:03.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:03.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:03.408492+0000) 2019-10-13T14:59:03.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:03.513+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:03.515291+0000) 2019-10-13T14:59:03.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:03.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:03.657887+0000) 2019-10-13T14:59:03.718 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:59:03.718 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:03.722 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:59:03.722 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:04.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:04.252+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.257698+0000) 2019-10-13T14:59:04.261 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:04.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.258438+0000) 2019-10-13T14:59:04.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:04.257+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.258960+0000) 2019-10-13T14:59:04.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:04.252+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.259031+0000) 2019-10-13T14:59:04.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:04.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.259532+0000) 2019-10-13T14:59:04.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:04.256+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.260071+0000) 2019-10-13T14:59:04.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:04.256+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.261065+0000) 2019-10-13T14:59:04.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:04.261+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.266125+0000) 2019-10-13T14:59:04.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:04.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:05.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:05.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:05.682052+0000) 2019-10-13T14:59:05.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:05.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:05.693847+0000) 2019-10-13T14:59:08.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:08.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:08.931406+0000) 2019-10-13T14:59:09.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:09.263+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.270089+0000) 2019-10-13T14:59:09.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:09.263+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.270899+0000) 2019-10-13T14:59:09.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:09.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.271814+0000) 2019-10-13T14:59:09.274 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:09.267+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.272052+0000) 2019-10-13T14:59:09.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:09.269+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.272589+0000) 2019-10-13T14:59:09.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:09.267+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.273071+0000) 2019-10-13T14:59:09.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:09.267+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.273500+0000) 2019-10-13T14:59:09.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:09.277+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:09.280428+0000) 2019-10-13T14:59:09.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:09.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:12.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:12.667+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:12.670369+0000) 2019-10-13T14:59:12.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:12.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:12.732489+0000) 2019-10-13T14:59:13.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:13.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:13.408655+0000) 2019-10-13T14:59:13.524 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:13.517+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:13.521837+0000) 2019-10-13T14:59:13.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:13.653+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:13.658087+0000) 2019-10-13T14:59:14.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:14.279+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.284970+0000) 2019-10-13T14:59:14.288 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:14.279+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.285639+0000) 2019-10-13T14:59:14.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:14.279+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.285300+0000) 2019-10-13T14:59:14.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:14.279+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.286538+0000) 2019-10-13T14:59:14.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:14.279+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.287026+0000) 2019-10-13T14:59:14.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:14.283+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.287807+0000) 2019-10-13T14:59:14.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:14.285+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.288614+0000) 2019-10-13T14:59:14.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:14.293+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.294580+0000) 2019-10-13T14:59:14.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:14.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:15.684 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:15.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:15.682221+0000) 2019-10-13T14:59:15.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:15.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:15.693998+0000) 2019-10-13T14:59:18.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:18.927+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:18.931555+0000) 2019-10-13T14:59:19.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:19.291+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.298430+0000) 2019-10-13T14:59:19.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:19.291+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.298125+0000) 2019-10-13T14:59:19.301 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:19.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.298565+0000) 2019-10-13T14:59:19.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:19.291+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.299181+0000) 2019-10-13T14:59:19.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:19.295+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.299431+0000) 2019-10-13T14:59:19.303 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:19.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.300048+0000) 2019-10-13T14:59:19.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:19.296+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.300673+0000) 2019-10-13T14:59:19.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:19.304+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:19.306452+0000) 2019-10-13T14:59:19.310 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:19.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:22.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:22.667+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:22.670594+0000) 2019-10-13T14:59:22.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:22.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:22.732731+0000) 2019-10-13T14:59:23.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:23.403+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:23.408831+0000) 2019-10-13T14:59:23.535 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:23.532+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:23.533221+0000) 2019-10-13T14:59:23.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:23.656+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:23.658306+0000) 2019-10-13T14:59:24.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:24.308+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.311922+0000) 2019-10-13T14:59:24.315 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:24.307+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.312459+0000) 2019-10-13T14:59:24.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:24.307+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.312959+0000) 2019-10-13T14:59:24.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:24.307+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.313297+0000) 2019-10-13T14:59:24.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:24.307+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.313646+0000) 2019-10-13T14:59:24.317 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:24.307+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.314064+0000) 2019-10-13T14:59:24.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:24.307+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.314877+0000) 2019-10-13T14:59:24.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:24.316+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.320812+0000) 2019-10-13T14:59:24.324 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:24.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:25.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:25.675+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:25.682412+0000) 2019-10-13T14:59:25.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:25.687+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:25.694186+0000) 2019-10-13T14:59:28.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:28.931+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:28.931753+0000) 2019-10-13T14:59:29.328 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:29.319+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.325470+0000) 2019-10-13T14:59:29.328 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:29.324+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.326270+0000) 2019-10-13T14:59:29.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:29.319+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.326686+0000) 2019-10-13T14:59:29.329 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:29.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.326847+0000) 2019-10-13T14:59:29.330 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:29.323+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.327626+0000) 2019-10-13T14:59:29.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:29.323+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.327384+0000) 2019-10-13T14:59:29.331 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:29.323+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.329124+0000) 2019-10-13T14:59:29.338 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:29.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:29.338 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:29.332+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:29.335816+0000) 2019-10-13T14:59:32.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:32.667+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:32.670794+0000) 2019-10-13T14:59:32.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:32.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:32.732929+0000) 2019-10-13T14:59:33.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:33.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:33.409018+0000) 2019-10-13T14:59:33.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:33.536+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:33.539847+0000) 2019-10-13T14:59:33.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:33.656+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:33.658508+0000) 2019-10-13T14:59:33.893 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T14:59:33.894 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:33.897 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T14:59:33.898 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:34.341 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:34.335+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.338582+0000) 2019-10-13T14:59:34.342 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:34.335+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.338781+0000) 2019-10-13T14:59:34.342 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:34.336+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.339898+0000) 2019-10-13T14:59:34.343 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:34.335+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.340339+0000) 2019-10-13T14:59:34.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:34.335+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.341306+0000) 2019-10-13T14:59:34.344 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:34.335+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.341549+0000) 2019-10-13T14:59:34.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:34.335+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.342319+0000) 2019-10-13T14:59:34.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:34.343+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:34.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:34.348+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.350424+0000) 2019-10-13T14:59:35.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:35.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:35.682611+0000) 2019-10-13T14:59:35.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:35.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:35.694404+0000) 2019-10-13T14:59:38.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:38.931+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:38.931943+0000) 2019-10-13T14:59:39.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:39.351+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.352921+0000) 2019-10-13T14:59:39.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:39.351+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.353668+0000) 2019-10-13T14:59:39.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:39.351+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.354043+0000) 2019-10-13T14:59:39.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:39.351+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.354589+0000) 2019-10-13T14:59:39.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:39.351+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.354999+0000) 2019-10-13T14:59:39.358 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:39.352+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.355438+0000) 2019-10-13T14:59:39.358 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:39.355+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.355987+0000) 2019-10-13T14:59:39.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:39.356+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:39.362119+0000) 2019-10-13T14:59:39.366 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:39.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209284 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:42.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:42.667+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:42.670995+0000) 2019-10-13T14:59:42.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:42.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:42.733132+0000) 2019-10-13T14:59:43.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:43.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:43.409199+0000) 2019-10-13T14:59:43.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:43.544+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:43.546147+0000) 2019-10-13T14:59:43.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:43.656+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:43.658706+0000) 2019-10-13T14:59:44.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:44.367+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.367447+0000) 2019-10-13T14:59:44.370 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:44.367+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.367794+0000) 2019-10-13T14:59:44.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:44.364+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.368256+0000) 2019-10-13T14:59:44.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:44.367+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.369057+0000) 2019-10-13T14:59:44.372 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:44.367+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.369347+0000) 2019-10-13T14:59:44.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:44.367+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.370011+0000) 2019-10-13T14:59:44.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:44.367+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.370407+0000) 2019-10-13T14:59:44.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:44.372+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.376883+0000) 2019-10-13T14:59:44.380 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:44.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:45.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:45.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:45.682838+0000) 2019-10-13T14:59:45.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:45.691+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:45.694601+0000) 2019-10-13T14:59:48.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:48.930+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:48.932139+0000) 2019-10-13T14:59:49.384 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:49.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.381680+0000) 2019-10-13T14:59:49.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:49.378+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.382194+0000) 2019-10-13T14:59:49.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:49.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.382399+0000) 2019-10-13T14:59:49.385 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:49.380+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.382957+0000) 2019-10-13T14:59:49.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:49.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.383496+0000) 2019-10-13T14:59:49.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:49.382+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.384857+0000) 2019-10-13T14:59:49.387 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:49.382+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.385330+0000) 2019-10-13T14:59:49.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:49.388+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:49.390471+0000) 2019-10-13T14:59:49.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:49.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209353 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:52.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:52.670+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:52.671215+0000) 2019-10-13T14:59:52.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:52.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:52.733334+0000) 2019-10-13T14:59:53.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:53.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:53.409401+0000) 2019-10-13T14:59:53.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:53.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:53.552570+0000) 2019-10-13T14:59:53.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:53.656+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:53.658906+0000) 2019-10-13T14:59:54.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:54.390+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.394873+0000) 2019-10-13T14:59:54.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:54.390+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.395063+0000) 2019-10-13T14:59:54.399 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:54.394+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.397203+0000) 2019-10-13T14:59:54.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:54.394+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.397284+0000) 2019-10-13T14:59:54.400 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:54.394+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.397811+0000) 2019-10-13T14:59:54.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:54.394+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.397587+0000) 2019-10-13T14:59:54.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:54.396+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.398393+0000) 2019-10-13T14:59:54.408 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:54.400+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.406029+0000) 2019-10-13T14:59:54.409 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:54.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T14:59:55.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:55.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:55.683067+0000) 2019-10-13T14:59:55.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:55.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:55.694789+0000) 2019-10-13T14:59:58.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:58.930+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:58.932333+0000) 2019-10-13T14:59:59.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T14:59:59.406+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.410083+0000) 2019-10-13T14:59:59.413 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T14:59:59.406+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.410331+0000) 2019-10-13T14:59:59.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T14:59:59.406+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.410615+0000) 2019-10-13T14:59:59.414 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T14:59:59.410+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.411528+0000) 2019-10-13T14:59:59.414 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T14:59:59.408+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.412147+0000) 2019-10-13T14:59:59.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T14:59:59.410+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.412726+0000) 2019-10-13T14:59:59.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T14:59:59.410+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.413211+0000) 2019-10-13T14:59:59.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T14:59:59.416+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:59.418721+0000) 2019-10-13T14:59:59.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T14:59:59.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:02.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:02.670+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:02.671481+0000) 2019-10-13T15:00:02.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:02.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:02.733552+0000) 2019-10-13T15:00:03.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:03.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:03.409579+0000) 2019-10-13T15:00:03.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:03.556+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:03.559055+0000) 2019-10-13T15:00:03.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:03.656+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:03.659100+0000) 2019-10-13T15:00:03.975 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:00:03.975 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:03.979 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:00:03.979 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:04.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:04.418+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.423050+0000) 2019-10-13T15:00:04.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:04.422+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.423874+0000) 2019-10-13T15:00:04.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:04.420+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.424485+0000) 2019-10-13T15:00:04.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:04.422+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.424933+0000) 2019-10-13T15:00:04.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:04.422+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.424658+0000) 2019-10-13T15:00:04.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:04.422+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.425352+0000) 2019-10-13T15:00:04.429 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:04.422+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.425865+0000) 2019-10-13T15:00:04.434 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:04.428+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.432529+0000) 2019-10-13T15:00:04.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:04.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:05.685 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:05.678+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:05.683260+0000) 2019-10-13T15:00:05.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:05.690+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:05.694940+0000) 2019-10-13T15:00:08.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:08.930+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:08.932548+0000) 2019-10-13T15:00:09.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:09.432+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.437241+0000) 2019-10-13T15:00:09.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:09.434+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.437716+0000) 2019-10-13T15:00:09.440 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:09.434+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.438021+0000) 2019-10-13T15:00:09.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:09.434+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.438353+0000) 2019-10-13T15:00:09.441 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:09.434+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.439012+0000) 2019-10-13T15:00:09.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:09.438+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.439334+0000) 2019-10-13T15:00:09.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:09.438+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.440380+0000) 2019-10-13T15:00:09.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:09.444+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:09.446289+0000) 2019-10-13T15:00:09.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:09.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:12.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:12.670+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:12.671706+0000) 2019-10-13T15:00:12.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:12.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:12.733790+0000) 2019-10-13T15:00:13.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:13.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:13.409791+0000) 2019-10-13T15:00:13.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:13.559+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:13.565717+0000) 2019-10-13T15:00:13.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:13.655+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:13.659290+0000) 2019-10-13T15:00:14.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:14.447+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.451699+0000) 2019-10-13T15:00:14.454 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:14.450+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.451923+0000) 2019-10-13T15:00:14.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:14.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.452554+0000) 2019-10-13T15:00:14.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:14.450+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.453107+0000) 2019-10-13T15:00:14.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:14.450+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.453316+0000) 2019-10-13T15:00:14.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:14.450+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.453628+0000) 2019-10-13T15:00:14.456 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:14.450+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.454006+0000) 2019-10-13T15:00:14.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:14.455+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.459252+0000) 2019-10-13T15:00:14.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:14.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:15.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:15.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:15.683521+0000) 2019-10-13T15:00:15.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:15.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:15.695136+0000) 2019-10-13T15:00:18.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:18.930+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:18.932743+0000) 2019-10-13T15:00:19.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:19.459+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.464415+0000) 2019-10-13T15:00:19.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:19.462+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.464660+0000) 2019-10-13T15:00:19.468 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:19.462+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.465344+0000) 2019-10-13T15:00:19.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:19.462+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.465542+0000) 2019-10-13T15:00:19.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:19.462+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.466190+0000) 2019-10-13T15:00:19.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:19.462+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.466626+0000) 2019-10-13T15:00:19.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:19.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.467366+0000) 2019-10-13T15:00:19.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:19.470+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:19.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:19.471+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:19.476056+0000) 2019-10-13T15:00:22.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:22.670+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:22.671907+0000) 2019-10-13T15:00:22.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:22.730+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:22.733993+0000) 2019-10-13T15:00:23.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:23.406+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:23.409976+0000) 2019-10-13T15:00:23.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:23.567+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:23.572255+0000) 2019-10-13T15:00:23.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:23.655+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:23.659484+0000) 2019-10-13T15:00:24.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:24.474+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.477817+0000) 2019-10-13T15:00:24.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:24.475+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.478510+0000) 2019-10-13T15:00:24.481 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:24.474+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.478744+0000) 2019-10-13T15:00:24.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:24.478+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.479432+0000) 2019-10-13T15:00:24.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:24.478+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.480339+0000) 2019-10-13T15:00:24.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:24.478+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.480840+0000) 2019-10-13T15:00:24.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:24.478+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.481459+0000) 2019-10-13T15:00:24.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:24.483+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.487452+0000) 2019-10-13T15:00:24.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:24.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:25.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:25.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:25.683717+0000) 2019-10-13T15:00:25.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:25.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:25.695365+0000) 2019-10-13T15:00:28.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:28.930+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:28.932938+0000) 2019-10-13T15:00:29.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:29.490+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.492807+0000) 2019-10-13T15:00:29.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:29.490+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.493084+0000) 2019-10-13T15:00:29.496 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:29.487+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.494034+0000) 2019-10-13T15:00:29.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:29.490+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.493443+0000) 2019-10-13T15:00:29.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:29.490+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.494777+0000) 2019-10-13T15:00:29.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:29.490+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.494859+0000) 2019-10-13T15:00:29.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:29.494+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.495469+0000) 2019-10-13T15:00:29.502 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:29.495+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:29.500037+0000) 2019-10-13T15:00:29.506 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:29.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209628 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:32.674 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:32.669+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:32.672118+0000) 2019-10-13T15:00:32.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:32.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:32.734192+0000) 2019-10-13T15:00:33.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:33.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:33.410202+0000) 2019-10-13T15:00:33.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:33.575+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:33.578702+0000) 2019-10-13T15:00:33.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:33.655+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:33.659674+0000) 2019-10-13T15:00:34.067 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:00:34.068 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:34.071 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:00:34.071 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:34.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:34.501+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.506943+0000) 2019-10-13T15:00:34.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:34.503+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.507350+0000) 2019-10-13T15:00:34.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:34.505+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.507549+0000) 2019-10-13T15:00:34.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:34.505+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.508667+0000) 2019-10-13T15:00:34.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:34.505+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.509236+0000) 2019-10-13T15:00:34.512 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:34.505+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.509652+0000) 2019-10-13T15:00:34.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:34.505+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.510473+0000) 2019-10-13T15:00:34.516 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:34.511+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.514281+0000) 2019-10-13T15:00:34.521 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:34.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:35.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:35.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:35.683918+0000) 2019-10-13T15:00:35.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:35.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:35.695563+0000) 2019-10-13T15:00:38.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:38.929+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:38.933143+0000) 2019-10-13T15:00:39.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:39.515+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.522007+0000) 2019-10-13T15:00:39.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:39.517+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.522530+0000) 2019-10-13T15:00:39.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:39.517+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.523019+0000) 2019-10-13T15:00:39.526 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:39.521+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.523871+0000) 2019-10-13T15:00:39.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:39.521+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.524769+0000) 2019-10-13T15:00:39.527 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:39.521+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.525035+0000) 2019-10-13T15:00:39.528 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:39.521+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.525446+0000) 2019-10-13T15:00:39.533 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:39.527+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:39.530605+0000) 2019-10-13T15:00:39.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:39.529+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209697 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:42.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:42.669+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:42.672337+0000) 2019-10-13T15:00:42.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:42.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:42.734410+0000) 2019-10-13T15:00:43.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:43.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:43.410396+0000) 2019-10-13T15:00:43.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:43.583+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:43.585017+0000) 2019-10-13T15:00:43.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:43.659+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:43.659861+0000) 2019-10-13T15:00:44.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:44.531+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.535555+0000) 2019-10-13T15:00:44.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:44.533+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.536100+0000) 2019-10-13T15:00:44.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:44.533+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.536481+0000) 2019-10-13T15:00:44.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:44.533+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.537330+0000) 2019-10-13T15:00:44.540 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:44.533+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.538134+0000) 2019-10-13T15:00:44.541 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:44.533+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.538394+0000) 2019-10-13T15:00:44.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:44.533+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.539001+0000) 2019-10-13T15:00:44.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:44.539+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.543206+0000) 2019-10-13T15:00:44.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:44.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:45.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:45.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:45.684072+0000) 2019-10-13T15:00:45.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:45.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:45.695735+0000) 2019-10-13T15:00:48.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:48.929+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:48.933343+0000) 2019-10-13T15:00:49.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:49.545+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.550125+0000) 2019-10-13T15:00:49.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:49.547+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.550833+0000) 2019-10-13T15:00:49.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:49.545+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.551008+0000) 2019-10-13T15:00:49.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:49.549+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.551344+0000) 2019-10-13T15:00:49.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:49.549+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.551940+0000) 2019-10-13T15:00:49.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:49.549+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.552132+0000) 2019-10-13T15:00:49.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:49.549+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.552455+0000) 2019-10-13T15:00:49.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:49.551+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:49.556064+0000) 2019-10-13T15:00:49.562 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:49.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209766 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:52.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:52.669+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:52.672564+0000) 2019-10-13T15:00:52.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:52.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:52.734607+0000) 2019-10-13T15:00:53.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:53.405+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:53.410615+0000) 2019-10-13T15:00:53.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:53.591+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:53.591433+0000) 2019-10-13T15:00:53.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:53.659+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:53.660007+0000) 2019-10-13T15:00:54.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:54.557+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.562885+0000) 2019-10-13T15:00:54.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:54.557+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.563234+0000) 2019-10-13T15:00:54.566 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:54.561+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.563718+0000) 2019-10-13T15:00:54.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:54.559+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.564916+0000) 2019-10-13T15:00:54.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:54.561+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.565132+0000) 2019-10-13T15:00:54.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:54.561+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.566462+0000) 2019-10-13T15:00:54.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:54.561+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.565822+0000) 2019-10-13T15:00:54.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:54.567+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.573364+0000) 2019-10-13T15:00:54.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:54.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:55.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:55.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:55.684285+0000) 2019-10-13T15:00:55.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:55.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:55.695941+0000) 2019-10-13T15:00:58.936 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:58.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:58.933552+0000) 2019-10-13T15:00:59.580 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:00:59.571+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.577648+0000) 2019-10-13T15:00:59.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:00:59.573+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.578235+0000) 2019-10-13T15:00:59.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:00:59.573+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.578055+0000) 2019-10-13T15:00:59.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:00:59.573+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.578603+0000) 2019-10-13T15:00:59.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:00:59.577+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.579764+0000) 2019-10-13T15:00:59.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:00:59.577+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.580503+0000) 2019-10-13T15:00:59.583 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:00:59.577+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.580698+0000) 2019-10-13T15:00:59.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:00:59.585+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:00:59.592 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:00:59.583+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:59.590045+0000) 2019-10-13T15:01:02.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:02.669+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:02.672825+0000) 2019-10-13T15:01:02.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:02.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:02.734810+0000) 2019-10-13T15:01:03.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:03.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:03.410804+0000) 2019-10-13T15:01:03.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:03.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:03.597935+0000) 2019-10-13T15:01:03.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:03.655+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:03.660229+0000) 2019-10-13T15:01:04.178 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:01:04.179 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:04.182 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:01:04.182 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:04.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:04.589+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.592037+0000) 2019-10-13T15:01:04.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:04.589+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.592639+0000) 2019-10-13T15:01:04.596 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:04.589+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.593415+0000) 2019-10-13T15:01:04.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:04.589+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.594084+0000) 2019-10-13T15:01:04.597 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:04.589+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.594818+0000) 2019-10-13T15:01:04.597 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:04.591+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.595021+0000) 2019-10-13T15:01:04.598 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:04.593+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.595558+0000) 2019-10-13T15:01:04.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:04.599+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.602557+0000) 2019-10-13T15:01:04.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:04.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:05.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:05.681+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:05.684486+0000) 2019-10-13T15:01:05.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:05.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:05.696138+0000) 2019-10-13T15:01:08.936 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:08.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:08.933795+0000) 2019-10-13T15:01:09.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:09.601+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.606457+0000) 2019-10-13T15:01:09.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:09.602+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.607387+0000) 2019-10-13T15:01:09.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:09.605+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.607475+0000) 2019-10-13T15:01:09.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:09.605+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.608015+0000) 2019-10-13T15:01:09.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:09.605+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.608412+0000) 2019-10-13T15:01:09.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:09.605+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.608840+0000) 2019-10-13T15:01:09.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:09.605+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.609164+0000) 2019-10-13T15:01:09.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:09.606+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:09.613578+0000) 2019-10-13T15:01:09.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:09.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209904 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:12.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:12.669+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:12.673063+0000) 2019-10-13T15:01:12.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:12.729+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:12.735024+0000) 2019-10-13T15:01:13.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:13.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:13.410993+0000) 2019-10-13T15:01:13.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:13.602+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:13.604476+0000) 2019-10-13T15:01:13.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:13.658+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:13.660458+0000) 2019-10-13T15:01:14.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:14.617+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.620025+0000) 2019-10-13T15:01:14.623 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:14.617+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.620262+0000) 2019-10-13T15:01:14.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:14.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.621800+0000) 2019-10-13T15:01:14.624 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:14.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.622250+0000) 2019-10-13T15:01:14.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:14.617+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.622464+0000) 2019-10-13T15:01:14.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:14.617+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.622838+0000) 2019-10-13T15:01:14.626 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:14.621+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.623460+0000) 2019-10-13T15:01:14.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:14.622+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.626804+0000) 2019-10-13T15:01:14.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:14.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:15.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:15.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:15.684673+0000) 2019-10-13T15:01:15.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:15.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:15.696337+0000) 2019-10-13T15:01:18.936 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:18.932+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:18.933990+0000) 2019-10-13T15:01:19.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:19.628+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.633504+0000) 2019-10-13T15:01:19.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:19.628+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.634108+0000) 2019-10-13T15:01:19.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:19.628+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.634759+0000) 2019-10-13T15:01:19.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:19.628+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.635229+0000) 2019-10-13T15:01:19.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:19.628+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.635113+0000) 2019-10-13T15:01:19.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:19.630+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.636176+0000) 2019-10-13T15:01:19.638 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:19.632+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.636325+0000) 2019-10-13T15:01:19.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:19.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 209973 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:19.646 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:19.638+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:19.644411+0000) 2019-10-13T15:01:22.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:22.672+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:22.673267+0000) 2019-10-13T15:01:22.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:22.728+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:22.735248+0000) 2019-10-13T15:01:23.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:23.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:23.411183+0000) 2019-10-13T15:01:23.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:23.610+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:23.610877+0000) 2019-10-13T15:01:23.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:23.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:23.660640+0000) 2019-10-13T15:01:24.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:24.640+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.646930+0000) 2019-10-13T15:01:24.650 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:24.644+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.647898+0000) 2019-10-13T15:01:24.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:24.644+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.648377+0000) 2019-10-13T15:01:24.651 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:24.644+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.648647+0000) 2019-10-13T15:01:24.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:24.644+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.649035+0000) 2019-10-13T15:01:24.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:24.642+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.649886+0000) 2019-10-13T15:01:24.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:24.644+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.650134+0000) 2019-10-13T15:01:24.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:24.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:24.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:24.650+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.657622+0000) 2019-10-13T15:01:25.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:25.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:25.684882+0000) 2019-10-13T15:01:25.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:25.692+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:25.696562+0000) 2019-10-13T15:01:28.936 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:28.928+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:28.934190+0000) 2019-10-13T15:01:29.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:29.656+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.660639+0000) 2019-10-13T15:01:29.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:29.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.660987+0000) 2019-10-13T15:01:29.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:29.656+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.661737+0000) 2019-10-13T15:01:29.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:29.656+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.661394+0000) 2019-10-13T15:01:29.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:29.656+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.662750+0000) 2019-10-13T15:01:29.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:29.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.663026+0000) 2019-10-13T15:01:29.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:29.658+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.663876+0000) 2019-10-13T15:01:29.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:29.662+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:29.668604+0000) 2019-10-13T15:01:29.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:29.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:32.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:32.672+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:32.673500+0000) 2019-10-13T15:01:32.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:32.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:32.735477+0000) 2019-10-13T15:01:33.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:33.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:33.411378+0000) 2019-10-13T15:01:33.620 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:33.614+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:33.617704+0000) 2019-10-13T15:01:33.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:33.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:33.660817+0000) 2019-10-13T15:01:34.249 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:01:34.249 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:34.253 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:01:34.253 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:34.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:34.668+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.674929+0000) 2019-10-13T15:01:34.678 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:34.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.675719+0000) 2019-10-13T15:01:34.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:34.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.675948+0000) 2019-10-13T15:01:34.679 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:34.672+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.676573+0000) 2019-10-13T15:01:34.679 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:34.672+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.676882+0000) 2019-10-13T15:01:34.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:34.672+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.677905+0000) 2019-10-13T15:01:34.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:34.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.678717+0000) 2019-10-13T15:01:34.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:34.678+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.682259+0000) 2019-10-13T15:01:34.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:34.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:35.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:35.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:35.685098+0000) 2019-10-13T15:01:35.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:35.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:35.696799+0000) 2019-10-13T15:01:38.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:38.928+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:38.934431+0000) 2019-10-13T15:01:39.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:39.684+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.688440+0000) 2019-10-13T15:01:39.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:39.684+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.688911+0000) 2019-10-13T15:01:39.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:39.686+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.690767+0000) 2019-10-13T15:01:39.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:39.684+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.690930+0000) 2019-10-13T15:01:39.694 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:39.688+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.691451+0000) 2019-10-13T15:01:39.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:39.688+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.691988+0000) 2019-10-13T15:01:39.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:39.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.693277+0000) 2019-10-13T15:01:39.701 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:39.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:39.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:39.694+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:39.699271+0000) 2019-10-13T15:01:42.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:42.672+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:42.673698+0000) 2019-10-13T15:01:42.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:42.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:42.735678+0000) 2019-10-13T15:01:43.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:43.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:43.411570+0000) 2019-10-13T15:01:43.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:43.622+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:43.624021+0000) 2019-10-13T15:01:43.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:43.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:43.661009+0000) 2019-10-13T15:01:44.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:44.696+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.701146+0000) 2019-10-13T15:01:44.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:44.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.702937+0000) 2019-10-13T15:01:44.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:44.700+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.703347+0000) 2019-10-13T15:01:44.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:44.700+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.703905+0000) 2019-10-13T15:01:44.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:44.698+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.704298+0000) 2019-10-13T15:01:44.707 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:44.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.704981+0000) 2019-10-13T15:01:44.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:44.700+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.705615+0000) 2019-10-13T15:01:44.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:44.706+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.712136+0000) 2019-10-13T15:01:44.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:44.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:45.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:45.680+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:45.685318+0000) 2019-10-13T15:01:45.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:45.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:45.697059+0000) 2019-10-13T15:01:48.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:48.928+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:48.934618+0000) 2019-10-13T15:01:49.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:49.710+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.716852+0000) 2019-10-13T15:01:49.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:49.712+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.718061+0000) 2019-10-13T15:01:49.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:49.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.718160+0000) 2019-10-13T15:01:49.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:49.712+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.718438+0000) 2019-10-13T15:01:49.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:49.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.718714+0000) 2019-10-13T15:01:49.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:49.712+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.719065+0000) 2019-10-13T15:01:49.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:49.716+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.720151+0000) 2019-10-13T15:01:49.728 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:49.722+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:49.726348+0000) 2019-10-13T15:01:49.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:49.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210181 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:52.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:52.668+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:52.673898+0000) 2019-10-13T15:01:52.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:52.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:52.735886+0000) 2019-10-13T15:01:53.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:53.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:53.411765+0000) 2019-10-13T15:01:53.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:53.630+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:53.630405+0000) 2019-10-13T15:01:53.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:53.654+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:53.661195+0000) 2019-10-13T15:01:54.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:54.730+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.730280+0000) 2019-10-13T15:01:54.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:54.724+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.730408+0000) 2019-10-13T15:01:54.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:54.728+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.731434+0000) 2019-10-13T15:01:54.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:54.728+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.732010+0000) 2019-10-13T15:01:54.735 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:54.728+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.732496+0000) 2019-10-13T15:01:54.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:54.728+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.733239+0000) 2019-10-13T15:01:54.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:54.728+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.733586+0000) 2019-10-13T15:01:54.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:54.738+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.739313+0000) 2019-10-13T15:01:54.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:54.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210217 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:01:55.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:55.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:55.685496+0000) 2019-10-13T15:01:55.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:55.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:55.697264+0000) 2019-10-13T15:01:58.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:58.932+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:58.934808+0000) 2019-10-13T15:01:59.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:01:59.739+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.743656+0000) 2019-10-13T15:01:59.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:01:59.742+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.744182+0000) 2019-10-13T15:01:59.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:01:59.739+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.744953+0000) 2019-10-13T15:01:59.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:01:59.739+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.745520+0000) 2019-10-13T15:01:59.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:01:59.739+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.746709+0000) 2019-10-13T15:01:59.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:01:59.739+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.747157+0000) 2019-10-13T15:01:59.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:01:59.743+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.747629+0000) 2019-10-13T15:01:59.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:01:59.750+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:59.752690+0000) 2019-10-13T15:01:59.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:01:59.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:02.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:02.671+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:02.674139+0000) 2019-10-13T15:02:02.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:02.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:02.736130+0000) 2019-10-13T15:02:03.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:03.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:03.411963+0000) 2019-10-13T15:02:03.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:03.633+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:03.636857+0000) 2019-10-13T15:02:03.663 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:03.657+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:03.661394+0000) 2019-10-13T15:02:04.377 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:02:04.377 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:04.381 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:02:04.381 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:04.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:04.751+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.758669+0000) 2019-10-13T15:02:04.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:04.755+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.760120+0000) 2019-10-13T15:02:04.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:04.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.761523+0000) 2019-10-13T15:02:04.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:04.755+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.761746+0000) 2019-10-13T15:02:04.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:04.755+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.762027+0000) 2019-10-13T15:02:04.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:04.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.762479+0000) 2019-10-13T15:02:04.765 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:04.755+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.762757+0000) 2019-10-13T15:02:04.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:04.765+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.769195+0000) 2019-10-13T15:02:04.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:04.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:05.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:05.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:05.685705+0000) 2019-10-13T15:02:05.699 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:05.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:05.697469+0000) 2019-10-13T15:02:08.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:08.931+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:08.934997+0000) 2019-10-13T15:02:09.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:09.767+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.772850+0000) 2019-10-13T15:02:09.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:09.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.774722+0000) 2019-10-13T15:02:09.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:09.767+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.775090+0000) 2019-10-13T15:02:09.778 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:09.773+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.775736+0000) 2019-10-13T15:02:09.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:09.771+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.775716+0000) 2019-10-13T15:02:09.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:09.771+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.776778+0000) 2019-10-13T15:02:09.779 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:09.771+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.777319+0000) 2019-10-13T15:02:09.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:09.781+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:09.782763+0000) 2019-10-13T15:02:09.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:09.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:12.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:12.671+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:12.674338+0000) 2019-10-13T15:02:12.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:12.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:12.736393+0000) 2019-10-13T15:02:13.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:13.407+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:13.412196+0000) 2019-10-13T15:02:13.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:13.641+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:13.643265+0000) 2019-10-13T15:02:13.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:13.657+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:13.661616+0000) 2019-10-13T15:02:14.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:14.785+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.786896+0000) 2019-10-13T15:02:14.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:14.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.787498+0000) 2019-10-13T15:02:14.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:14.783+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.787721+0000) 2019-10-13T15:02:14.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:14.783+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.788079+0000) 2019-10-13T15:02:14.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:14.783+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.788854+0000) 2019-10-13T15:02:14.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:14.783+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.789384+0000) 2019-10-13T15:02:14.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:14.783+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.789871+0000) 2019-10-13T15:02:14.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:14.793+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.797384+0000) 2019-10-13T15:02:14.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:14.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:15.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:15.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:15.685833+0000) 2019-10-13T15:02:15.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:15.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:15.697678+0000) 2019-10-13T15:02:18.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:18.931+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:18.935231+0000) 2019-10-13T15:02:19.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:19.795+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.801770+0000) 2019-10-13T15:02:19.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:19.795+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.802372+0000) 2019-10-13T15:02:19.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:19.795+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.802472+0000) 2019-10-13T15:02:19.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:19.795+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.802815+0000) 2019-10-13T15:02:19.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:19.801+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.803391+0000) 2019-10-13T15:02:19.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:19.799+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.804140+0000) 2019-10-13T15:02:19.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:19.799+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.804577+0000) 2019-10-13T15:02:19.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:19.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:19.807791+0000) 2019-10-13T15:02:19.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:19.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:22.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:22.671+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:22.674571+0000) 2019-10-13T15:02:22.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:22.731+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:22.736590+0000) 2019-10-13T15:02:23.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:23.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:23.412384+0000) 2019-10-13T15:02:23.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:23.649+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:23.653779+0000) 2019-10-13T15:02:23.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:23.657+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:23.661835+0000) 2019-10-13T15:02:24.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:24.811+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.815564+0000) 2019-10-13T15:02:24.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:24.811+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.816017+0000) 2019-10-13T15:02:24.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:24.811+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.816753+0000) 2019-10-13T15:02:24.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:24.811+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.817658+0000) 2019-10-13T15:02:24.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:24.813+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.818113+0000) 2019-10-13T15:02:24.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:24.811+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.818395+0000) 2019-10-13T15:02:24.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:24.811+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.818918+0000) 2019-10-13T15:02:24.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:24.821+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.825934+0000) 2019-10-13T15:02:24.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:24.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:25.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:25.679+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:25.686014+0000) 2019-10-13T15:02:25.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:25.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:25.697852+0000) 2019-10-13T15:02:28.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:28.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:28.935495+0000) 2019-10-13T15:02:29.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:29.827+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.829588+0000) 2019-10-13T15:02:29.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:29.831+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.831463+0000) 2019-10-13T15:02:29.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:29.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.832017+0000) 2019-10-13T15:02:29.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:29.829+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.832523+0000) 2019-10-13T15:02:29.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:29.831+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.832658+0000) 2019-10-13T15:02:29.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:29.831+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.832877+0000) 2019-10-13T15:02:29.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:29.831+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.833917+0000) 2019-10-13T15:02:29.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:29.837+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:29.839107+0000) 2019-10-13T15:02:29.843 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:29.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:32.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:32.671+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:32.674806+0000) 2019-10-13T15:02:32.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:32.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:32.736787+0000) 2019-10-13T15:02:33.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:33.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:33.412575+0000) 2019-10-13T15:02:33.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:33.657+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:33.660545+0000) 2019-10-13T15:02:33.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:33.657+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:33.662005+0000) 2019-10-13T15:02:34.488 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:02:34.488 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:34.492 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:02:34.492 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:34.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:34.841+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.845292+0000) 2019-10-13T15:02:34.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:34.843+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.845456+0000) 2019-10-13T15:02:34.848 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:34.843+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.845828+0000) 2019-10-13T15:02:34.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:34.843+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.846841+0000) 2019-10-13T15:02:34.849 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:34.843+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.846941+0000) 2019-10-13T15:02:34.850 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:34.843+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.847233+0000) 2019-10-13T15:02:34.850 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:34.847+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.847623+0000) 2019-10-13T15:02:34.853 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:34.849+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.851147+0000) 2019-10-13T15:02:34.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:34.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210492 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:35.688 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:35.683+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:35.686193+0000) 2019-10-13T15:02:35.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:35.695+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:35.698027+0000) 2019-10-13T15:02:38.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:38.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:38.935709+0000) 2019-10-13T15:02:39.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:39.855+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.858994+0000) 2019-10-13T15:02:39.862 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:39.859+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.859850+0000) 2019-10-13T15:02:39.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:39.857+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.860175+0000) 2019-10-13T15:02:39.864 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:39.859+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.861826+0000) 2019-10-13T15:02:39.865 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:39.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.862376+0000) 2019-10-13T15:02:39.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:39.859+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.861527+0000) 2019-10-13T15:02:39.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:39.859+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.863168+0000) 2019-10-13T15:02:39.868 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:39.861+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:39.865998+0000) 2019-10-13T15:02:39.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:39.867+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:42.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:42.670+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.675003+0000) 2019-10-13T15:02:42.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:42.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.736985+0000) 2019-10-13T15:02:43.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:43.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:43.412766+0000) 2019-10-13T15:02:43.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:43.661+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:43.662216+0000) 2019-10-13T15:02:43.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:43.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:43.667255+0000) 2019-10-13T15:02:44.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:44.869+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.873219+0000) 2019-10-13T15:02:44.876 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:44.870+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.874242+0000) 2019-10-13T15:02:44.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:44.870+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.875094+0000) 2019-10-13T15:02:44.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:44.874+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.875602+0000) 2019-10-13T15:02:44.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:44.874+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.876101+0000) 2019-10-13T15:02:44.879 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:44.874+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.876663+0000) 2019-10-13T15:02:44.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:44.874+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.877557+0000) 2019-10-13T15:02:44.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:44.881+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.882361+0000) 2019-10-13T15:02:44.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:44.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210560 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:45.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:45.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:45.686375+0000) 2019-10-13T15:02:45.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:45.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:45.698205+0000) 2019-10-13T15:02:48.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:48.934+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:48.935904+0000) 2019-10-13T15:02:49.890 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:49.886+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.887636+0000) 2019-10-13T15:02:49.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:49.886+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.887885+0000) 2019-10-13T15:02:49.891 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:49.886+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.888820+0000) 2019-10-13T15:02:49.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:49.886+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.889069+0000) 2019-10-13T15:02:49.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:49.886+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.890029+0000) 2019-10-13T15:02:49.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:49.889+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.890514+0000) 2019-10-13T15:02:49.893 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:49.886+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.890851+0000) 2019-10-13T15:02:49.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:49.893+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:49.895144+0000) 2019-10-13T15:02:49.900 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:49.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:52.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:52.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.675224+0000) 2019-10-13T15:02:52.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:52.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.737214+0000) 2019-10-13T15:02:53.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:53.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:53.412987+0000) 2019-10-13T15:02:53.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:53.661+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:53.662437+0000) 2019-10-13T15:02:53.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:53.673+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:53.674079+0000) 2019-10-13T15:02:54.903 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:54.898+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.901081+0000) 2019-10-13T15:02:54.904 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:54.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.901782+0000) 2019-10-13T15:02:54.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:54.898+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.902558+0000) 2019-10-13T15:02:54.905 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:54.898+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.902891+0000) 2019-10-13T15:02:54.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:54.902+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.903926+0000) 2019-10-13T15:02:54.907 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:54.902+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.904702+0000) 2019-10-13T15:02:54.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:54.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.905179+0000) 2019-10-13T15:02:54.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:54.909+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.910625+0000) 2019-10-13T15:02:54.914 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:54.910+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210630 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:02:55.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:55.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:55.686561+0000) 2019-10-13T15:02:55.940 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:55.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:55.698385+0000) 2019-10-13T15:02:58.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:58.934+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:58.936104+0000) 2019-10-13T15:02:59.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:02:59.914+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.916221+0000) 2019-10-13T15:02:59.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:02:59.914+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.916655+0000) 2019-10-13T15:02:59.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:02:59.914+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.916922+0000) 2019-10-13T15:02:59.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:02:59.912+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.917743+0000) 2019-10-13T15:02:59.921 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:02:59.914+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.918050+0000) 2019-10-13T15:02:59.921 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:02:59.914+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.918375+0000) 2019-10-13T15:02:59.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:02:59.914+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.919124+0000) 2019-10-13T15:02:59.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:02:59.924+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:59.926664+0000) 2019-10-13T15:02:59.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:02:59.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210664 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:02.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:02.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.675422+0000) 2019-10-13T15:03:02.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:02.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.737415+0000) 2019-10-13T15:03:03.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:03.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:03.413183+0000) 2019-10-13T15:03:03.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:03.660+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:03.662645+0000) 2019-10-13T15:03:03.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:03.676+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:03.680977+0000) 2019-10-13T15:03:04.823 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:03:04.824 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:04.827 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:03:04.827 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:04.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:04.926+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.929952+0000) 2019-10-13T15:03:04.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:04.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.930615+0000) 2019-10-13T15:03:04.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:04.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.930943+0000) 2019-10-13T15:03:04.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:04.930+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.932144+0000) 2019-10-13T15:03:04.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:04.930+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.932883+0000) 2019-10-13T15:03:04.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:04.930+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.933209+0000) 2019-10-13T15:03:04.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:04.930+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.933640+0000) 2019-10-13T15:03:04.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:04.936+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.938496+0000) 2019-10-13T15:03:04.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:04.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:05.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:05.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:05.686772+0000) 2019-10-13T15:03:05.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:05.694+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:05.698560+0000) 2019-10-13T15:03:08.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:08.934+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:08.936306+0000) 2019-10-13T15:03:09.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:09.942+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.944218+0000) 2019-10-13T15:03:09.947 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:09.942+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.944641+0000) 2019-10-13T15:03:09.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:09.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.945113+0000) 2019-10-13T15:03:09.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:09.942+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.945474+0000) 2019-10-13T15:03:09.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:09.942+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.946075+0000) 2019-10-13T15:03:09.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:09.944+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.946887+0000) 2019-10-13T15:03:09.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:09.946+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.947469+0000) 2019-10-13T15:03:09.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:09.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:09.954015+0000) 2019-10-13T15:03:09.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:09.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:12.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:12.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.675631+0000) 2019-10-13T15:03:12.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:12.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.737618+0000) 2019-10-13T15:03:13.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:13.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:13.413378+0000) 2019-10-13T15:03:13.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:13.660+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:13.662854+0000) 2019-10-13T15:03:13.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:13.684+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:13.687688+0000) 2019-10-13T15:03:14.960 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:14.954+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.957887+0000) 2019-10-13T15:03:14.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:14.954+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.958326+0000) 2019-10-13T15:03:14.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:14.954+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.958698+0000) 2019-10-13T15:03:14.962 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:14.958+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.959380+0000) 2019-10-13T15:03:14.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:14.956+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.959707+0000) 2019-10-13T15:03:14.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:14.958+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.960021+0000) 2019-10-13T15:03:14.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:14.958+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.960758+0000) 2019-10-13T15:03:14.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:14.960+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.965080+0000) 2019-10-13T15:03:14.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:14.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:15.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:15.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:15.686956+0000) 2019-10-13T15:03:15.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:15.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:15.698751+0000) 2019-10-13T15:03:18.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:18.934+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:18.936503+0000) 2019-10-13T15:03:19.974 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:19.970+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.971602+0000) 2019-10-13T15:03:19.975 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:19.970+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.972151+0000) 2019-10-13T15:03:19.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:19.968+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.973267+0000) 2019-10-13T15:03:19.976 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:19.970+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.973649+0000) 2019-10-13T15:03:19.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:19.970+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.973685+0000) 2019-10-13T15:03:19.977 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:19.970+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.974260+0000) 2019-10-13T15:03:19.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:19.972+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.974954+0000) 2019-10-13T15:03:19.977 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:19.970+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:19.974833+0000) 2019-10-13T15:03:19.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:19.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:22.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:22.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.675797+0000) 2019-10-13T15:03:22.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:22.734+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.737815+0000) 2019-10-13T15:03:23.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:23.410+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:23.413602+0000) 2019-10-13T15:03:23.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:23.660+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:23.663069+0000) 2019-10-13T15:03:23.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:23.692+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:23.694083+0000) 2019-10-13T15:03:24.988 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:24.982+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.985847+0000) 2019-10-13T15:03:24.989 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:24.982+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.986888+0000) 2019-10-13T15:03:24.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:24.986+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.987902+0000) 2019-10-13T15:03:24.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:24.986+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.988886+0000) 2019-10-13T15:03:24.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:24.986+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.989197+0000) 2019-10-13T15:03:24.992 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:24.984+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.990018+0000) 2019-10-13T15:03:24.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:24.986+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.990476+0000) 2019-10-13T15:03:24.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:24.988+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.993686+0000) 2019-10-13T15:03:24.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:24.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:25.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:25.682+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:25.687132+0000) 2019-10-13T15:03:25.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:25.693+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:25.698954+0000) 2019-10-13T15:03:28.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:28.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:28.936698+0000) 2019-10-13T15:03:30.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:29.997+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.000187+0000) 2019-10-13T15:03:30.003 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:29.996+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.000692+0000) 2019-10-13T15:03:30.003 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:29.997+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.000938+0000) 2019-10-13T15:03:30.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:29.997+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.001385+0000) 2019-10-13T15:03:30.004 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:29.997+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.002423+0000) 2019-10-13T15:03:30.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:30.001+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.003334+0000) 2019-10-13T15:03:30.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:30.001+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.004072+0000) 2019-10-13T15:03:30.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:30.008+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:30.011125+0000) 2019-10-13T15:03:30.013 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:30.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:32.678 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:32.673+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.676088+0000) 2019-10-13T15:03:32.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:32.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.738014+0000) 2019-10-13T15:03:33.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:33.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:33.413799+0000) 2019-10-13T15:03:33.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:33.660+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:33.663292+0000) 2019-10-13T15:03:33.702 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:33.700+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:33.700473+0000) 2019-10-13T15:03:34.868 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:03:34.868 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:34.872 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:03:34.872 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:35.017 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:35.009+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.014894+0000) 2019-10-13T15:03:35.018 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:35.013+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.015836+0000) 2019-10-13T15:03:35.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:35.013+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.016418+0000) 2019-10-13T15:03:35.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:35.013+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.016560+0000) 2019-10-13T15:03:35.019 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:35.013+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.017081+0000) 2019-10-13T15:03:35.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:35.012+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.017714+0000) 2019-10-13T15:03:35.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:35.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.018215+0000) 2019-10-13T15:03:35.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:35.020+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.023308+0000) 2019-10-13T15:03:35.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:35.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210906 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:35.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:35.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.687345+0000) 2019-10-13T15:03:35.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:35.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.699126+0000) 2019-10-13T15:03:38.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:38.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:38.936906+0000) 2019-10-13T15:03:40.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:40.025+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.028574+0000) 2019-10-13T15:03:40.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:40.025+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.028934+0000) 2019-10-13T15:03:40.033 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:40.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.029655+0000) 2019-10-13T15:03:40.033 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:40.025+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.029921+0000) 2019-10-13T15:03:40.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:40.025+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.030223+0000) 2019-10-13T15:03:40.034 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:40.025+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.031147+0000) 2019-10-13T15:03:40.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:40.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.031208+0000) 2019-10-13T15:03:40.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:40.032+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:40.037983+0000) 2019-10-13T15:03:40.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:40.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:42.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:42.673+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.676304+0000) 2019-10-13T15:03:42.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:42.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.738233+0000) 2019-10-13T15:03:43.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:43.409+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:43.413992+0000) 2019-10-13T15:03:43.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:43.660+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:43.663499+0000) 2019-10-13T15:03:43.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:43.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:43.706984+0000) 2019-10-13T15:03:45.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:45.037+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.041584+0000) 2019-10-13T15:03:45.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:45.037+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.042875+0000) 2019-10-13T15:03:45.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:45.041+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.043591+0000) 2019-10-13T15:03:45.046 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:45.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.043735+0000) 2019-10-13T15:03:45.046 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:45.041+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.044318+0000) 2019-10-13T15:03:45.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:45.040+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.044954+0000) 2019-10-13T15:03:45.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:45.041+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.045097+0000) 2019-10-13T15:03:45.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:45.044+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.049476+0000) 2019-10-13T15:03:45.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:45.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 210975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:45.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:45.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.687492+0000) 2019-10-13T15:03:45.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:45.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.699334+0000) 2019-10-13T15:03:48.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:48.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:48.937104+0000) 2019-10-13T15:03:50.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:50.049+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.054364+0000) 2019-10-13T15:03:50.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:50.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.055123+0000) 2019-10-13T15:03:50.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:50.053+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.055778+0000) 2019-10-13T15:03:50.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:50.053+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.056161+0000) 2019-10-13T15:03:50.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:50.053+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.057107+0000) 2019-10-13T15:03:50.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:50.053+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.057343+0000) 2019-10-13T15:03:50.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:50.056+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.058651+0000) 2019-10-13T15:03:50.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:50.056+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:50.061023+0000) 2019-10-13T15:03:50.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:50.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:52.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:52.673+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.676513+0000) 2019-10-13T15:03:52.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:52.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.738467+0000) 2019-10-13T15:03:53.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:53.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:53.414217+0000) 2019-10-13T15:03:53.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:53.659+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:53.663688+0000) 2019-10-13T15:03:53.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:53.711+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:53.713715+0000) 2019-10-13T15:03:55.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:03:55.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.067211+0000) 2019-10-13T15:03:55.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:03:55.063+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.067987+0000) 2019-10-13T15:03:55.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:03:55.065+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.068350+0000) 2019-10-13T15:03:55.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:55.065+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.069132+0000) 2019-10-13T15:03:55.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:55.065+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.069455+0000) 2019-10-13T15:03:55.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:55.065+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.070844+0000) 2019-10-13T15:03:55.074 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:03:55.069+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.071342+0000) 2019-10-13T15:03:55.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:03:55.071+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.075327+0000) 2019-10-13T15:03:55.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:03:55.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:03:55.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:03:55.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.687672+0000) 2019-10-13T15:03:55.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:03:55.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.699515+0000) 2019-10-13T15:03:58.939 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:03:58.937+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:58.937304+0000) 2019-10-13T15:04:00.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:00.075+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.081428+0000) 2019-10-13T15:04:00.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:00.077+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.081905+0000) 2019-10-13T15:04:00.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:00.077+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.083281+0000) 2019-10-13T15:04:00.086 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:00.081+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.083517+0000) 2019-10-13T15:04:00.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:00.081+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.083797+0000) 2019-10-13T15:04:00.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:00.081+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.084145+0000) 2019-10-13T15:04:00.087 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:00.081+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.084686+0000) 2019-10-13T15:04:00.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:00.087+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:00.093384+0000) 2019-10-13T15:04:00.096 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:00.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:02.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:02.673+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.676744+0000) 2019-10-13T15:04:02.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:02.733+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.738673+0000) 2019-10-13T15:04:03.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:03.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:03.414413+0000) 2019-10-13T15:04:03.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:03.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:03.663884+0000) 2019-10-13T15:04:03.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:03.719+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:03.720075+0000) 2019-10-13T15:04:04.988 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:04:04.988 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:04.992 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:04:04.992 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:05.098 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:05.093+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.096271+0000) 2019-10-13T15:04:05.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:05.093+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.097025+0000) 2019-10-13T15:04:05.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:05.093+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.097478+0000) 2019-10-13T15:04:05.101 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:05.093+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.097764+0000) 2019-10-13T15:04:05.102 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:05.095+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.098453+0000) 2019-10-13T15:04:05.102 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:05.093+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.098665+0000) 2019-10-13T15:04:05.102 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:05.093+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.098985+0000) 2019-10-13T15:04:05.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:05.099+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.104340+0000) 2019-10-13T15:04:05.110 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:05.105+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:05.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:05.685+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.687851+0000) 2019-10-13T15:04:05.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:05.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.699689+0000) 2019-10-13T15:04:08.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:08.933+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:08.937524+0000) 2019-10-13T15:04:10.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:10.107+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.110891+0000) 2019-10-13T15:04:10.114 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:10.104+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.111022+0000) 2019-10-13T15:04:10.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:10.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.112020+0000) 2019-10-13T15:04:10.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:10.108+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.112532+0000) 2019-10-13T15:04:10.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:10.108+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.112798+0000) 2019-10-13T15:04:10.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:10.108+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.113268+0000) 2019-10-13T15:04:10.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:10.108+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.114200+0000) 2019-10-13T15:04:10.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:10.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:10.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:10.119+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:10.122354+0000) 2019-10-13T15:04:12.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:12.672+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.676941+0000) 2019-10-13T15:04:12.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:12.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.738912+0000) 2019-10-13T15:04:13.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:13.408+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:13.414604+0000) 2019-10-13T15:04:13.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:13.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:13.664076+0000) 2019-10-13T15:04:13.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:13.723+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:13.726351+0000) 2019-10-13T15:04:15.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:15.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.125321+0000) 2019-10-13T15:04:15.128 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:15.120+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.125518+0000) 2019-10-13T15:04:15.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:15.120+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.126226+0000) 2019-10-13T15:04:15.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:15.120+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.126710+0000) 2019-10-13T15:04:15.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:15.120+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.127221+0000) 2019-10-13T15:04:15.131 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:15.123+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.128235+0000) 2019-10-13T15:04:15.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:15.124+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.128413+0000) 2019-10-13T15:04:15.137 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:15.131+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.135093+0000) 2019-10-13T15:04:15.138 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:15.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211176 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:15.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:15.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.688063+0000) 2019-10-13T15:04:15.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:15.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.699868+0000) 2019-10-13T15:04:18.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:18.936+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:18.937738+0000) 2019-10-13T15:04:20.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:20.132+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.139109+0000) 2019-10-13T15:04:20.142 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:20.136+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.140004+0000) 2019-10-13T15:04:20.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:20.136+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.140390+0000) 2019-10-13T15:04:20.143 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:20.136+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.140500+0000) 2019-10-13T15:04:20.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:20.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.140637+0000) 2019-10-13T15:04:20.144 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:20.136+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.141077+0000) 2019-10-13T15:04:20.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:20.135+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.141363+0000) 2019-10-13T15:04:20.150 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:20.143+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:20.148437+0000) 2019-10-13T15:04:20.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:20.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:22.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:22.676+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.677147+0000) 2019-10-13T15:04:22.741 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:22.732+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.739077+0000) 2019-10-13T15:04:23.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:23.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:23.414795+0000) 2019-10-13T15:04:23.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:23.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:23.664288+0000) 2019-10-13T15:04:23.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:23.727+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:23.732829+0000) 2019-10-13T15:04:25.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:25.148+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.152998+0000) 2019-10-13T15:04:25.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:25.148+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.153536+0000) 2019-10-13T15:04:25.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:25.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.153788+0000) 2019-10-13T15:04:25.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:25.148+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.154078+0000) 2019-10-13T15:04:25.157 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:25.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.154957+0000) 2019-10-13T15:04:25.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:25.152+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.155336+0000) 2019-10-13T15:04:25.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:25.151+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.156087+0000) 2019-10-13T15:04:25.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:25.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:25.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:25.159+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.164241+0000) 2019-10-13T15:04:25.690 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:25.684+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.688245+0000) 2019-10-13T15:04:25.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:25.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.700050+0000) 2019-10-13T15:04:28.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:28.932+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:28.937925+0000) 2019-10-13T15:04:30.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:30.160+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.166900+0000) 2019-10-13T15:04:30.170 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:30.163+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.167400+0000) 2019-10-13T15:04:30.170 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:30.164+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.167771+0000) 2019-10-13T15:04:30.171 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:30.164+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.168408+0000) 2019-10-13T15:04:30.171 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:30.164+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.168895+0000) 2019-10-13T15:04:30.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:30.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.169107+0000) 2019-10-13T15:04:30.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:30.164+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.169497+0000) 2019-10-13T15:04:30.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:30.171+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:30.176496+0000) 2019-10-13T15:04:30.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:30.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:32.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:32.676+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.677348+0000) 2019-10-13T15:04:32.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:32.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.739295+0000) 2019-10-13T15:04:33.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:33.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:33.415014+0000) 2019-10-13T15:04:33.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:33.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:33.664484+0000) 2019-10-13T15:04:33.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:33.735+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:33.738865+0000) 2019-10-13T15:04:35.110 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:04:35.110 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:35.114 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:04:35.114 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:35.182 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:35.176+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.180142+0000) 2019-10-13T15:04:35.183 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:35.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.180703+0000) 2019-10-13T15:04:35.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:35.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.180703+0000) 2019-10-13T15:04:35.183 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:35.176+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.180947+0000) 2019-10-13T15:04:35.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:35.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.181763+0000) 2019-10-13T15:04:35.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:35.176+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.182029+0000) 2019-10-13T15:04:35.185 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:35.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.183041+0000) 2019-10-13T15:04:35.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:35.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:35.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:35.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.194778+0000) 2019-10-13T15:04:35.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:35.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.688433+0000) 2019-10-13T15:04:35.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:35.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.700233+0000) 2019-10-13T15:04:38.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:38.932+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:38.938154+0000) 2019-10-13T15:04:40.196 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:40.188+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.193503+0000) 2019-10-13T15:04:40.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:40.188+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.193929+0000) 2019-10-13T15:04:40.197 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:40.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.193803+0000) 2019-10-13T15:04:40.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:40.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.194723+0000) 2019-10-13T15:04:40.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:40.188+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.194773+0000) 2019-10-13T15:04:40.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:40.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.195698+0000) 2019-10-13T15:04:40.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:40.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.195934+0000) 2019-10-13T15:04:40.205 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:40.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:40.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:40.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:40.203183+0000) 2019-10-13T15:04:42.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:42.676+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.677577+0000) 2019-10-13T15:04:42.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:42.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.739487+0000) 2019-10-13T15:04:43.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:43.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:43.415203+0000) 2019-10-13T15:04:43.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:43.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:43.664705+0000) 2019-10-13T15:04:43.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:43.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:43.745240+0000) 2019-10-13T15:04:45.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:45.200+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.206860+0000) 2019-10-13T15:04:45.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:45.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.207264+0000) 2019-10-13T15:04:45.220 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:45.203+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.208190+0000) 2019-10-13T15:04:45.221 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:45.204+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.208310+0000) 2019-10-13T15:04:45.221 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:45.204+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.209066+0000) 2019-10-13T15:04:45.221 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:45.204+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.209476+0000) 2019-10-13T15:04:45.222 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:45.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.210404+0000) 2019-10-13T15:04:45.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:45.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:45.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:45.211+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.217421+0000) 2019-10-13T15:04:45.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:45.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.688644+0000) 2019-10-13T15:04:45.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:45.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.700411+0000) 2019-10-13T15:04:48.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:48.936+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:48.938354+0000) 2019-10-13T15:04:50.222 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:50.216+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.220069+0000) 2019-10-13T15:04:50.223 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:50.216+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.220428+0000) 2019-10-13T15:04:50.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:50.216+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.220776+0000) 2019-10-13T15:04:50.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:50.214+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.221278+0000) 2019-10-13T15:04:50.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:50.216+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.221885+0000) 2019-10-13T15:04:50.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:50.216+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.222538+0000) 2019-10-13T15:04:50.225 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:50.216+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.223225+0000) 2019-10-13T15:04:50.228 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:50.222+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:50.226753+0000) 2019-10-13T15:04:50.231 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:50.224+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211408 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:52.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:52.675+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.677794+0000) 2019-10-13T15:04:52.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:52.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.739716+0000) 2019-10-13T15:04:53.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:53.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:53.415392+0000) 2019-10-13T15:04:53.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:53.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:53.664928+0000) 2019-10-13T15:04:53.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:53.746+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:53.751976+0000) 2019-10-13T15:04:55.234 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:04:55.227+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.231961+0000) 2019-10-13T15:04:55.235 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:55.227+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.232168+0000) 2019-10-13T15:04:55.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:55.227+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.233537+0000) 2019-10-13T15:04:55.236 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:04:55.226+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.233930+0000) 2019-10-13T15:04:55.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:04:55.227+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.234228+0000) 2019-10-13T15:04:55.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:04:55.227+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.234688+0000) 2019-10-13T15:04:55.237 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:55.227+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.235020+0000) 2019-10-13T15:04:55.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:04:55.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.241044+0000) 2019-10-13T15:04:55.244 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:04:55.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211443 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:04:55.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:04:55.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.688841+0000) 2019-10-13T15:04:55.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:04:55.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.700630+0000) 2019-10-13T15:04:58.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:04:58.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:58.938569+0000) 2019-10-13T15:05:00.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:00.239+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.245951+0000) 2019-10-13T15:05:00.249 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:00.242+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.246571+0000) 2019-10-13T15:05:00.249 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:00.239+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.247059+0000) 2019-10-13T15:05:00.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:00.243+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.247976+0000) 2019-10-13T15:05:00.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:00.243+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.248263+0000) 2019-10-13T15:05:00.254 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:00.243+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.248795+0000) 2019-10-13T15:05:00.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:00.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.249844+0000) 2019-10-13T15:05:00.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:00.246+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:00.253905+0000) 2019-10-13T15:05:00.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:00.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:02.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:02.675+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.678002+0000) 2019-10-13T15:05:02.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:02.735+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.739908+0000) 2019-10-13T15:05:03.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:03.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:03.415541+0000) 2019-10-13T15:05:03.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:03.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:03.665122+0000) 2019-10-13T15:05:03.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:03.754+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:03.758431+0000) 2019-10-13T15:05:05.245 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:05:05.245 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:05.248 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:05:05.249 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:05.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:05.251+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.257796+0000) 2019-10-13T15:05:05.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:05.251+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.257937+0000) 2019-10-13T15:05:05.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:05.251+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.258210+0000) 2019-10-13T15:05:05.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:05.251+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.258890+0000) 2019-10-13T15:05:05.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:05.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.259164+0000) 2019-10-13T15:05:05.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:05.254+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.259490+0000) 2019-10-13T15:05:05.262 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:05.255+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.260594+0000) 2019-10-13T15:05:05.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:05.262+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.266705+0000) 2019-10-13T15:05:05.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:05.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:05.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:05.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.689048+0000) 2019-10-13T15:05:05.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:05.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.700840+0000) 2019-10-13T15:05:08.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:08.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:08.938782+0000) 2019-10-13T15:05:10.279 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:10.266+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.272274+0000) 2019-10-13T15:05:10.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:10.267+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.272624+0000) 2019-10-13T15:05:10.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:10.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.273398+0000) 2019-10-13T15:05:10.281 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:10.267+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.273683+0000) 2019-10-13T15:05:10.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:10.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.274566+0000) 2019-10-13T15:05:10.282 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:10.267+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.274888+0000) 2019-10-13T15:05:10.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:10.271+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.275726+0000) 2019-10-13T15:05:10.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:10.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:10.282735+0000) 2019-10-13T15:05:10.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:10.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:12.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:12.671+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.678181+0000) 2019-10-13T15:05:12.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:12.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.740111+0000) 2019-10-13T15:05:13.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:13.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:13.415739+0000) 2019-10-13T15:05:13.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:13.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:13.665332+0000) 2019-10-13T15:05:13.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:13.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:13.764838+0000) 2019-10-13T15:05:15.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:15.279+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.285503+0000) 2019-10-13T15:05:15.288 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:15.282+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.286129+0000) 2019-10-13T15:05:15.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:15.279+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.286533+0000) 2019-10-13T15:05:15.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:15.279+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.286512+0000) 2019-10-13T15:05:15.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:15.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.286991+0000) 2019-10-13T15:05:15.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:15.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.287837+0000) 2019-10-13T15:05:15.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:15.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.287860+0000) 2019-10-13T15:05:15.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:15.290+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.296421+0000) 2019-10-13T15:05:15.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:15.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:15.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:15.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.689231+0000) 2019-10-13T15:05:15.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:15.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.701061+0000) 2019-10-13T15:05:18.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:18.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:18.938980+0000) 2019-10-13T15:05:20.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:20.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.300074+0000) 2019-10-13T15:05:20.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:20.295+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.300725+0000) 2019-10-13T15:05:20.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:20.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.301342+0000) 2019-10-13T15:05:20.304 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:20.295+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.301966+0000) 2019-10-13T15:05:20.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:20.298+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.302749+0000) 2019-10-13T15:05:20.305 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:20.295+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.302916+0000) 2019-10-13T15:05:20.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:20.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.303262+0000) 2019-10-13T15:05:20.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:20.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:20.304590+0000) 2019-10-13T15:05:20.313 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:20.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:22.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:22.675+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.678380+0000) 2019-10-13T15:05:22.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:22.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.740335+0000) 2019-10-13T15:05:23.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:23.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:23.415911+0000) 2019-10-13T15:05:23.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:23.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:23.665533+0000) 2019-10-13T15:05:23.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:23.774+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:23.775965+0000) 2019-10-13T15:05:25.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:25.311+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.313381+0000) 2019-10-13T15:05:25.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:25.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.314144+0000) 2019-10-13T15:05:25.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:25.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.314406+0000) 2019-10-13T15:05:25.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:25.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.314965+0000) 2019-10-13T15:05:25.318 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:25.311+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.315148+0000) 2019-10-13T15:05:25.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:25.315+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.315865+0000) 2019-10-13T15:05:25.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:25.310+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.316626+0000) 2019-10-13T15:05:25.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:25.314+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.320930+0000) 2019-10-13T15:05:25.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:25.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211644 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:25.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:25.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.689409+0000) 2019-10-13T15:05:25.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:25.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.701268+0000) 2019-10-13T15:05:28.941 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:28.935+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:28.939175+0000) 2019-10-13T15:05:30.329 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:30.322+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.327349+0000) 2019-10-13T15:05:30.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:30.327+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.327746+0000) 2019-10-13T15:05:30.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:30.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.328161+0000) 2019-10-13T15:05:30.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:30.327+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.328888+0000) 2019-10-13T15:05:30.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:30.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.329229+0000) 2019-10-13T15:05:30.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:30.327+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.329949+0000) 2019-10-13T15:05:30.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:30.327+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.330600+0000) 2019-10-13T15:05:30.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:30.330+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:30.336626+0000) 2019-10-13T15:05:30.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:30.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:32.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:32.675+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.678580+0000) 2019-10-13T15:05:32.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:32.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.740544+0000) 2019-10-13T15:05:33.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:33.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:33.416133+0000) 2019-10-13T15:05:33.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:33.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:33.665730+0000) 2019-10-13T15:05:33.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:33.782+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:33.782355+0000) 2019-10-13T15:05:35.298 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:05:35.299 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:35.302 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:05:35.303 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:35.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:35.339+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.342116+0000) 2019-10-13T15:05:35.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:35.339+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.342897+0000) 2019-10-13T15:05:35.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:35.343+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.343558+0000) 2019-10-13T15:05:35.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:35.343+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.343895+0000) 2019-10-13T15:05:35.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:35.343+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.344682+0000) 2019-10-13T15:05:35.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:35.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.345380+0000) 2019-10-13T15:05:35.348 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:35.346+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.346452+0000) 2019-10-13T15:05:35.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:35.346+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.347887+0000) 2019-10-13T15:05:35.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:35.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211710 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:35.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:35.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.689539+0000) 2019-10-13T15:05:35.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:35.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.701446+0000) 2019-10-13T15:05:38.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:38.938+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:38.939371+0000) 2019-10-13T15:05:40.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:40.354+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.355878+0000) 2019-10-13T15:05:40.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:40.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.355718+0000) 2019-10-13T15:05:40.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:40.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.356201+0000) 2019-10-13T15:05:40.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:40.354+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.357224+0000) 2019-10-13T15:05:40.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:40.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.357689+0000) 2019-10-13T15:05:40.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:40.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.357919+0000) 2019-10-13T15:05:40.361 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:40.354+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.357948+0000) 2019-10-13T15:05:40.369 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:40.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:40.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:40.366+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:40.367752+0000) 2019-10-13T15:05:42.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:42.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.678800+0000) 2019-10-13T15:05:42.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:42.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.740778+0000) 2019-10-13T15:05:43.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:43.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:43.416332+0000) 2019-10-13T15:05:43.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:43.661+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:43.665922+0000) 2019-10-13T15:05:43.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:43.785+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:43.788750+0000) 2019-10-13T15:05:45.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:45.366+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.369429+0000) 2019-10-13T15:05:45.373 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:45.366+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.370314+0000) 2019-10-13T15:05:45.373 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:45.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.370597+0000) 2019-10-13T15:05:45.374 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:45.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.370989+0000) 2019-10-13T15:05:45.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:45.366+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.370974+0000) 2019-10-13T15:05:45.374 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:45.370+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.371480+0000) 2019-10-13T15:05:45.374 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:45.370+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.372256+0000) 2019-10-13T15:05:45.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:45.377+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.380097+0000) 2019-10-13T15:05:45.384 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:45.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211777 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:45.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:45.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.689712+0000) 2019-10-13T15:05:45.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:45.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.701621+0000) 2019-10-13T15:05:48.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:48.938+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:48.939571+0000) 2019-10-13T15:05:50.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:50.382+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.384249+0000) 2019-10-13T15:05:50.387 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:50.382+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.384532+0000) 2019-10-13T15:05:50.387 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:50.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.384943+0000) 2019-10-13T15:05:50.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:50.382+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.385352+0000) 2019-10-13T15:05:50.388 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:50.381+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.385537+0000) 2019-10-13T15:05:50.388 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:50.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.385917+0000) 2019-10-13T15:05:50.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:50.382+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.386570+0000) 2019-10-13T15:05:50.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:50.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:50.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:50.393+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:50.398148+0000) 2019-10-13T15:05:52.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:52.674+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.678999+0000) 2019-10-13T15:05:52.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:52.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.740979+0000) 2019-10-13T15:05:53.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:53.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:53.416525+0000) 2019-10-13T15:05:53.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:53.665+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:53.666115+0000) 2019-10-13T15:05:53.797 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:53.793+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:53.795123+0000) 2019-10-13T15:05:55.400 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:55.394+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.397816+0000) 2019-10-13T15:05:55.401 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:05:55.397+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.398368+0000) 2019-10-13T15:05:55.401 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:55.394+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.399179+0000) 2019-10-13T15:05:55.402 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:05:55.398+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.400109+0000) 2019-10-13T15:05:55.403 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:05:55.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.400419+0000) 2019-10-13T15:05:55.403 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:05:55.398+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.400655+0000) 2019-10-13T15:05:55.403 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:55.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.401237+0000) 2019-10-13T15:05:55.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:05:55.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:05:55.412 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:05:55.405+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.410044+0000) 2019-10-13T15:05:55.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:05:55.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.689891+0000) 2019-10-13T15:05:55.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:05:55.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.701794+0000) 2019-10-13T15:05:58.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:05:58.938+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:58.939722+0000) 2019-10-13T15:06:00.415 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:00.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.412578+0000) 2019-10-13T15:06:00.415 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:00.410+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.412585+0000) 2019-10-13T15:06:00.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:00.410+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.413100+0000) 2019-10-13T15:06:00.416 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:00.410+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.414031+0000) 2019-10-13T15:06:00.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:00.413+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.414558+0000) 2019-10-13T15:06:00.417 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:00.410+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.414777+0000) 2019-10-13T15:06:00.417 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:00.410+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.415180+0000) 2019-10-13T15:06:00.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:00.417+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:00.420216+0000) 2019-10-13T15:06:00.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:00.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211874 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:02.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:02.678+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.679213+0000) 2019-10-13T15:06:02.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:02.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.741216+0000) 2019-10-13T15:06:03.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:03.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:03.416740+0000) 2019-10-13T15:06:03.668 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:03.665+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:03.666326+0000) 2019-10-13T15:06:03.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:03.797+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:03.801476+0000) 2019-10-13T15:06:05.425 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:06:05.426 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:05.429 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:06:05.429 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:05.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:05.425+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.426578+0000) 2019-10-13T15:06:05.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:05.426+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.427467+0000) 2019-10-13T15:06:05.431 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:05.426+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.427945+0000) 2019-10-13T15:06:05.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:05.426+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.428201+0000) 2019-10-13T15:06:05.431 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:05.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.428558+0000) 2019-10-13T15:06:05.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:05.426+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.429163+0000) 2019-10-13T15:06:05.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:05.426+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.429799+0000) 2019-10-13T15:06:05.435 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:05.429+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.433421+0000) 2019-10-13T15:06:05.439 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:05.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:05.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:05.686+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.690105+0000) 2019-10-13T15:06:05.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:05.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.702002+0000) 2019-10-13T15:06:08.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:08.938+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:08.939961+0000) 2019-10-13T15:06:10.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:10.438+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.440425+0000) 2019-10-13T15:06:10.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:10.437+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.441186+0000) 2019-10-13T15:06:10.445 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:10.438+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.441506+0000) 2019-10-13T15:06:10.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:10.438+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.441665+0000) 2019-10-13T15:06:10.446 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:10.438+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.442326+0000) 2019-10-13T15:06:10.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:10.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.442783+0000) 2019-10-13T15:06:10.447 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:10.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.443774+0000) 2019-10-13T15:06:10.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:10.445+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:10.447413+0000) 2019-10-13T15:06:10.453 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:10.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:12.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:12.678+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.679415+0000) 2019-10-13T15:06:12.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:12.738+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.741416+0000) 2019-10-13T15:06:13.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:13.414+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:13.416933+0000) 2019-10-13T15:06:13.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:13.665+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:13.666555+0000) 2019-10-13T15:06:13.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:13.805+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:13.807862+0000) 2019-10-13T15:06:15.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:15.450+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.454456+0000) 2019-10-13T15:06:15.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:15.450+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.454865+0000) 2019-10-13T15:06:15.458 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:15.454+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.455961+0000) 2019-10-13T15:06:15.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:15.454+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.456325+0000) 2019-10-13T15:06:15.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:15.453+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.456895+0000) 2019-10-13T15:06:15.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:15.454+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.457006+0000) 2019-10-13T15:06:15.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:15.454+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.457541+0000) 2019-10-13T15:06:15.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:15.457+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.461740+0000) 2019-10-13T15:06:15.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:15.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 211978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:15.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:15.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.690293+0000) 2019-10-13T15:06:15.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:15.698+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.702137+0000) 2019-10-13T15:06:18.942 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:18.938+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:18.940163+0000) 2019-10-13T15:06:20.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:20.465+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.468933+0000) 2019-10-13T15:06:20.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:20.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.469700+0000) 2019-10-13T15:06:20.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:20.465+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.470346+0000) 2019-10-13T15:06:20.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:20.469+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.471373+0000) 2019-10-13T15:06:20.474 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:20.469+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.471500+0000) 2019-10-13T15:06:20.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:20.469+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.472184+0000) 2019-10-13T15:06:20.475 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:20.469+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.472842+0000) 2019-10-13T15:06:20.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:20.477+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:20.478183+0000) 2019-10-13T15:06:20.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:20.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:22.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:22.677+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.679575+0000) 2019-10-13T15:06:22.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:22.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.741618+0000) 2019-10-13T15:06:23.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:23.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:23.417130+0000) 2019-10-13T15:06:23.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:23.665+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:23.666744+0000) 2019-10-13T15:06:23.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:23.813+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:23.814244+0000) 2019-10-13T15:06:25.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:25.481+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.484039+0000) 2019-10-13T15:06:25.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:25.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.484199+0000) 2019-10-13T15:06:25.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:25.481+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.484981+0000) 2019-10-13T15:06:25.491 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:25.481+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.485487+0000) 2019-10-13T15:06:25.492 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:25.485+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.486567+0000) 2019-10-13T15:06:25.493 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:25.481+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.486736+0000) 2019-10-13T15:06:25.493 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:25.481+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.486943+0000) 2019-10-13T15:06:25.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:25.489+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.491415+0000) 2019-10-13T15:06:25.501 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:25.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:25.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:25.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.690493+0000) 2019-10-13T15:06:25.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:25.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.702316+0000) 2019-10-13T15:06:28.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:28.937+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:28.940388+0000) 2019-10-13T15:06:30.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:30.493+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.496621+0000) 2019-10-13T15:06:30.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:30.493+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.497667+0000) 2019-10-13T15:06:30.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:30.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.498011+0000) 2019-10-13T15:06:30.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:30.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.498347+0000) 2019-10-13T15:06:30.501 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:30.493+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.498664+0000) 2019-10-13T15:06:30.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:30.493+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.499011+0000) 2019-10-13T15:06:30.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:30.497+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.499616+0000) 2019-10-13T15:06:30.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:30.505+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:30.506807+0000) 2019-10-13T15:06:30.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:30.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:32.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:32.677+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.679805+0000) 2019-10-13T15:06:32.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:32.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.741809+0000) 2019-10-13T15:06:33.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:33.413+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:33.417325+0000) 2019-10-13T15:06:33.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:33.665+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:33.666938+0000) 2019-10-13T15:06:33.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:33.817+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:33.820654+0000) 2019-10-13T15:06:35.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:35.509+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.512383+0000) 2019-10-13T15:06:35.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:35.509+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.513056+0000) 2019-10-13T15:06:35.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:35.509+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.513272+0000) 2019-10-13T15:06:35.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:35.509+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.513680+0000) 2019-10-13T15:06:35.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:35.509+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.514036+0000) 2019-10-13T15:06:35.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:35.509+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.514396+0000) 2019-10-13T15:06:35.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:35.509+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.515074+0000) 2019-10-13T15:06:35.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:35.517+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.519152+0000) 2019-10-13T15:06:35.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:35.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212111 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:35.545 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:06:35.546 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:35.549 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:06:35.549 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:35.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:35.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.690685+0000) 2019-10-13T15:06:35.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:35.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.702495+0000) 2019-10-13T15:06:38.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:38.937+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:38.940615+0000) 2019-10-13T15:06:40.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:40.521+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.525276+0000) 2019-10-13T15:06:40.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:40.524+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.526457+0000) 2019-10-13T15:06:40.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:40.521+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.526521+0000) 2019-10-13T15:06:40.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:40.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.527523+0000) 2019-10-13T15:06:40.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:40.525+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.527949+0000) 2019-10-13T15:06:40.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:40.525+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.528204+0000) 2019-10-13T15:06:40.531 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:40.525+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.528469+0000) 2019-10-13T15:06:40.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:40.528+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:40.532513+0000) 2019-10-13T15:06:40.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:40.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:42.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:42.677+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.680006+0000) 2019-10-13T15:06:42.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:42.737+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.742017+0000) 2019-10-13T15:06:43.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:43.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:43.417512+0000) 2019-10-13T15:06:43.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:43.664+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:43.667139+0000) 2019-10-13T15:06:43.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:43.828+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:43.832516+0000) 2019-10-13T15:06:45.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:45.537+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.540719+0000) 2019-10-13T15:06:45.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:45.536+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.541744+0000) 2019-10-13T15:06:45.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:45.537+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.541876+0000) 2019-10-13T15:06:45.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:45.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.542028+0000) 2019-10-13T15:06:45.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:45.537+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.542847+0000) 2019-10-13T15:06:45.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:45.537+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.542731+0000) 2019-10-13T15:06:45.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:45.537+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.543140+0000) 2019-10-13T15:06:45.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:45.544+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.548035+0000) 2019-10-13T15:06:45.552 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:45.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:45.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:45.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.690927+0000) 2019-10-13T15:06:45.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:45.697+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.702706+0000) 2019-10-13T15:06:48.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:48.937+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:48.940808+0000) 2019-10-13T15:06:50.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:50.549+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.554441+0000) 2019-10-13T15:06:50.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:50.552+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.556809+0000) 2019-10-13T15:06:50.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:50.560+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.564115+0000) 2019-10-13T15:06:50.588 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:50.553+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.555577+0000) 2019-10-13T15:06:50.589 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:50.553+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.555920+0000) 2019-10-13T15:06:50.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:50.553+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.556222+0000) 2019-10-13T15:06:50.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:50.553+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.556996+0000) 2019-10-13T15:06:50.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:50.553+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:50.558176+0000) 2019-10-13T15:06:50.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:50.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212210 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:52.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:52.677+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.680205+0000) 2019-10-13T15:06:52.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:52.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.742291+0000) 2019-10-13T15:06:53.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:53.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:53.417705+0000) 2019-10-13T15:06:53.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:53.664+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:53.667331+0000) 2019-10-13T15:06:53.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:53.836+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:53.838953+0000) 2019-10-13T15:06:55.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:06:55.565+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.568621+0000) 2019-10-13T15:06:55.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:06:55.564+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.569438+0000) 2019-10-13T15:06:55.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:55.565+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.570464+0000) 2019-10-13T15:06:55.573 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:06:55.565+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.570895+0000) 2019-10-13T15:06:55.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:55.569+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.571367+0000) 2019-10-13T15:06:55.574 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:55.569+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.571977+0000) 2019-10-13T15:06:55.574 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:06:55.569+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.572448+0000) 2019-10-13T15:06:55.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:06:55.572+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.576195+0000) 2019-10-13T15:06:55.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:06:55.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:06:55.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:06:55.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.691133+0000) 2019-10-13T15:06:55.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:06:55.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.702883+0000) 2019-10-13T15:06:58.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:06:58.937+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:58.941002+0000) 2019-10-13T15:07:00.585 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:00.577+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.582844+0000) 2019-10-13T15:07:00.586 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:00.577+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.583244+0000) 2019-10-13T15:07:00.586 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:00.581+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.583820+0000) 2019-10-13T15:07:00.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:00.581+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.584222+0000) 2019-10-13T15:07:00.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:00.580+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.585287+0000) 2019-10-13T15:07:00.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:00.581+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.585418+0000) 2019-10-13T15:07:00.588 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:00.581+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.586135+0000) 2019-10-13T15:07:00.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:00.584+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:00.588934+0000) 2019-10-13T15:07:00.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:00.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:02.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:02.676+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.680405+0000) 2019-10-13T15:07:02.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:02.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.742523+0000) 2019-10-13T15:07:03.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:03.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:03.417945+0000) 2019-10-13T15:07:03.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:03.664+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:03.667523+0000) 2019-10-13T15:07:03.852 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:03.848+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:03.850410+0000) 2019-10-13T15:07:05.599 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:05.592+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.597010+0000) 2019-10-13T15:07:05.601 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:05.592+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.598048+0000) 2019-10-13T15:07:05.601 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:05.592+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.598342+0000) 2019-10-13T15:07:05.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:05.592+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.598538+0000) 2019-10-13T15:07:05.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:05.592+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.599008+0000) 2019-10-13T15:07:05.602 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:05.592+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.599184+0000) 2019-10-13T15:07:05.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:05.596+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.599808+0000) 2019-10-13T15:07:05.608 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:05.600+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.605889+0000) 2019-10-13T15:07:05.610 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:05.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:05.648 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:07:05.649 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:05.652 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:07:05.652 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:05.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:05.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.691326+0000) 2019-10-13T15:07:05.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:05.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.703047+0000) 2019-10-13T15:07:08.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:08.936+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:08.941195+0000) 2019-10-13T15:07:10.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:10.608+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.610378+0000) 2019-10-13T15:07:10.613 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:10.604+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.610957+0000) 2019-10-13T15:07:10.614 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:10.608+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.611992+0000) 2019-10-13T15:07:10.615 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:10.608+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.612986+0000) 2019-10-13T15:07:10.616 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:10.608+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.613698+0000) 2019-10-13T15:07:10.616 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:10.608+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.613953+0000) 2019-10-13T15:07:10.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:10.608+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.614677+0000) 2019-10-13T15:07:10.622 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:10.616+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:10.619949+0000) 2019-10-13T15:07:10.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:10.616+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:12.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:12.676+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.680640+0000) 2019-10-13T15:07:12.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:12.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.742724+0000) 2019-10-13T15:07:13.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:13.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:13.418141+0000) 2019-10-13T15:07:13.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:13.664+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:13.667721+0000) 2019-10-13T15:07:13.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:13.852+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:13.856581+0000) 2019-10-13T15:07:15.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:15.620+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.624940+0000) 2019-10-13T15:07:15.628 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:15.620+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.625404+0000) 2019-10-13T15:07:15.628 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:15.620+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.626335+0000) 2019-10-13T15:07:15.629 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:15.620+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.626752+0000) 2019-10-13T15:07:15.629 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:15.624+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.627469+0000) 2019-10-13T15:07:15.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:15.624+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.628191+0000) 2019-10-13T15:07:15.631 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:15.624+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.628605+0000) 2019-10-13T15:07:15.636 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:15.632+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.634503+0000) 2019-10-13T15:07:15.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:15.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:15.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:15.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.691520+0000) 2019-10-13T15:07:15.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:15.696+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.703215+0000) 2019-10-13T15:07:18.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:18.940+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:18.941434+0000) 2019-10-13T15:07:20.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:20.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.640093+0000) 2019-10-13T15:07:20.643 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:20.636+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.640927+0000) 2019-10-13T15:07:20.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:20.636+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.640522+0000) 2019-10-13T15:07:20.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:20.636+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.641996+0000) 2019-10-13T15:07:20.645 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:20.636+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.642279+0000) 2019-10-13T15:07:20.645 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:20.636+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.642925+0000) 2019-10-13T15:07:20.646 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:20.636+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.643106+0000) 2019-10-13T15:07:20.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:20.644+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:20.647949+0000) 2019-10-13T15:07:20.653 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:20.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:22.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:22.680+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.680867+0000) 2019-10-13T15:07:22.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:22.736+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.742959+0000) 2019-10-13T15:07:23.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:23.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:23.418336+0000) 2019-10-13T15:07:23.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:23.664+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:23.667915+0000) 2019-10-13T15:07:23.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:23.860+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:23.863594+0000) 2019-10-13T15:07:25.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:25.648+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.654567+0000) 2019-10-13T15:07:25.657 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:25.648+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.654934+0000) 2019-10-13T15:07:25.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:25.652+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.655655+0000) 2019-10-13T15:07:25.658 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:25.652+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.656121+0000) 2019-10-13T15:07:25.659 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:25.652+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.656715+0000) 2019-10-13T15:07:25.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:25.652+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.657065+0000) 2019-10-13T15:07:25.660 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:25.652+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.657642+0000) 2019-10-13T15:07:25.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:25.660+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.665387+0000) 2019-10-13T15:07:25.668 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:25.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:25.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:25.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.691729+0000) 2019-10-13T15:07:25.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:25.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.703393+0000) 2019-10-13T15:07:28.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:28.940+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:28.941699+0000) 2019-10-13T15:07:30.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:30.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.668127+0000) 2019-10-13T15:07:30.671 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:30.664+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.668369+0000) 2019-10-13T15:07:30.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:30.664+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.669081+0000) 2019-10-13T15:07:30.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:30.663+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.669979+0000) 2019-10-13T15:07:30.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:30.664+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.670471+0000) 2019-10-13T15:07:30.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:30.664+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.671052+0000) 2019-10-13T15:07:30.674 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:30.668+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.671355+0000) 2019-10-13T15:07:30.678 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:30.671+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:30.676035+0000) 2019-10-13T15:07:30.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:30.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212478 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:32.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:32.680+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.681094+0000) 2019-10-13T15:07:32.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:32.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.743162+0000) 2019-10-13T15:07:33.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:33.412+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:33.418528+0000) 2019-10-13T15:07:33.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:33.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:33.668107+0000) 2019-10-13T15:07:33.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:33.867+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:33.870124+0000) 2019-10-13T15:07:35.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:35.679+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.682615+0000) 2019-10-13T15:07:35.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:35.676+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.683233+0000) 2019-10-13T15:07:35.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:35.680+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.684077+0000) 2019-10-13T15:07:35.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:35.680+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.684501+0000) 2019-10-13T15:07:35.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:35.680+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.685392+0000) 2019-10-13T15:07:35.688 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:35.680+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.685729+0000) 2019-10-13T15:07:35.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:35.680+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.686599+0000) 2019-10-13T15:07:35.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:35.683+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.689817+0000) 2019-10-13T15:07:35.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:35.688+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.691878+0000) 2019-10-13T15:07:35.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:35.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:35.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:35.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.703545+0000) 2019-10-13T15:07:35.727 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:07:35.727 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:35.731 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:07:35.731 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:38.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:38.936+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:38.941894+0000) 2019-10-13T15:07:40.700 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:40.692+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.697748+0000) 2019-10-13T15:07:40.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:40.692+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.697998+0000) 2019-10-13T15:07:40.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:40.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.698599+0000) 2019-10-13T15:07:40.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:40.692+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.699074+0000) 2019-10-13T15:07:40.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:40.695+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.699778+0000) 2019-10-13T15:07:40.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:40.696+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.700151+0000) 2019-10-13T15:07:40.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:40.696+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.702307+0000) 2019-10-13T15:07:40.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:40.703+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:40.707321+0000) 2019-10-13T15:07:40.710 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:40.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:42.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:42.680+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.681331+0000) 2019-10-13T15:07:42.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:42.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.743362+0000) 2019-10-13T15:07:43.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:43.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:43.418727+0000) 2019-10-13T15:07:43.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:43.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:43.668298+0000) 2019-10-13T15:07:43.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:43.875+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:43.881627+0000) 2019-10-13T15:07:45.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:45.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.692057+0000) 2019-10-13T15:07:45.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:45.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.703736+0000) 2019-10-13T15:07:45.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:45.708+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.711426+0000) 2019-10-13T15:07:45.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:45.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.713449+0000) 2019-10-13T15:07:45.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:45.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.713620+0000) 2019-10-13T15:07:45.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:45.708+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.713959+0000) 2019-10-13T15:07:45.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:45.711+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.714551+0000) 2019-10-13T15:07:45.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:45.708+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.714842+0000) 2019-10-13T15:07:45.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:45.712+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.715789+0000) 2019-10-13T15:07:45.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:45.715+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.719040+0000) 2019-10-13T15:07:45.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:45.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:48.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:48.939+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:48.942093+0000) 2019-10-13T15:07:50.730 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:50.719+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.727241+0000) 2019-10-13T15:07:50.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:50.723+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.727460+0000) 2019-10-13T15:07:50.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:50.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.728049+0000) 2019-10-13T15:07:50.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:50.723+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.728369+0000) 2019-10-13T15:07:50.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:50.723+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.729461+0000) 2019-10-13T15:07:50.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:50.723+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.730229+0000) 2019-10-13T15:07:50.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:50.727+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.731871+0000) 2019-10-13T15:07:50.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:50.731+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:50.737356+0000) 2019-10-13T15:07:50.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:50.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:52.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:52.679+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.681533+0000) 2019-10-13T15:07:52.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:52.739+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.743560+0000) 2019-10-13T15:07:53.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:53.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:53.418899+0000) 2019-10-13T15:07:53.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:53.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:53.668507+0000) 2019-10-13T15:07:53.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:53.883+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:53.888057+0000) 2019-10-13T15:07:55.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:55.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.692261+0000) 2019-10-13T15:07:55.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:55.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.703950+0000) 2019-10-13T15:07:55.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:07:55.735+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.740642+0000) 2019-10-13T15:07:55.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:07:55.735+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.741824+0000) 2019-10-13T15:07:55.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:07:55.739+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.742210+0000) 2019-10-13T15:07:55.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:07:55.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.742461+0000) 2019-10-13T15:07:55.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:07:55.735+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.743125+0000) 2019-10-13T15:07:55.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:07:55.739+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.743513+0000) 2019-10-13T15:07:55.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:55.739+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.744289+0000) 2019-10-13T15:07:55.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:07:55.747+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.751071+0000) 2019-10-13T15:07:55.754 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:07:55.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:07:58.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:07:58.939+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:58.942303+0000) 2019-10-13T15:08:00.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:00.751+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.754959+0000) 2019-10-13T15:08:00.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:00.751+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.755411+0000) 2019-10-13T15:08:00.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:00.751+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.756139+0000) 2019-10-13T15:08:00.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:00.751+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.757263+0000) 2019-10-13T15:08:00.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:00.751+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.757419+0000) 2019-10-13T15:08:00.761 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:00.751+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.757901+0000) 2019-10-13T15:08:00.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:00.751+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.758677+0000) 2019-10-13T15:08:00.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:00.759+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:00.762978+0000) 2019-10-13T15:08:00.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:00.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:02.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:02.679+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.681741+0000) 2019-10-13T15:08:02.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:02.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.743763+0000) 2019-10-13T15:08:03.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:03.411+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:03.419061+0000) 2019-10-13T15:08:03.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:03.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:03.668731+0000) 2019-10-13T15:08:03.902 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:03.899+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:03.899731+0000) 2019-10-13T15:08:05.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:05.687+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.692458+0000) 2019-10-13T15:08:05.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:05.699+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.704170+0000) 2019-10-13T15:08:05.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:05.763+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.768542+0000) 2019-10-13T15:08:05.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:05.763+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.770413+0000) 2019-10-13T15:08:05.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:05.763+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.770536+0000) 2019-10-13T15:08:05.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:05.767+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.771816+0000) 2019-10-13T15:08:05.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:05.767+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.771769+0000) 2019-10-13T15:08:05.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:05.767+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.772078+0000) 2019-10-13T15:08:05.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:05.767+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.772598+0000) 2019-10-13T15:08:05.779 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:05.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.777450+0000) 2019-10-13T15:08:05.781 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:05.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212714 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:05.934 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:08:05.934 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:05.938 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:08:05.938 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:08.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:08.939+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:08.942502+0000) 2019-10-13T15:08:10.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:10.779+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.782657+0000) 2019-10-13T15:08:10.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:10.779+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.783614+0000) 2019-10-13T15:08:10.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:10.779+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.783922+0000) 2019-10-13T15:08:10.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:10.779+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.784792+0000) 2019-10-13T15:08:10.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:10.779+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.785393+0000) 2019-10-13T15:08:10.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:10.779+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.785759+0000) 2019-10-13T15:08:10.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:10.779+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.786448+0000) 2019-10-13T15:08:10.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:10.787+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:10.791084+0000) 2019-10-13T15:08:10.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:10.787+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212746 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:12.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:12.679+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.681941+0000) 2019-10-13T15:08:12.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:12.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.743975+0000) 2019-10-13T15:08:13.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:13.415+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:13.419254+0000) 2019-10-13T15:08:13.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:13.663+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:13.668933+0000) 2019-10-13T15:08:13.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:13.903+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:13.906055+0000) 2019-10-13T15:08:15.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:15.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.692691+0000) 2019-10-13T15:08:15.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:15.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.704370+0000) 2019-10-13T15:08:15.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:15.791+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.796379+0000) 2019-10-13T15:08:15.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:15.791+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.797471+0000) 2019-10-13T15:08:15.800 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:15.791+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.797787+0000) 2019-10-13T15:08:15.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:15.791+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.798051+0000) 2019-10-13T15:08:15.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:15.791+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.799083+0000) 2019-10-13T15:08:15.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:15.795+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.799445+0000) 2019-10-13T15:08:15.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:15.795+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.799911+0000) 2019-10-13T15:08:15.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:15.799+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.804981+0000) 2019-10-13T15:08:15.809 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:15.799+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212780 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:18.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:18.939+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:18.942723+0000) 2019-10-13T15:08:20.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:20.807+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.809606+0000) 2019-10-13T15:08:20.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:20.807+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.810966+0000) 2019-10-13T15:08:20.813 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:20.807+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.811246+0000) 2019-10-13T15:08:20.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:20.811+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.811631+0000) 2019-10-13T15:08:20.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:20.811+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.812509+0000) 2019-10-13T15:08:20.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:20.811+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.812781+0000) 2019-10-13T15:08:20.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:20.807+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.813385+0000) 2019-10-13T15:08:20.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:20.815+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:20.818713+0000) 2019-10-13T15:08:20.823 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:20.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212812 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:22.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:22.679+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.682163+0000) 2019-10-13T15:08:22.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:22.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.744223+0000) 2019-10-13T15:08:23.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:23.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:23.419476+0000) 2019-10-13T15:08:23.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:23.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:23.669127+0000) 2019-10-13T15:08:23.932 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:23.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:23.917740+0000) 2019-10-13T15:08:25.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:25.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.692918+0000) 2019-10-13T15:08:25.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:25.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.704572+0000) 2019-10-13T15:08:25.826 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:25.823+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.823985+0000) 2019-10-13T15:08:25.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:25.823+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.824578+0000) 2019-10-13T15:08:25.827 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:25.823+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.825101+0000) 2019-10-13T15:08:25.827 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:25.823+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.825385+0000) 2019-10-13T15:08:25.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:25.823+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.826467+0000) 2019-10-13T15:08:25.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:25.823+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.827054+0000) 2019-10-13T15:08:25.830 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:25.822+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.828132+0000) 2019-10-13T15:08:25.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:25.826+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.833189+0000) 2019-10-13T15:08:25.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:25.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212848 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:28.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:28.939+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:28.942958+0000) 2019-10-13T15:08:30.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:30.834+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.838694+0000) 2019-10-13T15:08:30.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:30.834+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.838954+0000) 2019-10-13T15:08:30.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:30.838+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.839380+0000) 2019-10-13T15:08:30.842 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:30.838+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.840168+0000) 2019-10-13T15:08:30.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:30.838+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.840543+0000) 2019-10-13T15:08:30.843 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:30.838+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.841243+0000) 2019-10-13T15:08:30.844 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:30.838+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.841728+0000) 2019-10-13T15:08:30.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:30.842+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:30.847286+0000) 2019-10-13T15:08:30.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:30.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212880 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:32.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:32.678+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:32.682390+0000) 2019-10-13T15:08:32.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:32.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:32.744422+0000) 2019-10-13T15:08:33.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:33.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.419669+0000) 2019-10-13T15:08:33.671 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:33.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.669354+0000) 2019-10-13T15:08:33.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:33.922+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.924295+0000) 2019-10-13T15:08:35.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:35.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.693118+0000) 2019-10-13T15:08:35.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:35.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.704767+0000) 2019-10-13T15:08:35.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:35.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.851665+0000) 2019-10-13T15:08:35.855 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:35.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.852823+0000) 2019-10-13T15:08:35.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:35.850+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.852973+0000) 2019-10-13T15:08:35.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:35.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.853403+0000) 2019-10-13T15:08:35.856 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:35.850+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.853806+0000) 2019-10-13T15:08:35.856 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:35.850+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.853522+0000) 2019-10-13T15:08:35.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:35.850+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.854221+0000) 2019-10-13T15:08:35.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:35.854+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.858926+0000) 2019-10-13T15:08:35.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:35.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212914 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:36.049 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:08:36.049 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:36.056 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:08:36.056 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:38.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:38.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.943155+0000) 2019-10-13T15:08:40.868 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:40.862+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.865841+0000) 2019-10-13T15:08:40.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:40.862+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.866645+0000) 2019-10-13T15:08:40.869 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:40.866+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.867351+0000) 2019-10-13T15:08:40.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:40.862+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.867786+0000) 2019-10-13T15:08:40.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:40.866+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.868425+0000) 2019-10-13T15:08:40.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:40.866+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.868223+0000) 2019-10-13T15:08:40.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:40.866+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.869023+0000) 2019-10-13T15:08:40.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:40.866+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:40.872623+0000) 2019-10-13T15:08:40.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:40.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212946 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:42.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:42.678+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:42.682594+0000) 2019-10-13T15:08:42.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:42.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:42.744631+0000) 2019-10-13T15:08:43.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:43.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.419857+0000) 2019-10-13T15:08:43.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:43.666+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.669575+0000) 2019-10-13T15:08:43.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:43.926+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.930689+0000) 2019-10-13T15:08:45.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:45.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.693337+0000) 2019-10-13T15:08:45.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:45.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.704964+0000) 2019-10-13T15:08:45.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:45.878+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.879402+0000) 2019-10-13T15:08:45.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:45.878+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.880141+0000) 2019-10-13T15:08:45.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:45.878+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.880866+0000) 2019-10-13T15:08:45.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:45.874+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.881431+0000) 2019-10-13T15:08:45.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:45.878+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.881438+0000) 2019-10-13T15:08:45.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:45.878+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.881822+0000) 2019-10-13T15:08:45.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:45.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.882452+0000) 2019-10-13T15:08:45.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:45.882+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.889389+0000) 2019-10-13T15:08:45.892 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:45.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 212982 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:48.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:48.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.943330+0000) 2019-10-13T15:08:50.896 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:50.890+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.893729+0000) 2019-10-13T15:08:50.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:50.890+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.894188+0000) 2019-10-13T15:08:50.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:50.890+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.894406+0000) 2019-10-13T15:08:50.897 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:50.890+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.895166+0000) 2019-10-13T15:08:50.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:50.890+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.895400+0000) 2019-10-13T15:08:50.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:50.894+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.895784+0000) 2019-10-13T15:08:50.898 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:50.894+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.896046+0000) 2019-10-13T15:08:50.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:50.898+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:50.903073+0000) 2019-10-13T15:08:50.906 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:50.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:52.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:52.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:52.682803+0000) 2019-10-13T15:08:52.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:52.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:52.744832+0000) 2019-10-13T15:08:53.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:53.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.420084+0000) 2019-10-13T15:08:53.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:53.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.669798+0000) 2019-10-13T15:08:53.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:53.934+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.937095+0000) 2019-10-13T15:08:55.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:55.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.693523+0000) 2019-10-13T15:08:55.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:55.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.705159+0000) 2019-10-13T15:08:55.910 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:08:55.902+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.907095+0000) 2019-10-13T15:08:55.910 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:55.906+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.907420+0000) 2019-10-13T15:08:55.911 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:08:55.906+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.908989+0000) 2019-10-13T15:08:55.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:08:55.906+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.909559+0000) 2019-10-13T15:08:55.912 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:08:55.906+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.910400+0000) 2019-10-13T15:08:55.913 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:08:55.906+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.910778+0000) 2019-10-13T15:08:55.913 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:08:55.906+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.911112+0000) 2019-10-13T15:08:55.917 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:08:55.910+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.915329+0000) 2019-10-13T15:08:55.920 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:08:55.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:08:58.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:08:58.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.943528+0000) 2019-10-13T15:09:00.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:00.918+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.921453+0000) 2019-10-13T15:09:00.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:00.918+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.922023+0000) 2019-10-13T15:09:00.925 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:00.918+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.922701+0000) 2019-10-13T15:09:00.925 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:00.918+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.922702+0000) 2019-10-13T15:09:00.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:00.922+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.923458+0000) 2019-10-13T15:09:00.926 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:00.922+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.923965+0000) 2019-10-13T15:09:00.926 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:00.918+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.924582+0000) 2019-10-13T15:09:00.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:00.926+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:00.931133+0000) 2019-10-13T15:09:00.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:00.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:02.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:02.678+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:02.683025+0000) 2019-10-13T15:09:02.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:02.742+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:02.745062+0000) 2019-10-13T15:09:03.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:03.418+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.420279+0000) 2019-10-13T15:09:03.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:03.662+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.669995+0000) 2019-10-13T15:09:03.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:03.942+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.943426+0000) 2019-10-13T15:09:05.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:05.690+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.693719+0000) 2019-10-13T15:09:05.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:05.702+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.705412+0000) 2019-10-13T15:09:05.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:05.934+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.936623+0000) 2019-10-13T15:09:05.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:05.934+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.937096+0000) 2019-10-13T15:09:05.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:05.934+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.937616+0000) 2019-10-13T15:09:05.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:05.934+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.938866+0000) 2019-10-13T15:09:05.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:05.934+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.939276+0000) 2019-10-13T15:09:05.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:05.934+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.939809+0000) 2019-10-13T15:09:05.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:05.938+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.940041+0000) 2019-10-13T15:09:05.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:05.938+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.944273+0000) 2019-10-13T15:09:05.950 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:05.946+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:06.179 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:09:06.180 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:06.186 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:09:06.186 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:08.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:08.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.943764+0000) 2019-10-13T15:09:10.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:10.946+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.950432+0000) 2019-10-13T15:09:10.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:10.950+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.952671+0000) 2019-10-13T15:09:10.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:10.950+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.953036+0000) 2019-10-13T15:09:10.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:10.950+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.953659+0000) 2019-10-13T15:09:10.961 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:10.950+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.954121+0000) 2019-10-13T15:09:10.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:10.950+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.954301+0000) 2019-10-13T15:09:10.963 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:10.950+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.954970+0000) 2019-10-13T15:09:10.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:10.954+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:10.958418+0000) 2019-10-13T15:09:10.975 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:10.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:12.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:12.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:12.683251+0000) 2019-10-13T15:09:12.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:12.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:12.745261+0000) 2019-10-13T15:09:13.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:13.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.420429+0000) 2019-10-13T15:09:13.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:13.670+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.670214+0000) 2019-10-13T15:09:13.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:13.946+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.949823+0000) 2019-10-13T15:09:15.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:15.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.693916+0000) 2019-10-13T15:09:15.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:15.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.705612+0000) 2019-10-13T15:09:15.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:15.961+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.965995+0000) 2019-10-13T15:09:15.969 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:15.961+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.967016+0000) 2019-10-13T15:09:15.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:15.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.967117+0000) 2019-10-13T15:09:15.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:15.966+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.967763+0000) 2019-10-13T15:09:15.970 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:15.965+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.967900+0000) 2019-10-13T15:09:15.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:15.965+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.969159+0000) 2019-10-13T15:09:15.971 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:15.965+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.969524+0000) 2019-10-13T15:09:15.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:15.974+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.974852+0000) 2019-10-13T15:09:15.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:15.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213182 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:18.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:18.941+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.943989+0000) 2019-10-13T15:09:20.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:20.977+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.980280+0000) 2019-10-13T15:09:20.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:20.977+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.980545+0000) 2019-10-13T15:09:20.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:20.977+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.981777+0000) 2019-10-13T15:09:20.984 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:20.981+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.982597+0000) 2019-10-13T15:09:20.985 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:20.977+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.983202+0000) 2019-10-13T15:09:20.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:20.981+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.983554+0000) 2019-10-13T15:09:20.986 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:20.981+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.984052+0000) 2019-10-13T15:09:20.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:20.985+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:20.989277+0000) 2019-10-13T15:09:20.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:20.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213213 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:22.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:22.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:22.683537+0000) 2019-10-13T15:09:22.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:22.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:22.745463+0000) 2019-10-13T15:09:23.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:23.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.420618+0000) 2019-10-13T15:09:23.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:23.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.670410+0000) 2019-10-13T15:09:23.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:23.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.956218+0000) 2019-10-13T15:09:25.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:25.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.694109+0000) 2019-10-13T15:09:25.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:25.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.705812+0000) 2019-10-13T15:09:26.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:25.993+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.996755+0000) 2019-10-13T15:09:26.008 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:25.993+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.997507+0000) 2019-10-13T15:09:26.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:25.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.998132+0000) 2019-10-13T15:09:26.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:25.993+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.998368+0000) 2019-10-13T15:09:26.009 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:25.993+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.999022+0000) 2019-10-13T15:09:26.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:25.993+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.999266+0000) 2019-10-13T15:09:26.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:26.001+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:26.003358+0000) 2019-10-13T15:09:26.018 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:25.997+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:26.000135+0000) 2019-10-13T15:09:26.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:26.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213250 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:28.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:28.941+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.944203+0000) 2019-10-13T15:09:31.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:31.009+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.010604+0000) 2019-10-13T15:09:31.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:31.005+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.010818+0000) 2019-10-13T15:09:31.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:31.009+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.011995+0000) 2019-10-13T15:09:31.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:31.009+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.012518+0000) 2019-10-13T15:09:31.015 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:31.009+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.013014+0000) 2019-10-13T15:09:31.015 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:31.009+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.013378+0000) 2019-10-13T15:09:31.016 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:31.009+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.013726+0000) 2019-10-13T15:09:31.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:31.013+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.016597+0000) 2019-10-13T15:09:31.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:31.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213281 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:32.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:32.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:32.683738+0000) 2019-10-13T15:09:32.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:32.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:32.745713+0000) 2019-10-13T15:09:33.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:33.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.420811+0000) 2019-10-13T15:09:33.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:33.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.670603+0000) 2019-10-13T15:09:33.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:33.965+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.967499+0000) 2019-10-13T15:09:35.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:35.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:35.694304+0000) 2019-10-13T15:09:35.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:35.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:35.706007+0000) 2019-10-13T15:09:36.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:36.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.023993+0000) 2019-10-13T15:09:36.028 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:36.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.025788+0000) 2019-10-13T15:09:36.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:36.021+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.026551+0000) 2019-10-13T15:09:36.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:36.021+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.026821+0000) 2019-10-13T15:09:36.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:36.025+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.027465+0000) 2019-10-13T15:09:36.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:36.025+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.028361+0000) 2019-10-13T15:09:36.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:36.025+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.029394+0000) 2019-10-13T15:09:36.036 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:36.033+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:36.034383+0000) 2019-10-13T15:09:36.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:36.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:36.312 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:09:36.312 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:36.316 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:09:36.316 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:38.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:38.941+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.944431+0000) 2019-10-13T15:09:41.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:41.033+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.037954+0000) 2019-10-13T15:09:41.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:41.033+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.038636+0000) 2019-10-13T15:09:41.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:41.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.039446+0000) 2019-10-13T15:09:41.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:41.037+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.039910+0000) 2019-10-13T15:09:41.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:41.037+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.040574+0000) 2019-10-13T15:09:41.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:41.037+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.041205+0000) 2019-10-13T15:09:41.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:41.037+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.041648+0000) 2019-10-13T15:09:41.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:41.045+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.047694+0000) 2019-10-13T15:09:41.051 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:41.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:42.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:42.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:42.683940+0000) 2019-10-13T15:09:42.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:42.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:42.745923+0000) 2019-10-13T15:09:43.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:43.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.420999+0000) 2019-10-13T15:09:43.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:43.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.670812+0000) 2019-10-13T15:09:43.976 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:43.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.973939+0000) 2019-10-13T15:09:45.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:45.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:45.694513+0000) 2019-10-13T15:09:45.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:45.701+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:45.706220+0000) 2019-10-13T15:09:46.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:46.049+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.052335+0000) 2019-10-13T15:09:46.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:46.049+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.053640+0000) 2019-10-13T15:09:46.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:46.049+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.054121+0000) 2019-10-13T15:09:46.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:46.049+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.054262+0000) 2019-10-13T15:09:46.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:46.049+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.054791+0000) 2019-10-13T15:09:46.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:46.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.055044+0000) 2019-10-13T15:09:46.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:46.053+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.056048+0000) 2019-10-13T15:09:46.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:46.061+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:46.062583+0000) 2019-10-13T15:09:46.065 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:46.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213387 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:48.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:48.941+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.944635+0000) 2019-10-13T15:09:51.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:51.061+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.066623+0000) 2019-10-13T15:09:51.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:51.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.067396+0000) 2019-10-13T15:09:51.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:51.065+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.067912+0000) 2019-10-13T15:09:51.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:51.065+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.068350+0000) 2019-10-13T15:09:51.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:51.065+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.068560+0000) 2019-10-13T15:09:51.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:51.065+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.068995+0000) 2019-10-13T15:09:51.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:51.065+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.069587+0000) 2019-10-13T15:09:51.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:51.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:51.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:51.077+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.079810+0000) 2019-10-13T15:09:52.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:52.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:52.684138+0000) 2019-10-13T15:09:52.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:52.741+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:52.746174+0000) 2019-10-13T15:09:53.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:53.417+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.421252+0000) 2019-10-13T15:09:53.673 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:53.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.671017+0000) 2019-10-13T15:09:53.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:53.977+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.980360+0000) 2019-10-13T15:09:55.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:55.689+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:55.694743+0000) 2019-10-13T15:09:55.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:55.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:55.706435+0000) 2019-10-13T15:09:56.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:09:56.076+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.080015+0000) 2019-10-13T15:09:56.083 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:09:56.076+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.080365+0000) 2019-10-13T15:09:56.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:09:56.076+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.081187+0000) 2019-10-13T15:09:56.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:09:56.076+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.081723+0000) 2019-10-13T15:09:56.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:56.076+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.082175+0000) 2019-10-13T15:09:56.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:09:56.076+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.083210+0000) 2019-10-13T15:09:56.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:09:56.081+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.083971+0000) 2019-10-13T15:09:56.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:09:56.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:09:56.096 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:09:56.089+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:56.093794+0000) 2019-10-13T15:09:58.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:09:58.940+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.944839+0000) 2019-10-13T15:10:01.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:01.088+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.094088+0000) 2019-10-13T15:10:01.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:01.093+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.095017+0000) 2019-10-13T15:10:01.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:01.092+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.096496+0000) 2019-10-13T15:10:01.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:01.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.096903+0000) 2019-10-13T15:10:01.099 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:01.092+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.097204+0000) 2019-10-13T15:10:01.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:01.092+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.097862+0000) 2019-10-13T15:10:01.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:01.092+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.098381+0000) 2019-10-13T15:10:01.104 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:01.101+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.102293+0000) 2019-10-13T15:10:01.107 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:01.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213485 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:02.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:02.680+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:02.684339+0000) 2019-10-13T15:10:02.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:02.740+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:02.746387+0000) 2019-10-13T15:10:03.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:03.416+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.421476+0000) 2019-10-13T15:10:03.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:03.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.671205+0000) 2019-10-13T15:10:03.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:03.985+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.986670+0000) 2019-10-13T15:10:05.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:05.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:05.694960+0000) 2019-10-13T15:10:05.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:05.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:05.706636+0000) 2019-10-13T15:10:06.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:06.105+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.108118+0000) 2019-10-13T15:10:06.111 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:06.104+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.108404+0000) 2019-10-13T15:10:06.112 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:06.104+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.109912+0000) 2019-10-13T15:10:06.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:06.104+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.110120+0000) 2019-10-13T15:10:06.113 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:06.104+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.110580+0000) 2019-10-13T15:10:06.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:06.104+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.110925+0000) 2019-10-13T15:10:06.115 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:06.108+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.112318+0000) 2019-10-13T15:10:06.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:06.117+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:06.118795+0000) 2019-10-13T15:10:06.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:06.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213525 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:06.415 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:10:06.415 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:06.419 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:10:06.419 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:08.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:08.944+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.945070+0000) 2019-10-13T15:10:11.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:11.121+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.123093+0000) 2019-10-13T15:10:11.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:11.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.123925+0000) 2019-10-13T15:10:11.126 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:11.120+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.124257+0000) 2019-10-13T15:10:11.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:11.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.124623+0000) 2019-10-13T15:10:11.127 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:11.120+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.125193+0000) 2019-10-13T15:10:11.128 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:11.120+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.125746+0000) 2019-10-13T15:10:11.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:11.120+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.126512+0000) 2019-10-13T15:10:11.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:11.129+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.132979+0000) 2019-10-13T15:10:11.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:11.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:12.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:12.680+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:12.684574+0000) 2019-10-13T15:10:12.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:12.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:12.746591+0000) 2019-10-13T15:10:13.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:13.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.421687+0000) 2019-10-13T15:10:13.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:13.669+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.671351+0000) 2019-10-13T15:10:14.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:13.993+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.996239+0000) 2019-10-13T15:10:15.697 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:15.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:15.695141+0000) 2019-10-13T15:10:15.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:15.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:15.706835+0000) 2019-10-13T15:10:16.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:16.132+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.136778+0000) 2019-10-13T15:10:16.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:16.132+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.137209+0000) 2019-10-13T15:10:16.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:16.132+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.138002+0000) 2019-10-13T15:10:16.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:16.136+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.139261+0000) 2019-10-13T15:10:16.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:16.136+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.139578+0000) 2019-10-13T15:10:16.142 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:16.136+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.139858+0000) 2019-10-13T15:10:16.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:16.136+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.140109+0000) 2019-10-13T15:10:16.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:16.140+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:16.145243+0000) 2019-10-13T15:10:16.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:16.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:18.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:18.944+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.945251+0000) 2019-10-13T15:10:21.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:21.148+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.152607+0000) 2019-10-13T15:10:21.155 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:21.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.153153+0000) 2019-10-13T15:10:21.156 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:21.148+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.153982+0000) 2019-10-13T15:10:21.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:21.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.154287+0000) 2019-10-13T15:10:21.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:21.148+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.154671+0000) 2019-10-13T15:10:21.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:21.152+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.155404+0000) 2019-10-13T15:10:21.158 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:21.148+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.154949+0000) 2019-10-13T15:10:21.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:21.156+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.160130+0000) 2019-10-13T15:10:21.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:21.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:22.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:22.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:22.684774+0000) 2019-10-13T15:10:22.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:22.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:22.746802+0000) 2019-10-13T15:10:23.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:23.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.421916+0000) 2019-10-13T15:10:23.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:23.668+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.671547+0000) 2019-10-13T15:10:24.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:24.000+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:24.002585+0000) 2019-10-13T15:10:25.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:25.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:25.695367+0000) 2019-10-13T15:10:25.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:25.700+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:25.707029+0000) 2019-10-13T15:10:26.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:26.160+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.166651+0000) 2019-10-13T15:10:26.169 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:26.160+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.167122+0000) 2019-10-13T15:10:26.170 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:26.164+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.167588+0000) 2019-10-13T15:10:26.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:26.164+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.167863+0000) 2019-10-13T15:10:26.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:26.164+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.168326+0000) 2019-10-13T15:10:26.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:26.164+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.169499+0000) 2019-10-13T15:10:26.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:26.164+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.169532+0000) 2019-10-13T15:10:26.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:26.172+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:26.176206+0000) 2019-10-13T15:10:26.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:26.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213663 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:28.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:28.944+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.945443+0000) 2019-10-13T15:10:31.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:31.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.181129+0000) 2019-10-13T15:10:31.184 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:31.176+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.182368+0000) 2019-10-13T15:10:31.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:31.180+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.183169+0000) 2019-10-13T15:10:31.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:31.180+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.184350+0000) 2019-10-13T15:10:31.187 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:31.180+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.184896+0000) 2019-10-13T15:10:31.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:31.180+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.185515+0000) 2019-10-13T15:10:31.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:31.180+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.186149+0000) 2019-10-13T15:10:31.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:31.188+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.190273+0000) 2019-10-13T15:10:31.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:31.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:32.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:32.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:32.684976+0000) 2019-10-13T15:10:32.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:32.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:32.747037+0000) 2019-10-13T15:10:33.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:33.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.422107+0000) 2019-10-13T15:10:33.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:33.668+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.671740+0000) 2019-10-13T15:10:34.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:34.004+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:34.008971+0000) 2019-10-13T15:10:35.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:35.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:35.695561+0000) 2019-10-13T15:10:35.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:35.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:35.707233+0000) 2019-10-13T15:10:36.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:36.192+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.195827+0000) 2019-10-13T15:10:36.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:36.192+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.196212+0000) 2019-10-13T15:10:36.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:36.192+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.196879+0000) 2019-10-13T15:10:36.200 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:36.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.196689+0000) 2019-10-13T15:10:36.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:36.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.198234+0000) 2019-10-13T15:10:36.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:36.196+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.198929+0000) 2019-10-13T15:10:36.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:36.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.199016+0000) 2019-10-13T15:10:36.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:36.200+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:36.204802+0000) 2019-10-13T15:10:36.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:36.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:36.535 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:10:36.536 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:36.539 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:10:36.539 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:38.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:38.940+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.945657+0000) 2019-10-13T15:10:41.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:41.208+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.210214+0000) 2019-10-13T15:10:41.213 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:41.203+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.210631+0000) 2019-10-13T15:10:41.213 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:41.203+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.210841+0000) 2019-10-13T15:10:41.214 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:41.203+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.211195+0000) 2019-10-13T15:10:41.214 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:41.207+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.212000+0000) 2019-10-13T15:10:41.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:41.207+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.213227+0000) 2019-10-13T15:10:41.215 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:41.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.213359+0000) 2019-10-13T15:10:41.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:41.216+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.218690+0000) 2019-10-13T15:10:41.222 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:41.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213761 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:42.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:42.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:42.685180+0000) 2019-10-13T15:10:42.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:42.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:42.747230+0000) 2019-10-13T15:10:43.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:43.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.422338+0000) 2019-10-13T15:10:43.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:43.668+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.671965+0000) 2019-10-13T15:10:44.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:44.012+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:44.015570+0000) 2019-10-13T15:10:45.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:45.691+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:45.695759+0000) 2019-10-13T15:10:45.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:45.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:45.707518+0000) 2019-10-13T15:10:46.226 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:46.219+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.223569+0000) 2019-10-13T15:10:46.226 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:46.220+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.224331+0000) 2019-10-13T15:10:46.227 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:46.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.223951+0000) 2019-10-13T15:10:46.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:46.219+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.224744+0000) 2019-10-13T15:10:46.228 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:46.219+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.225306+0000) 2019-10-13T15:10:46.228 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:46.219+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.226115+0000) 2019-10-13T15:10:46.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:46.223+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.227335+0000) 2019-10-13T15:10:46.234 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:46.228+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:46.232090+0000) 2019-10-13T15:10:46.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:46.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213801 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:48.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:48.943+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.945859+0000) 2019-10-13T15:10:51.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:51.231+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.238047+0000) 2019-10-13T15:10:51.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:51.235+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.239291+0000) 2019-10-13T15:10:51.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:51.236+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.239971+0000) 2019-10-13T15:10:51.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:51.235+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.240080+0000) 2019-10-13T15:10:51.243 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:51.235+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.240374+0000) 2019-10-13T15:10:51.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:51.235+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.241356+0000) 2019-10-13T15:10:51.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:51.235+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.242061+0000) 2019-10-13T15:10:51.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:51.244+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.246725+0000) 2019-10-13T15:10:51.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:51.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:52.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:52.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:52.685383+0000) 2019-10-13T15:10:52.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:52.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:52.747428+0000) 2019-10-13T15:10:53.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:53.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.422553+0000) 2019-10-13T15:10:53.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:53.668+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.672190+0000) 2019-10-13T15:10:54.024 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:54.016+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:54.022076+0000) 2019-10-13T15:10:55.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:55.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:55.695961+0000) 2019-10-13T15:10:55.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:55.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:55.707715+0000) 2019-10-13T15:10:56.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:56.247+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.252479+0000) 2019-10-13T15:10:56.255 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:10:56.247+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.253333+0000) 2019-10-13T15:10:56.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:10:56.247+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.254243+0000) 2019-10-13T15:10:56.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:10:56.247+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.254344+0000) 2019-10-13T15:10:56.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:10:56.247+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.254728+0000) 2019-10-13T15:10:56.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:10:56.247+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.255105+0000) 2019-10-13T15:10:56.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:10:56.252+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.255864+0000) 2019-10-13T15:10:56.262 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:10:56.256+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:56.260644+0000) 2019-10-13T15:10:56.264 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:10:56.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:10:58.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:10:58.943+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.946080+0000) 2019-10-13T15:11:01.268 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:01.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.265530+0000) 2019-10-13T15:11:01.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:01.259+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.265999+0000) 2019-10-13T15:11:01.269 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:01.259+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.266470+0000) 2019-10-13T15:11:01.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:01.263+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.267729+0000) 2019-10-13T15:11:01.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:01.263+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.268124+0000) 2019-10-13T15:11:01.271 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:01.264+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.269058+0000) 2019-10-13T15:11:01.271 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:01.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.269322+0000) 2019-10-13T15:11:01.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:01.272+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.274906+0000) 2019-10-13T15:11:01.278 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:01.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:02.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:02.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:02.685584+0000) 2019-10-13T15:11:02.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:02.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:02.747591+0000) 2019-10-13T15:11:03.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:03.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.422753+0000) 2019-10-13T15:11:03.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:03.668+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.672408+0000) 2019-10-13T15:11:04.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:04.032+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:04.033446+0000) 2019-10-13T15:11:05.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:05.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:05.696193+0000) 2019-10-13T15:11:05.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:05.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:05.707909+0000) 2019-10-13T15:11:06.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:06.275+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.279343+0000) 2019-10-13T15:11:06.282 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:06.275+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.279553+0000) 2019-10-13T15:11:06.283 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:06.275+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.280164+0000) 2019-10-13T15:11:06.283 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:06.275+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.280716+0000) 2019-10-13T15:11:06.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:06.276+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.281488+0000) 2019-10-13T15:11:06.284 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:06.275+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.281613+0000) 2019-10-13T15:11:06.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:06.275+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.282687+0000) 2019-10-13T15:11:06.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:06.284+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:06.289483+0000) 2019-10-13T15:11:06.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:06.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:06.637 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:11:06.638 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:06.641 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:11:06.641 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:08.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:08.943+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.946283+0000) 2019-10-13T15:11:11.296 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:11.291+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.293720+0000) 2019-10-13T15:11:11.297 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:11.291+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.295104+0000) 2019-10-13T15:11:11.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:11.295+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.295551+0000) 2019-10-13T15:11:11.298 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:11.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.295912+0000) 2019-10-13T15:11:11.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:11.295+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.296458+0000) 2019-10-13T15:11:11.299 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:11.295+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.296669+0000) 2019-10-13T15:11:11.299 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:11.291+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.297494+0000) 2019-10-13T15:11:11.303 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:11.295+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.301350+0000) 2019-10-13T15:11:11.306 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:11.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 213968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:12.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:12.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:12.685802+0000) 2019-10-13T15:11:12.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:12.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:12.747854+0000) 2019-10-13T15:11:13.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:13.419+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.422970+0000) 2019-10-13T15:11:13.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:13.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.672598+0000) 2019-10-13T15:11:14.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:14.039+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:14.040080+0000) 2019-10-13T15:11:15.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:15.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:15.696409+0000) 2019-10-13T15:11:15.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:15.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:15.708105+0000) 2019-10-13T15:11:16.312 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:16.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.308052+0000) 2019-10-13T15:11:16.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:16.303+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.309358+0000) 2019-10-13T15:11:16.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:16.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.309171+0000) 2019-10-13T15:11:16.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:16.307+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.309574+0000) 2019-10-13T15:11:16.315 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:16.307+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.309961+0000) 2019-10-13T15:11:16.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:16.307+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.310806+0000) 2019-10-13T15:11:16.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:16.311+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.312044+0000) 2019-10-13T15:11:16.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:16.311+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:16.317953+0000) 2019-10-13T15:11:16.321 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:16.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:18.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:18.943+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.946434+0000) 2019-10-13T15:11:21.323 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:21.319+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.321296+0000) 2019-10-13T15:11:21.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:21.319+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.322455+0000) 2019-10-13T15:11:21.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:21.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.322878+0000) 2019-10-13T15:11:21.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:21.323+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.323957+0000) 2019-10-13T15:11:21.327 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:21.319+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.324793+0000) 2019-10-13T15:11:21.327 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:21.323+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.325098+0000) 2019-10-13T15:11:21.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:21.323+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.326324+0000) 2019-10-13T15:11:21.335 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:21.327+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.332736+0000) 2019-10-13T15:11:21.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:21.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214037 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:22.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:22.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:22.686007+0000) 2019-10-13T15:11:22.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:22.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:22.748067+0000) 2019-10-13T15:11:23.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:23.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.423169+0000) 2019-10-13T15:11:23.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:23.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.672813+0000) 2019-10-13T15:11:24.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:24.043+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:24.046565+0000) 2019-10-13T15:11:25.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:25.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:25.696607+0000) 2019-10-13T15:11:25.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:25.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:25.708366+0000) 2019-10-13T15:11:26.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:26.334+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.335842+0000) 2019-10-13T15:11:26.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:26.334+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.336430+0000) 2019-10-13T15:11:26.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:26.334+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.337115+0000) 2019-10-13T15:11:26.340 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:26.334+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.337641+0000) 2019-10-13T15:11:26.341 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:26.335+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.338773+0000) 2019-10-13T15:11:26.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:26.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.339014+0000) 2019-10-13T15:11:26.341 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:26.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.339151+0000) 2019-10-13T15:11:26.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:26.339+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:26.346034+0000) 2019-10-13T15:11:26.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:26.342+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:28.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:28.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.946614+0000) 2019-10-13T15:11:31.370 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:31.343+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.349247+0000) 2019-10-13T15:11:31.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:31.346+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.350147+0000) 2019-10-13T15:11:31.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:31.346+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.350703+0000) 2019-10-13T15:11:31.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:31.346+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.351171+0000) 2019-10-13T15:11:31.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:31.350+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.351865+0000) 2019-10-13T15:11:31.371 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:31.350+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.352186+0000) 2019-10-13T15:11:31.374 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:31.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.352391+0000) 2019-10-13T15:11:31.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:31.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:31.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:31.351+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.355605+0000) 2019-10-13T15:11:32.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:32.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:32.686236+0000) 2019-10-13T15:11:32.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:32.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:32.748266+0000) 2019-10-13T15:11:33.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:33.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.423368+0000) 2019-10-13T15:11:33.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:33.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.673019+0000) 2019-10-13T15:11:34.055 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:34.051+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:34.052798+0000) 2019-10-13T15:11:35.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:35.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:35.696810+0000) 2019-10-13T15:11:35.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:35.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:35.708564+0000) 2019-10-13T15:11:36.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:36.358+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.362411+0000) 2019-10-13T15:11:36.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:36.362+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.363266+0000) 2019-10-13T15:11:36.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:36.362+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.364069+0000) 2019-10-13T15:11:36.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:36.362+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.364260+0000) 2019-10-13T15:11:36.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:36.359+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.364716+0000) 2019-10-13T15:11:36.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:36.362+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.364778+0000) 2019-10-13T15:11:36.368 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:36.362+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.365234+0000) 2019-10-13T15:11:36.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:36.363+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:36.369251+0000) 2019-10-13T15:11:36.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:36.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:36.766 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:11:36.766 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:36.770 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:11:36.770 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:38.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:38.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.946795+0000) 2019-10-13T15:11:41.379 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:41.374+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.377224+0000) 2019-10-13T15:11:41.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:41.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.377739+0000) 2019-10-13T15:11:41.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:41.374+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.377996+0000) 2019-10-13T15:11:41.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:41.374+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.378685+0000) 2019-10-13T15:11:41.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:41.374+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.378806+0000) 2019-10-13T15:11:41.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:41.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.379169+0000) 2019-10-13T15:11:41.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:41.378+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.379687+0000) 2019-10-13T15:11:41.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:41.379+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.383335+0000) 2019-10-13T15:11:41.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:41.386+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214175 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:42.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:42.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:42.686427+0000) 2019-10-13T15:11:42.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:42.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:42.748478+0000) 2019-10-13T15:11:43.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:43.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.423583+0000) 2019-10-13T15:11:43.675 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:43.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.673226+0000) 2019-10-13T15:11:44.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:44.055+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:44.059264+0000) 2019-10-13T15:11:45.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:45.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:45.697011+0000) 2019-10-13T15:11:45.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:45.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:45.708766+0000) 2019-10-13T15:11:46.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:46.387+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.393962+0000) 2019-10-13T15:11:46.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:46.395+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.399026+0000) 2019-10-13T15:11:46.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:46.386+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.390715+0000) 2019-10-13T15:11:46.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:46.390+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.392220+0000) 2019-10-13T15:11:46.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:46.390+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.392707+0000) 2019-10-13T15:11:46.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:46.390+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.393030+0000) 2019-10-13T15:11:46.412 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:46.390+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.393322+0000) 2019-10-13T15:11:46.412 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:46.390+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:46.394473+0000) 2019-10-13T15:11:46.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:46.398+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:48.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:48.942+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.946978+0000) 2019-10-13T15:11:51.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:51.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.405206+0000) 2019-10-13T15:11:51.408 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:51.402+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.405593+0000) 2019-10-13T15:11:51.408 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:51.403+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.406238+0000) 2019-10-13T15:11:51.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:51.402+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.406504+0000) 2019-10-13T15:11:51.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:51.406+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.407616+0000) 2019-10-13T15:11:51.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:51.406+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.407927+0000) 2019-10-13T15:11:51.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:51.406+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.408488+0000) 2019-10-13T15:11:51.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:51.411+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.416135+0000) 2019-10-13T15:11:51.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:51.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:52.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:52.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:52.686631+0000) 2019-10-13T15:11:52.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:52.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:52.748677+0000) 2019-10-13T15:11:53.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:53.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.423817+0000) 2019-10-13T15:11:53.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:53.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.673445+0000) 2019-10-13T15:11:54.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:54.063+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:54.065606+0000) 2019-10-13T15:11:55.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:55.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:55.697216+0000) 2019-10-13T15:11:55.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:55.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:55.708968+0000) 2019-10-13T15:11:56.421 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:56.414+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.418741+0000) 2019-10-13T15:11:56.422 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:11:56.418+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.420157+0000) 2019-10-13T15:11:56.423 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:11:56.418+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.420051+0000) 2019-10-13T15:11:56.423 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:11:56.418+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.420831+0000) 2019-10-13T15:11:56.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:11:56.418+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.421155+0000) 2019-10-13T15:11:56.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:11:56.415+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.421924+0000) 2019-10-13T15:11:56.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:11:56.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.421504+0000) 2019-10-13T15:11:56.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:11:56.423+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:56.429448+0000) 2019-10-13T15:11:56.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:11:56.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214281 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:11:58.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:11:58.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.947174+0000) 2019-10-13T15:12:01.436 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:01.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.433523+0000) 2019-10-13T15:12:01.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:01.430+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.434054+0000) 2019-10-13T15:12:01.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:01.430+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.434380+0000) 2019-10-13T15:12:01.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:01.431+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.435156+0000) 2019-10-13T15:12:01.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:01.434+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.436453+0000) 2019-10-13T15:12:01.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:01.434+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.436726+0000) 2019-10-13T15:12:01.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:01.434+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.437068+0000) 2019-10-13T15:12:01.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:01.439+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.444148+0000) 2019-10-13T15:12:01.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:01.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214313 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:02.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:02.682+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:02.686830+0000) 2019-10-13T15:12:02.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:02.746+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:02.748872+0000) 2019-10-13T15:12:03.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:03.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.424014+0000) 2019-10-13T15:12:03.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:03.667+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.673658+0000) 2019-10-13T15:12:04.074 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:04.071+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:04.072113+0000) 2019-10-13T15:12:05.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:05.694+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:05.697418+0000) 2019-10-13T15:12:05.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:05.706+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:05.709166+0000) 2019-10-13T15:12:06.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:06.445+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.449165+0000) 2019-10-13T15:12:06.452 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:06.445+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.449426+0000) 2019-10-13T15:12:06.452 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:06.445+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.449825+0000) 2019-10-13T15:12:06.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:06.445+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.450123+0000) 2019-10-13T15:12:06.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:06.445+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.450493+0000) 2019-10-13T15:12:06.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:06.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.450812+0000) 2019-10-13T15:12:06.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:06.446+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.451519+0000) 2019-10-13T15:12:06.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:06.454+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:06.458443+0000) 2019-10-13T15:12:06.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:06.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214351 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:06.877 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:12:06.877 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:06.881 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:12:06.881 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:08.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:08.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.947409+0000) 2019-10-13T15:12:11.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:11.457+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.463212+0000) 2019-10-13T15:12:11.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:11.461+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.463986+0000) 2019-10-13T15:12:11.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:11.461+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.464287+0000) 2019-10-13T15:12:11.469 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:11.461+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.464641+0000) 2019-10-13T15:12:11.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:11.461+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.464889+0000) 2019-10-13T15:12:11.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:11.458+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.465884+0000) 2019-10-13T15:12:11.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:11.461+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.466356+0000) 2019-10-13T15:12:11.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:11.466+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.472597+0000) 2019-10-13T15:12:11.476 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:11.469+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:12.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:12.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:12.687051+0000) 2019-10-13T15:12:12.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:12.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:12.749076+0000) 2019-10-13T15:12:13.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:13.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.424228+0000) 2019-10-13T15:12:13.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:13.666+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.673876+0000) 2019-10-13T15:12:14.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:14.074+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:14.078623+0000) 2019-10-13T15:12:15.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:15.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:15.697623+0000) 2019-10-13T15:12:15.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:15.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:15.709363+0000) 2019-10-13T15:12:16.479 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:16.473+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.477169+0000) 2019-10-13T15:12:16.480 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:16.474+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.477842+0000) 2019-10-13T15:12:16.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:16.473+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.478284+0000) 2019-10-13T15:12:16.481 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:16.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.479204+0000) 2019-10-13T15:12:16.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:16.477+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.479638+0000) 2019-10-13T15:12:16.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:16.477+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.480306+0000) 2019-10-13T15:12:16.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:16.477+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.480692+0000) 2019-10-13T15:12:16.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:16.482+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:16.486932+0000) 2019-10-13T15:12:16.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:16.485+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:18.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:18.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.947599+0000) 2019-10-13T15:12:21.494 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:21.489+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.491429+0000) 2019-10-13T15:12:21.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:21.486+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.492101+0000) 2019-10-13T15:12:21.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:21.489+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.492344+0000) 2019-10-13T15:12:21.496 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:21.489+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.493037+0000) 2019-10-13T15:12:21.496 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:21.489+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.493121+0000) 2019-10-13T15:12:21.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:21.489+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.493453+0000) 2019-10-13T15:12:21.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:21.489+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.494329+0000) 2019-10-13T15:12:21.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:21.494+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.498818+0000) 2019-10-13T15:12:21.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:21.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:22.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:22.681+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:22.687259+0000) 2019-10-13T15:12:22.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:22.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:22.749230+0000) 2019-10-13T15:12:23.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:23.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.424413+0000) 2019-10-13T15:12:23.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:23.666+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.674063+0000) 2019-10-13T15:12:24.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:24.082+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:24.084918+0000) 2019-10-13T15:12:25.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:25.693+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:25.697817+0000) 2019-10-13T15:12:25.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:25.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:25.709562+0000) 2019-10-13T15:12:26.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:26.502+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.505591+0000) 2019-10-13T15:12:26.508 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:26.501+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.505933+0000) 2019-10-13T15:12:26.509 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:26.501+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.506524+0000) 2019-10-13T15:12:26.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:26.501+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.507133+0000) 2019-10-13T15:12:26.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:26.505+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.507650+0000) 2019-10-13T15:12:26.510 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:26.505+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.508042+0000) 2019-10-13T15:12:26.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:26.505+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.509030+0000) 2019-10-13T15:12:26.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:26.510+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:26.515345+0000) 2019-10-13T15:12:26.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:26.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214489 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:28.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:28.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.947796+0000) 2019-10-13T15:12:31.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:31.513+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.519007+0000) 2019-10-13T15:12:31.522 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:31.517+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.519438+0000) 2019-10-13T15:12:31.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:31.517+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.520463+0000) 2019-10-13T15:12:31.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:31.514+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.521140+0000) 2019-10-13T15:12:31.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:31.517+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.521261+0000) 2019-10-13T15:12:31.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:31.517+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.521691+0000) 2019-10-13T15:12:31.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:31.517+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.522714+0000) 2019-10-13T15:12:31.530 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:31.522+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.528264+0000) 2019-10-13T15:12:31.532 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:31.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:32.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:32.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:32.687475+0000) 2019-10-13T15:12:32.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:32.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:32.749429+0000) 2019-10-13T15:12:33.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:33.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.424615+0000) 2019-10-13T15:12:33.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:33.670+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.674250+0000) 2019-10-13T15:12:34.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:34.090+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:34.091426+0000) 2019-10-13T15:12:35.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:35.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:35.698034+0000) 2019-10-13T15:12:35.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:35.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:35.709754+0000) 2019-10-13T15:12:36.534 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:36.529+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.532256+0000) 2019-10-13T15:12:36.535 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:36.529+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.532731+0000) 2019-10-13T15:12:36.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:36.529+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.533087+0000) 2019-10-13T15:12:36.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:36.529+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.534422+0000) 2019-10-13T15:12:36.537 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:36.529+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.535038+0000) 2019-10-13T15:12:36.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:36.530+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.535482+0000) 2019-10-13T15:12:36.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:36.533+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.536130+0000) 2019-10-13T15:12:36.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:36.534+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:36.541043+0000) 2019-10-13T15:12:36.547 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:36.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:36.963 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:12:36.963 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:36.967 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:12:36.967 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:38.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:38.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.948002+0000) 2019-10-13T15:12:41.551 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:41.545+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.548485+0000) 2019-10-13T15:12:41.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:41.545+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.548779+0000) 2019-10-13T15:12:41.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:41.542+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.549647+0000) 2019-10-13T15:12:41.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:41.545+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.549767+0000) 2019-10-13T15:12:41.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:41.545+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.550562+0000) 2019-10-13T15:12:41.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:41.545+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.550547+0000) 2019-10-13T15:12:41.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:41.545+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.551260+0000) 2019-10-13T15:12:41.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:41.550+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.556613+0000) 2019-10-13T15:12:41.561 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:41.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214589 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:42.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:42.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:42.687717+0000) 2019-10-13T15:12:42.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:42.745+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:42.749625+0000) 2019-10-13T15:12:43.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:43.421+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.424814+0000) 2019-10-13T15:12:43.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:43.670+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.674444+0000) 2019-10-13T15:12:44.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:44.090+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:44.097806+0000) 2019-10-13T15:12:45.700 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:45.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:45.698236+0000) 2019-10-13T15:12:45.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:45.705+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:45.709946+0000) 2019-10-13T15:12:46.564 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:46.557+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.561995+0000) 2019-10-13T15:12:46.565 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:46.558+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.562554+0000) 2019-10-13T15:12:46.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:46.557+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.563142+0000) 2019-10-13T15:12:46.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:46.561+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.563747+0000) 2019-10-13T15:12:46.566 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:46.561+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.564462+0000) 2019-10-13T15:12:46.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:46.561+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.565198+0000) 2019-10-13T15:12:46.568 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:46.561+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.565962+0000) 2019-10-13T15:12:46.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:46.566+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:46.570906+0000) 2019-10-13T15:12:46.575 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:46.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:48.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:48.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.948226+0000) 2019-10-13T15:12:51.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:51.574+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.578598+0000) 2019-10-13T15:12:51.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:51.572+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.576515+0000) 2019-10-13T15:12:51.587 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:51.572+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.577136+0000) 2019-10-13T15:12:51.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:51.572+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.577686+0000) 2019-10-13T15:12:51.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:51.572+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.578725+0000) 2019-10-13T15:12:51.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:51.572+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.578999+0000) 2019-10-13T15:12:51.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:51.576+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.579620+0000) 2019-10-13T15:12:51.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:51.582+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.584712+0000) 2019-10-13T15:12:51.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:51.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214658 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:52.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:52.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:52.687927+0000) 2019-10-13T15:12:52.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:52.748+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:52.749824+0000) 2019-10-13T15:12:53.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:53.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.425013+0000) 2019-10-13T15:12:53.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:53.674+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.674593+0000) 2019-10-13T15:12:54.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:54.102+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:54.104110+0000) 2019-10-13T15:12:55.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:55.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:55.698436+0000) 2019-10-13T15:12:55.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:55.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:55.710157+0000) 2019-10-13T15:12:56.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:12:56.584+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.589477+0000) 2019-10-13T15:12:56.592 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:12:56.584+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.589835+0000) 2019-10-13T15:12:56.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:56.584+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.590375+0000) 2019-10-13T15:12:56.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:12:56.584+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.590795+0000) 2019-10-13T15:12:56.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:12:56.584+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.591128+0000) 2019-10-13T15:12:56.594 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:12:56.590+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.592392+0000) 2019-10-13T15:12:56.595 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:12:56.588+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.592621+0000) 2019-10-13T15:12:56.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:12:56.598+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:56.599350+0000) 2019-10-13T15:12:56.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:12:56.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214695 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:12:58.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:12:58.944+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.948460+0000) 2019-10-13T15:13:01.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:01.600+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.604319+0000) 2019-10-13T15:13:01.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:01.600+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.604581+0000) 2019-10-13T15:13:01.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:01.600+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.606256+0000) 2019-10-13T15:13:01.609 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:01.600+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.606714+0000) 2019-10-13T15:13:01.609 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:01.604+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.607424+0000) 2019-10-13T15:13:01.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:01.605+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.608118+0000) 2019-10-13T15:13:01.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:01.604+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.608593+0000) 2019-10-13T15:13:01.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:01.609+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.612596+0000) 2019-10-13T15:13:01.617 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:01.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:02.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:02.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:02.688146+0000) 2019-10-13T15:13:02.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:02.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:02.750036+0000) 2019-10-13T15:13:03.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:03.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.425232+0000) 2019-10-13T15:13:03.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:03.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.674803+0000) 2019-10-13T15:13:04.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:04.109+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:04.110421+0000) 2019-10-13T15:13:05.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:05.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:05.698635+0000) 2019-10-13T15:13:05.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:05.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:05.710386+0000) 2019-10-13T15:13:06.620 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:06.612+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.617793+0000) 2019-10-13T15:13:06.620 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:06.617+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.618608+0000) 2019-10-13T15:13:06.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:06.612+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.619194+0000) 2019-10-13T15:13:06.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:06.616+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.619701+0000) 2019-10-13T15:13:06.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:06.616+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.619886+0000) 2019-10-13T15:13:06.622 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:06.616+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.620403+0000) 2019-10-13T15:13:06.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:06.616+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.620766+0000) 2019-10-13T15:13:06.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:06.624+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:06.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:06.629+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:06.630502+0000) 2019-10-13T15:13:07.032 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:13:07.033 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:07.036 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:13:07.036 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:08.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:08.948+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.948662+0000) 2019-10-13T15:13:11.634 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:11.624+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.630940+0000) 2019-10-13T15:13:11.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:11.629+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.631837+0000) 2019-10-13T15:13:11.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:11.628+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.632319+0000) 2019-10-13T15:13:11.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:11.628+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.632934+0000) 2019-10-13T15:13:11.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:11.628+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.633578+0000) 2019-10-13T15:13:11.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:11.628+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.634205+0000) 2019-10-13T15:13:11.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:11.628+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.634573+0000) 2019-10-13T15:13:11.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:11.637+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.640531+0000) 2019-10-13T15:13:11.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:11.640+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:12.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:12.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:12.688376+0000) 2019-10-13T15:13:12.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:12.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:12.750242+0000) 2019-10-13T15:13:13.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:13.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.425432+0000) 2019-10-13T15:13:13.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:13.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.675025+0000) 2019-10-13T15:13:14.119 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:14.113+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:14.116924+0000) 2019-10-13T15:13:15.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:15.692+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:15.698868+0000) 2019-10-13T15:13:15.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:15.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:15.710582+0000) 2019-10-13T15:13:16.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:16.640+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.646950+0000) 2019-10-13T15:13:16.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:16.645+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.647468+0000) 2019-10-13T15:13:16.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:16.644+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.648182+0000) 2019-10-13T15:13:16.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:16.644+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.648559+0000) 2019-10-13T15:13:16.652 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:16.644+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.649788+0000) 2019-10-13T15:13:16.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:16.644+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.650152+0000) 2019-10-13T15:13:16.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:16.644+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.650700+0000) 2019-10-13T15:13:16.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:16.653+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:16.655770+0000) 2019-10-13T15:13:16.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:16.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:18.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:18.948+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.948859+0000) 2019-10-13T15:13:21.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:21.657+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.661808+0000) 2019-10-13T15:13:21.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:21.656+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.662590+0000) 2019-10-13T15:13:21.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:21.660+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.663350+0000) 2019-10-13T15:13:21.666 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:21.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.663862+0000) 2019-10-13T15:13:21.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:21.660+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.664024+0000) 2019-10-13T15:13:21.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:21.660+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.664943+0000) 2019-10-13T15:13:21.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:21.660+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.665346+0000) 2019-10-13T15:13:21.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:21.665+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.668761+0000) 2019-10-13T15:13:21.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:21.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:22.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:22.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:22.688612+0000) 2019-10-13T15:13:22.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:22.744+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:22.750470+0000) 2019-10-13T15:13:23.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:23.420+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.425666+0000) 2019-10-13T15:13:23.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:23.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.675215+0000) 2019-10-13T15:13:24.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:24.121+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:24.123546+0000) 2019-10-13T15:13:25.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:25.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:25.699094+0000) 2019-10-13T15:13:25.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:25.704+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:25.710768+0000) 2019-10-13T15:13:26.678 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:26.672+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.676074+0000) 2019-10-13T15:13:26.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:26.673+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.677203+0000) 2019-10-13T15:13:26.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:26.672+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.677437+0000) 2019-10-13T15:13:26.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:26.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.677718+0000) 2019-10-13T15:13:26.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:26.672+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.678318+0000) 2019-10-13T15:13:26.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:26.672+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.677840+0000) 2019-10-13T15:13:26.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:26.672+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.678905+0000) 2019-10-13T15:13:26.688 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:26.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:26.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:26.685+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:26.688021+0000) 2019-10-13T15:13:28.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:28.948+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.949062+0000) 2019-10-13T15:13:31.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:31.684+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.689344+0000) 2019-10-13T15:13:31.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:31.685+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.689934+0000) 2019-10-13T15:13:31.693 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:31.684+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.690397+0000) 2019-10-13T15:13:31.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:31.684+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.690693+0000) 2019-10-13T15:13:31.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:31.688+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.692406+0000) 2019-10-13T15:13:31.695 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:31.688+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.692457+0000) 2019-10-13T15:13:31.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:31.688+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.692716+0000) 2019-10-13T15:13:31.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:31.697+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.698425+0000) 2019-10-13T15:13:31.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:31.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214933 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:32.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:32.684+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:32.688815+0000) 2019-10-13T15:13:32.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:32.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:32.750677+0000) 2019-10-13T15:13:33.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:33.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.425855+0000) 2019-10-13T15:13:33.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:33.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.675423+0000) 2019-10-13T15:13:34.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:34.129+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:34.130104+0000) 2019-10-13T15:13:35.701 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:35.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:35.699293+0000) 2019-10-13T15:13:35.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:35.703+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:35.710969+0000) 2019-10-13T15:13:36.708 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:36.699+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.705423+0000) 2019-10-13T15:13:36.708 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:36.699+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.706375+0000) 2019-10-13T15:13:36.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:36.705+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.707201+0000) 2019-10-13T15:13:36.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:36.703+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.707496+0000) 2019-10-13T15:13:36.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:36.703+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.707807+0000) 2019-10-13T15:13:36.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:36.703+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.708424+0000) 2019-10-13T15:13:36.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:36.703+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.708691+0000) 2019-10-13T15:13:36.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:36.709+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:36.712991+0000) 2019-10-13T15:13:36.719 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:36.711+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 214971 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:37.143 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:13:37.144 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:37.151 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:13:37.151 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:38.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:38.947+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.949278+0000) 2019-10-13T15:13:41.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:41.715+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.720516+0000) 2019-10-13T15:13:41.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:41.715+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.720848+0000) 2019-10-13T15:13:41.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:41.717+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.721828+0000) 2019-10-13T15:13:41.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:41.715+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.721416+0000) 2019-10-13T15:13:41.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:41.725+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.729656+0000) 2019-10-13T15:13:41.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:41.715+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.722201+0000) 2019-10-13T15:13:41.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:41.715+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.722578+0000) 2019-10-13T15:13:41.741 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:41.719+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.723362+0000) 2019-10-13T15:13:41.742 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:41.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:42.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:42.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:42.688984+0000) 2019-10-13T15:13:42.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:42.743+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:42.750880+0000) 2019-10-13T15:13:43.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:43.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.426049+0000) 2019-10-13T15:13:43.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:43.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.675619+0000) 2019-10-13T15:13:44.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:44.133+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:44.136000+0000) 2019-10-13T15:13:45.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:45.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:45.699547+0000) 2019-10-13T15:13:45.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:45.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:45.711174+0000) 2019-10-13T15:13:46.737 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:46.731+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.734910+0000) 2019-10-13T15:13:46.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:46.733+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.735314+0000) 2019-10-13T15:13:46.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:46.731+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.735518+0000) 2019-10-13T15:13:46.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:46.731+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.735934+0000) 2019-10-13T15:13:46.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:46.731+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.736312+0000) 2019-10-13T15:13:46.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:46.731+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.736745+0000) 2019-10-13T15:13:46.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:46.731+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.738032+0000) 2019-10-13T15:13:46.748 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:46.739+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215041 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:46.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:46.745+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:46.746514+0000) 2019-10-13T15:13:48.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:48.947+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.949479+0000) 2019-10-13T15:13:51.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:51.743+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.749008+0000) 2019-10-13T15:13:51.752 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:51.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.749871+0000) 2019-10-13T15:13:51.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:51.743+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.749847+0000) 2019-10-13T15:13:51.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:51.743+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.750746+0000) 2019-10-13T15:13:51.753 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:51.743+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.751112+0000) 2019-10-13T15:13:51.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:51.749+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.752391+0000) 2019-10-13T15:13:51.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:51.747+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.752652+0000) 2019-10-13T15:13:51.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:51.757+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.759641+0000) 2019-10-13T15:13:51.763 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:51.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:52.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:52.683+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:52.689158+0000) 2019-10-13T15:13:52.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:52.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:52.751081+0000) 2019-10-13T15:13:53.428 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:53.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.426232+0000) 2019-10-13T15:13:53.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:53.673+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.675841+0000) 2019-10-13T15:13:54.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:54.141+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:54.142334+0000) 2019-10-13T15:13:55.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:55.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:55.699728+0000) 2019-10-13T15:13:55.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:55.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:55.711372+0000) 2019-10-13T15:13:56.767 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:13:56.760+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.765080+0000) 2019-10-13T15:13:56.768 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:13:56.759+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.766242+0000) 2019-10-13T15:13:56.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:13:56.759+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.766831+0000) 2019-10-13T15:13:56.769 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:13:56.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.767276+0000) 2019-10-13T15:13:56.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:13:56.763+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.767494+0000) 2019-10-13T15:13:56.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:13:56.763+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.767922+0000) 2019-10-13T15:13:56.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:56.763+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.768305+0000) 2019-10-13T15:13:56.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:13:56.768+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:56.771661+0000) 2019-10-13T15:13:56.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:13:56.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:13:58.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:13:58.947+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.949677+0000) 2019-10-13T15:14:01.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:01.780+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.783293+0000) 2019-10-13T15:14:01.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:01.788+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.790495+0000) 2019-10-13T15:14:01.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:01.775+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.780515+0000) 2019-10-13T15:14:01.816 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:01.775+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.781145+0000) 2019-10-13T15:14:01.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:01.775+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.781448+0000) 2019-10-13T15:14:01.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:01.775+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.782815+0000) 2019-10-13T15:14:01.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:01.775+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.782826+0000) 2019-10-13T15:14:01.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:01.779+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.783663+0000) 2019-10-13T15:14:01.828 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:01.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:02.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:02.687+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:02.689360+0000) 2019-10-13T15:14:02.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:02.747+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:02.751295+0000) 2019-10-13T15:14:03.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:03.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.426430+0000) 2019-10-13T15:14:03.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:03.672+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.676033+0000) 2019-10-13T15:14:04.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:04.152+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:04.154179+0000) 2019-10-13T15:14:05.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:05.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:05.699926+0000) 2019-10-13T15:14:05.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:05.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:05.711528+0000) 2019-10-13T15:14:06.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:06.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.794572+0000) 2019-10-13T15:14:06.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:06.791+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.795239+0000) 2019-10-13T15:14:06.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:06.795+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.795978+0000) 2019-10-13T15:14:06.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:06.795+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.796794+0000) 2019-10-13T15:14:06.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:06.795+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.797369+0000) 2019-10-13T15:14:06.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:06.795+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.796379+0000) 2019-10-13T15:14:06.800 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:06.792+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.797808+0000) 2019-10-13T15:14:06.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:06.800+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:06.804463+0000) 2019-10-13T15:14:06.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:06.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215179 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:07.272 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:14:07.272 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:07.277 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:14:07.277 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:08.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:08.947+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.949874+0000) 2019-10-13T15:14:11.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:11.807+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.808428+0000) 2019-10-13T15:14:11.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:11.804+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.809347+0000) 2019-10-13T15:14:11.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:11.807+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.809499+0000) 2019-10-13T15:14:11.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:11.807+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.810155+0000) 2019-10-13T15:14:11.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:11.807+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.810864+0000) 2019-10-13T15:14:11.813 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:11.807+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.811214+0000) 2019-10-13T15:14:11.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:11.811+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.811852+0000) 2019-10-13T15:14:11.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:11.812+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.816859+0000) 2019-10-13T15:14:11.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:11.819+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:12.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:12.687+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:12.689573+0000) 2019-10-13T15:14:12.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:12.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:12.751561+0000) 2019-10-13T15:14:13.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:13.423+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.426648+0000) 2019-10-13T15:14:13.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:13.672+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.676233+0000) 2019-10-13T15:14:14.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:14.164+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:14.165862+0000) 2019-10-13T15:14:15.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:15.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:15.700106+0000) 2019-10-13T15:14:15.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:15.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:15.711755+0000) 2019-10-13T15:14:16.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:16.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.823145+0000) 2019-10-13T15:14:16.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:16.822+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.823612+0000) 2019-10-13T15:14:16.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:16.822+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.824115+0000) 2019-10-13T15:14:16.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:16.820+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.824922+0000) 2019-10-13T15:14:16.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:16.822+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.825056+0000) 2019-10-13T15:14:16.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:16.822+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.825440+0000) 2019-10-13T15:14:16.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:16.822+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.826430+0000) 2019-10-13T15:14:16.834 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:16.828+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:16.832353+0000) 2019-10-13T15:14:16.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:16.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215247 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:18.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:18.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:18.950091+0000) 2019-10-13T15:14:21.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:21.834+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.837321+0000) 2019-10-13T15:14:21.840 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:21.834+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.837737+0000) 2019-10-13T15:14:21.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:21.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.838303+0000) 2019-10-13T15:14:21.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:21.834+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.839043+0000) 2019-10-13T15:14:21.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:21.838+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.839676+0000) 2019-10-13T15:14:21.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:21.836+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.840405+0000) 2019-10-13T15:14:21.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:21.838+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.841268+0000) 2019-10-13T15:14:21.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:21.844+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.846918+0000) 2019-10-13T15:14:21.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:21.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:22.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:22.686+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:22.689746+0000) 2019-10-13T15:14:22.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:22.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:22.751741+0000) 2019-10-13T15:14:23.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:23.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:23.426833+0000) 2019-10-13T15:14:23.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:23.672+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:23.676397+0000) 2019-10-13T15:14:24.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:24.168+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.172200+0000) 2019-10-13T15:14:25.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:25.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:25.700328+0000) 2019-10-13T15:14:25.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:25.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:25.711950+0000) 2019-10-13T15:14:26.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:26.850+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.851393+0000) 2019-10-13T15:14:26.855 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:26.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.852306+0000) 2019-10-13T15:14:26.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:26.850+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.852710+0000) 2019-10-13T15:14:26.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:26.850+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.853312+0000) 2019-10-13T15:14:26.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:26.848+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.853529+0000) 2019-10-13T15:14:26.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:26.850+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.853729+0000) 2019-10-13T15:14:26.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:26.850+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.854251+0000) 2019-10-13T15:14:26.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:26.856+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:26.860519+0000) 2019-10-13T15:14:26.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:26.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:28.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:28.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:28.950295+0000) 2019-10-13T15:14:31.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:31.862+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.866149+0000) 2019-10-13T15:14:31.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:31.862+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.866654+0000) 2019-10-13T15:14:31.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:31.862+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.866773+0000) 2019-10-13T15:14:31.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:31.866+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.867621+0000) 2019-10-13T15:14:31.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:31.866+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.867994+0000) 2019-10-13T15:14:31.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:31.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.868413+0000) 2019-10-13T15:14:31.871 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:31.864+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.869522+0000) 2019-10-13T15:14:31.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:31.872+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.876182+0000) 2019-10-13T15:14:31.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:31.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215346 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:32.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:32.686+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:32.689925+0000) 2019-10-13T15:14:32.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:32.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:32.751952+0000) 2019-10-13T15:14:33.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:33.422+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:33.427032+0000) 2019-10-13T15:14:33.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:33.672+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:33.676594+0000) 2019-10-13T15:14:34.183 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:34.176+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.181076+0000) 2019-10-13T15:14:35.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:35.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:35.700532+0000) 2019-10-13T15:14:35.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:35.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:35.712151+0000) 2019-10-13T15:14:36.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:36.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.880248+0000) 2019-10-13T15:14:36.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:36.878+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.880491+0000) 2019-10-13T15:14:36.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:36.878+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.881087+0000) 2019-10-13T15:14:36.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:36.878+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.881266+0000) 2019-10-13T15:14:36.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:36.878+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.882538+0000) 2019-10-13T15:14:36.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:36.880+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.883614+0000) 2019-10-13T15:14:36.886 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:36.882+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.883732+0000) 2019-10-13T15:14:36.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:36.884+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:36.888893+0000) 2019-10-13T15:14:36.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:36.886+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215385 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:37.399 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:14:37.400 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:37.403 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:14:37.403 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:38.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:38.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:38.950531+0000) 2019-10-13T15:14:41.896 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:41.890+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.893980+0000) 2019-10-13T15:14:41.897 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:41.890+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.894752+0000) 2019-10-13T15:14:41.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:41.890+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.895162+0000) 2019-10-13T15:14:41.898 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:41.894+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.895862+0000) 2019-10-13T15:14:41.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:41.892+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.896568+0000) 2019-10-13T15:14:41.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:41.894+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.896704+0000) 2019-10-13T15:14:41.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:41.894+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.897365+0000) 2019-10-13T15:14:41.907 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:41.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:41.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:41.904+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.907376+0000) 2019-10-13T15:14:42.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:42.686+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:42.690093+0000) 2019-10-13T15:14:42.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:42.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:42.752134+0000) 2019-10-13T15:14:43.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:43.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:43.427225+0000) 2019-10-13T15:14:43.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:43.672+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:43.676796+0000) 2019-10-13T15:14:44.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:44.184+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.187483+0000) 2019-10-13T15:14:45.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:45.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:45.700684+0000) 2019-10-13T15:14:45.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:45.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:45.712343+0000) 2019-10-13T15:14:46.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:46.906+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.907806+0000) 2019-10-13T15:14:46.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:46.906+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.908401+0000) 2019-10-13T15:14:46.912 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:46.904+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.909769+0000) 2019-10-13T15:14:46.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:46.906+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.909909+0000) 2019-10-13T15:14:46.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:46.906+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.910513+0000) 2019-10-13T15:14:46.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:46.906+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.910728+0000) 2019-10-13T15:14:46.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:46.906+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.911178+0000) 2019-10-13T15:14:46.922 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:46.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:46.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:46.916+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:46.918371+0000) 2019-10-13T15:14:48.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:48.950+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:48.950750+0000) 2019-10-13T15:14:51.925 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:51.918+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.922320+0000) 2019-10-13T15:14:51.925 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:51.918+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.922849+0000) 2019-10-13T15:14:51.926 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:51.922+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.923420+0000) 2019-10-13T15:14:51.927 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:51.919+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.924432+0000) 2019-10-13T15:14:51.927 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:51.922+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.924093+0000) 2019-10-13T15:14:51.927 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:51.922+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.924625+0000) 2019-10-13T15:14:51.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:51.922+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.925107+0000) 2019-10-13T15:14:51.934 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:51.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.932416+0000) 2019-10-13T15:14:51.935 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:51.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215484 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:52.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:52.686+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:52.690267+0000) 2019-10-13T15:14:52.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:52.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:52.752313+0000) 2019-10-13T15:14:53.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:53.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:53.427413+0000) 2019-10-13T15:14:53.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:53.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:53.677013+0000) 2019-10-13T15:14:54.196 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:54.187+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.193845+0000) 2019-10-13T15:14:55.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:55.698+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:55.700893+0000) 2019-10-13T15:14:55.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:55.710+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:55.712576+0000) 2019-10-13T15:14:56.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:14:56.934+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.935977+0000) 2019-10-13T15:14:56.939 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:14:56.934+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.936668+0000) 2019-10-13T15:14:56.940 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:56.934+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.938083+0000) 2019-10-13T15:14:56.941 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:14:56.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.938681+0000) 2019-10-13T15:14:56.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:14:56.938+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.939417+0000) 2019-10-13T15:14:56.942 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:14:56.935+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.939779+0000) 2019-10-13T15:14:56.942 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:14:56.938+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.940074+0000) 2019-10-13T15:14:56.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:14:56.943+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:56.947077+0000) 2019-10-13T15:14:56.949 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:14:56.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215523 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:14:58.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:14:58.946+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:58.950974+0000) 2019-10-13T15:15:01.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:01.945+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.950182+0000) 2019-10-13T15:15:01.953 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:01.945+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.950614+0000) 2019-10-13T15:15:01.954 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:01.949+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.951435+0000) 2019-10-13T15:15:01.954 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:01.949+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.952016+0000) 2019-10-13T15:15:01.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:01.949+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.952977+0000) 2019-10-13T15:15:01.956 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:01.949+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.953450+0000) 2019-10-13T15:15:01.956 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:01.947+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.954027+0000) 2019-10-13T15:15:01.961 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:01.955+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.959396+0000) 2019-10-13T15:15:01.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:01.957+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:02.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:02.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:02.690441+0000) 2019-10-13T15:15:02.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:02.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:02.752497+0000) 2019-10-13T15:15:03.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:03.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:03.427598+0000) 2019-10-13T15:15:03.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:03.671+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:03.677224+0000) 2019-10-13T15:15:04.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:04.199+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.200638+0000) 2019-10-13T15:15:05.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:05.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:05.701097+0000) 2019-10-13T15:15:05.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:05.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:05.712777+0000) 2019-10-13T15:15:06.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:06.961+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.964816+0000) 2019-10-13T15:15:06.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:06.961+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.965233+0000) 2019-10-13T15:15:06.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:06.961+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.965721+0000) 2019-10-13T15:15:06.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:06.961+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.966220+0000) 2019-10-13T15:15:06.969 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:06.961+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.966476+0000) 2019-10-13T15:15:06.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:06.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.967048+0000) 2019-10-13T15:15:06.970 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:06.963+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.967731+0000) 2019-10-13T15:15:06.977 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:06.971+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:06.975317+0000) 2019-10-13T15:15:06.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:06.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:07.444 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:15:07.444 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:07.448 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:15:07.448 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:08.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:08.945+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:08.951177+0000) 2019-10-13T15:15:11.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:11.973+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.979076+0000) 2019-10-13T15:15:11.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:11.975+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.979663+0000) 2019-10-13T15:15:11.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:11.977+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.980023+0000) 2019-10-13T15:15:11.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:11.977+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.980590+0000) 2019-10-13T15:15:11.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:11.977+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.980984+0000) 2019-10-13T15:15:11.985 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:11.977+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.982054+0000) 2019-10-13T15:15:11.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:11.977+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.982864+0000) 2019-10-13T15:15:11.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:11.983+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.990002+0000) 2019-10-13T15:15:11.993 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:11.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:12.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:12.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:12.690624+0000) 2019-10-13T15:15:12.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:12.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:12.752681+0000) 2019-10-13T15:15:13.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:13.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:13.427788+0000) 2019-10-13T15:15:13.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:13.671+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:13.677431+0000) 2019-10-13T15:15:14.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:14.203+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.207479+0000) 2019-10-13T15:15:15.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:15.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:15.701340+0000) 2019-10-13T15:15:15.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:15.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:15.712966+0000) 2019-10-13T15:15:16.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:16.989+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.992478+0000) 2019-10-13T15:15:16.995 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:16.989+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.993410+0000) 2019-10-13T15:15:16.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:16.989+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.993854+0000) 2019-10-13T15:15:16.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:16.991+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.994592+0000) 2019-10-13T15:15:16.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:16.989+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.994494+0000) 2019-10-13T15:15:16.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:16.989+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.994952+0000) 2019-10-13T15:15:16.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:16.993+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:16.995345+0000) 2019-10-13T15:15:17.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:16.995+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:17.000774+0000) 2019-10-13T15:15:17.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:17.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:18.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:18.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:18.951409+0000) 2019-10-13T15:15:22.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:22.005+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.007381+0000) 2019-10-13T15:15:22.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:22.005+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.007835+0000) 2019-10-13T15:15:22.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:22.005+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.007928+0000) 2019-10-13T15:15:22.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:22.005+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.008614+0000) 2019-10-13T15:15:22.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:22.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.009213+0000) 2019-10-13T15:15:22.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:22.003+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.009619+0000) 2019-10-13T15:15:22.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:22.005+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.009923+0000) 2019-10-13T15:15:22.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:22.007+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.013635+0000) 2019-10-13T15:15:22.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:22.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:22.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:22.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.690842+0000) 2019-10-13T15:15:22.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:22.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.752890+0000) 2019-10-13T15:15:23.430 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:23.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:23.427955+0000) 2019-10-13T15:15:23.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:23.671+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:23.677621+0000) 2019-10-13T15:15:24.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:24.207+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.213835+0000) 2019-10-13T15:15:25.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:25.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:25.701566+0000) 2019-10-13T15:15:25.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:25.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:25.713205+0000) 2019-10-13T15:15:27.023 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:27.017+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.021117+0000) 2019-10-13T15:15:27.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:27.017+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.022114+0000) 2019-10-13T15:15:27.025 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:27.017+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.022545+0000) 2019-10-13T15:15:27.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:27.017+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.022755+0000) 2019-10-13T15:15:27.031 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:27.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.024409+0000) 2019-10-13T15:15:27.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:27.021+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.024597+0000) 2019-10-13T15:15:27.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:27.019+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.024097+0000) 2019-10-13T15:15:27.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:27.023+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:27.029519+0000) 2019-10-13T15:15:27.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:27.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215731 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:28.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:28.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:28.951644+0000) 2019-10-13T15:15:32.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:32.033+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.035390+0000) 2019-10-13T15:15:32.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:32.033+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.035816+0000) 2019-10-13T15:15:32.039 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:32.031+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.036669+0000) 2019-10-13T15:15:32.039 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:32.033+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.037332+0000) 2019-10-13T15:15:32.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:32.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.037424+0000) 2019-10-13T15:15:32.040 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:32.033+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.037768+0000) 2019-10-13T15:15:32.041 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:32.033+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.038902+0000) 2019-10-13T15:15:32.047 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:32.039+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.045580+0000) 2019-10-13T15:15:32.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:32.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:32.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:32.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.691036+0000) 2019-10-13T15:15:32.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:32.749+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.753075+0000) 2019-10-13T15:15:33.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:33.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:33.428174+0000) 2019-10-13T15:15:33.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:33.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:33.677813+0000) 2019-10-13T15:15:34.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:34.219+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.220142+0000) 2019-10-13T15:15:35.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:35.697+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:35.701767+0000) 2019-10-13T15:15:35.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:35.709+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:35.713428+0000) 2019-10-13T15:15:37.052 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:37.045+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.049367+0000) 2019-10-13T15:15:37.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:37.045+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.050734+0000) 2019-10-13T15:15:37.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:37.047+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.051338+0000) 2019-10-13T15:15:37.055 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:37.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.052322+0000) 2019-10-13T15:15:37.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:37.049+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.052063+0000) 2019-10-13T15:15:37.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:37.049+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.053053+0000) 2019-10-13T15:15:37.056 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:37.049+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.053355+0000) 2019-10-13T15:15:37.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:37.055+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:37.058460+0000) 2019-10-13T15:15:37.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:37.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215799 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:37.547 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:15:37.547 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:37.551 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:15:37.551 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:38.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:38.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:38.951845+0000) 2019-10-13T15:15:42.068 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:42.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.064732+0000) 2019-10-13T15:15:42.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:42.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.064923+0000) 2019-10-13T15:15:42.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:42.061+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.065594+0000) 2019-10-13T15:15:42.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:42.061+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.065903+0000) 2019-10-13T15:15:42.069 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:42.061+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.067097+0000) 2019-10-13T15:15:42.070 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:42.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.067464+0000) 2019-10-13T15:15:42.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:42.063+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.067912+0000) 2019-10-13T15:15:42.077 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:42.071+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.074678+0000) 2019-10-13T15:15:42.077 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:42.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215829 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:42.693 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:42.685+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.691237+0000) 2019-10-13T15:15:42.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:42.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.753290+0000) 2019-10-13T15:15:43.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:43.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:43.428368+0000) 2019-10-13T15:15:43.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:43.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:43.677962+0000) 2019-10-13T15:15:44.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:44.222+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.226775+0000) 2019-10-13T15:15:45.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:45.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:45.701968+0000) 2019-10-13T15:15:45.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:45.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:45.713658+0000) 2019-10-13T15:15:47.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:47.072+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.079012+0000) 2019-10-13T15:15:47.082 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:47.076+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.079471+0000) 2019-10-13T15:15:47.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:47.074+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.080623+0000) 2019-10-13T15:15:47.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:47.076+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.080659+0000) 2019-10-13T15:15:47.083 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:47.076+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.081215+0000) 2019-10-13T15:15:47.084 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:47.076+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.082004+0000) 2019-10-13T15:15:47.084 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:47.076+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.082382+0000) 2019-10-13T15:15:47.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:47.082+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:47.086700+0000) 2019-10-13T15:15:47.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:47.084+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215867 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:48.954 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:48.948+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:48.952048+0000) 2019-10-13T15:15:52.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:52.088+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.095105+0000) 2019-10-13T15:15:52.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:52.088+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.095257+0000) 2019-10-13T15:15:52.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:52.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.095609+0000) 2019-10-13T15:15:52.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:52.090+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.097418+0000) 2019-10-13T15:15:52.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:52.092+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.096099+0000) 2019-10-13T15:15:52.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:52.092+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.096845+0000) 2019-10-13T15:15:52.113 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:52.092+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.097609+0000) 2019-10-13T15:15:52.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:52.098+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.104509+0000) 2019-10-13T15:15:52.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:52.100+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:52.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:52.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.691438+0000) 2019-10-13T15:15:52.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:52.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.753471+0000) 2019-10-13T15:15:53.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:53.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:53.428565+0000) 2019-10-13T15:15:53.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:53.674+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:53.678193+0000) 2019-10-13T15:15:54.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:54.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.233130+0000) 2019-10-13T15:15:55.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:55.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:55.702163+0000) 2019-10-13T15:15:55.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:55.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:55.713853+0000) 2019-10-13T15:15:57.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:15:57.104+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.109124+0000) 2019-10-13T15:15:57.113 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:15:57.104+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.110794+0000) 2019-10-13T15:15:57.113 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:15:57.106+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.111305+0000) 2019-10-13T15:15:57.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:15:57.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.111483+0000) 2019-10-13T15:15:57.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:57.108+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.112436+0000) 2019-10-13T15:15:57.115 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:15:57.108+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.112839+0000) 2019-10-13T15:15:57.115 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:15:57.108+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.113455+0000) 2019-10-13T15:15:57.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:15:57.110+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:57.116160+0000) 2019-10-13T15:15:57.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:15:57.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:15:58.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:15:58.948+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:58.952241+0000) 2019-10-13T15:16:02.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:02.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.123990+0000) 2019-10-13T15:16:02.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:02.120+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.124473+0000) 2019-10-13T15:16:02.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:02.120+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.125271+0000) 2019-10-13T15:16:02.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:02.120+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.125538+0000) 2019-10-13T15:16:02.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:02.122+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.126888+0000) 2019-10-13T15:16:02.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:02.120+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.127244+0000) 2019-10-13T15:16:02.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:02.124+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.127785+0000) 2019-10-13T15:16:02.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:02.126+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.132802+0000) 2019-10-13T15:16:02.137 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:02.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 215965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:02.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:02.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.691573+0000) 2019-10-13T15:16:02.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:02.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.753650+0000) 2019-10-13T15:16:03.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:03.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:03.428789+0000) 2019-10-13T15:16:03.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:03.674+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:03.678385+0000) 2019-10-13T15:16:04.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:04.238+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.239754+0000) 2019-10-13T15:16:05.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:05.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:05.702359+0000) 2019-10-13T15:16:05.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:05.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:05.714067+0000) 2019-10-13T15:16:07.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:07.132+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.138166+0000) 2019-10-13T15:16:07.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:07.132+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.138650+0000) 2019-10-13T15:16:07.142 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:07.136+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.139352+0000) 2019-10-13T15:16:07.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:07.136+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.139621+0000) 2019-10-13T15:16:07.143 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:07.136+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.140667+0000) 2019-10-13T15:16:07.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:07.134+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.141516+0000) 2019-10-13T15:16:07.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:07.136+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.141586+0000) 2019-10-13T15:16:07.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:07.142+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:07.147339+0000) 2019-10-13T15:16:07.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:07.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216001 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:07.600 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:16:07.600 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:07.604 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:16:07.604 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:08.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:08.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:08.952441+0000) 2019-10-13T15:16:12.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:12.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.153447+0000) 2019-10-13T15:16:12.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:12.148+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.153866+0000) 2019-10-13T15:16:12.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:12.148+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.154183+0000) 2019-10-13T15:16:12.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:12.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.154972+0000) 2019-10-13T15:16:12.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:12.150+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.155535+0000) 2019-10-13T15:16:12.158 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:12.152+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.155671+0000) 2019-10-13T15:16:12.158 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:12.152+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.156394+0000) 2019-10-13T15:16:12.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:12.158+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.163178+0000) 2019-10-13T15:16:12.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:12.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:12.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:12.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.691718+0000) 2019-10-13T15:16:12.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:12.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.753787+0000) 2019-10-13T15:16:13.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:13.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:13.428986+0000) 2019-10-13T15:16:13.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:13.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:13.678576+0000) 2019-10-13T15:16:14.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:14.250+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.250869+0000) 2019-10-13T15:16:15.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:15.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:15.702557+0000) 2019-10-13T15:16:15.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:15.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:15.714264+0000) 2019-10-13T15:16:17.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:17.164+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.168891+0000) 2019-10-13T15:16:17.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:17.164+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.169912+0000) 2019-10-13T15:16:17.173 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:17.164+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.170363+0000) 2019-10-13T15:16:17.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:17.164+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.170857+0000) 2019-10-13T15:16:17.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:17.166+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.171799+0000) 2019-10-13T15:16:17.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:17.164+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.171228+0000) 2019-10-13T15:16:17.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:17.168+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.171936+0000) 2019-10-13T15:16:17.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:17.170+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:17.176054+0000) 2019-10-13T15:16:17.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:17.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:18.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:18.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:18.952641+0000) 2019-10-13T15:16:22.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:22.180+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.184547+0000) 2019-10-13T15:16:22.188 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:22.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.184831+0000) 2019-10-13T15:16:22.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:22.180+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.185413+0000) 2019-10-13T15:16:22.190 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:22.180+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.185996+0000) 2019-10-13T15:16:22.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:22.180+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.186016+0000) 2019-10-13T15:16:22.190 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:22.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.187250+0000) 2019-10-13T15:16:22.191 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:22.182+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.187686+0000) 2019-10-13T15:16:22.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:22.186+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.192906+0000) 2019-10-13T15:16:22.197 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:22.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:22.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:22.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.691856+0000) 2019-10-13T15:16:22.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:22.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.753986+0000) 2019-10-13T15:16:23.431 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:23.424+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:23.429176+0000) 2019-10-13T15:16:23.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:23.674+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:23.678782+0000) 2019-10-13T15:16:24.259 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:24.250+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.257253+0000) 2019-10-13T15:16:25.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:25.696+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:25.702762+0000) 2019-10-13T15:16:25.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:25.708+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:25.714492+0000) 2019-10-13T15:16:27.200 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:27.191+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.198076+0000) 2019-10-13T15:16:27.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:27.191+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.198536+0000) 2019-10-13T15:16:27.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:27.195+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.199500+0000) 2019-10-13T15:16:27.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:27.195+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.199772+0000) 2019-10-13T15:16:27.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:27.195+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.200520+0000) 2019-10-13T15:16:27.203 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:27.195+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.200979+0000) 2019-10-13T15:16:27.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:27.198+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.202234+0000) 2019-10-13T15:16:27.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:27.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:27.213 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:27.206+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:27.211537+0000) 2019-10-13T15:16:28.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:28.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:28.952843+0000) 2019-10-13T15:16:32.216 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:32.210+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.214118+0000) 2019-10-13T15:16:32.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:32.207+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.214425+0000) 2019-10-13T15:16:32.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:32.207+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.214970+0000) 2019-10-13T15:16:32.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:32.211+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.215912+0000) 2019-10-13T15:16:32.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:32.211+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.216078+0000) 2019-10-13T15:16:32.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:32.211+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.216700+0000) 2019-10-13T15:16:32.219 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:32.211+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.217041+0000) 2019-10-13T15:16:32.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:32.218+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.221736+0000) 2019-10-13T15:16:32.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:32.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:32.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:32.687+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.692115+0000) 2019-10-13T15:16:32.756 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:32.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.754169+0000) 2019-10-13T15:16:33.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:33.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:33.429370+0000) 2019-10-13T15:16:33.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:33.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:33.678993+0000) 2019-10-13T15:16:34.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:34.262+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.263704+0000) 2019-10-13T15:16:35.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:35.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:35.702993+0000) 2019-10-13T15:16:35.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:35.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:35.714693+0000) 2019-10-13T15:16:37.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:37.223+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.227538+0000) 2019-10-13T15:16:37.230 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:37.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.228093+0000) 2019-10-13T15:16:37.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:37.223+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.228519+0000) 2019-10-13T15:16:37.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:37.223+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.229275+0000) 2019-10-13T15:16:37.232 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:37.223+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.229569+0000) 2019-10-13T15:16:37.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:37.223+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.230256+0000) 2019-10-13T15:16:37.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:37.223+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.230907+0000) 2019-10-13T15:16:37.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:37.234+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:37.234694+0000) 2019-10-13T15:16:37.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:37.235+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:37.702 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:16:37.702 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:37.706 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:16:37.706 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:38.955 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:38.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:38.953061+0000) 2019-10-13T15:16:42.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:42.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.242666+0000) 2019-10-13T15:16:42.245 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:42.239+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.243016+0000) 2019-10-13T15:16:42.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:42.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.243484+0000) 2019-10-13T15:16:42.247 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:42.239+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.245344+0000) 2019-10-13T15:16:42.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:42.239+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.245839+0000) 2019-10-13T15:16:42.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:42.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.246349+0000) 2019-10-13T15:16:42.249 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:42.239+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.246905+0000) 2019-10-13T15:16:42.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:42.249+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.250999+0000) 2019-10-13T15:16:42.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:42.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216234 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:42.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:42.691+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.692292+0000) 2019-10-13T15:16:42.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:42.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.754368+0000) 2019-10-13T15:16:43.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:43.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:43.429561+0000) 2019-10-13T15:16:43.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:43.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:43.679187+0000) 2019-10-13T15:16:44.272 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:44.269+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.270334+0000) 2019-10-13T15:16:45.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:45.695+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:45.703190+0000) 2019-10-13T15:16:45.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:45.707+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:45.714889+0000) 2019-10-13T15:16:47.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:47.251+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.257076+0000) 2019-10-13T15:16:47.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:47.257+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.258479+0000) 2019-10-13T15:16:47.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:47.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.258666+0000) 2019-10-13T15:16:47.261 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:47.255+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.259530+0000) 2019-10-13T15:16:47.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:47.255+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.260355+0000) 2019-10-13T15:16:47.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:47.255+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.260678+0000) 2019-10-13T15:16:47.264 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:47.255+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.261321+0000) 2019-10-13T15:16:47.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:47.261+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:47.264641+0000) 2019-10-13T15:16:47.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:47.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216268 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:48.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:48.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:48.953244+0000) 2019-10-13T15:16:52.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:52.267+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.271122+0000) 2019-10-13T15:16:52.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:52.267+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.271885+0000) 2019-10-13T15:16:52.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:52.267+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.272302+0000) 2019-10-13T15:16:52.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:52.267+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.272592+0000) 2019-10-13T15:16:52.276 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:52.273+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.274394+0000) 2019-10-13T15:16:52.277 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:52.267+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.274482+0000) 2019-10-13T15:16:52.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:52.267+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.275275+0000) 2019-10-13T15:16:52.283 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:52.277+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.281351+0000) 2019-10-13T15:16:52.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:52.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:52.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:52.687+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.692470+0000) 2019-10-13T15:16:52.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:52.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.754553+0000) 2019-10-13T15:16:53.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:53.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:53.429748+0000) 2019-10-13T15:16:53.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:53.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:53.679361+0000) 2019-10-13T15:16:54.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:54.273+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.277098+0000) 2019-10-13T15:16:55.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:55.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:55.703393+0000) 2019-10-13T15:16:55.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:55.711+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:55.715118+0000) 2019-10-13T15:16:57.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:16:57.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.285621+0000) 2019-10-13T15:16:57.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:16:57.283+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.285991+0000) 2019-10-13T15:16:57.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:16:57.283+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.286319+0000) 2019-10-13T15:16:57.289 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:16:57.285+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.286945+0000) 2019-10-13T15:16:57.290 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:57.283+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.287215+0000) 2019-10-13T15:16:57.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:16:57.287+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.287727+0000) 2019-10-13T15:16:57.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:16:57.287+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.289337+0000) 2019-10-13T15:16:57.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:16:57.293+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:57.294855+0000) 2019-10-13T15:16:57.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:16:57.295+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:16:58.956 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:16:58.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:58.953441+0000) 2019-10-13T15:17:02.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:02.299+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.299494+0000) 2019-10-13T15:17:02.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:02.299+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.299797+0000) 2019-10-13T15:17:02.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:02.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.300209+0000) 2019-10-13T15:17:02.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:02.297+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.301122+0000) 2019-10-13T15:17:02.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:02.299+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.300670+0000) 2019-10-13T15:17:02.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:02.299+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.301452+0000) 2019-10-13T15:17:02.304 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:02.299+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.301502+0000) 2019-10-13T15:17:02.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:02.305+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.309774+0000) 2019-10-13T15:17:02.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:02.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:02.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:02.691+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.692653+0000) 2019-10-13T15:17:02.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:02.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.754744+0000) 2019-10-13T15:17:03.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:03.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:03.429935+0000) 2019-10-13T15:17:03.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:03.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:03.679513+0000) 2019-10-13T15:17:04.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:04.281+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.283421+0000) 2019-10-13T15:17:05.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:05.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:05.703596+0000) 2019-10-13T15:17:05.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:05.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:05.715342+0000) 2019-10-13T15:17:07.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:07.311+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.314114+0000) 2019-10-13T15:17:07.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:07.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.314766+0000) 2019-10-13T15:17:07.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:07.311+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.315011+0000) 2019-10-13T15:17:07.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:07.315+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.315465+0000) 2019-10-13T15:17:07.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:07.315+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.315853+0000) 2019-10-13T15:17:07.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:07.315+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.316269+0000) 2019-10-13T15:17:07.319 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:07.313+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.317028+0000) 2019-10-13T15:17:07.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:07.317+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:07.322032+0000) 2019-10-13T15:17:07.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:07.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216402 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:07.781 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:17:07.782 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:07.785 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:17:07.786 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:08.956 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:08.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:08.953675+0000) 2019-10-13T15:17:12.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:12.326+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.328196+0000) 2019-10-13T15:17:12.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:12.325+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.328751+0000) 2019-10-13T15:17:12.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:12.326+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.329623+0000) 2019-10-13T15:17:12.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:12.326+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.330335+0000) 2019-10-13T15:17:12.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:12.326+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.330474+0000) 2019-10-13T15:17:12.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:12.326+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.330811+0000) 2019-10-13T15:17:12.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:12.326+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.331225+0000) 2019-10-13T15:17:12.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:12.333+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.334402+0000) 2019-10-13T15:17:12.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:12.334+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:12.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:12.690+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.692839+0000) 2019-10-13T15:17:12.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:12.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.754966+0000) 2019-10-13T15:17:13.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:13.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:13.430128+0000) 2019-10-13T15:17:13.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:13.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:13.679712+0000) 2019-10-13T15:17:14.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:14.289+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.289912+0000) 2019-10-13T15:17:15.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:15.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:15.703827+0000) 2019-10-13T15:17:15.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:15.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:15.715534+0000) 2019-10-13T15:17:17.344 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:17.337+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.341682+0000) 2019-10-13T15:17:17.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:17.338+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.342248+0000) 2019-10-13T15:17:17.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:17.338+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.342753+0000) 2019-10-13T15:17:17.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:17.338+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.343223+0000) 2019-10-13T15:17:17.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:17.342+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.343692+0000) 2019-10-13T15:17:17.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:17.342+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.344869+0000) 2019-10-13T15:17:17.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:17.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.345352+0000) 2019-10-13T15:17:17.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:17.349+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:17.350324+0000) 2019-10-13T15:17:17.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:17.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216469 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:18.956 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:18.950+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:18.953875+0000) 2019-10-13T15:17:22.358 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:22.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.355192+0000) 2019-10-13T15:17:22.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:22.354+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.355890+0000) 2019-10-13T15:17:22.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:22.354+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.356311+0000) 2019-10-13T15:17:22.359 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:22.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.357305+0000) 2019-10-13T15:17:22.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:22.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.357567+0000) 2019-10-13T15:17:22.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:22.354+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.358353+0000) 2019-10-13T15:17:22.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:22.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.358908+0000) 2019-10-13T15:17:22.368 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:22.361+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.365521+0000) 2019-10-13T15:17:22.369 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:22.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:22.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:22.690+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.693022+0000) 2019-10-13T15:17:22.757 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:22.750+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.755174+0000) 2019-10-13T15:17:23.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:23.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:23.430327+0000) 2019-10-13T15:17:23.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:23.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:23.679913+0000) 2019-10-13T15:17:24.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:24.293+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.295633+0000) 2019-10-13T15:17:25.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:25.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:25.704017+0000) 2019-10-13T15:17:25.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:25.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:25.715742+0000) 2019-10-13T15:17:27.372 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:27.366+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.370010+0000) 2019-10-13T15:17:27.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:27.366+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.370273+0000) 2019-10-13T15:17:27.373 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:27.369+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.371232+0000) 2019-10-13T15:17:27.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:27.370+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.371915+0000) 2019-10-13T15:17:27.375 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:27.370+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.372025+0000) 2019-10-13T15:17:27.375 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:27.370+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.372368+0000) 2019-10-13T15:17:27.375 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:27.370+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.372853+0000) 2019-10-13T15:17:27.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:27.377+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:27.379728+0000) 2019-10-13T15:17:27.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:27.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216536 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:28.956 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:28.950+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:28.954072+0000) 2019-10-13T15:17:32.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:32.382+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.383665+0000) 2019-10-13T15:17:32.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:32.382+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.384548+0000) 2019-10-13T15:17:32.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:32.382+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.384789+0000) 2019-10-13T15:17:32.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:32.381+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.385463+0000) 2019-10-13T15:17:32.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:32.382+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.385569+0000) 2019-10-13T15:17:32.388 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:32.382+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.386468+0000) 2019-10-13T15:17:32.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:32.386+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.387802+0000) 2019-10-13T15:17:32.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:32.389+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.393380+0000) 2019-10-13T15:17:32.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:32.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:32.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:32.690+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.693198+0000) 2019-10-13T15:17:32.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:32.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.755355+0000) 2019-10-13T15:17:33.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:33.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:33.430544+0000) 2019-10-13T15:17:33.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:33.677+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:33.680102+0000) 2019-10-13T15:17:34.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:34.304+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.306773+0000) 2019-10-13T15:17:35.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:35.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:35.704176+0000) 2019-10-13T15:17:35.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:35.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:35.715898+0000) 2019-10-13T15:17:37.401 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:37.394+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.398531+0000) 2019-10-13T15:17:37.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:37.394+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.399066+0000) 2019-10-13T15:17:37.402 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:37.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.399470+0000) 2019-10-13T15:17:37.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:37.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.399898+0000) 2019-10-13T15:17:37.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:37.396+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.400550+0000) 2019-10-13T15:17:37.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:37.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.400746+0000) 2019-10-13T15:17:37.404 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:37.398+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.401630+0000) 2019-10-13T15:17:37.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:37.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216603 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:37.414 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:37.404+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:37.409675+0000) 2019-10-13T15:17:37.901 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:17:37.901 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:37.905 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:17:37.905 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:38.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:38.950+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:38.954270+0000) 2019-10-13T15:17:42.416 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:42.410+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.413213+0000) 2019-10-13T15:17:42.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:42.410+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.413898+0000) 2019-10-13T15:17:42.417 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:42.410+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.414141+0000) 2019-10-13T15:17:42.417 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:42.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.415276+0000) 2019-10-13T15:17:42.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:42.414+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.415661+0000) 2019-10-13T15:17:42.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:42.412+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.416208+0000) 2019-10-13T15:17:42.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:42.414+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.416292+0000) 2019-10-13T15:17:42.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:42.416+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.421530+0000) 2019-10-13T15:17:42.425 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:42.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216636 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:42.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:42.690+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.693386+0000) 2019-10-13T15:17:42.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:42.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.755540+0000) 2019-10-13T15:17:43.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:43.426+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:43.430735+0000) 2019-10-13T15:17:43.682 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:43.676+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:43.680342+0000) 2019-10-13T15:17:44.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:44.316+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.318624+0000) 2019-10-13T15:17:45.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:45.702+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:45.704334+0000) 2019-10-13T15:17:45.718 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:45.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:45.716105+0000) 2019-10-13T15:17:47.429 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:47.422+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.426330+0000) 2019-10-13T15:17:47.429 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:47.422+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.426789+0000) 2019-10-13T15:17:47.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:47.426+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.427401+0000) 2019-10-13T15:17:47.430 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:47.424+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.428060+0000) 2019-10-13T15:17:47.431 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:47.426+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.428744+0000) 2019-10-13T15:17:47.432 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:47.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.429581+0000) 2019-10-13T15:17:47.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:47.426+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.430822+0000) 2019-10-13T15:17:47.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:47.432+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:47.436272+0000) 2019-10-13T15:17:47.440 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:47.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:48.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:48.950+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:48.954476+0000) 2019-10-13T15:17:52.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:52.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.440971+0000) 2019-10-13T15:17:52.444 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:52.436+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.441744+0000) 2019-10-13T15:17:52.444 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:52.438+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.442075+0000) 2019-10-13T15:17:52.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:52.438+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.442689+0000) 2019-10-13T15:17:52.445 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:52.442+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.443442+0000) 2019-10-13T15:17:52.447 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:52.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.444170+0000) 2019-10-13T15:17:52.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:52.442+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.444761+0000) 2019-10-13T15:17:52.451 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:52.444+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.448910+0000) 2019-10-13T15:17:52.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:52.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:52.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:52.690+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.693562+0000) 2019-10-13T15:17:52.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:52.754+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.755722+0000) 2019-10-13T15:17:53.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:53.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:53.430955+0000) 2019-10-13T15:17:53.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:53.676+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:53.680488+0000) 2019-10-13T15:17:54.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:54.320+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.325074+0000) 2019-10-13T15:17:55.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:55.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:55.704565+0000) 2019-10-13T15:17:55.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:55.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:55.716324+0000) 2019-10-13T15:17:57.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:17:57.453+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.456736+0000) 2019-10-13T15:17:57.459 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:17:57.453+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.457100+0000) 2019-10-13T15:17:57.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:17:57.452+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.457672+0000) 2019-10-13T15:17:57.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:17:57.453+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.457997+0000) 2019-10-13T15:17:57.461 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:17:57.453+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.458550+0000) 2019-10-13T15:17:57.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:57.453+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.458887+0000) 2019-10-13T15:17:57.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:17:57.453+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.459028+0000) 2019-10-13T15:17:57.467 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:17:57.460+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:57.464928+0000) 2019-10-13T15:17:57.469 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:17:57.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:17:58.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:17:58.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:58.954673+0000) 2019-10-13T15:18:02.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:02.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.470834+0000) 2019-10-13T15:18:02.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:02.465+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.471116+0000) 2019-10-13T15:18:02.474 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:02.469+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.472080+0000) 2019-10-13T15:18:02.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:02.469+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.473065+0000) 2019-10-13T15:18:02.476 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:02.468+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.473604+0000) 2019-10-13T15:18:02.476 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:02.469+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.474100+0000) 2019-10-13T15:18:02.477 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:02.469+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.474467+0000) 2019-10-13T15:18:02.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:02.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.479514+0000) 2019-10-13T15:18:02.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:02.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:02.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:02.689+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.693769+0000) 2019-10-13T15:18:02.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:02.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.755927+0000) 2019-10-13T15:18:03.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:03.425+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:03.431127+0000) 2019-10-13T15:18:03.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:03.676+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:03.680681+0000) 2019-10-13T15:18:04.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:04.328+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.331624+0000) 2019-10-13T15:18:05.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:05.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:05.704772+0000) 2019-10-13T15:18:05.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:05.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:05.716517+0000) 2019-10-13T15:18:07.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:07.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.485302+0000) 2019-10-13T15:18:07.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:07.481+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.486662+0000) 2019-10-13T15:18:07.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:07.484+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.487154+0000) 2019-10-13T15:18:07.490 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:07.485+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.487932+0000) 2019-10-13T15:18:07.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:07.485+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.488525+0000) 2019-10-13T15:18:07.491 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:07.485+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.489073+0000) 2019-10-13T15:18:07.491 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:07.485+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.489562+0000) 2019-10-13T15:18:07.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:07.488+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:07.492768+0000) 2019-10-13T15:18:07.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:07.493+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:07.953 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:18:07.954 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:07.957 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:18:07.957 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:08.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:08.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:08.954898+0000) 2019-10-13T15:18:12.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:12.497+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.499489+0000) 2019-10-13T15:18:12.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:12.497+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.500323+0000) 2019-10-13T15:18:12.503 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:12.496+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.501182+0000) 2019-10-13T15:18:12.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:12.497+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.500971+0000) 2019-10-13T15:18:12.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:12.497+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.501848+0000) 2019-10-13T15:18:12.504 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:12.497+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.502118+0000) 2019-10-13T15:18:12.505 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:12.497+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.503243+0000) 2019-10-13T15:18:12.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:12.500+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.505738+0000) 2019-10-13T15:18:12.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:12.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:12.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:12.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.693953+0000) 2019-10-13T15:18:12.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:12.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.756130+0000) 2019-10-13T15:18:13.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:13.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:13.431331+0000) 2019-10-13T15:18:13.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:13.680+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:13.680872+0000) 2019-10-13T15:18:14.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:14.336+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.338264+0000) 2019-10-13T15:18:15.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:15.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:15.704981+0000) 2019-10-13T15:18:15.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:15.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:15.716737+0000) 2019-10-13T15:18:17.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:17.509+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.512655+0000) 2019-10-13T15:18:17.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:17.508+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.513375+0000) 2019-10-13T15:18:17.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:17.509+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.513457+0000) 2019-10-13T15:18:17.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:17.509+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.514077+0000) 2019-10-13T15:18:17.517 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:17.509+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.514275+0000) 2019-10-13T15:18:17.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:17.509+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.515083+0000) 2019-10-13T15:18:17.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:17.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.515560+0000) 2019-10-13T15:18:17.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:17.516+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:17.520922+0000) 2019-10-13T15:18:17.527 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:17.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:18.957 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:18.949+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:18.955101+0000) 2019-10-13T15:18:22.531 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:22.525+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.528012+0000) 2019-10-13T15:18:22.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:22.525+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.528733+0000) 2019-10-13T15:18:22.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:22.524+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.529683+0000) 2019-10-13T15:18:22.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:22.525+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.529230+0000) 2019-10-13T15:18:22.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:22.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.530270+0000) 2019-10-13T15:18:22.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:22.525+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.530328+0000) 2019-10-13T15:18:22.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:22.529+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.531959+0000) 2019-10-13T15:18:22.540 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:22.532+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.537792+0000) 2019-10-13T15:18:22.542 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:22.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:22.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:22.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.694133+0000) 2019-10-13T15:18:22.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:22.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.756342+0000) 2019-10-13T15:18:23.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:23.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:23.431507+0000) 2019-10-13T15:18:23.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:23.680+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:23.681094+0000) 2019-10-13T15:18:24.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:24.344+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.344864+0000) 2019-10-13T15:18:25.707 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:25.701+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:25.705213+0000) 2019-10-13T15:18:25.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:25.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:25.716929+0000) 2019-10-13T15:18:27.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:27.537+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.543070+0000) 2019-10-13T15:18:27.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:27.541+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.543470+0000) 2019-10-13T15:18:27.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:27.541+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.544093+0000) 2019-10-13T15:18:27.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:27.541+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.544536+0000) 2019-10-13T15:18:27.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:27.541+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.545144+0000) 2019-10-13T15:18:27.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:27.541+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.545466+0000) 2019-10-13T15:18:27.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:27.540+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.546058+0000) 2019-10-13T15:18:27.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:27.548+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:27.553047+0000) 2019-10-13T15:18:27.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:27.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:28.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:28.953+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:28.955344+0000) 2019-10-13T15:18:32.559 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:32.553+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.556642+0000) 2019-10-13T15:18:32.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:32.553+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.558421+0000) 2019-10-13T15:18:32.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:32.553+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.559026+0000) 2019-10-13T15:18:32.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:32.557+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.559626+0000) 2019-10-13T15:18:32.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:32.557+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.560020+0000) 2019-10-13T15:18:32.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:32.557+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.560893+0000) 2019-10-13T15:18:32.564 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:32.555+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.561904+0000) 2019-10-13T15:18:32.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:32.559+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.564347+0000) 2019-10-13T15:18:32.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:32.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 216968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:32.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:32.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.694309+0000) 2019-10-13T15:18:32.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:32.753+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.756545+0000) 2019-10-13T15:18:33.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:33.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:33.431681+0000) 2019-10-13T15:18:33.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:33.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:33.681350+0000) 2019-10-13T15:18:34.358 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:34.351+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.356155+0000) 2019-10-13T15:18:35.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:35.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:35.705410+0000) 2019-10-13T15:18:35.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:35.713+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:35.717159+0000) 2019-10-13T15:18:37.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:37.568+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.572304+0000) 2019-10-13T15:18:37.575 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:37.567+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.572967+0000) 2019-10-13T15:18:37.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:37.568+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.573308+0000) 2019-10-13T15:18:37.580 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:37.568+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.574563+0000) 2019-10-13T15:18:37.581 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:37.568+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.574814+0000) 2019-10-13T15:18:37.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:37.572+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.575378+0000) 2019-10-13T15:18:37.581 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:37.572+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.576202+0000) 2019-10-13T15:18:37.585 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:37.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217004 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:37.587 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:37.579+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:37.585343+0000) 2019-10-13T15:18:38.082 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:18:38.082 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:38.086 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:18:38.086 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:38.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:38.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:38.955564+0000) 2019-10-13T15:18:42.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:42.580+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.585749+0000) 2019-10-13T15:18:42.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:42.580+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.586589+0000) 2019-10-13T15:18:42.589 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:42.580+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.586679+0000) 2019-10-13T15:18:42.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:42.580+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.587029+0000) 2019-10-13T15:18:42.590 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:42.584+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.587361+0000) 2019-10-13T15:18:42.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:42.583+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.588941+0000) 2019-10-13T15:18:42.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:42.584+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.589408+0000) 2019-10-13T15:18:42.596 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:42.591+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.594184+0000) 2019-10-13T15:18:42.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:42.592+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:42.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:42.692+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.694507+0000) 2019-10-13T15:18:42.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:42.752+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.756720+0000) 2019-10-13T15:18:43.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:43.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:43.431849+0000) 2019-10-13T15:18:43.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:43.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:43.681556+0000) 2019-10-13T15:18:44.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:44.359+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.362711+0000) 2019-10-13T15:18:45.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:45.700+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:45.705611+0000) 2019-10-13T15:18:45.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:45.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:45.717364+0000) 2019-10-13T15:18:47.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:47.596+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.600505+0000) 2019-10-13T15:18:47.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:47.596+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.600967+0000) 2019-10-13T15:18:47.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:47.596+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.601425+0000) 2019-10-13T15:18:47.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:47.596+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.602069+0000) 2019-10-13T15:18:47.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:47.596+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.603096+0000) 2019-10-13T15:18:47.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:47.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.604486+0000) 2019-10-13T15:18:47.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:47.599+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.604834+0000) 2019-10-13T15:18:47.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:47.603+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:47.610049+0000) 2019-10-13T15:18:47.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:47.608+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217071 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:48.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:48.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:48.955795+0000) 2019-10-13T15:18:52.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:52.612+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.615871+0000) 2019-10-13T15:18:52.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:52.612+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.616225+0000) 2019-10-13T15:18:52.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:52.612+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.617061+0000) 2019-10-13T15:18:52.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:52.612+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.617122+0000) 2019-10-13T15:18:52.620 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:52.612+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.617538+0000) 2019-10-13T15:18:52.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:52.615+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.618276+0000) 2019-10-13T15:18:52.621 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:52.612+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.618627+0000) 2019-10-13T15:18:52.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:52.619+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.625009+0000) 2019-10-13T15:18:52.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:52.624+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:52.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:52.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.694703+0000) 2019-10-13T15:18:52.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:52.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.756906+0000) 2019-10-13T15:18:53.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:53.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:53.432035+0000) 2019-10-13T15:18:53.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:53.675+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:53.681765+0000) 2019-10-13T15:18:54.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:54.371+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.374117+0000) 2019-10-13T15:18:55.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:55.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:55.705829+0000) 2019-10-13T15:18:55.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:55.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:55.717556+0000) 2019-10-13T15:18:57.633 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:18:57.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.631048+0000) 2019-10-13T15:18:57.634 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:18:57.627+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.632207+0000) 2019-10-13T15:18:57.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:18:57.628+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.632507+0000) 2019-10-13T15:18:57.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:57.628+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.632864+0000) 2019-10-13T15:18:57.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:18:57.628+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.633500+0000) 2019-10-13T15:18:57.636 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:18:57.628+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.634362+0000) 2019-10-13T15:18:57.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:18:57.628+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.634751+0000) 2019-10-13T15:18:57.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:18:57.636+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217138 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:18:57.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:18:57.639+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:57.643573+0000) 2019-10-13T15:18:58.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:18:58.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:58.956040+0000) 2019-10-13T15:19:02.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:02.640+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.645623+0000) 2019-10-13T15:19:02.649 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:02.640+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.646229+0000) 2019-10-13T15:19:02.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:02.640+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.646527+0000) 2019-10-13T15:19:02.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:02.643+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.647380+0000) 2019-10-13T15:19:02.650 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:02.644+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.647391+0000) 2019-10-13T15:19:02.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:02.644+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.647967+0000) 2019-10-13T15:19:02.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:02.644+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.648818+0000) 2019-10-13T15:19:02.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:02.647+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.653278+0000) 2019-10-13T15:19:02.658 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:02.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:02.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:02.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.694912+0000) 2019-10-13T15:19:02.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:02.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.757090+0000) 2019-10-13T15:19:03.434 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:03.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:03.432200+0000) 2019-10-13T15:19:03.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:03.679+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:03.681960+0000) 2019-10-13T15:19:04.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:04.383+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.385739+0000) 2019-10-13T15:19:05.708 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:05.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:05.706031+0000) 2019-10-13T15:19:05.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:05.712+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:05.717741+0000) 2019-10-13T15:19:07.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:07.652+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.658513+0000) 2019-10-13T15:19:07.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:07.652+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.658897+0000) 2019-10-13T15:19:07.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:07.656+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.660326+0000) 2019-10-13T15:19:07.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:07.656+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.660640+0000) 2019-10-13T15:19:07.663 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:07.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.660970+0000) 2019-10-13T15:19:07.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:07.655+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.661796+0000) 2019-10-13T15:19:07.664 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:07.656+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.662089+0000) 2019-10-13T15:19:07.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:07.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217205 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:07.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:07.663+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:07.669759+0000) 2019-10-13T15:19:08.201 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:19:08.201 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:08.205 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:19:08.205 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:08.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:08.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:08.956200+0000) 2019-10-13T15:19:12.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:12.668+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.672687+0000) 2019-10-13T15:19:12.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:12.668+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.673195+0000) 2019-10-13T15:19:12.676 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:12.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.673566+0000) 2019-10-13T15:19:12.677 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:12.671+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.674589+0000) 2019-10-13T15:19:12.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:12.668+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.674185+0000) 2019-10-13T15:19:12.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:12.668+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.675101+0000) 2019-10-13T15:19:12.680 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:12.672+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.677654+0000) 2019-10-13T15:19:12.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:12.679+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.682472+0000) 2019-10-13T15:19:12.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:12.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217235 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:12.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:12.688+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.695037+0000) 2019-10-13T15:19:12.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:12.756+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.757268+0000) 2019-10-13T15:19:13.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:13.428+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:13.432328+0000) 2019-10-13T15:19:13.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:13.679+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:13.682179+0000) 2019-10-13T15:19:14.394 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:14.387+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.391989+0000) 2019-10-13T15:19:15.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:15.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:15.706233+0000) 2019-10-13T15:19:15.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:15.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:15.717941+0000) 2019-10-13T15:19:17.689 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:17.680+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.686167+0000) 2019-10-13T15:19:17.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:17.680+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.686455+0000) 2019-10-13T15:19:17.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:17.680+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.686742+0000) 2019-10-13T15:19:17.690 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:17.684+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.688365+0000) 2019-10-13T15:19:17.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:17.684+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.688544+0000) 2019-10-13T15:19:17.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:17.683+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.689201+0000) 2019-10-13T15:19:17.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:17.684+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.689626+0000) 2019-10-13T15:19:17.699 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:17.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:17.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:17.695+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:17.698248+0000) 2019-10-13T15:19:18.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:18.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:18.956458+0000) 2019-10-13T15:19:22.697 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:22.691+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.695220+0000) 2019-10-13T15:19:22.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:22.695+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.700934+0000) 2019-10-13T15:19:22.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:22.695+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.701501+0000) 2019-10-13T15:19:22.704 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:22.695+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.701332+0000) 2019-10-13T15:19:22.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:22.695+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.701782+0000) 2019-10-13T15:19:22.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:22.695+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.702146+0000) 2019-10-13T15:19:22.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:22.695+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.702602+0000) 2019-10-13T15:19:22.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:22.695+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.702937+0000) 2019-10-13T15:19:22.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:22.707+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.710643+0000) 2019-10-13T15:19:22.713 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:22.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217303 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:22.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:22.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.757448+0000) 2019-10-13T15:19:23.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:23.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:23.432497+0000) 2019-10-13T15:19:23.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:23.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:23.682390+0000) 2019-10-13T15:19:24.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:24.394+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.398544+0000) 2019-10-13T15:19:25.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:25.699+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:25.706465+0000) 2019-10-13T15:19:25.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:25.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:25.718169+0000) 2019-10-13T15:19:27.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:27.707+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.714836+0000) 2019-10-13T15:19:27.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:27.710+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.715575+0000) 2019-10-13T15:19:27.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:27.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.716094+0000) 2019-10-13T15:19:27.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:27.711+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.716058+0000) 2019-10-13T15:19:27.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:27.711+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.716449+0000) 2019-10-13T15:19:27.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:27.711+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.716733+0000) 2019-10-13T15:19:27.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:27.711+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.717403+0000) 2019-10-13T15:19:27.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:27.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217339 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:27.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:27.718+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:27.725313+0000) 2019-10-13T15:19:28.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:28.955+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:28.956638+0000) 2019-10-13T15:19:32.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:32.691+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.695424+0000) 2019-10-13T15:19:32.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:32.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.728464+0000) 2019-10-13T15:19:32.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:32.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.729189+0000) 2019-10-13T15:19:32.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:32.723+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.729349+0000) 2019-10-13T15:19:32.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:32.723+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.729845+0000) 2019-10-13T15:19:32.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:32.723+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.730758+0000) 2019-10-13T15:19:32.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:32.727+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.731750+0000) 2019-10-13T15:19:32.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:32.726+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.731490+0000) 2019-10-13T15:19:32.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:32.730+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.736136+0000) 2019-10-13T15:19:32.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:32.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217369 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:32.760 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:32.751+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.757646+0000) 2019-10-13T15:19:33.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:33.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:33.432684+0000) 2019-10-13T15:19:33.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:33.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:33.682575+0000) 2019-10-13T15:19:34.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:34.402+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.404998+0000) 2019-10-13T15:19:35.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:35.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:35.706682+0000) 2019-10-13T15:19:35.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:35.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:35.718364+0000) 2019-10-13T15:19:37.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:37.735+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.742580+0000) 2019-10-13T15:19:37.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:37.738+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.743689+0000) 2019-10-13T15:19:37.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:37.735+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.743130+0000) 2019-10-13T15:19:37.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:37.739+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.743897+0000) 2019-10-13T15:19:37.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:37.739+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.744288+0000) 2019-10-13T15:19:37.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:37.739+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.745411+0000) 2019-10-13T15:19:37.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:37.739+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.745842+0000) 2019-10-13T15:19:37.753 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:37.746+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:37.751220+0000) 2019-10-13T15:19:37.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:37.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:38.262 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:19:38.263 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:38.266 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:19:38.266 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:38.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:38.955+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:38.956837+0000) 2019-10-13T15:19:42.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:42.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.695629+0000) 2019-10-13T15:19:42.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:42.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.757355+0000) 2019-10-13T15:19:42.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:42.751+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.757892+0000) 2019-10-13T15:19:42.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:42.754+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.758428+0000) 2019-10-13T15:19:42.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:42.751+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.758626+0000) 2019-10-13T15:19:42.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:42.751+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.759193+0000) 2019-10-13T15:19:42.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:42.755+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.759646+0000) 2019-10-13T15:19:42.763 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:42.755+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.761370+0000) 2019-10-13T15:19:42.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:42.762+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.767044+0000) 2019-10-13T15:19:42.770 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:42.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:43.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:43.427+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:43.432866+0000) 2019-10-13T15:19:43.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:43.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:43.682707+0000) 2019-10-13T15:19:44.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:44.410+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.411415+0000) 2019-10-13T15:19:45.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:45.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:45.706883+0000) 2019-10-13T15:19:45.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:45.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:45.718559+0000) 2019-10-13T15:19:47.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:47.767+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.771180+0000) 2019-10-13T15:19:47.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:47.767+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.772580+0000) 2019-10-13T15:19:47.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:47.767+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.772843+0000) 2019-10-13T15:19:47.776 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:47.767+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.774063+0000) 2019-10-13T15:19:47.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:47.770+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.774760+0000) 2019-10-13T15:19:47.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:47.771+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.775548+0000) 2019-10-13T15:19:47.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:47.771+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.776057+0000) 2019-10-13T15:19:47.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:47.774+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:47.781632+0000) 2019-10-13T15:19:47.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:47.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:48.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:48.951+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:48.957052+0000) 2019-10-13T15:19:49.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:49.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.508198+0000) 2019-10-13T15:19:52.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:52.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.695785+0000) 2019-10-13T15:19:52.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:52.783+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.785365+0000) 2019-10-13T15:19:52.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:52.783+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.786161+0000) 2019-10-13T15:19:52.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:52.783+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.786635+0000) 2019-10-13T15:19:52.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:52.783+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.786857+0000) 2019-10-13T15:19:52.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:52.787+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.787588+0000) 2019-10-13T15:19:52.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:52.782+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.788129+0000) 2019-10-13T15:19:52.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:52.787+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.788288+0000) 2019-10-13T15:19:52.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:52.786+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.792329+0000) 2019-10-13T15:19:52.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:52.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217503 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:53.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:53.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:53.433045+0000) 2019-10-13T15:19:53.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:53.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:53.682835+0000) 2019-10-13T15:19:54.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:54.414+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.417834+0000) 2019-10-13T15:19:55.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:55.703+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:55.707089+0000) 2019-10-13T15:19:55.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:55.715+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:55.718758+0000) 2019-10-13T15:19:57.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:19:57.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.799937+0000) 2019-10-13T15:19:57.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:19:57.799+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.800491+0000) 2019-10-13T15:19:57.803 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:19:57.799+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.801062+0000) 2019-10-13T15:19:57.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:19:57.794+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.801558+0000) 2019-10-13T15:19:57.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:57.799+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.801723+0000) 2019-10-13T15:19:57.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:57.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.802407+0000) 2019-10-13T15:19:57.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:19:57.803+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.803278+0000) 2019-10-13T15:19:57.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:19:57.802+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:57.809045+0000) 2019-10-13T15:19:57.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:19:57.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:19:58.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:19:58.955+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:58.957269+0000) 2019-10-13T15:19:59.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:19:59.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.508400+0000) 2019-10-13T15:20:02.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:02.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.696011+0000) 2019-10-13T15:20:02.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:02.811+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.814098+0000) 2019-10-13T15:20:02.817 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:02.811+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.814674+0000) 2019-10-13T15:20:02.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:02.815+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.815633+0000) 2019-10-13T15:20:02.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:02.815+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.816253+0000) 2019-10-13T15:20:02.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:02.810+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.816952+0000) 2019-10-13T15:20:02.819 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:02.815+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.817336+0000) 2019-10-13T15:20:02.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:02.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.817898+0000) 2019-10-13T15:20:02.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:02.818+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.822687+0000) 2019-10-13T15:20:02.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:02.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:03.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:03.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:03.433222+0000) 2019-10-13T15:20:03.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:03.678+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:03.683019+0000) 2019-10-13T15:20:04.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:04.426+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:04.429057+0000) 2019-10-13T15:20:05.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:05.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.707288+0000) 2019-10-13T15:20:05.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:05.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.718975+0000) 2019-10-13T15:20:07.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:07.826+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.828174+0000) 2019-10-13T15:20:07.832 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:07.826+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.829780+0000) 2019-10-13T15:20:07.832 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:07.826+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.830401+0000) 2019-10-13T15:20:07.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:07.826+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.830758+0000) 2019-10-13T15:20:07.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:07.830+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.831427+0000) 2019-10-13T15:20:07.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:07.826+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.832052+0000) 2019-10-13T15:20:07.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:07.830+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.832188+0000) 2019-10-13T15:20:07.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:07.830+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:07.837217+0000) 2019-10-13T15:20:07.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:07.838+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:08.399 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:20:08.399 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:08.403 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:20:08.403 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:08.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:08.954+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:08.957467+0000) 2019-10-13T15:20:09.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:09.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:09.508614+0000) 2019-10-13T15:20:12.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:12.694+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.696215+0000) 2019-10-13T15:20:12.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:12.838+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.842561+0000) 2019-10-13T15:20:12.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:12.838+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.843083+0000) 2019-10-13T15:20:12.847 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:12.842+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.844967+0000) 2019-10-13T15:20:12.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:12.842+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.845192+0000) 2019-10-13T15:20:12.848 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:12.842+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.845761+0000) 2019-10-13T15:20:12.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:12.842+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.845991+0000) 2019-10-13T15:20:12.849 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:12.842+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.846290+0000) 2019-10-13T15:20:12.856 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:12.850+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:12.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:12.854+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.854835+0000) 2019-10-13T15:20:13.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:13.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:13.433399+0000) 2019-10-13T15:20:13.685 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:13.682+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:13.683194+0000) 2019-10-13T15:20:14.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:14.434+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:14.435735+0000) 2019-10-13T15:20:15.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:15.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.707523+0000) 2019-10-13T15:20:15.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:15.714+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.719193+0000) 2019-10-13T15:20:17.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:17.854+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.857001+0000) 2019-10-13T15:20:17.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:17.854+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.857612+0000) 2019-10-13T15:20:17.861 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:17.858+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.858568+0000) 2019-10-13T15:20:17.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:17.854+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.858013+0000) 2019-10-13T15:20:17.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:17.854+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.858865+0000) 2019-10-13T15:20:17.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:17.854+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.859252+0000) 2019-10-13T15:20:17.862 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:17.858+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.859667+0000) 2019-10-13T15:20:17.870 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:17.866+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217672 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:17.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:17.866+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:17.869119+0000) 2019-10-13T15:20:18.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:18.954+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:18.957665+0000) 2019-10-13T15:20:19.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:19.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:19.508814+0000) 2019-10-13T15:20:22.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:22.694+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.696416+0000) 2019-10-13T15:20:22.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:22.870+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.871386+0000) 2019-10-13T15:20:22.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:22.869+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.871928+0000) 2019-10-13T15:20:22.875 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:22.870+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.872840+0000) 2019-10-13T15:20:22.875 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:22.870+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.873248+0000) 2019-10-13T15:20:22.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:22.870+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.873778+0000) 2019-10-13T15:20:22.878 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:22.870+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.874319+0000) 2019-10-13T15:20:22.879 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:22.870+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.874874+0000) 2019-10-13T15:20:22.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:22.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:22.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:22.881+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.882614+0000) 2019-10-13T15:20:23.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:23.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:23.433633+0000) 2019-10-13T15:20:23.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:23.681+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:23.683375+0000) 2019-10-13T15:20:24.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:24.441+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:24.442244+0000) 2019-10-13T15:20:25.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:25.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.707724+0000) 2019-10-13T15:20:25.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:25.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.719395+0000) 2019-10-13T15:20:27.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:27.882+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.884600+0000) 2019-10-13T15:20:27.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:27.882+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.886286+0000) 2019-10-13T15:20:27.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:27.882+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.886622+0000) 2019-10-13T15:20:27.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:27.885+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.887432+0000) 2019-10-13T15:20:27.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:27.882+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.887265+0000) 2019-10-13T15:20:27.890 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:27.886+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.887783+0000) 2019-10-13T15:20:27.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:27.886+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.888110+0000) 2019-10-13T15:20:27.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:27.889+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:27.891625+0000) 2019-10-13T15:20:27.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:27.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:28.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:28.954+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:28.957826+0000) 2019-10-13T15:20:29.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:29.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:29.509048+0000) 2019-10-13T15:20:32.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:32.694+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.696649+0000) 2019-10-13T15:20:32.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:32.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.898587+0000) 2019-10-13T15:20:32.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:32.894+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.898021+0000) 2019-10-13T15:20:32.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:32.894+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.898771+0000) 2019-10-13T15:20:32.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:32.894+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.899265+0000) 2019-10-13T15:20:32.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:32.898+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.899944+0000) 2019-10-13T15:20:32.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:32.898+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.901092+0000) 2019-10-13T15:20:32.904 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:32.898+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.901473+0000) 2019-10-13T15:20:32.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:32.905+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.907007+0000) 2019-10-13T15:20:32.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:32.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:33.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:33.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:33.433809+0000) 2019-10-13T15:20:33.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:33.681+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:33.683568+0000) 2019-10-13T15:20:34.451 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:34.445+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:34.448432+0000) 2019-10-13T15:20:35.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:35.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.707925+0000) 2019-10-13T15:20:35.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:35.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.719590+0000) 2019-10-13T15:20:37.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:37.910+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.912464+0000) 2019-10-13T15:20:37.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:37.910+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.912567+0000) 2019-10-13T15:20:37.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:37.910+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.913162+0000) 2019-10-13T15:20:37.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:37.910+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.913514+0000) 2019-10-13T15:20:37.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:37.913+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.914294+0000) 2019-10-13T15:20:37.918 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:37.910+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.914277+0000) 2019-10-13T15:20:37.919 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:37.914+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.915447+0000) 2019-10-13T15:20:37.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:37.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:37.921485+0000) 2019-10-13T15:20:37.924 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:37.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:38.453 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:20:38.453 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:38.457 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:20:38.457 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:38.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:38.954+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:38.958046+0000) 2019-10-13T15:20:39.511 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:39.506+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:39.509199+0000) 2019-10-13T15:20:42.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:42.694+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.696849+0000) 2019-10-13T15:20:42.927 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:42.922+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.924480+0000) 2019-10-13T15:20:42.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:42.921+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.925337+0000) 2019-10-13T15:20:42.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:42.922+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.925582+0000) 2019-10-13T15:20:42.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:42.922+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.926624+0000) 2019-10-13T15:20:42.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:42.922+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.927012+0000) 2019-10-13T15:20:42.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:42.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.927433+0000) 2019-10-13T15:20:42.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:42.926+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.928547+0000) 2019-10-13T15:20:42.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:42.929+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.933432+0000) 2019-10-13T15:20:42.938 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:42.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:43.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:43.430+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:43.433982+0000) 2019-10-13T15:20:43.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:43.681+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:43.683775+0000) 2019-10-13T15:20:44.457 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:44.453+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:44.454777+0000) 2019-10-13T15:20:45.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:45.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.708142+0000) 2019-10-13T15:20:45.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:45.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.719819+0000) 2019-10-13T15:20:47.942 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:47.937+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.939357+0000) 2019-10-13T15:20:47.943 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:47.937+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.940574+0000) 2019-10-13T15:20:47.943 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:47.937+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.941126+0000) 2019-10-13T15:20:47.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:47.937+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.941714+0000) 2019-10-13T15:20:47.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:47.941+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.942205+0000) 2019-10-13T15:20:47.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:47.937+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.942388+0000) 2019-10-13T15:20:47.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:47.941+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.943402+0000) 2019-10-13T15:20:47.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:47.945+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:47.946843+0000) 2019-10-13T15:20:47.952 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:47.945+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217868 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:48.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:48.953+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:48.958244+0000) 2019-10-13T15:20:49.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:49.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:49.509412+0000) 2019-10-13T15:20:52.699 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:52.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.697044+0000) 2019-10-13T15:20:52.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:52.949+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.953098+0000) 2019-10-13T15:20:52.957 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:52.949+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.954618+0000) 2019-10-13T15:20:52.957 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:52.949+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.955089+0000) 2019-10-13T15:20:52.958 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:52.953+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.955734+0000) 2019-10-13T15:20:52.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:52.953+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.955967+0000) 2019-10-13T15:20:52.958 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:52.953+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.956311+0000) 2019-10-13T15:20:52.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:52.953+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.957057+0000) 2019-10-13T15:20:52.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:52.957+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.960041+0000) 2019-10-13T15:20:52.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:52.961+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:53.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:53.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:53.434157+0000) 2019-10-13T15:20:53.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:53.681+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:53.683969+0000) 2019-10-13T15:20:54.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:54.457+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:54.461037+0000) 2019-10-13T15:20:55.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:55.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.708374+0000) 2019-10-13T15:20:55.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:55.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.720014+0000) 2019-10-13T15:20:57.970 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:20:57.965+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.967949+0000) 2019-10-13T15:20:57.971 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:20:57.965+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.968708+0000) 2019-10-13T15:20:57.972 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:20:57.965+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.969646+0000) 2019-10-13T15:20:57.972 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:20:57.969+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.970328+0000) 2019-10-13T15:20:57.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:57.965+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.970506+0000) 2019-10-13T15:20:57.977 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:20:57.965+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.971076+0000) 2019-10-13T15:20:57.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:57.969+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.971373+0000) 2019-10-13T15:20:57.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:20:57.977+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:57.978251+0000) 2019-10-13T15:20:57.981 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:20:57.973+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:20:58.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:20:58.953+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:58.958475+0000) 2019-10-13T15:20:59.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:20:59.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:59.509642+0000) 2019-10-13T15:21:02.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:02.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.697245+0000) 2019-10-13T15:21:02.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:02.977+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.982091+0000) 2019-10-13T15:21:02.985 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:02.977+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.982520+0000) 2019-10-13T15:21:02.986 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:02.981+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.983852+0000) 2019-10-13T15:21:02.986 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:02.981+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.983580+0000) 2019-10-13T15:21:02.987 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:02.981+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.984635+0000) 2019-10-13T15:21:02.987 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:02.981+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.984882+0000) 2019-10-13T15:21:02.987 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:02.981+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.985441+0000) 2019-10-13T15:21:02.995 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:02.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 217966 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:02.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:02.989+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.993771+0000) 2019-10-13T15:21:03.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:03.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:03.434333+0000) 2019-10-13T15:21:03.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:03.681+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:03.684148+0000) 2019-10-13T15:21:04.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:04.465+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:04.467454+0000) 2019-10-13T15:21:05.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:05.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.708576+0000) 2019-10-13T15:21:05.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:05.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.720240+0000) 2019-10-13T15:21:07.999 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:07.993+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.996808+0000) 2019-10-13T15:21:08.000 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:07.993+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.997618+0000) 2019-10-13T15:21:08.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:07.993+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.997843+0000) 2019-10-13T15:21:08.001 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:07.993+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.998572+0000) 2019-10-13T15:21:08.002 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:07.993+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.999228+0000) 2019-10-13T15:21:08.002 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:07.997+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:07.999466+0000) 2019-10-13T15:21:08.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:07.997+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:08.000076+0000) 2019-10-13T15:21:08.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:08.001+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:08.005435+0000) 2019-10-13T15:21:08.010 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:08.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218000 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:08.564 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:21:08.564 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:08.576 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:21:08.576 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:08.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:08.953+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:08.958673+0000) 2019-10-13T15:21:09.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:09.505+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:09.509800+0000) 2019-10-13T15:21:12.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:12.697+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:12.697482+0000) 2019-10-13T15:21:13.013 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:13.005+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.011083+0000) 2019-10-13T15:21:13.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:13.009+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.012156+0000) 2019-10-13T15:21:13.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:13.009+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.012473+0000) 2019-10-13T15:21:13.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:13.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.013055+0000) 2019-10-13T15:21:13.016 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:13.009+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.012714+0000) 2019-10-13T15:21:13.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:13.009+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.014305+0000) 2019-10-13T15:21:13.017 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:13.009+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.014440+0000) 2019-10-13T15:21:13.022 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:13.017+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.020450+0000) 2019-10-13T15:21:13.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:13.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:13.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:13.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.434513+0000) 2019-10-13T15:21:13.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:13.680+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:13.684327+0000) 2019-10-13T15:21:14.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:14.476+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:14.478698+0000) 2019-10-13T15:21:15.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:15.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.708773+0000) 2019-10-13T15:21:15.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:15.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.720436+0000) 2019-10-13T15:21:18.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:18.021+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.026173+0000) 2019-10-13T15:21:18.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:18.024+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.026773+0000) 2019-10-13T15:21:18.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:18.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.026963+0000) 2019-10-13T15:21:18.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:18.025+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.028134+0000) 2019-10-13T15:21:18.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:18.025+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.028402+0000) 2019-10-13T15:21:18.031 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:18.025+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.028971+0000) 2019-10-13T15:21:18.031 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:18.025+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.029197+0000) 2019-10-13T15:21:18.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:18.032+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.035273+0000) 2019-10-13T15:21:18.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:18.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218067 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:18.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:18.953+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.958842+0000) 2019-10-13T15:21:19.512 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:19.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:19.510035+0000) 2019-10-13T15:21:22.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:22.693+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:22.697700+0000) 2019-10-13T15:21:23.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:23.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.040773+0000) 2019-10-13T15:21:23.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:23.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.041305+0000) 2019-10-13T15:21:23.044 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:23.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.042002+0000) 2019-10-13T15:21:23.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:23.037+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.042509+0000) 2019-10-13T15:21:23.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:23.041+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.043733+0000) 2019-10-13T15:21:23.046 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:23.041+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.044133+0000) 2019-10-13T15:21:23.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:23.041+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.044445+0000) 2019-10-13T15:21:23.055 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:23.049+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:23.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:23.048+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.053529+0000) 2019-10-13T15:21:23.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:23.429+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.434688+0000) 2019-10-13T15:21:23.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:23.680+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:23.684498+0000) 2019-10-13T15:21:24.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:24.488+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:24.490544+0000) 2019-10-13T15:21:25.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:25.705+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.708975+0000) 2019-10-13T15:21:25.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:25.717+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.720660+0000) 2019-10-13T15:21:28.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:28.053+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.056451+0000) 2019-10-13T15:21:28.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:28.053+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.056615+0000) 2019-10-13T15:21:28.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:28.052+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.057184+0000) 2019-10-13T15:21:28.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:28.053+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.057446+0000) 2019-10-13T15:21:28.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:28.053+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.058501+0000) 2019-10-13T15:21:28.061 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:28.053+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.058931+0000) 2019-10-13T15:21:28.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:28.057+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.059552+0000) 2019-10-13T15:21:28.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:28.060+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.065840+0000) 2019-10-13T15:21:28.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:28.065+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:28.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:28.957+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.959049+0000) 2019-10-13T15:21:29.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:29.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:29.510282+0000) 2019-10-13T15:21:32.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:32.692+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:32.697893+0000) 2019-10-13T15:21:33.074 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:33.064+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.071225+0000) 2019-10-13T15:21:33.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:33.068+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.071857+0000) 2019-10-13T15:21:33.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:33.068+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.072705+0000) 2019-10-13T15:21:33.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:33.068+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.072742+0000) 2019-10-13T15:21:33.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:33.068+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.073474+0000) 2019-10-13T15:21:33.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:33.068+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.073664+0000) 2019-10-13T15:21:33.077 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:33.068+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.074835+0000) 2019-10-13T15:21:33.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:33.076+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.079755+0000) 2019-10-13T15:21:33.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:33.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:33.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:33.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.434865+0000) 2019-10-13T15:21:33.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:33.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:33.684672+0000) 2019-10-13T15:21:34.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:34.492+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:34.496964+0000) 2019-10-13T15:21:35.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:35.704+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.709203+0000) 2019-10-13T15:21:35.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:35.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.720887+0000) 2019-10-13T15:21:38.088 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:38.080+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.086159+0000) 2019-10-13T15:21:38.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:38.080+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.086531+0000) 2019-10-13T15:21:38.089 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:38.080+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.087133+0000) 2019-10-13T15:21:38.090 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:38.084+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.087482+0000) 2019-10-13T15:21:38.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:38.084+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.088120+0000) 2019-10-13T15:21:38.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:38.084+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.088186+0000) 2019-10-13T15:21:38.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:38.084+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.089483+0000) 2019-10-13T15:21:38.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:38.092+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.094860+0000) 2019-10-13T15:21:38.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:38.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218198 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:38.633 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:21:38.633 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:38.637 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:21:38.637 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:38.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:38.952+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.959242+0000) 2019-10-13T15:21:39.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:39.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:39.510498+0000) 2019-10-13T15:21:42.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:42.696+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:42.698100+0000) 2019-10-13T15:21:43.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:43.096+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.100936+0000) 2019-10-13T15:21:43.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:43.096+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.101534+0000) 2019-10-13T15:21:43.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:43.096+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.102181+0000) 2019-10-13T15:21:43.105 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:43.096+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.102824+0000) 2019-10-13T15:21:43.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:43.096+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.103106+0000) 2019-10-13T15:21:43.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:43.100+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.103437+0000) 2019-10-13T15:21:43.106 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:43.100+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.103637+0000) 2019-10-13T15:21:43.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:43.104+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.108249+0000) 2019-10-13T15:21:43.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:43.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:43.437 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:43.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.435061+0000) 2019-10-13T15:21:43.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:43.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:43.684846+0000) 2019-10-13T15:21:44.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:44.500+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:44.503309+0000) 2019-10-13T15:21:45.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:45.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.709410+0000) 2019-10-13T15:21:45.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:45.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.721079+0000) 2019-10-13T15:21:48.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:48.112+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.116204+0000) 2019-10-13T15:21:48.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:48.112+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.116440+0000) 2019-10-13T15:21:48.120 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:48.112+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.116927+0000) 2019-10-13T15:21:48.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:48.112+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.117413+0000) 2019-10-13T15:21:48.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:48.116+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.118438+0000) 2019-10-13T15:21:48.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:48.112+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.118991+0000) 2019-10-13T15:21:48.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:48.116+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.119443+0000) 2019-10-13T15:21:48.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:48.120+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.123876+0000) 2019-10-13T15:21:48.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:48.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:48.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:48.956+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.959504+0000) 2019-10-13T15:21:49.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:49.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:49.510712+0000) 2019-10-13T15:21:52.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:52.696+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:52.698329+0000) 2019-10-13T15:21:53.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:53.124+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.131196+0000) 2019-10-13T15:21:53.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:53.128+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.131780+0000) 2019-10-13T15:21:53.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:53.128+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.132370+0000) 2019-10-13T15:21:53.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:53.128+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.132718+0000) 2019-10-13T15:21:53.136 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:53.128+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.132858+0000) 2019-10-13T15:21:53.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:53.128+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.133628+0000) 2019-10-13T15:21:53.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:53.132+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.134560+0000) 2019-10-13T15:21:53.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:53.132+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.136905+0000) 2019-10-13T15:21:53.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:53.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218298 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:53.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:53.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.435251+0000) 2019-10-13T15:21:53.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:53.680+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:53.685016+0000) 2019-10-13T15:21:54.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:54.504+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:54.509685+0000) 2019-10-13T15:21:55.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:55.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.709565+0000) 2019-10-13T15:21:55.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:55.716+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.721224+0000) 2019-10-13T15:21:58.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:58.140+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.144029+0000) 2019-10-13T15:21:58.147 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:21:58.140+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.144896+0000) 2019-10-13T15:21:58.147 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:21:58.140+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.145253+0000) 2019-10-13T15:21:58.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:58.140+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.146114+0000) 2019-10-13T15:21:58.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:21:58.140+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.146613+0000) 2019-10-13T15:21:58.149 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:21:58.144+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.147534+0000) 2019-10-13T15:21:58.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:21:58.144+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.148409+0000) 2019-10-13T15:21:58.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:21:58.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218331 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:21:58.159 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:21:58.152+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.156497+0000) 2019-10-13T15:21:58.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:21:58.956+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.959675+0000) 2019-10-13T15:21:59.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:21:59.504+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:59.510884+0000) 2019-10-13T15:22:02.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:02.696+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:02.698547+0000) 2019-10-13T15:22:03.161 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:03.156+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.159070+0000) 2019-10-13T15:22:03.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:03.156+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.160482+0000) 2019-10-13T15:22:03.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:03.156+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.160820+0000) 2019-10-13T15:22:03.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:03.156+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.161458+0000) 2019-10-13T15:22:03.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:03.156+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.162067+0000) 2019-10-13T15:22:03.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:03.156+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.162666+0000) 2019-10-13T15:22:03.165 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:03.160+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.163414+0000) 2019-10-13T15:22:03.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:03.160+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.166016+0000) 2019-10-13T15:22:03.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:03.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218365 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:03.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:03.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.435456+0000) 2019-10-13T15:22:03.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:03.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:03.685192+0000) 2019-10-13T15:22:04.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:04.516+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:04.516285+0000) 2019-10-13T15:22:05.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:05.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.709785+0000) 2019-10-13T15:22:05.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:05.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.721418+0000) 2019-10-13T15:22:08.177 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:08.171+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.175287+0000) 2019-10-13T15:22:08.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:08.172+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.175532+0000) 2019-10-13T15:22:08.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:08.172+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.175996+0000) 2019-10-13T15:22:08.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:08.172+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.177169+0000) 2019-10-13T15:22:08.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:08.172+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.177532+0000) 2019-10-13T15:22:08.180 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:08.172+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.178234+0000) 2019-10-13T15:22:08.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:08.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.178995+0000) 2019-10-13T15:22:08.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:08.179+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.183133+0000) 2019-10-13T15:22:08.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:08.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218397 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:08.681 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:22:08.681 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:08.685 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:22:08.685 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:08.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:08.956+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.959857+0000) 2019-10-13T15:22:09.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:09.508+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:09.511064+0000) 2019-10-13T15:22:12.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:12.692+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:12.698782+0000) 2019-10-13T15:22:13.192 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:13.184+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.189588+0000) 2019-10-13T15:22:13.192 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:13.187+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.190277+0000) 2019-10-13T15:22:13.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:13.184+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.190554+0000) 2019-10-13T15:22:13.193 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:13.188+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.191398+0000) 2019-10-13T15:22:13.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:13.188+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.191661+0000) 2019-10-13T15:22:13.194 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:13.188+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.191936+0000) 2019-10-13T15:22:13.195 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:13.188+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.193148+0000) 2019-10-13T15:22:13.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:13.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.197910+0000) 2019-10-13T15:22:13.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:13.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218432 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:13.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:13.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.435630+0000) 2019-10-13T15:22:13.687 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:13.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:13.685365+0000) 2019-10-13T15:22:14.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:14.519+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:14.522765+0000) 2019-10-13T15:22:15.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:15.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.710020+0000) 2019-10-13T15:22:15.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:15.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.721614+0000) 2019-10-13T15:22:18.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:18.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.204295+0000) 2019-10-13T15:22:18.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:18.199+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.204908+0000) 2019-10-13T15:22:18.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:18.199+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.205594+0000) 2019-10-13T15:22:18.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:18.199+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.205993+0000) 2019-10-13T15:22:18.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:18.199+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.206204+0000) 2019-10-13T15:22:18.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:18.203+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.207360+0000) 2019-10-13T15:22:18.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:18.203+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.207564+0000) 2019-10-13T15:22:18.217 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:18.207+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218465 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:18.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:18.211+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.215343+0000) 2019-10-13T15:22:18.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:18.955+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.960050+0000) 2019-10-13T15:22:19.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:19.503+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:19.511321+0000) 2019-10-13T15:22:22.701 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:22.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:22.699032+0000) 2019-10-13T15:22:23.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:23.211+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.217560+0000) 2019-10-13T15:22:23.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:23.211+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.218454+0000) 2019-10-13T15:22:23.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:23.215+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.219528+0000) 2019-10-13T15:22:23.222 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:23.215+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.220162+0000) 2019-10-13T15:22:23.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:23.215+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.220570+0000) 2019-10-13T15:22:23.223 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:23.215+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.220810+0000) 2019-10-13T15:22:23.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:23.215+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.221651+0000) 2019-10-13T15:22:23.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:23.219+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.224742+0000) 2019-10-13T15:22:23.232 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:23.223+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:23.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:23.431+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.435824+0000) 2019-10-13T15:22:23.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:23.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:23.685539+0000) 2019-10-13T15:22:24.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:24.527+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:24.529244+0000) 2019-10-13T15:22:25.712 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:25.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.710199+0000) 2019-10-13T15:22:25.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:25.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.721829+0000) 2019-10-13T15:22:28.235 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:28.227+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.232821+0000) 2019-10-13T15:22:28.236 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:28.227+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.233839+0000) 2019-10-13T15:22:28.236 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:28.227+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.234082+0000) 2019-10-13T15:22:28.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:28.227+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.234386+0000) 2019-10-13T15:22:28.237 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:28.231+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.235251+0000) 2019-10-13T15:22:28.238 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:28.231+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.235396+0000) 2019-10-13T15:22:28.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:28.231+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.235734+0000) 2019-10-13T15:22:28.241 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:28.235+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.238874+0000) 2019-10-13T15:22:28.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:28.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:28.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:28.959+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.960244+0000) 2019-10-13T15:22:29.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:29.507+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:29.511530+0000) 2019-10-13T15:22:32.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:32.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:32.699237+0000) 2019-10-13T15:22:33.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:33.243+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.247910+0000) 2019-10-13T15:22:33.251 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:33.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.248390+0000) 2019-10-13T15:22:33.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:33.243+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.248727+0000) 2019-10-13T15:22:33.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:33.243+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.249661+0000) 2019-10-13T15:22:33.252 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:33.243+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.250493+0000) 2019-10-13T15:22:33.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:33.243+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.250947+0000) 2019-10-13T15:22:33.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:33.247+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.251550+0000) 2019-10-13T15:22:33.260 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:33.251+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218566 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:33.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:33.255+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.258246+0000) 2019-10-13T15:22:33.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:33.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.436036+0000) 2019-10-13T15:22:33.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:33.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:33.685726+0000) 2019-10-13T15:22:34.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:34.535+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:34.535741+0000) 2019-10-13T15:22:35.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:35.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.710395+0000) 2019-10-13T15:22:35.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:35.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.722025+0000) 2019-10-13T15:22:38.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:38.255+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.260298+0000) 2019-10-13T15:22:38.264 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:38.255+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.261818+0000) 2019-10-13T15:22:38.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:38.255+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.262256+0000) 2019-10-13T15:22:38.265 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:38.259+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.263456+0000) 2019-10-13T15:22:38.267 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:38.259+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.264331+0000) 2019-10-13T15:22:38.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:38.259+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.264568+0000) 2019-10-13T15:22:38.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:38.259+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.264940+0000) 2019-10-13T15:22:38.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:38.267+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.271407+0000) 2019-10-13T15:22:38.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:38.267+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:38.738 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:22:38.738 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:38.742 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:22:38.742 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:38.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:38.959+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.960424+0000) 2019-10-13T15:22:39.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:39.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:39.511755+0000) 2019-10-13T15:22:42.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:42.695+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:42.699471+0000) 2019-10-13T15:22:43.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:43.271+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.275709+0000) 2019-10-13T15:22:43.279 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:43.271+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.276156+0000) 2019-10-13T15:22:43.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:43.271+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.277684+0000) 2019-10-13T15:22:43.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:43.271+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.277848+0000) 2019-10-13T15:22:43.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:43.271+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.278245+0000) 2019-10-13T15:22:43.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:43.271+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.278834+0000) 2019-10-13T15:22:43.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:43.271+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.278883+0000) 2019-10-13T15:22:43.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:43.279+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.283483+0000) 2019-10-13T15:22:43.290 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:43.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:43.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:43.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.436211+0000) 2019-10-13T15:22:43.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:43.679+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:43.685931+0000) 2019-10-13T15:22:44.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:44.539+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:44.542206+0000) 2019-10-13T15:22:45.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:45.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.710594+0000) 2019-10-13T15:22:45.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:45.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.722220+0000) 2019-10-13T15:22:48.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:48.287+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.292611+0000) 2019-10-13T15:22:48.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:48.291+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.291403+0000) 2019-10-13T15:22:48.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:48.291+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.293301+0000) 2019-10-13T15:22:48.297 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:48.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.293534+0000) 2019-10-13T15:22:48.297 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:48.291+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.293943+0000) 2019-10-13T15:22:48.297 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:48.291+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.294190+0000) 2019-10-13T15:22:48.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:48.291+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.294888+0000) 2019-10-13T15:22:48.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:48.295+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.299408+0000) 2019-10-13T15:22:48.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:48.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218665 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:48.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:48.959+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.960596+0000) 2019-10-13T15:22:49.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:49.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:49.511965+0000) 2019-10-13T15:22:52.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:52.699+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:52.699671+0000) 2019-10-13T15:22:53.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:53.303+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.305988+0000) 2019-10-13T15:22:53.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:53.303+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.307009+0000) 2019-10-13T15:22:53.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:53.307+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.307500+0000) 2019-10-13T15:22:53.310 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:53.307+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.307749+0000) 2019-10-13T15:22:53.310 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:53.307+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.308493+0000) 2019-10-13T15:22:53.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:53.303+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.309151+0000) 2019-10-13T15:22:53.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:53.307+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.310048+0000) 2019-10-13T15:22:53.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:53.311+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.315202+0000) 2019-10-13T15:22:53.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:53.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218700 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:53.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:53.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.436385+0000) 2019-10-13T15:22:53.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:53.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:53.686125+0000) 2019-10-13T15:22:54.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:54.543+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:54.548734+0000) 2019-10-13T15:22:55.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:55.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.710821+0000) 2019-10-13T15:22:55.724 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:55.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.722431+0000) 2019-10-13T15:22:58.324 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:22:58.319+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.321201+0000) 2019-10-13T15:22:58.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:22:58.319+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.321797+0000) 2019-10-13T15:22:58.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:22:58.319+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.322459+0000) 2019-10-13T15:22:58.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:22:58.319+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.323028+0000) 2019-10-13T15:22:58.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:58.323+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.323418+0000) 2019-10-13T15:22:58.326 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:58.323+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.323597+0000) 2019-10-13T15:22:58.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:22:58.323+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.323873+0000) 2019-10-13T15:22:58.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:22:58.327+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.330368+0000) 2019-10-13T15:22:58.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:22:58.331+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218733 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:22:58.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:22:58.958+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.960771+0000) 2019-10-13T15:22:59.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:22:59.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:59.512186+0000) 2019-10-13T15:23:02.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:02.698+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:02.699872+0000) 2019-10-13T15:23:03.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:03.330+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.335500+0000) 2019-10-13T15:23:03.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:03.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.335628+0000) 2019-10-13T15:23:03.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:03.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.336391+0000) 2019-10-13T15:23:03.339 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:03.334+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.336869+0000) 2019-10-13T15:23:03.340 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:03.334+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.337235+0000) 2019-10-13T15:23:03.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:03.334+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.337787+0000) 2019-10-13T15:23:03.340 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:03.334+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.338159+0000) 2019-10-13T15:23:03.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:03.338+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.344044+0000) 2019-10-13T15:23:03.350 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:03.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218765 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:03.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:03.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.436567+0000) 2019-10-13T15:23:03.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:03.682+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:03.686292+0000) 2019-10-13T15:23:04.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:04.554+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:04.555103+0000) 2019-10-13T15:23:05.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:05.706+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.711068+0000) 2019-10-13T15:23:05.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:05.718+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.722630+0000) 2019-10-13T15:23:08.353 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:08.346+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.350800+0000) 2019-10-13T15:23:08.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:08.346+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.351408+0000) 2019-10-13T15:23:08.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:08.350+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.352014+0000) 2019-10-13T15:23:08.355 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:08.350+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.352638+0000) 2019-10-13T15:23:08.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:08.350+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.352143+0000) 2019-10-13T15:23:08.356 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:08.350+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.353077+0000) 2019-10-13T15:23:08.356 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:08.350+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.353577+0000) 2019-10-13T15:23:08.362 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:08.354+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.360591+0000) 2019-10-13T15:23:08.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:08.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218798 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:08.804 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:23:08.804 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:08.808 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:23:08.808 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:08.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:08.958+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.960943+0000) 2019-10-13T15:23:09.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:09.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:09.512401+0000) 2019-10-13T15:23:12.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:12.698+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:12.700114+0000) 2019-10-13T15:23:13.369 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:13.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.365408+0000) 2019-10-13T15:23:13.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:13.362+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.365887+0000) 2019-10-13T15:23:13.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:13.362+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.366245+0000) 2019-10-13T15:23:13.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:13.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.368000+0000) 2019-10-13T15:23:13.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:13.366+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.368159+0000) 2019-10-13T15:23:13.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:13.366+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.368410+0000) 2019-10-13T15:23:13.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:13.362+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.368993+0000) 2019-10-13T15:23:13.377 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:13.370+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.375234+0000) 2019-10-13T15:23:13.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:13.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218831 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:13.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:13.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.436749+0000) 2019-10-13T15:23:13.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:13.682+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:13.686464+0000) 2019-10-13T15:23:14.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:14.558+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:14.561349+0000) 2019-10-13T15:23:15.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:15.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.711251+0000) 2019-10-13T15:23:15.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:15.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.722834+0000) 2019-10-13T15:23:18.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:18.378+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.381628+0000) 2019-10-13T15:23:18.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:18.378+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.381836+0000) 2019-10-13T15:23:18.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:18.378+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.382057+0000) 2019-10-13T15:23:18.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:18.378+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.382789+0000) 2019-10-13T15:23:18.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:18.378+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.382975+0000) 2019-10-13T15:23:18.386 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:18.378+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.383276+0000) 2019-10-13T15:23:18.387 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:18.378+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.383828+0000) 2019-10-13T15:23:18.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:18.382+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.387373+0000) 2019-10-13T15:23:18.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:18.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218865 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:18.963 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:18.958+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.961135+0000) 2019-10-13T15:23:19.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:19.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:19.512622+0000) 2019-10-13T15:23:22.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:22.698+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:22.700351+0000) 2019-10-13T15:23:23.397 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:23.390+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.394758+0000) 2019-10-13T15:23:23.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:23.390+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.395028+0000) 2019-10-13T15:23:23.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:23.390+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.395568+0000) 2019-10-13T15:23:23.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:23.394+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.396133+0000) 2019-10-13T15:23:23.399 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:23.394+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.396586+0000) 2019-10-13T15:23:23.399 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:23.394+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.397177+0000) 2019-10-13T15:23:23.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:23.394+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.397857+0000) 2019-10-13T15:23:23.405 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:23.398+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.403613+0000) 2019-10-13T15:23:23.408 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:23.402+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218897 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:23.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:23.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.436907+0000) 2019-10-13T15:23:23.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:23.682+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:23.686636+0000) 2019-10-13T15:23:24.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:24.562+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:24.567726+0000) 2019-10-13T15:23:25.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:25.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.711514+0000) 2019-10-13T15:23:25.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:25.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.723063+0000) 2019-10-13T15:23:28.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:28.406+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.409072+0000) 2019-10-13T15:23:28.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:28.406+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.410306+0000) 2019-10-13T15:23:28.413 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:28.406+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.410742+0000) 2019-10-13T15:23:28.413 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:28.406+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.410964+0000) 2019-10-13T15:23:28.414 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:28.410+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.411336+0000) 2019-10-13T15:23:28.414 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:28.410+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.412081+0000) 2019-10-13T15:23:28.415 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:28.410+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.412478+0000) 2019-10-13T15:23:28.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:28.410+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.416164+0000) 2019-10-13T15:23:28.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:28.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:28.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:28.958+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.961361+0000) 2019-10-13T15:23:29.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:29.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:29.512823+0000) 2019-10-13T15:23:32.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:32.698+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:32.700555+0000) 2019-10-13T15:23:33.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:33.418+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.423285+0000) 2019-10-13T15:23:33.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:33.422+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.424247+0000) 2019-10-13T15:23:33.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:33.422+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.424948+0000) 2019-10-13T15:23:33.427 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:33.422+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.424807+0000) 2019-10-13T15:23:33.428 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:33.422+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.425684+0000) 2019-10-13T15:23:33.428 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:33.422+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.426218+0000) 2019-10-13T15:23:33.429 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:33.422+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.426670+0000) 2019-10-13T15:23:33.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:33.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218963 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:33.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:33.430+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.435167+0000) 2019-10-13T15:23:33.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:33.434+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.437012+0000) 2019-10-13T15:23:33.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:33.686+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:33.686807+0000) 2019-10-13T15:23:34.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:34.570+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:34.574106+0000) 2019-10-13T15:23:35.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:35.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.711713+0000) 2019-10-13T15:23:35.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:35.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.723263+0000) 2019-10-13T15:23:38.441 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:38.434+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.438643+0000) 2019-10-13T15:23:38.441 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:38.434+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.439194+0000) 2019-10-13T15:23:38.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:38.438+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.439764+0000) 2019-10-13T15:23:38.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:38.434+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.440208+0000) 2019-10-13T15:23:38.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:38.438+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.441048+0000) 2019-10-13T15:23:38.444 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:38.438+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.441394+0000) 2019-10-13T15:23:38.444 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:38.438+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.441768+0000) 2019-10-13T15:23:38.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:38.442+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.447573+0000) 2019-10-13T15:23:38.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:38.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 218997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:38.853 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:23:38.853 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:38.857 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:23:38.857 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:38.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:38.958+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.961541+0000) 2019-10-13T15:23:39.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:39.510+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:39.513021+0000) 2019-10-13T15:23:42.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:42.697+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:42.700758+0000) 2019-10-13T15:23:43.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:43.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.437208+0000) 2019-10-13T15:23:43.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:43.449+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.452978+0000) 2019-10-13T15:23:43.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:43.449+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.453461+0000) 2019-10-13T15:23:43.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:43.449+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.454377+0000) 2019-10-13T15:23:43.457 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:43.449+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.454736+0000) 2019-10-13T15:23:43.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:43.449+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.455043+0000) 2019-10-13T15:23:43.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:43.450+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.455945+0000) 2019-10-13T15:23:43.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:43.453+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.456268+0000) 2019-10-13T15:23:43.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:43.458+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.462772+0000) 2019-10-13T15:23:43.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:43.461+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:43.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:43.686+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:43.686976+0000) 2019-10-13T15:23:44.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:44.578+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:44.580730+0000) 2019-10-13T15:23:45.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:45.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.711938+0000) 2019-10-13T15:23:45.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:45.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.723517+0000) 2019-10-13T15:23:48.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:48.465+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.468683+0000) 2019-10-13T15:23:48.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:48.465+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.469169+0000) 2019-10-13T15:23:48.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:48.465+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.469391+0000) 2019-10-13T15:23:48.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:48.465+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.470218+0000) 2019-10-13T15:23:48.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:48.465+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.470685+0000) 2019-10-13T15:23:48.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:48.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.471285+0000) 2019-10-13T15:23:48.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:48.465+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.471271+0000) 2019-10-13T15:23:48.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:48.474+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.478852+0000) 2019-10-13T15:23:48.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:48.477+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219062 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:48.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:48.957+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.961718+0000) 2019-10-13T15:23:49.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:49.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:49.513205+0000) 2019-10-13T15:23:52.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:52.697+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:52.700993+0000) 2019-10-13T15:23:53.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:53.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.437440+0000) 2019-10-13T15:23:53.485 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:53.477+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.482498+0000) 2019-10-13T15:23:53.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:53.477+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.483268+0000) 2019-10-13T15:23:53.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:53.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.484364+0000) 2019-10-13T15:23:53.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:53.481+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.484663+0000) 2019-10-13T15:23:53.488 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:53.482+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.485619+0000) 2019-10-13T15:23:53.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:53.481+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.485737+0000) 2019-10-13T15:23:53.489 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:53.481+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.486255+0000) 2019-10-13T15:23:53.493 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:53.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.491434+0000) 2019-10-13T15:23:53.496 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:53.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219094 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:53.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:53.686+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:53.687151+0000) 2019-10-13T15:23:54.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:54.590+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:54.592355+0000) 2019-10-13T15:23:55.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:55.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.712155+0000) 2019-10-13T15:23:55.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:55.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.723720+0000) 2019-10-13T15:23:58.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:23:58.493+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.497306+0000) 2019-10-13T15:23:58.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:23:58.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.498646+0000) 2019-10-13T15:23:58.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:23:58.497+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.499181+0000) 2019-10-13T15:23:58.502 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:58.497+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.499463+0000) 2019-10-13T15:23:58.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:23:58.497+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.499690+0000) 2019-10-13T15:23:58.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:23:58.497+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.500012+0000) 2019-10-13T15:23:58.503 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:58.497+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.501488+0000) 2019-10-13T15:23:58.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:23:58.505+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.507540+0000) 2019-10-13T15:23:58.511 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:23:58.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:23:58.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:23:58.961+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.961873+0000) 2019-10-13T15:23:59.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:23:59.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:59.513355+0000) 2019-10-13T15:24:02.703 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:02.697+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:02.701193+0000) 2019-10-13T15:24:03.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:03.433+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.437641+0000) 2019-10-13T15:24:03.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:03.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.512137+0000) 2019-10-13T15:24:03.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:03.509+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.512490+0000) 2019-10-13T15:24:03.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:03.509+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.512727+0000) 2019-10-13T15:24:03.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:03.509+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.513263+0000) 2019-10-13T15:24:03.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:03.509+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.513582+0000) 2019-10-13T15:24:03.517 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:03.509+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.514456+0000) 2019-10-13T15:24:03.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:03.513+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.515831+0000) 2019-10-13T15:24:03.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:03.517+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.520357+0000) 2019-10-13T15:24:03.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:03.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219160 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:03.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:03.685+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:03.687349+0000) 2019-10-13T15:24:04.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:04.597+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:04.598857+0000) 2019-10-13T15:24:05.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:05.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.712354+0000) 2019-10-13T15:24:05.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:05.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.723915+0000) 2019-10-13T15:24:08.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:08.521+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.526347+0000) 2019-10-13T15:24:08.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:08.521+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.527064+0000) 2019-10-13T15:24:08.530 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:08.525+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.527620+0000) 2019-10-13T15:24:08.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:08.525+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.528015+0000) 2019-10-13T15:24:08.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:08.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.529420+0000) 2019-10-13T15:24:08.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:08.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.529762+0000) 2019-10-13T15:24:08.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:08.525+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.530072+0000) 2019-10-13T15:24:08.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:08.533+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.536977+0000) 2019-10-13T15:24:08.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:08.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219194 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:08.910 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:24:08.910 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:08.914 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:24:08.914 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:08.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:08.957+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.962078+0000) 2019-10-13T15:24:09.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:09.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:09.513574+0000) 2019-10-13T15:24:12.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:12.697+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:12.701393+0000) 2019-10-13T15:24:13.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:13.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.437835+0000) 2019-10-13T15:24:13.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:13.537+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.540337+0000) 2019-10-13T15:24:13.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:13.537+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.540853+0000) 2019-10-13T15:24:13.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:13.537+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.541037+0000) 2019-10-13T15:24:13.544 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:13.537+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.541586+0000) 2019-10-13T15:24:13.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:13.537+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.542388+0000) 2019-10-13T15:24:13.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:13.537+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.543220+0000) 2019-10-13T15:24:13.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:13.541+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.543615+0000) 2019-10-13T15:24:13.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:13.545+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.548312+0000) 2019-10-13T15:24:13.553 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:13.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:13.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:13.685+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:13.687521+0000) 2019-10-13T15:24:14.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:14.601+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:14.605272+0000) 2019-10-13T15:24:15.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:15.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.712552+0000) 2019-10-13T15:24:15.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:15.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.724130+0000) 2019-10-13T15:24:18.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:18.549+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.553958+0000) 2019-10-13T15:24:18.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:18.549+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.554711+0000) 2019-10-13T15:24:18.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:18.549+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.555208+0000) 2019-10-13T15:24:18.558 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:18.553+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.555363+0000) 2019-10-13T15:24:18.558 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:18.553+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.556351+0000) 2019-10-13T15:24:18.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:18.553+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.557021+0000) 2019-10-13T15:24:18.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:18.553+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.557504+0000) 2019-10-13T15:24:18.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:18.557+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.560589+0000) 2019-10-13T15:24:18.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:18.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219262 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:18.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:18.961+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.962295+0000) 2019-10-13T15:24:19.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:19.509+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:19.513764+0000) 2019-10-13T15:24:22.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:22.701+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:22.701622+0000) 2019-10-13T15:24:23.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:23.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.438040+0000) 2019-10-13T15:24:23.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:23.565+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.568443+0000) 2019-10-13T15:24:23.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:23.565+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.569055+0000) 2019-10-13T15:24:23.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:23.565+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.569161+0000) 2019-10-13T15:24:23.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:23.565+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.569501+0000) 2019-10-13T15:24:23.573 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:23.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.570858+0000) 2019-10-13T15:24:23.573 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:23.565+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.570953+0000) 2019-10-13T15:24:23.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:23.569+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.571933+0000) 2019-10-13T15:24:23.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:23.573+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.577050+0000) 2019-10-13T15:24:23.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:23.573+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:23.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:23.685+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:23.687728+0000) 2019-10-13T15:24:24.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:24.609+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:24.611647+0000) 2019-10-13T15:24:25.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:25.709+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.712766+0000) 2019-10-13T15:24:25.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:25.721+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.724353+0000) 2019-10-13T15:24:28.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:28.577+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.581910+0000) 2019-10-13T15:24:28.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:28.576+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.582268+0000) 2019-10-13T15:24:28.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:28.576+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.582593+0000) 2019-10-13T15:24:28.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:28.576+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.583144+0000) 2019-10-13T15:24:28.586 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:28.580+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.583649+0000) 2019-10-13T15:24:28.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:28.580+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.583826+0000) 2019-10-13T15:24:28.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:28.580+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.584288+0000) 2019-10-13T15:24:28.593 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:28.589+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.591348+0000) 2019-10-13T15:24:28.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:28.588+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:28.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:28.960+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.962503+0000) 2019-10-13T15:24:29.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:29.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:29.513948+0000) 2019-10-13T15:24:32.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:32.696+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:32.701823+0000) 2019-10-13T15:24:33.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:33.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.438264+0000) 2019-10-13T15:24:33.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:33.593+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.597418+0000) 2019-10-13T15:24:33.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:33.592+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.597803+0000) 2019-10-13T15:24:33.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:33.592+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.598288+0000) 2019-10-13T15:24:33.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:33.592+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.599229+0000) 2019-10-13T15:24:33.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:33.596+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.600004+0000) 2019-10-13T15:24:33.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:33.596+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.600363+0000) 2019-10-13T15:24:33.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:33.596+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.600730+0000) 2019-10-13T15:24:33.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:33.601+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.604315+0000) 2019-10-13T15:24:33.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:33.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:33.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:33.685+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:33.687904+0000) 2019-10-13T15:24:34.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:34.621+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:34.622863+0000) 2019-10-13T15:24:35.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:35.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.712961+0000) 2019-10-13T15:24:35.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:35.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.724549+0000) 2019-10-13T15:24:38.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:38.608+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.612180+0000) 2019-10-13T15:24:38.615 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:38.608+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.612821+0000) 2019-10-13T15:24:38.616 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:38.608+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.613585+0000) 2019-10-13T15:24:38.616 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:38.613+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.614528+0000) 2019-10-13T15:24:38.617 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:38.608+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.614696+0000) 2019-10-13T15:24:38.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:38.608+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.615172+0000) 2019-10-13T15:24:38.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:38.612+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.615501+0000) 2019-10-13T15:24:38.623 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:38.617+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.620858+0000) 2019-10-13T15:24:38.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:38.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:38.959 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:24:38.960 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:38.963 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:24:38.964 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:38.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:38.960+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.962701+0000) 2019-10-13T15:24:39.516 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:39.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:39.514162+0000) 2019-10-13T15:24:42.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:42.700+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:42.702019+0000) 2019-10-13T15:24:43.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:43.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.438468+0000) 2019-10-13T15:24:43.629 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:43.620+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.627164+0000) 2019-10-13T15:24:43.630 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:43.624+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.627619+0000) 2019-10-13T15:24:43.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:43.624+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.627975+0000) 2019-10-13T15:24:43.631 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:43.624+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.629074+0000) 2019-10-13T15:24:43.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:43.624+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.629631+0000) 2019-10-13T15:24:43.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:43.629+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.630576+0000) 2019-10-13T15:24:43.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:43.624+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.631146+0000) 2019-10-13T15:24:43.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:43.633+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.635201+0000) 2019-10-13T15:24:43.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:43.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219424 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:43.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:43.685+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:43.688081+0000) 2019-10-13T15:24:44.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:44.625+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:44.628688+0000) 2019-10-13T15:24:45.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:45.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.713164+0000) 2019-10-13T15:24:45.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:45.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.724798+0000) 2019-10-13T15:24:48.643 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:48.636+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.641291+0000) 2019-10-13T15:24:48.644 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:48.637+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.641681+0000) 2019-10-13T15:24:48.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:48.636+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.641967+0000) 2019-10-13T15:24:48.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:48.636+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.642880+0000) 2019-10-13T15:24:48.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:48.640+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.643367+0000) 2019-10-13T15:24:48.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:48.640+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.644214+0000) 2019-10-13T15:24:48.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:48.640+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.645024+0000) 2019-10-13T15:24:48.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:48.649+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.650776+0000) 2019-10-13T15:24:48.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:48.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:48.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:48.960+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.962878+0000) 2019-10-13T15:24:49.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:49.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:49.514358+0000) 2019-10-13T15:24:52.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:52.700+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:52.702231+0000) 2019-10-13T15:24:53.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:53.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.438706+0000) 2019-10-13T15:24:53.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:53.648+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.655064+0000) 2019-10-13T15:24:53.658 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:53.652+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.655736+0000) 2019-10-13T15:24:53.658 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:53.652+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.656069+0000) 2019-10-13T15:24:53.660 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:53.652+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.657316+0000) 2019-10-13T15:24:53.660 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:53.652+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.657139+0000) 2019-10-13T15:24:53.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:53.652+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.658125+0000) 2019-10-13T15:24:53.661 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:53.652+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.658856+0000) 2019-10-13T15:24:53.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:53.664+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.667181+0000) 2019-10-13T15:24:53.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:53.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219490 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:53.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:53.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:53.688243+0000) 2019-10-13T15:24:54.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:54.632+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:54.635220+0000) 2019-10-13T15:24:55.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:55.708+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.713373+0000) 2019-10-13T15:24:55.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:55.720+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.724986+0000) 2019-10-13T15:24:58.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:24:58.664+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.671272+0000) 2019-10-13T15:24:58.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:24:58.668+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.671795+0000) 2019-10-13T15:24:58.674 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:24:58.668+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.671988+0000) 2019-10-13T15:24:58.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:24:58.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.672381+0000) 2019-10-13T15:24:58.675 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:24:58.668+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.672919+0000) 2019-10-13T15:24:58.676 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:58.668+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.673441+0000) 2019-10-13T15:24:58.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:58.668+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.673582+0000) 2019-10-13T15:24:58.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:24:58.672+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.677531+0000) 2019-10-13T15:24:58.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:24:58.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219526 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:24:58.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:24:58.960+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.963065+0000) 2019-10-13T15:24:59.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:24:59.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:59.514604+0000) 2019-10-13T15:25:02.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:02.700+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:02.702460+0000) 2019-10-13T15:25:03.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:03.432+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.438887+0000) 2019-10-13T15:25:03.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:03.680+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.685112+0000) 2019-10-13T15:25:03.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:03.680+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.685614+0000) 2019-10-13T15:25:03.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:03.684+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.686266+0000) 2019-10-13T15:25:03.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:03.680+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.686487+0000) 2019-10-13T15:25:03.690 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:03.684+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.687616+0000) 2019-10-13T15:25:03.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:03.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.688372+0000) 2019-10-13T15:25:03.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:03.684+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.688365+0000) 2019-10-13T15:25:03.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:03.684+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.688492+0000) 2019-10-13T15:25:03.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:03.688+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:03.693587+0000) 2019-10-13T15:25:03.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:03.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219556 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:04.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:04.640+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:04.641568+0000) 2019-10-13T15:25:05.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:05.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.713568+0000) 2019-10-13T15:25:05.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:05.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.725180+0000) 2019-10-13T15:25:08.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:08.696+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.700954+0000) 2019-10-13T15:25:08.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:08.696+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.701441+0000) 2019-10-13T15:25:08.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:08.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.702123+0000) 2019-10-13T15:25:08.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:08.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.702327+0000) 2019-10-13T15:25:08.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:08.696+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.702492+0000) 2019-10-13T15:25:08.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:08.700+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.703481+0000) 2019-10-13T15:25:08.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:08.700+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.704528+0000) 2019-10-13T15:25:08.712 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:08.708+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.710525+0000) 2019-10-13T15:25:08.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:08.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219591 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:08.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:08.960+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.963237+0000) 2019-10-13T15:25:09.077 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:25:09.078 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:09.081 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:25:09.081 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:09.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:09.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:09.514805+0000) 2019-10-13T15:25:12.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:12.699+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:12.702665+0000) 2019-10-13T15:25:13.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:13.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.439082+0000) 2019-10-13T15:25:13.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:13.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.688536+0000) 2019-10-13T15:25:13.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:13.712+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.715319+0000) 2019-10-13T15:25:13.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:13.711+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.716201+0000) 2019-10-13T15:25:13.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:13.711+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.715955+0000) 2019-10-13T15:25:13.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:13.711+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.716431+0000) 2019-10-13T15:25:13.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:13.711+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.716856+0000) 2019-10-13T15:25:13.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:13.711+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.717131+0000) 2019-10-13T15:25:13.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:13.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.718933+0000) 2019-10-13T15:25:13.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:13.720+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:13.725021+0000) 2019-10-13T15:25:13.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:13.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219623 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:14.650 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:14.644+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:14.648019+0000) 2019-10-13T15:25:15.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:15.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.713765+0000) 2019-10-13T15:25:15.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:15.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.725380+0000) 2019-10-13T15:25:18.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:18.723+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.728966+0000) 2019-10-13T15:25:18.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:18.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.729759+0000) 2019-10-13T15:25:18.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:18.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.730276+0000) 2019-10-13T15:25:18.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:18.723+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.731053+0000) 2019-10-13T15:25:18.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:18.727+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.731634+0000) 2019-10-13T15:25:18.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:18.727+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.731511+0000) 2019-10-13T15:25:18.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:18.727+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.732382+0000) 2019-10-13T15:25:18.742 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:18.736+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.739763+0000) 2019-10-13T15:25:18.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:18.735+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219657 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:18.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:18.959+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.963430+0000) 2019-10-13T15:25:19.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:19.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:19.514991+0000) 2019-10-13T15:25:22.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:22.699+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:22.702812+0000) 2019-10-13T15:25:23.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:23.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.439282+0000) 2019-10-13T15:25:23.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:23.688+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.688731+0000) 2019-10-13T15:25:23.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:23.739+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.744157+0000) 2019-10-13T15:25:23.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:23.739+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.744629+0000) 2019-10-13T15:25:23.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:23.739+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.744770+0000) 2019-10-13T15:25:23.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:23.740+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.745501+0000) 2019-10-13T15:25:23.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:23.739+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.746532+0000) 2019-10-13T15:25:23.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:23.739+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.747167+0000) 2019-10-13T15:25:23.750 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:23.743+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.747730+0000) 2019-10-13T15:25:23.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:23.752+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:23.754382+0000) 2019-10-13T15:25:23.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:23.751+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:24.656 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:24.652+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:24.654199+0000) 2019-10-13T15:25:25.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:25.707+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.713997+0000) 2019-10-13T15:25:25.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:25.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.725584+0000) 2019-10-13T15:25:28.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:28.755+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.758801+0000) 2019-10-13T15:25:28.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:28.755+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.759536+0000) 2019-10-13T15:25:28.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:28.755+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.760010+0000) 2019-10-13T15:25:28.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:28.755+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.760704+0000) 2019-10-13T15:25:28.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:28.755+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.761819+0000) 2019-10-13T15:25:28.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:28.760+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.762284+0000) 2019-10-13T15:25:28.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:28.755+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.762954+0000) 2019-10-13T15:25:28.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:28.764+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.769090+0000) 2019-10-13T15:25:28.772 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:28.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:28.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:28.963+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.963609+0000) 2019-10-13T15:25:29.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:29.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:29.515176+0000) 2019-10-13T15:25:32.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:32.699+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:32.703045+0000) 2019-10-13T15:25:33.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:33.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.439505+0000) 2019-10-13T15:25:33.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:33.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.688943+0000) 2019-10-13T15:25:33.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:33.767+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.773352+0000) 2019-10-13T15:25:33.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:33.767+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.774417+0000) 2019-10-13T15:25:33.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:33.771+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.775350+0000) 2019-10-13T15:25:33.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:33.771+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.775789+0000) 2019-10-13T15:25:33.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:33.772+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.776639+0000) 2019-10-13T15:25:33.779 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:33.771+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.776257+0000) 2019-10-13T15:25:33.779 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:33.771+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.777240+0000) 2019-10-13T15:25:33.785 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:33.780+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:33.782954+0000) 2019-10-13T15:25:33.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:33.779+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219755 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:34.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:34.660+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:34.660563+0000) 2019-10-13T15:25:35.716 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:35.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.714194+0000) 2019-10-13T15:25:35.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:35.719+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.725794+0000) 2019-10-13T15:25:38.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:38.783+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.788716+0000) 2019-10-13T15:25:38.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:38.783+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.789155+0000) 2019-10-13T15:25:38.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:38.783+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.790430+0000) 2019-10-13T15:25:38.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:38.783+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.790044+0000) 2019-10-13T15:25:38.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:38.783+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.790749+0000) 2019-10-13T15:25:38.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:38.788+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.791719+0000) 2019-10-13T15:25:38.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:38.787+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.792088+0000) 2019-10-13T15:25:38.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:38.792+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.797573+0000) 2019-10-13T15:25:38.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:38.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:38.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:38.963+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.963791+0000) 2019-10-13T15:25:39.206 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:25:39.206 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:39.209 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:25:39.209 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:39.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:39.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:39.515373+0000) 2019-10-13T15:25:42.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:42.703+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:42.703275+0000) 2019-10-13T15:25:43.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:43.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.439675+0000) 2019-10-13T15:25:43.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:43.684+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.689172+0000) 2019-10-13T15:25:43.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:43.799+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.802840+0000) 2019-10-13T15:25:43.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:43.803+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.803629+0000) 2019-10-13T15:25:43.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:43.803+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.804361+0000) 2019-10-13T15:25:43.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:43.803+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.804549+0000) 2019-10-13T15:25:43.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:43.803+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.805624+0000) 2019-10-13T15:25:43.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:43.803+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.806520+0000) 2019-10-13T15:25:43.809 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:43.804+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.807259+0000) 2019-10-13T15:25:43.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:43.808+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:43.811699+0000) 2019-10-13T15:25:43.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:43.815+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219821 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:44.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:44.664+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:44.666982+0000) 2019-10-13T15:25:45.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:45.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.714422+0000) 2019-10-13T15:25:45.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:45.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.725999+0000) 2019-10-13T15:25:48.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:48.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.818589+0000) 2019-10-13T15:25:48.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:48.819+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.820180+0000) 2019-10-13T15:25:48.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:48.819+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.820465+0000) 2019-10-13T15:25:48.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:48.819+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.821074+0000) 2019-10-13T15:25:48.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:48.815+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.822056+0000) 2019-10-13T15:25:48.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:48.819+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.821713+0000) 2019-10-13T15:25:48.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:48.819+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.822572+0000) 2019-10-13T15:25:48.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:48.823+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.828038+0000) 2019-10-13T15:25:48.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:48.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:48.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:48.963+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.963970+0000) 2019-10-13T15:25:49.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:49.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:49.515572+0000) 2019-10-13T15:25:52.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:52.703+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:52.703531+0000) 2019-10-13T15:25:53.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:53.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.439860+0000) 2019-10-13T15:25:53.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:53.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.689364+0000) 2019-10-13T15:25:53.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:53.830+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.832989+0000) 2019-10-13T15:25:53.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:53.830+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.833808+0000) 2019-10-13T15:25:53.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:53.830+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.834732+0000) 2019-10-13T15:25:53.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:53.834+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.835360+0000) 2019-10-13T15:25:53.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:53.834+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.836270+0000) 2019-10-13T15:25:53.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:53.834+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.836545+0000) 2019-10-13T15:25:53.839 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:53.831+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.836967+0000) 2019-10-13T15:25:53.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:53.835+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:53.840154+0000) 2019-10-13T15:25:53.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:53.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219889 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:54.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:54.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:54.673338+0000) 2019-10-13T15:25:55.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:55.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:55.714620+0000) 2019-10-13T15:25:55.728 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:55.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:55.726192+0000) 2019-10-13T15:25:58.851 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:58.846+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.848753+0000) 2019-10-13T15:25:58.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:25:58.846+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.849040+0000) 2019-10-13T15:25:58.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:58.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.849941+0000) 2019-10-13T15:25:58.852 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:25:58.846+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.850081+0000) 2019-10-13T15:25:58.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:25:58.846+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.850383+0000) 2019-10-13T15:25:58.853 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:25:58.846+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.850979+0000) 2019-10-13T15:25:58.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:25:58.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.852164+0000) 2019-10-13T15:25:58.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:25:58.851+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.855710+0000) 2019-10-13T15:25:58.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:25:58.854+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:25:58.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:25:58.962+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.964158+0000) 2019-10-13T15:25:59.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:25:59.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:59.515759+0000) 2019-10-13T15:26:02.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:02.702+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:02.703730+0000) 2019-10-13T15:26:03.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:03.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.440050+0000) 2019-10-13T15:26:03.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:03.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.689562+0000) 2019-10-13T15:26:03.864 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:03.858+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.861300+0000) 2019-10-13T15:26:03.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:03.858+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.862058+0000) 2019-10-13T15:26:03.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:03.858+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.862532+0000) 2019-10-13T15:26:03.866 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:03.859+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.863660+0000) 2019-10-13T15:26:03.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:03.862+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.863855+0000) 2019-10-13T15:26:03.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:03.862+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.864465+0000) 2019-10-13T15:26:03.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:03.862+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.865430+0000) 2019-10-13T15:26:03.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:03.863+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:03.869887+0000) 2019-10-13T15:26:03.876 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:03.870+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219955 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:04.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:04.675+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:04.679699+0000) 2019-10-13T15:26:05.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:05.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:05.714849+0000) 2019-10-13T15:26:05.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:05.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:05.726425+0000) 2019-10-13T15:26:08.878 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:08.871+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.875610+0000) 2019-10-13T15:26:08.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:08.874+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.876716+0000) 2019-10-13T15:26:08.880 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:08.874+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.877133+0000) 2019-10-13T15:26:08.880 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:08.874+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.877464+0000) 2019-10-13T15:26:08.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:08.874+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.877888+0000) 2019-10-13T15:26:08.881 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:08.874+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.878185+0000) 2019-10-13T15:26:08.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:08.874+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.878666+0000) 2019-10-13T15:26:08.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:08.875+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.881844+0000) 2019-10-13T15:26:08.889 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:08.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 219988 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:08.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:08.962+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.964334+0000) 2019-10-13T15:26:09.317 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:26:09.317 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:09.320 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:26:09.321 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:09.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:09.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:09.515949+0000) 2019-10-13T15:26:12.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:12.702+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:12.703933+0000) 2019-10-13T15:26:13.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:13.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.440251+0000) 2019-10-13T15:26:13.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:13.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.689777+0000) 2019-10-13T15:26:13.892 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:13.883+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.889952+0000) 2019-10-13T15:26:13.893 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:13.886+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.890405+0000) 2019-10-13T15:26:13.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:13.886+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.891227+0000) 2019-10-13T15:26:13.894 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:13.890+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.891438+0000) 2019-10-13T15:26:13.895 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:13.890+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.892793+0000) 2019-10-13T15:26:13.895 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:13.890+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.893003+0000) 2019-10-13T15:26:13.896 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:13.890+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.893646+0000) 2019-10-13T15:26:13.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:13.895+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:13.898580+0000) 2019-10-13T15:26:13.904 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:13.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220022 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:14.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:14.683+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:14.686460+0000) 2019-10-13T15:26:15.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:15.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:15.715053+0000) 2019-10-13T15:26:15.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:15.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:15.726647+0000) 2019-10-13T15:26:18.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:18.902+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.905221+0000) 2019-10-13T15:26:18.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:18.902+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.905738+0000) 2019-10-13T15:26:18.909 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:18.902+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.907063+0000) 2019-10-13T15:26:18.910 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:18.902+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.907100+0000) 2019-10-13T15:26:18.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:18.906+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.907917+0000) 2019-10-13T15:26:18.911 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:18.903+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.908911+0000) 2019-10-13T15:26:18.912 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:18.906+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.909673+0000) 2019-10-13T15:26:18.918 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:18.911+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.915824+0000) 2019-10-13T15:26:18.919 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:18.914+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:18.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:18.962+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.964457+0000) 2019-10-13T15:26:19.518 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:19.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:19.516124+0000) 2019-10-13T15:26:22.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:22.702+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:22.704152+0000) 2019-10-13T15:26:23.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:23.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.440481+0000) 2019-10-13T15:26:23.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:23.683+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.690006+0000) 2019-10-13T15:26:23.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:23.915+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.921443+0000) 2019-10-13T15:26:23.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:23.918+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.920256+0000) 2019-10-13T15:26:23.929 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:23.918+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.920673+0000) 2019-10-13T15:26:23.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:23.918+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.920975+0000) 2019-10-13T15:26:23.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:23.918+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.922576+0000) 2019-10-13T15:26:23.929 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:23.918+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.922696+0000) 2019-10-13T15:26:23.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:23.922+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.923277+0000) 2019-10-13T15:26:23.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:23.923+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:23.928270+0000) 2019-10-13T15:26:23.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:23.930+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:24.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:24.687+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:24.692873+0000) 2019-10-13T15:26:25.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:25.710+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:25.715283+0000) 2019-10-13T15:26:25.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:25.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:25.726843+0000) 2019-10-13T15:26:28.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:28.930+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.934979+0000) 2019-10-13T15:26:28.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:28.934+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.935597+0000) 2019-10-13T15:26:28.938 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:28.934+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.936012+0000) 2019-10-13T15:26:28.939 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:28.931+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.936658+0000) 2019-10-13T15:26:28.939 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:28.934+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.936973+0000) 2019-10-13T15:26:28.940 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:28.934+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.937428+0000) 2019-10-13T15:26:28.940 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:28.934+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.937825+0000) 2019-10-13T15:26:28.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:28.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220120 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:28.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:28.943+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.947339+0000) 2019-10-13T15:26:28.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:28.962+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.964563+0000) 2019-10-13T15:26:29.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:29.514+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:29.516349+0000) 2019-10-13T15:26:32.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:32.702+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:32.704376+0000) 2019-10-13T15:26:33.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:33.438+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.440674+0000) 2019-10-13T15:26:33.692 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:33.687+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.690212+0000) 2019-10-13T15:26:33.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:33.946+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.949169+0000) 2019-10-13T15:26:33.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:33.946+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.949878+0000) 2019-10-13T15:26:33.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:33.946+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.950579+0000) 2019-10-13T15:26:33.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:33.947+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.951309+0000) 2019-10-13T15:26:33.954 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:33.950+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.951778+0000) 2019-10-13T15:26:33.954 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:33.950+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.952136+0000) 2019-10-13T15:26:33.955 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:33.950+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.952912+0000) 2019-10-13T15:26:33.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:33.951+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:33.958056+0000) 2019-10-13T15:26:33.967 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:33.958+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220154 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:34.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:34.699+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:34.699310+0000) 2019-10-13T15:26:35.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:35.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:35.715546+0000) 2019-10-13T15:26:35.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:35.722+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:35.727044+0000) 2019-10-13T15:26:38.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:38.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.964270+0000) 2019-10-13T15:26:38.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:38.961+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.964734+0000) 2019-10-13T15:26:38.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:38.959+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.965248+0000) 2019-10-13T15:26:38.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:38.961+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.965850+0000) 2019-10-13T15:26:38.969 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:38.965+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.967485+0000) 2019-10-13T15:26:38.970 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:38.965+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.968064+0000) 2019-10-13T15:26:38.971 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:38.965+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.968800+0000) 2019-10-13T15:26:38.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:38.967+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.972809+0000) 2019-10-13T15:26:38.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:38.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220187 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:39.429 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:26:39.429 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:39.433 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:26:39.433 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:39.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:39.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:39.516572+0000) 2019-10-13T15:26:42.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:42.701+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:42.704581+0000) 2019-10-13T15:26:43.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:43.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.440899+0000) 2019-10-13T15:26:43.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:43.686+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.690426+0000) 2019-10-13T15:26:43.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:43.973+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.978592+0000) 2019-10-13T15:26:43.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:43.973+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.979244+0000) 2019-10-13T15:26:43.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:43.977+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.979872+0000) 2019-10-13T15:26:43.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:43.977+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.980656+0000) 2019-10-13T15:26:43.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:43.977+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.980669+0000) 2019-10-13T15:26:43.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:43.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.981281+0000) 2019-10-13T15:26:43.984 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:43.977+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.981823+0000) 2019-10-13T15:26:43.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:43.978+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:43.984683+0000) 2019-10-13T15:26:43.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:43.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:44.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:44.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:44.705511+0000) 2019-10-13T15:26:45.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:45.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:45.715212+0000) 2019-10-13T15:26:45.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:45.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:45.715699+0000) 2019-10-13T15:26:45.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:45.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:45.727279+0000) 2019-10-13T15:26:48.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:48.989+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.993569+0000) 2019-10-13T15:26:48.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:48.989+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.992915+0000) 2019-10-13T15:26:48.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:48.989+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.994597+0000) 2019-10-13T15:26:48.997 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:48.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.995070+0000) 2019-10-13T15:26:48.998 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:48.993+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.995416+0000) 2019-10-13T15:26:48.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:48.993+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.996313+0000) 2019-10-13T15:26:48.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:48.993+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.996655+0000) 2019-10-13T15:26:49.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:48.998+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:49.003808+0000) 2019-10-13T15:26:49.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:49.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220251 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:49.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:49.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:49.516777+0000) 2019-10-13T15:26:52.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:52.701+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:52.704804+0000) 2019-10-13T15:26:53.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:53.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:53.441126+0000) 2019-10-13T15:26:53.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:53.686+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:53.690698+0000) 2019-10-13T15:26:54.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:54.005+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.007814+0000) 2019-10-13T15:26:54.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:54.005+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.008073+0000) 2019-10-13T15:26:54.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:54.005+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.008814+0000) 2019-10-13T15:26:54.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:54.002+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.009440+0000) 2019-10-13T15:26:54.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:54.005+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.009628+0000) 2019-10-13T15:26:54.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:54.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.010808+0000) 2019-10-13T15:26:54.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:54.009+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.011334+0000) 2019-10-13T15:26:54.023 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:54.014+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.020669+0000) 2019-10-13T15:26:54.023 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:54.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:54.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:54.710+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.711902+0000) 2019-10-13T15:26:55.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:55.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:55.715434+0000) 2019-10-13T15:26:55.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:55.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:55.715860+0000) 2019-10-13T15:26:55.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:55.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:55.727505+0000) 2019-10-13T15:26:59.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:26:59.021+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.024052+0000) 2019-10-13T15:26:59.027 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:59.021+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.024771+0000) 2019-10-13T15:26:59.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:26:59.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.025065+0000) 2019-10-13T15:26:59.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:26:59.021+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.025435+0000) 2019-10-13T15:26:59.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:26:59.021+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.025973+0000) 2019-10-13T15:26:59.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:26:59.021+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.026843+0000) 2019-10-13T15:26:59.031 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:26:59.022+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.027402+0000) 2019-10-13T15:26:59.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:26:59.026+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.033181+0000) 2019-10-13T15:26:59.038 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:26:59.033+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220317 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:26:59.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:26:59.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:59.516990+0000) 2019-10-13T15:27:02.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:02.701+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:02.705021+0000) 2019-10-13T15:27:03.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:03.437+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:03.441338+0000) 2019-10-13T15:27:03.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:03.690+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:03.690899+0000) 2019-10-13T15:27:04.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:04.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.038544+0000) 2019-10-13T15:27:04.042 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:04.037+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.040157+0000) 2019-10-13T15:27:04.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:04.037+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.040762+0000) 2019-10-13T15:27:04.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:04.037+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.040486+0000) 2019-10-13T15:27:04.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:04.034+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.041252+0000) 2019-10-13T15:27:04.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:04.037+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.041679+0000) 2019-10-13T15:27:04.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:04.037+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.042063+0000) 2019-10-13T15:27:04.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:04.042+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.049388+0000) 2019-10-13T15:27:04.052 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:04.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:04.720 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:04.714+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.718273+0000) 2019-10-13T15:27:05.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:05.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:05.715653+0000) 2019-10-13T15:27:05.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:05.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:05.716017+0000) 2019-10-13T15:27:05.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:05.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:05.727697+0000) 2019-10-13T15:27:09.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:09.049+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.052957+0000) 2019-10-13T15:27:09.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:09.049+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.053959+0000) 2019-10-13T15:27:09.056 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:09.049+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.054136+0000) 2019-10-13T15:27:09.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:09.049+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.054831+0000) 2019-10-13T15:27:09.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:09.050+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.055861+0000) 2019-10-13T15:27:09.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:09.053+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.055901+0000) 2019-10-13T15:27:09.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:09.053+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.056078+0000) 2019-10-13T15:27:09.065 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:09.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:09.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:09.058+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.064454+0000) 2019-10-13T15:27:09.519 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:09.513+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:09.517180+0000) 2019-10-13T15:27:09.555 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:27:09.555 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:09.559 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:27:09.559 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:12.707 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:12.701+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:12.705252+0000) 2019-10-13T15:27:13.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:13.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:13.441532+0000) 2019-10-13T15:27:13.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:13.690+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:13.691094+0000) 2019-10-13T15:27:14.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:14.061+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.066579+0000) 2019-10-13T15:27:14.073 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:14.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.067378+0000) 2019-10-13T15:27:14.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:14.065+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.068129+0000) 2019-10-13T15:27:14.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:14.065+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.068700+0000) 2019-10-13T15:27:14.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:14.065+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.069219+0000) 2019-10-13T15:27:14.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:14.065+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.069775+0000) 2019-10-13T15:27:14.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:14.066+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.070313+0000) 2019-10-13T15:27:14.080 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:14.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:14.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:14.070+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.077583+0000) 2019-10-13T15:27:14.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:14.718+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.724883+0000) 2019-10-13T15:27:15.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:15.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:15.715881+0000) 2019-10-13T15:27:15.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:15.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:15.716182+0000) 2019-10-13T15:27:15.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:15.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:15.727890+0000) 2019-10-13T15:27:19.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:19.077+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.080919+0000) 2019-10-13T15:27:19.084 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:19.077+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.081006+0000) 2019-10-13T15:27:19.084 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:19.077+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.081263+0000) 2019-10-13T15:27:19.085 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:19.077+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.083146+0000) 2019-10-13T15:27:19.086 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:19.078+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.083640+0000) 2019-10-13T15:27:19.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:19.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.084146+0000) 2019-10-13T15:27:19.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:19.081+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.084399+0000) 2019-10-13T15:27:19.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:19.086+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.092605+0000) 2019-10-13T15:27:19.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:19.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:19.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:19.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:19.517362+0000) 2019-10-13T15:27:22.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:22.704+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:22.705483+0000) 2019-10-13T15:27:23.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:23.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:23.441750+0000) 2019-10-13T15:27:23.693 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:23.690+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:23.691293+0000) 2019-10-13T15:27:24.098 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:24.092+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.095717+0000) 2019-10-13T15:27:24.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:24.092+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.096442+0000) 2019-10-13T15:27:24.099 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:24.092+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.096974+0000) 2019-10-13T15:27:24.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:24.092+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.097359+0000) 2019-10-13T15:27:24.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:24.094+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.098247+0000) 2019-10-13T15:27:24.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:24.092+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.097756+0000) 2019-10-13T15:27:24.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:24.092+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.098745+0000) 2019-10-13T15:27:24.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:24.098+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.105182+0000) 2019-10-13T15:27:24.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:24.104+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220481 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:24.733 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:24.730+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.730967+0000) 2019-10-13T15:27:25.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:25.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:25.716112+0000) 2019-10-13T15:27:25.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:25.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:25.716371+0000) 2019-10-13T15:27:25.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:25.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:25.728101+0000) 2019-10-13T15:27:29.113 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:29.104+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.110832+0000) 2019-10-13T15:27:29.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:29.104+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.111248+0000) 2019-10-13T15:27:29.114 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:29.108+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.111796+0000) 2019-10-13T15:27:29.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:29.106+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.112569+0000) 2019-10-13T15:27:29.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:29.108+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.112795+0000) 2019-10-13T15:27:29.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:29.108+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.113167+0000) 2019-10-13T15:27:29.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:29.108+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.113564+0000) 2019-10-13T15:27:29.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:29.114+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.121454+0000) 2019-10-13T15:27:29.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:29.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220514 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:29.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:29.516+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:29.517545+0000) 2019-10-13T15:27:32.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:32.704+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:32.705688+0000) 2019-10-13T15:27:33.444 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:33.436+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:33.441946+0000) 2019-10-13T15:27:33.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:33.690+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:33.691517+0000) 2019-10-13T15:27:34.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:34.120+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.125664+0000) 2019-10-13T15:27:34.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:34.122+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.126034+0000) 2019-10-13T15:27:34.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:34.120+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.127052+0000) 2019-10-13T15:27:34.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:34.120+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.127118+0000) 2019-10-13T15:27:34.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:34.124+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.127605+0000) 2019-10-13T15:27:34.131 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:34.124+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.128185+0000) 2019-10-13T15:27:34.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:34.124+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.128631+0000) 2019-10-13T15:27:34.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:34.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:34.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:34.134+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.136341+0000) 2019-10-13T15:27:34.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:34.734+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.737316+0000) 2019-10-13T15:27:35.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:35.712+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:35.716329+0000) 2019-10-13T15:27:35.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:35.712+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:35.716565+0000) 2019-10-13T15:27:35.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:35.724+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:35.728297+0000) 2019-10-13T15:27:39.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:39.137+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.140582+0000) 2019-10-13T15:27:39.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:39.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.140728+0000) 2019-10-13T15:27:39.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:39.136+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.141089+0000) 2019-10-13T15:27:39.144 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:39.136+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.141503+0000) 2019-10-13T15:27:39.145 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:39.136+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.142700+0000) 2019-10-13T15:27:39.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:39.140+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.143828+0000) 2019-10-13T15:27:39.146 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:39.140+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.144093+0000) 2019-10-13T15:27:39.153 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:39.149+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.151211+0000) 2019-10-13T15:27:39.154 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:39.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220580 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:39.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:39.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:39.517749+0000) 2019-10-13T15:27:39.618 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:27:39.618 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:39.622 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:27:39.622 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:42.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:42.700+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:42.705846+0000) 2019-10-13T15:27:43.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:43.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:43.442148+0000) 2019-10-13T15:27:43.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:43.689+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:43.691711+0000) 2019-10-13T15:27:44.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:44.152+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.155498+0000) 2019-10-13T15:27:44.158 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:44.152+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.155945+0000) 2019-10-13T15:27:44.159 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:44.152+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.156264+0000) 2019-10-13T15:27:44.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:44.152+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.157207+0000) 2019-10-13T15:27:44.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:44.153+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.157882+0000) 2019-10-13T15:27:44.160 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:44.152+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.158084+0000) 2019-10-13T15:27:44.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:44.156+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.159528+0000) 2019-10-13T15:27:44.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:44.161+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.163418+0000) 2019-10-13T15:27:44.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:44.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220612 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:44.747 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:44.741+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.743652+0000) 2019-10-13T15:27:45.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:45.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:45.716525+0000) 2019-10-13T15:27:45.719 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:45.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:45.716752+0000) 2019-10-13T15:27:45.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:45.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:45.728523+0000) 2019-10-13T15:27:49.172 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:49.164+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.169212+0000) 2019-10-13T15:27:49.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:49.164+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.169526+0000) 2019-10-13T15:27:49.173 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:49.164+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.170738+0000) 2019-10-13T15:27:49.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:49.164+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.171195+0000) 2019-10-13T15:27:49.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:49.168+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.171594+0000) 2019-10-13T15:27:49.175 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:49.168+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.171922+0000) 2019-10-13T15:27:49.175 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:49.169+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.172524+0000) 2019-10-13T15:27:49.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:49.173+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.177741+0000) 2019-10-13T15:27:49.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:49.176+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:49.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:49.516+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:49.517929+0000) 2019-10-13T15:27:52.708 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:52.704+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:52.706039+0000) 2019-10-13T15:27:53.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:53.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:53.442342+0000) 2019-10-13T15:27:53.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:53.689+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:53.691911+0000) 2019-10-13T15:27:54.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:54.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.183979+0000) 2019-10-13T15:27:54.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:54.180+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.185459+0000) 2019-10-13T15:27:54.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:54.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.185756+0000) 2019-10-13T15:27:54.189 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:54.185+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.186674+0000) 2019-10-13T15:27:54.189 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:54.180+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.186517+0000) 2019-10-13T15:27:54.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:54.184+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.187667+0000) 2019-10-13T15:27:54.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:54.184+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.187979+0000) 2019-10-13T15:27:54.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:54.189+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.191453+0000) 2019-10-13T15:27:54.198 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:54.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:54.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:54.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.750047+0000) 2019-10-13T15:27:55.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:55.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:55.716726+0000) 2019-10-13T15:27:55.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:55.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:55.716948+0000) 2019-10-13T15:27:55.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:55.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:55.728744+0000) 2019-10-13T15:27:59.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:27:59.192+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.198882+0000) 2019-10-13T15:27:59.202 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:59.196+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.200050+0000) 2019-10-13T15:27:59.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:27:59.196+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.200216+0000) 2019-10-13T15:27:59.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:27:59.196+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.200783+0000) 2019-10-13T15:27:59.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:27:59.196+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.201152+0000) 2019-10-13T15:27:59.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:27:59.197+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.201713+0000) 2019-10-13T15:27:59.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:27:59.196+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.202653+0000) 2019-10-13T15:27:59.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:27:59.205+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.208468+0000) 2019-10-13T15:27:59.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:27:59.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:27:59.520 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:27:59.512+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:59.518108+0000) 2019-10-13T15:28:02.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:02.704+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:02.706258+0000) 2019-10-13T15:28:03.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:03.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:03.442552+0000) 2019-10-13T15:28:03.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:03.689+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:03.692105+0000) 2019-10-13T15:28:04.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:04.207+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.213955+0000) 2019-10-13T15:28:04.217 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:04.207+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.214509+0000) 2019-10-13T15:28:04.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:04.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.215500+0000) 2019-10-13T15:28:04.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:04.211+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.216047+0000) 2019-10-13T15:28:04.219 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:04.211+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.216479+0000) 2019-10-13T15:28:04.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:04.211+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.217277+0000) 2019-10-13T15:28:04.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:04.211+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.217976+0000) 2019-10-13T15:28:04.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:04.217+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.221446+0000) 2019-10-13T15:28:04.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:04.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220744 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:04.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:04.757+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.761410+0000) 2019-10-13T15:28:05.719 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:05.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:05.716927+0000) 2019-10-13T15:28:05.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:05.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:05.717152+0000) 2019-10-13T15:28:05.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:05.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:05.728941+0000) 2019-10-13T15:28:09.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:09.223+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.228851+0000) 2019-10-13T15:28:09.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:09.223+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.228586+0000) 2019-10-13T15:28:09.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:09.223+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.229474+0000) 2019-10-13T15:28:09.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:09.223+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.229764+0000) 2019-10-13T15:28:09.233 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:09.229+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.230388+0000) 2019-10-13T15:28:09.233 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:09.223+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.231077+0000) 2019-10-13T15:28:09.234 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:09.227+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.231628+0000) 2019-10-13T15:28:09.241 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:09.237+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.238609+0000) 2019-10-13T15:28:09.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:09.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220778 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:09.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:09.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:09.518303+0000) 2019-10-13T15:28:09.742 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:28:09.742 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:09.746 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:28:09.746 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:12.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:12.699+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:12.706454+0000) 2019-10-13T15:28:13.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:13.435+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:13.442791+0000) 2019-10-13T15:28:13.694 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:13.689+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:13.692311+0000) 2019-10-13T15:28:14.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:14.235+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.242574+0000) 2019-10-13T15:28:14.245 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:14.239+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.243415+0000) 2019-10-13T15:28:14.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:14.241+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.244225+0000) 2019-10-13T15:28:14.246 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:14.239+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.244235+0000) 2019-10-13T15:28:14.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:14.239+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.244726+0000) 2019-10-13T15:28:14.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:14.239+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.245164+0000) 2019-10-13T15:28:14.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:14.239+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.245693+0000) 2019-10-13T15:28:14.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:14.249+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.253276+0000) 2019-10-13T15:28:14.256 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:14.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:14.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:14.765+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.767823+0000) 2019-10-13T15:28:15.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:15.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:15.717128+0000) 2019-10-13T15:28:15.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:15.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:15.717383+0000) 2019-10-13T15:28:15.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:15.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:15.729165+0000) 2019-10-13T15:28:19.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:19.251+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.256166+0000) 2019-10-13T15:28:19.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:19.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.256978+0000) 2019-10-13T15:28:19.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:19.251+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.257952+0000) 2019-10-13T15:28:19.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:19.251+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.258711+0000) 2019-10-13T15:28:19.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:19.257+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.259158+0000) 2019-10-13T15:28:19.262 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:19.255+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.259557+0000) 2019-10-13T15:28:19.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:19.255+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.259564+0000) 2019-10-13T15:28:19.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:19.261+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.263714+0000) 2019-10-13T15:28:19.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:19.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220844 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:19.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:19.511+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:19.518518+0000) 2019-10-13T15:28:22.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:22.703+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:22.706648+0000) 2019-10-13T15:28:23.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:23.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:23.442983+0000) 2019-10-13T15:28:23.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:23.689+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:23.692508+0000) 2019-10-13T15:28:24.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:24.263+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.270404+0000) 2019-10-13T15:28:24.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:24.263+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.271123+0000) 2019-10-13T15:28:24.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:24.269+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.272190+0000) 2019-10-13T15:28:24.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:24.267+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.272650+0000) 2019-10-13T15:28:24.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:24.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.273052+0000) 2019-10-13T15:28:24.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:24.267+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.273405+0000) 2019-10-13T15:28:24.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:24.267+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.273525+0000) 2019-10-13T15:28:24.283 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:24.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:24.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:24.281+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.282856+0000) 2019-10-13T15:28:24.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:24.773+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.774268+0000) 2019-10-13T15:28:25.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:25.711+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:25.717327+0000) 2019-10-13T15:28:25.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:25.711+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:25.717578+0000) 2019-10-13T15:28:25.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:25.723+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:25.729359+0000) 2019-10-13T15:28:29.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:29.279+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.284047+0000) 2019-10-13T15:28:29.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:29.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.284433+0000) 2019-10-13T15:28:29.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:29.281+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.285330+0000) 2019-10-13T15:28:29.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:29.279+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.285704+0000) 2019-10-13T15:28:29.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:29.279+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.286411+0000) 2019-10-13T15:28:29.293 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:29.279+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.286424+0000) 2019-10-13T15:28:29.294 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:29.279+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.287157+0000) 2019-10-13T15:28:29.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:29.289+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.291316+0000) 2019-10-13T15:28:29.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:29.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220910 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:29.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:29.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:29.518725+0000) 2019-10-13T15:28:32.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:32.703+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:32.706850+0000) 2019-10-13T15:28:33.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:33.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:33.443184+0000) 2019-10-13T15:28:33.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:33.688+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:33.692699+0000) 2019-10-13T15:28:34.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:34.295+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.298453+0000) 2019-10-13T15:28:34.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:34.295+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.298853+0000) 2019-10-13T15:28:34.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:34.299+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.299668+0000) 2019-10-13T15:28:34.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:34.296+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.300528+0000) 2019-10-13T15:28:34.303 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:34.299+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.300513+0000) 2019-10-13T15:28:34.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:34.299+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.300914+0000) 2019-10-13T15:28:34.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:34.299+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.301670+0000) 2019-10-13T15:28:34.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:34.304+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.307944+0000) 2019-10-13T15:28:34.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:34.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:34.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:34.776+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.780524+0000) 2019-10-13T15:28:35.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:35.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:35.717523+0000) 2019-10-13T15:28:35.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:35.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:35.717767+0000) 2019-10-13T15:28:35.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:35.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:35.729577+0000) 2019-10-13T15:28:39.314 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:39.311+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.312266+0000) 2019-10-13T15:28:39.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:39.311+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.313185+0000) 2019-10-13T15:28:39.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:39.311+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.313816+0000) 2019-10-13T15:28:39.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:39.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.314587+0000) 2019-10-13T15:28:39.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:39.312+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.314850+0000) 2019-10-13T15:28:39.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:39.311+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.314897+0000) 2019-10-13T15:28:39.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:39.311+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.315181+0000) 2019-10-13T15:28:39.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:39.316+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.319358+0000) 2019-10-13T15:28:39.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:39.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 220976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:39.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:39.515+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:39.518910+0000) 2019-10-13T15:28:39.848 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:28:39.848 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:39.852 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:28:39.852 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:42.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:42.703+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:42.707068+0000) 2019-10-13T15:28:43.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:43.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:43.443392+0000) 2019-10-13T15:28:43.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:43.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:43.692924+0000) 2019-10-13T15:28:44.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:44.323+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.327071+0000) 2019-10-13T15:28:44.330 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:44.327+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.327422+0000) 2019-10-13T15:28:44.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:44.327+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.327995+0000) 2019-10-13T15:28:44.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:44.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.328534+0000) 2019-10-13T15:28:44.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:44.324+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.329032+0000) 2019-10-13T15:28:44.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:44.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.329249+0000) 2019-10-13T15:28:44.332 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:44.327+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.329735+0000) 2019-10-13T15:28:44.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:44.332+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.334674+0000) 2019-10-13T15:28:44.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:44.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:44.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:44.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.786902+0000) 2019-10-13T15:28:45.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:45.715+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:45.717718+0000) 2019-10-13T15:28:45.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:45.715+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:45.717942+0000) 2019-10-13T15:28:45.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:45.727+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:45.729751+0000) 2019-10-13T15:28:49.344 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:49.338+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.341745+0000) 2019-10-13T15:28:49.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:49.338+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.343124+0000) 2019-10-13T15:28:49.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:49.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.343967+0000) 2019-10-13T15:28:49.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:49.340+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.344590+0000) 2019-10-13T15:28:49.347 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:49.342+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.344814+0000) 2019-10-13T15:28:49.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:49.342+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.345180+0000) 2019-10-13T15:28:49.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:49.342+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.345592+0000) 2019-10-13T15:28:49.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:49.344+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.348680+0000) 2019-10-13T15:28:49.357 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:49.350+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221043 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:49.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:49.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:49.519127+0000) 2019-10-13T15:28:52.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:52.706+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:52.707271+0000) 2019-10-13T15:28:53.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:53.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:53.443621+0000) 2019-10-13T15:28:53.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:53.688+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:53.693116+0000) 2019-10-13T15:28:54.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:54.354+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.357526+0000) 2019-10-13T15:28:54.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:54.354+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.357983+0000) 2019-10-13T15:28:54.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:54.358+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.359383+0000) 2019-10-13T15:28:54.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:54.358+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.360055+0000) 2019-10-13T15:28:54.363 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:54.358+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.360602+0000) 2019-10-13T15:28:54.363 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:54.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.361231+0000) 2019-10-13T15:28:54.363 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:54.358+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.361439+0000) 2019-10-13T15:28:54.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:54.364+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.368259+0000) 2019-10-13T15:28:54.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:54.366+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221075 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:54.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:54.788+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.793105+0000) 2019-10-13T15:28:55.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:55.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:55.717947+0000) 2019-10-13T15:28:55.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:55.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:55.718114+0000) 2019-10-13T15:28:55.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:55.726+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:55.729940+0000) 2019-10-13T15:28:59.376 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:28:59.370+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.373403+0000) 2019-10-13T15:28:59.376 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:28:59.370+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.373832+0000) 2019-10-13T15:28:59.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:28:59.370+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.374189+0000) 2019-10-13T15:28:59.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:59.370+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.375266+0000) 2019-10-13T15:28:59.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:28:59.374+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.375510+0000) 2019-10-13T15:28:59.378 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:28:59.374+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.376042+0000) 2019-10-13T15:28:59.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:28:59.372+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.377681+0000) 2019-10-13T15:28:59.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:28:59.380+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.383270+0000) 2019-10-13T15:28:59.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:28:59.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:28:59.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:28:59.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:59.519340+0000) 2019-10-13T15:29:02.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:02.706+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:02.707510+0000) 2019-10-13T15:29:03.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:03.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:03.443816+0000) 2019-10-13T15:29:03.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:03.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:03.693347+0000) 2019-10-13T15:29:04.389 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:04.382+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.387164+0000) 2019-10-13T15:29:04.390 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:04.386+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.387869+0000) 2019-10-13T15:29:04.390 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:04.386+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.388219+0000) 2019-10-13T15:29:04.391 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:04.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.389003+0000) 2019-10-13T15:29:04.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:04.386+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.389641+0000) 2019-10-13T15:29:04.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:04.388+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.390588+0000) 2019-10-13T15:29:04.393 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:04.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.391241+0000) 2019-10-13T15:29:04.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:04.392+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.395075+0000) 2019-10-13T15:29:04.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:04.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:04.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:04.800+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.804055+0000) 2019-10-13T15:29:05.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:05.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:05.718143+0000) 2019-10-13T15:29:05.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:05.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:05.718294+0000) 2019-10-13T15:29:05.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:05.726+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:05.730137+0000) 2019-10-13T15:29:09.403 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:09.398+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.401146+0000) 2019-10-13T15:29:09.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:09.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.401507+0000) 2019-10-13T15:29:09.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:09.398+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.401780+0000) 2019-10-13T15:29:09.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:09.400+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.403035+0000) 2019-10-13T15:29:09.406 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:09.402+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.403647+0000) 2019-10-13T15:29:09.406 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:09.402+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.404089+0000) 2019-10-13T15:29:09.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:09.402+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.404302+0000) 2019-10-13T15:29:09.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:09.408+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.410822+0000) 2019-10-13T15:29:09.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:09.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:09.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:09.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:09.519544+0000) 2019-10-13T15:29:09.972 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:29:09.973 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:09.976 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:29:09.977 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:12.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:12.706+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:12.707711+0000) 2019-10-13T15:29:13.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:13.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:13.444018+0000) 2019-10-13T15:29:13.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:13.688+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:13.693580+0000) 2019-10-13T15:29:14.418 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:14.414+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.415842+0000) 2019-10-13T15:29:14.419 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:14.414+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.416778+0000) 2019-10-13T15:29:14.419 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:14.414+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.417088+0000) 2019-10-13T15:29:14.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:14.414+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.417378+0000) 2019-10-13T15:29:14.420 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:14.412+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.418020+0000) 2019-10-13T15:29:14.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:14.414+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.418835+0000) 2019-10-13T15:29:14.421 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:14.414+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.418866+0000) 2019-10-13T15:29:14.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:14.424+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.426344+0000) 2019-10-13T15:29:14.429 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:14.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:14.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:14.808+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.810622+0000) 2019-10-13T15:29:15.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:15.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:15.718372+0000) 2019-10-13T15:29:15.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:15.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:15.718488+0000) 2019-10-13T15:29:15.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:15.726+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:15.730329+0000) 2019-10-13T15:29:19.432 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:19.426+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.429558+0000) 2019-10-13T15:29:19.433 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:19.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.430904+0000) 2019-10-13T15:29:19.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:19.430+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.431418+0000) 2019-10-13T15:29:19.434 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:19.430+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.431686+0000) 2019-10-13T15:29:19.435 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:19.428+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.433243+0000) 2019-10-13T15:29:19.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:19.430+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.433615+0000) 2019-10-13T15:29:19.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:19.430+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.433820+0000) 2019-10-13T15:29:19.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:19.436+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.439095+0000) 2019-10-13T15:29:19.444 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:19.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221244 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:19.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:19.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:19.519744+0000) 2019-10-13T15:29:22.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:22.706+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:22.707942+0000) 2019-10-13T15:29:23.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:23.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:23.444210+0000) 2019-10-13T15:29:23.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:23.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:23.693776+0000) 2019-10-13T15:29:24.447 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:24.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.444974+0000) 2019-10-13T15:29:24.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:24.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.445360+0000) 2019-10-13T15:29:24.449 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:24.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.446695+0000) 2019-10-13T15:29:24.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:24.444+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.447421+0000) 2019-10-13T15:29:24.450 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:24.446+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.447534+0000) 2019-10-13T15:29:24.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:24.446+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.447818+0000) 2019-10-13T15:29:24.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:24.446+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.448216+0000) 2019-10-13T15:29:24.455 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:24.448+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.452730+0000) 2019-10-13T15:29:24.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:24.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:24.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:24.812+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.816987+0000) 2019-10-13T15:29:25.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:25.714+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:25.718569+0000) 2019-10-13T15:29:25.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:25.714+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:25.718698+0000) 2019-10-13T15:29:25.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:25.730+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:25.730582+0000) 2019-10-13T15:29:29.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:29.458+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.459542+0000) 2019-10-13T15:29:29.462 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:29.458+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.459866+0000) 2019-10-13T15:29:29.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:29.458+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.461144+0000) 2019-10-13T15:29:29.464 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:29.458+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.461762+0000) 2019-10-13T15:29:29.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:29.458+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.462252+0000) 2019-10-13T15:29:29.465 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:29.459+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.462823+0000) 2019-10-13T15:29:29.465 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:29.458+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.462874+0000) 2019-10-13T15:29:29.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:29.463+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.468268+0000) 2019-10-13T15:29:29.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:29.470+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:29.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:29.518+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:29.519905+0000) 2019-10-13T15:29:32.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:32.705+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:32.708175+0000) 2019-10-13T15:29:33.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:33.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:33.444407+0000) 2019-10-13T15:29:33.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:33.687+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:33.693996+0000) 2019-10-13T15:29:34.478 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:34.469+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.474855+0000) 2019-10-13T15:29:34.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:34.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.476239+0000) 2019-10-13T15:29:34.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:34.473+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.476556+0000) 2019-10-13T15:29:34.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:34.471+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.476914+0000) 2019-10-13T15:29:34.479 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:34.473+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.477194+0000) 2019-10-13T15:29:34.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:34.473+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.477892+0000) 2019-10-13T15:29:34.480 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:34.473+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.478428+0000) 2019-10-13T15:29:34.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:34.479+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.484688+0000) 2019-10-13T15:29:34.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:34.481+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:34.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:34.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.823290+0000) 2019-10-13T15:29:35.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:35.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:35.718811+0000) 2019-10-13T15:29:35.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:35.717+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:35.718911+0000) 2019-10-13T15:29:35.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:35.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:35.730777+0000) 2019-10-13T15:29:39.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:39.487+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.491592+0000) 2019-10-13T15:29:39.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:39.485+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.490334+0000) 2019-10-13T15:29:39.495 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:39.485+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.490723+0000) 2019-10-13T15:29:39.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:39.489+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.491825+0000) 2019-10-13T15:29:39.495 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:39.489+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.492313+0000) 2019-10-13T15:29:39.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:39.489+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.493097+0000) 2019-10-13T15:29:39.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:39.489+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.492529+0000) 2019-10-13T15:29:39.501 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:39.495+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.499564+0000) 2019-10-13T15:29:39.503 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:39.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:39.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:39.517+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:39.520056+0000) 2019-10-13T15:29:40.087 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:29:40.087 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:40.091 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:29:40.091 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:42.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:42.705+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:42.708386+0000) 2019-10-13T15:29:43.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:43.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:43.444602+0000) 2019-10-13T15:29:43.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:43.691+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:43.694207+0000) 2019-10-13T15:29:44.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:44.501+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.504481+0000) 2019-10-13T15:29:44.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:44.499+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.505564+0000) 2019-10-13T15:29:44.509 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:44.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.506355+0000) 2019-10-13T15:29:44.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:44.501+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.506050+0000) 2019-10-13T15:29:44.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:44.501+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.506682+0000) 2019-10-13T15:29:44.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:44.505+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.507621+0000) 2019-10-13T15:29:44.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:44.505+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.508255+0000) 2019-10-13T15:29:44.515 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:44.507+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.513487+0000) 2019-10-13T15:29:44.518 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:44.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:44.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:44.823+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.829866+0000) 2019-10-13T15:29:45.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:45.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:45.719013+0000) 2019-10-13T15:29:45.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:45.713+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:45.719126+0000) 2019-10-13T15:29:45.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:45.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:45.730975+0000) 2019-10-13T15:29:49.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:49.515+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.519053+0000) 2019-10-13T15:29:49.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:49.517+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.519732+0000) 2019-10-13T15:29:49.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:49.517+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.520496+0000) 2019-10-13T15:29:49.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:49.517+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.521137+0000) 2019-10-13T15:29:49.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:49.517+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.520199+0000) 2019-10-13T15:29:49.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:49.517+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.522339+0000) 2019-10-13T15:29:49.525 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:49.517+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.523149+0000) 2019-10-13T15:29:49.534 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:49.525+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:49.534 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:49.527+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:49.531468+0000) 2019-10-13T15:29:52.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:52.705+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:52.708590+0000) 2019-10-13T15:29:53.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:53.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:53.444818+0000) 2019-10-13T15:29:53.697 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:53.691+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:53.694415+0000) 2019-10-13T15:29:54.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:54.531+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.534536+0000) 2019-10-13T15:29:54.537 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:54.529+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.534827+0000) 2019-10-13T15:29:54.538 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:54.533+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.535468+0000) 2019-10-13T15:29:54.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:54.533+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.536034+0000) 2019-10-13T15:29:54.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:54.533+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.536773+0000) 2019-10-13T15:29:54.541 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:54.533+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.537546+0000) 2019-10-13T15:29:54.541 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:54.533+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.538460+0000) 2019-10-13T15:29:54.545 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:54.539+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.542789+0000) 2019-10-13T15:29:54.548 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:54.541+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:29:54.838 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:54.835+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.836283+0000) 2019-10-13T15:29:55.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:55.713+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:55.719218+0000) 2019-10-13T15:29:55.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:55.717+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:55.719364+0000) 2019-10-13T15:29:55.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:55.725+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:55.731170+0000) 2019-10-13T15:29:56.273 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:56.265+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.270697+0000) 2019-10-13T15:29:59.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:29:59.545+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.548920+0000) 2019-10-13T15:29:59.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:29:59.545+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.549227+0000) 2019-10-13T15:29:59.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:29:59.545+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.549576+0000) 2019-10-13T15:29:59.553 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:29:59.545+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.550934+0000) 2019-10-13T15:29:59.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:29:59.547+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.551599+0000) 2019-10-13T15:29:59.554 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:29:59.549+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.551705+0000) 2019-10-13T15:29:59.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:29:59.549+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.552236+0000) 2019-10-13T15:29:59.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:29:59.555+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:59.560147+0000) 2019-10-13T15:29:59.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:29:59.557+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221512 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:02.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:02.705+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:02.708822+0000) 2019-10-13T15:30:03.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:03.441+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:03.445014+0000) 2019-10-13T15:30:03.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:03.691+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:03.694618+0000) 2019-10-13T15:30:04.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:04.561+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.564180+0000) 2019-10-13T15:30:04.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:04.561+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.564623+0000) 2019-10-13T15:30:04.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:04.561+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.565046+0000) 2019-10-13T15:30:04.568 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:04.559+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.565577+0000) 2019-10-13T15:30:04.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:04.561+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.566667+0000) 2019-10-13T15:30:04.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:04.561+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.566935+0000) 2019-10-13T15:30:04.570 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:04.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.567888+0000) 2019-10-13T15:30:04.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:04.571+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.574582+0000) 2019-10-13T15:30:04.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:04.569+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:04.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:04.839+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.842852+0000) 2019-10-13T15:30:05.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:05.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:05.719417+0000) 2019-10-13T15:30:05.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:05.717+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:05.719568+0000) 2019-10-13T15:30:05.733 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:05.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:05.731374+0000) 2019-10-13T15:30:06.273 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:06.269+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.270895+0000) 2019-10-13T15:30:09.581 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:09.573+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.577866+0000) 2019-10-13T15:30:09.581 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:09.575+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.578592+0000) 2019-10-13T15:30:09.581 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:09.573+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.578954+0000) 2019-10-13T15:30:09.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:09.577+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.579528+0000) 2019-10-13T15:30:09.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:09.577+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.580067+0000) 2019-10-13T15:30:09.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:09.577+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.580395+0000) 2019-10-13T15:30:09.583 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:09.577+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.580835+0000) 2019-10-13T15:30:09.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:09.583+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:09.587899+0000) 2019-10-13T15:30:09.591 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:09.585+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:10.211 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:30:10.211 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:10.215 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:30:10.215 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:12.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:12.705+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:12.709057+0000) 2019-10-13T15:30:13.447 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:13.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:13.445248+0000) 2019-10-13T15:30:13.697 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:13.691+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:13.694839+0000) 2019-10-13T15:30:14.595 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:14.588+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.592073+0000) 2019-10-13T15:30:14.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:14.588+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.592396+0000) 2019-10-13T15:30:14.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:14.588+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.592829+0000) 2019-10-13T15:30:14.596 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:14.588+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.593484+0000) 2019-10-13T15:30:14.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:14.587+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.593802+0000) 2019-10-13T15:30:14.597 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:14.588+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.594001+0000) 2019-10-13T15:30:14.597 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:14.588+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.594573+0000) 2019-10-13T15:30:14.603 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:14.595+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.601490+0000) 2019-10-13T15:30:14.605 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:14.600+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221611 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:14.852 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:14.847+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.849426+0000) 2019-10-13T15:30:15.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:15.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:15.719646+0000) 2019-10-13T15:30:15.722 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:15.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:15.719766+0000) 2019-10-13T15:30:15.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:15.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:15.731572+0000) 2019-10-13T15:30:16.273 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:16.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.271097+0000) 2019-10-13T15:30:19.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:19.600+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.606974+0000) 2019-10-13T15:30:19.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:19.600+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.606983+0000) 2019-10-13T15:30:19.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:19.604+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.607710+0000) 2019-10-13T15:30:19.611 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:19.604+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.608021+0000) 2019-10-13T15:30:19.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:19.607+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.610384+0000) 2019-10-13T15:30:19.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:19.604+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.608896+0000) 2019-10-13T15:30:19.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:19.604+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.609530+0000) 2019-10-13T15:30:19.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:19.611+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:19.616626+0000) 2019-10-13T15:30:19.621 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:19.612+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:22.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:22.708+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:22.709256+0000) 2019-10-13T15:30:23.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:23.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:23.445416+0000) 2019-10-13T15:30:23.697 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:23.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:23.695040+0000) 2019-10-13T15:30:24.625 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:24.616+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.622802+0000) 2019-10-13T15:30:24.626 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:24.616+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.622916+0000) 2019-10-13T15:30:24.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:24.620+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.623444+0000) 2019-10-13T15:30:24.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:24.620+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.624010+0000) 2019-10-13T15:30:24.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:24.620+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.624299+0000) 2019-10-13T15:30:24.627 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:24.620+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.624664+0000) 2019-10-13T15:30:24.629 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:24.622+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.626714+0000) 2019-10-13T15:30:24.633 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:24.626+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.631556+0000) 2019-10-13T15:30:24.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:24.628+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:24.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:24.854+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.855976+0000) 2019-10-13T15:30:25.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:25.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:25.719847+0000) 2019-10-13T15:30:25.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:25.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:25.719967+0000) 2019-10-13T15:30:25.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:25.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:25.731802+0000) 2019-10-13T15:30:26.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:26.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.271356+0000) 2019-10-13T15:30:29.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:29.632+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.637413+0000) 2019-10-13T15:30:29.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:29.632+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.638045+0000) 2019-10-13T15:30:29.642 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:29.632+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.638069+0000) 2019-10-13T15:30:29.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:29.632+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.638824+0000) 2019-10-13T15:30:29.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:29.634+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.640137+0000) 2019-10-13T15:30:29.643 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:29.636+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.639784+0000) 2019-10-13T15:30:29.643 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:29.636+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.640664+0000) 2019-10-13T15:30:29.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:29.642+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:29.646471+0000) 2019-10-13T15:30:29.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:29.644+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:32.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:32.708+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:32.709489+0000) 2019-10-13T15:30:33.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:33.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:33.445624+0000) 2019-10-13T15:30:33.697 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:33.690+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:33.695249+0000) 2019-10-13T15:30:34.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:34.648+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.652362+0000) 2019-10-13T15:30:34.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:34.648+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.652864+0000) 2019-10-13T15:30:34.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:34.648+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.653858+0000) 2019-10-13T15:30:34.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:34.648+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.654804+0000) 2019-10-13T15:30:34.658 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:34.648+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.655112+0000) 2019-10-13T15:30:34.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:34.652+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.655664+0000) 2019-10-13T15:30:34.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:34.650+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.656231+0000) 2019-10-13T15:30:34.663 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:34.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.660990+0000) 2019-10-13T15:30:34.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:34.660+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:34.864 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:34.858+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.862372+0000) 2019-10-13T15:30:35.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:35.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:35.720052+0000) 2019-10-13T15:30:35.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:35.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:35.720209+0000) 2019-10-13T15:30:35.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:35.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:35.732012+0000) 2019-10-13T15:30:36.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:36.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.271560+0000) 2019-10-13T15:30:39.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:39.664+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.668800+0000) 2019-10-13T15:30:39.672 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:39.664+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.669159+0000) 2019-10-13T15:30:39.673 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:39.664+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.670448+0000) 2019-10-13T15:30:39.673 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:39.664+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.670507+0000) 2019-10-13T15:30:39.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:39.664+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.670871+0000) 2019-10-13T15:30:39.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:39.664+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.671157+0000) 2019-10-13T15:30:39.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:39.666+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.672520+0000) 2019-10-13T15:30:39.679 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:39.670+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:39.676646+0000) 2019-10-13T15:30:39.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:39.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:40.317 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:30:40.317 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:40.321 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:30:40.321 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:42.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:42.708+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:42.709692+0000) 2019-10-13T15:30:43.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:43.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:43.445846+0000) 2019-10-13T15:30:43.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:43.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:43.695479+0000) 2019-10-13T15:30:44.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:44.680+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.683607+0000) 2019-10-13T15:30:44.686 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:44.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.683862+0000) 2019-10-13T15:30:44.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:44.680+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.684810+0000) 2019-10-13T15:30:44.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:44.678+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.685885+0000) 2019-10-13T15:30:44.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:44.680+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.686343+0000) 2019-10-13T15:30:44.689 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:44.680+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.686889+0000) 2019-10-13T15:30:44.689 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:44.684+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.687373+0000) 2019-10-13T15:30:44.697 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:44.688+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:44.697 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:44.690+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.695262+0000) 2019-10-13T15:30:44.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:44.866+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.868992+0000) 2019-10-13T15:30:45.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:45.716+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:45.720261+0000) 2019-10-13T15:30:45.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:45.716+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:45.720417+0000) 2019-10-13T15:30:45.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:45.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:45.732207+0000) 2019-10-13T15:30:46.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:46.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.271762+0000) 2019-10-13T15:30:49.701 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:49.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.698924+0000) 2019-10-13T15:30:49.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:49.692+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.699128+0000) 2019-10-13T15:30:49.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:49.696+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.699678+0000) 2019-10-13T15:30:49.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:49.694+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.700298+0000) 2019-10-13T15:30:49.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:49.696+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.700584+0000) 2019-10-13T15:30:49.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:49.696+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.700951+0000) 2019-10-13T15:30:49.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:49.696+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.701604+0000) 2019-10-13T15:30:49.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:49.702+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:49.707131+0000) 2019-10-13T15:30:49.711 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:49.704+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:52.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:52.708+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:52.709895+0000) 2019-10-13T15:30:53.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:53.440+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:53.446043+0000) 2019-10-13T15:30:53.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:53.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:53.695699+0000) 2019-10-13T15:30:54.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:54.708+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.712229+0000) 2019-10-13T15:30:54.715 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:54.708+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.712724+0000) 2019-10-13T15:30:54.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:54.708+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.713466+0000) 2019-10-13T15:30:54.716 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:54.708+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.713881+0000) 2019-10-13T15:30:54.716 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:54.708+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.714430+0000) 2019-10-13T15:30:54.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:54.708+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.715118+0000) 2019-10-13T15:30:54.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:54.710+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.715989+0000) 2019-10-13T15:30:54.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:54.714+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.721204+0000) 2019-10-13T15:30:54.726 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:54.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:30:54.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:54.874+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.875287+0000) 2019-10-13T15:30:55.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:55.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:55.720485+0000) 2019-10-13T15:30:55.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:55.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:55.720633+0000) 2019-10-13T15:30:55.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:55.728+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:55.732420+0000) 2019-10-13T15:30:56.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:56.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.271994+0000) 2019-10-13T15:30:59.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:30:59.719+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.726911+0000) 2019-10-13T15:30:59.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:30:59.719+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.727191+0000) 2019-10-13T15:30:59.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:30:59.723+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.727932+0000) 2019-10-13T15:30:59.731 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:30:59.723+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.728377+0000) 2019-10-13T15:30:59.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:30:59.723+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.728040+0000) 2019-10-13T15:30:59.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:30:59.722+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.729884+0000) 2019-10-13T15:30:59.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:30:59.723+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.730057+0000) 2019-10-13T15:30:59.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:30:59.730+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:59.735090+0000) 2019-10-13T15:30:59.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:30:59.731+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221912 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:02.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:02.707+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:02.710094+0000) 2019-10-13T15:31:03.448 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:03.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:03.446240+0000) 2019-10-13T15:31:03.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:03.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:03.695907+0000) 2019-10-13T15:31:04.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:04.735+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.740913+0000) 2019-10-13T15:31:04.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:04.735+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.741399+0000) 2019-10-13T15:31:04.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:04.735+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.742864+0000) 2019-10-13T15:31:04.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:04.735+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.743231+0000) 2019-10-13T15:31:04.746 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:04.739+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.743750+0000) 2019-10-13T15:31:04.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:04.738+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.744214+0000) 2019-10-13T15:31:04.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:04.739+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.744740+0000) 2019-10-13T15:31:04.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:04.742+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.747830+0000) 2019-10-13T15:31:04.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:04.747+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:04.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:04.874+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.881757+0000) 2019-10-13T15:31:05.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:05.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:05.720687+0000) 2019-10-13T15:31:05.723 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:05.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:05.720841+0000) 2019-10-13T15:31:05.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:05.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:05.732645+0000) 2019-10-13T15:31:06.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:06.271+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.272201+0000) 2019-10-13T15:31:09.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:09.751+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.756282+0000) 2019-10-13T15:31:09.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:09.751+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.757395+0000) 2019-10-13T15:31:09.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:09.751+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.757812+0000) 2019-10-13T15:31:09.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:09.758+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.758376+0000) 2019-10-13T15:31:09.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:09.751+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.758843+0000) 2019-10-13T15:31:09.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:09.751+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.759216+0000) 2019-10-13T15:31:09.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:09.755+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.759883+0000) 2019-10-13T15:31:09.766 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:09.762+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:09.763974+0000) 2019-10-13T15:31:09.769 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:09.763+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 221980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:10.458 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:31:10.458 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:10.461 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:31:10.462 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:12.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:12.707+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:12.710313+0000) 2019-10-13T15:31:13.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:13.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:13.446478+0000) 2019-10-13T15:31:13.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:13.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:13.696118+0000) 2019-10-13T15:31:14.773 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:14.770+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.770520+0000) 2019-10-13T15:31:14.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:14.767+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.771482+0000) 2019-10-13T15:31:14.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:14.767+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.771715+0000) 2019-10-13T15:31:14.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:14.767+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.772600+0000) 2019-10-13T15:31:14.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:14.767+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.772728+0000) 2019-10-13T15:31:14.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:14.767+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.773290+0000) 2019-10-13T15:31:14.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:14.767+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.773679+0000) 2019-10-13T15:31:14.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:14.778+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.778485+0000) 2019-10-13T15:31:14.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:14.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:14.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:14.886+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.888210+0000) 2019-10-13T15:31:15.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:15.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:15.720865+0000) 2019-10-13T15:31:15.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:15.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:15.721048+0000) 2019-10-13T15:31:15.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:15.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:15.732827+0000) 2019-10-13T15:31:16.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:16.271+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.272431+0000) 2019-10-13T15:31:19.787 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:19.781+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.785076+0000) 2019-10-13T15:31:19.788 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:19.779+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.785223+0000) 2019-10-13T15:31:19.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:19.779+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.785551+0000) 2019-10-13T15:31:19.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:19.779+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.786959+0000) 2019-10-13T15:31:19.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:19.783+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.787663+0000) 2019-10-13T15:31:19.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:19.783+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.788335+0000) 2019-10-13T15:31:19.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:19.783+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.788389+0000) 2019-10-13T15:31:19.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:19.789+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:19.793252+0000) 2019-10-13T15:31:19.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:19.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:22.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:22.707+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:22.710516+0000) 2019-10-13T15:31:23.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:23.439+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:23.446704+0000) 2019-10-13T15:31:23.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:23.693+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:23.696320+0000) 2019-10-13T15:31:24.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:24.795+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.800760+0000) 2019-10-13T15:31:24.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:24.795+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.800409+0000) 2019-10-13T15:31:24.804 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:24.795+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.801350+0000) 2019-10-13T15:31:24.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:24.795+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.801883+0000) 2019-10-13T15:31:24.805 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:24.801+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.802257+0000) 2019-10-13T15:31:24.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:24.795+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.802396+0000) 2019-10-13T15:31:24.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:24.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.803791+0000) 2019-10-13T15:31:24.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:24.809+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.812523+0000) 2019-10-13T15:31:24.815 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:24.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:24.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:24.893+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.895223+0000) 2019-10-13T15:31:25.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:25.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:25.721060+0000) 2019-10-13T15:31:25.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:25.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:25.721235+0000) 2019-10-13T15:31:25.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:25.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:25.733001+0000) 2019-10-13T15:31:26.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:26.271+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:26.272641+0000) 2019-10-13T15:31:29.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:29.815+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.815819+0000) 2019-10-13T15:31:29.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:29.813+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.816339+0000) 2019-10-13T15:31:29.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:29.815+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.817057+0000) 2019-10-13T15:31:29.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:29.815+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.817674+0000) 2019-10-13T15:31:29.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:29.815+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.818297+0000) 2019-10-13T15:31:29.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:29.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.818939+0000) 2019-10-13T15:31:29.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:29.815+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.819226+0000) 2019-10-13T15:31:29.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:29.825+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:29.827378+0000) 2019-10-13T15:31:29.830 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:29.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222114 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:32.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:32.707+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.710715+0000) 2019-10-13T15:31:33.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:33.443+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:33.446931+0000) 2019-10-13T15:31:33.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:33.693+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:33.696526+0000) 2019-10-13T15:31:34.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:34.827+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.830982+0000) 2019-10-13T15:31:34.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:34.831+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.831604+0000) 2019-10-13T15:31:34.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:34.831+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.831819+0000) 2019-10-13T15:31:34.834 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:34.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.832473+0000) 2019-10-13T15:31:34.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:34.831+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.832805+0000) 2019-10-13T15:31:34.836 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:34.831+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.833655+0000) 2019-10-13T15:31:34.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:34.833+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.834444+0000) 2019-10-13T15:31:34.843 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:34.837+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.841177+0000) 2019-10-13T15:31:34.844 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:34.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:34.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:34.901+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.902710+0000) 2019-10-13T15:31:35.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:35.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:35.721261+0000) 2019-10-13T15:31:35.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:35.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:35.721433+0000) 2019-10-13T15:31:35.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:35.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:35.733178+0000) 2019-10-13T15:31:36.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:36.271+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:36.272879+0000) 2019-10-13T15:31:39.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:39.843+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.844475+0000) 2019-10-13T15:31:39.847 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:39.843+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.845248+0000) 2019-10-13T15:31:39.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:39.843+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.846212+0000) 2019-10-13T15:31:39.849 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:39.843+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.846240+0000) 2019-10-13T15:31:39.849 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:39.843+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.846569+0000) 2019-10-13T15:31:39.850 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:39.843+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.847084+0000) 2019-10-13T15:31:39.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:39.845+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.848796+0000) 2019-10-13T15:31:39.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:39.853+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:39.854200+0000) 2019-10-13T15:31:39.859 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:39.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:40.590 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:31:40.590 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:40.593 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:31:40.594 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:42.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:42.706+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.710944+0000) 2019-10-13T15:31:43.449 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:43.442+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:43.447126+0000) 2019-10-13T15:31:43.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:43.693+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:43.696727+0000) 2019-10-13T15:31:44.862 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:44.858+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.859538+0000) 2019-10-13T15:31:44.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:44.857+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.860346+0000) 2019-10-13T15:31:44.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:44.858+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.860525+0000) 2019-10-13T15:31:44.864 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:44.858+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.861741+0000) 2019-10-13T15:31:44.864 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:44.858+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.862128+0000) 2019-10-13T15:31:44.865 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:44.858+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.862810+0000) 2019-10-13T15:31:44.865 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:44.862+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.863387+0000) 2019-10-13T15:31:44.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:44.865+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.868928+0000) 2019-10-13T15:31:44.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:44.870+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:44.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:44.909+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.909609+0000) 2019-10-13T15:31:45.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:45.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:45.721442+0000) 2019-10-13T15:31:45.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:45.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:45.721582+0000) 2019-10-13T15:31:45.735 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:45.730+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:45.733354+0000) 2019-10-13T15:31:46.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:46.270+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:46.273141+0000) 2019-10-13T15:31:49.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:49.870+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.874828+0000) 2019-10-13T15:31:49.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:49.874+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.875865+0000) 2019-10-13T15:31:49.878 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:49.874+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.876180+0000) 2019-10-13T15:31:49.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:49.873+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.876602+0000) 2019-10-13T15:31:49.879 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:49.874+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.876782+0000) 2019-10-13T15:31:49.879 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:49.874+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.877161+0000) 2019-10-13T15:31:49.881 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:49.874+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.878675+0000) 2019-10-13T15:31:49.886 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:49.881+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:49.884510+0000) 2019-10-13T15:31:49.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:49.882+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:52.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:52.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.711141+0000) 2019-10-13T15:31:53.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:53.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:53.447348+0000) 2019-10-13T15:31:53.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:53.693+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:53.696930+0000) 2019-10-13T15:31:54.891 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:54.886+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.888513+0000) 2019-10-13T15:31:54.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:54.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.889154+0000) 2019-10-13T15:31:54.892 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:54.886+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.889472+0000) 2019-10-13T15:31:54.893 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:54.886+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.890424+0000) 2019-10-13T15:31:54.894 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:54.886+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.890900+0000) 2019-10-13T15:31:54.894 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:54.886+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.891241+0000) 2019-10-13T15:31:54.894 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:54.890+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.891677+0000) 2019-10-13T15:31:54.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:54.897+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.899345+0000) 2019-10-13T15:31:54.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:54.898+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:31:54.917 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:54.913+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.914841+0000) 2019-10-13T15:31:55.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:55.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:55.721655+0000) 2019-10-13T15:31:55.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:55.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:55.721773+0000) 2019-10-13T15:31:55.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:55.730+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:55.733552+0000) 2019-10-13T15:31:56.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:56.270+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:56.273358+0000) 2019-10-13T15:31:59.907 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:31:59.902+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.904489+0000) 2019-10-13T15:31:59.908 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:31:59.902+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.905443+0000) 2019-10-13T15:31:59.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:31:59.902+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.905651+0000) 2019-10-13T15:31:59.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:31:59.901+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.906114+0000) 2019-10-13T15:31:59.910 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:31:59.902+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.906519+0000) 2019-10-13T15:31:59.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:31:59.906+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.907295+0000) 2019-10-13T15:31:59.910 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:31:59.906+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.908092+0000) 2019-10-13T15:31:59.915 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:31:59.909+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:59.913566+0000) 2019-10-13T15:31:59.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:31:59.910+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222315 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:02.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:02.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.711352+0000) 2019-10-13T15:32:03.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:03.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:03.447536+0000) 2019-10-13T15:32:03.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:03.693+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:03.697126+0000) 2019-10-13T15:32:04.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:04.914+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.918965+0000) 2019-10-13T15:32:04.922 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:04.918+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.919649+0000) 2019-10-13T15:32:04.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:04.918+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.919772+0000) 2019-10-13T15:32:04.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:04.917+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.920189+0000) 2019-10-13T15:32:04.923 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:04.918+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.920411+0000) 2019-10-13T15:32:04.923 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:04.918+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.921199+0000) 2019-10-13T15:32:04.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:04.918+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.922214+0000) 2019-10-13T15:32:04.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:04.925+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.928107+0000) 2019-10-13T15:32:04.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:04.925+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.928159+0000) 2019-10-13T15:32:04.932 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:04.926+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:05.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:05.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:05.721836+0000) 2019-10-13T15:32:05.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:05.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:05.721972+0000) 2019-10-13T15:32:05.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:05.730+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:05.733744+0000) 2019-10-13T15:32:06.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:06.270+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:06.273553+0000) 2019-10-13T15:32:09.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:09.930+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.932624+0000) 2019-10-13T15:32:09.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:09.929+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.933495+0000) 2019-10-13T15:32:09.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:09.930+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.934042+0000) 2019-10-13T15:32:09.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:09.930+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.934828+0000) 2019-10-13T15:32:09.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:09.930+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.935271+0000) 2019-10-13T15:32:09.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:09.934+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.935682+0000) 2019-10-13T15:32:09.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:09.934+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.936109+0000) 2019-10-13T15:32:09.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:09.937+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:09.940777+0000) 2019-10-13T15:32:09.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:09.942+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:10.726 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:32:10.726 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:10.730 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:32:10.730 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:12.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:12.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.711556+0000) 2019-10-13T15:32:13.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:13.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:13.447765+0000) 2019-10-13T15:32:13.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:13.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:13.697347+0000) 2019-10-13T15:32:14.931 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:14.924+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.928375+0000) 2019-10-13T15:32:14.950 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:14.946+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.947711+0000) 2019-10-13T15:32:14.951 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:14.944+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.948537+0000) 2019-10-13T15:32:14.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:14.946+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.949141+0000) 2019-10-13T15:32:14.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:14.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.949596+0000) 2019-10-13T15:32:14.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:14.946+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.950083+0000) 2019-10-13T15:32:14.953 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:14.946+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.950691+0000) 2019-10-13T15:32:14.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:14.946+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.951154+0000) 2019-10-13T15:32:14.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:14.952+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.954952+0000) 2019-10-13T15:32:14.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:14.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:15.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:15.718+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:15.722007+0000) 2019-10-13T15:32:15.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:15.718+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:15.722171+0000) 2019-10-13T15:32:15.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:15.730+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:15.733922+0000) 2019-10-13T15:32:16.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:16.270+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:16.273747+0000) 2019-10-13T15:32:19.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:19.960+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.962243+0000) 2019-10-13T15:32:19.965 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:19.958+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.962736+0000) 2019-10-13T15:32:19.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:19.958+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.962850+0000) 2019-10-13T15:32:19.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:19.962+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.963613+0000) 2019-10-13T15:32:19.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:19.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.964922+0000) 2019-10-13T15:32:19.967 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:19.962+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.965402+0000) 2019-10-13T15:32:19.968 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:19.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.965947+0000) 2019-10-13T15:32:19.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:19.964+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:19.969942+0000) 2019-10-13T15:32:19.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:19.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:22.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:22.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.711792+0000) 2019-10-13T15:32:23.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:23.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:23.447959+0000) 2019-10-13T15:32:23.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:23.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:23.697543+0000) 2019-10-13T15:32:24.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:24.932+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.934752+0000) 2019-10-13T15:32:24.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:24.970+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.975083+0000) 2019-10-13T15:32:24.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:24.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.975621+0000) 2019-10-13T15:32:24.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:24.972+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.977471+0000) 2019-10-13T15:32:24.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:24.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.977738+0000) 2019-10-13T15:32:24.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:24.974+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.977826+0000) 2019-10-13T15:32:24.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:24.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.978143+0000) 2019-10-13T15:32:24.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:24.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.979090+0000) 2019-10-13T15:32:24.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:24.980+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.983045+0000) 2019-10-13T15:32:24.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:24.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:25.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:25.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:25.722195+0000) 2019-10-13T15:32:25.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:25.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:25.722322+0000) 2019-10-13T15:32:25.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:25.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:25.734127+0000) 2019-10-13T15:32:26.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:26.269+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:26.273979+0000) 2019-10-13T15:32:29.993 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:29.985+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.990847+0000) 2019-10-13T15:32:29.994 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:29.985+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.991070+0000) 2019-10-13T15:32:29.994 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:29.989+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.991424+0000) 2019-10-13T15:32:29.995 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:29.988+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.992092+0000) 2019-10-13T15:32:29.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:29.989+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.992506+0000) 2019-10-13T15:32:29.995 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:29.989+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.992900+0000) 2019-10-13T15:32:29.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:29.989+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:29.993495+0000) 2019-10-13T15:32:30.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:29.996+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:30.000224+0000) 2019-10-13T15:32:30.004 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:29.997+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:32.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:32.709+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.712019+0000) 2019-10-13T15:32:33.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:33.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:33.448160+0000) 2019-10-13T15:32:33.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:33.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:33.697734+0000) 2019-10-13T15:32:34.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:34.940+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:34.941278+0000) 2019-10-13T15:32:35.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:35.004+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.006263+0000) 2019-10-13T15:32:35.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:35.001+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.005733+0000) 2019-10-13T15:32:35.009 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:35.001+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.006681+0000) 2019-10-13T15:32:35.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:35.005+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.007355+0000) 2019-10-13T15:32:35.010 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:35.005+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.007680+0000) 2019-10-13T15:32:35.010 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:35.005+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.007958+0000) 2019-10-13T15:32:35.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:35.005+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.008925+0000) 2019-10-13T15:32:35.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:35.008+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.013505+0000) 2019-10-13T15:32:35.019 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:35.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:35.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:35.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.722381+0000) 2019-10-13T15:32:35.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:35.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.722516+0000) 2019-10-13T15:32:35.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:35.733+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.734337+0000) 2019-10-13T15:32:36.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:36.269+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:36.274211+0000) 2019-10-13T15:32:40.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:40.017+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.019859+0000) 2019-10-13T15:32:40.023 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:40.017+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.019994+0000) 2019-10-13T15:32:40.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:40.017+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.021512+0000) 2019-10-13T15:32:40.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:40.017+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.021879+0000) 2019-10-13T15:32:40.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:40.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.022349+0000) 2019-10-13T15:32:40.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:40.017+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.022453+0000) 2019-10-13T15:32:40.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:40.017+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.022784+0000) 2019-10-13T15:32:40.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:40.024+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:40.026997+0000) 2019-10-13T15:32:40.033 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:40.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:40.794 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:32:40.794 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:40.798 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:32:40.798 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:42.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:42.709+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.712218+0000) 2019-10-13T15:32:43.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:43.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:43.448394+0000) 2019-10-13T15:32:43.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:43.692+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:43.697938+0000) 2019-10-13T15:32:44.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:44.944+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:44.947505+0000) 2019-10-13T15:32:45.037 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:45.029+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.034341+0000) 2019-10-13T15:32:45.037 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:45.029+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.034921+0000) 2019-10-13T15:32:45.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:45.032+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.035167+0000) 2019-10-13T15:32:45.038 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:45.033+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.036188+0000) 2019-10-13T15:32:45.039 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:45.033+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.036697+0000) 2019-10-13T15:32:45.039 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:45.033+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.036317+0000) 2019-10-13T15:32:45.040 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:45.033+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.037460+0000) 2019-10-13T15:32:45.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:45.036+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.041956+0000) 2019-10-13T15:32:45.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:45.041+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222621 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:45.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:45.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.722574+0000) 2019-10-13T15:32:45.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:45.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.722706+0000) 2019-10-13T15:32:45.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:45.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.734515+0000) 2019-10-13T15:32:46.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:46.273+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:46.274429+0000) 2019-10-13T15:32:50.051 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:50.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.048523+0000) 2019-10-13T15:32:50.051 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:50.044+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.049379+0000) 2019-10-13T15:32:50.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:50.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.049913+0000) 2019-10-13T15:32:50.052 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:50.045+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.050144+0000) 2019-10-13T15:32:50.053 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:50.045+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.050556+0000) 2019-10-13T15:32:50.053 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:50.045+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.051193+0000) 2019-10-13T15:32:50.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:50.049+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.052009+0000) 2019-10-13T15:32:50.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:50.052+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:50.057820+0000) 2019-10-13T15:32:50.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:50.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:52.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:52.709+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.712414+0000) 2019-10-13T15:32:53.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:53.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:53.448619+0000) 2019-10-13T15:32:53.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:53.696+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:53.698103+0000) 2019-10-13T15:32:54.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:54.948+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:54.954035+0000) 2019-10-13T15:32:55.067 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:32:55.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.064421+0000) 2019-10-13T15:32:55.067 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:55.061+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.064739+0000) 2019-10-13T15:32:55.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:55.061+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.065789+0000) 2019-10-13T15:32:55.068 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:55.061+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.065944+0000) 2019-10-13T15:32:55.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:32:55.064+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.066510+0000) 2019-10-13T15:32:55.069 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:32:55.061+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.066736+0000) 2019-10-13T15:32:55.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:55.065+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.068075+0000) 2019-10-13T15:32:55.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:32:55.068+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.073600+0000) 2019-10-13T15:32:55.078 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:32:55.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222690 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:32:55.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:32:55.717+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.722769+0000) 2019-10-13T15:32:55.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:32:55.717+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.722862+0000) 2019-10-13T15:32:55.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:32:55.729+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.734692+0000) 2019-10-13T15:32:56.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:32:56.269+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:56.274662+0000) 2019-10-13T15:33:00.081 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:00.073+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.078936+0000) 2019-10-13T15:33:00.082 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:00.077+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.079571+0000) 2019-10-13T15:33:00.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:00.077+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.080064+0000) 2019-10-13T15:33:00.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:00.077+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.081203+0000) 2019-10-13T15:33:00.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:00.076+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.081692+0000) 2019-10-13T15:33:00.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:00.077+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.082375+0000) 2019-10-13T15:33:00.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:00.077+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.082832+0000) 2019-10-13T15:33:00.089 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:00.084+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:00.086983+0000) 2019-10-13T15:33:00.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:00.085+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:02.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:02.709+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.712611+0000) 2019-10-13T15:33:03.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:03.445+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:03.448840+0000) 2019-10-13T15:33:03.700 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:03.696+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:03.698309+0000) 2019-10-13T15:33:04.962 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:04.960+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:04.960381+0000) 2019-10-13T15:33:05.096 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:05.089+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.093668+0000) 2019-10-13T15:33:05.096 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:05.089+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.094049+0000) 2019-10-13T15:33:05.097 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:05.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.094601+0000) 2019-10-13T15:33:05.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:05.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.094961+0000) 2019-10-13T15:33:05.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:05.093+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.095379+0000) 2019-10-13T15:33:05.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:05.092+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.096742+0000) 2019-10-13T15:33:05.100 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:05.093+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.097956+0000) 2019-10-13T15:33:05.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:05.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:05.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:05.100+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.105843+0000) 2019-10-13T15:33:05.725 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:05.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.722954+0000) 2019-10-13T15:33:05.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:05.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.723041+0000) 2019-10-13T15:33:05.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:05.733+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.734888+0000) 2019-10-13T15:33:06.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:06.269+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:06.274904+0000) 2019-10-13T15:33:10.111 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:10.104+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.108543+0000) 2019-10-13T15:33:10.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:10.104+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.109567+0000) 2019-10-13T15:33:10.112 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:10.104+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.110244+0000) 2019-10-13T15:33:10.113 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:10.104+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.110492+0000) 2019-10-13T15:33:10.113 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:10.104+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.110860+0000) 2019-10-13T15:33:10.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:10.107+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.112051+0000) 2019-10-13T15:33:10.114 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:10.108+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.112223+0000) 2019-10-13T15:33:10.121 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:10.115+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:10.118834+0000) 2019-10-13T15:33:10.122 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:10.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:10.859 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:33:10.860 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:10.863 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:33:10.863 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:12.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:12.708+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.712810+0000) 2019-10-13T15:33:13.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:13.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:13.449079+0000) 2019-10-13T15:33:13.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:13.695+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:13.698482+0000) 2019-10-13T15:33:14.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:14.963+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:14.966793+0000) 2019-10-13T15:33:15.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:15.119+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.122600+0000) 2019-10-13T15:33:15.127 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:15.120+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.124118+0000) 2019-10-13T15:33:15.127 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:15.120+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.124510+0000) 2019-10-13T15:33:15.127 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:15.120+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.124617+0000) 2019-10-13T15:33:15.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:15.120+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.125563+0000) 2019-10-13T15:33:15.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:15.120+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.126245+0000) 2019-10-13T15:33:15.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:15.120+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.126468+0000) 2019-10-13T15:33:15.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:15.123+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.129983+0000) 2019-10-13T15:33:15.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:15.128+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:15.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:15.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.723142+0000) 2019-10-13T15:33:15.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:15.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.723237+0000) 2019-10-13T15:33:15.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:15.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.735062+0000) 2019-10-13T15:33:16.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:16.268+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:16.275131+0000) 2019-10-13T15:33:20.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:20.131+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.137578+0000) 2019-10-13T15:33:20.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:20.132+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.137982+0000) 2019-10-13T15:33:20.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:20.132+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.138264+0000) 2019-10-13T15:33:20.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:20.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.138874+0000) 2019-10-13T15:33:20.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:20.136+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.139367+0000) 2019-10-13T15:33:20.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:20.136+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.140248+0000) 2019-10-13T15:33:20.143 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:20.136+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.140656+0000) 2019-10-13T15:33:20.148 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:20.139+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:20.146041+0000) 2019-10-13T15:33:20.151 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:20.144+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:22.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:22.712+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.713014+0000) 2019-10-13T15:33:23.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:23.444+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:23.449271+0000) 2019-10-13T15:33:23.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:23.695+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:23.698681+0000) 2019-10-13T15:33:24.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:24.971+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:24.973160+0000) 2019-10-13T15:33:25.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:25.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.151497+0000) 2019-10-13T15:33:25.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:25.148+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.151728+0000) 2019-10-13T15:33:25.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:25.148+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.152741+0000) 2019-10-13T15:33:25.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:25.148+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.153161+0000) 2019-10-13T15:33:25.156 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:25.147+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.153728+0000) 2019-10-13T15:33:25.157 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:25.148+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.154340+0000) 2019-10-13T15:33:25.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:25.148+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.154651+0000) 2019-10-13T15:33:25.163 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:25.155+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.160796+0000) 2019-10-13T15:33:25.167 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:25.160+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222887 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:25.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:25.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.723343+0000) 2019-10-13T15:33:25.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:25.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.723424+0000) 2019-10-13T15:33:25.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:25.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.735199+0000) 2019-10-13T15:33:26.277 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:26.272+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:26.275358+0000) 2019-10-13T15:33:30.171 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:30.164+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.168379+0000) 2019-10-13T15:33:30.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:30.164+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.169083+0000) 2019-10-13T15:33:30.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:30.163+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.169533+0000) 2019-10-13T15:33:30.172 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:30.164+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.170032+0000) 2019-10-13T15:33:30.174 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:30.164+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.171088+0000) 2019-10-13T15:33:30.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:30.164+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.170034+0000) 2019-10-13T15:33:30.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:30.164+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.170598+0000) 2019-10-13T15:33:30.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:30.171+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:30.177860+0000) 2019-10-13T15:33:30.181 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:30.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222919 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:32.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:32.712+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.713229+0000) 2019-10-13T15:33:33.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:33.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:33.449462+0000) 2019-10-13T15:33:33.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:33.695+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:33.698860+0000) 2019-10-13T15:33:34.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:34.983+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:34.984939+0000) 2019-10-13T15:33:35.184 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:35.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.182023+0000) 2019-10-13T15:33:35.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:35.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.183138+0000) 2019-10-13T15:33:35.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:35.180+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.183751+0000) 2019-10-13T15:33:35.186 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:35.180+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.184355+0000) 2019-10-13T15:33:35.187 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:35.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.184449+0000) 2019-10-13T15:33:35.188 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:35.180+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.185556+0000) 2019-10-13T15:33:35.188 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:35.180+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.185249+0000) 2019-10-13T15:33:35.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:35.187+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.191093+0000) 2019-10-13T15:33:35.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:35.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:35.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:35.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.723553+0000) 2019-10-13T15:33:35.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:35.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.723562+0000) 2019-10-13T15:33:35.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:35.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.735373+0000) 2019-10-13T15:33:36.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:36.272+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:36.275573+0000) 2019-10-13T15:33:40.200 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:40.192+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.197100+0000) 2019-10-13T15:33:40.201 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:40.192+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.198050+0000) 2019-10-13T15:33:40.201 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:40.195+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.198442+0000) 2019-10-13T15:33:40.201 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:40.192+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.198930+0000) 2019-10-13T15:33:40.202 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:40.196+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.199489+0000) 2019-10-13T15:33:40.202 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:40.196+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.199757+0000) 2019-10-13T15:33:40.202 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:40.196+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.199985+0000) 2019-10-13T15:33:40.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:40.199+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:40.204036+0000) 2019-10-13T15:33:40.211 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:40.204+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 222983 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:40.907 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:33:40.907 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:40.911 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:33:40.911 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:42.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:42.712+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.713460+0000) 2019-10-13T15:33:43.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:43.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:43.449668+0000) 2019-10-13T15:33:43.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:43.695+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:43.699081+0000) 2019-10-13T15:33:44.994 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:44.991+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:44.991501+0000) 2019-10-13T15:33:45.214 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:45.208+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.211768+0000) 2019-10-13T15:33:45.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:45.208+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.212084+0000) 2019-10-13T15:33:45.215 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:45.207+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.212852+0000) 2019-10-13T15:33:45.216 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:45.208+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.213458+0000) 2019-10-13T15:33:45.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:45.208+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.213890+0000) 2019-10-13T15:33:45.217 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:45.208+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.214943+0000) 2019-10-13T15:33:45.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:45.208+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.215247+0000) 2019-10-13T15:33:45.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:45.215+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.220601+0000) 2019-10-13T15:33:45.225 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:45.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223017 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:45.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:45.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.723757+0000) 2019-10-13T15:33:45.726 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:45.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.723738+0000) 2019-10-13T15:33:45.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:45.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.735580+0000) 2019-10-13T15:33:46.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:46.272+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:46.275786+0000) 2019-10-13T15:33:50.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:50.220+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.225780+0000) 2019-10-13T15:33:50.229 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:50.220+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.226239+0000) 2019-10-13T15:33:50.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:50.220+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.226667+0000) 2019-10-13T15:33:50.230 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:50.223+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.227135+0000) 2019-10-13T15:33:50.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:50.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.227666+0000) 2019-10-13T15:33:50.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:50.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.228379+0000) 2019-10-13T15:33:50.231 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:50.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.228769+0000) 2019-10-13T15:33:50.239 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:50.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223049 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:50.241 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:50.235+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:50.238913+0000) 2019-10-13T15:33:52.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:52.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.713670+0000) 2019-10-13T15:33:53.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:53.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:53.449871+0000) 2019-10-13T15:33:53.701 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:53.695+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:53.699273+0000) 2019-10-13T15:33:55.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:54.999+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.002661+0000) 2019-10-13T15:33:55.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:33:55.235+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.239756+0000) 2019-10-13T15:33:55.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:55.235+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.240158+0000) 2019-10-13T15:33:55.244 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:55.235+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.241597+0000) 2019-10-13T15:33:55.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:55.235+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.242059+0000) 2019-10-13T15:33:55.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:33:55.239+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.242803+0000) 2019-10-13T15:33:55.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:33:55.239+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.243433+0000) 2019-10-13T15:33:55.246 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:55.239+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.243745+0000) 2019-10-13T15:33:55.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:33:55.243+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.249697+0000) 2019-10-13T15:33:55.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:33:55.243+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:33:55.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:33:55.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.723963+0000) 2019-10-13T15:33:55.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:33:55.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.723950+0000) 2019-10-13T15:33:55.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:33:55.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.735778+0000) 2019-10-13T15:33:56.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:33:56.271+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:56.275986+0000) 2019-10-13T15:34:00.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:00.247+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.254376+0000) 2019-10-13T15:34:00.257 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:00.247+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.254706+0000) 2019-10-13T15:34:00.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:00.247+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.255111+0000) 2019-10-13T15:34:00.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:00.251+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.256024+0000) 2019-10-13T15:34:00.258 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:00.251+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.256463+0000) 2019-10-13T15:34:00.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:00.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.256871+0000) 2019-10-13T15:34:00.259 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:00.251+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.257443+0000) 2019-10-13T15:34:00.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:00.259+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:00.265033+0000) 2019-10-13T15:34:00.268 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:00.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:02.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:02.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.713912+0000) 2019-10-13T15:34:03.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:03.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:03.450067+0000) 2019-10-13T15:34:03.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:03.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:03.699473+0000) 2019-10-13T15:34:05.011 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:05.002+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.009395+0000) 2019-10-13T15:34:05.272 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:05.263+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.269559+0000) 2019-10-13T15:34:05.273 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:05.263+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.270217+0000) 2019-10-13T15:34:05.273 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:05.263+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.270168+0000) 2019-10-13T15:34:05.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:05.263+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.271222+0000) 2019-10-13T15:34:05.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:05.267+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.271870+0000) 2019-10-13T15:34:05.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:05.266+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.272378+0000) 2019-10-13T15:34:05.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:05.267+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.272473+0000) 2019-10-13T15:34:05.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:05.270+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.276821+0000) 2019-10-13T15:34:05.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:05.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223146 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:05.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:05.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.724172+0000) 2019-10-13T15:34:05.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:05.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.724175+0000) 2019-10-13T15:34:05.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:05.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.735988+0000) 2019-10-13T15:34:06.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:06.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:06.276197+0000) 2019-10-13T15:34:10.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:10.275+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.282914+0000) 2019-10-13T15:34:10.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:10.279+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.283580+0000) 2019-10-13T15:34:10.286 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:10.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.284064+0000) 2019-10-13T15:34:10.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:10.279+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.284752+0000) 2019-10-13T15:34:10.287 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:10.279+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.284956+0000) 2019-10-13T15:34:10.288 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:10.279+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.286075+0000) 2019-10-13T15:34:10.288 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:10.279+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.286562+0000) 2019-10-13T15:34:10.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:10.286+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:10.292298+0000) 2019-10-13T15:34:10.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:10.287+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223178 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:10.964 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:34:10.965 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:10.968 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:34:10.969 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:12.716 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:12.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.714164+0000) 2019-10-13T15:34:13.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:13.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:13.450280+0000) 2019-10-13T15:34:13.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:13.694+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:13.699651+0000) 2019-10-13T15:34:15.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:15.010+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.016020+0000) 2019-10-13T15:34:15.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:15.291+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.298074+0000) 2019-10-13T15:34:15.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:15.291+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.298769+0000) 2019-10-13T15:34:15.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:15.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.299447+0000) 2019-10-13T15:34:15.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:15.295+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.299572+0000) 2019-10-13T15:34:15.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:15.295+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.300458+0000) 2019-10-13T15:34:15.303 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:15.295+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.300793+0000) 2019-10-13T15:34:15.304 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:15.295+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.301587+0000) 2019-10-13T15:34:15.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:15.298+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.304345+0000) 2019-10-13T15:34:15.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:15.303+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:15.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:15.719+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.724363+0000) 2019-10-13T15:34:15.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:15.719+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.724371+0000) 2019-10-13T15:34:15.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:15.731+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.736175+0000) 2019-10-13T15:34:16.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:16.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:16.276427+0000) 2019-10-13T15:34:20.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:20.307+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.312137+0000) 2019-10-13T15:34:20.316 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:20.307+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.313542+0000) 2019-10-13T15:34:20.317 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:20.310+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.314607+0000) 2019-10-13T15:34:20.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:20.307+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.314782+0000) 2019-10-13T15:34:20.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:20.307+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.315141+0000) 2019-10-13T15:34:20.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:20.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.315744+0000) 2019-10-13T15:34:20.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:20.311+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.315871+0000) 2019-10-13T15:34:20.322 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:20.314+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:20.320435+0000) 2019-10-13T15:34:20.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:20.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:22.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:22.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.714358+0000) 2019-10-13T15:34:23.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:23.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:23.450482+0000) 2019-10-13T15:34:23.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:23.698+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:23.699828+0000) 2019-10-13T15:34:25.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:25.018+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.022575+0000) 2019-10-13T15:34:25.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:25.323+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.326751+0000) 2019-10-13T15:34:25.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:25.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.328369+0000) 2019-10-13T15:34:25.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:25.322+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.329004+0000) 2019-10-13T15:34:25.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:25.327+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.329465+0000) 2019-10-13T15:34:25.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:25.327+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.329911+0000) 2019-10-13T15:34:25.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:25.327+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.330526+0000) 2019-10-13T15:34:25.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:25.327+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.331128+0000) 2019-10-13T15:34:25.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:25.326+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.333606+0000) 2019-10-13T15:34:25.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:25.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:25.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:25.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.724552+0000) 2019-10-13T15:34:25.727 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:25.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.724573+0000) 2019-10-13T15:34:25.738 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:25.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.736326+0000) 2019-10-13T15:34:26.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:26.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:26.276609+0000) 2019-10-13T15:34:30.343 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:30.334+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.340961+0000) 2019-10-13T15:34:30.344 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:30.339+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.341549+0000) 2019-10-13T15:34:30.344 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:30.339+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.341889+0000) 2019-10-13T15:34:30.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:30.339+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.342574+0000) 2019-10-13T15:34:30.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:30.339+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.343027+0000) 2019-10-13T15:34:30.346 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:30.343+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.343411+0000) 2019-10-13T15:34:30.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:30.343+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.343988+0000) 2019-10-13T15:34:30.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:30.346+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:30.350549+0000) 2019-10-13T15:34:30.355 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:30.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:32.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:32.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.714591+0000) 2019-10-13T15:34:33.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:33.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:33.450694+0000) 2019-10-13T15:34:33.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:33.698+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:33.699997+0000) 2019-10-13T15:34:35.031 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:35.026+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.029064+0000) 2019-10-13T15:34:35.358 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:35.355+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.355629+0000) 2019-10-13T15:34:35.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:35.355+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.356013+0000) 2019-10-13T15:34:35.359 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:35.355+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.356322+0000) 2019-10-13T15:34:35.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:35.355+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.357183+0000) 2019-10-13T15:34:35.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:35.355+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.357307+0000) 2019-10-13T15:34:35.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:35.355+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.357996+0000) 2019-10-13T15:34:35.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:35.354+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.358603+0000) 2019-10-13T15:34:35.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:35.358+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.364051+0000) 2019-10-13T15:34:35.370 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:35.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:35.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:35.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.724730+0000) 2019-10-13T15:34:35.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:35.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.724754+0000) 2019-10-13T15:34:35.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:35.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.736505+0000) 2019-10-13T15:34:36.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:36.274+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:36.276823+0000) 2019-10-13T15:34:40.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:40.366+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.370504+0000) 2019-10-13T15:34:40.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:40.370+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.371655+0000) 2019-10-13T15:34:40.374 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:40.370+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.372171+0000) 2019-10-13T15:34:40.375 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:40.370+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.372903+0000) 2019-10-13T15:34:40.376 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:40.370+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.373552+0000) 2019-10-13T15:34:40.376 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:40.370+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.374319+0000) 2019-10-13T15:34:40.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:40.370+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.374961+0000) 2019-10-13T15:34:40.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:40.374+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:40.379446+0000) 2019-10-13T15:34:40.386 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:40.382+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:41.021 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:34:41.022 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:41.025 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:34:41.026 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:42.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:42.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.714784+0000) 2019-10-13T15:34:43.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:43.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:43.450885+0000) 2019-10-13T15:34:43.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:43.698+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:43.700193+0000) 2019-10-13T15:34:45.038 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:45.034+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.035575+0000) 2019-10-13T15:34:45.389 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:45.382+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.386891+0000) 2019-10-13T15:34:45.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:45.382+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.387206+0000) 2019-10-13T15:34:45.390 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:45.386+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.387647+0000) 2019-10-13T15:34:45.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:45.386+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.388205+0000) 2019-10-13T15:34:45.391 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:45.386+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.388779+0000) 2019-10-13T15:34:45.391 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:45.386+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.389089+0000) 2019-10-13T15:34:45.392 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:45.382+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.389496+0000) 2019-10-13T15:34:45.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:45.390+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.396241+0000) 2019-10-13T15:34:45.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:45.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:45.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:45.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.724944+0000) 2019-10-13T15:34:45.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:45.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.724946+0000) 2019-10-13T15:34:45.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:45.734+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.736713+0000) 2019-10-13T15:34:46.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:46.274+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:46.277002+0000) 2019-10-13T15:34:50.404 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:50.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.401149+0000) 2019-10-13T15:34:50.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:50.398+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.401570+0000) 2019-10-13T15:34:50.404 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:50.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.401870+0000) 2019-10-13T15:34:50.405 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:50.402+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.402580+0000) 2019-10-13T15:34:50.405 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:50.402+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.403489+0000) 2019-10-13T15:34:50.406 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:50.402+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.403418+0000) 2019-10-13T15:34:50.406 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:50.402+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.404500+0000) 2019-10-13T15:34:50.424 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:50.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223440 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:50.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:50.406+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:50.409158+0000) 2019-10-13T15:34:52.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:52.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.714983+0000) 2019-10-13T15:34:53.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:53.446+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:53.451086+0000) 2019-10-13T15:34:53.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:53.698+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:53.700370+0000) 2019-10-13T15:34:55.044 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:55.038+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.041849+0000) 2019-10-13T15:34:55.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:55.414+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.417371+0000) 2019-10-13T15:34:55.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:34:55.414+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.418164+0000) 2019-10-13T15:34:55.421 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:34:55.418+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.419536+0000) 2019-10-13T15:34:55.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:34:55.418+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.420089+0000) 2019-10-13T15:34:55.422 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:55.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.420279+0000) 2019-10-13T15:34:55.423 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:55.418+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.421014+0000) 2019-10-13T15:34:55.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:55.418+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.421288+0000) 2019-10-13T15:34:55.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:34:55.422+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.425250+0000) 2019-10-13T15:34:55.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:34:55.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:34:55.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:34:55.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.725166+0000) 2019-10-13T15:34:55.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:34:55.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.725166+0000) 2019-10-13T15:34:55.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:34:55.734+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.736891+0000) 2019-10-13T15:34:56.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:34:56.274+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:56.277187+0000) 2019-10-13T15:35:00.436 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:00.430+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.433582+0000) 2019-10-13T15:35:00.436 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:00.430+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.433337+0000) 2019-10-13T15:35:00.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:00.430+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.434867+0000) 2019-10-13T15:35:00.437 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:00.430+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.435280+0000) 2019-10-13T15:35:00.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:00.434+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.436053+0000) 2019-10-13T15:35:00.438 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:00.434+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.436290+0000) 2019-10-13T15:35:00.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:00.433+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.436889+0000) 2019-10-13T15:35:00.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:00.437+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:00.441026+0000) 2019-10-13T15:35:00.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:00.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:02.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:02.710+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.715182+0000) 2019-10-13T15:35:03.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:03.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:03.451349+0000) 2019-10-13T15:35:03.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:03.697+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:03.700575+0000) 2019-10-13T15:35:05.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:05.045+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.048258+0000) 2019-10-13T15:35:05.451 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:05.446+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.448879+0000) 2019-10-13T15:35:05.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:05.446+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.449669+0000) 2019-10-13T15:35:05.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:05.446+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.450318+0000) 2019-10-13T15:35:05.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:05.446+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.450842+0000) 2019-10-13T15:35:05.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:05.449+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.451281+0000) 2019-10-13T15:35:05.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:05.450+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.451487+0000) 2019-10-13T15:35:05.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:05.450+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.452019+0000) 2019-10-13T15:35:05.461 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:05.457+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.459028+0000) 2019-10-13T15:35:05.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:05.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:05.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:05.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.725395+0000) 2019-10-13T15:35:05.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:05.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.725387+0000) 2019-10-13T15:35:05.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:05.734+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.737062+0000) 2019-10-13T15:35:06.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:06.274+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:06.277360+0000) 2019-10-13T15:35:10.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:10.462+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.464189+0000) 2019-10-13T15:35:10.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:10.462+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.464671+0000) 2019-10-13T15:35:10.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:10.462+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.464974+0000) 2019-10-13T15:35:10.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:10.462+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.465542+0000) 2019-10-13T15:35:10.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:10.462+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.466475+0000) 2019-10-13T15:35:10.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:10.462+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.467177+0000) 2019-10-13T15:35:10.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:10.465+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.468076+0000) 2019-10-13T15:35:10.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:10.473+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:10.474154+0000) 2019-10-13T15:35:10.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:10.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:11.070 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:35:11.070 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:11.074 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:35:11.074 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:12.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:12.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.715384+0000) 2019-10-13T15:35:13.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:13.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:13.451545+0000) 2019-10-13T15:35:13.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:13.697+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:13.700749+0000) 2019-10-13T15:35:15.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:15.053+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.054685+0000) 2019-10-13T15:35:15.481 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:15.474+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.478420+0000) 2019-10-13T15:35:15.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:15.474+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.479266+0000) 2019-10-13T15:35:15.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:15.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.479873+0000) 2019-10-13T15:35:15.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:15.478+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.480300+0000) 2019-10-13T15:35:15.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:15.478+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.481057+0000) 2019-10-13T15:35:15.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:15.478+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.481461+0000) 2019-10-13T15:35:15.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:15.478+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.481982+0000) 2019-10-13T15:35:15.489 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:15.485+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.487182+0000) 2019-10-13T15:35:15.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:15.486+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:15.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:15.722+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.725595+0000) 2019-10-13T15:35:15.728 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:15.722+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.725595+0000) 2019-10-13T15:35:15.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:15.734+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.737248+0000) 2019-10-13T15:35:16.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:16.274+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:16.277564+0000) 2019-10-13T15:35:20.496 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:20.489+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.493622+0000) 2019-10-13T15:35:20.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:20.489+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.494553+0000) 2019-10-13T15:35:20.497 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:20.493+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.495454+0000) 2019-10-13T15:35:20.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:20.489+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.495145+0000) 2019-10-13T15:35:20.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:20.493+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.496161+0000) 2019-10-13T15:35:20.499 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:20.493+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.495805+0000) 2019-10-13T15:35:20.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:20.493+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.496905+0000) 2019-10-13T15:35:20.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:20.501+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:20.503664+0000) 2019-10-13T15:35:20.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:20.501+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:22.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:22.713+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.715620+0000) 2019-10-13T15:35:23.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:23.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:23.451743+0000) 2019-10-13T15:35:23.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:23.697+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:23.700986+0000) 2019-10-13T15:35:25.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:25.061+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.064279+0000) 2019-10-13T15:35:25.511 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:25.505+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.509002+0000) 2019-10-13T15:35:25.512 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:25.505+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.509294+0000) 2019-10-13T15:35:25.513 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:25.505+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.510437+0000) 2019-10-13T15:35:25.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:25.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.510949+0000) 2019-10-13T15:35:25.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:25.509+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.511538+0000) 2019-10-13T15:35:25.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:25.509+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.511730+0000) 2019-10-13T15:35:25.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:25.509+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.512367+0000) 2019-10-13T15:35:25.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:25.517+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.518653+0000) 2019-10-13T15:35:25.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:25.517+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223673 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:25.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:25.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.725807+0000) 2019-10-13T15:35:25.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:25.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.725808+0000) 2019-10-13T15:35:25.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:25.733+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.737421+0000) 2019-10-13T15:35:26.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:26.273+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:26.277742+0000) 2019-10-13T15:35:30.527 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:30.521+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.524796+0000) 2019-10-13T15:35:30.528 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:30.521+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.525431+0000) 2019-10-13T15:35:30.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:30.521+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.526441+0000) 2019-10-13T15:35:30.529 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:30.521+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.527004+0000) 2019-10-13T15:35:30.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:30.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.526294+0000) 2019-10-13T15:35:30.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:30.525+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.527289+0000) 2019-10-13T15:35:30.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:30.525+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.527688+0000) 2019-10-13T15:35:30.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:30.529+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:30.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:30.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:30.534637+0000) 2019-10-13T15:35:32.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:32.713+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.715823+0000) 2019-10-13T15:35:33.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:33.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:33.451937+0000) 2019-10-13T15:35:33.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:33.697+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:33.701168+0000) 2019-10-13T15:35:35.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:35.069+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.070589+0000) 2019-10-13T15:35:35.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:35.533+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.537567+0000) 2019-10-13T15:35:35.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:35.537+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.538578+0000) 2019-10-13T15:35:35.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:35.533+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.539066+0000) 2019-10-13T15:35:35.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:35.537+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.540027+0000) 2019-10-13T15:35:35.542 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:35.537+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.540284+0000) 2019-10-13T15:35:35.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:35.537+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.540627+0000) 2019-10-13T15:35:35.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:35.537+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.541295+0000) 2019-10-13T15:35:35.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:35.541+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.545613+0000) 2019-10-13T15:35:35.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:35.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:35.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:35.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.725989+0000) 2019-10-13T15:35:35.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:35.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.725989+0000) 2019-10-13T15:35:35.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:35.733+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.737627+0000) 2019-10-13T15:35:36.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:36.277+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:36.277920+0000) 2019-10-13T15:35:40.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:40.549+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.552417+0000) 2019-10-13T15:35:40.555 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:40.549+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.552777+0000) 2019-10-13T15:35:40.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:40.549+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.553616+0000) 2019-10-13T15:35:40.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:40.549+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.554072+0000) 2019-10-13T15:35:40.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:40.549+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.554403+0000) 2019-10-13T15:35:40.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:40.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.554822+0000) 2019-10-13T15:35:40.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:40.549+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.554964+0000) 2019-10-13T15:35:40.564 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:40.557+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:40.561726+0000) 2019-10-13T15:35:40.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:40.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:41.204 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:35:41.205 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:41.208 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:35:41.209 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:42.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:42.713+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.716049+0000) 2019-10-13T15:35:43.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:43.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:43.452165+0000) 2019-10-13T15:35:43.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:43.701+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:43.701370+0000) 2019-10-13T15:35:45.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:45.073+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.077180+0000) 2019-10-13T15:35:45.569 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:45.561+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.566836+0000) 2019-10-13T15:35:45.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:45.565+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.568092+0000) 2019-10-13T15:35:45.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:45.565+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.568452+0000) 2019-10-13T15:35:45.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:45.565+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.568952+0000) 2019-10-13T15:35:45.571 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:45.565+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.569516+0000) 2019-10-13T15:35:45.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:45.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.570004+0000) 2019-10-13T15:35:45.573 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:45.565+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.570381+0000) 2019-10-13T15:35:45.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:45.573+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.574202+0000) 2019-10-13T15:35:45.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:45.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:45.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:45.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.726185+0000) 2019-10-13T15:35:45.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:45.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.726189+0000) 2019-10-13T15:35:45.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:45.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.737797+0000) 2019-10-13T15:35:46.280 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:46.277+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:46.278122+0000) 2019-10-13T15:35:50.585 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:50.577+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.582322+0000) 2019-10-13T15:35:50.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:50.581+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.583359+0000) 2019-10-13T15:35:50.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:50.581+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.584663+0000) 2019-10-13T15:35:50.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:50.581+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.584798+0000) 2019-10-13T15:35:50.588 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:50.580+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.585175+0000) 2019-10-13T15:35:50.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:50.581+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.585948+0000) 2019-10-13T15:35:50.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:50.581+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.586466+0000) 2019-10-13T15:35:50.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:50.588+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:50.590258+0000) 2019-10-13T15:35:50.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:50.589+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:52.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:52.713+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.716252+0000) 2019-10-13T15:35:53.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:53.449+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:53.452356+0000) 2019-10-13T15:35:53.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:53.696+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:53.701583+0000) 2019-10-13T15:35:55.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:55.080+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.083608+0000) 2019-10-13T15:35:55.600 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:35:55.593+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.597288+0000) 2019-10-13T15:35:55.603 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:35:55.596+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.599966+0000) 2019-10-13T15:35:55.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:55.593+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.598560+0000) 2019-10-13T15:35:55.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:55.593+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.598910+0000) 2019-10-13T15:35:55.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:55.597+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.599509+0000) 2019-10-13T15:35:55.603 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:35:55.597+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.600244+0000) 2019-10-13T15:35:55.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:55.597+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.600942+0000) 2019-10-13T15:35:55.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:35:55.604+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.607139+0000) 2019-10-13T15:35:55.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:35:55.605+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:35:55.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:35:55.721+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.726367+0000) 2019-10-13T15:35:55.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:35:55.721+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.726366+0000) 2019-10-13T15:35:55.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:35:55.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.737980+0000) 2019-10-13T15:35:56.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:35:56.273+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:56.278332+0000) 2019-10-13T15:36:00.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:00.609+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.614317+0000) 2019-10-13T15:36:00.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:00.609+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.614755+0000) 2019-10-13T15:36:00.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:00.612+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.615240+0000) 2019-10-13T15:36:00.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:00.613+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.615670+0000) 2019-10-13T15:36:00.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:00.613+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.616444+0000) 2019-10-13T15:36:00.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:00.613+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.616850+0000) 2019-10-13T15:36:00.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:00.613+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.617287+0000) 2019-10-13T15:36:00.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:00.620+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:00.622487+0000) 2019-10-13T15:36:00.627 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:00.621+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223900 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:02.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:02.713+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.716451+0000) 2019-10-13T15:36:03.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:03.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:03.452548+0000) 2019-10-13T15:36:03.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:03.700+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:03.701795+0000) 2019-10-13T15:36:05.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:05.084+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.090052+0000) 2019-10-13T15:36:05.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:05.624+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.628103+0000) 2019-10-13T15:36:05.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:05.624+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.629736+0000) 2019-10-13T15:36:05.632 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:05.624+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.630018+0000) 2019-10-13T15:36:05.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:05.624+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.630424+0000) 2019-10-13T15:36:05.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:05.624+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.630887+0000) 2019-10-13T15:36:05.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:05.628+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.631748+0000) 2019-10-13T15:36:05.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:05.628+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.632681+0000) 2019-10-13T15:36:05.637 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:05.632+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.634852+0000) 2019-10-13T15:36:05.642 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:05.636+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223937 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:05.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:05.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.726509+0000) 2019-10-13T15:36:05.729 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:05.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.726542+0000) 2019-10-13T15:36:05.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:05.736+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.738190+0000) 2019-10-13T15:36:06.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:06.272+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:06.278518+0000) 2019-10-13T15:36:10.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:10.636+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.643149+0000) 2019-10-13T15:36:10.646 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:10.640+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.643450+0000) 2019-10-13T15:36:10.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:10.640+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.643971+0000) 2019-10-13T15:36:10.647 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:10.640+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.645128+0000) 2019-10-13T15:36:10.648 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:10.640+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.645587+0000) 2019-10-13T15:36:10.648 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:10.640+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.645896+0000) 2019-10-13T15:36:10.648 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:10.640+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.646156+0000) 2019-10-13T15:36:10.653 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:10.648+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:10.650985+0000) 2019-10-13T15:36:10.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:10.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 223967 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:11.316 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:36:11.316 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:11.320 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:36:11.320 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:12.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:12.712+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.716649+0000) 2019-10-13T15:36:13.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:13.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:13.452740+0000) 2019-10-13T15:36:13.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:13.696+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:13.702019+0000) 2019-10-13T15:36:15.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:15.092+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.096429+0000) 2019-10-13T15:36:15.660 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:15.652+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.657765+0000) 2019-10-13T15:36:15.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:15.652+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.658156+0000) 2019-10-13T15:36:15.661 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:15.652+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.658961+0000) 2019-10-13T15:36:15.661 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:15.652+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.658393+0000) 2019-10-13T15:36:15.662 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:15.656+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.659757+0000) 2019-10-13T15:36:15.662 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:15.656+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.660369+0000) 2019-10-13T15:36:15.663 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:15.656+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.660782+0000) 2019-10-13T15:36:15.669 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:15.664+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.667384+0000) 2019-10-13T15:36:15.671 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:15.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:15.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:15.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.726720+0000) 2019-10-13T15:36:15.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:15.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.726746+0000) 2019-10-13T15:36:15.740 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:15.736+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.738366+0000) 2019-10-13T15:36:16.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:16.272+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:16.278701+0000) 2019-10-13T15:36:20.675 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:20.668+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.672787+0000) 2019-10-13T15:36:20.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:20.668+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.672956+0000) 2019-10-13T15:36:20.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:20.668+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.673760+0000) 2019-10-13T15:36:20.677 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:20.668+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.673440+0000) 2019-10-13T15:36:20.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:20.668+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.674103+0000) 2019-10-13T15:36:20.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:20.668+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.674973+0000) 2019-10-13T15:36:20.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:20.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.675400+0000) 2019-10-13T15:36:20.686 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:20.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224031 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:20.687 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:20.680+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:20.684820+0000) 2019-10-13T15:36:22.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:22.716+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.716848+0000) 2019-10-13T15:36:23.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:23.448+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:23.452932+0000) 2019-10-13T15:36:23.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:23.700+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:23.702229+0000) 2019-10-13T15:36:25.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:25.100+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.102990+0000) 2019-10-13T15:36:25.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:25.680+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.687200+0000) 2019-10-13T15:36:25.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:25.684+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.688612+0000) 2019-10-13T15:36:25.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:25.684+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.688880+0000) 2019-10-13T15:36:25.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:25.684+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.689282+0000) 2019-10-13T15:36:25.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:25.684+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.689501+0000) 2019-10-13T15:36:25.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:25.684+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.689848+0000) 2019-10-13T15:36:25.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:25.684+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.690282+0000) 2019-10-13T15:36:25.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:25.692+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.697689+0000) 2019-10-13T15:36:25.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:25.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:25.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:25.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.726931+0000) 2019-10-13T15:36:25.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:25.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.726929+0000) 2019-10-13T15:36:25.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:25.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.738535+0000) 2019-10-13T15:36:26.281 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:26.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:26.278884+0000) 2019-10-13T15:36:30.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:30.696+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.701762+0000) 2019-10-13T15:36:30.705 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:30.696+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.702336+0000) 2019-10-13T15:36:30.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:30.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.702851+0000) 2019-10-13T15:36:30.706 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:30.696+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.703112+0000) 2019-10-13T15:36:30.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:30.700+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.703411+0000) 2019-10-13T15:36:30.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:30.700+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.704020+0000) 2019-10-13T15:36:30.707 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:30.700+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.704377+0000) 2019-10-13T15:36:30.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:30.708+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:30.710934+0000) 2019-10-13T15:36:30.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:30.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:32.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:32.716+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.717054+0000) 2019-10-13T15:36:33.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:33.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:33.453154+0000) 2019-10-13T15:36:33.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:33.700+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:33.702423+0000) 2019-10-13T15:36:35.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:35.108+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.109374+0000) 2019-10-13T15:36:35.718 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:35.712+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.715631+0000) 2019-10-13T15:36:35.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:35.712+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.716261+0000) 2019-10-13T15:36:35.719 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:35.712+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.716741+0000) 2019-10-13T15:36:35.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:35.712+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.717470+0000) 2019-10-13T15:36:35.720 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:35.712+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.717922+0000) 2019-10-13T15:36:35.721 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:35.712+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.718607+0000) 2019-10-13T15:36:35.721 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:35.712+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.718951+0000) 2019-10-13T15:36:35.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:35.720+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.724528+0000) 2019-10-13T15:36:35.729 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:35.720+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224133 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:35.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:35.720+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.727080+0000) 2019-10-13T15:36:35.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:35.720+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.727054+0000) 2019-10-13T15:36:35.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:35.732+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.738679+0000) 2019-10-13T15:36:36.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:36.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:36.279089+0000) 2019-10-13T15:36:40.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:40.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.730430+0000) 2019-10-13T15:36:40.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:40.724+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.730845+0000) 2019-10-13T15:36:40.734 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:40.728+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.731487+0000) 2019-10-13T15:36:40.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:40.728+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.732829+0000) 2019-10-13T15:36:40.735 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:40.728+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.733449+0000) 2019-10-13T15:36:40.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:40.728+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.733774+0000) 2019-10-13T15:36:40.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:40.728+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.734150+0000) 2019-10-13T15:36:40.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:40.736+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:40.739414+0000) 2019-10-13T15:36:40.743 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:40.736+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:41.448 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:36:41.448 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:41.452 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:36:41.452 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:42.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:42.712+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.717282+0000) 2019-10-13T15:36:43.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:43.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:43.453356+0000) 2019-10-13T15:36:43.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:43.700+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:43.702647+0000) 2019-10-13T15:36:45.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:45.112+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.115745+0000) 2019-10-13T15:36:45.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:45.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.727251+0000) 2019-10-13T15:36:45.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:45.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.727251+0000) 2019-10-13T15:36:45.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:45.736+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.738872+0000) 2019-10-13T15:36:45.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:45.740+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.744981+0000) 2019-10-13T15:36:45.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:45.740+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.745533+0000) 2019-10-13T15:36:45.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:45.740+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.746128+0000) 2019-10-13T15:36:45.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:45.740+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.746793+0000) 2019-10-13T15:36:45.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:45.744+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.747626+0000) 2019-10-13T15:36:45.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:45.743+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.748162+0000) 2019-10-13T15:36:45.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:45.744+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.748531+0000) 2019-10-13T15:36:45.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:45.747+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.751722+0000) 2019-10-13T15:36:45.759 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:45.752+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:46.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:46.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:46.279337+0000) 2019-10-13T15:36:50.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:50.755+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.760000+0000) 2019-10-13T15:36:50.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:50.755+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.760268+0000) 2019-10-13T15:36:50.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:50.755+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.760613+0000) 2019-10-13T15:36:50.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:50.755+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.762543+0000) 2019-10-13T15:36:50.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:50.755+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.762855+0000) 2019-10-13T15:36:50.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:50.759+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.764284+0000) 2019-10-13T15:36:50.767 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:50.759+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.764725+0000) 2019-10-13T15:36:50.770 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:50.763+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:50.768062+0000) 2019-10-13T15:36:50.774 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:50.767+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224227 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:52.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:52.715+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.717485+0000) 2019-10-13T15:36:53.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:53.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:53.453551+0000) 2019-10-13T15:36:53.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:53.699+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:53.702839+0000) 2019-10-13T15:36:55.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:55.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.122351+0000) 2019-10-13T15:36:55.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:55.723+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.727499+0000) 2019-10-13T15:36:55.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:55.723+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.727506+0000) 2019-10-13T15:36:55.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:55.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.739075+0000) 2019-10-13T15:36:55.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:36:55.771+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.775657+0000) 2019-10-13T15:36:55.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:36:55.771+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.776256+0000) 2019-10-13T15:36:55.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:36:55.771+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.777247+0000) 2019-10-13T15:36:55.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:36:55.771+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.777642+0000) 2019-10-13T15:36:55.780 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:36:55.771+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.778127+0000) 2019-10-13T15:36:55.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:36:55.771+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.778533+0000) 2019-10-13T15:36:55.781 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:55.775+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.779585+0000) 2019-10-13T15:36:55.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:36:55.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.782354+0000) 2019-10-13T15:36:55.789 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:36:55.783+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:36:56.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:36:56.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:56.279565+0000) 2019-10-13T15:37:00.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:00.783+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.790449+0000) 2019-10-13T15:37:00.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:00.783+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.789927+0000) 2019-10-13T15:37:00.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:00.787+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.791597+0000) 2019-10-13T15:37:00.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:00.787+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.792201+0000) 2019-10-13T15:37:00.795 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:00.787+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.792713+0000) 2019-10-13T15:37:00.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:00.787+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.792966+0000) 2019-10-13T15:37:00.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:00.787+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.793058+0000) 2019-10-13T15:37:00.804 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:00.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224291 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:00.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:00.799+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:00.803576+0000) 2019-10-13T15:37:02.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:02.711+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:02.717715+0000) 2019-10-13T15:37:03.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:03.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.453776+0000) 2019-10-13T15:37:03.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:03.699+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.703047+0000) 2019-10-13T15:37:05.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:05.123+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.128846+0000) 2019-10-13T15:37:05.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:05.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.727734+0000) 2019-10-13T15:37:05.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:05.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.727729+0000) 2019-10-13T15:37:05.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:05.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.739272+0000) 2019-10-13T15:37:05.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:05.799+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.805118+0000) 2019-10-13T15:37:05.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:05.799+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.805720+0000) 2019-10-13T15:37:05.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:05.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.806261+0000) 2019-10-13T15:37:05.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:05.799+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.806408+0000) 2019-10-13T15:37:05.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:05.799+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.806993+0000) 2019-10-13T15:37:05.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:05.803+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.807287+0000) 2019-10-13T15:37:05.811 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:05.803+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.808911+0000) 2019-10-13T15:37:05.816 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:05.807+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.814049+0000) 2019-10-13T15:37:05.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:05.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224329 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:06.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:06.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:06.279756+0000) 2019-10-13T15:37:10.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:10.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.820207+0000) 2019-10-13T15:37:10.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:10.815+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.820683+0000) 2019-10-13T15:37:10.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:10.815+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.821269+0000) 2019-10-13T15:37:10.824 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:10.815+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.821880+0000) 2019-10-13T15:37:10.824 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:10.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.822345+0000) 2019-10-13T15:37:10.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:10.819+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.822843+0000) 2019-10-13T15:37:10.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:10.819+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.823434+0000) 2019-10-13T15:37:10.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:10.823+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:10.827598+0000) 2019-10-13T15:37:10.835 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:10.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:11.580 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:37:11.580 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:37:11.585 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:37:11.585 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:37:12.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:12.715+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:12.717921+0000) 2019-10-13T15:37:13.456 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:13.447+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:13.453977+0000) 2019-10-13T15:37:13.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:13.699+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:13.703208+0000) 2019-10-13T15:37:15.138 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:15.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.135620+0000) 2019-10-13T15:37:15.730 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:15.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.727936+0000) 2019-10-13T15:37:15.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:15.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.727930+0000) 2019-10-13T15:37:15.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:15.739+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.739533+0000) 2019-10-13T15:37:15.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:15.831+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.835704+0000) 2019-10-13T15:37:15.839 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:15.831+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.836187+0000) 2019-10-13T15:37:15.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:15.831+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.837086+0000) 2019-10-13T15:37:15.839 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:15.831+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.837025+0000) 2019-10-13T15:37:15.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:15.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.837907+0000) 2019-10-13T15:37:15.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:15.831+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.838924+0000) 2019-10-13T15:37:15.842 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:15.835+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.839594+0000) 2019-10-13T15:37:15.847 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:15.839+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:15.844620+0000) 2019-10-13T15:37:15.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:15.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:16.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:16.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:16.279943+0000) 2019-10-13T15:37:20.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:20.851+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.851357+0000) 2019-10-13T15:37:20.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:20.851+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.851615+0000) 2019-10-13T15:37:20.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:20.851+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.851960+0000) 2019-10-13T15:37:20.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:20.851+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.852388+0000) 2019-10-13T15:37:20.856 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:20.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.853513+0000) 2019-10-13T15:37:20.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:20.851+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.853738+0000) 2019-10-13T15:37:20.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:20.851+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.854291+0000) 2019-10-13T15:37:20.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:20.855+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:20.860488+0000) 2019-10-13T15:37:20.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:20.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:22.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:22.715+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:22.718135+0000) 2019-10-13T15:37:23.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:23.451+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:23.454191+0000) 2019-10-13T15:37:23.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:23.699+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:23.703402+0000) 2019-10-13T15:37:25.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:25.139+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.142401+0000) 2019-10-13T15:37:25.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:25.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.728147+0000) 2019-10-13T15:37:25.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:25.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.728137+0000) 2019-10-13T15:37:25.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:25.739+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.739734+0000) 2019-10-13T15:37:25.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:25.863+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.866798+0000) 2019-10-13T15:37:25.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:25.863+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.866868+0000) 2019-10-13T15:37:25.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:25.863+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.867259+0000) 2019-10-13T15:37:25.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:25.867+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.867738+0000) 2019-10-13T15:37:25.871 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:25.867+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.868192+0000) 2019-10-13T15:37:25.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:25.863+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.869684+0000) 2019-10-13T15:37:25.872 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:25.867+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.869819+0000) 2019-10-13T15:37:25.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:25.871+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:25.875468+0000) 2019-10-13T15:37:25.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:25.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:26.282 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:26.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:26.280098+0000) 2019-10-13T15:37:30.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:30.878+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.882468+0000) 2019-10-13T15:37:30.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:30.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.883198+0000) 2019-10-13T15:37:30.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:30.882+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.883474+0000) 2019-10-13T15:37:30.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:30.879+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.884729+0000) 2019-10-13T15:37:30.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:30.882+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.885131+0000) 2019-10-13T15:37:30.888 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:30.882+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.886029+0000) 2019-10-13T15:37:30.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:30.882+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.886562+0000) 2019-10-13T15:37:30.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:30.887+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:30.892215+0000) 2019-10-13T15:37:30.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:30.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:32.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:32.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:32.718365+0000) 2019-10-13T15:37:33.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:33.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:33.454387+0000) 2019-10-13T15:37:33.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:33.703+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:33.703595+0000) 2019-10-13T15:37:35.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:35.143+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.148745+0000) 2019-10-13T15:37:35.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:35.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.728309+0000) 2019-10-13T15:37:35.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:35.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.728331+0000) 2019-10-13T15:37:35.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:35.738+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.739962+0000) 2019-10-13T15:37:35.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:35.894+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.897281+0000) 2019-10-13T15:37:35.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:35.894+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.898407+0000) 2019-10-13T15:37:35.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:35.895+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.899154+0000) 2019-10-13T15:37:35.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:35.894+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.898792+0000) 2019-10-13T15:37:35.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:35.898+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.899622+0000) 2019-10-13T15:37:35.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:35.898+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.899879+0000) 2019-10-13T15:37:35.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:35.898+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.900331+0000) 2019-10-13T15:37:35.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:35.903+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:35.907301+0000) 2019-10-13T15:37:35.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:35.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224522 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:36.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:36.278+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:36.280325+0000) 2019-10-13T15:37:40.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:40.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.912054+0000) 2019-10-13T15:37:40.916 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:40.910+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.913691+0000) 2019-10-13T15:37:40.917 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:40.910+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.914897+0000) 2019-10-13T15:37:40.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:40.910+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.915424+0000) 2019-10-13T15:37:40.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:40.914+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.915975+0000) 2019-10-13T15:37:40.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:40.914+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.916122+0000) 2019-10-13T15:37:40.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:40.914+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.916636+0000) 2019-10-13T15:37:40.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:40.914+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:40.919790+0000) 2019-10-13T15:37:40.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:40.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:41.716 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:37:41.716 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:37:41.720 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:37:41.720 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:37:42.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:42.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:42.718568+0000) 2019-10-13T15:37:43.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:43.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:43.454588+0000) 2019-10-13T15:37:43.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:43.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:43.703789+0000) 2019-10-13T15:37:45.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:45.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.155207+0000) 2019-10-13T15:37:45.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:45.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.728514+0000) 2019-10-13T15:37:45.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:45.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.728536+0000) 2019-10-13T15:37:45.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:45.738+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.740158+0000) 2019-10-13T15:37:45.929 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:45.922+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.927101+0000) 2019-10-13T15:37:45.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:45.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.927707+0000) 2019-10-13T15:37:45.931 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:45.926+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.928201+0000) 2019-10-13T15:37:45.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:45.926+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.928563+0000) 2019-10-13T15:37:45.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:45.926+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.929025+0000) 2019-10-13T15:37:45.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:45.926+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.929796+0000) 2019-10-13T15:37:45.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:45.926+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.931019+0000) 2019-10-13T15:37:45.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:45.930+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:45.936509+0000) 2019-10-13T15:37:45.941 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:45.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:46.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:46.278+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:46.280538+0000) 2019-10-13T15:37:50.944 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:50.938+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.942175+0000) 2019-10-13T15:37:50.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:50.938+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.942908+0000) 2019-10-13T15:37:50.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:50.942+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.943772+0000) 2019-10-13T15:37:50.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:50.942+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.944330+0000) 2019-10-13T15:37:50.947 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:50.938+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.944801+0000) 2019-10-13T15:37:50.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:50.942+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.945237+0000) 2019-10-13T15:37:50.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:50.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.945743+0000) 2019-10-13T15:37:50.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:50.946+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:50.950673+0000) 2019-10-13T15:37:50.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:50.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:52.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:52.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:52.718805+0000) 2019-10-13T15:37:53.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:53.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:53.454819+0000) 2019-10-13T15:37:53.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:53.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:53.704008+0000) 2019-10-13T15:37:55.169 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:55.162+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.166766+0000) 2019-10-13T15:37:55.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:55.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.728727+0000) 2019-10-13T15:37:55.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:55.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.728733+0000) 2019-10-13T15:37:55.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:55.738+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.740359+0000) 2019-10-13T15:37:55.959 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:37:55.954+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.957153+0000) 2019-10-13T15:37:55.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:37:55.954+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.958956+0000) 2019-10-13T15:37:55.962 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:37:55.958+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.959589+0000) 2019-10-13T15:37:55.962 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:55.958+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.960087+0000) 2019-10-13T15:37:55.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:37:55.958+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.960423+0000) 2019-10-13T15:37:55.963 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:37:55.954+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.961123+0000) 2019-10-13T15:37:55.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:37:55.958+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.961452+0000) 2019-10-13T15:37:55.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:37:55.958+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:55.965340+0000) 2019-10-13T15:37:55.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:37:55.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:37:56.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:37:56.278+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:56.280728+0000) 2019-10-13T15:38:00.974 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:00.970+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.971734+0000) 2019-10-13T15:38:00.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:00.966+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.972930+0000) 2019-10-13T15:38:00.976 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:00.970+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.973280+0000) 2019-10-13T15:38:00.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:00.970+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.974136+0000) 2019-10-13T15:38:00.977 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:00.970+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.974687+0000) 2019-10-13T15:38:00.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:00.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.975368+0000) 2019-10-13T15:38:00.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:00.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.975924+0000) 2019-10-13T15:38:00.981 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:00.974+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:00.978970+0000) 2019-10-13T15:38:00.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:00.978+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224681 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:02.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:02.714+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:02.719007+0000) 2019-10-13T15:38:03.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:03.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:03.455014+0000) 2019-10-13T15:38:03.706 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:03.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:03.704206+0000) 2019-10-13T15:38:05.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:05.166+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.173161+0000) 2019-10-13T15:38:05.731 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:05.726+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.728930+0000) 2019-10-13T15:38:05.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:05.726+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.728937+0000) 2019-10-13T15:38:05.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:05.738+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.740556+0000) 2019-10-13T15:38:05.988 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:05.982+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.986073+0000) 2019-10-13T15:38:05.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:05.982+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.986967+0000) 2019-10-13T15:38:05.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:05.986+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.987997+0000) 2019-10-13T15:38:05.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:05.986+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.988126+0000) 2019-10-13T15:38:05.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:05.986+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.988495+0000) 2019-10-13T15:38:05.991 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:05.982+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.989060+0000) 2019-10-13T15:38:05.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:05.986+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.990575+0000) 2019-10-13T15:38:05.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:05.990+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:05.995986+0000) 2019-10-13T15:38:06.001 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:05.994+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:06.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:06.278+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:06.280937+0000) 2019-10-13T15:38:11.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:10.998+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.002208+0000) 2019-10-13T15:38:11.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:11.002+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.003390+0000) 2019-10-13T15:38:11.006 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:11.002+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.003655+0000) 2019-10-13T15:38:11.006 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:10.998+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.004443+0000) 2019-10-13T15:38:11.007 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:11.002+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.004644+0000) 2019-10-13T15:38:11.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:11.002+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.005536+0000) 2019-10-13T15:38:11.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:11.002+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.006184+0000) 2019-10-13T15:38:11.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:11.006+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:11.010683+0000) 2019-10-13T15:38:11.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:11.010+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224747 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:11.770 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:38:11.770 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:38:11.774 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:38:11.774 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:38:12.721 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:12.718+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:12.719237+0000) 2019-10-13T15:38:13.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:13.450+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:13.455225+0000) 2019-10-13T15:38:13.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:13.698+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:13.704400+0000) 2019-10-13T15:38:15.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:15.178+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:15.179710+0000) 2019-10-13T15:38:15.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:15.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:15.729158+0000) 2019-10-13T15:38:15.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:15.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:15.729148+0000) 2019-10-13T15:38:15.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:15.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:15.740754+0000) 2019-10-13T15:38:16.019 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:16.013+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.016357+0000) 2019-10-13T15:38:16.019 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:16.013+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.017001+0000) 2019-10-13T15:38:16.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:16.013+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.017053+0000) 2019-10-13T15:38:16.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:16.013+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.017661+0000) 2019-10-13T15:38:16.020 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:16.013+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.018079+0000) 2019-10-13T15:38:16.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:16.014+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.018628+0000) 2019-10-13T15:38:16.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:16.017+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.020514+0000) 2019-10-13T15:38:16.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:16.022+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.026913+0000) 2019-10-13T15:38:16.029 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:16.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:16.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:16.277+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:16.281126+0000) 2019-10-13T15:38:21.032 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:21.025+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.029634+0000) 2019-10-13T15:38:21.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:21.026+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.030538+0000) 2019-10-13T15:38:21.033 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:21.025+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.031205+0000) 2019-10-13T15:38:21.034 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:21.029+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.031609+0000) 2019-10-13T15:38:21.034 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:21.029+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.032083+0000) 2019-10-13T15:38:21.034 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:21.029+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.032295+0000) 2019-10-13T15:38:21.035 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:21.029+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.033280+0000) 2019-10-13T15:38:21.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:21.030+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:21.037630+0000) 2019-10-13T15:38:21.044 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:21.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:22.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:22.717+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:22.719506+0000) 2019-10-13T15:38:23.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:23.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:23.455422+0000) 2019-10-13T15:38:23.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:23.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:23.704602+0000) 2019-10-13T15:38:25.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:25.182+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:25.186071+0000) 2019-10-13T15:38:25.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:25.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:25.729386+0000) 2019-10-13T15:38:25.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:25.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:25.729383+0000) 2019-10-13T15:38:25.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:25.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:25.740962+0000) 2019-10-13T15:38:26.048 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:26.041+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.045305+0000) 2019-10-13T15:38:26.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:26.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.045697+0000) 2019-10-13T15:38:26.049 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:26.041+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.046643+0000) 2019-10-13T15:38:26.050 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:26.045+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.047411+0000) 2019-10-13T15:38:26.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:26.042+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.048836+0000) 2019-10-13T15:38:26.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:26.046+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.052602+0000) 2019-10-13T15:38:26.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:26.045+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.048316+0000) 2019-10-13T15:38:26.063 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:26.045+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.049361+0000) 2019-10-13T15:38:26.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:26.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224847 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:26.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:26.277+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:26.281325+0000) 2019-10-13T15:38:31.062 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:31.057+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.060048+0000) 2019-10-13T15:38:31.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:31.057+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.060565+0000) 2019-10-13T15:38:31.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:31.057+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.060930+0000) 2019-10-13T15:38:31.064 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:31.058+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.061474+0000) 2019-10-13T15:38:31.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:31.057+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.061904+0000) 2019-10-13T15:38:31.064 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:31.057+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.062359+0000) 2019-10-13T15:38:31.065 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:31.057+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.062777+0000) 2019-10-13T15:38:31.072 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:31.070+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:31.070559+0000) 2019-10-13T15:38:31.074 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:31.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:32.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:32.717+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:32.719704+0000) 2019-10-13T15:38:33.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:33.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:33.455615+0000) 2019-10-13T15:38:33.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:33.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:33.704787+0000) 2019-10-13T15:38:35.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:35.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:35.192524+0000) 2019-10-13T15:38:35.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:35.725+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:35.729622+0000) 2019-10-13T15:38:35.732 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:35.725+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:35.729623+0000) 2019-10-13T15:38:35.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:35.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:35.741154+0000) 2019-10-13T15:38:36.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:36.073+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.074810+0000) 2019-10-13T15:38:36.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:36.069+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.075098+0000) 2019-10-13T15:38:36.079 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:36.073+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.077081+0000) 2019-10-13T15:38:36.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:36.073+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.077322+0000) 2019-10-13T15:38:36.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:36.073+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.077887+0000) 2019-10-13T15:38:36.080 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:36.073+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.078241+0000) 2019-10-13T15:38:36.081 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:36.073+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.078651+0000) 2019-10-13T15:38:36.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:36.077+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.081844+0000) 2019-10-13T15:38:36.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:36.085+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224911 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:36.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:36.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:36.281541+0000) 2019-10-13T15:38:41.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:41.089+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.090506+0000) 2019-10-13T15:38:41.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:41.085+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.090662+0000) 2019-10-13T15:38:41.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:41.089+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.091292+0000) 2019-10-13T15:38:41.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:41.089+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.091902+0000) 2019-10-13T15:38:41.100 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:41.089+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.092354+0000) 2019-10-13T15:38:41.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:41.089+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.092745+0000) 2019-10-13T15:38:41.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:41.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.093469+0000) 2019-10-13T15:38:41.106 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:41.097+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:41.099776+0000) 2019-10-13T15:38:41.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:41.097+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:41.897 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:38:41.897 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:38:41.901 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:38:41.901 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:38:42.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:42.717+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:42.719909+0000) 2019-10-13T15:38:43.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:43.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:43.455813+0000) 2019-10-13T15:38:43.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:43.701+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:43.704988+0000) 2019-10-13T15:38:45.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:45.197+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:45.199076+0000) 2019-10-13T15:38:45.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:45.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:45.729825+0000) 2019-10-13T15:38:45.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:45.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:45.729822+0000) 2019-10-13T15:38:45.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:45.737+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:45.741356+0000) 2019-10-13T15:38:46.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:46.101+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.105180+0000) 2019-10-13T15:38:46.108 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:46.101+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.105846+0000) 2019-10-13T15:38:46.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:46.101+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.106186+0000) 2019-10-13T15:38:46.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:46.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.106714+0000) 2019-10-13T15:38:46.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:46.101+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.107260+0000) 2019-10-13T15:38:46.110 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:46.101+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.107008+0000) 2019-10-13T15:38:46.110 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:46.105+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.107617+0000) 2019-10-13T15:38:46.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:46.113+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.115687+0000) 2019-10-13T15:38:46.119 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:46.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 224980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:46.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:46.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:46.281726+0000) 2019-10-13T15:38:51.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:51.117+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.119626+0000) 2019-10-13T15:38:51.123 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:51.117+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.121054+0000) 2019-10-13T15:38:51.123 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:51.117+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.121131+0000) 2019-10-13T15:38:51.124 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:51.117+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.121844+0000) 2019-10-13T15:38:51.125 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:51.121+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.122626+0000) 2019-10-13T15:38:51.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:51.117+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.122255+0000) 2019-10-13T15:38:51.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:51.117+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.122717+0000) 2019-10-13T15:38:51.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:51.125+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:51.129996+0000) 2019-10-13T15:38:51.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:51.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:52.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:52.717+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:52.720128+0000) 2019-10-13T15:38:53.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:53.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:53.456011+0000) 2019-10-13T15:38:53.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:53.701+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:53.705226+0000) 2019-10-13T15:38:55.208 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:55.201+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:55.205488+0000) 2019-10-13T15:38:55.732 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:55.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:55.730042+0000) 2019-10-13T15:38:55.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:55.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:55.730042+0000) 2019-10-13T15:38:55.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:55.741+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:55.741549+0000) 2019-10-13T15:38:56.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:38:56.133+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.135620+0000) 2019-10-13T15:38:56.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:38:56.133+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.136174+0000) 2019-10-13T15:38:56.140 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:56.133+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.137184+0000) 2019-10-13T15:38:56.140 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:38:56.133+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.137785+0000) 2019-10-13T15:38:56.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:38:56.133+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.138118+0000) 2019-10-13T15:38:56.141 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:38:56.137+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.138472+0000) 2019-10-13T15:38:56.141 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:38:56.133+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.139136+0000) 2019-10-13T15:38:56.145 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:38:56.141+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.142923+0000) 2019-10-13T15:38:56.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:38:56.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225047 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:38:56.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:38:56.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:38:56.281913+0000) 2019-10-13T15:39:01.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:01.144+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.150459+0000) 2019-10-13T15:39:01.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:01.144+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.150481+0000) 2019-10-13T15:39:01.154 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:01.144+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.150931+0000) 2019-10-13T15:39:01.154 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:01.148+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.151361+0000) 2019-10-13T15:39:01.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:01.149+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.152552+0000) 2019-10-13T15:39:01.156 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:01.148+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.152129+0000) 2019-10-13T15:39:01.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:01.148+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.153594+0000) 2019-10-13T15:39:01.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:01.157+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:01.160487+0000) 2019-10-13T15:39:01.164 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:01.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225079 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:02.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:02.716+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:02.720335+0000) 2019-10-13T15:39:03.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:03.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:03.456226+0000) 2019-10-13T15:39:03.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:03.701+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:03.705421+0000) 2019-10-13T15:39:05.214 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:05.209+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:05.212127+0000) 2019-10-13T15:39:05.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:05.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:05.730223+0000) 2019-10-13T15:39:05.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:05.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:05.730273+0000) 2019-10-13T15:39:05.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:05.740+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:05.741744+0000) 2019-10-13T15:39:06.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:06.160+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.166040+0000) 2019-10-13T15:39:06.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:06.160+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.165766+0000) 2019-10-13T15:39:06.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:06.160+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.166370+0000) 2019-10-13T15:39:06.170 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:06.160+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.166843+0000) 2019-10-13T15:39:06.170 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:06.164+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.167574+0000) 2019-10-13T15:39:06.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:06.164+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.168027+0000) 2019-10-13T15:39:06.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:06.165+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.169148+0000) 2019-10-13T15:39:06.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:06.169+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.173490+0000) 2019-10-13T15:39:06.179 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:06.172+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:06.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:06.280+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:06.282112+0000) 2019-10-13T15:39:11.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:11.177+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.181989+0000) 2019-10-13T15:39:11.193 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:11.176+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.180991+0000) 2019-10-13T15:39:11.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:11.176+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.182456+0000) 2019-10-13T15:39:11.194 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:11.176+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.182633+0000) 2019-10-13T15:39:11.194 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:11.176+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.182888+0000) 2019-10-13T15:39:11.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:11.180+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.183287+0000) 2019-10-13T15:39:11.196 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:11.180+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.183650+0000) 2019-10-13T15:39:11.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:11.185+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:11.188384+0000) 2019-10-13T15:39:11.202 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:11.188+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:11.991 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:39:11.992 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:39:11.997 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:39:11.997 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:39:12.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:12.716+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:12.720543+0000) 2019-10-13T15:39:13.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:13.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:13.456429+0000) 2019-10-13T15:39:13.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:13.705+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:13.705619+0000) 2019-10-13T15:39:15.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:15.217+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:15.218462+0000) 2019-10-13T15:39:15.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:15.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:15.730417+0000) 2019-10-13T15:39:15.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:15.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:15.730500+0000) 2019-10-13T15:39:15.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:15.740+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:15.741940+0000) 2019-10-13T15:39:16.198 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:16.193+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.195545+0000) 2019-10-13T15:39:16.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:16.192+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.196080+0000) 2019-10-13T15:39:16.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:16.192+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.196814+0000) 2019-10-13T15:39:16.199 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:16.192+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.197211+0000) 2019-10-13T15:39:16.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:16.192+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.198113+0000) 2019-10-13T15:39:16.201 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:16.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.198431+0000) 2019-10-13T15:39:16.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:16.192+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.198767+0000) 2019-10-13T15:39:16.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:16.201+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.203362+0000) 2019-10-13T15:39:16.209 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:16.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:16.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:16.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:16.282284+0000) 2019-10-13T15:39:21.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:21.209+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.210519+0000) 2019-10-13T15:39:21.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:21.204+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.211245+0000) 2019-10-13T15:39:21.214 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:21.208+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.211625+0000) 2019-10-13T15:39:21.214 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:21.208+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.211898+0000) 2019-10-13T15:39:21.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:21.208+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.212529+0000) 2019-10-13T15:39:21.215 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:21.208+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.212777+0000) 2019-10-13T15:39:21.215 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:21.208+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.213044+0000) 2019-10-13T15:39:21.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:21.217+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:21.219701+0000) 2019-10-13T15:39:21.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:21.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225215 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:22.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:22.720+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:22.720722+0000) 2019-10-13T15:39:23.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:23.452+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:23.456657+0000) 2019-10-13T15:39:23.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:23.705+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:23.705818+0000) 2019-10-13T15:39:25.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:25.221+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:25.224574+0000) 2019-10-13T15:39:25.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:25.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:25.730619+0000) 2019-10-13T15:39:25.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:25.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:25.730711+0000) 2019-10-13T15:39:25.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:25.740+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:25.742144+0000) 2019-10-13T15:39:26.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:26.221+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.225646+0000) 2019-10-13T15:39:26.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:26.220+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.225969+0000) 2019-10-13T15:39:26.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:26.220+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.226598+0000) 2019-10-13T15:39:26.230 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:26.220+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.227057+0000) 2019-10-13T15:39:26.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:26.224+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.227473+0000) 2019-10-13T15:39:26.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:26.224+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.227738+0000) 2019-10-13T15:39:26.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:26.224+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.228176+0000) 2019-10-13T15:39:26.241 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:26.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:26.241 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:26.233+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.234917+0000) 2019-10-13T15:39:26.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:26.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:26.282457+0000) 2019-10-13T15:39:31.242 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:31.236+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.240148+0000) 2019-10-13T15:39:31.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:31.236+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.240862+0000) 2019-10-13T15:39:31.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:31.236+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.241603+0000) 2019-10-13T15:39:31.244 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:31.236+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.241800+0000) 2019-10-13T15:39:31.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:31.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.242906+0000) 2019-10-13T15:39:31.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:31.240+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.243706+0000) 2019-10-13T15:39:31.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:31.240+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.244302+0000) 2019-10-13T15:39:31.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:31.244+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:31.249099+0000) 2019-10-13T15:39:31.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:31.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:32.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:32.716+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:32.720918+0000) 2019-10-13T15:39:33.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:33.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:33.456886+0000) 2019-10-13T15:39:33.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:33.700+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:33.706007+0000) 2019-10-13T15:39:35.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:35.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:35.236090+0000) 2019-10-13T15:39:35.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:35.724+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:35.730822+0000) 2019-10-13T15:39:35.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:35.724+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:35.730910+0000) 2019-10-13T15:39:35.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:35.736+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:35.742355+0000) 2019-10-13T15:39:36.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:36.252+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.255583+0000) 2019-10-13T15:39:36.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:36.252+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.256059+0000) 2019-10-13T15:39:36.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:36.252+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.257029+0000) 2019-10-13T15:39:36.260 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:36.252+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.257935+0000) 2019-10-13T15:39:36.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:36.252+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.258437+0000) 2019-10-13T15:39:36.261 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:36.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.258609+0000) 2019-10-13T15:39:36.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:36.256+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.259491+0000) 2019-10-13T15:39:36.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:36.260+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.264048+0000) 2019-10-13T15:39:36.269 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:36.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225319 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:36.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:36.276+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:36.282601+0000) 2019-10-13T15:39:41.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:41.263+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.270822+0000) 2019-10-13T15:39:41.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:41.263+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.271063+0000) 2019-10-13T15:39:41.275 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:41.268+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.271960+0000) 2019-10-13T15:39:41.275 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:41.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.272185+0000) 2019-10-13T15:39:41.275 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:41.267+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.272615+0000) 2019-10-13T15:39:41.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:41.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.273485+0000) 2019-10-13T15:39:41.276 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:41.267+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.274533+0000) 2019-10-13T15:39:41.282 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:41.276+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:41.279736+0000) 2019-10-13T15:39:41.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:41.279+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225350 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:42.120 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:39:42.120 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:39:42.147 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:39:42.148 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:39:42.723 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:42.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:42.721128+0000) 2019-10-13T15:39:43.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:43.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:43.457077+0000) 2019-10-13T15:39:43.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:43.704+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:43.706233+0000) 2019-10-13T15:39:45.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:45.240+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:45.242487+0000) 2019-10-13T15:39:45.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:45.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:45.731039+0000) 2019-10-13T15:39:45.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:45.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:45.731138+0000) 2019-10-13T15:39:45.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:45.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:45.742573+0000) 2019-10-13T15:39:46.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:46.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.282765+0000) 2019-10-13T15:39:46.289 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:46.279+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.287004+0000) 2019-10-13T15:39:46.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:46.283+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.287436+0000) 2019-10-13T15:39:46.290 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:46.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.287856+0000) 2019-10-13T15:39:46.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:46.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.288566+0000) 2019-10-13T15:39:46.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:46.283+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.289178+0000) 2019-10-13T15:39:46.292 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:46.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.289412+0000) 2019-10-13T15:39:46.293 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:46.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.290516+0000) 2019-10-13T15:39:46.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:46.292+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:46.297860+0000) 2019-10-13T15:39:46.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:46.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225388 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:51.304 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:51.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.301771+0000) 2019-10-13T15:39:51.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:51.295+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.302610+0000) 2019-10-13T15:39:51.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:51.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.302850+0000) 2019-10-13T15:39:51.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:51.299+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.303439+0000) 2019-10-13T15:39:51.306 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:51.299+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.304151+0000) 2019-10-13T15:39:51.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:51.300+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.304530+0000) 2019-10-13T15:39:51.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:51.299+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.305058+0000) 2019-10-13T15:39:51.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:51.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225418 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:39:51.316 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:51.312+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:51.314426+0000) 2019-10-13T15:39:52.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:52.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:52.721325+0000) 2019-10-13T15:39:53.459 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:53.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:53.457279+0000) 2019-10-13T15:39:53.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:53.704+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:53.706447+0000) 2019-10-13T15:39:55.251 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:55.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:55.248843+0000) 2019-10-13T15:39:55.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:55.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:55.731274+0000) 2019-10-13T15:39:55.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:55.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:55.731356+0000) 2019-10-13T15:39:55.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:55.735+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:55.742718+0000) 2019-10-13T15:39:56.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:56.275+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.282983+0000) 2019-10-13T15:39:56.319 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:39:56.311+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.316892+0000) 2019-10-13T15:39:56.320 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:39:56.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.317236+0000) 2019-10-13T15:39:56.321 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:39:56.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.317762+0000) 2019-10-13T15:39:56.321 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:39:56.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.318569+0000) 2019-10-13T15:39:56.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:39:56.315+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.319793+0000) 2019-10-13T15:39:56.322 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:39:56.316+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.320242+0000) 2019-10-13T15:39:56.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:39:56.315+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.320556+0000) 2019-10-13T15:39:56.328 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:39:56.324+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:39:56.326248+0000) 2019-10-13T15:39:56.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:39:56.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:01.336 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:01.327+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.333021+0000) 2019-10-13T15:40:01.336 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:01.327+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.333466+0000) 2019-10-13T15:40:01.336 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:01.327+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.334271+0000) 2019-10-13T15:40:01.337 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:01.332+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.334699+0000) 2019-10-13T15:40:01.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:01.327+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.335088+0000) 2019-10-13T15:40:01.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:01.331+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.335941+0000) 2019-10-13T15:40:01.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:01.331+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.336076+0000) 2019-10-13T15:40:01.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:01.336+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:01.340460+0000) 2019-10-13T15:40:01.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:01.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225486 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:02.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:02.715+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:02.721547+0000) 2019-10-13T15:40:03.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:03.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:03.457476+0000) 2019-10-13T15:40:03.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:03.704+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:03.706673+0000) 2019-10-13T15:40:05.258 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:05.252+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:05.255255+0000) 2019-10-13T15:40:05.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:05.727+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:05.731518+0000) 2019-10-13T15:40:05.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:05.727+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:05.731563+0000) 2019-10-13T15:40:05.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:05.739+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:05.742927+0000) 2019-10-13T15:40:06.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:06.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.283189+0000) 2019-10-13T15:40:06.350 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:06.343+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.347956+0000) 2019-10-13T15:40:06.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:06.344+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.348514+0000) 2019-10-13T15:40:06.352 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:06.343+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.348656+0000) 2019-10-13T15:40:06.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:06.343+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.348895+0000) 2019-10-13T15:40:06.353 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:06.343+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.349726+0000) 2019-10-13T15:40:06.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:06.343+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.350126+0000) 2019-10-13T15:40:06.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:06.343+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.350893+0000) 2019-10-13T15:40:06.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:06.348+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:06.353756+0000) 2019-10-13T15:40:06.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:06.355+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225524 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:11.365 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:11.359+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.362801+0000) 2019-10-13T15:40:11.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:11.359+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.363181+0000) 2019-10-13T15:40:11.366 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:11.360+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.364079+0000) 2019-10-13T15:40:11.367 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:11.363+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.364634+0000) 2019-10-13T15:40:11.367 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:11.363+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.363618+0000) 2019-10-13T15:40:11.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:11.363+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.365190+0000) 2019-10-13T15:40:11.369 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:11.363+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.366680+0000) 2019-10-13T15:40:11.375 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:11.368+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:11.372514+0000) 2019-10-13T15:40:11.376 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:11.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225554 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:12.255 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:40:12.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:40:12.259 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:40:12.259 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:40:12.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:12.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:12.721756+0000) 2019-10-13T15:40:13.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:13.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:13.457676+0000) 2019-10-13T15:40:13.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:13.704+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:13.706887+0000) 2019-10-13T15:40:15.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:15.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:15.261528+0000) 2019-10-13T15:40:15.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:15.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:15.731751+0000) 2019-10-13T15:40:15.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:15.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:15.731791+0000) 2019-10-13T15:40:15.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:15.739+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:15.743129+0000) 2019-10-13T15:40:16.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:16.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.283390+0000) 2019-10-13T15:40:16.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:16.375+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.376688+0000) 2019-10-13T15:40:16.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:16.372+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.377133+0000) 2019-10-13T15:40:16.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:16.375+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.378542+0000) 2019-10-13T15:40:16.381 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:16.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.378280+0000) 2019-10-13T15:40:16.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:16.375+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.379155+0000) 2019-10-13T15:40:16.383 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:16.379+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.379601+0000) 2019-10-13T15:40:16.383 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:16.379+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.379975+0000) 2019-10-13T15:40:16.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:16.380+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:16.383896+0000) 2019-10-13T15:40:16.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:16.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225591 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:21.395 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:21.391+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.392452+0000) 2019-10-13T15:40:21.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:21.391+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.392928+0000) 2019-10-13T15:40:21.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:21.388+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.393490+0000) 2019-10-13T15:40:21.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:21.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.393631+0000) 2019-10-13T15:40:21.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:21.391+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.393864+0000) 2019-10-13T15:40:21.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:21.391+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.394557+0000) 2019-10-13T15:40:21.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:21.395+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.395494+0000) 2019-10-13T15:40:21.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:21.396+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:21.400621+0000) 2019-10-13T15:40:21.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:21.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:22.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:22.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:22.721984+0000) 2019-10-13T15:40:23.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:23.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:23.457851+0000) 2019-10-13T15:40:23.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:23.704+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:23.707117+0000) 2019-10-13T15:40:25.270 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:25.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:25.268298+0000) 2019-10-13T15:40:25.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:25.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:25.731962+0000) 2019-10-13T15:40:25.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:25.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:25.732015+0000) 2019-10-13T15:40:25.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:25.742+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:25.743353+0000) 2019-10-13T15:40:26.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:26.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.283592+0000) 2019-10-13T15:40:26.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:26.402+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.406847+0000) 2019-10-13T15:40:26.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:26.403+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.407205+0000) 2019-10-13T15:40:26.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:26.406+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.407669+0000) 2019-10-13T15:40:26.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:26.406+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.408477+0000) 2019-10-13T15:40:26.411 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:26.406+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.409070+0000) 2019-10-13T15:40:26.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:26.406+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.409902+0000) 2019-10-13T15:40:26.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:26.406+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.410354+0000) 2019-10-13T15:40:26.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:26.411+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:26.415848+0000) 2019-10-13T15:40:26.421 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:26.414+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225660 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:31.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:31.418+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.422280+0000) 2019-10-13T15:40:31.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:31.422+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.423348+0000) 2019-10-13T15:40:31.426 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:31.419+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.423806+0000) 2019-10-13T15:40:31.426 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:31.422+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.424042+0000) 2019-10-13T15:40:31.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:31.422+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.424439+0000) 2019-10-13T15:40:31.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:31.422+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.424781+0000) 2019-10-13T15:40:31.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:31.422+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.426031+0000) 2019-10-13T15:40:31.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:31.431+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:31.434632+0000) 2019-10-13T15:40:31.437 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:31.430+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:32.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:32.718+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:32.722202+0000) 2019-10-13T15:40:33.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:33.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:33.458063+0000) 2019-10-13T15:40:33.709 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:33.703+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:33.707313+0000) 2019-10-13T15:40:35.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:35.271+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:35.275076+0000) 2019-10-13T15:40:35.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:35.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:35.732181+0000) 2019-10-13T15:40:35.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:35.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:35.732210+0000) 2019-10-13T15:40:35.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:35.742+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:35.743572+0000) 2019-10-13T15:40:36.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:36.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.283791+0000) 2019-10-13T15:40:36.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:36.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.438203+0000) 2019-10-13T15:40:36.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:36.434+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.438833+0000) 2019-10-13T15:40:36.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:36.434+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.439146+0000) 2019-10-13T15:40:36.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:36.438+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.439639+0000) 2019-10-13T15:40:36.442 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:36.438+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.440415+0000) 2019-10-13T15:40:36.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:36.438+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.440749+0000) 2019-10-13T15:40:36.443 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:36.438+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.441329+0000) 2019-10-13T15:40:36.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:36.439+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:36.444730+0000) 2019-10-13T15:40:36.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:36.446+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225727 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:41.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:41.450+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.453438+0000) 2019-10-13T15:40:41.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:41.450+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.453882+0000) 2019-10-13T15:40:41.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:41.450+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.454561+0000) 2019-10-13T15:40:41.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:41.450+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.455078+0000) 2019-10-13T15:40:41.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:41.450+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.455185+0000) 2019-10-13T15:40:41.458 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:41.454+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.455587+0000) 2019-10-13T15:40:41.458 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:41.451+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.456441+0000) 2019-10-13T15:40:41.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:41.459+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:41.462511+0000) 2019-10-13T15:40:41.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:41.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:42.508 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:40:42.508 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:40:42.511 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:40:42.512 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:40:42.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:42.718+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:42.722409+0000) 2019-10-13T15:40:43.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:43.454+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:43.458294+0000) 2019-10-13T15:40:43.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:43.703+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:43.707533+0000) 2019-10-13T15:40:45.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:45.279+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:45.281541+0000) 2019-10-13T15:40:45.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:45.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:45.732379+0000) 2019-10-13T15:40:45.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:45.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:45.732407+0000) 2019-10-13T15:40:45.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:45.742+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:45.743767+0000) 2019-10-13T15:40:46.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:46.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.284021+0000) 2019-10-13T15:40:46.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:46.466+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.468094+0000) 2019-10-13T15:40:46.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:46.466+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.467822+0000) 2019-10-13T15:40:46.471 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:46.466+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.469015+0000) 2019-10-13T15:40:46.472 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:46.467+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.470192+0000) 2019-10-13T15:40:46.473 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:46.466+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.470821+0000) 2019-10-13T15:40:46.473 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:46.466+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.471031+0000) 2019-10-13T15:40:46.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:46.470+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.471568+0000) 2019-10-13T15:40:46.477 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:46.471+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:46.475183+0000) 2019-10-13T15:40:46.482 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:46.478+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225796 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:51.485 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:51.479+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.483315+0000) 2019-10-13T15:40:51.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:51.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.483916+0000) 2019-10-13T15:40:51.487 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:51.482+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.484744+0000) 2019-10-13T15:40:51.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:51.482+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.485129+0000) 2019-10-13T15:40:51.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:51.482+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.484968+0000) 2019-10-13T15:40:51.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:51.482+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.485606+0000) 2019-10-13T15:40:51.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:51.482+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.486110+0000) 2019-10-13T15:40:51.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:51.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225825 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:40:51.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:51.491+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:51.495037+0000) 2019-10-13T15:40:52.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:52.718+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:52.722631+0000) 2019-10-13T15:40:53.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:53.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:53.458521+0000) 2019-10-13T15:40:53.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:53.703+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:53.707729+0000) 2019-10-13T15:40:55.290 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:55.283+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:55.288088+0000) 2019-10-13T15:40:55.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:55.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:55.732583+0000) 2019-10-13T15:40:55.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:55.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:55.732597+0000) 2019-10-13T15:40:55.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:55.742+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:55.743960+0000) 2019-10-13T15:40:56.286 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:56.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.284222+0000) 2019-10-13T15:40:56.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:40:56.494+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.498595+0000) 2019-10-13T15:40:56.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:40:56.494+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.499053+0000) 2019-10-13T15:40:56.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:40:56.498+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.499380+0000) 2019-10-13T15:40:56.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:40:56.495+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.500063+0000) 2019-10-13T15:40:56.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:40:56.498+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.500787+0000) 2019-10-13T15:40:56.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:40:56.498+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.501046+0000) 2019-10-13T15:40:56.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:40:56.498+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.501277+0000) 2019-10-13T15:40:56.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:40:56.499+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:40:56.504420+0000) 2019-10-13T15:40:56.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:40:56.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:01.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:01.510+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.513174+0000) 2019-10-13T15:41:01.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:01.511+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.514786+0000) 2019-10-13T15:41:01.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:01.514+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.515339+0000) 2019-10-13T15:41:01.518 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:01.514+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.515740+0000) 2019-10-13T15:41:01.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:01.514+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.516070+0000) 2019-10-13T15:41:01.519 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:01.514+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.516391+0000) 2019-10-13T15:41:01.519 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:01.514+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.516717+0000) 2019-10-13T15:41:01.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:01.515+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:01.520897+0000) 2019-10-13T15:41:01.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:01.522+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225893 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:02.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:02.722+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:02.722849+0000) 2019-10-13T15:41:03.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:03.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:03.458718+0000) 2019-10-13T15:41:03.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:03.707+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:03.707925+0000) 2019-10-13T15:41:05.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:05.295+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:05.298942+0000) 2019-10-13T15:41:05.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:05.730+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:05.732832+0000) 2019-10-13T15:41:05.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:05.730+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:05.732832+0000) 2019-10-13T15:41:05.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:05.742+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:05.744147+0000) 2019-10-13T15:41:06.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:06.282+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.284455+0000) 2019-10-13T15:41:06.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:06.526+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.529896+0000) 2019-10-13T15:41:06.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:06.526+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.530464+0000) 2019-10-13T15:41:06.533 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:06.526+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.530651+0000) 2019-10-13T15:41:06.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:06.526+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.530949+0000) 2019-10-13T15:41:06.534 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:06.530+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.531452+0000) 2019-10-13T15:41:06.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:06.527+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.531760+0000) 2019-10-13T15:41:06.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:06.530+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.532446+0000) 2019-10-13T15:41:06.542 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:06.535+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:06.539916+0000) 2019-10-13T15:41:06.543 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:06.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225932 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:11.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:11.541+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.544131+0000) 2019-10-13T15:41:11.548 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:11.541+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.545721+0000) 2019-10-13T15:41:11.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:11.541+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.545807+0000) 2019-10-13T15:41:11.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:11.541+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.546108+0000) 2019-10-13T15:41:11.549 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:11.541+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.546655+0000) 2019-10-13T15:41:11.550 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:11.545+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.548083+0000) 2019-10-13T15:41:11.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:11.543+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.548501+0000) 2019-10-13T15:41:11.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:11.547+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:11.553593+0000) 2019-10-13T15:41:11.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:11.553+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:12.644 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:41:12.645 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:41:12.648 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:41:12.648 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:41:12.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:12.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:12.723048+0000) 2019-10-13T15:41:13.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:13.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:13.458911+0000) 2019-10-13T15:41:13.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:13.703+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:13.708139+0000) 2019-10-13T15:41:15.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:15.299+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:15.305648+0000) 2019-10-13T15:41:15.735 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:15.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:15.733070+0000) 2019-10-13T15:41:15.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:15.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:15.733071+0000) 2019-10-13T15:41:15.746 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:15.741+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:15.744350+0000) 2019-10-13T15:41:16.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:16.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.284693+0000) 2019-10-13T15:41:16.565 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:16.555+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.560723+0000) 2019-10-13T15:41:16.565 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:16.553+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.558809+0000) 2019-10-13T15:41:16.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:16.557+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.559630+0000) 2019-10-13T15:41:16.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:16.557+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.559759+0000) 2019-10-13T15:41:16.566 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:16.557+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.560007+0000) 2019-10-13T15:41:16.566 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:16.557+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.560842+0000) 2019-10-13T15:41:16.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:16.557+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.561517+0000) 2019-10-13T15:41:16.572 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:16.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 225999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:16.573 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:16.567+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:16.571201+0000) 2019-10-13T15:41:21.575 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:21.569+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.573068+0000) 2019-10-13T15:41:21.576 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:21.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.574011+0000) 2019-10-13T15:41:21.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:21.569+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.574378+0000) 2019-10-13T15:41:21.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:21.573+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.575558+0000) 2019-10-13T15:41:21.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:21.573+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.575574+0000) 2019-10-13T15:41:21.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:21.573+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.576027+0000) 2019-10-13T15:41:21.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:21.573+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.576192+0000) 2019-10-13T15:41:21.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:21.574+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:21.581305+0000) 2019-10-13T15:41:21.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:21.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:22.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:22.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:22.723251+0000) 2019-10-13T15:41:23.461 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:23.453+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:23.459112+0000) 2019-10-13T15:41:23.710 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:23.706+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:23.708340+0000) 2019-10-13T15:41:25.314 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:25.310+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:25.312047+0000) 2019-10-13T15:41:25.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:25.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:25.733287+0000) 2019-10-13T15:41:25.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:25.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:25.733280+0000) 2019-10-13T15:41:25.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:25.741+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:25.744545+0000) 2019-10-13T15:41:26.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:26.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.284898+0000) 2019-10-13T15:41:26.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:26.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.588317+0000) 2019-10-13T15:41:26.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:26.585+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.588851+0000) 2019-10-13T15:41:26.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:26.582+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.589145+0000) 2019-10-13T15:41:26.592 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:26.585+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.589390+0000) 2019-10-13T15:41:26.592 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:26.585+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.589774+0000) 2019-10-13T15:41:26.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:26.585+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.590385+0000) 2019-10-13T15:41:26.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:26.585+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.591262+0000) 2019-10-13T15:41:26.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:26.590+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:26.597104+0000) 2019-10-13T15:41:26.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:26.593+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226068 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:31.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:31.597+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.602297+0000) 2019-10-13T15:41:31.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:31.597+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.602695+0000) 2019-10-13T15:41:31.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:31.601+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.603445+0000) 2019-10-13T15:41:31.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:31.601+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.604103+0000) 2019-10-13T15:41:31.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:31.601+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.604313+0000) 2019-10-13T15:41:31.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:31.598+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.604714+0000) 2019-10-13T15:41:31.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:31.601+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.605510+0000) 2019-10-13T15:41:31.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:31.606+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:31.611208+0000) 2019-10-13T15:41:31.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:31.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226096 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:32.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:32.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:32.723491+0000) 2019-10-13T15:41:33.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:33.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:33.459330+0000) 2019-10-13T15:41:33.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:33.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:33.708543+0000) 2019-10-13T15:41:35.321 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:35.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:35.318427+0000) 2019-10-13T15:41:35.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:35.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:35.733498+0000) 2019-10-13T15:41:35.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:35.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:35.733522+0000) 2019-10-13T15:41:35.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:35.741+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:35.744755+0000) 2019-10-13T15:41:36.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:36.281+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.285094+0000) 2019-10-13T15:41:36.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:36.613+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.616668+0000) 2019-10-13T15:41:36.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:36.613+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.616937+0000) 2019-10-13T15:41:36.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:36.613+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.617983+0000) 2019-10-13T15:41:36.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:36.613+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.618567+0000) 2019-10-13T15:41:36.621 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:36.617+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.619358+0000) 2019-10-13T15:41:36.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:36.614+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.619836+0000) 2019-10-13T15:41:36.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:36.617+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.619961+0000) 2019-10-13T15:41:36.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:36.618+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:36.625138+0000) 2019-10-13T15:41:36.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:36.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:41.635 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:41.629+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.632410+0000) 2019-10-13T15:41:41.635 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:41.629+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.632942+0000) 2019-10-13T15:41:41.636 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:41.629+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.633600+0000) 2019-10-13T15:41:41.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:41.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.633980+0000) 2019-10-13T15:41:41.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:41.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.634452+0000) 2019-10-13T15:41:41.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:41.629+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.634817+0000) 2019-10-13T15:41:41.638 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:41.633+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.635994+0000) 2019-10-13T15:41:41.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:41.634+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:41.642095+0000) 2019-10-13T15:41:41.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:41.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226164 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:42.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:42.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:42.723723+0000) 2019-10-13T15:41:42.780 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:41:42.780 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:41:42.784 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:41:42.784 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:41:43.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:43.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:43.459527+0000) 2019-10-13T15:41:43.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:43.706+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:43.708767+0000) 2019-10-13T15:41:45.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:45.322+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:45.324875+0000) 2019-10-13T15:41:45.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:45.729+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:45.733720+0000) 2019-10-13T15:41:45.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:45.729+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:45.733726+0000) 2019-10-13T15:41:45.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:45.741+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:45.744955+0000) 2019-10-13T15:41:46.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:46.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.285297+0000) 2019-10-13T15:41:46.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:46.641+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.646930+0000) 2019-10-13T15:41:46.650 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:46.645+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.647791+0000) 2019-10-13T15:41:46.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:46.645+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.648258+0000) 2019-10-13T15:41:46.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:46.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.647399+0000) 2019-10-13T15:41:46.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:46.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.648535+0000) 2019-10-13T15:41:46.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:46.645+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.649175+0000) 2019-10-13T15:41:46.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:46.646+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.650278+0000) 2019-10-13T15:41:46.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:46.650+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:46.656912+0000) 2019-10-13T15:41:46.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:46.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:51.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:51.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.661865+0000) 2019-10-13T15:41:51.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:51.656+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.662660+0000) 2019-10-13T15:41:51.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:51.660+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.663481+0000) 2019-10-13T15:41:51.666 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:51.658+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.663812+0000) 2019-10-13T15:41:51.666 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:51.660+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.664199+0000) 2019-10-13T15:41:51.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:51.660+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.664628+0000) 2019-10-13T15:41:51.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:51.660+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.665331+0000) 2019-10-13T15:41:51.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:51.666+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:51.671223+0000) 2019-10-13T15:41:51.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:51.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:41:52.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:52.720+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:52.723925+0000) 2019-10-13T15:41:53.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:53.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:53.459745+0000) 2019-10-13T15:41:53.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:53.702+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:53.708960+0000) 2019-10-13T15:41:55.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:55.330+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:55.331512+0000) 2019-10-13T15:41:55.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:55.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:55.733883+0000) 2019-10-13T15:41:55.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:55.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:55.733922+0000) 2019-10-13T15:41:55.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:55.744+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:55.745154+0000) 2019-10-13T15:41:56.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:56.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.285511+0000) 2019-10-13T15:41:56.679 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:41:56.670+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.676868+0000) 2019-10-13T15:41:56.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:41:56.672+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.678040+0000) 2019-10-13T15:41:56.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:41:56.672+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.678278+0000) 2019-10-13T15:41:56.681 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:41:56.672+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.679199+0000) 2019-10-13T15:41:56.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:41:56.672+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.678947+0000) 2019-10-13T15:41:56.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:41:56.676+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.679974+0000) 2019-10-13T15:41:56.682 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:41:56.676+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.679954+0000) 2019-10-13T15:41:56.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:41:56.678+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:41:56.683977+0000) 2019-10-13T15:41:56.690 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:41:56.684+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:01.694 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:01.688+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.691822+0000) 2019-10-13T15:42:01.695 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:01.688+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.692123+0000) 2019-10-13T15:42:01.695 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:01.688+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.692561+0000) 2019-10-13T15:42:01.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:01.686+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.693625+0000) 2019-10-13T15:42:01.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:01.688+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.694121+0000) 2019-10-13T15:42:01.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:01.688+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.694870+0000) 2019-10-13T15:42:01.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:01.692+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.695798+0000) 2019-10-13T15:42:01.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:01.694+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:01.701237+0000) 2019-10-13T15:42:01.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:01.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:02.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:02.720+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:02.724141+0000) 2019-10-13T15:42:03.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:03.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:03.459936+0000) 2019-10-13T15:42:03.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:03.706+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:03.709155+0000) 2019-10-13T15:42:05.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:05.334+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:05.337827+0000) 2019-10-13T15:42:05.736 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:05.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:05.734079+0000) 2019-10-13T15:42:05.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:05.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:05.734143+0000) 2019-10-13T15:42:05.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:05.740+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:05.745378+0000) 2019-10-13T15:42:06.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:06.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.285720+0000) 2019-10-13T15:42:06.708 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:06.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.705543+0000) 2019-10-13T15:42:06.709 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:06.700+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.707102+0000) 2019-10-13T15:42:06.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:06.704+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.707404+0000) 2019-10-13T15:42:06.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:06.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.707669+0000) 2019-10-13T15:42:06.710 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:06.704+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.708134+0000) 2019-10-13T15:42:06.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:06.702+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.708620+0000) 2019-10-13T15:42:06.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:06.704+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.708789+0000) 2019-10-13T15:42:06.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:06.710+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:06.715095+0000) 2019-10-13T15:42:06.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:06.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:11.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:11.716+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.720701+0000) 2019-10-13T15:42:11.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:11.716+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.721318+0000) 2019-10-13T15:42:11.724 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:11.716+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.721669+0000) 2019-10-13T15:42:11.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:11.722+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.722761+0000) 2019-10-13T15:42:11.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:11.716+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.722088+0000) 2019-10-13T15:42:11.726 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:11.716+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.722992+0000) 2019-10-13T15:42:11.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:11.720+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.724497+0000) 2019-10-13T15:42:11.732 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:11.726+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:11.729674+0000) 2019-10-13T15:42:11.735 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:11.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:12.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:12.720+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:12.724384+0000) 2019-10-13T15:42:12.908 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:42:12.909 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:42:12.913 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:42:12.913 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:42:13.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:13.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:13.460161+0000) 2019-10-13T15:42:13.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:13.706+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:13.709350+0000) 2019-10-13T15:42:15.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:15.341+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:15.344286+0000) 2019-10-13T15:42:15.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:15.728+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:15.734281+0000) 2019-10-13T15:42:15.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:15.728+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:15.734358+0000) 2019-10-13T15:42:15.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:15.744+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:15.745575+0000) 2019-10-13T15:42:16.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:16.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.285950+0000) 2019-10-13T15:42:16.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:16.732+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.736976+0000) 2019-10-13T15:42:16.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:16.732+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.737365+0000) 2019-10-13T15:42:16.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:16.732+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.737926+0000) 2019-10-13T15:42:16.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:16.737+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.738406+0000) 2019-10-13T15:42:16.741 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:16.732+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.738534+0000) 2019-10-13T15:42:16.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:16.736+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.739912+0000) 2019-10-13T15:42:16.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:16.736+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.740629+0000) 2019-10-13T15:42:16.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:16.745+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:16.747203+0000) 2019-10-13T15:42:16.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:16.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:21.753 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:21.744+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.750872+0000) 2019-10-13T15:42:21.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:21.748+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.751767+0000) 2019-10-13T15:42:21.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:21.748+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.752192+0000) 2019-10-13T15:42:21.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:21.749+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.752802+0000) 2019-10-13T15:42:21.756 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:21.748+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.753641+0000) 2019-10-13T15:42:21.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:21.748+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.753906+0000) 2019-10-13T15:42:21.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:21.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.754375+0000) 2019-10-13T15:42:21.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:21.761+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:21.762970+0000) 2019-10-13T15:42:21.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:21.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226440 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:22.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:22.720+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:22.724619+0000) 2019-10-13T15:42:23.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:23.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:23.460376+0000) 2019-10-13T15:42:23.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:23.705+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:23.709548+0000) 2019-10-13T15:42:25.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:25.349+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:25.350909+0000) 2019-10-13T15:42:25.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:25.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:25.734432+0000) 2019-10-13T15:42:25.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:25.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:25.734551+0000) 2019-10-13T15:42:25.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:25.744+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:25.745772+0000) 2019-10-13T15:42:26.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:26.284+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.286183+0000) 2019-10-13T15:42:26.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:26.760+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.765232+0000) 2019-10-13T15:42:26.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:26.760+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.765759+0000) 2019-10-13T15:42:26.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:26.760+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.766227+0000) 2019-10-13T15:42:26.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:26.764+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.767804+0000) 2019-10-13T15:42:26.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:26.765+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.768570+0000) 2019-10-13T15:42:26.771 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:26.764+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.769404+0000) 2019-10-13T15:42:26.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:26.764+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.769825+0000) 2019-10-13T15:42:26.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:26.773+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:26.774764+0000) 2019-10-13T15:42:26.779 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:26.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:31.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:31.776+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.780726+0000) 2019-10-13T15:42:31.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:31.776+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.780859+0000) 2019-10-13T15:42:31.784 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:31.777+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.781215+0000) 2019-10-13T15:42:31.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:31.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.781757+0000) 2019-10-13T15:42:31.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:31.776+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.782412+0000) 2019-10-13T15:42:31.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:31.776+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.782789+0000) 2019-10-13T15:42:31.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:31.780+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.783416+0000) 2019-10-13T15:42:31.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:31.789+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:31.790453+0000) 2019-10-13T15:42:31.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:31.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:32.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:32.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:32.724796+0000) 2019-10-13T15:42:33.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:33.456+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:33.460565+0000) 2019-10-13T15:42:33.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:33.705+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:33.709741+0000) 2019-10-13T15:42:35.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:35.353+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:35.357405+0000) 2019-10-13T15:42:35.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:35.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:35.734670+0000) 2019-10-13T15:42:35.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:35.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:35.734778+0000) 2019-10-13T15:42:35.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:35.743+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:35.746000+0000) 2019-10-13T15:42:36.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:36.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.286393+0000) 2019-10-13T15:42:36.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:36.791+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.796533+0000) 2019-10-13T15:42:36.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:36.791+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.797036+0000) 2019-10-13T15:42:36.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:36.791+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.797277+0000) 2019-10-13T15:42:36.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:36.791+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.797739+0000) 2019-10-13T15:42:36.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:36.791+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.797909+0000) 2019-10-13T15:42:36.801 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:36.791+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.799054+0000) 2019-10-13T15:42:36.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:36.797+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.799632+0000) 2019-10-13T15:42:36.807 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:36.801+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:36.805095+0000) 2019-10-13T15:42:36.810 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:36.803+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:41.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:41.803+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.811023+0000) 2019-10-13T15:42:41.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:41.807+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.811526+0000) 2019-10-13T15:42:41.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:41.807+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.812317+0000) 2019-10-13T15:42:41.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:41.807+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.812892+0000) 2019-10-13T15:42:41.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:41.809+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.813599+0000) 2019-10-13T15:42:41.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:41.807+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.813889+0000) 2019-10-13T15:42:41.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:41.807+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.814280+0000) 2019-10-13T15:42:41.822 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:41.817+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:41.820387+0000) 2019-10-13T15:42:41.825 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:41.815+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:42.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:42.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:42.724977+0000) 2019-10-13T15:42:43.037 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:42:43.037 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:42:43.041 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:42:43.041 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:42:43.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:43.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:43.460798+0000) 2019-10-13T15:42:43.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:43.705+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:43.709978+0000) 2019-10-13T15:42:45.366 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:45.361+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:45.363722+0000) 2019-10-13T15:42:45.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:45.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:45.734880+0000) 2019-10-13T15:42:45.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:45.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:45.734959+0000) 2019-10-13T15:42:45.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:45.739+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:45.746201+0000) 2019-10-13T15:42:46.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:46.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.286594+0000) 2019-10-13T15:42:46.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:46.819+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.825898+0000) 2019-10-13T15:42:46.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:46.819+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.827279+0000) 2019-10-13T15:42:46.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:46.823+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.827884+0000) 2019-10-13T15:42:46.831 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:46.823+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.828756+0000) 2019-10-13T15:42:46.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:46.823+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.828203+0000) 2019-10-13T15:42:46.832 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:46.825+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.828958+0000) 2019-10-13T15:42:46.832 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:46.823+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.829193+0000) 2019-10-13T15:42:46.839 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:46.833+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:46.836723+0000) 2019-10-13T15:42:46.839 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:46.831+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:51.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:51.837+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.840422+0000) 2019-10-13T15:42:51.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:51.835+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.840672+0000) 2019-10-13T15:42:51.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:51.835+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.841907+0000) 2019-10-13T15:42:51.844 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:51.835+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.842159+0000) 2019-10-13T15:42:51.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:51.835+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.842658+0000) 2019-10-13T15:42:51.845 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:51.835+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.842878+0000) 2019-10-13T15:42:51.845 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:51.835+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.843109+0000) 2019-10-13T15:42:51.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:51.845+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:51.847600+0000) 2019-10-13T15:42:51.854 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:51.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226647 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:42:52.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:52.723+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:52.725183+0000) 2019-10-13T15:42:53.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:53.455+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:53.461029+0000) 2019-10-13T15:42:53.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:53.709+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:53.710181+0000) 2019-10-13T15:42:55.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:55.369+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:55.370103+0000) 2019-10-13T15:42:55.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:55.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:55.735098+0000) 2019-10-13T15:42:55.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:55.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:55.735200+0000) 2019-10-13T15:42:55.748 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:55.743+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:55.746399+0000) 2019-10-13T15:42:56.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:56.279+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.286802+0000) 2019-10-13T15:42:56.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:42:56.853+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.855310+0000) 2019-10-13T15:42:56.858 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:42:56.851+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.855844+0000) 2019-10-13T15:42:56.858 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:42:56.851+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.856120+0000) 2019-10-13T15:42:56.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:42:56.851+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.856389+0000) 2019-10-13T15:42:56.859 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:42:56.851+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.857297+0000) 2019-10-13T15:42:56.860 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:42:56.851+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.857992+0000) 2019-10-13T15:42:56.860 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:42:56.851+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.858405+0000) 2019-10-13T15:42:56.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:42:56.861+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:42:56.863067+0000) 2019-10-13T15:42:56.869 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:42:56.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:01.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:01.867+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.871216+0000) 2019-10-13T15:43:01.875 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:01.869+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.872699+0000) 2019-10-13T15:43:01.875 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:01.867+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.873080+0000) 2019-10-13T15:43:01.876 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:01.867+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.873589+0000) 2019-10-13T15:43:01.876 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:01.867+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.874090+0000) 2019-10-13T15:43:01.877 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:01.867+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.874507+0000) 2019-10-13T15:43:01.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:01.867+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.875172+0000) 2019-10-13T15:43:01.883 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:01.877+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:01.880618+0000) 2019-10-13T15:43:01.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:01.875+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:02.727 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:02.719+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:02.725358+0000) 2019-10-13T15:43:03.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:03.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:03.461284+0000) 2019-10-13T15:43:03.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:03.709+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:03.710378+0000) 2019-10-13T15:43:05.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:05.373+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:05.376376+0000) 2019-10-13T15:43:05.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:05.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:05.735330+0000) 2019-10-13T15:43:05.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:05.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:05.735415+0000) 2019-10-13T15:43:05.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:05.743+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:05.746605+0000) 2019-10-13T15:43:06.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:06.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.287049+0000) 2019-10-13T15:43:06.887 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:06.883+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.884647+0000) 2019-10-13T15:43:06.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:06.883+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.885070+0000) 2019-10-13T15:43:06.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:06.883+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.885387+0000) 2019-10-13T15:43:06.889 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:06.883+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.887129+0000) 2019-10-13T15:43:06.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:06.887+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.887344+0000) 2019-10-13T15:43:06.890 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:06.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.888112+0000) 2019-10-13T15:43:06.891 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:06.887+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.888549+0000) 2019-10-13T15:43:06.896 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:06.893+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:06.894571+0000) 2019-10-13T15:43:06.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:06.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226754 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:11.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:11.896+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.899588+0000) 2019-10-13T15:43:11.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:11.899+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.900319+0000) 2019-10-13T15:43:11.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:11.899+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.900650+0000) 2019-10-13T15:43:11.903 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:11.899+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.900883+0000) 2019-10-13T15:43:11.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:11.899+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.901430+0000) 2019-10-13T15:43:11.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:11.899+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.902232+0000) 2019-10-13T15:43:11.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:11.899+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.903051+0000) 2019-10-13T15:43:11.912 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:11.908+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:11.910438+0000) 2019-10-13T15:43:11.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:11.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:12.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:12.723+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:12.725565+0000) 2019-10-13T15:43:13.099 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:43:13.100 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:43:13.103 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:43:13.103 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:43:13.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:13.459+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:13.461483+0000) 2019-10-13T15:43:13.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:13.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:13.710595+0000) 2019-10-13T15:43:15.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:15.380+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:15.382792+0000) 2019-10-13T15:43:15.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:15.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:15.735526+0000) 2019-10-13T15:43:15.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:15.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:15.735640+0000) 2019-10-13T15:43:15.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:15.743+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:15.746832+0000) 2019-10-13T15:43:16.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:16.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.287259+0000) 2019-10-13T15:43:16.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:16.915+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.915943+0000) 2019-10-13T15:43:16.919 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:16.915+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.916070+0000) 2019-10-13T15:43:16.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:16.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.916867+0000) 2019-10-13T15:43:16.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:16.915+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.916306+0000) 2019-10-13T15:43:16.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:16.912+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.917395+0000) 2019-10-13T15:43:16.920 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:16.915+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.917671+0000) 2019-10-13T15:43:16.920 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:16.915+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.917969+0000) 2019-10-13T15:43:16.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:16.916+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:16.921145+0000) 2019-10-13T15:43:16.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:16.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226822 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:21.931 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:21.926+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.928919+0000) 2019-10-13T15:43:21.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:21.926+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.929315+0000) 2019-10-13T15:43:21.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:21.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.930488+0000) 2019-10-13T15:43:21.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:21.926+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.930585+0000) 2019-10-13T15:43:21.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:21.926+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.931042+0000) 2019-10-13T15:43:21.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:21.928+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.931910+0000) 2019-10-13T15:43:21.935 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:21.930+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.931378+0000) 2019-10-13T15:43:21.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:21.936+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:21.939099+0000) 2019-10-13T15:43:21.943 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:21.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:22.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:22.722+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:22.725745+0000) 2019-10-13T15:43:23.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:23.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:23.461719+0000) 2019-10-13T15:43:23.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:23.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:23.710810+0000) 2019-10-13T15:43:25.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:25.388+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:25.389129+0000) 2019-10-13T15:43:25.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:25.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:25.735728+0000) 2019-10-13T15:43:25.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:25.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:25.735882+0000) 2019-10-13T15:43:25.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:25.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:25.747042+0000) 2019-10-13T15:43:26.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:26.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.287512+0000) 2019-10-13T15:43:26.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:26.942+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.944720+0000) 2019-10-13T15:43:26.947 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:26.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.945105+0000) 2019-10-13T15:43:26.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:26.942+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.945744+0000) 2019-10-13T15:43:26.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:26.944+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.946363+0000) 2019-10-13T15:43:26.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:26.942+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.946543+0000) 2019-10-13T15:43:26.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:26.942+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.947179+0000) 2019-10-13T15:43:26.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:26.946+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.947946+0000) 2019-10-13T15:43:26.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:26.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:26.951946+0000) 2019-10-13T15:43:26.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:26.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226892 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:31.962 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:31.956+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.959972+0000) 2019-10-13T15:43:31.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:31.954+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.959199+0000) 2019-10-13T15:43:31.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:31.958+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.960216+0000) 2019-10-13T15:43:31.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:31.958+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.961633+0000) 2019-10-13T15:43:31.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:31.958+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.961779+0000) 2019-10-13T15:43:31.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:31.958+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.962145+0000) 2019-10-13T15:43:31.965 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:31.958+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.962986+0000) 2019-10-13T15:43:31.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:31.968+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:31.970681+0000) 2019-10-13T15:43:31.974 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:31.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226923 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:32.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:32.722+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:32.725929+0000) 2019-10-13T15:43:33.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:33.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:33.461930+0000) 2019-10-13T15:43:33.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:33.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:33.711036+0000) 2019-10-13T15:43:35.398 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:35.392+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:35.395783+0000) 2019-10-13T15:43:35.738 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:35.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:35.735934+0000) 2019-10-13T15:43:35.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:35.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:35.736056+0000) 2019-10-13T15:43:35.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:35.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:35.747266+0000) 2019-10-13T15:43:36.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:36.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.287719+0000) 2019-10-13T15:43:36.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:36.974+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.975463+0000) 2019-10-13T15:43:36.978 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:36.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.975882+0000) 2019-10-13T15:43:36.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:36.972+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.976461+0000) 2019-10-13T15:43:36.979 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:36.974+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.976688+0000) 2019-10-13T15:43:36.980 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:36.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.977382+0000) 2019-10-13T15:43:36.980 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:36.974+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.977188+0000) 2019-10-13T15:43:36.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:36.974+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.977985+0000) 2019-10-13T15:43:36.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:36.980+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:36.984023+0000) 2019-10-13T15:43:36.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:36.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226960 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:41.994 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:41.986+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.989992+0000) 2019-10-13T15:43:41.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:41.986+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.991003+0000) 2019-10-13T15:43:41.995 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:41.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.992078+0000) 2019-10-13T15:43:41.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:41.990+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.992102+0000) 2019-10-13T15:43:41.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:41.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.992828+0000) 2019-10-13T15:43:41.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:41.988+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.993805+0000) 2019-10-13T15:43:41.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:41.990+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.994178+0000) 2019-10-13T15:43:41.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:41.992+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:41.997165+0000) 2019-10-13T15:43:42.005 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:41.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 226993 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:42.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:42.722+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:42.726121+0000) 2019-10-13T15:43:43.207 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:43:43.207 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:43:43.211 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:43:43.211 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:43:43.464 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:43.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:43.462141+0000) 2019-10-13T15:43:43.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:43.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:43.711233+0000) 2019-10-13T15:43:45.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:45.400+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:45.402126+0000) 2019-10-13T15:43:45.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:45.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:45.736130+0000) 2019-10-13T15:43:45.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:45.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:45.736260+0000) 2019-10-13T15:43:45.749 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:45.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:45.747540+0000) 2019-10-13T15:43:46.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:46.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:46.287883+0000) 2019-10-13T15:43:47.009 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:47.002+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.006799+0000) 2019-10-13T15:43:47.009 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:47.002+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.007112+0000) 2019-10-13T15:43:47.010 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:47.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.007857+0000) 2019-10-13T15:43:47.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:47.006+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.007850+0000) 2019-10-13T15:43:47.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:47.006+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.008861+0000) 2019-10-13T15:43:47.011 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:47.006+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.008736+0000) 2019-10-13T15:43:47.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:47.004+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.009258+0000) 2019-10-13T15:43:47.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:47.012+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:47.015322+0000) 2019-10-13T15:43:47.020 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:47.014+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:52.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:52.018+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.022146+0000) 2019-10-13T15:43:52.025 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:52.018+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.021939+0000) 2019-10-13T15:43:52.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:52.022+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.023642+0000) 2019-10-13T15:43:52.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:52.022+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.023930+0000) 2019-10-13T15:43:52.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:52.020+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.024343+0000) 2019-10-13T15:43:52.027 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:52.022+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.024512+0000) 2019-10-13T15:43:52.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:52.022+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.024918+0000) 2019-10-13T15:43:52.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:52.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.032026+0000) 2019-10-13T15:43:52.035 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:52.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:52.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:52.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:52.726333+0000) 2019-10-13T15:43:53.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:53.458+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:53.462363+0000) 2019-10-13T15:43:53.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:53.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:53.711442+0000) 2019-10-13T15:43:55.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:55.404+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:55.408630+0000) 2019-10-13T15:43:55.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:55.734+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:55.736368+0000) 2019-10-13T15:43:55.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:55.734+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:55.736506+0000) 2019-10-13T15:43:55.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:55.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:55.747752+0000) 2019-10-13T15:43:56.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:56.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:56.288104+0000) 2019-10-13T15:43:57.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:43:57.034+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.036431+0000) 2019-10-13T15:43:57.039 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:43:57.034+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.037386+0000) 2019-10-13T15:43:57.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:43:57.034+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.038101+0000) 2019-10-13T15:43:57.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:43:57.034+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.038884+0000) 2019-10-13T15:43:57.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:43:57.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.039665+0000) 2019-10-13T15:43:57.042 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:43:57.038+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.039865+0000) 2019-10-13T15:43:57.043 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:43:57.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.040529+0000) 2019-10-13T15:43:57.049 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:43:57.046+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:43:57.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:43:57.044+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:43:57.049416+0000) 2019-10-13T15:44:02.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:02.045+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.050735+0000) 2019-10-13T15:44:02.054 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:02.049+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.051633+0000) 2019-10-13T15:44:02.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:02.049+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.052233+0000) 2019-10-13T15:44:02.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:02.049+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.052626+0000) 2019-10-13T15:44:02.055 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:02.049+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.053498+0000) 2019-10-13T15:44:02.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:02.049+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.054238+0000) 2019-10-13T15:44:02.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:02.052+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.054437+0000) 2019-10-13T15:44:02.060 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:02.056+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.058360+0000) 2019-10-13T15:44:02.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:02.057+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:02.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:02.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:02.726525+0000) 2019-10-13T15:44:03.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:03.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:03.462559+0000) 2019-10-13T15:44:03.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:03.708+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:03.711669+0000) 2019-10-13T15:44:05.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:05.415+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:05.417929+0000) 2019-10-13T15:44:05.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:05.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:05.736578+0000) 2019-10-13T15:44:05.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:05.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:05.736709+0000) 2019-10-13T15:44:05.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:05.745+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:05.747951+0000) 2019-10-13T15:44:06.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:06.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:06.288297+0000) 2019-10-13T15:44:07.068 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:07.061+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.066106+0000) 2019-10-13T15:44:07.069 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:07.063+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.066551+0000) 2019-10-13T15:44:07.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:07.061+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.066876+0000) 2019-10-13T15:44:07.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:07.065+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.067553+0000) 2019-10-13T15:44:07.071 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:07.065+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.068100+0000) 2019-10-13T15:44:07.071 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:07.065+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.068302+0000) 2019-10-13T15:44:07.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:07.065+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.068563+0000) 2019-10-13T15:44:07.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:07.071+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:07.075867+0000) 2019-10-13T15:44:07.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:07.073+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227168 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:12.084 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:12.075+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.080675+0000) 2019-10-13T15:44:12.084 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:12.077+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.079806+0000) 2019-10-13T15:44:12.084 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:12.077+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.080286+0000) 2019-10-13T15:44:12.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:12.077+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.081581+0000) 2019-10-13T15:44:12.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:12.077+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.082235+0000) 2019-10-13T15:44:12.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:12.077+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.082561+0000) 2019-10-13T15:44:12.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:12.077+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.083180+0000) 2019-10-13T15:44:12.090 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:12.083+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.087618+0000) 2019-10-13T15:44:12.093 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:12.089+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:12.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:12.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:12.726733+0000) 2019-10-13T15:44:13.319 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:44:13.319 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:44:13.323 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:44:13.323 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:44:13.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:13.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:13.462780+0000) 2019-10-13T15:44:13.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:13.707+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:13.711868+0000) 2019-10-13T15:44:15.432 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:15.427+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:15.429715+0000) 2019-10-13T15:44:15.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:15.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:15.736778+0000) 2019-10-13T15:44:15.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:15.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:15.736914+0000) 2019-10-13T15:44:15.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:15.745+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:15.748103+0000) 2019-10-13T15:44:16.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:16.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:16.288454+0000) 2019-10-13T15:44:17.098 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:17.093+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.095481+0000) 2019-10-13T15:44:17.098 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:17.091+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.095897+0000) 2019-10-13T15:44:17.099 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:17.093+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.096545+0000) 2019-10-13T15:44:17.099 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:17.093+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.096540+0000) 2019-10-13T15:44:17.099 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:17.093+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.097176+0000) 2019-10-13T15:44:17.100 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:17.093+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.097471+0000) 2019-10-13T15:44:17.100 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:17.093+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.097762+0000) 2019-10-13T15:44:17.105 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:17.099+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:17.103516+0000) 2019-10-13T15:44:17.109 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:17.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227236 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:22.112 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:22.103+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.110018+0000) 2019-10-13T15:44:22.113 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:22.105+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.110649+0000) 2019-10-13T15:44:22.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:22.109+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.111451+0000) 2019-10-13T15:44:22.114 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:22.109+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.112116+0000) 2019-10-13T15:44:22.115 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:22.109+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.112470+0000) 2019-10-13T15:44:22.115 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:22.109+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.113025+0000) 2019-10-13T15:44:22.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:22.109+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.113162+0000) 2019-10-13T15:44:22.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:22.115+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.118212+0000) 2019-10-13T15:44:22.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:22.117+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:22.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:22.721+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:22.726913+0000) 2019-10-13T15:44:23.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:23.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:23.462984+0000) 2019-10-13T15:44:23.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:23.711+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:23.712083+0000) 2019-10-13T15:44:25.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:25.431+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:25.436091+0000) 2019-10-13T15:44:25.739 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:25.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:25.736999+0000) 2019-10-13T15:44:25.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:25.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:25.737155+0000) 2019-10-13T15:44:25.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:25.745+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:25.748320+0000) 2019-10-13T15:44:26.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:26.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:26.288661+0000) 2019-10-13T15:44:27.128 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:27.121+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.126122+0000) 2019-10-13T15:44:27.129 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:27.121+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.126472+0000) 2019-10-13T15:44:27.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:27.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.126879+0000) 2019-10-13T15:44:27.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:27.125+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.127470+0000) 2019-10-13T15:44:27.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:27.125+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.128040+0000) 2019-10-13T15:44:27.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:27.125+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.128682+0000) 2019-10-13T15:44:27.131 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:27.125+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.128974+0000) 2019-10-13T15:44:27.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:27.127+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:27.133819+0000) 2019-10-13T15:44:27.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:27.133+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:32.142 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:32.137+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.140007+0000) 2019-10-13T15:44:32.143 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:32.137+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.140506+0000) 2019-10-13T15:44:32.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:32.135+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.141875+0000) 2019-10-13T15:44:32.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:32.137+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.142073+0000) 2019-10-13T15:44:32.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:32.137+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.142558+0000) 2019-10-13T15:44:32.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:32.137+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.142694+0000) 2019-10-13T15:44:32.145 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:32.137+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.143137+0000) 2019-10-13T15:44:32.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:32.143+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.147552+0000) 2019-10-13T15:44:32.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:32.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227335 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:32.729 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:32.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:32.727096+0000) 2019-10-13T15:44:33.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:33.457+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:33.463204+0000) 2019-10-13T15:44:33.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:33.711+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:33.712285+0000) 2019-10-13T15:44:35.444 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:35.435+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:35.441916+0000) 2019-10-13T15:44:35.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:35.733+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:35.737196+0000) 2019-10-13T15:44:35.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:35.733+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:35.737364+0000) 2019-10-13T15:44:35.750 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:35.745+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:35.748517+0000) 2019-10-13T15:44:36.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:36.285+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:36.288895+0000) 2019-10-13T15:44:37.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:37.149+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.154494+0000) 2019-10-13T15:44:37.157 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:37.151+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.154863+0000) 2019-10-13T15:44:37.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:37.153+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.155533+0000) 2019-10-13T15:44:37.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:37.153+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.157218+0000) 2019-10-13T15:44:37.160 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:37.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.157501+0000) 2019-10-13T15:44:37.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:37.153+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.157867+0000) 2019-10-13T15:44:37.161 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:37.153+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.158916+0000) 2019-10-13T15:44:37.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:37.159+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:37.164957+0000) 2019-10-13T15:44:37.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:37.161+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:42.172 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:42.163+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.169630+0000) 2019-10-13T15:44:42.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:42.165+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.170045+0000) 2019-10-13T15:44:42.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:42.165+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.170781+0000) 2019-10-13T15:44:42.174 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:42.169+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.171655+0000) 2019-10-13T15:44:42.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:42.169+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.172989+0000) 2019-10-13T15:44:42.176 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:42.169+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.173651+0000) 2019-10-13T15:44:42.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:42.169+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.174034+0000) 2019-10-13T15:44:42.180 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:42.171+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.177773+0000) 2019-10-13T15:44:42.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:42.177+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227404 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:42.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:42.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:42.727330+0000) 2019-10-13T15:44:43.462 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:44:43.462 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:44:43.465 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:44:43.466 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:44:43.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:43.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:43.463380+0000) 2019-10-13T15:44:43.714 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:43.707+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:43.712483+0000) 2019-10-13T15:44:45.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:45.451+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:45.453737+0000) 2019-10-13T15:44:45.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:45.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:45.737393+0000) 2019-10-13T15:44:45.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:45.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:45.737567+0000) 2019-10-13T15:44:45.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:45.744+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:45.748729+0000) 2019-10-13T15:44:46.291 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:46.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:46.289099+0000) 2019-10-13T15:44:47.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:47.179+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.185669+0000) 2019-10-13T15:44:47.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:47.180+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.186162+0000) 2019-10-13T15:44:47.189 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:47.180+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.186576+0000) 2019-10-13T15:44:47.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:47.180+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.186862+0000) 2019-10-13T15:44:47.190 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:47.180+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.187277+0000) 2019-10-13T15:44:47.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:47.184+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.187842+0000) 2019-10-13T15:44:47.191 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:47.184+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.188804+0000) 2019-10-13T15:44:47.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:47.191+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:47.194774+0000) 2019-10-13T15:44:47.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:47.192+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:52.202 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:52.195+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.200125+0000) 2019-10-13T15:44:52.203 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:52.196+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.201295+0000) 2019-10-13T15:44:52.204 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:52.196+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.201697+0000) 2019-10-13T15:44:52.204 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:52.196+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.202174+0000) 2019-10-13T15:44:52.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:52.196+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.202787+0000) 2019-10-13T15:44:52.206 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:52.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.204121+0000) 2019-10-13T15:44:52.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:52.200+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.204288+0000) 2019-10-13T15:44:52.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:52.203+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.206984+0000) 2019-10-13T15:44:52.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:52.208+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:44:52.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:52.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:52.727516+0000) 2019-10-13T15:44:53.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:53.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:53.463545+0000) 2019-10-13T15:44:53.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:53.707+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:53.712691+0000) 2019-10-13T15:44:55.473 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:55.463+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:55.465512+0000) 2019-10-13T15:44:55.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:55.732+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:55.737600+0000) 2019-10-13T15:44:55.740 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:55.732+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:55.737768+0000) 2019-10-13T15:44:55.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:55.748+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:55.748931+0000) 2019-10-13T15:44:56.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:56.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:56.289339+0000) 2019-10-13T15:44:57.217 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:44:57.208+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.214242+0000) 2019-10-13T15:44:57.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:44:57.211+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.215104+0000) 2019-10-13T15:44:57.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:44:57.212+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.215732+0000) 2019-10-13T15:44:57.219 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:44:57.212+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.216946+0000) 2019-10-13T15:44:57.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:44:57.212+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.216669+0000) 2019-10-13T15:44:57.220 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:44:57.212+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.217177+0000) 2019-10-13T15:44:57.220 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:44:57.212+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.217647+0000) 2019-10-13T15:44:57.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:44:57.215+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:44:57.221150+0000) 2019-10-13T15:44:57.228 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:44:57.220+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:02.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:02.224+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.229413+0000) 2019-10-13T15:45:02.232 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:02.224+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.230446+0000) 2019-10-13T15:45:02.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:02.224+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.231125+0000) 2019-10-13T15:45:02.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:02.228+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.231734+0000) 2019-10-13T15:45:02.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:02.226+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.231991+0000) 2019-10-13T15:45:02.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:02.228+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.232177+0000) 2019-10-13T15:45:02.235 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:02.228+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.232536+0000) 2019-10-13T15:45:02.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:02.234+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.239863+0000) 2019-10-13T15:45:02.243 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:02.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227540 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:02.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:02.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:02.727701+0000) 2019-10-13T15:45:03.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:03.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:03.463777+0000) 2019-10-13T15:45:03.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:03.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:03.712885+0000) 2019-10-13T15:45:05.485 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:05.470+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:05.472220+0000) 2019-10-13T15:45:05.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:05.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:05.737800+0000) 2019-10-13T15:45:05.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:05.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:05.737967+0000) 2019-10-13T15:45:05.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:05.748+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:05.749133+0000) 2019-10-13T15:45:06.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:06.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:06.289545+0000) 2019-10-13T15:45:07.247 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:07.240+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.244862+0000) 2019-10-13T15:45:07.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:07.238+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.245213+0000) 2019-10-13T15:45:07.248 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:07.240+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.245832+0000) 2019-10-13T15:45:07.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:07.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.246265+0000) 2019-10-13T15:45:07.249 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:07.240+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.246718+0000) 2019-10-13T15:45:07.249 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:07.240+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.247030+0000) 2019-10-13T15:45:07.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:07.244+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.247644+0000) 2019-10-13T15:45:07.256 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:07.250+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:07.254270+0000) 2019-10-13T15:45:07.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:07.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:12.261 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:12.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.258456+0000) 2019-10-13T15:45:12.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:12.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.258996+0000) 2019-10-13T15:45:12.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:12.254+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.259780+0000) 2019-10-13T15:45:12.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:12.256+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.260128+0000) 2019-10-13T15:45:12.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:12.256+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.261072+0000) 2019-10-13T15:45:12.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:12.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.261173+0000) 2019-10-13T15:45:12.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:12.256+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.261608+0000) 2019-10-13T15:45:12.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:12.262+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.266674+0000) 2019-10-13T15:45:12.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:12.264+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:12.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:12.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:12.727898+0000) 2019-10-13T15:45:13.466 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:13.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:13.463971+0000) 2019-10-13T15:45:13.560 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:45:13.560 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:45:13.564 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:45:13.564 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:45:13.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:13.706+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:13.713088+0000) 2019-10-13T15:45:15.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:15.474+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:15.478691+0000) 2019-10-13T15:45:15.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:15.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:15.738011+0000) 2019-10-13T15:45:15.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:15.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:15.738147+0000) 2019-10-13T15:45:15.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:15.744+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:15.749352+0000) 2019-10-13T15:45:16.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:16.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:16.289739+0000) 2019-10-13T15:45:17.276 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:17.268+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.274013+0000) 2019-10-13T15:45:17.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:17.270+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.274429+0000) 2019-10-13T15:45:17.277 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:17.268+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.274758+0000) 2019-10-13T15:45:17.278 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:17.268+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.275158+0000) 2019-10-13T15:45:17.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:17.272+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.275831+0000) 2019-10-13T15:45:17.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:17.272+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.276482+0000) 2019-10-13T15:45:17.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:17.272+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.276453+0000) 2019-10-13T15:45:17.285 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:17.278+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:17.283401+0000) 2019-10-13T15:45:17.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:17.280+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:22.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:22.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.286916+0000) 2019-10-13T15:45:22.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:22.284+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.287426+0000) 2019-10-13T15:45:22.290 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:22.284+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.288233+0000) 2019-10-13T15:45:22.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:22.284+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.288599+0000) 2019-10-13T15:45:22.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:22.284+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.288806+0000) 2019-10-13T15:45:22.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:22.284+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.289745+0000) 2019-10-13T15:45:22.292 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:22.286+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.290195+0000) 2019-10-13T15:45:22.300 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:22.290+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.297690+0000) 2019-10-13T15:45:22.301 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:22.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:22.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:22.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:22.728108+0000) 2019-10-13T15:45:23.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:23.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:23.464185+0000) 2019-10-13T15:45:23.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:23.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:23.713291+0000) 2019-10-13T15:45:25.487 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:25.482+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:25.485125+0000) 2019-10-13T15:45:25.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:25.736+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:25.738234+0000) 2019-10-13T15:45:25.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:25.736+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:25.738365+0000) 2019-10-13T15:45:25.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:25.748+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:25.749558+0000) 2019-10-13T15:45:26.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:26.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:26.289970+0000) 2019-10-13T15:45:27.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:27.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.301958+0000) 2019-10-13T15:45:27.305 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:27.296+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.302314+0000) 2019-10-13T15:45:27.305 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:27.298+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.302728+0000) 2019-10-13T15:45:27.305 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:27.300+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.303293+0000) 2019-10-13T15:45:27.306 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:27.300+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.303677+0000) 2019-10-13T15:45:27.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:27.300+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.304996+0000) 2019-10-13T15:45:27.307 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:27.300+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.305305+0000) 2019-10-13T15:45:27.313 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:27.306+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:27.311430+0000) 2019-10-13T15:45:27.315 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:27.308+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:32.319 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:32.311+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.316130+0000) 2019-10-13T15:45:32.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:32.311+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.316533+0000) 2019-10-13T15:45:32.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:32.311+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.317296+0000) 2019-10-13T15:45:32.320 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:32.310+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.317736+0000) 2019-10-13T15:45:32.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:32.311+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.317081+0000) 2019-10-13T15:45:32.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:32.311+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.318141+0000) 2019-10-13T15:45:32.321 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:32.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.319021+0000) 2019-10-13T15:45:32.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:32.318+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.324940+0000) 2019-10-13T15:45:32.331 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:32.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:32.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:32.727+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:32.728291+0000) 2019-10-13T15:45:33.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:33.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:33.464414+0000) 2019-10-13T15:45:33.715 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:33.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:33.713488+0000) 2019-10-13T15:45:35.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:35.486+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:35.491566+0000) 2019-10-13T15:45:35.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:35.731+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:35.738433+0000) 2019-10-13T15:45:35.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:35.731+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:35.738545+0000) 2019-10-13T15:45:35.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:35.747+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:35.749792+0000) 2019-10-13T15:45:36.292 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:36.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:36.290185+0000) 2019-10-13T15:45:37.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:37.327+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.331285+0000) 2019-10-13T15:45:37.334 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:37.327+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.331952+0000) 2019-10-13T15:45:37.335 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:37.327+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.332592+0000) 2019-10-13T15:45:37.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:37.326+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.333453+0000) 2019-10-13T15:45:37.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:37.327+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.334362+0000) 2019-10-13T15:45:37.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:37.327+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.334840+0000) 2019-10-13T15:45:37.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:37.327+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.335069+0000) 2019-10-13T15:45:37.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:37.334+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:37.340061+0000) 2019-10-13T15:45:37.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:37.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227781 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:42.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:42.343+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.346916+0000) 2019-10-13T15:45:42.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:42.343+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.347683+0000) 2019-10-13T15:45:42.350 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:42.343+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.347813+0000) 2019-10-13T15:45:42.351 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:42.343+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.348882+0000) 2019-10-13T15:45:42.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:42.342+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.349219+0000) 2019-10-13T15:45:42.352 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:42.343+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.349444+0000) 2019-10-13T15:45:42.352 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:42.343+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.350124+0000) 2019-10-13T15:45:42.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:42.350+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.357452+0000) 2019-10-13T15:45:42.361 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:42.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:42.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:42.727+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:42.728466+0000) 2019-10-13T15:45:43.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:43.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:43.464565+0000) 2019-10-13T15:45:43.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:43.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:43.713738+0000) 2019-10-13T15:45:43.833 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:45:43.833 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:45:43.837 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:45:43.837 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:45:45.500 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:45.494+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:45.498089+0000) 2019-10-13T15:45:45.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:45.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:45.738660+0000) 2019-10-13T15:45:45.741 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:45.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:45.738710+0000) 2019-10-13T15:45:45.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:45.747+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:45.750022+0000) 2019-10-13T15:45:46.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:46.283+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:46.290348+0000) 2019-10-13T15:45:47.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:47.355+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.361907+0000) 2019-10-13T15:45:47.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:47.355+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.362438+0000) 2019-10-13T15:45:47.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:47.355+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.362729+0000) 2019-10-13T15:45:47.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:47.359+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.363917+0000) 2019-10-13T15:45:47.367 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:47.359+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.364536+0000) 2019-10-13T15:45:47.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:47.362+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.364935+0000) 2019-10-13T15:45:47.368 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:47.359+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.365889+0000) 2019-10-13T15:45:47.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:47.370+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:47.371071+0000) 2019-10-13T15:45:47.377 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:47.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:52.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:52.371+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.378478+0000) 2019-10-13T15:45:52.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:52.371+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.379053+0000) 2019-10-13T15:45:52.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:52.378+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.379718+0000) 2019-10-13T15:45:52.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:52.375+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.380112+0000) 2019-10-13T15:45:52.382 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:52.375+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.380533+0000) 2019-10-13T15:45:52.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:52.375+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.380841+0000) 2019-10-13T15:45:52.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:52.375+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.381668+0000) 2019-10-13T15:45:52.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:52.382+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.384998+0000) 2019-10-13T15:45:52.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:52.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227878 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:45:52.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:52.727+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:52.728671+0000) 2019-10-13T15:45:53.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:53.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:53.464766+0000) 2019-10-13T15:45:53.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:53.709+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:53.713932+0000) 2019-10-13T15:45:55.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:55.501+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:55.504621+0000) 2019-10-13T15:45:55.741 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:55.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:55.738870+0000) 2019-10-13T15:45:55.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:55.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:55.738938+0000) 2019-10-13T15:45:55.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:55.747+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:55.750213+0000) 2019-10-13T15:45:56.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:56.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:56.290560+0000) 2019-10-13T15:45:57.394 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:45:57.387+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.392210+0000) 2019-10-13T15:45:57.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:45:57.387+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.392888+0000) 2019-10-13T15:45:57.396 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:45:57.387+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.393653+0000) 2019-10-13T15:45:57.397 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:45:57.393+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.394850+0000) 2019-10-13T15:45:57.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:45:57.387+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.395128+0000) 2019-10-13T15:45:57.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:45:57.391+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.395628+0000) 2019-10-13T15:45:57.398 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:45:57.391+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.396043+0000) 2019-10-13T15:45:57.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:45:57.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:45:57.400680+0000) 2019-10-13T15:45:57.407 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:45:57.399+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:02.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:02.407+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.408074+0000) 2019-10-13T15:46:02.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:02.407+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.408679+0000) 2019-10-13T15:46:02.412 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:02.407+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.409670+0000) 2019-10-13T15:46:02.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:02.407+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.409846+0000) 2019-10-13T15:46:02.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:02.407+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.410521+0000) 2019-10-13T15:46:02.413 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:02.409+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.410842+0000) 2019-10-13T15:46:02.413 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:02.407+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.411200+0000) 2019-10-13T15:46:02.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:02.417+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.419613+0000) 2019-10-13T15:46:02.422 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:02.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:02.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:02.727+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:02.728853+0000) 2019-10-13T15:46:03.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:03.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:03.464955+0000) 2019-10-13T15:46:03.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:03.713+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:03.714199+0000) 2019-10-13T15:46:05.513 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:05.509+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:05.511124+0000) 2019-10-13T15:46:05.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:05.735+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:05.739097+0000) 2019-10-13T15:46:05.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:05.735+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:05.739141+0000) 2019-10-13T15:46:05.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:05.747+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:05.750407+0000) 2019-10-13T15:46:06.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:06.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:06.290760+0000) 2019-10-13T15:46:07.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:07.419+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.423014+0000) 2019-10-13T15:46:07.426 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:07.423+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.423496+0000) 2019-10-13T15:46:07.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:07.423+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.424195+0000) 2019-10-13T15:46:07.427 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:07.423+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.424708+0000) 2019-10-13T15:46:07.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:07.423+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.425552+0000) 2019-10-13T15:46:07.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:07.421+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.425946+0000) 2019-10-13T15:46:07.429 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:07.423+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.426337+0000) 2019-10-13T15:46:07.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:07.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 227985 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:07.440 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:07.437+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:07.438304+0000) 2019-10-13T15:46:12.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:12.435+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.438122+0000) 2019-10-13T15:46:12.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:12.435+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.438345+0000) 2019-10-13T15:46:12.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:12.435+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.439244+0000) 2019-10-13T15:46:12.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:12.439+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.439842+0000) 2019-10-13T15:46:12.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:12.439+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.440008+0000) 2019-10-13T15:46:12.443 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:12.439+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.441055+0000) 2019-10-13T15:46:12.443 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:12.437+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.441501+0000) 2019-10-13T15:46:12.450 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:12.445+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.448237+0000) 2019-10-13T15:46:12.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:12.447+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228014 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:12.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:12.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:12.729030+0000) 2019-10-13T15:46:13.467 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:13.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:13.465186+0000) 2019-10-13T15:46:13.716 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:13.713+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:13.714353+0000) 2019-10-13T15:46:13.888 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:46:13.888 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:46:13.892 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:46:13.892 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:46:15.519 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:15.513+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:15.517449+0000) 2019-10-13T15:46:15.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:15.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:15.739346+0000) 2019-10-13T15:46:15.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:15.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:15.739338+0000) 2019-10-13T15:46:15.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:15.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:15.750618+0000) 2019-10-13T15:46:16.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:16.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:16.291011+0000) 2019-10-13T15:46:17.457 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:17.450+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.454296+0000) 2019-10-13T15:46:17.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:17.450+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.454793+0000) 2019-10-13T15:46:17.458 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:17.450+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.454989+0000) 2019-10-13T15:46:17.458 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:17.454+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.455467+0000) 2019-10-13T15:46:17.458 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:17.454+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.455743+0000) 2019-10-13T15:46:17.459 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:17.454+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.456365+0000) 2019-10-13T15:46:17.459 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:17.453+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.456734+0000) 2019-10-13T15:46:17.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:17.457+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:17.460685+0000) 2019-10-13T15:46:17.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:17.462+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228054 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:22.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:22.466+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.468081+0000) 2019-10-13T15:46:22.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:22.466+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.468613+0000) 2019-10-13T15:46:22.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:22.466+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.469559+0000) 2019-10-13T15:46:22.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:22.466+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.470179+0000) 2019-10-13T15:46:22.473 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:22.469+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.470768+0000) 2019-10-13T15:46:22.473 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:22.470+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.471397+0000) 2019-10-13T15:46:22.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:22.470+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.471757+0000) 2019-10-13T15:46:22.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:22.477+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.478939+0000) 2019-10-13T15:46:22.481 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:22.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228083 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:22.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:22.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:22.729211+0000) 2019-10-13T15:46:23.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:23.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:23.465393+0000) 2019-10-13T15:46:23.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:23.713+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:23.714546+0000) 2019-10-13T15:46:25.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:25.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:25.523348+0000) 2019-10-13T15:46:25.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:25.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:25.739574+0000) 2019-10-13T15:46:25.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:25.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:25.739574+0000) 2019-10-13T15:46:25.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:25.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:25.750815+0000) 2019-10-13T15:46:26.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:26.286+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:26.291272+0000) 2019-10-13T15:46:27.485 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:27.478+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.482222+0000) 2019-10-13T15:46:27.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:27.478+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.482684+0000) 2019-10-13T15:46:27.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:27.482+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.483488+0000) 2019-10-13T15:46:27.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:27.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.484012+0000) 2019-10-13T15:46:27.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:27.482+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.484766+0000) 2019-10-13T15:46:27.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:27.481+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.485455+0000) 2019-10-13T15:46:27.488 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:27.482+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.485743+0000) 2019-10-13T15:46:27.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:27.489+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:27.492675+0000) 2019-10-13T15:46:27.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:27.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228122 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:32.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:32.494+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.496500+0000) 2019-10-13T15:46:32.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:32.494+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.497234+0000) 2019-10-13T15:46:32.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:32.494+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.497484+0000) 2019-10-13T15:46:32.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:32.494+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.497943+0000) 2019-10-13T15:46:32.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:32.498+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.499469+0000) 2019-10-13T15:46:32.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:32.497+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.500053+0000) 2019-10-13T15:46:32.502 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:32.498+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.500470+0000) 2019-10-13T15:46:32.509 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:32.505+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.507485+0000) 2019-10-13T15:46:32.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:32.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228150 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:32.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:32.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:32.729386+0000) 2019-10-13T15:46:33.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:33.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:33.465570+0000) 2019-10-13T15:46:33.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:33.713+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:33.714741+0000) 2019-10-13T15:46:35.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:35.525+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:35.529869+0000) 2019-10-13T15:46:35.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:35.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:35.739780+0000) 2019-10-13T15:46:35.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:35.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:35.739782+0000) 2019-10-13T15:46:35.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:35.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:35.751010+0000) 2019-10-13T15:46:36.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:36.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:36.291514+0000) 2019-10-13T15:46:37.513 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:37.506+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.511128+0000) 2019-10-13T15:46:37.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:37.510+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.511566+0000) 2019-10-13T15:46:37.514 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:37.510+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.511831+0000) 2019-10-13T15:46:37.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:37.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.512246+0000) 2019-10-13T15:46:37.515 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:37.510+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.512680+0000) 2019-10-13T15:46:37.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:37.510+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.513120+0000) 2019-10-13T15:46:37.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:37.513+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.514324+0000) 2019-10-13T15:46:37.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:37.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228190 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:37.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:37.521+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:37.523705+0000) 2019-10-13T15:46:42.529 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:42.522+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.526353+0000) 2019-10-13T15:46:42.529 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:42.522+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.526982+0000) 2019-10-13T15:46:42.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:42.525+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.527483+0000) 2019-10-13T15:46:42.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:42.526+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.528223+0000) 2019-10-13T15:46:42.531 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:42.526+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.528854+0000) 2019-10-13T15:46:42.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:42.526+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.529648+0000) 2019-10-13T15:46:42.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:42.526+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.529786+0000) 2019-10-13T15:46:42.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:42.529+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.534061+0000) 2019-10-13T15:46:42.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:42.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:42.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:42.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:42.729568+0000) 2019-10-13T15:46:43.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:43.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:43.465729+0000) 2019-10-13T15:46:43.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:43.713+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:43.714934+0000) 2019-10-13T15:46:44.020 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:46:44.021 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:46:44.024 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:46:44.025 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:46:45.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:45.533+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:45.536742+0000) 2019-10-13T15:46:45.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:45.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:45.739999+0000) 2019-10-13T15:46:45.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:45.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:45.740015+0000) 2019-10-13T15:46:45.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:45.746+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:45.751246+0000) 2019-10-13T15:46:46.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:46.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:46.291719+0000) 2019-10-13T15:46:47.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:47.538+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.542453+0000) 2019-10-13T15:46:47.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:47.538+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.542875+0000) 2019-10-13T15:46:47.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:47.542+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.543588+0000) 2019-10-13T15:46:47.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:47.542+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.544187+0000) 2019-10-13T15:46:47.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:47.542+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.544435+0000) 2019-10-13T15:46:47.547 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:47.542+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.544765+0000) 2019-10-13T15:46:47.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:47.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.545230+0000) 2019-10-13T15:46:47.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:47.545+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:47.549755+0000) 2019-10-13T15:46:47.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:47.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228258 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:52.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:52.554+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.557386+0000) 2019-10-13T15:46:52.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:52.554+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.558096+0000) 2019-10-13T15:46:52.561 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:52.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.558679+0000) 2019-10-13T15:46:52.561 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:52.554+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.558864+0000) 2019-10-13T15:46:52.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:52.554+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.559251+0000) 2019-10-13T15:46:52.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:52.558+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.559932+0000) 2019-10-13T15:46:52.562 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:52.558+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.560516+0000) 2019-10-13T15:46:52.566 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:52.560+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.564049+0000) 2019-10-13T15:46:52.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:52.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228286 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:46:52.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:52.726+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:52.729764+0000) 2019-10-13T15:46:53.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:53.462+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:53.465871+0000) 2019-10-13T15:46:53.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:53.712+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:53.715121+0000) 2019-10-13T15:46:55.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:55.540+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:55.543526+0000) 2019-10-13T15:46:55.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:55.738+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:55.740232+0000) 2019-10-13T15:46:55.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:55.738+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:55.740232+0000) 2019-10-13T15:46:55.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:55.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:55.751497+0000) 2019-10-13T15:46:56.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:56.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:56.291919+0000) 2019-10-13T15:46:57.575 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:46:57.569+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.572760+0000) 2019-10-13T15:46:57.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:46:57.569+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.573307+0000) 2019-10-13T15:46:57.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:46:57.569+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.573670+0000) 2019-10-13T15:46:57.576 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:46:57.572+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.574168+0000) 2019-10-13T15:46:57.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:46:57.569+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.574767+0000) 2019-10-13T15:46:57.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:46:57.573+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.575943+0000) 2019-10-13T15:46:57.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:46:57.573+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.576529+0000) 2019-10-13T15:46:57.584 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:46:57.576+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:46:57.581655+0000) 2019-10-13T15:46:57.587 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:46:57.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228325 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:02.591 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:02.585+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.588318+0000) 2019-10-13T15:47:02.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:02.585+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.588980+0000) 2019-10-13T15:47:02.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:02.585+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.589372+0000) 2019-10-13T15:47:02.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:02.585+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.590451+0000) 2019-10-13T15:47:02.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:02.588+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.591324+0000) 2019-10-13T15:47:02.594 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:02.589+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.591633+0000) 2019-10-13T15:47:02.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:02.589+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.592218+0000) 2019-10-13T15:47:02.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:02.592+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.596172+0000) 2019-10-13T15:47:02.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:02.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228355 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:02.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:02.729+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:02.729960+0000) 2019-10-13T15:47:03.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:03.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:03.466010+0000) 2019-10-13T15:47:03.717 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:03.712+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:03.715315+0000) 2019-10-13T15:47:05.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:05.548+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:05.549975+0000) 2019-10-13T15:47:05.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:05.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:05.740448+0000) 2019-10-13T15:47:05.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:05.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:05.740447+0000) 2019-10-13T15:47:05.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:05.749+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:05.751695+0000) 2019-10-13T15:47:06.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:06.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:06.292135+0000) 2019-10-13T15:47:07.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:07.601+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.604247+0000) 2019-10-13T15:47:07.607 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:07.601+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.604762+0000) 2019-10-13T15:47:07.608 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:07.600+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.605177+0000) 2019-10-13T15:47:07.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:07.601+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.605439+0000) 2019-10-13T15:47:07.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:07.601+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.605682+0000) 2019-10-13T15:47:07.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:07.601+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.606471+0000) 2019-10-13T15:47:07.610 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:07.605+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.608043+0000) 2019-10-13T15:47:07.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:07.608+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:07.613756+0000) 2019-10-13T15:47:07.617 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:07.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228393 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:12.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:12.613+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.619005+0000) 2019-10-13T15:47:12.622 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:12.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.619557+0000) 2019-10-13T15:47:12.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:12.616+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.620532+0000) 2019-10-13T15:47:12.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:12.617+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.620217+0000) 2019-10-13T15:47:12.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:12.617+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.621385+0000) 2019-10-13T15:47:12.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:12.617+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.621813+0000) 2019-10-13T15:47:12.624 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:12.617+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.622278+0000) 2019-10-13T15:47:12.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:12.624+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.626880+0000) 2019-10-13T15:47:12.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:12.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228422 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:12.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:12.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:12.730141+0000) 2019-10-13T15:47:13.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:13.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:13.466242+0000) 2019-10-13T15:47:13.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:13.712+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:13.715502+0000) 2019-10-13T15:47:14.092 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:47:14.093 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:47:14.096 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:47:14.096 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:47:15.558 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:15.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:15.556402+0000) 2019-10-13T15:47:15.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:15.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:15.740662+0000) 2019-10-13T15:47:15.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:15.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:15.740653+0000) 2019-10-13T15:47:15.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:15.749+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:15.751894+0000) 2019-10-13T15:47:16.294 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:16.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:16.292340+0000) 2019-10-13T15:47:17.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:17.629+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.633159+0000) 2019-10-13T15:47:17.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:17.629+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.634192+0000) 2019-10-13T15:47:17.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:17.632+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.634607+0000) 2019-10-13T15:47:17.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:17.629+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.635136+0000) 2019-10-13T15:47:17.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:17.633+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.635568+0000) 2019-10-13T15:47:17.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:17.633+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.635894+0000) 2019-10-13T15:47:17.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:17.633+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.636570+0000) 2019-10-13T15:47:17.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:17.636+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:17.640103+0000) 2019-10-13T15:47:17.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:17.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:22.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:22.645+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.648074+0000) 2019-10-13T15:47:22.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:22.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.648277+0000) 2019-10-13T15:47:22.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:22.644+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.649240+0000) 2019-10-13T15:47:22.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:22.645+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.649340+0000) 2019-10-13T15:47:22.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:22.645+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.649678+0000) 2019-10-13T15:47:22.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:22.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.650364+0000) 2019-10-13T15:47:22.653 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:22.645+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.650884+0000) 2019-10-13T15:47:22.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:22.652+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.657867+0000) 2019-10-13T15:47:22.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:22.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228491 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:22.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:22.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:22.730337+0000) 2019-10-13T15:47:23.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:23.461+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:23.466430+0000) 2019-10-13T15:47:23.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:23.712+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:23.715700+0000) 2019-10-13T15:47:25.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:25.560+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:25.562905+0000) 2019-10-13T15:47:25.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:25.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:25.740868+0000) 2019-10-13T15:47:25.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:25.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:25.740860+0000) 2019-10-13T15:47:25.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:25.749+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:25.752092+0000) 2019-10-13T15:47:26.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:26.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:26.292578+0000) 2019-10-13T15:47:27.664 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:27.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.662409+0000) 2019-10-13T15:47:27.665 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:27.657+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.663219+0000) 2019-10-13T15:47:27.666 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:27.661+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.663949+0000) 2019-10-13T15:47:27.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:27.661+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.664288+0000) 2019-10-13T15:47:27.667 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:27.661+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.664777+0000) 2019-10-13T15:47:27.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:27.660+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.665375+0000) 2019-10-13T15:47:27.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:27.661+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.665614+0000) 2019-10-13T15:47:27.671 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:27.664+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:27.669364+0000) 2019-10-13T15:47:27.675 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:27.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:32.679 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:32.673+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.677248+0000) 2019-10-13T15:47:32.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:32.672+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.677674+0000) 2019-10-13T15:47:32.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:32.673+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.678254+0000) 2019-10-13T15:47:32.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:32.673+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.679141+0000) 2019-10-13T15:47:32.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:32.677+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.679786+0000) 2019-10-13T15:47:32.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:32.677+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.680226+0000) 2019-10-13T15:47:32.683 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:32.677+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.680600+0000) 2019-10-13T15:47:32.685 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:32.680+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.683544+0000) 2019-10-13T15:47:32.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:32.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228558 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:32.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:32.725+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:32.730514+0000) 2019-10-13T15:47:33.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:33.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:33.466632+0000) 2019-10-13T15:47:33.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:33.712+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:33.715941+0000) 2019-10-13T15:47:35.577 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:35.572+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:35.574391+0000) 2019-10-13T15:47:35.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:35.737+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:35.741078+0000) 2019-10-13T15:47:35.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:35.737+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:35.741070+0000) 2019-10-13T15:47:35.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:35.749+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:35.752287+0000) 2019-10-13T15:47:36.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:36.289+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:36.292818+0000) 2019-10-13T15:47:37.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:37.685+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.691178+0000) 2019-10-13T15:47:37.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:37.685+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.691279+0000) 2019-10-13T15:47:37.694 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:37.689+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.691845+0000) 2019-10-13T15:47:37.695 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:37.689+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.692309+0000) 2019-10-13T15:47:37.695 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:37.688+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.693055+0000) 2019-10-13T15:47:37.695 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:37.689+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.693476+0000) 2019-10-13T15:47:37.696 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:37.689+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.693912+0000) 2019-10-13T15:47:37.705 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:37.697+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:37.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:37.700+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:37.703214+0000) 2019-10-13T15:47:42.709 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:42.700+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.706643+0000) 2019-10-13T15:47:42.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:42.704+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.707445+0000) 2019-10-13T15:47:42.710 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:42.704+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.707740+0000) 2019-10-13T15:47:42.710 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:42.704+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.708281+0000) 2019-10-13T15:47:42.711 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:42.704+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.708591+0000) 2019-10-13T15:47:42.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:42.704+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.708864+0000) 2019-10-13T15:47:42.711 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:42.704+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.709362+0000) 2019-10-13T15:47:42.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:42.712+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.716547+0000) 2019-10-13T15:47:42.720 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:42.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228627 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:42.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:42.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:42.730657+0000) 2019-10-13T15:47:43.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:43.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:43.466802+0000) 2019-10-13T15:47:43.718 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:43.711+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:43.716151+0000) 2019-10-13T15:47:44.207 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:47:44.207 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:47:44.211 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:47:44.211 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:47:45.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:45.579+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:45.580866+0000) 2019-10-13T15:47:45.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:45.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:45.741283+0000) 2019-10-13T15:47:45.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:45.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:45.741283+0000) 2019-10-13T15:47:45.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:45.748+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:45.752488+0000) 2019-10-13T15:47:46.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:46.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:46.293056+0000) 2019-10-13T15:47:47.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:47.716+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.720732+0000) 2019-10-13T15:47:47.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:47.716+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.721374+0000) 2019-10-13T15:47:47.724 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:47.716+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.721522+0000) 2019-10-13T15:47:47.724 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:47.716+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.722040+0000) 2019-10-13T15:47:47.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:47.719+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.722811+0000) 2019-10-13T15:47:47.726 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:47.720+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.723670+0000) 2019-10-13T15:47:47.726 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:47.720+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.724268+0000) 2019-10-13T15:47:47.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:47.723+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:47.729109+0000) 2019-10-13T15:47:47.734 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:47.728+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228664 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:52.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:52.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.730833+0000) 2019-10-13T15:47:52.737 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:52.732+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.735285+0000) 2019-10-13T15:47:52.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:52.731+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.735694+0000) 2019-10-13T15:47:52.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:52.732+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.736900+0000) 2019-10-13T15:47:52.739 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:52.732+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.737005+0000) 2019-10-13T15:47:52.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:52.732+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.737653+0000) 2019-10-13T15:47:52.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:52.732+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.738353+0000) 2019-10-13T15:47:52.741 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:52.736+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.739379+0000) 2019-10-13T15:47:52.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:52.739+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:52.742351+0000) 2019-10-13T15:47:52.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:52.744+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228694 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:47:53.469 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:53.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:53.466975+0000) 2019-10-13T15:47:53.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:53.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:53.716352+0000) 2019-10-13T15:47:55.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:55.583+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:55.587285+0000) 2019-10-13T15:47:55.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:55.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:55.741525+0000) 2019-10-13T15:47:55.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:55.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:55.741529+0000) 2019-10-13T15:47:55.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:55.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:55.752681+0000) 2019-10-13T15:47:56.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:56.288+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:56.293258+0000) 2019-10-13T15:47:57.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:47:57.744+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.750818+0000) 2019-10-13T15:47:57.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:47:57.748+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.751739+0000) 2019-10-13T15:47:57.754 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:47:57.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.752253+0000) 2019-10-13T15:47:57.755 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:47:57.748+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.752749+0000) 2019-10-13T15:47:57.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:47:57.747+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.753458+0000) 2019-10-13T15:47:57.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:47:57.748+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.753759+0000) 2019-10-13T15:47:57.756 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:47:57.748+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.753994+0000) 2019-10-13T15:47:57.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:47:57.755+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:47:57.761154+0000) 2019-10-13T15:47:57.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:47:57.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228732 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:02.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:02.724+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.731096+0000) 2019-10-13T15:48:02.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:02.760+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.764900+0000) 2019-10-13T15:48:02.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:02.760+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.765888+0000) 2019-10-13T15:48:02.769 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:02.760+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.766319+0000) 2019-10-13T15:48:02.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:02.760+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.765963+0000) 2019-10-13T15:48:02.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:02.760+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.766759+0000) 2019-10-13T15:48:02.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:02.764+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.767844+0000) 2019-10-13T15:48:02.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:02.763+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.768639+0000) 2019-10-13T15:48:02.777 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:02.771+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:02.774703+0000) 2019-10-13T15:48:02.780 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:02.772+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228763 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:03.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:03.460+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:03.467174+0000) 2019-10-13T15:48:03.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:03.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:03.716554+0000) 2019-10-13T15:48:05.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:05.595+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:05.598553+0000) 2019-10-13T15:48:05.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:05.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:05.741794+0000) 2019-10-13T15:48:05.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:05.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:05.741791+0000) 2019-10-13T15:48:05.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:05.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:05.752867+0000) 2019-10-13T15:48:06.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:06.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:06.293458+0000) 2019-10-13T15:48:07.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:07.776+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.781305+0000) 2019-10-13T15:48:07.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:07.776+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.781934+0000) 2019-10-13T15:48:07.785 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:07.776+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.783168+0000) 2019-10-13T15:48:07.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:07.779+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.783392+0000) 2019-10-13T15:48:07.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:07.780+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.783840+0000) 2019-10-13T15:48:07.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:07.780+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.784088+0000) 2019-10-13T15:48:07.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:07.780+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.784900+0000) 2019-10-13T15:48:07.791 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:07.783+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:07.788923+0000) 2019-10-13T15:48:07.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:07.788+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:12.733 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:12.728+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.731295+0000) 2019-10-13T15:48:12.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:12.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.796025+0000) 2019-10-13T15:48:12.799 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:12.792+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.797254+0000) 2019-10-13T15:48:12.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:12.792+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.797813+0000) 2019-10-13T15:48:12.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:12.792+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.798346+0000) 2019-10-13T15:48:12.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:12.792+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.798462+0000) 2019-10-13T15:48:12.801 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:12.795+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.799388+0000) 2019-10-13T15:48:12.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:12.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.799582+0000) 2019-10-13T15:48:12.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:12.803+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:12.806400+0000) 2019-10-13T15:48:12.812 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:12.804+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:13.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:13.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:13.467374+0000) 2019-10-13T15:48:13.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:13.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:13.716730+0000) 2019-10-13T15:48:14.309 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:48:14.309 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:48:14.313 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:48:14.313 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:48:15.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:15.603+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:15.604919+0000) 2019-10-13T15:48:15.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:15.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:15.741977+0000) 2019-10-13T15:48:15.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:15.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:15.742002+0000) 2019-10-13T15:48:15.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:15.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:15.753056+0000) 2019-10-13T15:48:16.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:16.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:16.293659+0000) 2019-10-13T15:48:17.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:17.808+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.812885+0000) 2019-10-13T15:48:17.816 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:17.807+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.813282+0000) 2019-10-13T15:48:17.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:17.808+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.813743+0000) 2019-10-13T15:48:17.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:17.808+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.814439+0000) 2019-10-13T15:48:17.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:17.808+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.814935+0000) 2019-10-13T15:48:17.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:17.812+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.815488+0000) 2019-10-13T15:48:17.818 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:17.812+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.815966+0000) 2019-10-13T15:48:17.827 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:17.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228869 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:17.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:17.823+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:17.826380+0000) 2019-10-13T15:48:22.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:22.728+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.731535+0000) 2019-10-13T15:48:22.831 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:22.824+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.828545+0000) 2019-10-13T15:48:22.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:22.824+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.829213+0000) 2019-10-13T15:48:22.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:22.824+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.829643+0000) 2019-10-13T15:48:22.832 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:22.824+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.830036+0000) 2019-10-13T15:48:22.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:22.827+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.831519+0000) 2019-10-13T15:48:22.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:22.824+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.830640+0000) 2019-10-13T15:48:22.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:22.824+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.831141+0000) 2019-10-13T15:48:22.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:22.835+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:22.838360+0000) 2019-10-13T15:48:22.842 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:22.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228899 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:23.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:23.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:23.467551+0000) 2019-10-13T15:48:23.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:23.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:23.716958+0000) 2019-10-13T15:48:25.616 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:25.611+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:25.613881+0000) 2019-10-13T15:48:25.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:25.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:25.742199+0000) 2019-10-13T15:48:25.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:25.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:25.742182+0000) 2019-10-13T15:48:25.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:25.751+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:25.753284+0000) 2019-10-13T15:48:26.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:26.287+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:26.293858+0000) 2019-10-13T15:48:27.845 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:27.839+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.843110+0000) 2019-10-13T15:48:27.846 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:27.839+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.843764+0000) 2019-10-13T15:48:27.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:27.839+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.843954+0000) 2019-10-13T15:48:27.847 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:27.839+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.844432+0000) 2019-10-13T15:48:27.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:27.839+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.844846+0000) 2019-10-13T15:48:27.848 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:27.839+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.845615+0000) 2019-10-13T15:48:27.848 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:27.839+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.845851+0000) 2019-10-13T15:48:27.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:27.847+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:27.853214+0000) 2019-10-13T15:48:27.857 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:27.847+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:32.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:32.727+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.731736+0000) 2019-10-13T15:48:32.860 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:32.851+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.858103+0000) 2019-10-13T15:48:32.861 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:32.851+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.858484+0000) 2019-10-13T15:48:32.861 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:32.851+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.858935+0000) 2019-10-13T15:48:32.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:32.855+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.859582+0000) 2019-10-13T15:48:32.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:32.851+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.859157+0000) 2019-10-13T15:48:32.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:32.855+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.860313+0000) 2019-10-13T15:48:32.863 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:32.855+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.860584+0000) 2019-10-13T15:48:32.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:32.863+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:32.867733+0000) 2019-10-13T15:48:32.872 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:32.863+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 228965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:33.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:33.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:33.467739+0000) 2019-10-13T15:48:33.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:33.711+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:33.717125+0000) 2019-10-13T15:48:35.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:35.623+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:35.624064+0000) 2019-10-13T15:48:35.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:35.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:35.742404+0000) 2019-10-13T15:48:35.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:35.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:35.742400+0000) 2019-10-13T15:48:35.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:35.751+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:35.753484+0000) 2019-10-13T15:48:36.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:36.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:36.294091+0000) 2019-10-13T15:48:37.875 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:37.867+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.872661+0000) 2019-10-13T15:48:37.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:37.867+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.872994+0000) 2019-10-13T15:48:37.876 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:37.867+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.873940+0000) 2019-10-13T15:48:37.877 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:37.867+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.874191+0000) 2019-10-13T15:48:37.877 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:37.867+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.874929+0000) 2019-10-13T15:48:37.877 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:37.871+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.875459+0000) 2019-10-13T15:48:37.878 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:37.871+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.876427+0000) 2019-10-13T15:48:37.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:37.879+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:37.882316+0000) 2019-10-13T15:48:37.886 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:37.879+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229003 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:42.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:42.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.731942+0000) 2019-10-13T15:48:42.890 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:42.883+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.887898+0000) 2019-10-13T15:48:42.891 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:42.883+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.888932+0000) 2019-10-13T15:48:42.891 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:42.882+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.889237+0000) 2019-10-13T15:48:42.892 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:42.883+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.889528+0000) 2019-10-13T15:48:42.892 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:42.883+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.890158+0000) 2019-10-13T15:48:42.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:42.883+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.889878+0000) 2019-10-13T15:48:42.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:42.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.892193+0000) 2019-10-13T15:48:42.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:42.890+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:42.897974+0000) 2019-10-13T15:48:42.902 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:42.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:43.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:43.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:43.467970+0000) 2019-10-13T15:48:43.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:43.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:43.717296+0000) 2019-10-13T15:48:44.440 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:48:44.440 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:48:44.444 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:48:44.444 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:48:45.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:45.626+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:45.629862+0000) 2019-10-13T15:48:45.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:45.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:45.742610+0000) 2019-10-13T15:48:45.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:45.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:45.742610+0000) 2019-10-13T15:48:45.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:45.751+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:45.753691+0000) 2019-10-13T15:48:46.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:46.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:46.294293+0000) 2019-10-13T15:48:47.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:47.895+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.902713+0000) 2019-10-13T15:48:47.906 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:47.899+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.903962+0000) 2019-10-13T15:48:47.907 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:47.899+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.904621+0000) 2019-10-13T15:48:47.907 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:47.898+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.904938+0000) 2019-10-13T15:48:47.908 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:47.899+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.905126+0000) 2019-10-13T15:48:47.908 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:47.899+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.905758+0000) 2019-10-13T15:48:47.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:47.899+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.905354+0000) 2019-10-13T15:48:47.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:47.910+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:47.914539+0000) 2019-10-13T15:48:47.917 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:47.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:52.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:52.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.732172+0000) 2019-10-13T15:48:52.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:52.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.918072+0000) 2019-10-13T15:48:52.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:52.911+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.918534+0000) 2019-10-13T15:48:52.921 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:52.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.918680+0000) 2019-10-13T15:48:52.922 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:52.915+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.919556+0000) 2019-10-13T15:48:52.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:52.915+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.920016+0000) 2019-10-13T15:48:52.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:52.914+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.921157+0000) 2019-10-13T15:48:52.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:52.915+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.921339+0000) 2019-10-13T15:48:52.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:52.922+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:52.927210+0000) 2019-10-13T15:48:52.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:52.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229099 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:48:53.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:53.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:53.468160+0000) 2019-10-13T15:48:53.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:53.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:53.717465+0000) 2019-10-13T15:48:55.638 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:55.630+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:55.636166+0000) 2019-10-13T15:48:55.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:55.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:55.742793+0000) 2019-10-13T15:48:55.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:55.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:55.742791+0000) 2019-10-13T15:48:55.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:55.751+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:55.753891+0000) 2019-10-13T15:48:56.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:56.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:56.294489+0000) 2019-10-13T15:48:57.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:48:57.931+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.934848+0000) 2019-10-13T15:48:57.938 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:48:57.935+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.935397+0000) 2019-10-13T15:48:57.938 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:48:57.935+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.935748+0000) 2019-10-13T15:48:57.939 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:48:57.935+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.936507+0000) 2019-10-13T15:48:57.939 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:48:57.935+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.936858+0000) 2019-10-13T15:48:57.940 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:48:57.930+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.938062+0000) 2019-10-13T15:48:57.941 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:48:57.935+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.937831+0000) 2019-10-13T15:48:57.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:48:57.938+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:48:57.942828+0000) 2019-10-13T15:48:57.948 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:48:57.943+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:02.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:02.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.732382+0000) 2019-10-13T15:49:02.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:02.947+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.948692+0000) 2019-10-13T15:49:02.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:02.947+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.949195+0000) 2019-10-13T15:49:02.952 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:02.947+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.950199+0000) 2019-10-13T15:49:02.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:02.946+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.950938+0000) 2019-10-13T15:49:02.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:02.951+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.951363+0000) 2019-10-13T15:49:02.954 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:02.951+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.951616+0000) 2019-10-13T15:49:02.954 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:02.951+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.952018+0000) 2019-10-13T15:49:02.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:02.950+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:02.956803+0000) 2019-10-13T15:49:02.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:02.959+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:03.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:03.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:03.468350+0000) 2019-10-13T15:49:03.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:03.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:03.717632+0000) 2019-10-13T15:49:05.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:05.638+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:05.642479+0000) 2019-10-13T15:49:05.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:05.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:05.742990+0000) 2019-10-13T15:49:05.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:05.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:05.742990+0000) 2019-10-13T15:49:05.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:05.751+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:05.754098+0000) 2019-10-13T15:49:06.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:06.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:06.294726+0000) 2019-10-13T15:49:07.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:07.962+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.964067+0000) 2019-10-13T15:49:07.968 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:07.962+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.964634+0000) 2019-10-13T15:49:07.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:07.962+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.964858+0000) 2019-10-13T15:49:07.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:07.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.965558+0000) 2019-10-13T15:49:07.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:07.962+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.966689+0000) 2019-10-13T15:49:07.969 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:07.962+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.965957+0000) 2019-10-13T15:49:07.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:07.962+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.966305+0000) 2019-10-13T15:49:07.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:07.966+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:07.971672+0000) 2019-10-13T15:49:07.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:07.974+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229203 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:12.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:12.730+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.732610+0000) 2019-10-13T15:49:12.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:12.974+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.978490+0000) 2019-10-13T15:49:12.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:12.978+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.979478+0000) 2019-10-13T15:49:12.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:12.974+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.980675+0000) 2019-10-13T15:49:12.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:12.978+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.980931+0000) 2019-10-13T15:49:12.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:12.978+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.981232+0000) 2019-10-13T15:49:12.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:12.978+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.981940+0000) 2019-10-13T15:49:12.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:12.978+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.982379+0000) 2019-10-13T15:49:12.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:12.982+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:12.988597+0000) 2019-10-13T15:49:12.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:12.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:13.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:13.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:13.468531+0000) 2019-10-13T15:49:13.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:13.710+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:13.717818+0000) 2019-10-13T15:49:14.580 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:49:14.581 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:49:14.584 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:49:14.584 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:49:15.651 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:15.646+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:15.648991+0000) 2019-10-13T15:49:15.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:15.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:15.743167+0000) 2019-10-13T15:49:15.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:15.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:15.743167+0000) 2019-10-13T15:49:15.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:15.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:15.754284+0000) 2019-10-13T15:49:16.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:16.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:16.294941+0000) 2019-10-13T15:49:17.995 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:17.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.993259+0000) 2019-10-13T15:49:17.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:17.990+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.994172+0000) 2019-10-13T15:49:17.998 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:17.990+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.994627+0000) 2019-10-13T15:49:17.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:17.990+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.994878+0000) 2019-10-13T15:49:17.998 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:17.994+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.995540+0000) 2019-10-13T15:49:17.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:17.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.997014+0000) 2019-10-13T15:49:17.999 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:17.994+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:17.997371+0000) 2019-10-13T15:49:18.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:17.998+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:18.002788+0000) 2019-10-13T15:49:18.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:18.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229271 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:22.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:22.730+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:22.732808+0000) 2019-10-13T15:49:23.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:23.006+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.009044+0000) 2019-10-13T15:49:23.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:23.006+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.010137+0000) 2019-10-13T15:49:23.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:23.006+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.010617+0000) 2019-10-13T15:49:23.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:23.006+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.010833+0000) 2019-10-13T15:49:23.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:23.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.011101+0000) 2019-10-13T15:49:23.014 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:23.010+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.011618+0000) 2019-10-13T15:49:23.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:23.010+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.011854+0000) 2019-10-13T15:49:23.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:23.014+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.019510+0000) 2019-10-13T15:49:23.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:23.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229300 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:23.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:23.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.468713+0000) 2019-10-13T15:49:23.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:23.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:23.717990+0000) 2019-10-13T15:49:25.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:25.654+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:25.655484+0000) 2019-10-13T15:49:25.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:25.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:25.743364+0000) 2019-10-13T15:49:25.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:25.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:25.743376+0000) 2019-10-13T15:49:25.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:25.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:25.754482+0000) 2019-10-13T15:49:26.297 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:26.290+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:26.295142+0000) 2019-10-13T15:49:28.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:28.022+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.024326+0000) 2019-10-13T15:49:28.028 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:28.022+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.025491+0000) 2019-10-13T15:49:28.028 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:28.022+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.025821+0000) 2019-10-13T15:49:28.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:28.026+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.026866+0000) 2019-10-13T15:49:28.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:28.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.027253+0000) 2019-10-13T15:49:28.030 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:28.022+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.027282+0000) 2019-10-13T15:49:28.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:28.026+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.028108+0000) 2019-10-13T15:49:28.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:28.030+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:28.032776+0000) 2019-10-13T15:49:28.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:28.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:32.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:32.730+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:32.733002+0000) 2019-10-13T15:49:33.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:33.038+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.039629+0000) 2019-10-13T15:49:33.043 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:33.038+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.040617+0000) 2019-10-13T15:49:33.043 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:33.038+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.041314+0000) 2019-10-13T15:49:33.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:33.038+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.042427+0000) 2019-10-13T15:49:33.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:33.038+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.041954+0000) 2019-10-13T15:49:33.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:33.038+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.041718+0000) 2019-10-13T15:49:33.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:33.042+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.043398+0000) 2019-10-13T15:49:33.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:33.046+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.049932+0000) 2019-10-13T15:49:33.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:33.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229366 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:33.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:33.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.468920+0000) 2019-10-13T15:49:33.720 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:33.718+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:33.718206+0000) 2019-10-13T15:49:35.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:35.661+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:35.662219+0000) 2019-10-13T15:49:35.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:35.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:35.743535+0000) 2019-10-13T15:49:35.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:35.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:35.743542+0000) 2019-10-13T15:49:35.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:35.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:35.754672+0000) 2019-10-13T15:49:36.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:36.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:36.295363+0000) 2019-10-13T15:49:38.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:38.054+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.056203+0000) 2019-10-13T15:49:38.059 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:38.054+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.056584+0000) 2019-10-13T15:49:38.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:38.054+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.056921+0000) 2019-10-13T15:49:38.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:38.054+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.057744+0000) 2019-10-13T15:49:38.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:38.057+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.058764+0000) 2019-10-13T15:49:38.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:38.054+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.059091+0000) 2019-10-13T15:49:38.062 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:38.058+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.060352+0000) 2019-10-13T15:49:38.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:38.061+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:38.062344+0000) 2019-10-13T15:49:38.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:38.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:42.735 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:42.730+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:42.733205+0000) 2019-10-13T15:49:43.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:43.069+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.071256+0000) 2019-10-13T15:49:43.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:43.070+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.072371+0000) 2019-10-13T15:49:43.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:43.070+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.072803+0000) 2019-10-13T15:49:43.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:43.070+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.073429+0000) 2019-10-13T15:49:43.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:43.070+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.073795+0000) 2019-10-13T15:49:43.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:43.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.074229+0000) 2019-10-13T15:49:43.077 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:43.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.074766+0000) 2019-10-13T15:49:43.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:43.073+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.077895+0000) 2019-10-13T15:49:43.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:43.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229434 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:43.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:43.466+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.469095+0000) 2019-10-13T15:49:43.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:43.717+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:43.718384+0000) 2019-10-13T15:49:44.712 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:49:44.712 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:49:44.716 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:49:44.716 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:49:45.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:45.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:45.668300+0000) 2019-10-13T15:49:45.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:45.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:45.743736+0000) 2019-10-13T15:49:45.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:45.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:45.743744+0000) 2019-10-13T15:49:45.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:45.750+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:45.754873+0000) 2019-10-13T15:49:46.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:46.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:46.295592+0000) 2019-10-13T15:49:48.088 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:48.082+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.085208+0000) 2019-10-13T15:49:48.088 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:48.082+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.085652+0000) 2019-10-13T15:49:48.088 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:48.082+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.086239+0000) 2019-10-13T15:49:48.089 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:48.082+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.087059+0000) 2019-10-13T15:49:48.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:48.085+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.088267+0000) 2019-10-13T15:49:48.091 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:48.086+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.088540+0000) 2019-10-13T15:49:48.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:48.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.088817+0000) 2019-10-13T15:49:48.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:48.089+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:48.093344+0000) 2019-10-13T15:49:48.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:48.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229471 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:52.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:52.729+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:52.733409+0000) 2019-10-13T15:49:53.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:53.097+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.102026+0000) 2019-10-13T15:49:53.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:53.097+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.102638+0000) 2019-10-13T15:49:53.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:53.097+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.103158+0000) 2019-10-13T15:49:53.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:53.101+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.103685+0000) 2019-10-13T15:49:53.106 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:53.101+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.104007+0000) 2019-10-13T15:49:53.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:53.101+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.104595+0000) 2019-10-13T15:49:53.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:53.101+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.104898+0000) 2019-10-13T15:49:53.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:53.109+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.110987+0000) 2019-10-13T15:49:53.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:53.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229501 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:53.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:53.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.469282+0000) 2019-10-13T15:49:53.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:53.717+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:53.718511+0000) 2019-10-13T15:49:55.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:55.673+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:55.674718+0000) 2019-10-13T15:49:55.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:55.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:55.743908+0000) 2019-10-13T15:49:55.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:55.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:55.743947+0000) 2019-10-13T15:49:55.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:55.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:55.755067+0000) 2019-10-13T15:49:56.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:56.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:56.295848+0000) 2019-10-13T15:49:58.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:49:58.113+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.116384+0000) 2019-10-13T15:49:58.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:49:58.113+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.116660+0000) 2019-10-13T15:49:58.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:49:58.113+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.117815+0000) 2019-10-13T15:49:58.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:49:58.117+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.118514+0000) 2019-10-13T15:49:58.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:49:58.113+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.118922+0000) 2019-10-13T15:49:58.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:49:58.113+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.119156+0000) 2019-10-13T15:49:58.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:49:58.117+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.120448+0000) 2019-10-13T15:49:58.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:49:58.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229539 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:49:58.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:49:58.125+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:49:58.128762+0000) 2019-10-13T15:50:02.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:02.729+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:02.733659+0000) 2019-10-13T15:50:03.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:03.129+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.131437+0000) 2019-10-13T15:50:03.134 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:03.129+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.131785+0000) 2019-10-13T15:50:03.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:03.129+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.132208+0000) 2019-10-13T15:50:03.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:03.129+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.133730+0000) 2019-10-13T15:50:03.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:03.129+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.133914+0000) 2019-10-13T15:50:03.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:03.129+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.134320+0000) 2019-10-13T15:50:03.137 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:03.129+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.135216+0000) 2019-10-13T15:50:03.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:03.137+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.140951+0000) 2019-10-13T15:50:03.145 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:03.137+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229569 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:03.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:03.465+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.469489+0000) 2019-10-13T15:50:03.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:03.717+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:03.718678+0000) 2019-10-13T15:50:05.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:05.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:05.681265+0000) 2019-10-13T15:50:05.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:05.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:05.744083+0000) 2019-10-13T15:50:05.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:05.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:05.744128+0000) 2019-10-13T15:50:05.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:05.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:05.755290+0000) 2019-10-13T15:50:06.298 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:06.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:06.296050+0000) 2019-10-13T15:50:08.148 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:08.141+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.146133+0000) 2019-10-13T15:50:08.149 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:08.141+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.146980+0000) 2019-10-13T15:50:08.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:08.145+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.147495+0000) 2019-10-13T15:50:08.150 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:08.145+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.148276+0000) 2019-10-13T15:50:08.151 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:08.145+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.148584+0000) 2019-10-13T15:50:08.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:08.145+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.149470+0000) 2019-10-13T15:50:08.152 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:08.145+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.149986+0000) 2019-10-13T15:50:08.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:08.153+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:08.155415+0000) 2019-10-13T15:50:08.160 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:08.153+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:12.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:12.729+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:12.733860+0000) 2019-10-13T15:50:13.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:13.157+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.160687+0000) 2019-10-13T15:50:13.164 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:13.157+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.161521+0000) 2019-10-13T15:50:13.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:13.157+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.162077+0000) 2019-10-13T15:50:13.165 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:13.157+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.162335+0000) 2019-10-13T15:50:13.165 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:13.157+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.162802+0000) 2019-10-13T15:50:13.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:13.161+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.163675+0000) 2019-10-13T15:50:13.167 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:13.161+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.165322+0000) 2019-10-13T15:50:13.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:13.169+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.170658+0000) 2019-10-13T15:50:13.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:13.169+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:13.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:13.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.469726+0000) 2019-10-13T15:50:13.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:13.717+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:13.718851+0000) 2019-10-13T15:50:14.821 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:50:14.822 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:50:14.825 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:50:14.826 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:50:15.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:15.685+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:15.687961+0000) 2019-10-13T15:50:15.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:15.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:15.744266+0000) 2019-10-13T15:50:15.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:15.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:15.744344+0000) 2019-10-13T15:50:15.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:15.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:15.755518+0000) 2019-10-13T15:50:16.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:16.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:16.296281+0000) 2019-10-13T15:50:18.179 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:18.173+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.176376+0000) 2019-10-13T15:50:18.180 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:18.173+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.177078+0000) 2019-10-13T15:50:18.181 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:18.173+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.177630+0000) 2019-10-13T15:50:18.181 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:18.173+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.177883+0000) 2019-10-13T15:50:18.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:18.173+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.178623+0000) 2019-10-13T15:50:18.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:18.173+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.178763+0000) 2019-10-13T15:50:18.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:18.177+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.179932+0000) 2019-10-13T15:50:18.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:18.185+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:18.186276+0000) 2019-10-13T15:50:18.191 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:18.185+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229673 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:22.736 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:22.729+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:22.734072+0000) 2019-10-13T15:50:23.194 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:23.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.191712+0000) 2019-10-13T15:50:23.195 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:23.189+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.192651+0000) 2019-10-13T15:50:23.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:23.189+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.193113+0000) 2019-10-13T15:50:23.196 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:23.189+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.193589+0000) 2019-10-13T15:50:23.196 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:23.189+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.194282+0000) 2019-10-13T15:50:23.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:23.189+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.194541+0000) 2019-10-13T15:50:23.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:23.193+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.195708+0000) 2019-10-13T15:50:23.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:23.197+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.201622+0000) 2019-10-13T15:50:23.206 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:23.201+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229702 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:23.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:23.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.469901+0000) 2019-10-13T15:50:23.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:23.717+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:23.719045+0000) 2019-10-13T15:50:25.697 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:25.693+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:25.694497+0000) 2019-10-13T15:50:25.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:25.741+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:25.744423+0000) 2019-10-13T15:50:25.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:25.741+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:25.744548+0000) 2019-10-13T15:50:25.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:25.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:25.755715+0000) 2019-10-13T15:50:26.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:26.293+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:26.296499+0000) 2019-10-13T15:50:28.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:28.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.207903+0000) 2019-10-13T15:50:28.210 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:28.205+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.208180+0000) 2019-10-13T15:50:28.211 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:28.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.208872+0000) 2019-10-13T15:50:28.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:28.205+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.209168+0000) 2019-10-13T15:50:28.212 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:28.205+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.209492+0000) 2019-10-13T15:50:28.212 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:28.205+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.209820+0000) 2019-10-13T15:50:28.212 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:28.205+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.210488+0000) 2019-10-13T15:50:28.215 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:28.209+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:28.213424+0000) 2019-10-13T15:50:28.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:28.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229739 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:32.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:32.733+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:32.734272+0000) 2019-10-13T15:50:33.224 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:33.220+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.222238+0000) 2019-10-13T15:50:33.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:33.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.222619+0000) 2019-10-13T15:50:33.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:33.217+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.223198+0000) 2019-10-13T15:50:33.227 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:33.221+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.224932+0000) 2019-10-13T15:50:33.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:33.221+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.225339+0000) 2019-10-13T15:50:33.228 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:33.221+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.225807+0000) 2019-10-13T15:50:33.228 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:33.221+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.226131+0000) 2019-10-13T15:50:33.231 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:33.224+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.229360+0000) 2019-10-13T15:50:33.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:33.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229769 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:33.472 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:33.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.470081+0000) 2019-10-13T15:50:33.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:33.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:33.719194+0000) 2019-10-13T15:50:35.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:35.696+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:35.700825+0000) 2019-10-13T15:50:35.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:35.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:35.744673+0000) 2019-10-13T15:50:35.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:35.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:35.744771+0000) 2019-10-13T15:50:35.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:35.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:35.755905+0000) 2019-10-13T15:50:36.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:36.292+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:36.296737+0000) 2019-10-13T15:50:38.241 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:38.232+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.237938+0000) 2019-10-13T15:50:38.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:38.232+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.238348+0000) 2019-10-13T15:50:38.241 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:38.232+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.238545+0000) 2019-10-13T15:50:38.242 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:38.236+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.239375+0000) 2019-10-13T15:50:38.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:38.232+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.239062+0000) 2019-10-13T15:50:38.243 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:38.236+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.240616+0000) 2019-10-13T15:50:38.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:38.236+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.241203+0000) 2019-10-13T15:50:38.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:38.244+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:38.247766+0000) 2019-10-13T15:50:38.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:38.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:42.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:42.732+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:42.734467+0000) 2019-10-13T15:50:43.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:43.248+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.253388+0000) 2019-10-13T15:50:43.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:43.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.254546+0000) 2019-10-13T15:50:43.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:43.248+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.254841+0000) 2019-10-13T15:50:43.258 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:43.248+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.255217+0000) 2019-10-13T15:50:43.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:43.252+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.255444+0000) 2019-10-13T15:50:43.259 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:43.252+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.255854+0000) 2019-10-13T15:50:43.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:43.252+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.256519+0000) 2019-10-13T15:50:43.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:43.256+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.260599+0000) 2019-10-13T15:50:43.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:43.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229836 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:43.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:43.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.470214+0000) 2019-10-13T15:50:43.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:43.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:43.719317+0000) 2019-10-13T15:50:44.867 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:50:44.867 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:50:44.871 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:50:44.871 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:50:45.710 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:45.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:45.707381+0000) 2019-10-13T15:50:45.747 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:45.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:45.744881+0000) 2019-10-13T15:50:45.747 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:45.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:45.745002+0000) 2019-10-13T15:50:45.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:45.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:45.756125+0000) 2019-10-13T15:50:46.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:46.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:46.296944+0000) 2019-10-13T15:50:48.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:48.264+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.267593+0000) 2019-10-13T15:50:48.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:48.264+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.268317+0000) 2019-10-13T15:50:48.271 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:48.264+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.268667+0000) 2019-10-13T15:50:48.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:48.264+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.269041+0000) 2019-10-13T15:50:48.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:48.264+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.269480+0000) 2019-10-13T15:50:48.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:48.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.269953+0000) 2019-10-13T15:50:48.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:48.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.271101+0000) 2019-10-13T15:50:48.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:48.272+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:48.277003+0000) 2019-10-13T15:50:48.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:48.272+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:52.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:52.728+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:52.734661+0000) 2019-10-13T15:50:53.285 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:53.276+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.282114+0000) 2019-10-13T15:50:53.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:53.276+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.282274+0000) 2019-10-13T15:50:53.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:53.280+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.282796+0000) 2019-10-13T15:50:53.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:53.276+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.283092+0000) 2019-10-13T15:50:53.286 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:53.280+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.283990+0000) 2019-10-13T15:50:53.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:53.280+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.284491+0000) 2019-10-13T15:50:53.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:53.280+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.284789+0000) 2019-10-13T15:50:53.294 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:53.288+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.292438+0000) 2019-10-13T15:50:53.297 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:53.288+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:50:53.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:53.464+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.470394+0000) 2019-10-13T15:50:53.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:53.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:53.719504+0000) 2019-10-13T15:50:55.715 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:55.708+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:55.713217+0000) 2019-10-13T15:50:55.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:55.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:55.745122+0000) 2019-10-13T15:50:55.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:55.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:55.745222+0000) 2019-10-13T15:50:55.758 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:55.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:55.756350+0000) 2019-10-13T15:50:56.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:56.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:56.297148+0000) 2019-10-13T15:50:58.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:50:58.292+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.298354+0000) 2019-10-13T15:50:58.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:50:58.296+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.298983+0000) 2019-10-13T15:50:58.302 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:50:58.292+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.299223+0000) 2019-10-13T15:50:58.302 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:50:58.296+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.299995+0000) 2019-10-13T15:50:58.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:50:58.296+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.300416+0000) 2019-10-13T15:50:58.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:50:58.296+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.301580+0000) 2019-10-13T15:50:58.304 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:50:58.296+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.301875+0000) 2019-10-13T15:50:58.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:50:58.300+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:50:58.306079+0000) 2019-10-13T15:50:58.313 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:50:58.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229940 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:02.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:02.728+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:02.734825+0000) 2019-10-13T15:51:03.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:03.308+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.314581+0000) 2019-10-13T15:51:03.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:03.312+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.315383+0000) 2019-10-13T15:51:03.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:03.312+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.315941+0000) 2019-10-13T15:51:03.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:03.312+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.316416+0000) 2019-10-13T15:51:03.319 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:03.312+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.317168+0000) 2019-10-13T15:51:03.320 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:03.312+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.317886+0000) 2019-10-13T15:51:03.320 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:03.312+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.318340+0000) 2019-10-13T15:51:03.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:03.320+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.323294+0000) 2019-10-13T15:51:03.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:03.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 229969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:03.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:03.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.470575+0000) 2019-10-13T15:51:03.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:03.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:03.719675+0000) 2019-10-13T15:51:05.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:05.716+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:05.719652+0000) 2019-10-13T15:51:05.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:05.740+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:05.745359+0000) 2019-10-13T15:51:05.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:05.740+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:05.745446+0000) 2019-10-13T15:51:05.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:05.752+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:05.756549+0000) 2019-10-13T15:51:06.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:06.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:06.297306+0000) 2019-10-13T15:51:08.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:08.324+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.330294+0000) 2019-10-13T15:51:08.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:08.328+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.330733+0000) 2019-10-13T15:51:08.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:08.328+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.331441+0000) 2019-10-13T15:51:08.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:08.328+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.331805+0000) 2019-10-13T15:51:08.335 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:08.328+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.332071+0000) 2019-10-13T15:51:08.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:08.328+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.332657+0000) 2019-10-13T15:51:08.335 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:08.328+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.333430+0000) 2019-10-13T15:51:08.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:08.336+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:08.340575+0000) 2019-10-13T15:51:08.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:08.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:12.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:12.728+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:12.735070+0000) 2019-10-13T15:51:13.347 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:13.340+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.344777+0000) 2019-10-13T15:51:13.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:13.340+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.345349+0000) 2019-10-13T15:51:13.349 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:13.340+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.346377+0000) 2019-10-13T15:51:13.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:13.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.346975+0000) 2019-10-13T15:51:13.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:13.344+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.347715+0000) 2019-10-13T15:51:13.351 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:13.344+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.348130+0000) 2019-10-13T15:51:13.351 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:13.344+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.348494+0000) 2019-10-13T15:51:13.359 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:13.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:13.360 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:13.352+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.357670+0000) 2019-10-13T15:51:13.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:13.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.470756+0000) 2019-10-13T15:51:13.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:13.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:13.719847+0000) 2019-10-13T15:51:15.002 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:51:15.002 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:51:15.005 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:51:15.005 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:51:15.731 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:15.728+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:15.729297+0000) 2019-10-13T15:51:15.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:15.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:15.745563+0000) 2019-10-13T15:51:15.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:15.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:15.745680+0000) 2019-10-13T15:51:15.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:15.756+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:15.756762+0000) 2019-10-13T15:51:16.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:16.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:16.297549+0000) 2019-10-13T15:51:18.362 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:18.355+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.359983+0000) 2019-10-13T15:51:18.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:18.355+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.361987+0000) 2019-10-13T15:51:18.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:18.355+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.362420+0000) 2019-10-13T15:51:18.365 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:18.355+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.362990+0000) 2019-10-13T15:51:18.365 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:18.360+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.363432+0000) 2019-10-13T15:51:18.366 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:18.359+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.363902+0000) 2019-10-13T15:51:18.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:18.359+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.364583+0000) 2019-10-13T15:51:18.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:18.364+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:18.367928+0000) 2019-10-13T15:51:18.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:18.367+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:22.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:22.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:22.735270+0000) 2019-10-13T15:51:23.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:23.371+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.376556+0000) 2019-10-13T15:51:23.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:23.371+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.377490+0000) 2019-10-13T15:51:23.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:23.371+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.377692+0000) 2019-10-13T15:51:23.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:23.371+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.378419+0000) 2019-10-13T15:51:23.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:23.371+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.378426+0000) 2019-10-13T15:51:23.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:23.371+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.379026+0000) 2019-10-13T15:51:23.382 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:23.376+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.379573+0000) 2019-10-13T15:51:23.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:23.380+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.383502+0000) 2019-10-13T15:51:23.390 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:23.383+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:23.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:23.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.470942+0000) 2019-10-13T15:51:23.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:23.716+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:23.720058+0000) 2019-10-13T15:51:25.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:25.731+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:25.735791+0000) 2019-10-13T15:51:25.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:25.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:25.745761+0000) 2019-10-13T15:51:25.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:25.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:25.745881+0000) 2019-10-13T15:51:25.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:25.755+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:25.756971+0000) 2019-10-13T15:51:26.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:26.291+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:26.297789+0000) 2019-10-13T15:51:28.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:28.387+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.391928+0000) 2019-10-13T15:51:28.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:28.387+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.392325+0000) 2019-10-13T15:51:28.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:28.387+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.393206+0000) 2019-10-13T15:51:28.396 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:28.387+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.393661+0000) 2019-10-13T15:51:28.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:28.387+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.394803+0000) 2019-10-13T15:51:28.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:28.387+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.395216+0000) 2019-10-13T15:51:28.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:28.391+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.395776+0000) 2019-10-13T15:51:28.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:28.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:28.400153+0000) 2019-10-13T15:51:28.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:28.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230140 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:32.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:32.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:32.735514+0000) 2019-10-13T15:51:33.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:33.399+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.405656+0000) 2019-10-13T15:51:33.409 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:33.399+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.406379+0000) 2019-10-13T15:51:33.409 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:33.399+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.406698+0000) 2019-10-13T15:51:33.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:33.403+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.407930+0000) 2019-10-13T15:51:33.410 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:33.403+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.408436+0000) 2019-10-13T15:51:33.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:33.403+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.408772+0000) 2019-10-13T15:51:33.411 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:33.403+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.409528+0000) 2019-10-13T15:51:33.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:33.411+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.414693+0000) 2019-10-13T15:51:33.418 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:33.411+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:33.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:33.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.471123+0000) 2019-10-13T15:51:33.722 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:33.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:33.720267+0000) 2019-10-13T15:51:35.744 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:35.739+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:35.742214+0000) 2019-10-13T15:51:35.748 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:35.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:35.745983+0000) 2019-10-13T15:51:35.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:35.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:35.746076+0000) 2019-10-13T15:51:35.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:35.755+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:35.757168+0000) 2019-10-13T15:51:36.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:36.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:36.297948+0000) 2019-10-13T15:51:38.423 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:38.415+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.420679+0000) 2019-10-13T15:51:38.423 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:38.415+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.421007+0000) 2019-10-13T15:51:38.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:38.415+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.421353+0000) 2019-10-13T15:51:38.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:38.415+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.421685+0000) 2019-10-13T15:51:38.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:38.419+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.422233+0000) 2019-10-13T15:51:38.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:38.415+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.422470+0000) 2019-10-13T15:51:38.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:38.415+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.422681+0000) 2019-10-13T15:51:38.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:38.419+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:38.425762+0000) 2019-10-13T15:51:38.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:38.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:42.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:42.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:42.735726+0000) 2019-10-13T15:51:43.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:43.431+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.435677+0000) 2019-10-13T15:51:43.439 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:43.431+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.436478+0000) 2019-10-13T15:51:43.439 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:43.431+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.437209+0000) 2019-10-13T15:51:43.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:43.431+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.437617+0000) 2019-10-13T15:51:43.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:43.435+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.438410+0000) 2019-10-13T15:51:43.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:43.431+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.438521+0000) 2019-10-13T15:51:43.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:43.431+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.438775+0000) 2019-10-13T15:51:43.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:43.439+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.444387+0000) 2019-10-13T15:51:43.449 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:43.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:43.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:43.463+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.471285+0000) 2019-10-13T15:51:43.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:43.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:43.720469+0000) 2019-10-13T15:51:45.113 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:51:45.113 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:51:45.117 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:51:45.117 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:51:45.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:45.739+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:45.746184+0000) 2019-10-13T15:51:45.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:45.739+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:45.746301+0000) 2019-10-13T15:51:45.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:45.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:45.748745+0000) 2019-10-13T15:51:45.759 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:45.755+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:45.757363+0000) 2019-10-13T15:51:46.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:46.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:46.298152+0000) 2019-10-13T15:51:48.453 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:48.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.451160+0000) 2019-10-13T15:51:48.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:48.451+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.452063+0000) 2019-10-13T15:51:48.455 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:48.451+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.452444+0000) 2019-10-13T15:51:48.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:48.451+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.453266+0000) 2019-10-13T15:51:48.456 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:48.451+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.453329+0000) 2019-10-13T15:51:48.456 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:48.451+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.454079+0000) 2019-10-13T15:51:48.457 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:48.451+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.454814+0000) 2019-10-13T15:51:48.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:48.455+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:48.458357+0000) 2019-10-13T15:51:48.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:48.459+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:52.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:52.735+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:52.735930+0000) 2019-10-13T15:51:53.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:53.463+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.465392+0000) 2019-10-13T15:51:53.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:53.463+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.466039+0000) 2019-10-13T15:51:53.469 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:53.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.466367+0000) 2019-10-13T15:51:53.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:53.463+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.466945+0000) 2019-10-13T15:51:53.471 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:53.463+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.467393+0000) 2019-10-13T15:51:53.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:53.467+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.468045+0000) 2019-10-13T15:51:53.471 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:53.467+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.468427+0000) 2019-10-13T15:51:53.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:53.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.471438+0000) 2019-10-13T15:51:53.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:53.471+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.474173+0000) 2019-10-13T15:51:53.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:53.475+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230304 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:51:53.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:53.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:53.720638+0000) 2019-10-13T15:51:55.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:55.743+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:55.746387+0000) 2019-10-13T15:51:55.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:55.743+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:55.746524+0000) 2019-10-13T15:51:55.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:55.755+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:55.757588+0000) 2019-10-13T15:51:55.762 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:55.755+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:55.760507+0000) 2019-10-13T15:51:56.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:56.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:56.298359+0000) 2019-10-13T15:51:58.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:51:58.479+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.480259+0000) 2019-10-13T15:51:58.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:51:58.479+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.481071+0000) 2019-10-13T15:51:58.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:51:58.479+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.481768+0000) 2019-10-13T15:51:58.484 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:51:58.479+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.481901+0000) 2019-10-13T15:51:58.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:51:58.479+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.482772+0000) 2019-10-13T15:51:58.486 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:51:58.479+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.482999+0000) 2019-10-13T15:51:58.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:51:58.479+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.483680+0000) 2019-10-13T15:51:58.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:51:58.487+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:51:58.492625+0000) 2019-10-13T15:51:58.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:51:58.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:02.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:02.734+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:02.736175+0000) 2019-10-13T15:52:03.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:03.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.471629+0000) 2019-10-13T15:52:03.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:03.494+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.495900+0000) 2019-10-13T15:52:03.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:03.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.496824+0000) 2019-10-13T15:52:03.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:03.494+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.497756+0000) 2019-10-13T15:52:03.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:03.494+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.498302+0000) 2019-10-13T15:52:03.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:03.494+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.498494+0000) 2019-10-13T15:52:03.501 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:03.494+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.498881+0000) 2019-10-13T15:52:03.502 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:03.498+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.499844+0000) 2019-10-13T15:52:03.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:03.499+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.503817+0000) 2019-10-13T15:52:03.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:03.506+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:03.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:03.719+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:03.720866+0000) 2019-10-13T15:52:05.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:05.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:05.746581+0000) 2019-10-13T15:52:05.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:05.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:05.746714+0000) 2019-10-13T15:52:05.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:05.754+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:05.757798+0000) 2019-10-13T15:52:05.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:05.763+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:05.767199+0000) 2019-10-13T15:52:06.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:06.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:06.298555+0000) 2019-10-13T15:52:08.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:08.510+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.511674+0000) 2019-10-13T15:52:08.514 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:08.510+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.512396+0000) 2019-10-13T15:52:08.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:08.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.512685+0000) 2019-10-13T15:52:08.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:08.510+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.513259+0000) 2019-10-13T15:52:08.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:08.507+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.514014+0000) 2019-10-13T15:52:08.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:08.510+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.514279+0000) 2019-10-13T15:52:08.517 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:08.510+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.515241+0000) 2019-10-13T15:52:08.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:08.515+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:08.521322+0000) 2019-10-13T15:52:08.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:08.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:12.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:12.734+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:12.736375+0000) 2019-10-13T15:52:13.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:13.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.471866+0000) 2019-10-13T15:52:13.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:13.522+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.526443+0000) 2019-10-13T15:52:13.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:13.526+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.527519+0000) 2019-10-13T15:52:13.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:13.523+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.528147+0000) 2019-10-13T15:52:13.531 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:13.526+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.529313+0000) 2019-10-13T15:52:13.532 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:13.526+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.529888+0000) 2019-10-13T15:52:13.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:13.526+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.530222+0000) 2019-10-13T15:52:13.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:13.526+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.530632+0000) 2019-10-13T15:52:13.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:13.531+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.534749+0000) 2019-10-13T15:52:13.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:13.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230438 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:13.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:13.715+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:13.721066+0000) 2019-10-13T15:52:15.232 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:52:15.232 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:52:15.236 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:52:15.236 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:52:15.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:15.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:15.746782+0000) 2019-10-13T15:52:15.749 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:15.742+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:15.746923+0000) 2019-10-13T15:52:15.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:15.754+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:15.757948+0000) 2019-10-13T15:52:15.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:15.771+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:15.773657+0000) 2019-10-13T15:52:16.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:16.298+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:16.298777+0000) 2019-10-13T15:52:18.545 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:18.538+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.542421+0000) 2019-10-13T15:52:18.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:18.542+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.543524+0000) 2019-10-13T15:52:18.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:18.542+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.544204+0000) 2019-10-13T15:52:18.547 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:18.542+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.543992+0000) 2019-10-13T15:52:18.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:18.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.544670+0000) 2019-10-13T15:52:18.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:18.539+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.545514+0000) 2019-10-13T15:52:18.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:18.542+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.545320+0000) 2019-10-13T15:52:18.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:18.547+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:18.551369+0000) 2019-10-13T15:52:18.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:18.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:22.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:22.734+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:22.736584+0000) 2019-10-13T15:52:23.474 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:23.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.472105+0000) 2019-10-13T15:52:23.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:23.554+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.557480+0000) 2019-10-13T15:52:23.560 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:23.554+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.557876+0000) 2019-10-13T15:52:23.561 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:23.554+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.558349+0000) 2019-10-13T15:52:23.561 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:23.554+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.558546+0000) 2019-10-13T15:52:23.562 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:23.558+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.559746+0000) 2019-10-13T15:52:23.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:23.558+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.559868+0000) 2019-10-13T15:52:23.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:23.558+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.560493+0000) 2019-10-13T15:52:23.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:23.562+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.567221+0000) 2019-10-13T15:52:23.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:23.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:23.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:23.718+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:23.721247+0000) 2019-10-13T15:52:25.749 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:25.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:25.747044+0000) 2019-10-13T15:52:25.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:25.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:25.747130+0000) 2019-10-13T15:52:25.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:25.754+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:25.758136+0000) 2019-10-13T15:52:25.782 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:25.774+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:25.780109+0000) 2019-10-13T15:52:26.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:26.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:26.298951+0000) 2019-10-13T15:52:28.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:28.570+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.573130+0000) 2019-10-13T15:52:28.576 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:28.570+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.573209+0000) 2019-10-13T15:52:28.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:28.570+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.573624+0000) 2019-10-13T15:52:28.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:28.570+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.574036+0000) 2019-10-13T15:52:28.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:28.570+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.575145+0000) 2019-10-13T15:52:28.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:28.574+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.575528+0000) 2019-10-13T15:52:28.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:28.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.576011+0000) 2019-10-13T15:52:28.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:28.578+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:28.583226+0000) 2019-10-13T15:52:28.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:28.578+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:32.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:32.734+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:32.736809+0000) 2019-10-13T15:52:33.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:33.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.472315+0000) 2019-10-13T15:52:33.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:33.582+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.586032+0000) 2019-10-13T15:52:33.590 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:33.582+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.587852+0000) 2019-10-13T15:52:33.590 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:33.586+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.587518+0000) 2019-10-13T15:52:33.591 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:33.586+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.588656+0000) 2019-10-13T15:52:33.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:33.586+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.588326+0000) 2019-10-13T15:52:33.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:33.586+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.589098+0000) 2019-10-13T15:52:33.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:33.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.590388+0000) 2019-10-13T15:52:33.599 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:33.590+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.597103+0000) 2019-10-13T15:52:33.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:33.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230572 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:33.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:33.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:33.721442+0000) 2019-10-13T15:52:35.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:35.742+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:35.747246+0000) 2019-10-13T15:52:35.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:35.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:35.747358+0000) 2019-10-13T15:52:35.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:35.758+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:35.758332+0000) 2019-10-13T15:52:35.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:35.782+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:35.786522+0000) 2019-10-13T15:52:36.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:36.294+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:36.299182+0000) 2019-10-13T15:52:38.603 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:38.598+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.601179+0000) 2019-10-13T15:52:38.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:38.598+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.602275+0000) 2019-10-13T15:52:38.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:38.598+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.602770+0000) 2019-10-13T15:52:38.606 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:38.598+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.603845+0000) 2019-10-13T15:52:38.606 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:38.602+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.603932+0000) 2019-10-13T15:52:38.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:38.602+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.605485+0000) 2019-10-13T15:52:38.608 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:38.602+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.606070+0000) 2019-10-13T15:52:38.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:38.602+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:38.609770+0000) 2019-10-13T15:52:38.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:38.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230608 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:42.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:42.734+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:42.737003+0000) 2019-10-13T15:52:43.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:43.470+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.472510+0000) 2019-10-13T15:52:43.620 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:43.614+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.617345+0000) 2019-10-13T15:52:43.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:43.614+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.618380+0000) 2019-10-13T15:52:43.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:43.618+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.619851+0000) 2019-10-13T15:52:43.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:43.618+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.620485+0000) 2019-10-13T15:52:43.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:43.618+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.620908+0000) 2019-10-13T15:52:43.624 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:43.614+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.621340+0000) 2019-10-13T15:52:43.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:43.618+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.621541+0000) 2019-10-13T15:52:43.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:43.618+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.625732+0000) 2019-10-13T15:52:43.632 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:43.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:43.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:43.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:43.721652+0000) 2019-10-13T15:52:45.343 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:52:45.343 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:52:45.347 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:52:45.347 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:52:45.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:45.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:45.747479+0000) 2019-10-13T15:52:45.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:45.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:45.747558+0000) 2019-10-13T15:52:45.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:45.757+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:45.758565+0000) 2019-10-13T15:52:45.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:45.790+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:45.792877+0000) 2019-10-13T15:52:46.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:46.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:46.299390+0000) 2019-10-13T15:52:48.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:48.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.633746+0000) 2019-10-13T15:52:48.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:48.630+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.634931+0000) 2019-10-13T15:52:48.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:48.629+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.634650+0000) 2019-10-13T15:52:48.638 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:48.633+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.635338+0000) 2019-10-13T15:52:48.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:48.633+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.635576+0000) 2019-10-13T15:52:48.638 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:48.633+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.636051+0000) 2019-10-13T15:52:48.639 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:48.633+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.636462+0000) 2019-10-13T15:52:48.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:48.634+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:48.641481+0000) 2019-10-13T15:52:48.648 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:48.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230674 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:52.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:52.733+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:52.737188+0000) 2019-10-13T15:52:53.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:53.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.472708+0000) 2019-10-13T15:52:53.652 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:53.645+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.649742+0000) 2019-10-13T15:52:53.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:53.645+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.650309+0000) 2019-10-13T15:52:53.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:53.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.650988+0000) 2019-10-13T15:52:53.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:53.649+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.651738+0000) 2019-10-13T15:52:53.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:53.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.652022+0000) 2019-10-13T15:52:53.655 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:53.649+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.652471+0000) 2019-10-13T15:52:53.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:53.649+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.653331+0000) 2019-10-13T15:52:53.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:53.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.658171+0000) 2019-10-13T15:52:53.663 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:53.657+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230706 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:52:53.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:53.714+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:53.721836+0000) 2019-10-13T15:52:55.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:55.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:55.747679+0000) 2019-10-13T15:52:55.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:55.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:55.747716+0000) 2019-10-13T15:52:55.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:55.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:55.758753+0000) 2019-10-13T15:52:55.801 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:55.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:55.799534+0000) 2019-10-13T15:52:56.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:56.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:56.299613+0000) 2019-10-13T15:52:58.667 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:52:58.661+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.664484+0000) 2019-10-13T15:52:58.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:52:58.661+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.665339+0000) 2019-10-13T15:52:58.668 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:52:58.661+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.665881+0000) 2019-10-13T15:52:58.668 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:52:58.661+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.665970+0000) 2019-10-13T15:52:58.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:52:58.661+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.666368+0000) 2019-10-13T15:52:58.669 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:52:58.661+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.666774+0000) 2019-10-13T15:52:58.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:52:58.662+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.667266+0000) 2019-10-13T15:52:58.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:52:58.670+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:52:58.674535+0000) 2019-10-13T15:52:58.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:52:58.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:02.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:02.733+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:02.737353+0000) 2019-10-13T15:53:03.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:03.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.472944+0000) 2019-10-13T15:53:03.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:03.673+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.678826+0000) 2019-10-13T15:53:03.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:03.677+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.679286+0000) 2019-10-13T15:53:03.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:03.677+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.680161+0000) 2019-10-13T15:53:03.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:03.677+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.680282+0000) 2019-10-13T15:53:03.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:03.674+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.680822+0000) 2019-10-13T15:53:03.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:03.677+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.681293+0000) 2019-10-13T15:53:03.684 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:03.677+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.681939+0000) 2019-10-13T15:53:03.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:03.682+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.686263+0000) 2019-10-13T15:53:03.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:03.685+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:03.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:03.718+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:03.722000+0000) 2019-10-13T15:53:05.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:05.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:05.747924+0000) 2019-10-13T15:53:05.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:05.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:05.747919+0000) 2019-10-13T15:53:05.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:05.757+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:05.758979+0000) 2019-10-13T15:53:05.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:05.802+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:05.806036+0000) 2019-10-13T15:53:06.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:06.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:06.299845+0000) 2019-10-13T15:53:08.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:08.689+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.693793+0000) 2019-10-13T15:53:08.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:08.689+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.694481+0000) 2019-10-13T15:53:08.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:08.689+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.695011+0000) 2019-10-13T15:53:08.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:08.693+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.695641+0000) 2019-10-13T15:53:08.698 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:08.694+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.696458+0000) 2019-10-13T15:53:08.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:08.693+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.696621+0000) 2019-10-13T15:53:08.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:08.693+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.696883+0000) 2019-10-13T15:53:08.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:08.702+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:08.702670+0000) 2019-10-13T15:53:08.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:08.701+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230808 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:12.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:12.733+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:12.737569+0000) 2019-10-13T15:53:13.475 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:13.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.473144+0000) 2019-10-13T15:53:13.711 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:13.705+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.708974+0000) 2019-10-13T15:53:13.712 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:13.705+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.709899+0000) 2019-10-13T15:53:13.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:13.710+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.710326+0000) 2019-10-13T15:53:13.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:13.705+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.710796+0000) 2019-10-13T15:53:13.713 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:13.709+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.711344+0000) 2019-10-13T15:53:13.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:13.709+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.711550+0000) 2019-10-13T15:53:13.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:13.709+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.711817+0000) 2019-10-13T15:53:13.721 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:13.718+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.719132+0000) 2019-10-13T15:53:13.723 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:13.717+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230840 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:13.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:13.722+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:13.722178+0000) 2019-10-13T15:53:15.404 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:53:15.404 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:53:15.408 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:53:15.408 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:53:15.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:15.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:15.748151+0000) 2019-10-13T15:53:15.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:15.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:15.748143+0000) 2019-10-13T15:53:15.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:15.753+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:15.759209+0000) 2019-10-13T15:53:15.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:15.809+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:15.812382+0000) 2019-10-13T15:53:16.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:16.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:16.300054+0000) 2019-10-13T15:53:18.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:18.721+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.724935+0000) 2019-10-13T15:53:18.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:18.721+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.725046+0000) 2019-10-13T15:53:18.728 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:18.721+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.725740+0000) 2019-10-13T15:53:18.729 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:18.721+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.726744+0000) 2019-10-13T15:53:18.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:18.725+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.727256+0000) 2019-10-13T15:53:18.730 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:18.725+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.727620+0000) 2019-10-13T15:53:18.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:18.725+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.728101+0000) 2019-10-13T15:53:18.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:18.729+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:18.732584+0000) 2019-10-13T15:53:18.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:18.733+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230876 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:22.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:22.733+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:22.737768+0000) 2019-10-13T15:53:23.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:23.469+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.473373+0000) 2019-10-13T15:53:23.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:23.721+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.722339+0000) 2019-10-13T15:53:23.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:23.737+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.739785+0000) 2019-10-13T15:53:23.742 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:23.737+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.740097+0000) 2019-10-13T15:53:23.743 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:23.737+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.740895+0000) 2019-10-13T15:53:23.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:23.737+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.741502+0000) 2019-10-13T15:53:23.744 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:23.737+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.742215+0000) 2019-10-13T15:53:23.744 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:23.737+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.741762+0000) 2019-10-13T15:53:23.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:23.737+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.743238+0000) 2019-10-13T15:53:23.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:23.745+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:23.750063+0000) 2019-10-13T15:53:23.754 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:23.749+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:25.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:25.745+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:25.748335+0000) 2019-10-13T15:53:25.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:25.745+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:25.748354+0000) 2019-10-13T15:53:25.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:25.757+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:25.759440+0000) 2019-10-13T15:53:25.821 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:25.817+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:25.818884+0000) 2019-10-13T15:53:26.302 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:26.297+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:26.300293+0000) 2019-10-13T15:53:28.757 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:28.749+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.754604+0000) 2019-10-13T15:53:28.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:28.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.755500+0000) 2019-10-13T15:53:28.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:28.753+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.756680+0000) 2019-10-13T15:53:28.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:28.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.757148+0000) 2019-10-13T15:53:28.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:28.753+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.757500+0000) 2019-10-13T15:53:28.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:28.753+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.758746+0000) 2019-10-13T15:53:28.761 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:28.753+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.758281+0000) 2019-10-13T15:53:28.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:28.761+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:28.763323+0000) 2019-10-13T15:53:28.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:28.761+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:32.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:32.732+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:32.737967+0000) 2019-10-13T15:53:33.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:33.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.473590+0000) 2019-10-13T15:53:33.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:33.721+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.722500+0000) 2019-10-13T15:53:33.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:33.764+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.769164+0000) 2019-10-13T15:53:33.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:33.764+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.769745+0000) 2019-10-13T15:53:33.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:33.764+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.770028+0000) 2019-10-13T15:53:33.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:33.764+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.771112+0000) 2019-10-13T15:53:33.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:33.768+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.771440+0000) 2019-10-13T15:53:33.774 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:33.768+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.772080+0000) 2019-10-13T15:53:33.775 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:33.769+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.773090+0000) 2019-10-13T15:53:33.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:33.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 230975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:33.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:33.781+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:33.782327+0000) 2019-10-13T15:53:35.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:35.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:35.748501+0000) 2019-10-13T15:53:35.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:35.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:35.748581+0000) 2019-10-13T15:53:35.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:35.756+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:35.759630+0000) 2019-10-13T15:53:35.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:35.821+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:35.825335+0000) 2019-10-13T15:53:36.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:36.296+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:36.300471+0000) 2019-10-13T15:53:38.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:38.780+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.783921+0000) 2019-10-13T15:53:38.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:38.780+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.784816+0000) 2019-10-13T15:53:38.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:38.780+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.786021+0000) 2019-10-13T15:53:38.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:38.785+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.786443+0000) 2019-10-13T15:53:38.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:38.780+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.787208+0000) 2019-10-13T15:53:38.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:38.784+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.787421+0000) 2019-10-13T15:53:38.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:38.784+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.787827+0000) 2019-10-13T15:53:38.794 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:38.789+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:38.791981+0000) 2019-10-13T15:53:38.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:38.792+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:42.740 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:42.732+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:42.738167+0000) 2019-10-13T15:53:43.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:43.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.473789+0000) 2019-10-13T15:53:43.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:43.721+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.722693+0000) 2019-10-13T15:53:43.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:43.792+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.799039+0000) 2019-10-13T15:53:43.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:43.796+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.799511+0000) 2019-10-13T15:53:43.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:43.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.800327+0000) 2019-10-13T15:53:43.803 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:43.796+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.801268+0000) 2019-10-13T15:53:43.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:43.801+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.802355+0000) 2019-10-13T15:53:43.805 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:43.796+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.802882+0000) 2019-10-13T15:53:43.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:43.800+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.803379+0000) 2019-10-13T15:53:43.809 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:43.805+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:43.807458+0000) 2019-10-13T15:53:43.814 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:43.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:45.531 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:53:45.531 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:53:45.535 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:53:45.535 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:53:45.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:45.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:45.748699+0000) 2019-10-13T15:53:45.751 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:45.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:45.748798+0000) 2019-10-13T15:53:45.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:45.756+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:45.759854+0000) 2019-10-13T15:53:45.833 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:45.829+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:45.831152+0000) 2019-10-13T15:53:46.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:46.300+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:46.300691+0000) 2019-10-13T15:53:48.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:48.812+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.815883+0000) 2019-10-13T15:53:48.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:48.813+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.816271+0000) 2019-10-13T15:53:48.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:48.812+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.817321+0000) 2019-10-13T15:53:48.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:48.812+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.817371+0000) 2019-10-13T15:53:48.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:48.812+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.817889+0000) 2019-10-13T15:53:48.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:48.812+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.818255+0000) 2019-10-13T15:53:48.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:48.812+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.818816+0000) 2019-10-13T15:53:48.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:48.825+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:48.826306+0000) 2019-10-13T15:53:48.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:48.820+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:52.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:52.732+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:52.738367+0000) 2019-10-13T15:53:53.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:53.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.474011+0000) 2019-10-13T15:53:53.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:53.721+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.722903+0000) 2019-10-13T15:53:53.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:53.824+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.830820+0000) 2019-10-13T15:53:53.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:53.828+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.831395+0000) 2019-10-13T15:53:53.834 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:53.828+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.831615+0000) 2019-10-13T15:53:53.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:53.828+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.833173+0000) 2019-10-13T15:53:53.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:53.829+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.833148+0000) 2019-10-13T15:53:53.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:53.828+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.833515+0000) 2019-10-13T15:53:53.837 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:53.828+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.834732+0000) 2019-10-13T15:53:53.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:53.837+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:53.840440+0000) 2019-10-13T15:53:53.844 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:53.836+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231110 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:53:55.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:55.744+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:55.748918+0000) 2019-10-13T15:53:55.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:55.744+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:55.749018+0000) 2019-10-13T15:53:55.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:55.756+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:55.760052+0000) 2019-10-13T15:53:55.840 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:55.837+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:55.837705+0000) 2019-10-13T15:53:56.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:56.300+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:56.300898+0000) 2019-10-13T15:53:58.847 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:53:58.840+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.844979+0000) 2019-10-13T15:53:58.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:53:58.840+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.846462+0000) 2019-10-13T15:53:58.849 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:53:58.840+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.846682+0000) 2019-10-13T15:53:58.849 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:53:58.840+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.847035+0000) 2019-10-13T15:53:58.850 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:53:58.844+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.847707+0000) 2019-10-13T15:53:58.850 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:53:58.845+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.848004+0000) 2019-10-13T15:53:58.851 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:53:58.844+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.848349+0000) 2019-10-13T15:53:58.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:53:58.853+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:53:58.855705+0000) 2019-10-13T15:53:58.858 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:53:58.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:02.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:02.732+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:02.738565+0000) 2019-10-13T15:54:03.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:03.468+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.474207+0000) 2019-10-13T15:54:03.725 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:03.721+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.723098+0000) 2019-10-13T15:54:03.862 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:03.857+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.859554+0000) 2019-10-13T15:54:03.862 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:03.856+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.859942+0000) 2019-10-13T15:54:03.863 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:03.856+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.860215+0000) 2019-10-13T15:54:03.863 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:03.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.860832+0000) 2019-10-13T15:54:03.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:03.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.861747+0000) 2019-10-13T15:54:03.864 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:03.856+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.862450+0000) 2019-10-13T15:54:03.865 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:03.856+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.862902+0000) 2019-10-13T15:54:03.870 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:03.865+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:03.867712+0000) 2019-10-13T15:54:03.873 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:03.864+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231177 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:05.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:05.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:05.749121+0000) 2019-10-13T15:54:05.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:05.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:05.749190+0000) 2019-10-13T15:54:05.762 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:05.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:05.760245+0000) 2019-10-13T15:54:05.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:05.841+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:05.844329+0000) 2019-10-13T15:54:06.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:06.300+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:06.301127+0000) 2019-10-13T15:54:08.878 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:08.872+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.875723+0000) 2019-10-13T15:54:08.879 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:08.872+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.876017+0000) 2019-10-13T15:54:08.879 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:08.873+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.876785+0000) 2019-10-13T15:54:08.880 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:08.872+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.877117+0000) 2019-10-13T15:54:08.880 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:08.872+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.877426+0000) 2019-10-13T15:54:08.880 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:08.872+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.878173+0000) 2019-10-13T15:54:08.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:08.872+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.878543+0000) 2019-10-13T15:54:08.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:08.877+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:08.881850+0000) 2019-10-13T15:54:08.888 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:08.880+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231211 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:12.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:12.736+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:12.738769+0000) 2019-10-13T15:54:13.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:13.467+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.474405+0000) 2019-10-13T15:54:13.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:13.720+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.723325+0000) 2019-10-13T15:54:13.892 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:13.883+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.890119+0000) 2019-10-13T15:54:13.893 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:13.883+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.890667+0000) 2019-10-13T15:54:13.893 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:13.883+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.890937+0000) 2019-10-13T15:54:13.894 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:13.887+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.891908+0000) 2019-10-13T15:54:13.894 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:13.887+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.892479+0000) 2019-10-13T15:54:13.895 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:13.888+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.892601+0000) 2019-10-13T15:54:13.895 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:13.887+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.893532+0000) 2019-10-13T15:54:13.901 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:13.896+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:13.899078+0000) 2019-10-13T15:54:13.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:13.895+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:15.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:15.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:15.749356+0000) 2019-10-13T15:54:15.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:15.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:15.749422+0000) 2019-10-13T15:54:15.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:15.755+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:15.760445+0000) 2019-10-13T15:54:15.839 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:54:15.839 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:54:15.842 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:54:15.843 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:54:15.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:15.852+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:15.855270+0000) 2019-10-13T15:54:16.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:16.295+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:16.301298+0000) 2019-10-13T15:54:18.906 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:18.899+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.903745+0000) 2019-10-13T15:54:18.908 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:18.899+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.905398+0000) 2019-10-13T15:54:18.908 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:18.899+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.905679+0000) 2019-10-13T15:54:18.909 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:18.899+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.906645+0000) 2019-10-13T15:54:18.909 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:18.904+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.907120+0000) 2019-10-13T15:54:18.910 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:18.903+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.907401+0000) 2019-10-13T15:54:18.910 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:18.903+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.907874+0000) 2019-10-13T15:54:18.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:18.908+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:18.911199+0000) 2019-10-13T15:54:18.918 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:18.911+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:22.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:22.731+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:22.738973+0000) 2019-10-13T15:54:23.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:23.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.474660+0000) 2019-10-13T15:54:23.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:23.720+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.723553+0000) 2019-10-13T15:54:23.921 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:23.915+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.919123+0000) 2019-10-13T15:54:23.922 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:23.915+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.920487+0000) 2019-10-13T15:54:23.923 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:23.916+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.920880+0000) 2019-10-13T15:54:23.923 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:23.915+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.921355+0000) 2019-10-13T15:54:23.924 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:23.915+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.921406+0000) 2019-10-13T15:54:23.924 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:23.915+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.921836+0000) 2019-10-13T15:54:23.925 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:23.915+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.922347+0000) 2019-10-13T15:54:23.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:23.924+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:23.926496+0000) 2019-10-13T15:54:23.933 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:23.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231312 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:25.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:25.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:25.749568+0000) 2019-10-13T15:54:25.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:25.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:25.749636+0000) 2019-10-13T15:54:25.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:25.759+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:25.760671+0000) 2019-10-13T15:54:25.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:25.856+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:25.857740+0000) 2019-10-13T15:54:26.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:26.299+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:26.301535+0000) 2019-10-13T15:54:28.936 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:28.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.933603+0000) 2019-10-13T15:54:28.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:28.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.934221+0000) 2019-10-13T15:54:28.937 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:28.927+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.934881+0000) 2019-10-13T15:54:28.937 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:28.927+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.934554+0000) 2019-10-13T15:54:28.938 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:28.931+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.935643+0000) 2019-10-13T15:54:28.938 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:28.931+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.936302+0000) 2019-10-13T15:54:28.940 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:28.931+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.937737+0000) 2019-10-13T15:54:28.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:28.940+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:28.943140+0000) 2019-10-13T15:54:28.947 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:28.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231347 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:32.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:32.735+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:32.739202+0000) 2019-10-13T15:54:33.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:33.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.474886+0000) 2019-10-13T15:54:33.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:33.720+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.723794+0000) 2019-10-13T15:54:33.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:33.943+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.948186+0000) 2019-10-13T15:54:33.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:33.943+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.949439+0000) 2019-10-13T15:54:33.952 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:33.943+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.949998+0000) 2019-10-13T15:54:33.952 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:33.943+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.950423+0000) 2019-10-13T15:54:33.953 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:33.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.950690+0000) 2019-10-13T15:54:33.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:33.948+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.951410+0000) 2019-10-13T15:54:33.955 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:33.947+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.952353+0000) 2019-10-13T15:54:33.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:33.952+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:33.957434+0000) 2019-10-13T15:54:33.963 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:33.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:35.752 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:35.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:35.749790+0000) 2019-10-13T15:54:35.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:35.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:35.749868+0000) 2019-10-13T15:54:35.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:35.759+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:35.760866+0000) 2019-10-13T15:54:35.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:35.860+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:35.864150+0000) 2019-10-13T15:54:36.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:36.299+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:36.301765+0000) 2019-10-13T15:54:38.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:38.959+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.965053+0000) 2019-10-13T15:54:38.968 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:38.959+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.964512+0000) 2019-10-13T15:54:38.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:38.959+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.966123+0000) 2019-10-13T15:54:38.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:38.959+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.966482+0000) 2019-10-13T15:54:38.969 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:38.963+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.967295+0000) 2019-10-13T15:54:38.970 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:38.963+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.967750+0000) 2019-10-13T15:54:38.971 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:38.964+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.968988+0000) 2019-10-13T15:54:38.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:38.968+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:38.971842+0000) 2019-10-13T15:54:38.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:38.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:42.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:42.735+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:42.739378+0000) 2019-10-13T15:54:43.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:43.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.475107+0000) 2019-10-13T15:54:43.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:43.720+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.723998+0000) 2019-10-13T15:54:43.982 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:43.979+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.979824+0000) 2019-10-13T15:54:43.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:43.979+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.980207+0000) 2019-10-13T15:54:43.983 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:43.979+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.980557+0000) 2019-10-13T15:54:43.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:43.979+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.981618+0000) 2019-10-13T15:54:43.984 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:43.979+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.982419+0000) 2019-10-13T15:54:43.985 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:43.980+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.982971+0000) 2019-10-13T15:54:43.985 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:43.979+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.982965+0000) 2019-10-13T15:54:43.991 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:43.984+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:43.989576+0000) 2019-10-13T15:54:43.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:43.987+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:45.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:45.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:45.750024+0000) 2019-10-13T15:54:45.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:45.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:45.750070+0000) 2019-10-13T15:54:45.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:45.759+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:45.761054+0000) 2019-10-13T15:54:45.872 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:45.868+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:45.870172+0000) 2019-10-13T15:54:45.895 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:54:45.895 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:54:45.899 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:54:45.899 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:54:46.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:46.299+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:46.301980+0000) 2019-10-13T15:54:48.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:48.991+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.993100+0000) 2019-10-13T15:54:48.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:48.991+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.994202+0000) 2019-10-13T15:54:48.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:48.995+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.996066+0000) 2019-10-13T15:54:48.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:48.992+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.996488+0000) 2019-10-13T15:54:48.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:48.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.996643+0000) 2019-10-13T15:54:48.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:48.995+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.997004+0000) 2019-10-13T15:54:49.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:48.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:48.997405+0000) 2019-10-13T15:54:49.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:48.996+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:49.001515+0000) 2019-10-13T15:54:49.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:49.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:52.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:52.739+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:52.739586+0000) 2019-10-13T15:54:53.477 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:53.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:53.475331+0000) 2019-10-13T15:54:53.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:53.720+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:53.724188+0000) 2019-10-13T15:54:54.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:54.007+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.009228+0000) 2019-10-13T15:54:54.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:54.007+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.009847+0000) 2019-10-13T15:54:54.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:54.007+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.010508+0000) 2019-10-13T15:54:54.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:54.011+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.011346+0000) 2019-10-13T15:54:54.014 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:54.011+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.011579+0000) 2019-10-13T15:54:54.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:54.008+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.012370+0000) 2019-10-13T15:54:54.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:54.011+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.012536+0000) 2019-10-13T15:54:54.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:54.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:54.017522+0000) 2019-10-13T15:54:54.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:54.019+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:54:55.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:55.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:55.750228+0000) 2019-10-13T15:54:55.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:55.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:55.750304+0000) 2019-10-13T15:54:55.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:55.759+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:55.761254+0000) 2019-10-13T15:54:55.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:55.872+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:55.876557+0000) 2019-10-13T15:54:56.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:56.299+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:56.302224+0000) 2019-10-13T15:54:59.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:54:59.022+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.025505+0000) 2019-10-13T15:54:59.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:54:59.024+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.026293+0000) 2019-10-13T15:54:59.029 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:54:59.022+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.025873+0000) 2019-10-13T15:54:59.029 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:54:59.022+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.027105+0000) 2019-10-13T15:54:59.030 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:54:59.026+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.027650+0000) 2019-10-13T15:54:59.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:54:59.026+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.027874+0000) 2019-10-13T15:54:59.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:54:59.026+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.028200+0000) 2019-10-13T15:54:59.036 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:54:59.028+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:54:59.033794+0000) 2019-10-13T15:54:59.039 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:54:59.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231552 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:02.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:02.738+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:02.739802+0000) 2019-10-13T15:55:03.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:03.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:03.475570+0000) 2019-10-13T15:55:03.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:03.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:03.724411+0000) 2019-10-13T15:55:04.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:04.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.039710+0000) 2019-10-13T15:55:04.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:04.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.040695+0000) 2019-10-13T15:55:04.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:04.038+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.041679+0000) 2019-10-13T15:55:04.045 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:04.038+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.042006+0000) 2019-10-13T15:55:04.045 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:04.038+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.042306+0000) 2019-10-13T15:55:04.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:04.038+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.042976+0000) 2019-10-13T15:55:04.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:04.038+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.042586+0000) 2019-10-13T15:55:04.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:04.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:04.047221+0000) 2019-10-13T15:55:04.054 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:04.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:05.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:05.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:05.750429+0000) 2019-10-13T15:55:05.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:05.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:05.750517+0000) 2019-10-13T15:55:05.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:05.758+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:05.761446+0000) 2019-10-13T15:55:05.885 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:05.880+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:05.882960+0000) 2019-10-13T15:55:06.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:06.298+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:06.302422+0000) 2019-10-13T15:55:09.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:09.054+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.055609+0000) 2019-10-13T15:55:09.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:09.054+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.056081+0000) 2019-10-13T15:55:09.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:09.051+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.056947+0000) 2019-10-13T15:55:09.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:09.054+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.057373+0000) 2019-10-13T15:55:09.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:09.054+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.057659+0000) 2019-10-13T15:55:09.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:09.054+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.057877+0000) 2019-10-13T15:55:09.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:09.054+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.058610+0000) 2019-10-13T15:55:09.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:09.059+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:09.064925+0000) 2019-10-13T15:55:09.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:09.062+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231619 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:12.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:12.738+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:12.740000+0000) 2019-10-13T15:55:13.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:13.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:13.475763+0000) 2019-10-13T15:55:13.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:13.719+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:13.724623+0000) 2019-10-13T15:55:14.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:14.066+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.070152+0000) 2019-10-13T15:55:14.073 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:14.066+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.070799+0000) 2019-10-13T15:55:14.073 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:14.066+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.071153+0000) 2019-10-13T15:55:14.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:14.070+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.071403+0000) 2019-10-13T15:55:14.074 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:14.070+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.072290+0000) 2019-10-13T15:55:14.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:14.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.073137+0000) 2019-10-13T15:55:14.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:14.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.074045+0000) 2019-10-13T15:55:14.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:14.075+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:14.079996+0000) 2019-10-13T15:55:14.084 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:14.078+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:15.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:15.746+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:15.750661+0000) 2019-10-13T15:55:15.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:15.746+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:15.750755+0000) 2019-10-13T15:55:15.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:15.758+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:15.761671+0000) 2019-10-13T15:55:15.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:15.887+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:15.889348+0000) 2019-10-13T15:55:15.942 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:55:15.942 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:55:15.946 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:55:15.946 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:55:16.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:16.298+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:16.302621+0000) 2019-10-13T15:55:19.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:19.082+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.085052+0000) 2019-10-13T15:55:19.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:19.082+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.086498+0000) 2019-10-13T15:55:19.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:19.082+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.086784+0000) 2019-10-13T15:55:19.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:19.083+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.087575+0000) 2019-10-13T15:55:19.090 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:19.086+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.087462+0000) 2019-10-13T15:55:19.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:19.086+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.087774+0000) 2019-10-13T15:55:19.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:19.086+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.088239+0000) 2019-10-13T15:55:19.094 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:19.087+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:19.092304+0000) 2019-10-13T15:55:19.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:19.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:22.742 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:22.738+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:22.740215+0000) 2019-10-13T15:55:23.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:23.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:23.475954+0000) 2019-10-13T15:55:23.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:23.723+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:23.724818+0000) 2019-10-13T15:55:24.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:24.098+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.099763+0000) 2019-10-13T15:55:24.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:24.098+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.100992+0000) 2019-10-13T15:55:24.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:24.098+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.101213+0000) 2019-10-13T15:55:24.104 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:24.098+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.102566+0000) 2019-10-13T15:55:24.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:24.098+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.102944+0000) 2019-10-13T15:55:24.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:24.099+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.103809+0000) 2019-10-13T15:55:24.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:24.102+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.104192+0000) 2019-10-13T15:55:24.110 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:24.103+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:24.108096+0000) 2019-10-13T15:55:24.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:24.110+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:25.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:25.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:25.750851+0000) 2019-10-13T15:55:25.753 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:25.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:25.750979+0000) 2019-10-13T15:55:25.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:25.758+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:25.761897+0000) 2019-10-13T15:55:25.898 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:25.891+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:25.895685+0000) 2019-10-13T15:55:26.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:26.298+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:26.302847+0000) 2019-10-13T15:55:29.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:29.110+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.114991+0000) 2019-10-13T15:55:29.118 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:29.114+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.115631+0000) 2019-10-13T15:55:29.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:29.111+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.116756+0000) 2019-10-13T15:55:29.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:29.114+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.116463+0000) 2019-10-13T15:55:29.120 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:29.114+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.117233+0000) 2019-10-13T15:55:29.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:29.114+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.116974+0000) 2019-10-13T15:55:29.120 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:29.114+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.118171+0000) 2019-10-13T15:55:29.126 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:29.119+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:29.124020+0000) 2019-10-13T15:55:29.129 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:29.122+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:32.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:32.738+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:32.740393+0000) 2019-10-13T15:55:33.478 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:33.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:33.476154+0000) 2019-10-13T15:55:33.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:33.719+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:33.725039+0000) 2019-10-13T15:55:34.133 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:34.127+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.130930+0000) 2019-10-13T15:55:34.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:34.126+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.130669+0000) 2019-10-13T15:55:34.134 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:34.130+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.131696+0000) 2019-10-13T15:55:34.134 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:34.130+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.132055+0000) 2019-10-13T15:55:34.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:34.130+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.132586+0000) 2019-10-13T15:55:34.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:34.130+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.132987+0000) 2019-10-13T15:55:34.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:34.130+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.133335+0000) 2019-10-13T15:55:34.140 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:34.131+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:34.137889+0000) 2019-10-13T15:55:34.143 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:34.138+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:35.753 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:35.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:35.751061+0000) 2019-10-13T15:55:35.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:35.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:35.751168+0000) 2019-10-13T15:55:35.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:35.758+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:35.762104+0000) 2019-10-13T15:55:35.904 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:35.899+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:35.902178+0000) 2019-10-13T15:55:36.305 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:36.302+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:36.303085+0000) 2019-10-13T15:55:39.148 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:39.142+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.145182+0000) 2019-10-13T15:55:39.148 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:39.142+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.145406+0000) 2019-10-13T15:55:39.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:39.139+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.145954+0000) 2019-10-13T15:55:39.149 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:39.142+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.146450+0000) 2019-10-13T15:55:39.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:39.142+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.146768+0000) 2019-10-13T15:55:39.150 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:39.146+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.147424+0000) 2019-10-13T15:55:39.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:39.146+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.148380+0000) 2019-10-13T15:55:39.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:39.147+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:39.153581+0000) 2019-10-13T15:55:39.158 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:39.154+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:42.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:42.737+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:42.740622+0000) 2019-10-13T15:55:43.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:43.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:43.476367+0000) 2019-10-13T15:55:43.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:43.723+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:43.725261+0000) 2019-10-13T15:55:44.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:44.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.159761+0000) 2019-10-13T15:55:44.163 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:44.157+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.159560+0000) 2019-10-13T15:55:44.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:44.157+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.160105+0000) 2019-10-13T15:55:44.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:44.157+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.160418+0000) 2019-10-13T15:55:44.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:44.157+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.161625+0000) 2019-10-13T15:55:44.164 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:44.157+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.162102+0000) 2019-10-13T15:55:44.165 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:44.157+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.162750+0000) 2019-10-13T15:55:44.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:44.167+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:44.171167+0000) 2019-10-13T15:55:44.174 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:44.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:45.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:45.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:45.751268+0000) 2019-10-13T15:55:45.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:45.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:45.751377+0000) 2019-10-13T15:55:45.764 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:45.757+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:45.762295+0000) 2019-10-13T15:55:45.911 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:45.907+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:45.908621+0000) 2019-10-13T15:55:46.001 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:55:46.001 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:55:46.005 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:55:46.005 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:55:46.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:46.301+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:46.303334+0000) 2019-10-13T15:55:49.178 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:49.169+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.174444+0000) 2019-10-13T15:55:49.178 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:49.169+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.175128+0000) 2019-10-13T15:55:49.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:49.173+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.176483+0000) 2019-10-13T15:55:49.179 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:49.171+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.176967+0000) 2019-10-13T15:55:49.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:49.173+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.177226+0000) 2019-10-13T15:55:49.180 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:49.173+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.177699+0000) 2019-10-13T15:55:49.181 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:49.173+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.178849+0000) 2019-10-13T15:55:49.187 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:49.179+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:49.185304+0000) 2019-10-13T15:55:49.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:49.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:52.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:52.737+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:52.740821+0000) 2019-10-13T15:55:53.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:53.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:53.476560+0000) 2019-10-13T15:55:53.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:53.719+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:53.725450+0000) 2019-10-13T15:55:54.191 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:54.185+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.189088+0000) 2019-10-13T15:55:54.192 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:54.185+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.189529+0000) 2019-10-13T15:55:54.193 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:54.185+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.190852+0000) 2019-10-13T15:55:54.194 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:54.189+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.191815+0000) 2019-10-13T15:55:54.194 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:54.189+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.192249+0000) 2019-10-13T15:55:54.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:54.189+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.192494+0000) 2019-10-13T15:55:54.195 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:54.187+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.193329+0000) 2019-10-13T15:55:54.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:54.195+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:54.199497+0000) 2019-10-13T15:55:54.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:54.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231928 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:55:55.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:55.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:55.751514+0000) 2019-10-13T15:55:55.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:55.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:55.751614+0000) 2019-10-13T15:55:55.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:55.761+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:55.762500+0000) 2019-10-13T15:55:55.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:55.915+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:55.918666+0000) 2019-10-13T15:55:56.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:56.301+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:56.303567+0000) 2019-10-13T15:55:59.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:55:59.201+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.205420+0000) 2019-10-13T15:55:59.208 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:55:59.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.206025+0000) 2019-10-13T15:55:59.208 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:55:59.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.206249+0000) 2019-10-13T15:55:59.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:55:59.201+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.206619+0000) 2019-10-13T15:55:59.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:55:59.205+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.207545+0000) 2019-10-13T15:55:59.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:55:59.205+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.208185+0000) 2019-10-13T15:55:59.211 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:55:59.203+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.209513+0000) 2019-10-13T15:55:59.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:55:59.211+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:55:59.214747+0000) 2019-10-13T15:55:59.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:55:59.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231964 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:02.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:02.737+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:02.741054+0000) 2019-10-13T15:56:03.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:03.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:03.476749+0000) 2019-10-13T15:56:03.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:03.718+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:03.725676+0000) 2019-10-13T15:56:04.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:04.217+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.221453+0000) 2019-10-13T15:56:04.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:04.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.221729+0000) 2019-10-13T15:56:04.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:04.217+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.222070+0000) 2019-10-13T15:56:04.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:04.217+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.222466+0000) 2019-10-13T15:56:04.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:04.217+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.222845+0000) 2019-10-13T15:56:04.226 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:04.221+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.223474+0000) 2019-10-13T15:56:04.227 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:04.218+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.224761+0000) 2019-10-13T15:56:04.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:04.222+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:04.227256+0000) 2019-10-13T15:56:04.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:04.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 231996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:05.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:05.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:05.751715+0000) 2019-10-13T15:56:05.754 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:05.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:05.751958+0000) 2019-10-13T15:56:05.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:05.761+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:05.762695+0000) 2019-10-13T15:56:05.930 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:05.926+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:05.927645+0000) 2019-10-13T15:56:06.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:06.301+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:06.303782+0000) 2019-10-13T15:56:09.238 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:09.233+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.236059+0000) 2019-10-13T15:56:09.239 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:09.233+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.236713+0000) 2019-10-13T15:56:09.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:09.233+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.237314+0000) 2019-10-13T15:56:09.240 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:09.230+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.237801+0000) 2019-10-13T15:56:09.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:09.233+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.237915+0000) 2019-10-13T15:56:09.240 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:09.233+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.238597+0000) 2019-10-13T15:56:09.241 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:09.237+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.239524+0000) 2019-10-13T15:56:09.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:09.238+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:09.245512+0000) 2019-10-13T15:56:09.250 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:09.245+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:12.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:12.737+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:12.741260+0000) 2019-10-13T15:56:13.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:13.473+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:13.476974+0000) 2019-10-13T15:56:13.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:13.718+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:13.725865+0000) 2019-10-13T15:56:14.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:14.245+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.250367+0000) 2019-10-13T15:56:14.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:14.245+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.251185+0000) 2019-10-13T15:56:14.254 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:14.249+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.251496+0000) 2019-10-13T15:56:14.255 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:14.246+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.252922+0000) 2019-10-13T15:56:14.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:14.249+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.253119+0000) 2019-10-13T15:56:14.256 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:14.249+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.253903+0000) 2019-10-13T15:56:14.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:14.249+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.254298+0000) 2019-10-13T15:56:14.264 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:14.254+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:14.261727+0000) 2019-10-13T15:56:14.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:14.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:15.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:15.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:15.751950+0000) 2019-10-13T15:56:15.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:15.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:15.752157+0000) 2019-10-13T15:56:15.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:15.761+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:15.762928+0000) 2019-10-13T15:56:15.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:15.930+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:15.934071+0000) 2019-10-13T15:56:16.122 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:56:16.122 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:56:16.126 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:56:16.126 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:56:16.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:16.301+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:16.303988+0000) 2019-10-13T15:56:19.269 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:19.261+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.266434+0000) 2019-10-13T15:56:19.269 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:19.261+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.266743+0000) 2019-10-13T15:56:19.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:19.265+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.267678+0000) 2019-10-13T15:56:19.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:19.265+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.268461+0000) 2019-10-13T15:56:19.271 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:19.265+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.268141+0000) 2019-10-13T15:56:19.271 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:19.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.269418+0000) 2019-10-13T15:56:19.272 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:19.265+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.269675+0000) 2019-10-13T15:56:19.275 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:19.266+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:19.273660+0000) 2019-10-13T15:56:19.280 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:19.273+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:22.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:22.737+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:22.741483+0000) 2019-10-13T15:56:23.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:23.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:23.477166+0000) 2019-10-13T15:56:23.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:23.722+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:23.726086+0000) 2019-10-13T15:56:24.284 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:24.277+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.281485+0000) 2019-10-13T15:56:24.284 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:24.277+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.281972+0000) 2019-10-13T15:56:24.285 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:24.277+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.282191+0000) 2019-10-13T15:56:24.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:24.277+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.282853+0000) 2019-10-13T15:56:24.286 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:24.281+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.283647+0000) 2019-10-13T15:56:24.286 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:24.278+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.284328+0000) 2019-10-13T15:56:24.287 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:24.281+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.284714+0000) 2019-10-13T15:56:24.295 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:24.286+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:24.293392+0000) 2019-10-13T15:56:24.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:24.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:25.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:25.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:25.752122+0000) 2019-10-13T15:56:25.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:25.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:25.752367+0000) 2019-10-13T15:56:25.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:25.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:25.763115+0000) 2019-10-13T15:56:25.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:25.938+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:25.940405+0000) 2019-10-13T15:56:26.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:26.300+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:26.304155+0000) 2019-10-13T15:56:29.300 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:29.292+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.297631+0000) 2019-10-13T15:56:29.300 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:29.292+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.298178+0000) 2019-10-13T15:56:29.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:29.292+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.298976+0000) 2019-10-13T15:56:29.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:29.296+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.299427+0000) 2019-10-13T15:56:29.302 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:29.296+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.299787+0000) 2019-10-13T15:56:29.302 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:29.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.300390+0000) 2019-10-13T15:56:29.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:29.296+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.300796+0000) 2019-10-13T15:56:29.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:29.298+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:29.304868+0000) 2019-10-13T15:56:29.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:29.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232171 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:32.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:32.740+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:32.741686+0000) 2019-10-13T15:56:33.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:33.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:33.477362+0000) 2019-10-13T15:56:33.728 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:33.722+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:33.726261+0000) 2019-10-13T15:56:34.314 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:34.306+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.312432+0000) 2019-10-13T15:56:34.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:34.308+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.313430+0000) 2019-10-13T15:56:34.316 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:34.308+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.313635+0000) 2019-10-13T15:56:34.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:34.308+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.314216+0000) 2019-10-13T15:56:34.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:34.308+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.314579+0000) 2019-10-13T15:56:34.317 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:34.312+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.315500+0000) 2019-10-13T15:56:34.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:34.312+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.316386+0000) 2019-10-13T15:56:34.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:34.318+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:34.322581+0000) 2019-10-13T15:56:34.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:34.320+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:35.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:35.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:35.752352+0000) 2019-10-13T15:56:35.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:35.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:35.752607+0000) 2019-10-13T15:56:35.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:35.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:35.763340+0000) 2019-10-13T15:56:35.951 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:35.946+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:35.948645+0000) 2019-10-13T15:56:36.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:36.300+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:36.304401+0000) 2019-10-13T15:56:39.331 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:39.324+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.328882+0000) 2019-10-13T15:56:39.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:39.324+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.330372+0000) 2019-10-13T15:56:39.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:39.324+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.330622+0000) 2019-10-13T15:56:39.333 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:39.326+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.331541+0000) 2019-10-13T15:56:39.334 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:39.328+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.331646+0000) 2019-10-13T15:56:39.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:39.328+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.332018+0000) 2019-10-13T15:56:39.335 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:39.328+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.332393+0000) 2019-10-13T15:56:39.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:39.330+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:39.337167+0000) 2019-10-13T15:56:39.342 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:39.336+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232239 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:42.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:42.736+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:42.741875+0000) 2019-10-13T15:56:43.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:43.472+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:43.477589+0000) 2019-10-13T15:56:43.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:43.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:43.726453+0000) 2019-10-13T15:56:44.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:44.338+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.343900+0000) 2019-10-13T15:56:44.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:44.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.344168+0000) 2019-10-13T15:56:44.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:44.340+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.344698+0000) 2019-10-13T15:56:44.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:44.340+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.345442+0000) 2019-10-13T15:56:44.348 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:44.340+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.345508+0000) 2019-10-13T15:56:44.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:44.340+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.346283+0000) 2019-10-13T15:56:44.350 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:44.344+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.347611+0000) 2019-10-13T15:56:44.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:44.350+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:44.354589+0000) 2019-10-13T15:56:44.358 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:44.352+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232272 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:45.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:45.748+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:45.752546+0000) 2019-10-13T15:56:45.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:45.748+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:45.752796+0000) 2019-10-13T15:56:45.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:45.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:45.763550+0000) 2019-10-13T15:56:45.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:45.954+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:45.955168+0000) 2019-10-13T15:56:46.172 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:56:46.172 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:56:46.175 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:56:46.176 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:56:46.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:46.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:46.304628+0000) 2019-10-13T15:56:49.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:49.356+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.360372+0000) 2019-10-13T15:56:49.363 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:49.356+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.360537+0000) 2019-10-13T15:56:49.364 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:49.358+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.361067+0000) 2019-10-13T15:56:49.364 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:49.356+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.361356+0000) 2019-10-13T15:56:49.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:49.356+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.361951+0000) 2019-10-13T15:56:49.365 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:49.356+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.362351+0000) 2019-10-13T15:56:49.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:49.356+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.363281+0000) 2019-10-13T15:56:49.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:49.370+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:49.371516+0000) 2019-10-13T15:56:49.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:49.368+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:52.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:52.736+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:52.742080+0000) 2019-10-13T15:56:53.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:53.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:53.477795+0000) 2019-10-13T15:56:53.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:53.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:53.726650+0000) 2019-10-13T15:56:54.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:54.368+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.374571+0000) 2019-10-13T15:56:54.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:54.372+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.375346+0000) 2019-10-13T15:56:54.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:54.372+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.375861+0000) 2019-10-13T15:56:54.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:54.372+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.376666+0000) 2019-10-13T15:56:54.379 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:54.372+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.377352+0000) 2019-10-13T15:56:54.380 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:54.374+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.378037+0000) 2019-10-13T15:56:54.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:54.372+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.378491+0000) 2019-10-13T15:56:54.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:54.382+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:54.382304+0000) 2019-10-13T15:56:54.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:54.380+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:56:55.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:55.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:55.752745+0000) 2019-10-13T15:56:55.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:55.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:55.752991+0000) 2019-10-13T15:56:55.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:55.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:55.763743+0000) 2019-10-13T15:56:55.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:55.958+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:55.961711+0000) 2019-10-13T15:56:56.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:56.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:56.304845+0000) 2019-10-13T15:56:59.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:56:59.384+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.389870+0000) 2019-10-13T15:56:59.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:56:59.384+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.390113+0000) 2019-10-13T15:56:59.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:56:59.384+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.390587+0000) 2019-10-13T15:56:59.393 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:56:59.389+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.391204+0000) 2019-10-13T15:56:59.394 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:56:59.388+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.391485+0000) 2019-10-13T15:56:59.394 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:56:59.388+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.392619+0000) 2019-10-13T15:56:59.395 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:56:59.388+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.393300+0000) 2019-10-13T15:56:59.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:56:59.397+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:56:59.401214+0000) 2019-10-13T15:56:59.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:56:59.396+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:02.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:02.736+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:02.742283+0000) 2019-10-13T15:57:03.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:03.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:03.477944+0000) 2019-10-13T15:57:03.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:03.725+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:03.726849+0000) 2019-10-13T15:57:04.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:04.400+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.404522+0000) 2019-10-13T15:57:04.407 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:04.400+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.405195+0000) 2019-10-13T15:57:04.408 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:04.400+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.405717+0000) 2019-10-13T15:57:04.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:04.400+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.406265+0000) 2019-10-13T15:57:04.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:04.405+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.408638+0000) 2019-10-13T15:57:04.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:04.404+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.407791+0000) 2019-10-13T15:57:04.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:04.404+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.409321+0000) 2019-10-13T15:57:04.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:04.409+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:04.413860+0000) 2019-10-13T15:57:04.419 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:04.412+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:05.755 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:05.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:05.752983+0000) 2019-10-13T15:57:05.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:05.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:05.753160+0000) 2019-10-13T15:57:05.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:05.760+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:05.763943+0000) 2019-10-13T15:57:05.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:05.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:05.973131+0000) 2019-10-13T15:57:06.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:06.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:06.305048+0000) 2019-10-13T15:57:09.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:09.417+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.421672+0000) 2019-10-13T15:57:09.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:09.415+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.421932+0000) 2019-10-13T15:57:09.425 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:09.415+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.422782+0000) 2019-10-13T15:57:09.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:09.415+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.423131+0000) 2019-10-13T15:57:09.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:09.419+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.423544+0000) 2019-10-13T15:57:09.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:09.419+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.423951+0000) 2019-10-13T15:57:09.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:09.419+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.424188+0000) 2019-10-13T15:57:09.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:09.429+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:09.431280+0000) 2019-10-13T15:57:09.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:09.427+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:12.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:12.735+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:12.742514+0000) 2019-10-13T15:57:13.480 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:13.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:13.478163+0000) 2019-10-13T15:57:13.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:13.725+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:13.727080+0000) 2019-10-13T15:57:14.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:14.431+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.436176+0000) 2019-10-13T15:57:14.439 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:14.431+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.436425+0000) 2019-10-13T15:57:14.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:14.431+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.436854+0000) 2019-10-13T15:57:14.440 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:14.431+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.437427+0000) 2019-10-13T15:57:14.441 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:14.437+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.438410+0000) 2019-10-13T15:57:14.441 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:14.431+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.438000+0000) 2019-10-13T15:57:14.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:14.431+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.438722+0000) 2019-10-13T15:57:14.446 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:14.441+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:14.444420+0000) 2019-10-13T15:57:14.449 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:14.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:15.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:15.747+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:15.753184+0000) 2019-10-13T15:57:15.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:15.747+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:15.753352+0000) 2019-10-13T15:57:15.766 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:15.759+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:15.764151+0000) 2019-10-13T15:57:15.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:15.977+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:15.979801+0000) 2019-10-13T15:57:16.271 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:57:16.271 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:57:16.275 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:57:16.275 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:57:16.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:16.299+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:16.305259+0000) 2019-10-13T15:57:19.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:19.443+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.450029+0000) 2019-10-13T15:57:19.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:19.447+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.451472+0000) 2019-10-13T15:57:19.454 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:19.447+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.452216+0000) 2019-10-13T15:57:19.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:19.447+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.452356+0000) 2019-10-13T15:57:19.455 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:19.447+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.452929+0000) 2019-10-13T15:57:19.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:19.449+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.453338+0000) 2019-10-13T15:57:19.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:19.447+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.453836+0000) 2019-10-13T15:57:19.464 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:19.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232515 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:19.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:19.461+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:19.463226+0000) 2019-10-13T15:57:22.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:22.735+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:22.742689+0000) 2019-10-13T15:57:23.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:23.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:23.478356+0000) 2019-10-13T15:57:23.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:23.725+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:23.727309+0000) 2019-10-13T15:57:24.468 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:24.461+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.465805+0000) 2019-10-13T15:57:24.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:24.459+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.466164+0000) 2019-10-13T15:57:24.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:24.459+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.466755+0000) 2019-10-13T15:57:24.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:24.463+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.467415+0000) 2019-10-13T15:57:24.470 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:24.459+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.467067+0000) 2019-10-13T15:57:24.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:24.463+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.467706+0000) 2019-10-13T15:57:24.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:24.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.468312+0000) 2019-10-13T15:57:24.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:24.473+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:24.475706+0000) 2019-10-13T15:57:24.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:24.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232548 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:25.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:25.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:25.753394+0000) 2019-10-13T15:57:25.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:25.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:25.753540+0000) 2019-10-13T15:57:25.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:25.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:25.764376+0000) 2019-10-13T15:57:25.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:25.985+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:25.986293+0000) 2019-10-13T15:57:26.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:26.303+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:26.305474+0000) 2019-10-13T15:57:29.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:29.481+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.482635+0000) 2019-10-13T15:57:29.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:29.475+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.479479+0000) 2019-10-13T15:57:29.487 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:29.475+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.480533+0000) 2019-10-13T15:57:29.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:29.475+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.480960+0000) 2019-10-13T15:57:29.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:29.475+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.481731+0000) 2019-10-13T15:57:29.488 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:29.475+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.482120+0000) 2019-10-13T15:57:29.488 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:29.475+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.482864+0000) 2019-10-13T15:57:29.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:29.485+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:29.489544+0000) 2019-10-13T15:57:29.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:29.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:32.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:32.739+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:32.742904+0000) 2019-10-13T15:57:33.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:33.471+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:33.478553+0000) 2019-10-13T15:57:33.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:33.725+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:33.727498+0000) 2019-10-13T15:57:34.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:34.493+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.496188+0000) 2019-10-13T15:57:34.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:34.491+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.496563+0000) 2019-10-13T15:57:34.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:34.491+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.497460+0000) 2019-10-13T15:57:34.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:34.491+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.497737+0000) 2019-10-13T15:57:34.501 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:34.491+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.498777+0000) 2019-10-13T15:57:34.501 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:34.495+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.499506+0000) 2019-10-13T15:57:34.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:34.495+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.499950+0000) 2019-10-13T15:57:34.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:34.501+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:34.503562+0000) 2019-10-13T15:57:34.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:34.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:35.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:35.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:35.753604+0000) 2019-10-13T15:57:35.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:35.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:35.753723+0000) 2019-10-13T15:57:35.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:35.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:35.764571+0000) 2019-10-13T15:57:35.995 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:35.989+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:35.992498+0000) 2019-10-13T15:57:36.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:36.303+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:36.305679+0000) 2019-10-13T15:57:39.522 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:39.507+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.511230+0000) 2019-10-13T15:57:39.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:39.511+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.511753+0000) 2019-10-13T15:57:39.523 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:39.511+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.512160+0000) 2019-10-13T15:57:39.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:39.509+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.513352+0000) 2019-10-13T15:57:39.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:39.511+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.513666+0000) 2019-10-13T15:57:39.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:39.511+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.514154+0000) 2019-10-13T15:57:39.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:39.511+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.515135+0000) 2019-10-13T15:57:39.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:39.517+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:39.521745+0000) 2019-10-13T15:57:39.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:39.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232652 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:42.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:42.739+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:42.743110+0000) 2019-10-13T15:57:43.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:43.475+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:43.478772+0000) 2019-10-13T15:57:43.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:43.725+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:43.727725+0000) 2019-10-13T15:57:44.530 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:44.527+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.527513+0000) 2019-10-13T15:57:44.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:44.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.528231+0000) 2019-10-13T15:57:44.531 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:44.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.528931+0000) 2019-10-13T15:57:44.531 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:44.527+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.529596+0000) 2019-10-13T15:57:44.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:44.529+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.530499+0000) 2019-10-13T15:57:44.533 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:44.527+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.529844+0000) 2019-10-13T15:57:44.534 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:44.531+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.531495+0000) 2019-10-13T15:57:44.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:44.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:44.534774+0000) 2019-10-13T15:57:44.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:44.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:45.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:45.751+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:45.753802+0000) 2019-10-13T15:57:45.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:45.751+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:45.753889+0000) 2019-10-13T15:57:45.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:45.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:45.764767+0000) 2019-10-13T15:57:46.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:45.997+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:45.999251+0000) 2019-10-13T15:57:46.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:46.303+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:46.305885+0000) 2019-10-13T15:57:46.539 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:57:46.539 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:57:46.543 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:57:46.543 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:57:49.545 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:49.539+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.543043+0000) 2019-10-13T15:57:49.546 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:49.543+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.543578+0000) 2019-10-13T15:57:49.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:49.543+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.543881+0000) 2019-10-13T15:57:49.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:49.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.544484+0000) 2019-10-13T15:57:49.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:49.543+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.545308+0000) 2019-10-13T15:57:49.548 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:49.543+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.545667+0000) 2019-10-13T15:57:49.549 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:49.547+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.547337+0000) 2019-10-13T15:57:49.555 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:49.549+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:49.552674+0000) 2019-10-13T15:57:49.556 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:49.551+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232721 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:52.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:52.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:52.743343+0000) 2019-10-13T15:57:53.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:53.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:53.478971+0000) 2019-10-13T15:57:53.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:53.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:53.727917+0000) 2019-10-13T15:57:54.560 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:54.554+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.557848+0000) 2019-10-13T15:57:54.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:54.554+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.558263+0000) 2019-10-13T15:57:54.561 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:54.554+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.558778+0000) 2019-10-13T15:57:54.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:54.558+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.559333+0000) 2019-10-13T15:57:54.562 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:54.556+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.560058+0000) 2019-10-13T15:57:54.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:54.558+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.560855+0000) 2019-10-13T15:57:54.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:54.558+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.561393+0000) 2019-10-13T15:57:54.570 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:54.564+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:54.568161+0000) 2019-10-13T15:57:54.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:54.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:57:55.756 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:55.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:55.754004+0000) 2019-10-13T15:57:55.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:55.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:55.754107+0000) 2019-10-13T15:57:55.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:55.762+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:55.764991+0000) 2019-10-13T15:57:56.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:56.008+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:56.010655+0000) 2019-10-13T15:57:56.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:56.302+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:56.306113+0000) 2019-10-13T15:57:59.576 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:57:59.570+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.573208+0000) 2019-10-13T15:57:59.576 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:57:59.570+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.573229+0000) 2019-10-13T15:57:59.576 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:57:59.570+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.574584+0000) 2019-10-13T15:57:59.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:57:59.570+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.574919+0000) 2019-10-13T15:57:59.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:57:59.572+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.575557+0000) 2019-10-13T15:57:59.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:57:59.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.576301+0000) 2019-10-13T15:57:59.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:57:59.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.577077+0000) 2019-10-13T15:57:59.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:57:59.576+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:57:59.580217+0000) 2019-10-13T15:57:59.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:57:59.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232790 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:02.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:02.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:02.743573+0000) 2019-10-13T15:58:03.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:03.474+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:03.479180+0000) 2019-10-13T15:58:03.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:03.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:03.728141+0000) 2019-10-13T15:58:04.590 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:04.586+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.587821+0000) 2019-10-13T15:58:04.591 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:04.584+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.588192+0000) 2019-10-13T15:58:04.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:04.586+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.588441+0000) 2019-10-13T15:58:04.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:04.586+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.589786+0000) 2019-10-13T15:58:04.592 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:04.586+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.590037+0000) 2019-10-13T15:58:04.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:04.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.590333+0000) 2019-10-13T15:58:04.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:04.586+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.590831+0000) 2019-10-13T15:58:04.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:04.592+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:04.595676+0000) 2019-10-13T15:58:04.601 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:04.594+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:05.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:05.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:05.754256+0000) 2019-10-13T15:58:05.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:05.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:05.754338+0000) 2019-10-13T15:58:05.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:05.762+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:05.765187+0000) 2019-10-13T15:58:06.020 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:06.012+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:06.017363+0000) 2019-10-13T15:58:06.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:06.302+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:06.306325+0000) 2019-10-13T15:58:09.604 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:09.598+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.602067+0000) 2019-10-13T15:58:09.605 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:09.598+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.602475+0000) 2019-10-13T15:58:09.606 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:09.602+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.603683+0000) 2019-10-13T15:58:09.606 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:09.602+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.604264+0000) 2019-10-13T15:58:09.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:09.602+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.604721+0000) 2019-10-13T15:58:09.607 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:09.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.605148+0000) 2019-10-13T15:58:09.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:09.602+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.605743+0000) 2019-10-13T15:58:09.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:09.608+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:09.611218+0000) 2019-10-13T15:58:09.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:09.610+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232859 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:12.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:12.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:12.743763+0000) 2019-10-13T15:58:13.481 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:13.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:13.479377+0000) 2019-10-13T15:58:13.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:13.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:13.728369+0000) 2019-10-13T15:58:14.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:14.614+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.618261+0000) 2019-10-13T15:58:14.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:14.614+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.618595+0000) 2019-10-13T15:58:14.621 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:14.614+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.618947+0000) 2019-10-13T15:58:14.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:14.616+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.619341+0000) 2019-10-13T15:58:14.623 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:14.618+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.619751+0000) 2019-10-13T15:58:14.623 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:14.618+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.620186+0000) 2019-10-13T15:58:14.623 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:14.618+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.620387+0000) 2019-10-13T15:58:14.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:14.620+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:14.624056+0000) 2019-10-13T15:58:14.633 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:14.626+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:15.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:15.750+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:15.754593+0000) 2019-10-13T15:58:15.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:15.750+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:15.754457+0000) 2019-10-13T15:58:15.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:15.762+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:15.765402+0000) 2019-10-13T15:58:16.026 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:16.020+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:16.023687+0000) 2019-10-13T15:58:16.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:16.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:16.306570+0000) 2019-10-13T15:58:16.601 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:58:16.601 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:58:16.605 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:58:16.605 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:58:19.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:19.630+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.633671+0000) 2019-10-13T15:58:19.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:19.630+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.634111+0000) 2019-10-13T15:58:19.637 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:19.630+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.634579+0000) 2019-10-13T15:58:19.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:19.634+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.635537+0000) 2019-10-13T15:58:19.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:19.632+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.636387+0000) 2019-10-13T15:58:19.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:19.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.636567+0000) 2019-10-13T15:58:19.639 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:19.634+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.637003+0000) 2019-10-13T15:58:19.644 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:19.636+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:19.641187+0000) 2019-10-13T15:58:19.649 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:19.642+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232928 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:22.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:22.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:22.744009+0000) 2019-10-13T15:58:23.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:23.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:23.479572+0000) 2019-10-13T15:58:23.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:23.728+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:23.728597+0000) 2019-10-13T15:58:24.652 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:24.644+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.650015+0000) 2019-10-13T15:58:24.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:24.646+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.650778+0000) 2019-10-13T15:58:24.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:24.646+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.651219+0000) 2019-10-13T15:58:24.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:24.650+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.651831+0000) 2019-10-13T15:58:24.654 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:24.650+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.652435+0000) 2019-10-13T15:58:24.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:24.650+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.652858+0000) 2019-10-13T15:58:24.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:24.650+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.653367+0000) 2019-10-13T15:58:24.659 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:24.652+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:24.656996+0000) 2019-10-13T15:58:24.664 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:24.658+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:25.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:25.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:25.754655+0000) 2019-10-13T15:58:25.757 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:25.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:25.754765+0000) 2019-10-13T15:58:25.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:25.762+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:25.765597+0000) 2019-10-13T15:58:26.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:26.028+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:26.030182+0000) 2019-10-13T15:58:26.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:26.302+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:26.306793+0000) 2019-10-13T15:58:29.668 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:29.662+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.665479+0000) 2019-10-13T15:58:29.669 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:29.662+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.666679+0000) 2019-10-13T15:58:29.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:29.664+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.667051+0000) 2019-10-13T15:58:29.669 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:29.666+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.667532+0000) 2019-10-13T15:58:29.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:29.666+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.667777+0000) 2019-10-13T15:58:29.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:29.666+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.668505+0000) 2019-10-13T15:58:29.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:29.666+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.668953+0000) 2019-10-13T15:58:29.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:29.672+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:29.674726+0000) 2019-10-13T15:58:29.680 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:29.674+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 232997 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:32.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:32.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:32.744189+0000) 2019-10-13T15:58:33.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:33.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:33.479766+0000) 2019-10-13T15:58:33.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:33.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:33.728787+0000) 2019-10-13T15:58:34.684 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:34.676+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.681637+0000) 2019-10-13T15:58:34.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:34.678+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.681915+0000) 2019-10-13T15:58:34.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:34.678+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.682353+0000) 2019-10-13T15:58:34.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:34.678+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.682633+0000) 2019-10-13T15:58:34.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:34.682+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.683591+0000) 2019-10-13T15:58:34.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:34.682+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.684395+0000) 2019-10-13T15:58:34.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:34.682+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.685355+0000) 2019-10-13T15:58:34.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:34.684+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:34.689870+0000) 2019-10-13T15:58:34.696 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:34.690+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:35.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:35.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:35.754875+0000) 2019-10-13T15:58:35.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:35.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:35.754991+0000) 2019-10-13T15:58:35.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:35.762+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:35.765792+0000) 2019-10-13T15:58:36.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:36.036+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:36.040919+0000) 2019-10-13T15:58:36.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:36.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:36.306992+0000) 2019-10-13T15:58:39.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:39.693+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.698126+0000) 2019-10-13T15:58:39.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:39.693+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.698348+0000) 2019-10-13T15:58:39.701 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:39.693+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.698839+0000) 2019-10-13T15:58:39.702 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:39.693+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.699141+0000) 2019-10-13T15:58:39.702 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:39.696+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.699781+0000) 2019-10-13T15:58:39.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:39.697+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.699413+0000) 2019-10-13T15:58:39.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:39.697+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.700108+0000) 2019-10-13T15:58:39.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:39.704+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:39.707513+0000) 2019-10-13T15:58:39.712 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:39.705+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:42.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:42.741+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:42.744351+0000) 2019-10-13T15:58:43.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:43.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:43.479961+0000) 2019-10-13T15:58:43.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:43.724+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:43.729033+0000) 2019-10-13T15:58:44.716 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:44.709+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.713753+0000) 2019-10-13T15:58:44.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:44.709+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.714237+0000) 2019-10-13T15:58:44.717 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:44.709+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.714840+0000) 2019-10-13T15:58:44.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:44.713+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.715815+0000) 2019-10-13T15:58:44.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:44.713+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.715589+0000) 2019-10-13T15:58:44.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:44.713+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.716257+0000) 2019-10-13T15:58:44.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:44.712+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.717006+0000) 2019-10-13T15:58:44.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:44.720+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:44.723051+0000) 2019-10-13T15:58:44.727 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:44.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:45.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:45.749+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:45.755073+0000) 2019-10-13T15:58:45.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:45.749+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:45.755225+0000) 2019-10-13T15:58:45.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:45.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:45.765987+0000) 2019-10-13T15:58:46.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:46.044+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:46.047455+0000) 2019-10-13T15:58:46.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:46.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:46.307196+0000) 2019-10-13T15:58:46.853 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:58:46.854 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:58:46.857 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:58:46.857 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:58:49.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:49.723+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.729261+0000) 2019-10-13T15:58:49.732 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:49.725+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.729487+0000) 2019-10-13T15:58:49.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:49.725+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.729706+0000) 2019-10-13T15:58:49.733 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:49.725+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.730012+0000) 2019-10-13T15:58:49.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:49.725+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.730594+0000) 2019-10-13T15:58:49.733 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:49.725+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.730798+0000) 2019-10-13T15:58:49.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:49.729+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.732096+0000) 2019-10-13T15:58:49.741 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:49.735+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:49.739345+0000) 2019-10-13T15:58:49.742 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:49.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:52.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:52.741+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:52.744556+0000) 2019-10-13T15:58:53.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:53.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:53.480155+0000) 2019-10-13T15:58:53.731 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:53.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:53.729264+0000) 2019-10-13T15:58:54.745 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:54.737+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.742639+0000) 2019-10-13T15:58:54.746 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:54.741+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.743632+0000) 2019-10-13T15:58:54.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:54.741+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.744647+0000) 2019-10-13T15:58:54.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:54.739+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.745128+0000) 2019-10-13T15:58:54.748 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:54.741+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.745693+0000) 2019-10-13T15:58:54.748 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:54.741+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.746346+0000) 2019-10-13T15:58:54.749 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:54.741+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.746509+0000) 2019-10-13T15:58:54.756 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:54.751+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:54.754424+0000) 2019-10-13T15:58:54.758 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:54.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:58:55.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:55.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:55.755331+0000) 2019-10-13T15:58:55.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:55.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:55.755486+0000) 2019-10-13T15:58:55.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:55.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:55.766175+0000) 2019-10-13T15:58:56.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:56.047+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:56.054023+0000) 2019-10-13T15:58:56.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:56.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:56.307393+0000) 2019-10-13T15:58:59.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:58:59.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.758537+0000) 2019-10-13T15:58:59.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:58:59.757+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.759492+0000) 2019-10-13T15:58:59.762 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:58:59.757+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.760190+0000) 2019-10-13T15:58:59.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:58:59.757+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.760584+0000) 2019-10-13T15:58:59.763 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:58:59.755+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.761424+0000) 2019-10-13T15:58:59.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:58:59.757+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.761597+0000) 2019-10-13T15:58:59.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:58:59.757+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.762007+0000) 2019-10-13T15:58:59.769 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:58:59.759+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:58:59.766114+0000) 2019-10-13T15:58:59.773 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:58:59.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233204 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:02.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:02.741+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:02.744786+0000) 2019-10-13T15:59:03.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:03.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:03.480345+0000) 2019-10-13T15:59:03.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:03.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:03.729463+0000) 2019-10-13T15:59:04.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:04.769+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.774628+0000) 2019-10-13T15:59:04.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:04.773+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.775421+0000) 2019-10-13T15:59:04.778 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:04.773+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.776004+0000) 2019-10-13T15:59:04.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:04.773+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.776127+0000) 2019-10-13T15:59:04.779 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:04.771+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.776882+0000) 2019-10-13T15:59:04.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:04.773+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.777520+0000) 2019-10-13T15:59:04.780 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:04.773+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.778496+0000) 2019-10-13T15:59:04.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:04.779+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:04.784547+0000) 2019-10-13T15:59:04.788 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:04.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233237 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:05.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:05.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:05.755542+0000) 2019-10-13T15:59:05.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:05.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:05.755644+0000) 2019-10-13T15:59:05.768 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:05.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:05.766401+0000) 2019-10-13T15:59:06.063 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:06.059+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:06.060704+0000) 2019-10-13T15:59:06.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:06.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:06.307599+0000) 2019-10-13T15:59:09.792 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:09.783+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.789914+0000) 2019-10-13T15:59:09.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:09.785+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.789802+0000) 2019-10-13T15:59:09.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:09.785+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.790399+0000) 2019-10-13T15:59:09.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:09.785+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.790980+0000) 2019-10-13T15:59:09.794 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:09.785+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.791280+0000) 2019-10-13T15:59:09.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:09.789+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.791730+0000) 2019-10-13T15:59:09.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:09.789+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.792161+0000) 2019-10-13T15:59:09.800 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:09.791+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:09.798014+0000) 2019-10-13T15:59:09.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:09.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233273 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:12.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:12.741+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:12.745020+0000) 2019-10-13T15:59:13.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:13.477+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:13.480574+0000) 2019-10-13T15:59:13.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:13.723+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:13.729652+0000) 2019-10-13T15:59:14.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:14.801+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.803863+0000) 2019-10-13T15:59:14.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:14.801+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.804941+0000) 2019-10-13T15:59:14.807 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:14.799+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.805269+0000) 2019-10-13T15:59:14.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:14.801+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.805605+0000) 2019-10-13T15:59:14.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:14.801+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.805907+0000) 2019-10-13T15:59:14.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:14.801+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.806999+0000) 2019-10-13T15:59:14.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:14.805+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.807606+0000) 2019-10-13T15:59:14.813 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:14.807+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:14.811470+0000) 2019-10-13T15:59:14.819 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:14.813+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:15.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:15.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:15.755710+0000) 2019-10-13T15:59:15.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:15.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:15.755801+0000) 2019-10-13T15:59:15.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:15.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:15.766593+0000) 2019-10-13T15:59:16.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:16.063+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:16.067149+0000) 2019-10-13T15:59:16.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:16.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:16.307797+0000) 2019-10-13T15:59:16.948 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:59:16.948 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:59:16.952 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:59:16.952 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:59:19.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:19.817+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.820429+0000) 2019-10-13T15:59:19.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:19.817+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.820831+0000) 2019-10-13T15:59:19.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:19.815+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.821389+0000) 2019-10-13T15:59:19.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:19.817+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.821781+0000) 2019-10-13T15:59:19.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:19.817+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.822154+0000) 2019-10-13T15:59:19.825 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:19.817+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.823019+0000) 2019-10-13T15:59:19.825 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:19.817+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.823281+0000) 2019-10-13T15:59:19.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:19.823+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:19.826960+0000) 2019-10-13T15:59:19.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:19.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:22.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:22.740+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:22.745219+0000) 2019-10-13T15:59:23.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:23.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:23.480767+0000) 2019-10-13T15:59:23.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:23.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:23.729848+0000) 2019-10-13T15:59:24.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:24.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.835145+0000) 2019-10-13T15:59:24.838 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:24.832+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.835760+0000) 2019-10-13T15:59:24.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:24.832+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.836115+0000) 2019-10-13T15:59:24.839 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:24.832+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.836474+0000) 2019-10-13T15:59:24.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:24.832+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.836903+0000) 2019-10-13T15:59:24.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:24.832+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.837506+0000) 2019-10-13T15:59:24.840 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:24.832+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.838175+0000) 2019-10-13T15:59:24.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:24.839+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:24.843981+0000) 2019-10-13T15:59:24.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:24.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233375 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:25.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:25.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:25.755838+0000) 2019-10-13T15:59:25.758 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:25.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:25.755966+0000) 2019-10-13T15:59:25.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:25.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:25.766765+0000) 2019-10-13T15:59:26.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:26.067+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:26.073694+0000) 2019-10-13T15:59:26.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:26.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:26.308018+0000) 2019-10-13T15:59:29.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:29.844+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.849505+0000) 2019-10-13T15:59:29.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:29.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.850228+0000) 2019-10-13T15:59:29.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:29.844+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.851170+0000) 2019-10-13T15:59:29.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:29.848+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.851611+0000) 2019-10-13T15:59:29.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:29.848+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.851893+0000) 2019-10-13T15:59:29.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:29.848+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.852251+0000) 2019-10-13T15:59:29.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:29.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.852632+0000) 2019-10-13T15:59:29.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:29.856+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:29.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:29.859+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:29.864524+0000) 2019-10-13T15:59:32.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:32.744+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:32.745437+0000) 2019-10-13T15:59:33.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:33.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:33.480960+0000) 2019-10-13T15:59:33.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:33.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:33.730043+0000) 2019-10-13T15:59:34.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:34.859+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.864724+0000) 2019-10-13T15:59:34.867 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:34.860+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.865146+0000) 2019-10-13T15:59:34.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:34.860+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.866145+0000) 2019-10-13T15:59:34.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:34.860+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.866574+0000) 2019-10-13T15:59:34.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:34.860+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.866642+0000) 2019-10-13T15:59:34.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:34.860+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.866883+0000) 2019-10-13T15:59:34.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:34.864+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.867452+0000) 2019-10-13T15:59:34.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:34.867+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:34.871990+0000) 2019-10-13T15:59:34.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:34.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:35.758 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:35.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:35.756015+0000) 2019-10-13T15:59:35.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:35.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:35.756162+0000) 2019-10-13T15:59:35.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:35.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:35.766941+0000) 2019-10-13T15:59:36.083 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:36.075+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:36.080427+0000) 2019-10-13T15:59:36.310 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:36.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:36.308196+0000) 2019-10-13T15:59:39.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:39.876+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.879437+0000) 2019-10-13T15:59:39.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:39.875+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.880943+0000) 2019-10-13T15:59:39.883 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:39.876+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.881260+0000) 2019-10-13T15:59:39.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:39.876+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.881807+0000) 2019-10-13T15:59:39.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:39.876+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.882291+0000) 2019-10-13T15:59:39.885 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:39.876+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.882895+0000) 2019-10-13T15:59:39.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:39.880+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.883438+0000) 2019-10-13T15:59:39.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:39.883+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:39.888659+0000) 2019-10-13T15:59:39.895 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:39.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233480 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:42.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:42.740+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:42.745670+0000) 2019-10-13T15:59:43.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:43.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:43.481204+0000) 2019-10-13T15:59:43.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:43.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:43.730240+0000) 2019-10-13T15:59:44.898 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:44.892+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.896058+0000) 2019-10-13T15:59:44.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:44.890+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.896492+0000) 2019-10-13T15:59:44.899 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:44.892+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.897210+0000) 2019-10-13T15:59:44.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:44.892+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.897758+0000) 2019-10-13T15:59:44.900 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:44.892+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.897459+0000) 2019-10-13T15:59:44.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:44.892+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.898830+0000) 2019-10-13T15:59:44.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:44.892+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.898626+0000) 2019-10-13T15:59:44.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:44.898+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:44.905325+0000) 2019-10-13T15:59:44.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:44.900+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:45.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:45.752+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:45.756194+0000) 2019-10-13T15:59:45.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:45.752+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:45.756372+0000) 2019-10-13T15:59:45.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:45.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:45.767149+0000) 2019-10-13T15:59:46.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:46.082+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:46.086228+0000) 2019-10-13T15:59:46.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:46.304+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:46.308404+0000) 2019-10-13T15:59:47.042 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T15:59:47.042 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:59:47.046 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T15:59:47.046 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:59:49.913 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:49.904+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.910813+0000) 2019-10-13T15:59:49.913 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:49.904+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.911098+0000) 2019-10-13T15:59:49.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:49.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.912056+0000) 2019-10-13T15:59:49.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:49.908+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.912591+0000) 2019-10-13T15:59:49.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:49.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.913722+0000) 2019-10-13T15:59:49.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:49.908+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.913882+0000) 2019-10-13T15:59:49.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:49.908+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.914107+0000) 2019-10-13T15:59:49.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:49.914+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:49.919813+0000) 2019-10-13T15:59:49.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:49.916+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:52.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:52.740+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:52.745886+0000) 2019-10-13T15:59:53.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:53.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:53.481434+0000) 2019-10-13T15:59:53.732 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:53.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:53.730431+0000) 2019-10-13T15:59:54.928 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:54.920+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.926156+0000) 2019-10-13T15:59:54.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:54.920+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.926473+0000) 2019-10-13T15:59:54.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:54.920+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.926946+0000) 2019-10-13T15:59:54.930 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:54.924+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.927823+0000) 2019-10-13T15:59:54.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:54.924+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.928358+0000) 2019-10-13T15:59:54.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:54.924+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.928692+0000) 2019-10-13T15:59:54.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:54.922+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.930026+0000) 2019-10-13T15:59:54.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:54.930+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:54.935784+0000) 2019-10-13T15:59:54.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:54.932+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T15:59:55.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:55.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:55.756375+0000) 2019-10-13T15:59:55.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:55.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:55.756551+0000) 2019-10-13T15:59:55.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:55.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:55.767364+0000) 2019-10-13T15:59:56.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:56.086+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:56.092969+0000) 2019-10-13T15:59:56.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:56.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:56.308617+0000) 2019-10-13T15:59:59.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T15:59:59.938+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.942130+0000) 2019-10-13T15:59:59.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T15:59:59.936+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.942491+0000) 2019-10-13T15:59:59.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T15:59:59.936+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.943074+0000) 2019-10-13T15:59:59.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T15:59:59.940+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.943448+0000) 2019-10-13T15:59:59.946 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T15:59:59.940+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.943836+0000) 2019-10-13T15:59:59.946 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T15:59:59.940+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.944366+0000) 2019-10-13T15:59:59.947 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T15:59:59.940+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.944960+0000) 2019-10-13T15:59:59.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T15:59:59.942+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:59:59.947895+0000) 2019-10-13T15:59:59.955 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T15:59:59.948+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:02.748 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:02.744+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:02.746089+0000) 2019-10-13T16:00:03.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:03.476+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:03.481627+0000) 2019-10-13T16:00:03.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:03.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:03.730635+0000) 2019-10-13T16:00:04.959 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:04.951+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.956500+0000) 2019-10-13T16:00:04.959 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:04.950+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.956936+0000) 2019-10-13T16:00:04.960 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:04.951+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.957232+0000) 2019-10-13T16:00:04.960 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:04.951+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.957792+0000) 2019-10-13T16:00:04.960 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:04.951+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.958395+0000) 2019-10-13T16:00:04.961 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:04.951+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.958763+0000) 2019-10-13T16:00:04.962 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:04.955+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.960088+0000) 2019-10-13T16:00:04.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:04.958+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:04.964216+0000) 2019-10-13T16:00:04.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:04.963+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:05.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:05.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:05.756555+0000) 2019-10-13T16:00:05.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:05.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:05.756736+0000) 2019-10-13T16:00:05.769 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:05.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:05.767540+0000) 2019-10-13T16:00:06.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:06.094+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:06.099612+0000) 2019-10-13T16:00:06.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:06.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:06.308851+0000) 2019-10-13T16:00:09.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:09.967+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.971650+0000) 2019-10-13T16:00:09.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:09.967+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.972432+0000) 2019-10-13T16:00:09.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:09.967+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.972912+0000) 2019-10-13T16:00:09.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:09.967+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.973422+0000) 2019-10-13T16:00:09.997 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:09.967+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.973955+0000) 2019-10-13T16:00:09.998 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:09.967+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.974351+0000) 2019-10-13T16:00:10.007 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:09.966+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.972030+0000) 2019-10-13T16:00:10.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:09.974+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:09.980368+0000) 2019-10-13T16:00:10.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:09.975+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:12.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:12.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:12.746321+0000) 2019-10-13T16:00:13.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:13.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:13.481841+0000) 2019-10-13T16:00:13.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:13.726+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:13.730862+0000) 2019-10-13T16:00:14.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:14.982+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.986746+0000) 2019-10-13T16:00:14.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:14.979+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.987067+0000) 2019-10-13T16:00:14.990 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:14.983+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.987589+0000) 2019-10-13T16:00:14.990 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:14.983+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.988483+0000) 2019-10-13T16:00:14.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:14.983+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.988997+0000) 2019-10-13T16:00:14.991 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:14.983+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.989042+0000) 2019-10-13T16:00:14.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:14.983+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.989367+0000) 2019-10-13T16:00:14.996 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:14.990+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:14.994228+0000) 2019-10-13T16:00:14.999 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:14.991+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233720 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:15.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:15.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:15.756733+0000) 2019-10-13T16:00:15.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:15.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:15.756916+0000) 2019-10-13T16:00:15.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:15.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:15.767716+0000) 2019-10-13T16:00:16.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:16.098+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:16.106015+0000) 2019-10-13T16:00:16.311 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:16.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:16.309062+0000) 2019-10-13T16:00:17.328 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:00:17.328 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:00:17.332 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:00:17.332 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:00:20.002 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:19.994+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.000411+0000) 2019-10-13T16:00:20.003 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:19.995+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.001162+0000) 2019-10-13T16:00:20.004 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:19.995+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.001870+0000) 2019-10-13T16:00:20.004 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:19.995+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.002472+0000) 2019-10-13T16:00:20.005 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:19.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.002835+0000) 2019-10-13T16:00:20.005 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:19.995+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.002674+0000) 2019-10-13T16:00:20.006 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:19.999+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.003484+0000) 2019-10-13T16:00:20.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:20.002+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:20.009780+0000) 2019-10-13T16:00:20.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:20.007+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:22.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:22.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:22.746524+0000) 2019-10-13T16:00:23.484 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:23.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:23.482037+0000) 2019-10-13T16:00:23.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:23.730+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:23.731093+0000) 2019-10-13T16:00:25.019 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:25.011+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.016551+0000) 2019-10-13T16:00:25.019 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:25.011+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.017101+0000) 2019-10-13T16:00:25.020 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:25.011+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.017897+0000) 2019-10-13T16:00:25.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:25.014+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.018568+0000) 2019-10-13T16:00:25.021 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:25.015+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.019417+0000) 2019-10-13T16:00:25.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:25.015+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.020091+0000) 2019-10-13T16:00:25.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:25.015+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.020422+0000) 2019-10-13T16:00:25.027 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:25.018+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.024699+0000) 2019-10-13T16:00:25.031 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:25.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:25.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:25.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.756937+0000) 2019-10-13T16:00:25.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:25.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.757094+0000) 2019-10-13T16:00:25.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:25.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:25.767892+0000) 2019-10-13T16:00:26.115 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:26.110+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:26.112571+0000) 2019-10-13T16:00:26.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:26.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:26.309274+0000) 2019-10-13T16:00:30.036 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:30.027+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.033333+0000) 2019-10-13T16:00:30.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:30.027+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.033500+0000) 2019-10-13T16:00:30.037 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:30.030+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.033773+0000) 2019-10-13T16:00:30.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:30.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.034452+0000) 2019-10-13T16:00:30.037 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:30.027+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.035048+0000) 2019-10-13T16:00:30.038 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:30.031+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.035506+0000) 2019-10-13T16:00:30.038 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:30.031+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.035818+0000) 2019-10-13T16:00:30.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:30.038+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:30.041073+0000) 2019-10-13T16:00:30.047 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:30.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:32.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:32.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:32.746725+0000) 2019-10-13T16:00:33.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:33.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:33.482264+0000) 2019-10-13T16:00:33.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:33.730+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:33.731287+0000) 2019-10-13T16:00:35.051 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:35.043+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.049158+0000) 2019-10-13T16:00:35.053 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:35.043+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.048732+0000) 2019-10-13T16:00:35.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:35.043+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.050117+0000) 2019-10-13T16:00:35.056 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:35.043+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.050437+0000) 2019-10-13T16:00:35.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:35.043+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.050851+0000) 2019-10-13T16:00:35.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:35.050+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.052462+0000) 2019-10-13T16:00:35.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:35.047+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.052767+0000) 2019-10-13T16:00:35.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:35.054+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.057804+0000) 2019-10-13T16:00:35.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:35.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233858 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:35.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:35.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.757113+0000) 2019-10-13T16:00:35.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:35.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.757279+0000) 2019-10-13T16:00:35.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:35.767+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:35.768067+0000) 2019-10-13T16:00:36.127 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:36.122+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:36.124363+0000) 2019-10-13T16:00:36.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:36.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:36.309490+0000) 2019-10-13T16:00:40.067 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:40.063+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.064582+0000) 2019-10-13T16:00:40.067 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:40.063+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.064924+0000) 2019-10-13T16:00:40.068 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:40.061+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.065481+0000) 2019-10-13T16:00:40.068 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:40.063+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.066308+0000) 2019-10-13T16:00:40.069 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:40.063+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.066329+0000) 2019-10-13T16:00:40.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:40.063+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.066674+0000) 2019-10-13T16:00:40.070 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:40.067+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.067923+0000) 2019-10-13T16:00:40.076 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:40.073+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:40.074189+0000) 2019-10-13T16:00:40.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:40.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:42.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:42.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:42.746946+0000) 2019-10-13T16:00:43.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:43.479+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:43.482460+0000) 2019-10-13T16:00:43.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:43.729+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:43.731486+0000) 2019-10-13T16:00:45.082 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:45.079+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.079979+0000) 2019-10-13T16:00:45.083 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:45.077+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.080587+0000) 2019-10-13T16:00:45.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:45.079+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.082101+0000) 2019-10-13T16:00:45.085 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:45.079+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.082655+0000) 2019-10-13T16:00:45.086 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:45.083+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.083363+0000) 2019-10-13T16:00:45.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:45.083+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.083752+0000) 2019-10-13T16:00:45.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:45.083+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.084477+0000) 2019-10-13T16:00:45.091 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:45.085+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.089062+0000) 2019-10-13T16:00:45.094 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:45.091+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233926 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:45.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:45.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.757251+0000) 2019-10-13T16:00:45.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:45.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.757430+0000) 2019-10-13T16:00:45.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:45.767+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:45.768239+0000) 2019-10-13T16:00:46.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:46.129+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:46.130711+0000) 2019-10-13T16:00:46.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:46.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:46.309713+0000) 2019-10-13T16:00:47.376 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:00:47.376 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:00:47.381 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:00:47.381 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:00:50.097 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:50.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.094292+0000) 2019-10-13T16:00:50.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:50.090+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.094849+0000) 2019-10-13T16:00:50.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:50.090+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.095126+0000) 2019-10-13T16:00:50.098 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:50.094+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.096307+0000) 2019-10-13T16:00:50.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:50.093+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.096437+0000) 2019-10-13T16:00:50.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:50.094+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.096561+0000) 2019-10-13T16:00:50.099 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:50.094+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.096736+0000) 2019-10-13T16:00:50.103 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:50.097+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:50.101633+0000) 2019-10-13T16:00:50.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:50.102+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:52.749 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:52.742+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:52.747163+0000) 2019-10-13T16:00:53.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:53.478+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:53.482652+0000) 2019-10-13T16:00:53.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:53.729+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:53.731681+0000) 2019-10-13T16:00:55.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:55.106+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.109567+0000) 2019-10-13T16:00:55.112 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:00:55.106+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.109846+0000) 2019-10-13T16:00:55.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:55.106+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.110999+0000) 2019-10-13T16:00:55.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:55.110+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.111583+0000) 2019-10-13T16:00:55.114 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:55.110+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.112097+0000) 2019-10-13T16:00:55.115 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:00:55.109+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.112710+0000) 2019-10-13T16:00:55.115 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:00:55.110+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.113060+0000) 2019-10-13T16:00:55.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:55.117+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.118568+0000) 2019-10-13T16:00:55.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:00:55.118+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 233996 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:00:55.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:00:55.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.757478+0000) 2019-10-13T16:00:55.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:00:55.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.757613+0000) 2019-10-13T16:00:55.770 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:00:55.766+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:55.768437+0000) 2019-10-13T16:00:56.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:00:56.133+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:56.136982+0000) 2019-10-13T16:00:56.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:00:56.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:00:56.309954+0000) 2019-10-13T16:01:00.128 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:00.122+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.125870+0000) 2019-10-13T16:01:00.129 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:00.122+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.126307+0000) 2019-10-13T16:01:00.129 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:00.125+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.126694+0000) 2019-10-13T16:01:00.130 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:00.122+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.127031+0000) 2019-10-13T16:01:00.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:00.126+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.127838+0000) 2019-10-13T16:01:00.130 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:00.126+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.127953+0000) 2019-10-13T16:01:00.131 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:00.126+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.128777+0000) 2019-10-13T16:01:00.134 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:00.129+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:00.132016+0000) 2019-10-13T16:01:00.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:00.134+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:02.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:02.746+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:02.747422+0000) 2019-10-13T16:01:03.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:03.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:03.482849+0000) 2019-10-13T16:01:03.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:03.729+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:03.731873+0000) 2019-10-13T16:01:05.142 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:05.138+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.140192+0000) 2019-10-13T16:01:05.143 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:05.138+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.140610+0000) 2019-10-13T16:01:05.144 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:05.137+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.141657+0000) 2019-10-13T16:01:05.144 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:05.138+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.141951+0000) 2019-10-13T16:01:05.144 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:05.138+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.142264+0000) 2019-10-13T16:01:05.145 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:05.142+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.143391+0000) 2019-10-13T16:01:05.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:05.142+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.144108+0000) 2019-10-13T16:01:05.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:05.145+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.149674+0000) 2019-10-13T16:01:05.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:05.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:05.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:05.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.757657+0000) 2019-10-13T16:01:05.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:05.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.757765+0000) 2019-10-13T16:01:05.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:05.766+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:05.768611+0000) 2019-10-13T16:01:06.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:06.141+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:06.143467+0000) 2019-10-13T16:01:06.312 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:06.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:06.310139+0000) 2019-10-13T16:01:10.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:10.150+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.153729+0000) 2019-10-13T16:01:10.157 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:10.150+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.154065+0000) 2019-10-13T16:01:10.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:10.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.154581+0000) 2019-10-13T16:01:10.158 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:10.154+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.155376+0000) 2019-10-13T16:01:10.158 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:10.154+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.155409+0000) 2019-10-13T16:01:10.158 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:10.153+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.156570+0000) 2019-10-13T16:01:10.159 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:10.154+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.156659+0000) 2019-10-13T16:01:10.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:10.161+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:10.165703+0000) 2019-10-13T16:01:10.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:10.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:12.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:12.746+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:12.747659+0000) 2019-10-13T16:01:13.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:13.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:13.483044+0000) 2019-10-13T16:01:13.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:13.729+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:13.732065+0000) 2019-10-13T16:01:15.172 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:15.166+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.170046+0000) 2019-10-13T16:01:15.173 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:15.166+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.170781+0000) 2019-10-13T16:01:15.173 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:15.170+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.171528+0000) 2019-10-13T16:01:15.174 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:15.170+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.171951+0000) 2019-10-13T16:01:15.174 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:15.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.172447+0000) 2019-10-13T16:01:15.175 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:15.170+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.173147+0000) 2019-10-13T16:01:15.176 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:15.169+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.173748+0000) 2019-10-13T16:01:15.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:15.177+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.180615+0000) 2019-10-13T16:01:15.183 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:15.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:15.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:15.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.757871+0000) 2019-10-13T16:01:15.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:15.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.757904+0000) 2019-10-13T16:01:15.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:15.766+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:15.768796+0000) 2019-10-13T16:01:16.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:16.149+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:16.149849+0000) 2019-10-13T16:01:16.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:16.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:16.310315+0000) 2019-10-13T16:01:17.492 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:01:17.493 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:01:17.496 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:01:17.496 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:01:20.187 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:20.182+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.184748+0000) 2019-10-13T16:01:20.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:20.182+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.185075+0000) 2019-10-13T16:01:20.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:20.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.185696+0000) 2019-10-13T16:01:20.188 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:20.181+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.185991+0000) 2019-10-13T16:01:20.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:20.182+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.186697+0000) 2019-10-13T16:01:20.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:20.182+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.186923+0000) 2019-10-13T16:01:20.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:20.186+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.187399+0000) 2019-10-13T16:01:20.195 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:20.189+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:20.192871+0000) 2019-10-13T16:01:20.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:20.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234165 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:22.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:22.746+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:22.747854+0000) 2019-10-13T16:01:23.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:23.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:23.483237+0000) 2019-10-13T16:01:23.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:23.729+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:23.732263+0000) 2019-10-13T16:01:25.203 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:25.197+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.200820+0000) 2019-10-13T16:01:25.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:25.198+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.201085+0000) 2019-10-13T16:01:25.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:25.198+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.201426+0000) 2019-10-13T16:01:25.204 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:25.198+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.202150+0000) 2019-10-13T16:01:25.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:25.198+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.202228+0000) 2019-10-13T16:01:25.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:25.198+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.202958+0000) 2019-10-13T16:01:25.205 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:25.202+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.203358+0000) 2019-10-13T16:01:25.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:25.205+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.209913+0000) 2019-10-13T16:01:25.213 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:25.206+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234197 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:25.760 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:25.754+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.758062+0000) 2019-10-13T16:01:25.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:25.754+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.758093+0000) 2019-10-13T16:01:25.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:25.766+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:25.768973+0000) 2019-10-13T16:01:26.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:26.153+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:26.156252+0000) 2019-10-13T16:01:26.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:26.306+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:26.310498+0000) 2019-10-13T16:01:30.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:30.210+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.213675+0000) 2019-10-13T16:01:30.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:30.210+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.214370+0000) 2019-10-13T16:01:30.217 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:30.210+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.214749+0000) 2019-10-13T16:01:30.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:30.214+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.216193+0000) 2019-10-13T16:01:30.219 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:30.214+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.216021+0000) 2019-10-13T16:01:30.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:30.214+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.216802+0000) 2019-10-13T16:01:30.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:30.213+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.217351+0000) 2019-10-13T16:01:30.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:30.221+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:30.222866+0000) 2019-10-13T16:01:30.227 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:30.222+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:32.750 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:32.745+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:32.748054+0000) 2019-10-13T16:01:33.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:33.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:33.483450+0000) 2019-10-13T16:01:33.734 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:33.728+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:33.732483+0000) 2019-10-13T16:01:35.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:35.225+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.228243+0000) 2019-10-13T16:01:35.231 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:35.225+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.228505+0000) 2019-10-13T16:01:35.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:35.225+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.229227+0000) 2019-10-13T16:01:35.233 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:35.225+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.230502+0000) 2019-10-13T16:01:35.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:35.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.229821+0000) 2019-10-13T16:01:35.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:35.228+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.231404+0000) 2019-10-13T16:01:35.234 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:35.225+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.230849+0000) 2019-10-13T16:01:35.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:35.232+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.236538+0000) 2019-10-13T16:01:35.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:35.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:35.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:35.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.758282+0000) 2019-10-13T16:01:35.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:35.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.758254+0000) 2019-10-13T16:01:35.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:35.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:35.769165+0000) 2019-10-13T16:01:36.165 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:36.160+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:36.162896+0000) 2019-10-13T16:01:36.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:36.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:36.310697+0000) 2019-10-13T16:01:40.245 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:40.237+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.242602+0000) 2019-10-13T16:01:40.245 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:40.237+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.243027+0000) 2019-10-13T16:01:40.246 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:40.241+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.243820+0000) 2019-10-13T16:01:40.246 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:40.240+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.244259+0000) 2019-10-13T16:01:40.247 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:40.241+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.244551+0000) 2019-10-13T16:01:40.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:40.241+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.245072+0000) 2019-10-13T16:01:40.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:40.241+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.246221+0000) 2019-10-13T16:01:40.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:40.248+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:40.252409+0000) 2019-10-13T16:01:40.257 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:40.249+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234296 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:42.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:42.745+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:42.748274+0000) 2019-10-13T16:01:43.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:43.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:43.483653+0000) 2019-10-13T16:01:43.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:43.728+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:43.732717+0000) 2019-10-13T16:01:45.261 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:45.253+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.258498+0000) 2019-10-13T16:01:45.262 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:45.256+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.258720+0000) 2019-10-13T16:01:45.263 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:45.253+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.259046+0000) 2019-10-13T16:01:45.263 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:45.257+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.259755+0000) 2019-10-13T16:01:45.264 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:45.257+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.260313+0000) 2019-10-13T16:01:45.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:45.257+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.261327+0000) 2019-10-13T16:01:45.265 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:45.257+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.261719+0000) 2019-10-13T16:01:45.268 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:45.264+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.266552+0000) 2019-10-13T16:01:45.272 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:45.265+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234329 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:45.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:45.753+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.758419+0000) 2019-10-13T16:01:45.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:45.753+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.758426+0000) 2019-10-13T16:01:45.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:45.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:45.769342+0000) 2019-10-13T16:01:46.172 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:46.164+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:46.169250+0000) 2019-10-13T16:01:46.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:46.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:46.310881+0000) 2019-10-13T16:01:47.604 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:01:47.604 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:01:47.608 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:01:47.608 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:01:50.277 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:50.269+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.274320+0000) 2019-10-13T16:01:50.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:50.269+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.274704+0000) 2019-10-13T16:01:50.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:50.272+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.275159+0000) 2019-10-13T16:01:50.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:50.273+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.275969+0000) 2019-10-13T16:01:50.278 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:50.273+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.276485+0000) 2019-10-13T16:01:50.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:50.273+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.276931+0000) 2019-10-13T16:01:50.279 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:50.273+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.277306+0000) 2019-10-13T16:01:50.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:50.280+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:50.284442+0000) 2019-10-13T16:01:50.288 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:50.281+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234363 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:52.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:52.745+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:52.748516+0000) 2019-10-13T16:01:53.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:53.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:53.483848+0000) 2019-10-13T16:01:53.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:53.728+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:53.732912+0000) 2019-10-13T16:01:55.292 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:01:55.285+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.289559+0000) 2019-10-13T16:01:55.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:55.285+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.290012+0000) 2019-10-13T16:01:55.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:01:55.285+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.290634+0000) 2019-10-13T16:01:55.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:55.289+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.291336+0000) 2019-10-13T16:01:55.294 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:55.289+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.292160+0000) 2019-10-13T16:01:55.295 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:55.289+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.292521+0000) 2019-10-13T16:01:55.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:01:55.288+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.293203+0000) 2019-10-13T16:01:55.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:55.296+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.299877+0000) 2019-10-13T16:01:55.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:01:55.297+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:01:55.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:01:55.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.758632+0000) 2019-10-13T16:01:55.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:01:55.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.758635+0000) 2019-10-13T16:01:55.771 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:01:55.765+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:55.769514+0000) 2019-10-13T16:01:56.178 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:01:56.172+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:56.175691+0000) 2019-10-13T16:01:56.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:01:56.309+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:01:56.311062+0000) 2019-10-13T16:02:00.306 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:00.301+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.304324+0000) 2019-10-13T16:02:00.307 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:00.301+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.305030+0000) 2019-10-13T16:02:00.307 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:00.301+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.305213+0000) 2019-10-13T16:02:00.308 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:00.301+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.306065+0000) 2019-10-13T16:02:00.309 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:00.301+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.306614+0000) 2019-10-13T16:02:00.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:00.301+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.306934+0000) 2019-10-13T16:02:00.310 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:00.304+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.307945+0000) 2019-10-13T16:02:00.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:00.312+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:00.316357+0000) 2019-10-13T16:02:00.319 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:00.313+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234428 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:02.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:02.745+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:02.748714+0000) 2019-10-13T16:02:03.486 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:03.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:03.484058+0000) 2019-10-13T16:02:03.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:03.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:03.733105+0000) 2019-10-13T16:02:05.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:05.317+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.319447+0000) 2019-10-13T16:02:05.322 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:05.317+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.320400+0000) 2019-10-13T16:02:05.323 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:05.317+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.320044+0000) 2019-10-13T16:02:05.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:05.316+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.321333+0000) 2019-10-13T16:02:05.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:05.317+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.321976+0000) 2019-10-13T16:02:05.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:05.317+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.322543+0000) 2019-10-13T16:02:05.325 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:05.317+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.322780+0000) 2019-10-13T16:02:05.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:05.324+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.328262+0000) 2019-10-13T16:02:05.333 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:05.325+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:05.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:05.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.758829+0000) 2019-10-13T16:02:05.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:05.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.758840+0000) 2019-10-13T16:02:05.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:05.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:05.769695+0000) 2019-10-13T16:02:06.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:06.176+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:06.181951+0000) 2019-10-13T16:02:06.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:06.305+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:06.311256+0000) 2019-10-13T16:02:10.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:10.332+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.334403+0000) 2019-10-13T16:02:10.337 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:10.329+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.334680+0000) 2019-10-13T16:02:10.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:10.329+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.335214+0000) 2019-10-13T16:02:10.338 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:10.333+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.336169+0000) 2019-10-13T16:02:10.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:10.333+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.336770+0000) 2019-10-13T16:02:10.340 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:10.333+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.337546+0000) 2019-10-13T16:02:10.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:10.333+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.337915+0000) 2019-10-13T16:02:10.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:10.336+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:10.341975+0000) 2019-10-13T16:02:10.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:10.341+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:12.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:12.745+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:12.748916+0000) 2019-10-13T16:02:13.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:13.481+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:13.484251+0000) 2019-10-13T16:02:13.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:13.728+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:13.733318+0000) 2019-10-13T16:02:15.351 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:15.345+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.349248+0000) 2019-10-13T16:02:15.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:15.345+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.350197+0000) 2019-10-13T16:02:15.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:15.345+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.351281+0000) 2019-10-13T16:02:15.354 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:15.348+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.351819+0000) 2019-10-13T16:02:15.354 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:15.349+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.352122+0000) 2019-10-13T16:02:15.355 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:15.349+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.352411+0000) 2019-10-13T16:02:15.355 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:15.349+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.352808+0000) 2019-10-13T16:02:15.364 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:15.357+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234527 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:15.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:15.360+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.363638+0000) 2019-10-13T16:02:15.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:15.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.759042+0000) 2019-10-13T16:02:15.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:15.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.759059+0000) 2019-10-13T16:02:15.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:15.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:15.769874+0000) 2019-10-13T16:02:16.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:16.188+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:16.188430+0000) 2019-10-13T16:02:16.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:16.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:16.311478+0000) 2019-10-13T16:02:17.873 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:02:17.874 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:02:17.877 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:02:17.877 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:02:20.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:20.360+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.365306+0000) 2019-10-13T16:02:20.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:20.360+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.365773+0000) 2019-10-13T16:02:20.368 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:20.360+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.365911+0000) 2019-10-13T16:02:20.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:20.360+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.366780+0000) 2019-10-13T16:02:20.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:20.360+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.367070+0000) 2019-10-13T16:02:20.370 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:20.364+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.367376+0000) 2019-10-13T16:02:20.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:20.364+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.368116+0000) 2019-10-13T16:02:20.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:20.372+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:20.374365+0000) 2019-10-13T16:02:20.379 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:20.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234559 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:22.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:22.744+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:22.749146+0000) 2019-10-13T16:02:23.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:23.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:23.484445+0000) 2019-10-13T16:02:23.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:23.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:23.733518+0000) 2019-10-13T16:02:25.382 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:25.376+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.380314+0000) 2019-10-13T16:02:25.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:25.376+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.380738+0000) 2019-10-13T16:02:25.383 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:25.376+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.381351+0000) 2019-10-13T16:02:25.384 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:25.376+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.381737+0000) 2019-10-13T16:02:25.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:25.376+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.382632+0000) 2019-10-13T16:02:25.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:25.376+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.382776+0000) 2019-10-13T16:02:25.386 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:25.380+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.383710+0000) 2019-10-13T16:02:25.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:25.384+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.388749+0000) 2019-10-13T16:02:25.394 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:25.388+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234593 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:25.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:25.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.759227+0000) 2019-10-13T16:02:25.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:25.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.759249+0000) 2019-10-13T16:02:25.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:25.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:25.770077+0000) 2019-10-13T16:02:26.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:26.192+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:26.194814+0000) 2019-10-13T16:02:26.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:26.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:26.311655+0000) 2019-10-13T16:02:30.397 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:30.388+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.394677+0000) 2019-10-13T16:02:30.397 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:30.392+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.395697+0000) 2019-10-13T16:02:30.398 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:30.392+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.396446+0000) 2019-10-13T16:02:30.399 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:30.391+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.397319+0000) 2019-10-13T16:02:30.400 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:30.392+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.397594+0000) 2019-10-13T16:02:30.400 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:30.392+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.397831+0000) 2019-10-13T16:02:30.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:30.392+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.398306+0000) 2019-10-13T16:02:30.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:30.395+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:30.401557+0000) 2019-10-13T16:02:30.410 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:30.404+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234626 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:32.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:32.744+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:32.749366+0000) 2019-10-13T16:02:33.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:33.480+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:33.484646+0000) 2019-10-13T16:02:33.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:33.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:33.733707+0000) 2019-10-13T16:02:35.414 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:35.408+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.411803+0000) 2019-10-13T16:02:35.415 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:35.408+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.412112+0000) 2019-10-13T16:02:35.415 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:35.408+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.412501+0000) 2019-10-13T16:02:35.416 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:35.408+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.414078+0000) 2019-10-13T16:02:35.416 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:35.408+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.414106+0000) 2019-10-13T16:02:35.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:35.411+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.414841+0000) 2019-10-13T16:02:35.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:35.408+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.415172+0000) 2019-10-13T16:02:35.423 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:35.415+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.421122+0000) 2019-10-13T16:02:35.426 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:35.420+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234659 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:35.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:35.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.759412+0000) 2019-10-13T16:02:35.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:35.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.759478+0000) 2019-10-13T16:02:35.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:35.768+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:35.770253+0000) 2019-10-13T16:02:36.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:36.199+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:36.201271+0000) 2019-10-13T16:02:36.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:36.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:36.311812+0000) 2019-10-13T16:02:40.429 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:40.420+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.426688+0000) 2019-10-13T16:02:40.429 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:40.420+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.427074+0000) 2019-10-13T16:02:40.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:40.424+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.428394+0000) 2019-10-13T16:02:40.431 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:40.424+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.428908+0000) 2019-10-13T16:02:40.432 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:40.423+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.429802+0000) 2019-10-13T16:02:40.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:40.424+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.430307+0000) 2019-10-13T16:02:40.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:40.424+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.430947+0000) 2019-10-13T16:02:40.437 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:40.431+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:40.435581+0000) 2019-10-13T16:02:40.441 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:40.432+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234691 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:42.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:42.744+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:42.749584+0000) 2019-10-13T16:02:43.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:43.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:43.484842+0000) 2019-10-13T16:02:43.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:43.731+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:43.733904+0000) 2019-10-13T16:02:45.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:45.436+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.442387+0000) 2019-10-13T16:02:45.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:45.436+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.442844+0000) 2019-10-13T16:02:45.449 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:45.440+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.443416+0000) 2019-10-13T16:02:45.450 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:45.440+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.443997+0000) 2019-10-13T16:02:45.450 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:45.440+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.444451+0000) 2019-10-13T16:02:45.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:45.439+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.445638+0000) 2019-10-13T16:02:45.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:45.440+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.445031+0000) 2019-10-13T16:02:45.459 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:45.448+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234725 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:45.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:45.451+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.455587+0000) 2019-10-13T16:02:45.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:45.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.759630+0000) 2019-10-13T16:02:45.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:45.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.759666+0000) 2019-10-13T16:02:45.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:45.764+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:45.770436+0000) 2019-10-13T16:02:46.210 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:46.203+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:46.207620+0000) 2019-10-13T16:02:46.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:46.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:46.312038+0000) 2019-10-13T16:02:47.927 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:02:47.927 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:02:47.930 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:02:47.931 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:02:50.460 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:50.452+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.457659+0000) 2019-10-13T16:02:50.460 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:50.451+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.458065+0000) 2019-10-13T16:02:50.461 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:50.452+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.458868+0000) 2019-10-13T16:02:50.461 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:50.452+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.459089+0000) 2019-10-13T16:02:50.462 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:50.456+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.460338+0000) 2019-10-13T16:02:50.463 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:50.456+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.460304+0000) 2019-10-13T16:02:50.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:50.456+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.461539+0000) 2019-10-13T16:02:50.470 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:50.463+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:50.467759+0000) 2019-10-13T16:02:50.474 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:50.464+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:52.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:52.748+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:52.749809+0000) 2019-10-13T16:02:53.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:53.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:53.485058+0000) 2019-10-13T16:02:53.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:53.727+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:53.734094+0000) 2019-10-13T16:02:55.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:55.468+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.473410+0000) 2019-10-13T16:02:55.477 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:02:55.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.474641+0000) 2019-10-13T16:02:55.477 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:55.468+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.474975+0000) 2019-10-13T16:02:55.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:55.472+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.476120+0000) 2019-10-13T16:02:55.479 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:02:55.471+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.476475+0000) 2019-10-13T16:02:55.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:02:55.472+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.476663+0000) 2019-10-13T16:02:55.480 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:55.472+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.477028+0000) 2019-10-13T16:02:55.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:55.475+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.481857+0000) 2019-10-13T16:02:55.488 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:02:55.480+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234791 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:02:55.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:02:55.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.759847+0000) 2019-10-13T16:02:55.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:02:55.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.759852+0000) 2019-10-13T16:02:55.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:02:55.768+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:55.770633+0000) 2019-10-13T16:02:56.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:02:56.211+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:56.214005+0000) 2019-10-13T16:02:56.314 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:02:56.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:02:56.312233+0000) 2019-10-13T16:03:00.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:00.483+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.489575+0000) 2019-10-13T16:03:00.492 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:00.483+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.490170+0000) 2019-10-13T16:03:00.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:00.483+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.490296+0000) 2019-10-13T16:03:00.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:00.487+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.491655+0000) 2019-10-13T16:03:00.494 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:00.487+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.491892+0000) 2019-10-13T16:03:00.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:00.487+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.492286+0000) 2019-10-13T16:03:00.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:00.487+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.492882+0000) 2019-10-13T16:03:00.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:00.491+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:00.497580+0000) 2019-10-13T16:03:00.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:00.495+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234823 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:02.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:02.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:02.750007+0000) 2019-10-13T16:03:03.487 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:03.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:03.485260+0000) 2019-10-13T16:03:03.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:03.731+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:03.734337+0000) 2019-10-13T16:03:05.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:05.503+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.506199+0000) 2019-10-13T16:03:05.509 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:05.499+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.505365+0000) 2019-10-13T16:03:05.510 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:05.499+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.507111+0000) 2019-10-13T16:03:05.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:05.499+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.507120+0000) 2019-10-13T16:03:05.510 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:05.503+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.507587+0000) 2019-10-13T16:03:05.511 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:05.503+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.507900+0000) 2019-10-13T16:03:05.511 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:05.503+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.508961+0000) 2019-10-13T16:03:05.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:05.511+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.515993+0000) 2019-10-13T16:03:05.520 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:05.511+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234857 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:05.762 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:05.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.759981+0000) 2019-10-13T16:03:05.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:05.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.760046+0000) 2019-10-13T16:03:05.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:05.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:05.770843+0000) 2019-10-13T16:03:06.223 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:06.219+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:06.220625+0000) 2019-10-13T16:03:06.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:06.311+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:06.312448+0000) 2019-10-13T16:03:10.539 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:10.515+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.521176+0000) 2019-10-13T16:03:10.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:10.515+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.521502+0000) 2019-10-13T16:03:10.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:10.515+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.522438+0000) 2019-10-13T16:03:10.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:10.519+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.522935+0000) 2019-10-13T16:03:10.542 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:10.515+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.523200+0000) 2019-10-13T16:03:10.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:10.527+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.530629+0000) 2019-10-13T16:03:10.553 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:10.519+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.523789+0000) 2019-10-13T16:03:10.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:10.519+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:10.524109+0000) 2019-10-13T16:03:10.554 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:10.527+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234888 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:12.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:12.747+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:12.750204+0000) 2019-10-13T16:03:13.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:13.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:13.485457+0000) 2019-10-13T16:03:13.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:13.731+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:13.734530+0000) 2019-10-13T16:03:15.538 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:15.531+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.536232+0000) 2019-10-13T16:03:15.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:15.531+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.536999+0000) 2019-10-13T16:03:15.539 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:15.531+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.536791+0000) 2019-10-13T16:03:15.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:15.531+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.537937+0000) 2019-10-13T16:03:15.540 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:15.531+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.538231+0000) 2019-10-13T16:03:15.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:15.531+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.539182+0000) 2019-10-13T16:03:15.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:15.535+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.539952+0000) 2019-10-13T16:03:15.548 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:15.543+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.546616+0000) 2019-10-13T16:03:15.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:15.543+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:15.762 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:15.755+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.760181+0000) 2019-10-13T16:03:15.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:15.755+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.760168+0000) 2019-10-13T16:03:15.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:15.767+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:15.771031+0000) 2019-10-13T16:03:16.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:16.223+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:16.227013+0000) 2019-10-13T16:03:16.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:16.311+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:16.312640+0000) 2019-10-13T16:03:18.022 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:03:18.022 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:03:18.026 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:03:18.026 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:03:20.554 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:20.547+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.551766+0000) 2019-10-13T16:03:20.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:20.547+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.553278+0000) 2019-10-13T16:03:20.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:20.547+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.553682+0000) 2019-10-13T16:03:20.556 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:20.551+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.554417+0000) 2019-10-13T16:03:20.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:20.547+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.554701+0000) 2019-10-13T16:03:20.557 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:20.547+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.555024+0000) 2019-10-13T16:03:20.559 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:20.551+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.555762+0000) 2019-10-13T16:03:20.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:20.555+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:20.560539+0000) 2019-10-13T16:03:20.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:20.559+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234952 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:22.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:22.743+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:22.750438+0000) 2019-10-13T16:03:23.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:23.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:23.485664+0000) 2019-10-13T16:03:23.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:23.730+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:23.734689+0000) 2019-10-13T16:03:25.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:25.559+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.567159+0000) 2019-10-13T16:03:25.570 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:25.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.567821+0000) 2019-10-13T16:03:25.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:25.563+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.567604+0000) 2019-10-13T16:03:25.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:25.563+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.568399+0000) 2019-10-13T16:03:25.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:25.563+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.568567+0000) 2019-10-13T16:03:25.571 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:25.563+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.569322+0000) 2019-10-13T16:03:25.572 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:25.563+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.569794+0000) 2019-10-13T16:03:25.578 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:25.570+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.576388+0000) 2019-10-13T16:03:25.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:25.571+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 234986 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:25.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:25.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.760371+0000) 2019-10-13T16:03:25.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:25.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.760365+0000) 2019-10-13T16:03:25.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:25.763+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:25.771177+0000) 2019-10-13T16:03:26.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:26.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:26.233365+0000) 2019-10-13T16:03:26.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:26.307+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:26.312830+0000) 2019-10-13T16:03:30.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:30.575+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.581994+0000) 2019-10-13T16:03:30.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:30.575+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.582948+0000) 2019-10-13T16:03:30.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:30.579+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.583732+0000) 2019-10-13T16:03:30.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:30.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.584208+0000) 2019-10-13T16:03:30.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:30.579+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.584643+0000) 2019-10-13T16:03:30.587 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:30.579+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.584880+0000) 2019-10-13T16:03:30.587 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:30.579+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.585238+0000) 2019-10-13T16:03:30.591 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:30.582+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:30.588736+0000) 2019-10-13T16:03:30.596 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:30.587+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:32.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:32.747+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:32.750671+0000) 2019-10-13T16:03:33.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:33.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:33.485866+0000) 2019-10-13T16:03:33.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:33.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:33.734885+0000) 2019-10-13T16:03:35.599 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:35.595+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.596882+0000) 2019-10-13T16:03:35.600 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:35.590+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.597962+0000) 2019-10-13T16:03:35.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:35.595+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.599005+0000) 2019-10-13T16:03:35.601 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:35.599+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.599395+0000) 2019-10-13T16:03:35.602 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:35.599+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.599997+0000) 2019-10-13T16:03:35.603 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:35.599+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.601264+0000) 2019-10-13T16:03:35.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:35.599+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.601524+0000) 2019-10-13T16:03:35.609 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:35.602+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.606646+0000) 2019-10-13T16:03:35.611 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:35.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235051 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:35.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:35.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.760586+0000) 2019-10-13T16:03:35.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:35.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.760586+0000) 2019-10-13T16:03:35.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:35.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:35.771370+0000) 2019-10-13T16:03:36.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:36.238+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:36.240345+0000) 2019-10-13T16:03:36.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:36.311+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:36.313010+0000) 2019-10-13T16:03:40.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:40.611+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.614463+0000) 2019-10-13T16:03:40.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:40.611+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.614862+0000) 2019-10-13T16:03:40.617 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:40.611+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.615270+0000) 2019-10-13T16:03:40.618 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:40.615+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.615681+0000) 2019-10-13T16:03:40.618 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:40.615+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.615977+0000) 2019-10-13T16:03:40.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:40.610+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.616372+0000) 2019-10-13T16:03:40.619 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:40.615+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.616905+0000) 2019-10-13T16:03:40.626 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:40.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235082 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:40.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:40.622+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:40.626501+0000) 2019-10-13T16:03:42.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:42.747+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:42.750902+0000) 2019-10-13T16:03:43.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:43.483+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:43.486066+0000) 2019-10-13T16:03:43.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:43.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:43.735080+0000) 2019-10-13T16:03:45.630 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:45.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.627585+0000) 2019-10-13T16:03:45.630 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:45.622+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.627968+0000) 2019-10-13T16:03:45.631 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:45.626+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.629040+0000) 2019-10-13T16:03:45.632 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:45.626+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.629680+0000) 2019-10-13T16:03:45.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:45.626+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.629748+0000) 2019-10-13T16:03:45.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:45.626+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.629961+0000) 2019-10-13T16:03:45.633 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:45.626+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.630306+0000) 2019-10-13T16:03:45.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:45.630+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.637611+0000) 2019-10-13T16:03:45.640 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:45.634+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:45.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:45.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.760707+0000) 2019-10-13T16:03:45.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:45.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.760749+0000) 2019-10-13T16:03:45.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:45.770+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:45.771553+0000) 2019-10-13T16:03:46.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:46.250+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:46.251798+0000) 2019-10-13T16:03:46.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:46.310+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:46.313194+0000) 2019-10-13T16:03:48.081 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:03:48.081 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:03:48.085 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:03:48.085 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:03:50.644 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:50.638+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.641812+0000) 2019-10-13T16:03:50.645 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:50.638+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.642354+0000) 2019-10-13T16:03:50.645 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:50.638+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.643085+0000) 2019-10-13T16:03:50.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:50.642+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.643677+0000) 2019-10-13T16:03:50.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:50.638+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.644025+0000) 2019-10-13T16:03:50.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:50.642+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.644223+0000) 2019-10-13T16:03:50.647 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:50.642+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.644819+0000) 2019-10-13T16:03:50.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:50.646+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:50.651510+0000) 2019-10-13T16:03:50.655 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:50.650+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235148 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:52.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:52.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:52.751100+0000) 2019-10-13T16:03:53.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:53.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:53.486285+0000) 2019-10-13T16:03:53.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:53.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:53.735276+0000) 2019-10-13T16:03:55.658 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:55.654+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.656183+0000) 2019-10-13T16:03:55.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:55.654+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.656205+0000) 2019-10-13T16:03:55.660 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:55.654+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.658175+0000) 2019-10-13T16:03:55.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:03:55.654+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.658609+0000) 2019-10-13T16:03:55.661 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:03:55.654+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.659006+0000) 2019-10-13T16:03:55.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:03:55.654+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.659618+0000) 2019-10-13T16:03:55.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:55.658+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.660152+0000) 2019-10-13T16:03:55.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:55.658+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.663021+0000) 2019-10-13T16:03:55.670 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:03:55.666+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:03:55.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:03:55.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.760917+0000) 2019-10-13T16:03:55.763 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:03:55.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.760922+0000) 2019-10-13T16:03:55.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:03:55.770+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:55.771749+0000) 2019-10-13T16:03:56.260 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:03:56.254+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:56.258197+0000) 2019-10-13T16:03:56.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:03:56.310+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:03:56.313385+0000) 2019-10-13T16:04:00.674 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:00.670+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.671938+0000) 2019-10-13T16:04:00.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:00.670+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.672254+0000) 2019-10-13T16:04:00.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:00.670+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.673141+0000) 2019-10-13T16:04:00.675 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:00.670+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.673452+0000) 2019-10-13T16:04:00.676 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:00.666+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.674054+0000) 2019-10-13T16:04:00.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:00.670+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.673750+0000) 2019-10-13T16:04:00.677 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:00.670+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.674325+0000) 2019-10-13T16:04:00.684 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:00.674+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:00.681794+0000) 2019-10-13T16:04:00.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:00.678+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:02.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:02.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:02.751343+0000) 2019-10-13T16:04:03.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:03.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:03.486479+0000) 2019-10-13T16:04:03.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:03.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:03.735486+0000) 2019-10-13T16:04:05.690 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:05.682+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.687877+0000) 2019-10-13T16:04:05.691 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:05.686+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.688367+0000) 2019-10-13T16:04:05.691 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:05.686+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.688629+0000) 2019-10-13T16:04:05.691 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:05.686+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.689070+0000) 2019-10-13T16:04:05.692 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:05.686+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.689516+0000) 2019-10-13T16:04:05.692 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:05.686+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.689744+0000) 2019-10-13T16:04:05.693 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:05.686+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.690063+0000) 2019-10-13T16:04:05.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:05.690+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.694262+0000) 2019-10-13T16:04:05.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:05.694+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235245 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:05.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:05.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.761116+0000) 2019-10-13T16:04:05.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:05.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.761108+0000) 2019-10-13T16:04:05.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:05.770+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:05.771943+0000) 2019-10-13T16:04:06.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:06.262+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:06.264416+0000) 2019-10-13T16:04:06.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:06.310+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:06.313633+0000) 2019-10-13T16:04:10.703 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:10.698+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.701170+0000) 2019-10-13T16:04:10.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:10.698+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.702253+0000) 2019-10-13T16:04:10.705 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:10.698+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.702716+0000) 2019-10-13T16:04:10.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:10.698+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.702954+0000) 2019-10-13T16:04:10.706 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:10.702+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.703280+0000) 2019-10-13T16:04:10.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:10.702+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.703843+0000) 2019-10-13T16:04:10.706 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:10.702+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.704133+0000) 2019-10-13T16:04:10.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:10.706+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:10.710880+0000) 2019-10-13T16:04:10.715 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:10.710+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:12.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:12.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:12.751499+0000) 2019-10-13T16:04:13.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:13.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:13.486677+0000) 2019-10-13T16:04:13.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:13.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:13.735679+0000) 2019-10-13T16:04:15.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:15.714+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.717071+0000) 2019-10-13T16:04:15.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:15.714+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.717496+0000) 2019-10-13T16:04:15.720 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:15.714+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.718278+0000) 2019-10-13T16:04:15.721 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:15.718+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.719062+0000) 2019-10-13T16:04:15.722 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:15.718+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.719632+0000) 2019-10-13T16:04:15.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:15.718+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.719617+0000) 2019-10-13T16:04:15.722 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:15.718+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.720007+0000) 2019-10-13T16:04:15.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:15.722+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.725236+0000) 2019-10-13T16:04:15.731 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:15.726+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235309 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:15.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:15.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.761314+0000) 2019-10-13T16:04:15.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:15.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.761312+0000) 2019-10-13T16:04:15.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:15.770+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:15.772136+0000) 2019-10-13T16:04:16.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:16.270+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:16.270848+0000) 2019-10-13T16:04:16.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:16.310+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:16.313827+0000) 2019-10-13T16:04:18.128 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:04:18.129 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:04:18.132 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:04:18.132 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:04:20.735 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:20.730+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.732546+0000) 2019-10-13T16:04:20.736 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:20.729+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.733225+0000) 2019-10-13T16:04:20.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:20.730+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.733001+0000) 2019-10-13T16:04:20.736 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:20.730+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.733667+0000) 2019-10-13T16:04:20.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:20.730+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.734947+0000) 2019-10-13T16:04:20.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:20.730+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.735156+0000) 2019-10-13T16:04:20.738 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:20.734+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.735828+0000) 2019-10-13T16:04:20.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:20.737+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:20.740875+0000) 2019-10-13T16:04:20.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:20.742+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235340 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:22.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:22.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:22.751733+0000) 2019-10-13T16:04:23.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:23.482+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:23.486871+0000) 2019-10-13T16:04:23.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:23.733+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:23.735872+0000) 2019-10-13T16:04:25.749 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:25.746+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.747280+0000) 2019-10-13T16:04:25.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:25.745+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.748010+0000) 2019-10-13T16:04:25.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:25.746+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.748649+0000) 2019-10-13T16:04:25.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:25.746+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.749163+0000) 2019-10-13T16:04:25.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:25.746+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.749496+0000) 2019-10-13T16:04:25.752 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:25.746+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.749907+0000) 2019-10-13T16:04:25.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:25.746+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.749599+0000) 2019-10-13T16:04:25.757 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:25.753+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.755073+0000) 2019-10-13T16:04:25.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:25.754+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:25.763 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:25.758+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.761441+0000) 2019-10-13T16:04:25.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:25.758+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.761465+0000) 2019-10-13T16:04:25.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:25.770+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:25.772303+0000) 2019-10-13T16:04:26.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:26.273+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:26.277364+0000) 2019-10-13T16:04:26.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:26.310+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:26.314073+0000) 2019-10-13T16:04:30.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:30.757+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.761419+0000) 2019-10-13T16:04:30.764 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:30.757+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.762499+0000) 2019-10-13T16:04:30.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:30.757+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.763035+0000) 2019-10-13T16:04:30.766 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:30.761+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.763731+0000) 2019-10-13T16:04:30.766 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:30.761+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.764426+0000) 2019-10-13T16:04:30.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:30.761+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.764009+0000) 2019-10-13T16:04:30.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:30.761+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.765176+0000) 2019-10-13T16:04:30.772 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:30.765+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:30.769822+0000) 2019-10-13T16:04:30.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:30.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235405 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:32.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:32.749+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:32.751970+0000) 2019-10-13T16:04:33.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:33.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:33.487068+0000) 2019-10-13T16:04:33.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:33.733+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:33.736075+0000) 2019-10-13T16:04:35.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:35.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.761626+0000) 2019-10-13T16:04:35.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:35.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.762244+0000) 2019-10-13T16:04:35.774 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:35.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.772511+0000) 2019-10-13T16:04:35.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:35.773+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.777092+0000) 2019-10-13T16:04:35.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:35.773+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.778221+0000) 2019-10-13T16:04:35.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:35.773+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.778484+0000) 2019-10-13T16:04:35.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:35.773+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.778951+0000) 2019-10-13T16:04:35.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:35.777+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.779627+0000) 2019-10-13T16:04:35.782 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:35.777+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.779419+0000) 2019-10-13T16:04:35.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:35.777+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.780358+0000) 2019-10-13T16:04:35.786 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:35.781+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:35.783995+0000) 2019-10-13T16:04:35.790 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:35.785+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235437 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:36.289 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:36.285+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:36.287431+0000) 2019-10-13T16:04:36.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:36.313+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:36.314262+0000) 2019-10-13T16:04:40.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:40.789+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.791532+0000) 2019-10-13T16:04:40.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:40.789+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.791936+0000) 2019-10-13T16:04:40.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:40.789+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.792395+0000) 2019-10-13T16:04:40.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:40.789+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.792808+0000) 2019-10-13T16:04:40.795 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:40.789+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.793521+0000) 2019-10-13T16:04:40.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:40.789+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.794135+0000) 2019-10-13T16:04:40.797 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:40.793+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.794753+0000) 2019-10-13T16:04:40.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:40.797+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:40.801338+0000) 2019-10-13T16:04:40.807 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:40.801+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235468 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:42.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:42.749+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:42.752190+0000) 2019-10-13T16:04:43.489 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:43.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:43.487327+0000) 2019-10-13T16:04:43.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:43.733+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:43.736247+0000) 2019-10-13T16:04:45.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:45.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.761826+0000) 2019-10-13T16:04:45.764 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:45.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.762399+0000) 2019-10-13T16:04:45.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:45.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.772742+0000) 2019-10-13T16:04:45.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:45.805+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.808210+0000) 2019-10-13T16:04:45.812 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:45.805+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.809123+0000) 2019-10-13T16:04:45.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:45.805+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.808498+0000) 2019-10-13T16:04:45.812 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:45.805+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.809644+0000) 2019-10-13T16:04:45.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:45.805+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.810559+0000) 2019-10-13T16:04:45.813 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:45.805+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.810492+0000) 2019-10-13T16:04:45.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:45.805+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.811155+0000) 2019-10-13T16:04:45.818 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:45.813+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:45.816329+0000) 2019-10-13T16:04:45.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:45.817+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235500 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:46.296 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:46.289+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:46.293859+0000) 2019-10-13T16:04:46.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:46.309+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:46.314448+0000) 2019-10-13T16:04:48.169 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:04:48.170 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:04:48.173 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:04:48.173 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:04:50.826 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:50.821+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.823512+0000) 2019-10-13T16:04:50.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:50.821+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.823885+0000) 2019-10-13T16:04:50.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:50.821+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.824240+0000) 2019-10-13T16:04:50.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:50.821+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.824841+0000) 2019-10-13T16:04:50.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:50.821+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.825152+0000) 2019-10-13T16:04:50.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:50.825+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.826434+0000) 2019-10-13T16:04:50.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:50.821+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.826807+0000) 2019-10-13T16:04:50.837 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:50.829+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235533 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:50.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:50.833+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:50.834521+0000) 2019-10-13T16:04:52.754 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:52.749+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:52.752388+0000) 2019-10-13T16:04:53.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:53.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:53.487534+0000) 2019-10-13T16:04:53.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:53.733+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:53.736407+0000) 2019-10-13T16:04:55.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:55.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.762030+0000) 2019-10-13T16:04:55.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:55.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.762553+0000) 2019-10-13T16:04:55.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:55.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.772943+0000) 2019-10-13T16:04:55.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:04:55.833+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.837887+0000) 2019-10-13T16:04:55.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:55.833+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.838501+0000) 2019-10-13T16:04:55.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:04:55.833+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.839071+0000) 2019-10-13T16:04:55.841 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:04:55.837+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.839469+0000) 2019-10-13T16:04:55.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:04:55.837+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.839988+0000) 2019-10-13T16:04:55.842 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:04:55.837+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.840205+0000) 2019-10-13T16:04:55.843 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:04:55.837+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.840994+0000) 2019-10-13T16:04:55.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:55.845+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:55.847423+0000) 2019-10-13T16:04:55.852 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:04:55.845+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235564 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:04:56.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:04:56.297+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:56.300289+0000) 2019-10-13T16:04:56.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:04:56.313+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:04:56.314657+0000) 2019-10-13T16:05:00.855 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:00.849+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.853313+0000) 2019-10-13T16:05:00.856 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:00.849+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.853660+0000) 2019-10-13T16:05:00.857 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:00.849+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.854830+0000) 2019-10-13T16:05:00.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:00.853+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.855427+0000) 2019-10-13T16:05:00.858 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:00.853+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.855764+0000) 2019-10-13T16:05:00.858 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:00.853+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.856111+0000) 2019-10-13T16:05:00.859 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:00.853+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.856699+0000) 2019-10-13T16:05:00.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:00.861+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:00.862916+0000) 2019-10-13T16:05:00.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:00.861+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235596 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:02.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:02.749+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:02.752582+0000) 2019-10-13T16:05:03.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:03.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:03.487755+0000) 2019-10-13T16:05:03.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:03.733+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:03.736599+0000) 2019-10-13T16:05:05.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:05.757+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.762277+0000) 2019-10-13T16:05:05.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:05.757+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.762711+0000) 2019-10-13T16:05:05.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:05.769+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.773140+0000) 2019-10-13T16:05:05.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:05.861+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.867138+0000) 2019-10-13T16:05:05.871 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:05.865+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.868962+0000) 2019-10-13T16:05:05.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:05.865+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.868843+0000) 2019-10-13T16:05:05.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:05.865+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.870181+0000) 2019-10-13T16:05:05.872 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:05.865+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.870443+0000) 2019-10-13T16:05:05.873 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:05.865+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.870960+0000) 2019-10-13T16:05:05.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:05.869+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.871419+0000) 2019-10-13T16:05:05.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:05.873+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:05.876259+0000) 2019-10-13T16:05:05.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:05.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235629 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:06.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:06.305+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:06.306750+0000) 2019-10-13T16:05:06.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:06.309+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:06.314843+0000) 2019-10-13T16:05:10.885 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:10.877+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.882497+0000) 2019-10-13T16:05:10.886 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:10.881+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.883736+0000) 2019-10-13T16:05:10.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:10.881+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.884172+0000) 2019-10-13T16:05:10.887 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:10.881+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.884646+0000) 2019-10-13T16:05:10.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:10.881+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.885166+0000) 2019-10-13T16:05:10.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:10.881+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.885331+0000) 2019-10-13T16:05:10.888 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:10.881+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.885651+0000) 2019-10-13T16:05:10.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:10.889+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:10.890170+0000) 2019-10-13T16:05:10.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:10.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235661 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:12.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:12.748+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:12.752789+0000) 2019-10-13T16:05:13.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:13.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:13.487989+0000) 2019-10-13T16:05:13.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:13.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:13.736810+0000) 2019-10-13T16:05:15.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:15.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.762516+0000) 2019-10-13T16:05:15.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:15.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.762863+0000) 2019-10-13T16:05:15.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:15.772+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.773335+0000) 2019-10-13T16:05:15.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:15.892+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.897687+0000) 2019-10-13T16:05:15.901 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:15.892+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.898670+0000) 2019-10-13T16:05:15.902 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:15.896+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.899042+0000) 2019-10-13T16:05:15.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:15.896+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.899369+0000) 2019-10-13T16:05:15.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:15.896+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.899887+0000) 2019-10-13T16:05:15.902 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:15.896+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.899911+0000) 2019-10-13T16:05:15.903 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:15.896+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.900698+0000) 2019-10-13T16:05:15.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:15.904+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:15.906533+0000) 2019-10-13T16:05:15.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:15.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235692 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:16.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:16.308+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:16.315061+0000) 2019-10-13T16:05:16.318 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:16.312+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:16.315574+0000) 2019-10-13T16:05:18.226 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:05:18.226 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:05:18.230 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:05:18.230 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:05:20.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:20.908+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.913242+0000) 2019-10-13T16:05:20.916 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:20.908+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.913559+0000) 2019-10-13T16:05:20.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:20.908+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.914372+0000) 2019-10-13T16:05:20.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:20.912+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.914691+0000) 2019-10-13T16:05:20.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:20.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.915034+0000) 2019-10-13T16:05:20.918 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:20.908+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.915257+0000) 2019-10-13T16:05:20.918 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:20.912+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.915541+0000) 2019-10-13T16:05:20.925 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:20.920+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:20.922743+0000) 2019-10-13T16:05:20.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:20.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:22.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:22.752+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:22.752997+0000) 2019-10-13T16:05:23.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:23.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:23.488180+0000) 2019-10-13T16:05:23.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:23.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:23.737009+0000) 2019-10-13T16:05:25.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:25.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.762703+0000) 2019-10-13T16:05:25.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:25.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.763015+0000) 2019-10-13T16:05:25.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:25.768+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.773530+0000) 2019-10-13T16:05:25.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:25.924+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.928769+0000) 2019-10-13T16:05:25.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:25.924+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.928860+0000) 2019-10-13T16:05:25.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:25.924+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.929195+0000) 2019-10-13T16:05:25.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:25.924+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.930307+0000) 2019-10-13T16:05:25.933 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:25.924+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.930989+0000) 2019-10-13T16:05:25.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:25.928+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.931868+0000) 2019-10-13T16:05:25.934 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:25.928+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.932512+0000) 2019-10-13T16:05:25.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:25.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235757 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:25.945 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:25.940+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:25.942670+0000) 2019-10-13T16:05:26.317 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:26.312+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:26.315283+0000) 2019-10-13T16:05:26.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:26.316+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:26.322061+0000) 2019-10-13T16:05:30.947 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:30.940+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.945075+0000) 2019-10-13T16:05:30.948 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:30.940+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.945589+0000) 2019-10-13T16:05:30.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:30.940+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.947190+0000) 2019-10-13T16:05:30.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:30.944+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.947608+0000) 2019-10-13T16:05:30.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:30.944+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.948071+0000) 2019-10-13T16:05:30.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:30.944+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.948567+0000) 2019-10-13T16:05:30.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:30.944+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.948969+0000) 2019-10-13T16:05:30.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:30.948+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:30.952970+0000) 2019-10-13T16:05:30.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:30.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235789 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:32.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:32.752+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:32.753201+0000) 2019-10-13T16:05:33.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:33.484+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:33.488367+0000) 2019-10-13T16:05:33.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:33.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:33.737201+0000) 2019-10-13T16:05:35.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:35.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.762927+0000) 2019-10-13T16:05:35.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:35.756+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.763196+0000) 2019-10-13T16:05:35.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:35.772+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.773733+0000) 2019-10-13T16:05:35.964 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:35.956+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.961552+0000) 2019-10-13T16:05:35.964 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:35.956+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.962098+0000) 2019-10-13T16:05:35.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:35.960+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.962491+0000) 2019-10-13T16:05:35.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:35.956+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.962854+0000) 2019-10-13T16:05:35.965 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:35.956+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.963255+0000) 2019-10-13T16:05:35.966 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:35.960+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.963440+0000) 2019-10-13T16:05:35.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:35.960+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.965093+0000) 2019-10-13T16:05:35.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:35.968+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:35.972765+0000) 2019-10-13T16:05:35.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:35.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235820 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:36.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:36.312+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:36.315521+0000) 2019-10-13T16:05:36.331 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:36.324+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:36.328423+0000) 2019-10-13T16:05:40.979 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:40.972+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.977134+0000) 2019-10-13T16:05:40.980 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:40.972+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.977696+0000) 2019-10-13T16:05:40.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:40.972+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.977937+0000) 2019-10-13T16:05:40.981 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:40.976+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.978878+0000) 2019-10-13T16:05:40.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:40.972+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.979102+0000) 2019-10-13T16:05:40.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:40.976+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.979473+0000) 2019-10-13T16:05:40.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:40.976+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.979765+0000) 2019-10-13T16:05:40.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:40.984+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:40.986831+0000) 2019-10-13T16:05:40.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:40.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235852 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:42.755 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:42.748+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:42.753419+0000) 2019-10-13T16:05:43.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:43.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:43.488568+0000) 2019-10-13T16:05:43.739 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:43.736+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:43.737401+0000) 2019-10-13T16:05:45.765 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:45.756+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.763125+0000) 2019-10-13T16:05:45.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:45.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.763419+0000) 2019-10-13T16:05:45.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:45.768+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.773930+0000) 2019-10-13T16:05:45.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:45.988+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.993142+0000) 2019-10-13T16:05:45.996 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:45.988+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.993768+0000) 2019-10-13T16:05:45.996 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:45.988+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.994001+0000) 2019-10-13T16:05:45.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:45.988+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.994845+0000) 2019-10-13T16:05:45.997 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:45.988+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.995105+0000) 2019-10-13T16:05:45.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:45.992+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.995879+0000) 2019-10-13T16:05:45.998 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:45.992+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:45.995631+0000) 2019-10-13T16:05:46.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:45.996+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:46.000872+0000) 2019-10-13T16:05:46.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:46.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235885 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:46.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:46.312+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:46.315705+0000) 2019-10-13T16:05:46.337 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:46.332+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:46.334824+0000) 2019-10-13T16:05:48.275 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:05:48.275 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:05:48.279 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:05:48.279 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:05:51.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:51.004+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.008342+0000) 2019-10-13T16:05:51.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:51.004+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.008742+0000) 2019-10-13T16:05:51.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:51.004+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.009131+0000) 2019-10-13T16:05:51.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:51.004+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.009504+0000) 2019-10-13T16:05:51.012 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:51.004+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.010383+0000) 2019-10-13T16:05:51.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:51.004+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.011213+0000) 2019-10-13T16:05:51.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:51.008+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.011485+0000) 2019-10-13T16:05:51.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:51.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:51.017090+0000) 2019-10-13T16:05:51.021 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:51.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:52.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:52.752+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:52.753649+0000) 2019-10-13T16:05:53.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:53.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:53.488766+0000) 2019-10-13T16:05:53.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:53.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:53.737608+0000) 2019-10-13T16:05:55.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:55.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:55.763350+0000) 2019-10-13T16:05:55.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:55.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:55.763648+0000) 2019-10-13T16:05:55.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:55.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:55.774127+0000) 2019-10-13T16:05:56.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:05:56.015+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.022665+0000) 2019-10-13T16:05:56.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:05:56.019+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.023824+0000) 2019-10-13T16:05:56.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:56.019+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.023747+0000) 2019-10-13T16:05:56.027 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:05:56.020+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.024644+0000) 2019-10-13T16:05:56.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:05:56.019+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.026035+0000) 2019-10-13T16:05:56.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:05:56.019+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.026402+0000) 2019-10-13T16:05:56.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:05:56.019+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.026963+0000) 2019-10-13T16:05:56.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:56.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.032037+0000) 2019-10-13T16:05:56.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:05:56.027+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235948 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:05:56.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:05:56.311+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.315889+0000) 2019-10-13T16:05:56.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:05:56.340+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:05:56.341106+0000) 2019-10-13T16:06:01.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:01.031+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.038042+0000) 2019-10-13T16:06:01.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:01.031+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.038485+0000) 2019-10-13T16:06:01.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:01.031+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.039189+0000) 2019-10-13T16:06:01.060 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:01.035+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.040130+0000) 2019-10-13T16:06:01.060 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:01.035+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.040410+0000) 2019-10-13T16:06:01.061 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:01.036+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.041009+0000) 2019-10-13T16:06:01.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:01.044+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.046240+0000) 2019-10-13T16:06:01.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:01.035+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:01.041555+0000) 2019-10-13T16:06:01.075 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:01.043+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 235980 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:02.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:02.751+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:02.753851+0000) 2019-10-13T16:06:03.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:03.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:03.488979+0000) 2019-10-13T16:06:03.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:03.732+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:03.737796+0000) 2019-10-13T16:06:05.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:05.759+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:05.763547+0000) 2019-10-13T16:06:05.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:05.759+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:05.763885+0000) 2019-10-13T16:06:05.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:05.767+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:05.774323+0000) 2019-10-13T16:06:06.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:06.047+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.053300+0000) 2019-10-13T16:06:06.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:06.047+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.053632+0000) 2019-10-13T16:06:06.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:06.047+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.053940+0000) 2019-10-13T16:06:06.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:06.047+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.054215+0000) 2019-10-13T16:06:06.058 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:06.047+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.054796+0000) 2019-10-13T16:06:06.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:06.051+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.055815+0000) 2019-10-13T16:06:06.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:06.052+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.055280+0000) 2019-10-13T16:06:06.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:06.059+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:06.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:06.060+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.064848+0000) 2019-10-13T16:06:06.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:06.315+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.316069+0000) 2019-10-13T16:06:06.350 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:06.344+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:06.347553+0000) 2019-10-13T16:06:11.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:11.063+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.067549+0000) 2019-10-13T16:06:11.070 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:11.063+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.067855+0000) 2019-10-13T16:06:11.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:11.063+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.068153+0000) 2019-10-13T16:06:11.071 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:11.063+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.068682+0000) 2019-10-13T16:06:11.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:11.063+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.069733+0000) 2019-10-13T16:06:11.072 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:11.063+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.070004+0000) 2019-10-13T16:06:11.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:11.063+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.070318+0000) 2019-10-13T16:06:11.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:11.071+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:11.076927+0000) 2019-10-13T16:06:11.081 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:11.075+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:12.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:12.751+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:12.754054+0000) 2019-10-13T16:06:13.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:13.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:13.489127+0000) 2019-10-13T16:06:13.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:13.731+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:13.738026+0000) 2019-10-13T16:06:15.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:15.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:15.763751+0000) 2019-10-13T16:06:15.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:15.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:15.764060+0000) 2019-10-13T16:06:15.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:15.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:15.774540+0000) 2019-10-13T16:06:16.085 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:16.079+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.082707+0000) 2019-10-13T16:06:16.085 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:16.079+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.083092+0000) 2019-10-13T16:06:16.086 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:16.079+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.083843+0000) 2019-10-13T16:06:16.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:16.079+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.084304+0000) 2019-10-13T16:06:16.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:16.079+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.085350+0000) 2019-10-13T16:06:16.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:16.079+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.085438+0000) 2019-10-13T16:06:16.088 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:16.079+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.086353+0000) 2019-10-13T16:06:16.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:16.091+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.094691+0000) 2019-10-13T16:06:16.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:16.087+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236076 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:16.318 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:16.311+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.316265+0000) 2019-10-13T16:06:16.356 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:16.347+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:16.354089+0000) 2019-10-13T16:06:18.563 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:06:18.564 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:06:18.567 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:06:18.567 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:06:21.100 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:21.095+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.098324+0000) 2019-10-13T16:06:21.101 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:21.091+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.098547+0000) 2019-10-13T16:06:21.101 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:21.091+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.098931+0000) 2019-10-13T16:06:21.102 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:21.095+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.099737+0000) 2019-10-13T16:06:21.102 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:21.095+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.100241+0000) 2019-10-13T16:06:21.103 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:21.095+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.100481+0000) 2019-10-13T16:06:21.103 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:21.095+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.101397+0000) 2019-10-13T16:06:21.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:21.103+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:21.106508+0000) 2019-10-13T16:06:21.112 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:21.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236107 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:22.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:22.751+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:22.754256+0000) 2019-10-13T16:06:23.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:23.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:23.489356+0000) 2019-10-13T16:06:23.740 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:23.735+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:23.738201+0000) 2019-10-13T16:06:25.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:25.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:25.763947+0000) 2019-10-13T16:06:25.766 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:25.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:25.764235+0000) 2019-10-13T16:06:25.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:25.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:25.774759+0000) 2019-10-13T16:06:26.116 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:26.107+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.113465+0000) 2019-10-13T16:06:26.117 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:26.111+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.114338+0000) 2019-10-13T16:06:26.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:26.107+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.114510+0000) 2019-10-13T16:06:26.117 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:26.107+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.114855+0000) 2019-10-13T16:06:26.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:26.111+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.115354+0000) 2019-10-13T16:06:26.119 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:26.111+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.115941+0000) 2019-10-13T16:06:26.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:26.111+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.116674+0000) 2019-10-13T16:06:26.122 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:26.115+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.120264+0000) 2019-10-13T16:06:26.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:26.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236141 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:26.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:26.315+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.316481+0000) 2019-10-13T16:06:26.363 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:26.355+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:26.360817+0000) 2019-10-13T16:06:31.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:31.127+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.127731+0000) 2019-10-13T16:06:31.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:31.123+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.128086+0000) 2019-10-13T16:06:31.132 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:31.127+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.129322+0000) 2019-10-13T16:06:31.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:31.127+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.129896+0000) 2019-10-13T16:06:31.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:31.127+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.131273+0000) 2019-10-13T16:06:31.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:31.131+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.131514+0000) 2019-10-13T16:06:31.134 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:31.131+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.131924+0000) 2019-10-13T16:06:31.139 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:31.131+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:31.136874+0000) 2019-10-13T16:06:31.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:31.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:32.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:32.751+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:32.754489+0000) 2019-10-13T16:06:33.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:33.487+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:33.489558+0000) 2019-10-13T16:06:33.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:33.735+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:33.738425+0000) 2019-10-13T16:06:35.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:35.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:35.764161+0000) 2019-10-13T16:06:35.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:35.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:35.764423+0000) 2019-10-13T16:06:35.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:35.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:35.774988+0000) 2019-10-13T16:06:36.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:36.143+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.143504+0000) 2019-10-13T16:06:36.146 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:36.139+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.144198+0000) 2019-10-13T16:06:36.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:36.143+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.144586+0000) 2019-10-13T16:06:36.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:36.143+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.144941+0000) 2019-10-13T16:06:36.147 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:36.143+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.145373+0000) 2019-10-13T16:06:36.148 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:36.143+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.145752+0000) 2019-10-13T16:06:36.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:36.143+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.146099+0000) 2019-10-13T16:06:36.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:36.147+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.150171+0000) 2019-10-13T16:06:36.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:36.151+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:36.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:36.315+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.316687+0000) 2019-10-13T16:06:36.369 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:36.363+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:36.367402+0000) 2019-10-13T16:06:41.160 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:41.154+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.158053+0000) 2019-10-13T16:06:41.161 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:41.154+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.158502+0000) 2019-10-13T16:06:41.161 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:41.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.158964+0000) 2019-10-13T16:06:41.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:41.158+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.160058+0000) 2019-10-13T16:06:41.162 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:41.158+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.160187+0000) 2019-10-13T16:06:41.163 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:41.158+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.160545+0000) 2019-10-13T16:06:41.163 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:41.158+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.161393+0000) 2019-10-13T16:06:41.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:41.159+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:41.165793+0000) 2019-10-13T16:06:41.173 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:41.166+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:42.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:42.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:42.754688+0000) 2019-10-13T16:06:43.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:43.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:43.489754+0000) 2019-10-13T16:06:43.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:43.735+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:43.738632+0000) 2019-10-13T16:06:45.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:45.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:45.764370+0000) 2019-10-13T16:06:45.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:45.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:45.764595+0000) 2019-10-13T16:06:45.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:45.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:45.775186+0000) 2019-10-13T16:06:46.176 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:46.170+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.173957+0000) 2019-10-13T16:06:46.177 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:46.170+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.174425+0000) 2019-10-13T16:06:46.177 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:46.170+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.174821+0000) 2019-10-13T16:06:46.178 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:46.171+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.175292+0000) 2019-10-13T16:06:46.178 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:46.174+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.176148+0000) 2019-10-13T16:06:46.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:46.174+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.176453+0000) 2019-10-13T16:06:46.179 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:46.174+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.177190+0000) 2019-10-13T16:06:46.185 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:46.179+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.182275+0000) 2019-10-13T16:06:46.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:46.182+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236275 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:46.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:46.314+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.316901+0000) 2019-10-13T16:06:46.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:46.371+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:46.373804+0000) 2019-10-13T16:06:48.666 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:06:48.666 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:06:48.670 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:06:48.670 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:06:51.191 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:51.186+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.188879+0000) 2019-10-13T16:06:51.191 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:51.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.189222+0000) 2019-10-13T16:06:51.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:51.186+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.190065+0000) 2019-10-13T16:06:51.192 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:51.186+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.190535+0000) 2019-10-13T16:06:51.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:51.186+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.190914+0000) 2019-10-13T16:06:51.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:51.190+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.192045+0000) 2019-10-13T16:06:51.194 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:51.190+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.192627+0000) 2019-10-13T16:06:51.204 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:51.195+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:51.201817+0000) 2019-10-13T16:06:51.204 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:51.198+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236306 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:52.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:52.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:52.754911+0000) 2019-10-13T16:06:53.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:53.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:53.489991+0000) 2019-10-13T16:06:53.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:53.735+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:53.738805+0000) 2019-10-13T16:06:55.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:55.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:55.764577+0000) 2019-10-13T16:06:55.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:55.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:55.764791+0000) 2019-10-13T16:06:55.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:55.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:55.775380+0000) 2019-10-13T16:06:56.207 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:06:56.202+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.205496+0000) 2019-10-13T16:06:56.208 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:06:56.199+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.205795+0000) 2019-10-13T16:06:56.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:06:56.202+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.206328+0000) 2019-10-13T16:06:56.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:56.202+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.206942+0000) 2019-10-13T16:06:56.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:06:56.202+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.206951+0000) 2019-10-13T16:06:56.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:06:56.206+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.207691+0000) 2019-10-13T16:06:56.210 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:06:56.206+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.207991+0000) 2019-10-13T16:06:56.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:56.211+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.214852+0000) 2019-10-13T16:06:56.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:06:56.214+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236341 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:06:56.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:06:56.314+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.317085+0000) 2019-10-13T16:06:56.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:06:56.379+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:06:56.379321+0000) 2019-10-13T16:07:01.223 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:01.218+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.221188+0000) 2019-10-13T16:07:01.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:01.218+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.222192+0000) 2019-10-13T16:07:01.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:01.218+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.222490+0000) 2019-10-13T16:07:01.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:01.218+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.223042+0000) 2019-10-13T16:07:01.225 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:01.219+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.223512+0000) 2019-10-13T16:07:01.226 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:01.222+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.223745+0000) 2019-10-13T16:07:01.226 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:01.222+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.224425+0000) 2019-10-13T16:07:01.235 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:01.230+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:01.237 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:01.231+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:01.235050+0000) 2019-10-13T16:07:02.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:02.750+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:02.755113+0000) 2019-10-13T16:07:03.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:03.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:03.490185+0000) 2019-10-13T16:07:03.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:03.735+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:03.738995+0000) 2019-10-13T16:07:05.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:05.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:05.764782+0000) 2019-10-13T16:07:05.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:05.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:05.764972+0000) 2019-10-13T16:07:05.777 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:05.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:05.775537+0000) 2019-10-13T16:07:06.239 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:06.234+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.236746+0000) 2019-10-13T16:07:06.239 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:06.234+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.237077+0000) 2019-10-13T16:07:06.240 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:06.234+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.238274+0000) 2019-10-13T16:07:06.241 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:06.234+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.238863+0000) 2019-10-13T16:07:06.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:06.234+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.239223+0000) 2019-10-13T16:07:06.242 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:06.238+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.239681+0000) 2019-10-13T16:07:06.242 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:06.238+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.240221+0000) 2019-10-13T16:07:06.249 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:06.242+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.247555+0000) 2019-10-13T16:07:06.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:06.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236409 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:06.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:06.314+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.317285+0000) 2019-10-13T16:07:06.388 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:06.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:06.385634+0000) 2019-10-13T16:07:11.254 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:11.246+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.251973+0000) 2019-10-13T16:07:11.255 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:11.250+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.252297+0000) 2019-10-13T16:07:11.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:11.250+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.252954+0000) 2019-10-13T16:07:11.255 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:11.250+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.253531+0000) 2019-10-13T16:07:11.256 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:11.250+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.254358+0000) 2019-10-13T16:07:11.257 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:11.250+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.254681+0000) 2019-10-13T16:07:11.258 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:11.254+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.255843+0000) 2019-10-13T16:07:11.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:11.254+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:11.259177+0000) 2019-10-13T16:07:11.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:11.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236439 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:12.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:12.754+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:12.755387+0000) 2019-10-13T16:07:13.492 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:13.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:13.490356+0000) 2019-10-13T16:07:13.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:13.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:13.739190+0000) 2019-10-13T16:07:15.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:15.762+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:15.764989+0000) 2019-10-13T16:07:15.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:15.762+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:15.765176+0000) 2019-10-13T16:07:15.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:15.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:15.775758+0000) 2019-10-13T16:07:16.270 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:16.266+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.267695+0000) 2019-10-13T16:07:16.270 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:16.262+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.268126+0000) 2019-10-13T16:07:16.271 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:16.266+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.268779+0000) 2019-10-13T16:07:16.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:16.266+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.269096+0000) 2019-10-13T16:07:16.272 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:16.266+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.270315+0000) 2019-10-13T16:07:16.273 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:16.266+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.270641+0000) 2019-10-13T16:07:16.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:16.270+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.271644+0000) 2019-10-13T16:07:16.277 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:16.270+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.275641+0000) 2019-10-13T16:07:16.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:16.274+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236475 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:16.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:16.314+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.317453+0000) 2019-10-13T16:07:16.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:16.390+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:16.392621+0000) 2019-10-13T16:07:18.785 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:07:18.785 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:07:18.789 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:07:18.789 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:07:21.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:21.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.282053+0000) 2019-10-13T16:07:21.285 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:21.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.282682+0000) 2019-10-13T16:07:21.285 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:21.278+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.283194+0000) 2019-10-13T16:07:21.286 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:21.282+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.283918+0000) 2019-10-13T16:07:21.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:21.282+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.284209+0000) 2019-10-13T16:07:21.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:21.282+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.284440+0000) 2019-10-13T16:07:21.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:21.282+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.284774+0000) 2019-10-13T16:07:21.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:21.286+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:21.291439+0000) 2019-10-13T16:07:21.296 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:21.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236506 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:22.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:22.754+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:22.755610+0000) 2019-10-13T16:07:23.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:23.486+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:23.490561+0000) 2019-10-13T16:07:23.741 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:23.734+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:23.739401+0000) 2019-10-13T16:07:25.767 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:25.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:25.765180+0000) 2019-10-13T16:07:25.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:25.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:25.765334+0000) 2019-10-13T16:07:25.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:25.773+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:25.776013+0000) 2019-10-13T16:07:26.300 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:26.293+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.297631+0000) 2019-10-13T16:07:26.300 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:26.293+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.298166+0000) 2019-10-13T16:07:26.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:26.293+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.298734+0000) 2019-10-13T16:07:26.301 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:26.293+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.298823+0000) 2019-10-13T16:07:26.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:26.297+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.299546+0000) 2019-10-13T16:07:26.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:26.297+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.300225+0000) 2019-10-13T16:07:26.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:26.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.301130+0000) 2019-10-13T16:07:26.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:26.302+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.307464+0000) 2019-10-13T16:07:26.311 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:26.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:26.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:26.313+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.317589+0000) 2019-10-13T16:07:26.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:26.398+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:26.399105+0000) 2019-10-13T16:07:31.315 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:31.309+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.312858+0000) 2019-10-13T16:07:31.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:31.309+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.313506+0000) 2019-10-13T16:07:31.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:31.309+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.314112+0000) 2019-10-13T16:07:31.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:31.309+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.314595+0000) 2019-10-13T16:07:31.317 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:31.309+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.314769+0000) 2019-10-13T16:07:31.317 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:31.309+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.315135+0000) 2019-10-13T16:07:31.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:31.310+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.315501+0000) 2019-10-13T16:07:31.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:31.318+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:31.323074+0000) 2019-10-13T16:07:31.326 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:31.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:32.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:32.753+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:32.755808+0000) 2019-10-13T16:07:33.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:33.485+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:33.490762+0000) 2019-10-13T16:07:33.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:33.738+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:33.739595+0000) 2019-10-13T16:07:35.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:35.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:35.765383+0000) 2019-10-13T16:07:35.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:35.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:35.765512+0000) 2019-10-13T16:07:35.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:35.773+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:35.776236+0000) 2019-10-13T16:07:36.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:36.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.317772+0000) 2019-10-13T16:07:36.330 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:36.325+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.327734+0000) 2019-10-13T16:07:36.330 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:36.325+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.328180+0000) 2019-10-13T16:07:36.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:36.325+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.328800+0000) 2019-10-13T16:07:36.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:36.322+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.329120+0000) 2019-10-13T16:07:36.332 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:36.325+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.329712+0000) 2019-10-13T16:07:36.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:36.325+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.331002+0000) 2019-10-13T16:07:36.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:36.329+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.331665+0000) 2019-10-13T16:07:36.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:36.330+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.338016+0000) 2019-10-13T16:07:36.341 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:36.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:36.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:36.402+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:36.405198+0000) 2019-10-13T16:07:41.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:41.337+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.342811+0000) 2019-10-13T16:07:41.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:41.337+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.343244+0000) 2019-10-13T16:07:41.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:41.338+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.343685+0000) 2019-10-13T16:07:41.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:41.341+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.344082+0000) 2019-10-13T16:07:41.347 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:41.341+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.345529+0000) 2019-10-13T16:07:41.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:41.341+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.345841+0000) 2019-10-13T16:07:41.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:41.341+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.346581+0000) 2019-10-13T16:07:41.352 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:41.346+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:41.350711+0000) 2019-10-13T16:07:41.357 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:41.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236640 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:42.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:42.753+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:42.756033+0000) 2019-10-13T16:07:43.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:43.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:43.490969+0000) 2019-10-13T16:07:43.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:43.738+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:43.739791+0000) 2019-10-13T16:07:45.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:45.761+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:45.765580+0000) 2019-10-13T16:07:45.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:45.761+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:45.765699+0000) 2019-10-13T16:07:45.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:45.773+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:45.776467+0000) 2019-10-13T16:07:46.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:46.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.317969+0000) 2019-10-13T16:07:46.360 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:46.353+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.357802+0000) 2019-10-13T16:07:46.360 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:46.353+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.358181+0000) 2019-10-13T16:07:46.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:46.354+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.358907+0000) 2019-10-13T16:07:46.361 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:46.357+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.359528+0000) 2019-10-13T16:07:46.362 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:46.357+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.359897+0000) 2019-10-13T16:07:46.362 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:46.357+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.360076+0000) 2019-10-13T16:07:46.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:46.357+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.361374+0000) 2019-10-13T16:07:46.370 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:46.362+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.367960+0000) 2019-10-13T16:07:46.371 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:46.365+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:46.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:46.406+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:46.411599+0000) 2019-10-13T16:07:48.880 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:07:48.880 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:07:48.885 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:07:48.885 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:07:51.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:51.369+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.373558+0000) 2019-10-13T16:07:51.377 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:51.366+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.373851+0000) 2019-10-13T16:07:51.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:51.369+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.374241+0000) 2019-10-13T16:07:51.377 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:51.369+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.374512+0000) 2019-10-13T16:07:51.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:51.369+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.374843+0000) 2019-10-13T16:07:51.378 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:51.373+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.375703+0000) 2019-10-13T16:07:51.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:51.373+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.376668+0000) 2019-10-13T16:07:51.384 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:51.378+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:51.382433+0000) 2019-10-13T16:07:51.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:51.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:52.758 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:52.753+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:52.756235+0000) 2019-10-13T16:07:53.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:53.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:53.491165+0000) 2019-10-13T16:07:53.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:53.738+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:53.740003+0000) 2019-10-13T16:07:55.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:55.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:55.765783+0000) 2019-10-13T16:07:55.768 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:55.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:55.765922+0000) 2019-10-13T16:07:55.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:55.773+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:55.776667+0000) 2019-10-13T16:07:56.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:56.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.318174+0000) 2019-10-13T16:07:56.391 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:07:56.385+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.388564+0000) 2019-10-13T16:07:56.391 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:07:56.386+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.388806+0000) 2019-10-13T16:07:56.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:07:56.385+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.389230+0000) 2019-10-13T16:07:56.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:07:56.385+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.390062+0000) 2019-10-13T16:07:56.392 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:07:56.385+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.390540+0000) 2019-10-13T16:07:56.393 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:07:56.385+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.391040+0000) 2019-10-13T16:07:56.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:07:56.389+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.391575+0000) 2019-10-13T16:07:56.399 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:56.394+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.397046+0000) 2019-10-13T16:07:56.402 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:07:56.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:07:56.420 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:07:56.414+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:07:56.417794+0000) 2019-10-13T16:08:01.406 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:01.401+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.403527+0000) 2019-10-13T16:08:01.406 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:01.401+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.404159+0000) 2019-10-13T16:08:01.407 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:01.401+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.404799+0000) 2019-10-13T16:08:01.408 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:01.401+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.406153+0000) 2019-10-13T16:08:01.408 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:01.401+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.406359+0000) 2019-10-13T16:08:01.409 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:01.401+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.406767+0000) 2019-10-13T16:08:01.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:01.401+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.407083+0000) 2019-10-13T16:08:01.413 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:01.409+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:01.411439+0000) 2019-10-13T16:08:01.417 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:01.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:02.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:02.753+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:02.756465+0000) 2019-10-13T16:08:03.493 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:03.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:03.491394+0000) 2019-10-13T16:08:03.742 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:03.737+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:03.740236+0000) 2019-10-13T16:08:05.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:05.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:05.765981+0000) 2019-10-13T16:08:05.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:05.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:05.766124+0000) 2019-10-13T16:08:05.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:05.773+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:05.776892+0000) 2019-10-13T16:08:06.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:06.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.318392+0000) 2019-10-13T16:08:06.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:06.413+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.418754+0000) 2019-10-13T16:08:06.422 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:06.417+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.419512+0000) 2019-10-13T16:08:06.422 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:06.413+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.419116+0000) 2019-10-13T16:08:06.423 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:06.417+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.420690+0000) 2019-10-13T16:08:06.424 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:06.417+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.421570+0000) 2019-10-13T16:08:06.424 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:06.417+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.421627+0000) 2019-10-13T16:08:06.424 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:06.417+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.422363+0000) 2019-10-13T16:08:06.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:06.425+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.427508+0000) 2019-10-13T16:08:06.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:06.425+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:06.427606+0000) 2019-10-13T16:08:06.432 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:06.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:11.436 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:11.428+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.434217+0000) 2019-10-13T16:08:11.437 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:11.428+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.434791+0000) 2019-10-13T16:08:11.437 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:11.428+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.435053+0000) 2019-10-13T16:08:11.438 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:11.432+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.435602+0000) 2019-10-13T16:08:11.439 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:11.433+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.436847+0000) 2019-10-13T16:08:11.439 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:11.432+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.437164+0000) 2019-10-13T16:08:11.439 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:11.432+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.437539+0000) 2019-10-13T16:08:11.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:11.441+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:11.443141+0000) 2019-10-13T16:08:11.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:11.440+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:12.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:12.752+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:12.756701+0000) 2019-10-13T16:08:13.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:13.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:13.491586+0000) 2019-10-13T16:08:13.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:13.737+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:13.740497+0000) 2019-10-13T16:08:15.768 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:15.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:15.766196+0000) 2019-10-13T16:08:15.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:15.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:15.766322+0000) 2019-10-13T16:08:15.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:15.776+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:15.777097+0000) 2019-10-13T16:08:16.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:16.316+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.318590+0000) 2019-10-13T16:08:16.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:16.425+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.427808+0000) 2019-10-13T16:08:16.451 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:16.444+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.449104+0000) 2019-10-13T16:08:16.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:16.444+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.449421+0000) 2019-10-13T16:08:16.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:16.445+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.450077+0000) 2019-10-13T16:08:16.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:16.444+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.450451+0000) 2019-10-13T16:08:16.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:16.444+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.451213+0000) 2019-10-13T16:08:16.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:16.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.451457+0000) 2019-10-13T16:08:16.454 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:16.448+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.451869+0000) 2019-10-13T16:08:16.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:16.453+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:16.456887+0000) 2019-10-13T16:08:16.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:16.456+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236879 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:18.950 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:08:18.951 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:08:18.954 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:08:18.955 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:08:21.466 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:21.460+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.463600+0000) 2019-10-13T16:08:21.466 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:21.460+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.463782+0000) 2019-10-13T16:08:21.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:21.460+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.464109+0000) 2019-10-13T16:08:21.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:21.461+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.464985+0000) 2019-10-13T16:08:21.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:21.460+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.464456+0000) 2019-10-13T16:08:21.468 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:21.460+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.466028+0000) 2019-10-13T16:08:21.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:21.460+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.466695+0000) 2019-10-13T16:08:21.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:21.469+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:21.473036+0000) 2019-10-13T16:08:21.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:21.472+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:22.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:22.756+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:22.756902+0000) 2019-10-13T16:08:23.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:23.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:23.491815+0000) 2019-10-13T16:08:23.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:23.737+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:23.740696+0000) 2019-10-13T16:08:25.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:25.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:25.766393+0000) 2019-10-13T16:08:25.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:25.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:25.766560+0000) 2019-10-13T16:08:25.779 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:25.776+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:25.777322+0000) 2019-10-13T16:08:26.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:26.312+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.318792+0000) 2019-10-13T16:08:26.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:26.433+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.434336+0000) 2019-10-13T16:08:26.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:26.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.479211+0000) 2019-10-13T16:08:26.482 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:26.476+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.479761+0000) 2019-10-13T16:08:26.482 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:26.476+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.480520+0000) 2019-10-13T16:08:26.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:26.476+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.481535+0000) 2019-10-13T16:08:26.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:26.476+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.481867+0000) 2019-10-13T16:08:26.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:26.476+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.482350+0000) 2019-10-13T16:08:26.485 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:26.476+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.482888+0000) 2019-10-13T16:08:26.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:26.481+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:26.486017+0000) 2019-10-13T16:08:26.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:26.488+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:31.497 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:31.492+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.495406+0000) 2019-10-13T16:08:31.498 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:31.493+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.495817+0000) 2019-10-13T16:08:31.499 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:31.492+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.496592+0000) 2019-10-13T16:08:31.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:31.492+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.496866+0000) 2019-10-13T16:08:31.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:31.492+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.497423+0000) 2019-10-13T16:08:31.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:31.492+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.497685+0000) 2019-10-13T16:08:31.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:31.492+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.498430+0000) 2019-10-13T16:08:31.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:31.505+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:31.506222+0000) 2019-10-13T16:08:31.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:31.500+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 236975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:32.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:32.756+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:32.757104+0000) 2019-10-13T16:08:33.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:33.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:33.492041+0000) 2019-10-13T16:08:33.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:33.737+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:33.740905+0000) 2019-10-13T16:08:35.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:35.764+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:35.766599+0000) 2019-10-13T16:08:35.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:35.764+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:35.766781+0000) 2019-10-13T16:08:35.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:35.776+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:35.777475+0000) 2019-10-13T16:08:36.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:36.316+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.318990+0000) 2019-10-13T16:08:36.443 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:36.437+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.440649+0000) 2019-10-13T16:08:36.512 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:36.504+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.510230+0000) 2019-10-13T16:08:36.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:36.504+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.510692+0000) 2019-10-13T16:08:36.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:36.504+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.511038+0000) 2019-10-13T16:08:36.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:36.509+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.511822+0000) 2019-10-13T16:08:36.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:36.508+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.511388+0000) 2019-10-13T16:08:36.515 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:36.508+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.512648+0000) 2019-10-13T16:08:36.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:36.508+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.512843+0000) 2019-10-13T16:08:36.521 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:36.517+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:36.518763+0000) 2019-10-13T16:08:36.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:36.516+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:41.527 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:41.520+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.524700+0000) 2019-10-13T16:08:41.527 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:41.520+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.525488+0000) 2019-10-13T16:08:41.528 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:41.520+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.526058+0000) 2019-10-13T16:08:41.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:41.520+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.526335+0000) 2019-10-13T16:08:41.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:41.525+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.526923+0000) 2019-10-13T16:08:41.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:41.524+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.527558+0000) 2019-10-13T16:08:41.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:41.524+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.527631+0000) 2019-10-13T16:08:41.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:41.533+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:41.534449+0000) 2019-10-13T16:08:41.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:41.532+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:42.759 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:42.752+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:42.757344+0000) 2019-10-13T16:08:43.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:43.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:43.492234+0000) 2019-10-13T16:08:43.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:43.737+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:43.741093+0000) 2019-10-13T16:08:45.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:45.760+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:45.766797+0000) 2019-10-13T16:08:45.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:45.760+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:45.766956+0000) 2019-10-13T16:08:45.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:45.772+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:45.777695+0000) 2019-10-13T16:08:46.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:46.312+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.319190+0000) 2019-10-13T16:08:46.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:46.445+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.447070+0000) 2019-10-13T16:08:46.542 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:46.536+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.540420+0000) 2019-10-13T16:08:46.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:46.536+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.540843+0000) 2019-10-13T16:08:46.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:46.536+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.541502+0000) 2019-10-13T16:08:46.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:46.536+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.541902+0000) 2019-10-13T16:08:46.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:46.541+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.542810+0000) 2019-10-13T16:08:46.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:46.536+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.542362+0000) 2019-10-13T16:08:46.545 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:46.540+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.543294+0000) 2019-10-13T16:08:46.551 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:46.545+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:46.549225+0000) 2019-10-13T16:08:46.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:46.548+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:49.089 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:08:49.089 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:08:49.093 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:08:49.093 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:08:51.558 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:51.551+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.555808+0000) 2019-10-13T16:08:51.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:51.551+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.556217+0000) 2019-10-13T16:08:51.559 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:51.551+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.556618+0000) 2019-10-13T16:08:51.559 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:51.551+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.557250+0000) 2019-10-13T16:08:51.560 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:51.553+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.557827+0000) 2019-10-13T16:08:51.560 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:51.551+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.558600+0000) 2019-10-13T16:08:51.561 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:51.551+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.559027+0000) 2019-10-13T16:08:51.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:51.561+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:51.563498+0000) 2019-10-13T16:08:51.570 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:51.563+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:08:52.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:52.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:52.757573+0000) 2019-10-13T16:08:53.494 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:53.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:53.492428+0000) 2019-10-13T16:08:53.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:53.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:53.741317+0000) 2019-10-13T16:08:55.769 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:55.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:55.767037+0000) 2019-10-13T16:08:55.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:55.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:55.767149+0000) 2019-10-13T16:08:55.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:55.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:55.777890+0000) 2019-10-13T16:08:56.321 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:56.315+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.319391+0000) 2019-10-13T16:08:56.456 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:56.448+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.453569+0000) 2019-10-13T16:08:56.575 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:08:56.567+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.572652+0000) 2019-10-13T16:08:56.576 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:08:56.567+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.573175+0000) 2019-10-13T16:08:56.576 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:08:56.567+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.573757+0000) 2019-10-13T16:08:56.577 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:08:56.572+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.574619+0000) 2019-10-13T16:08:56.577 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:08:56.567+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.574355+0000) 2019-10-13T16:08:56.578 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:08:56.567+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.575166+0000) 2019-10-13T16:08:56.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:08:56.571+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.576035+0000) 2019-10-13T16:08:56.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:08:56.576+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:08:56.578732+0000) 2019-10-13T16:08:56.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:08:56.579+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237146 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:01.590 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:01.583+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.587803+0000) 2019-10-13T16:09:01.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:01.583+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.588162+0000) 2019-10-13T16:09:01.591 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:01.583+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.588655+0000) 2019-10-13T16:09:01.591 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:01.583+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.589147+0000) 2019-10-13T16:09:01.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:01.584+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.589467+0000) 2019-10-13T16:09:01.593 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:01.583+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.590549+0000) 2019-10-13T16:09:01.593 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:01.587+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.591554+0000) 2019-10-13T16:09:01.602 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:01.596+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:01.599818+0000) 2019-10-13T16:09:01.602 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:01.595+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237176 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:02.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:02.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:02.757772+0000) 2019-10-13T16:09:03.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:03.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:03.492625+0000) 2019-10-13T16:09:03.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:03.736+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:03.741518+0000) 2019-10-13T16:09:05.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:05.763+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:05.767231+0000) 2019-10-13T16:09:05.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:05.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:05.767341+0000) 2019-10-13T16:09:05.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:05.775+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:05.778082+0000) 2019-10-13T16:09:06.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:06.315+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.319602+0000) 2019-10-13T16:09:06.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:06.456+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.460003+0000) 2019-10-13T16:09:06.605 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:06.599+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.602976+0000) 2019-10-13T16:09:06.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:06.599+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.603471+0000) 2019-10-13T16:09:06.606 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:06.599+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.604422+0000) 2019-10-13T16:09:06.607 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:06.599+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.604730+0000) 2019-10-13T16:09:06.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:06.599+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.605324+0000) 2019-10-13T16:09:06.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:06.599+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.605537+0000) 2019-10-13T16:09:06.608 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:06.600+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.606094+0000) 2019-10-13T16:09:06.616 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:06.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:06.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:06.612+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:06.615589+0000) 2019-10-13T16:09:11.620 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:11.611+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.617635+0000) 2019-10-13T16:09:11.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:11.611+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.618607+0000) 2019-10-13T16:09:11.621 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:11.611+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.618963+0000) 2019-10-13T16:09:11.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:11.611+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.619237+0000) 2019-10-13T16:09:11.622 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:11.615+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.619868+0000) 2019-10-13T16:09:11.622 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:11.615+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.620074+0000) 2019-10-13T16:09:11.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:11.616+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.620541+0000) 2019-10-13T16:09:11.629 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:11.624+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:11.627075+0000) 2019-10-13T16:09:11.631 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:11.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:12.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:12.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:12.757975+0000) 2019-10-13T16:09:13.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:13.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:13.492852+0000) 2019-10-13T16:09:13.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:13.740+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:13.741712+0000) 2019-10-13T16:09:15.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:15.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:15.767511+0000) 2019-10-13T16:09:15.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:15.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:15.767545+0000) 2019-10-13T16:09:15.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:15.771+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:15.778281+0000) 2019-10-13T16:09:16.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:16.319+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.319800+0000) 2019-10-13T16:09:16.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:16.464+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.466563+0000) 2019-10-13T16:09:16.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:16.627+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.632507+0000) 2019-10-13T16:09:16.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:16.627+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.633805+0000) 2019-10-13T16:09:16.636 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:16.627+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.633993+0000) 2019-10-13T16:09:16.637 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:16.627+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.634615+0000) 2019-10-13T16:09:16.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:16.627+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.634972+0000) 2019-10-13T16:09:16.637 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:16.631+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.635418+0000) 2019-10-13T16:09:16.638 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:16.632+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.635737+0000) 2019-10-13T16:09:16.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:16.636+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:16.640147+0000) 2019-10-13T16:09:16.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:16.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237280 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:19.189 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:09:19.189 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:09:19.192 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:09:19.192 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:09:21.651 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:21.643+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.648206+0000) 2019-10-13T16:09:21.651 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:21.643+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.648270+0000) 2019-10-13T16:09:21.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:21.643+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.648562+0000) 2019-10-13T16:09:21.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:21.643+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.649697+0000) 2019-10-13T16:09:21.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:21.643+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.650118+0000) 2019-10-13T16:09:21.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:21.648+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.651772+0000) 2019-10-13T16:09:21.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:21.643+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.650657+0000) 2019-10-13T16:09:21.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:21.652+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:21.655928+0000) 2019-10-13T16:09:21.662 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:21.655+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:22.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:22.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:22.758213+0000) 2019-10-13T16:09:23.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:23.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:23.493074+0000) 2019-10-13T16:09:23.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:23.736+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:23.741900+0000) 2019-10-13T16:09:25.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:25.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:25.767729+0000) 2019-10-13T16:09:25.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:25.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:25.767762+0000) 2019-10-13T16:09:25.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:25.775+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:25.778474+0000) 2019-10-13T16:09:26.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:26.319+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.320002+0000) 2019-10-13T16:09:26.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:26.468+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.473105+0000) 2019-10-13T16:09:26.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:26.663+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.664056+0000) 2019-10-13T16:09:26.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:26.663+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.664422+0000) 2019-10-13T16:09:26.667 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:26.663+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.664643+0000) 2019-10-13T16:09:26.667 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:26.663+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.665218+0000) 2019-10-13T16:09:26.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:26.663+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.665357+0000) 2019-10-13T16:09:26.668 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:26.663+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.665758+0000) 2019-10-13T16:09:26.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:26.664+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.666888+0000) 2019-10-13T16:09:26.675 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:26.668+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:26.673052+0000) 2019-10-13T16:09:26.678 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:26.675+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:31.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:31.675+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.678872+0000) 2019-10-13T16:09:31.682 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:31.679+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.679250+0000) 2019-10-13T16:09:31.682 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:31.679+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.680503+0000) 2019-10-13T16:09:31.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:31.676+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.680834+0000) 2019-10-13T16:09:31.683 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:31.679+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.681433+0000) 2019-10-13T16:09:31.684 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:31.679+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.682527+0000) 2019-10-13T16:09:31.685 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:31.679+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.683215+0000) 2019-10-13T16:09:31.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:31.684+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:31.688088+0000) 2019-10-13T16:09:31.694 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:31.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:32.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:32.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:32.758449+0000) 2019-10-13T16:09:33.495 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:33.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:33.493274+0000) 2019-10-13T16:09:33.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:33.740+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:33.742178+0000) 2019-10-13T16:09:35.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:35.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:35.767958+0000) 2019-10-13T16:09:35.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:35.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:35.767972+0000) 2019-10-13T16:09:35.780 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:35.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:35.778693+0000) 2019-10-13T16:09:36.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:36.318+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.320205+0000) 2019-10-13T16:09:36.482 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:36.476+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.479805+0000) 2019-10-13T16:09:36.698 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:36.694+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.695894+0000) 2019-10-13T16:09:36.698 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:36.694+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.696025+0000) 2019-10-13T16:09:36.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:36.692+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.696341+0000) 2019-10-13T16:09:36.699 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:36.694+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.697122+0000) 2019-10-13T16:09:36.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:36.694+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.697445+0000) 2019-10-13T16:09:36.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:36.694+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.698190+0000) 2019-10-13T16:09:36.701 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:36.694+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.698714+0000) 2019-10-13T16:09:36.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:36.700+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:36.704158+0000) 2019-10-13T16:09:36.709 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:36.706+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237414 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:41.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:41.708+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.710598+0000) 2019-10-13T16:09:41.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:41.706+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.711062+0000) 2019-10-13T16:09:41.713 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:41.710+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.711403+0000) 2019-10-13T16:09:41.714 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:41.710+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.711760+0000) 2019-10-13T16:09:41.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:41.710+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.712725+0000) 2019-10-13T16:09:41.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:41.710+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.713268+0000) 2019-10-13T16:09:41.715 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:41.710+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.713640+0000) 2019-10-13T16:09:41.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:41.716+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:41.720681+0000) 2019-10-13T16:09:41.725 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:41.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:42.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:42.754+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:42.758682+0000) 2019-10-13T16:09:43.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:43.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:43.493513+0000) 2019-10-13T16:09:43.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:43.740+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:43.742379+0000) 2019-10-13T16:09:45.770 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:45.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:45.768182+0000) 2019-10-13T16:09:45.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:45.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:45.768200+0000) 2019-10-13T16:09:45.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:45.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:45.778924+0000) 2019-10-13T16:09:46.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:46.318+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.320435+0000) 2019-10-13T16:09:46.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:46.484+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.486393+0000) 2019-10-13T16:09:46.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:46.722+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.726228+0000) 2019-10-13T16:09:46.729 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:46.724+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.727727+0000) 2019-10-13T16:09:46.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:46.726+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.727988+0000) 2019-10-13T16:09:46.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:46.726+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.728302+0000) 2019-10-13T16:09:46.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:46.726+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.728576+0000) 2019-10-13T16:09:46.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:46.726+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.729503+0000) 2019-10-13T16:09:46.732 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:46.726+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.729326+0000) 2019-10-13T16:09:46.738 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:46.732+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:46.735869+0000) 2019-10-13T16:09:46.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:46.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:49.294 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:09:49.294 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:09:49.298 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:09:49.298 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:09:51.743 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:51.738+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.740567+0000) 2019-10-13T16:09:51.743 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:51.738+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.741748+0000) 2019-10-13T16:09:51.745 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:51.738+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.742341+0000) 2019-10-13T16:09:51.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:51.738+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.742722+0000) 2019-10-13T16:09:51.745 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:51.738+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.743080+0000) 2019-10-13T16:09:51.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:51.739+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.743987+0000) 2019-10-13T16:09:51.747 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:51.742+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.744304+0000) 2019-10-13T16:09:51.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:51.747+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:51.750181+0000) 2019-10-13T16:09:51.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:51.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:09:52.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:52.754+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:52.758883+0000) 2019-10-13T16:09:53.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:53.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:53.493705+0000) 2019-10-13T16:09:53.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:53.739+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:53.742605+0000) 2019-10-13T16:09:55.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:55.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:55.768398+0000) 2019-10-13T16:09:55.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:55.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:55.768400+0000) 2019-10-13T16:09:55.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:55.774+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:55.779116+0000) 2019-10-13T16:09:56.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:56.318+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.320637+0000) 2019-10-13T16:09:56.499 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:56.495+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.496927+0000) 2019-10-13T16:09:56.759 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:09:56.754+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.756842+0000) 2019-10-13T16:09:56.760 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:09:56.751+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.757171+0000) 2019-10-13T16:09:56.760 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:09:56.754+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.757582+0000) 2019-10-13T16:09:56.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:09:56.754+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.757906+0000) 2019-10-13T16:09:56.761 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:09:56.754+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.758230+0000) 2019-10-13T16:09:56.761 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:09:56.754+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.758638+0000) 2019-10-13T16:09:56.761 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:09:56.754+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.759088+0000) 2019-10-13T16:09:56.768 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:09:56.763+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:09:56.766414+0000) 2019-10-13T16:09:56.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:09:56.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:01.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:01.770+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.772566+0000) 2019-10-13T16:10:01.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:01.770+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.773351+0000) 2019-10-13T16:10:01.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:01.770+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.773919+0000) 2019-10-13T16:10:01.776 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:01.771+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.774315+0000) 2019-10-13T16:10:01.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:01.770+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.774833+0000) 2019-10-13T16:10:01.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:01.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.775361+0000) 2019-10-13T16:10:01.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:01.774+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.775750+0000) 2019-10-13T16:10:01.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:01.779+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:01.782683+0000) 2019-10-13T16:10:01.787 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:01.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:02.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:02.758+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:02.759126+0000) 2019-10-13T16:10:03.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:03.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:03.493933+0000) 2019-10-13T16:10:03.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:03.739+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:03.742840+0000) 2019-10-13T16:10:05.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:05.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:05.768619+0000) 2019-10-13T16:10:05.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:05.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:05.768607+0000) 2019-10-13T16:10:05.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:05.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:05.779334+0000) 2019-10-13T16:10:06.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:06.318+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.320887+0000) 2019-10-13T16:10:06.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:06.499+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.503302+0000) 2019-10-13T16:10:06.791 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:06.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.789124+0000) 2019-10-13T16:10:06.792 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:06.786+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.789976+0000) 2019-10-13T16:10:06.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:06.786+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.790185+0000) 2019-10-13T16:10:06.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:06.786+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.790480+0000) 2019-10-13T16:10:06.794 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:06.787+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.791776+0000) 2019-10-13T16:10:06.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:06.790+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.792043+0000) 2019-10-13T16:10:06.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:06.790+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.792422+0000) 2019-10-13T16:10:06.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:06.791+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:06.797268+0000) 2019-10-13T16:10:06.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:06.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237618 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:11.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:11.802+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.803495+0000) 2019-10-13T16:10:11.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:11.802+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.804114+0000) 2019-10-13T16:10:11.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:11.802+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.804246+0000) 2019-10-13T16:10:11.807 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:11.802+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.805183+0000) 2019-10-13T16:10:11.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:11.803+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.806912+0000) 2019-10-13T16:10:11.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:11.802+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.806053+0000) 2019-10-13T16:10:11.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:11.802+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.806516+0000) 2019-10-13T16:10:11.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:11.807+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:11.813690+0000) 2019-10-13T16:10:11.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:11.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237648 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:12.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:12.754+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:12.759266+0000) 2019-10-13T16:10:13.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:13.490+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:13.494129+0000) 2019-10-13T16:10:13.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:13.739+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:13.743067+0000) 2019-10-13T16:10:15.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:15.766+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:15.768835+0000) 2019-10-13T16:10:15.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:15.766+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:15.768825+0000) 2019-10-13T16:10:15.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:15.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:15.779552+0000) 2019-10-13T16:10:16.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:16.318+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.321116+0000) 2019-10-13T16:10:16.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:16.507+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.509611+0000) 2019-10-13T16:10:16.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:16.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.819364+0000) 2019-10-13T16:10:16.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:16.818+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.819587+0000) 2019-10-13T16:10:16.823 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:16.818+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.820538+0000) 2019-10-13T16:10:16.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:16.818+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.820760+0000) 2019-10-13T16:10:16.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:16.818+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.821429+0000) 2019-10-13T16:10:16.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:16.818+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.821801+0000) 2019-10-13T16:10:16.824 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:16.819+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.822173+0000) 2019-10-13T16:10:16.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:16.823+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:16.827441+0000) 2019-10-13T16:10:16.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:16.826+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237686 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:19.418 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:10:19.418 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:10:19.421 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:10:19.422 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:10:21.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:21.829+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.835068+0000) 2019-10-13T16:10:21.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:21.833+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.835842+0000) 2019-10-13T16:10:21.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:21.833+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.836068+0000) 2019-10-13T16:10:21.839 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:21.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.836953+0000) 2019-10-13T16:10:21.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:21.833+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.837368+0000) 2019-10-13T16:10:21.840 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:21.831+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.837631+0000) 2019-10-13T16:10:21.840 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:21.833+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.837993+0000) 2019-10-13T16:10:21.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:21.839+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:21.842654+0000) 2019-10-13T16:10:21.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:21.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:22.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:22.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:22.759475+0000) 2019-10-13T16:10:23.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:23.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:23.494348+0000) 2019-10-13T16:10:23.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:23.739+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:23.743304+0000) 2019-10-13T16:10:25.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:25.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:25.769051+0000) 2019-10-13T16:10:25.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:25.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:25.769043+0000) 2019-10-13T16:10:25.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:25.777+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:25.779749+0000) 2019-10-13T16:10:26.323 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:26.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.321322+0000) 2019-10-13T16:10:26.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:26.511+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.515994+0000) 2019-10-13T16:10:26.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:26.845+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.849339+0000) 2019-10-13T16:10:26.852 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:26.845+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.849979+0000) 2019-10-13T16:10:26.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:26.847+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.850407+0000) 2019-10-13T16:10:26.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:26.845+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.850912+0000) 2019-10-13T16:10:26.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:26.849+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.851627+0000) 2019-10-13T16:10:26.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:26.849+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.852328+0000) 2019-10-13T16:10:26.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:26.849+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.852794+0000) 2019-10-13T16:10:26.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:26.851+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:26.857478+0000) 2019-10-13T16:10:26.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:26.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:31.869 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:31.861+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.865939+0000) 2019-10-13T16:10:31.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:31.861+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.866529+0000) 2019-10-13T16:10:31.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:31.861+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.867266+0000) 2019-10-13T16:10:31.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:31.865+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.867695+0000) 2019-10-13T16:10:31.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:31.865+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.868076+0000) 2019-10-13T16:10:31.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:31.863+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.868449+0000) 2019-10-13T16:10:31.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:31.865+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.869108+0000) 2019-10-13T16:10:31.880 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:31.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237786 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:31.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:31.875+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:31.879314+0000) 2019-10-13T16:10:32.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:32.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:32.759674+0000) 2019-10-13T16:10:33.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:33.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:33.494548+0000) 2019-10-13T16:10:33.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:33.739+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:33.743494+0000) 2019-10-13T16:10:35.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:35.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:35.769298+0000) 2019-10-13T16:10:35.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:35.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:35.769297+0000) 2019-10-13T16:10:35.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:35.777+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:35.779947+0000) 2019-10-13T16:10:36.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:36.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.321527+0000) 2019-10-13T16:10:36.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:36.519+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.522438+0000) 2019-10-13T16:10:36.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:36.877+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.881577+0000) 2019-10-13T16:10:36.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:36.877+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.881706+0000) 2019-10-13T16:10:36.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:36.875+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.882066+0000) 2019-10-13T16:10:36.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:36.877+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.882408+0000) 2019-10-13T16:10:36.885 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:36.877+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.882894+0000) 2019-10-13T16:10:36.886 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:36.881+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.883758+0000) 2019-10-13T16:10:36.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:36.881+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.884046+0000) 2019-10-13T16:10:36.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:36.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:36.890562+0000) 2019-10-13T16:10:36.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:36.889+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:41.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:41.893+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.895764+0000) 2019-10-13T16:10:41.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:41.893+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.896161+0000) 2019-10-13T16:10:41.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:41.893+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.896836+0000) 2019-10-13T16:10:41.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:41.893+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.897168+0000) 2019-10-13T16:10:41.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:41.891+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.897940+0000) 2019-10-13T16:10:41.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:41.893+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.898253+0000) 2019-10-13T16:10:41.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:41.893+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.898846+0000) 2019-10-13T16:10:41.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:41.903+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:41.907229+0000) 2019-10-13T16:10:41.909 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:41.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237856 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:42.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:42.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:42.759851+0000) 2019-10-13T16:10:43.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:43.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:43.494766+0000) 2019-10-13T16:10:43.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:43.743+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:43.743715+0000) 2019-10-13T16:10:45.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:45.765+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:45.769519+0000) 2019-10-13T16:10:45.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:45.765+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:45.769497+0000) 2019-10-13T16:10:45.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:45.777+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:45.780177+0000) 2019-10-13T16:10:46.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:46.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.321724+0000) 2019-10-13T16:10:46.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:46.522+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.528683+0000) 2019-10-13T16:10:46.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:46.905+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.911076+0000) 2019-10-13T16:10:46.914 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:46.909+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.911375+0000) 2019-10-13T16:10:46.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:46.906+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.912486+0000) 2019-10-13T16:10:46.915 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:46.909+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.912025+0000) 2019-10-13T16:10:46.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:46.909+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.913367+0000) 2019-10-13T16:10:46.916 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:46.909+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.913393+0000) 2019-10-13T16:10:46.916 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:46.909+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.913786+0000) 2019-10-13T16:10:46.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:46.914+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:46.919691+0000) 2019-10-13T16:10:46.925 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:46.917+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237894 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:49.533 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:10:49.533 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:10:49.537 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:10:49.537 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:10:51.928 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:51.921+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.926250+0000) 2019-10-13T16:10:51.929 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:51.921+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.926872+0000) 2019-10-13T16:10:51.929 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:51.925+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.927520+0000) 2019-10-13T16:10:51.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:51.922+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.927954+0000) 2019-10-13T16:10:51.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:51.925+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.928521+0000) 2019-10-13T16:10:51.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:51.925+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.929438+0000) 2019-10-13T16:10:51.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:51.925+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.929829+0000) 2019-10-13T16:10:51.937 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:51.930+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:51.934953+0000) 2019-10-13T16:10:51.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:51.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237924 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:10:52.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:52.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:52.760029+0000) 2019-10-13T16:10:53.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:53.489+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:53.494957+0000) 2019-10-13T16:10:53.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:53.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:53.743911+0000) 2019-10-13T16:10:55.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:55.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:55.769733+0000) 2019-10-13T16:10:55.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:55.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:55.769733+0000) 2019-10-13T16:10:55.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:55.777+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:55.780369+0000) 2019-10-13T16:10:56.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:56.317+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.321952+0000) 2019-10-13T16:10:56.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:56.530+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.534970+0000) 2019-10-13T16:10:56.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:10:56.934+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.941018+0000) 2019-10-13T16:10:56.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:10:56.937+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.941913+0000) 2019-10-13T16:10:56.944 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:10:56.937+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.941872+0000) 2019-10-13T16:10:56.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:10:56.937+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.942272+0000) 2019-10-13T16:10:56.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:10:56.937+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.942831+0000) 2019-10-13T16:10:56.945 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:10:56.937+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.943111+0000) 2019-10-13T16:10:56.947 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:10:56.941+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.944853+0000) 2019-10-13T16:10:56.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:10:56.946+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:10:56.950232+0000) 2019-10-13T16:10:56.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:10:56.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:01.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:01.953+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.955641+0000) 2019-10-13T16:11:01.958 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:01.953+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.955998+0000) 2019-10-13T16:11:01.958 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:01.953+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.956523+0000) 2019-10-13T16:11:01.959 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:01.953+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.957634+0000) 2019-10-13T16:11:01.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:01.954+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.958322+0000) 2019-10-13T16:11:01.961 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:01.953+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.958598+0000) 2019-10-13T16:11:01.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:01.953+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.958967+0000) 2019-10-13T16:11:01.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:01.958+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:01.963900+0000) 2019-10-13T16:11:01.969 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:01.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 237994 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:02.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:02.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:02.760206+0000) 2019-10-13T16:11:03.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:03.488+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:03.495147+0000) 2019-10-13T16:11:03.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:03.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:03.744092+0000) 2019-10-13T16:11:05.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:05.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:05.769948+0000) 2019-10-13T16:11:05.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:05.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:05.769951+0000) 2019-10-13T16:11:05.782 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:05.776+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:05.780567+0000) 2019-10-13T16:11:06.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:06.320+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.322174+0000) 2019-10-13T16:11:06.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:06.534+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.541415+0000) 2019-10-13T16:11:06.973 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:06.964+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.971174+0000) 2019-10-13T16:11:06.974 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:06.968+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.971820+0000) 2019-10-13T16:11:06.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:06.968+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.972187+0000) 2019-10-13T16:11:06.975 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:06.968+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.972838+0000) 2019-10-13T16:11:06.976 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:06.968+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.973794+0000) 2019-10-13T16:11:06.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:06.968+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.974264+0000) 2019-10-13T16:11:06.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:06.970+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.974578+0000) 2019-10-13T16:11:06.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:06.974+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:06.979780+0000) 2019-10-13T16:11:06.985 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:06.976+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:11.989 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:11.980+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.986368+0000) 2019-10-13T16:11:11.989 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:11.982+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.987086+0000) 2019-10-13T16:11:11.990 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:11.984+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.987613+0000) 2019-10-13T16:11:11.990 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:11.984+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.987991+0000) 2019-10-13T16:11:11.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:11.984+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.988475+0000) 2019-10-13T16:11:11.991 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:11.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.988946+0000) 2019-10-13T16:11:11.992 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:11.984+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.990036+0000) 2019-10-13T16:11:11.999 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:11.990+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:11.996947+0000) 2019-10-13T16:11:12.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:11.992+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238061 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:12.762 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:12.756+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:12.760401+0000) 2019-10-13T16:11:13.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:13.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:13.495343+0000) 2019-10-13T16:11:13.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:13.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:13.744288+0000) 2019-10-13T16:11:15.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:15.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:15.770169+0000) 2019-10-13T16:11:15.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:15.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:15.770182+0000) 2019-10-13T16:11:15.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:15.776+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:15.780753+0000) 2019-10-13T16:11:16.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:16.316+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:16.322390+0000) 2019-10-13T16:11:16.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:16.546+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:16.547920+0000) 2019-10-13T16:11:17.004 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:16.996+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.001621+0000) 2019-10-13T16:11:17.005 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:16.998+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.002702+0000) 2019-10-13T16:11:17.005 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:16.996+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.002327+0000) 2019-10-13T16:11:17.006 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:16.996+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.003224+0000) 2019-10-13T16:11:17.006 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:17.000+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.004141+0000) 2019-10-13T16:11:17.007 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:17.000+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.004871+0000) 2019-10-13T16:11:17.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:17.000+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.005917+0000) 2019-10-13T16:11:17.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:17.002+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:17.009943+0000) 2019-10-13T16:11:17.016 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:17.008+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238100 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:19.865 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:11:19.866 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:11:19.869 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:11:19.870 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:11:22.020 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:22.012+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.017864+0000) 2019-10-13T16:11:22.020 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:22.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.018399+0000) 2019-10-13T16:11:22.021 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:22.012+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.018717+0000) 2019-10-13T16:11:22.021 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:22.012+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.019216+0000) 2019-10-13T16:11:22.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:22.016+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.020198+0000) 2019-10-13T16:11:22.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:22.016+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.020576+0000) 2019-10-13T16:11:22.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:22.016+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.020997+0000) 2019-10-13T16:11:22.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:22.022+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.027556+0000) 2019-10-13T16:11:22.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:22.024+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238131 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:22.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:22.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:22.760600+0000) 2019-10-13T16:11:23.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:23.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:23.495631+0000) 2019-10-13T16:11:23.746 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:23.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:23.744483+0000) 2019-10-13T16:11:25.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:25.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:25.770393+0000) 2019-10-13T16:11:25.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:25.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:25.770392+0000) 2019-10-13T16:11:25.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:25.780+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:25.780916+0000) 2019-10-13T16:11:26.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:26.316+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:26.322579+0000) 2019-10-13T16:11:26.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:26.554+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:26.554663+0000) 2019-10-13T16:11:27.035 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:27.030+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.033500+0000) 2019-10-13T16:11:27.036 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:27.028+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.034175+0000) 2019-10-13T16:11:27.036 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:27.028+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.034637+0000) 2019-10-13T16:11:27.037 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:27.028+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.034958+0000) 2019-10-13T16:11:27.037 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:27.032+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.035565+0000) 2019-10-13T16:11:27.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:27.032+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.036427+0000) 2019-10-13T16:11:27.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:27.032+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.037099+0000) 2019-10-13T16:11:27.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:27.034+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:27.040131+0000) 2019-10-13T16:11:27.048 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:27.040+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238170 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:32.051 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:32.044+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.049215+0000) 2019-10-13T16:11:32.052 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:32.044+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.050399+0000) 2019-10-13T16:11:32.053 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:32.044+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.050720+0000) 2019-10-13T16:11:32.053 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:32.046+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.051251+0000) 2019-10-13T16:11:32.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:32.048+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.052095+0000) 2019-10-13T16:11:32.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:32.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.052546+0000) 2019-10-13T16:11:32.055 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:32.048+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.053406+0000) 2019-10-13T16:11:32.061 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:32.054+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.059651+0000) 2019-10-13T16:11:32.064 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:32.056+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238199 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:32.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:32.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:32.760780+0000) 2019-10-13T16:11:33.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:33.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:33.495827+0000) 2019-10-13T16:11:33.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:33.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:33.744690+0000) 2019-10-13T16:11:35.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:35.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:35.770632+0000) 2019-10-13T16:11:35.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:35.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:35.770636+0000) 2019-10-13T16:11:35.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:35.780+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:35.781115+0000) 2019-10-13T16:11:36.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:36.320+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:36.322802+0000) 2019-10-13T16:11:36.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:36.566+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:36.566276+0000) 2019-10-13T16:11:37.068 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:37.060+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.065847+0000) 2019-10-13T16:11:37.068 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:37.060+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.066316+0000) 2019-10-13T16:11:37.069 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:37.060+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.067187+0000) 2019-10-13T16:11:37.070 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:37.066+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.067898+0000) 2019-10-13T16:11:37.070 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:37.064+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.068194+0000) 2019-10-13T16:11:37.071 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:37.064+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.068465+0000) 2019-10-13T16:11:37.071 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:37.064+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.069072+0000) 2019-10-13T16:11:37.078 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:37.074+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:37.075835+0000) 2019-10-13T16:11:37.079 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:37.072+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:42.083 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:42.076+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.080368+0000) 2019-10-13T16:11:42.083 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:42.076+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.080880+0000) 2019-10-13T16:11:42.083 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:42.076+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.081445+0000) 2019-10-13T16:11:42.084 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:42.076+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.082248+0000) 2019-10-13T16:11:42.085 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:42.081+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.082835+0000) 2019-10-13T16:11:42.085 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:42.076+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.083211+0000) 2019-10-13T16:11:42.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:42.080+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.084317+0000) 2019-10-13T16:11:42.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:42.089+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.090215+0000) 2019-10-13T16:11:42.095 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:42.088+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238269 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:42.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:42.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:42.760960+0000) 2019-10-13T16:11:43.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:43.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:43.496042+0000) 2019-10-13T16:11:43.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:43.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:43.744910+0000) 2019-10-13T16:11:45.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:45.768+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:45.770848+0000) 2019-10-13T16:11:45.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:45.768+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:45.770860+0000) 2019-10-13T16:11:45.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:45.780+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:45.781341+0000) 2019-10-13T16:11:46.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:46.316+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:46.322996+0000) 2019-10-13T16:11:46.575 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:46.569+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:46.572887+0000) 2019-10-13T16:11:47.098 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:47.091+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.096169+0000) 2019-10-13T16:11:47.099 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:47.093+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.097103+0000) 2019-10-13T16:11:47.100 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:47.091+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.097877+0000) 2019-10-13T16:11:47.100 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:47.091+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.098637+0000) 2019-10-13T16:11:47.101 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:47.091+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.099174+0000) 2019-10-13T16:11:47.101 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:47.095+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.099517+0000) 2019-10-13T16:11:47.102 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:47.095+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.100654+0000) 2019-10-13T16:11:47.108 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:47.105+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:47.106593+0000) 2019-10-13T16:11:47.112 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:47.103+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238308 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:49.930 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:11:49.930 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:11:49.935 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:11:49.935 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:11:52.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:52.107+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.113295+0000) 2019-10-13T16:11:52.116 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:52.107+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.113968+0000) 2019-10-13T16:11:52.116 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:52.107+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.114323+0000) 2019-10-13T16:11:52.117 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:52.107+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.114775+0000) 2019-10-13T16:11:52.118 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:52.113+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.116129+0000) 2019-10-13T16:11:52.119 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:52.111+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.116679+0000) 2019-10-13T16:11:52.119 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:52.111+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.116704+0000) 2019-10-13T16:11:52.123 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:52.117+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.121074+0000) 2019-10-13T16:11:52.127 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:52.119+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238337 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:11:52.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:52.759+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:52.761138+0000) 2019-10-13T16:11:53.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:53.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:53.496253+0000) 2019-10-13T16:11:53.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:53.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:53.745102+0000) 2019-10-13T16:11:55.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:55.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:55.771067+0000) 2019-10-13T16:11:55.773 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:55.767+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:55.771068+0000) 2019-10-13T16:11:55.783 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:55.775+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:55.781532+0000) 2019-10-13T16:11:56.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:56.319+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:56.323191+0000) 2019-10-13T16:11:56.581 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:56.577+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:56.579241+0000) 2019-10-13T16:11:57.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:11:57.123+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.128171+0000) 2019-10-13T16:11:57.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:11:57.123+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.129438+0000) 2019-10-13T16:11:57.132 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:11:57.123+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.129951+0000) 2019-10-13T16:11:57.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:11:57.123+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.131145+0000) 2019-10-13T16:11:57.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:11:57.127+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.131547+0000) 2019-10-13T16:11:57.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:11:57.127+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.132670+0000) 2019-10-13T16:11:57.135 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:11:57.129+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.132754+0000) 2019-10-13T16:11:57.141 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:11:57.137+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:11:57.138792+0000) 2019-10-13T16:11:57.142 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:11:57.135+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238376 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:02.146 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:02.139+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.143955+0000) 2019-10-13T16:12:02.147 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:02.139+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.144888+0000) 2019-10-13T16:12:02.147 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:02.139+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.145195+0000) 2019-10-13T16:12:02.148 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:02.139+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.145518+0000) 2019-10-13T16:12:02.148 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:02.141+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.145820+0000) 2019-10-13T16:12:02.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:02.139+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.146893+0000) 2019-10-13T16:12:02.149 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:02.139+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.147231+0000) 2019-10-13T16:12:02.156 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:02.153+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.154675+0000) 2019-10-13T16:12:02.157 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:02.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:02.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:02.755+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:02.761315+0000) 2019-10-13T16:12:03.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:03.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:03.496444+0000) 2019-10-13T16:12:03.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:03.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:03.745307+0000) 2019-10-13T16:12:05.773 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:05.767+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:05.771287+0000) 2019-10-13T16:12:05.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:05.763+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:05.771269+0000) 2019-10-13T16:12:05.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:05.779+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:05.781724+0000) 2019-10-13T16:12:06.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:06.319+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:06.323362+0000) 2019-10-13T16:12:06.588 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:06.585+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:06.585565+0000) 2019-10-13T16:12:07.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:07.157+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.158342+0000) 2019-10-13T16:12:07.161 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:07.151+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.158705+0000) 2019-10-13T16:12:07.161 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:07.151+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.159170+0000) 2019-10-13T16:12:07.162 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:07.155+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.159772+0000) 2019-10-13T16:12:07.162 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:07.155+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.160301+0000) 2019-10-13T16:12:07.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:07.155+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.160761+0000) 2019-10-13T16:12:07.163 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:07.155+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.160607+0000) 2019-10-13T16:12:07.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:07.165+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:07.169591+0000) 2019-10-13T16:12:07.172 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:07.163+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238445 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:12.176 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:12.167+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.173640+0000) 2019-10-13T16:12:12.176 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:12.167+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.174169+0000) 2019-10-13T16:12:12.177 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:12.167+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.175239+0000) 2019-10-13T16:12:12.177 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:12.171+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.175493+0000) 2019-10-13T16:12:12.178 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:12.173+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.176159+0000) 2019-10-13T16:12:12.178 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:12.171+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.176092+0000) 2019-10-13T16:12:12.179 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:12.171+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.176876+0000) 2019-10-13T16:12:12.182 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:12.177+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.179838+0000) 2019-10-13T16:12:12.188 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:12.179+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:12.763 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:12.759+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:12.761496+0000) 2019-10-13T16:12:13.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:13.491+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:13.496638+0000) 2019-10-13T16:12:13.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:13.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:13.745503+0000) 2019-10-13T16:12:15.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:15.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:15.771514+0000) 2019-10-13T16:12:15.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:15.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:15.771506+0000) 2019-10-13T16:12:15.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:15.779+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:15.781944+0000) 2019-10-13T16:12:16.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:16.323+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:16.323558+0000) 2019-10-13T16:12:16.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:16.589+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:16.591869+0000) 2019-10-13T16:12:17.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:17.183+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.189268+0000) 2019-10-13T16:12:17.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:17.183+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.189862+0000) 2019-10-13T16:12:17.192 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:17.183+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.190473+0000) 2019-10-13T16:12:17.193 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:17.183+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.190975+0000) 2019-10-13T16:12:17.194 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:17.191+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.191984+0000) 2019-10-13T16:12:17.194 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:17.189+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.192396+0000) 2019-10-13T16:12:17.195 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:17.191+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.193526+0000) 2019-10-13T16:12:17.201 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:17.197+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:17.198983+0000) 2019-10-13T16:12:17.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:17.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:20.038 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:12:20.038 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:12:20.042 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:12:20.042 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:12:22.207 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:22.203+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.204504+0000) 2019-10-13T16:12:22.207 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:22.203+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.205170+0000) 2019-10-13T16:12:22.208 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:22.203+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.205870+0000) 2019-10-13T16:12:22.208 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:22.203+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.206491+0000) 2019-10-13T16:12:22.209 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:22.203+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.207056+0000) 2019-10-13T16:12:22.209 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:22.203+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.206869+0000) 2019-10-13T16:12:22.210 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:22.205+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.207835+0000) 2019-10-13T16:12:22.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:22.213+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.214956+0000) 2019-10-13T16:12:22.218 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:22.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:22.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:22.759+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:22.761674+0000) 2019-10-13T16:12:23.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:23.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:23.496849+0000) 2019-10-13T16:12:23.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:23.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:23.745675+0000) 2019-10-13T16:12:25.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:25.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:25.771691+0000) 2019-10-13T16:12:25.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:25.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:25.771720+0000) 2019-10-13T16:12:25.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:25.779+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:25.782130+0000) 2019-10-13T16:12:26.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:26.323+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:26.323717+0000) 2019-10-13T16:12:26.600 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:26.597+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:26.598198+0000) 2019-10-13T16:12:27.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:27.215+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.219148+0000) 2019-10-13T16:12:27.222 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:27.219+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.220098+0000) 2019-10-13T16:12:27.223 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:27.219+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.220994+0000) 2019-10-13T16:12:27.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:27.217+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.221447+0000) 2019-10-13T16:12:27.224 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:27.219+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.221873+0000) 2019-10-13T16:12:27.224 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:27.219+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.222519+0000) 2019-10-13T16:12:27.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:27.219+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.222703+0000) 2019-10-13T16:12:27.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:27.225+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:27.227518+0000) 2019-10-13T16:12:27.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:27.227+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238581 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:32.236 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:32.233+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.234371+0000) 2019-10-13T16:12:32.237 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:32.230+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.233810+0000) 2019-10-13T16:12:32.237 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:32.230+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.234866+0000) 2019-10-13T16:12:32.239 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:32.234+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.236055+0000) 2019-10-13T16:12:32.239 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:32.234+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.236573+0000) 2019-10-13T16:12:32.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:32.234+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.237239+0000) 2019-10-13T16:12:32.240 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:32.234+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.238003+0000) 2019-10-13T16:12:32.246 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:32.241+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.244391+0000) 2019-10-13T16:12:32.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:32.246+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:32.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:32.758+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:32.761856+0000) 2019-10-13T16:12:33.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:33.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:33.497045+0000) 2019-10-13T16:12:33.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:33.741+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:33.745861+0000) 2019-10-13T16:12:35.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:35.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:35.771851+0000) 2019-10-13T16:12:35.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:35.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:35.771902+0000) 2019-10-13T16:12:35.784 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:35.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:35.782282+0000) 2019-10-13T16:12:36.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:36.322+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:36.323920+0000) 2019-10-13T16:12:36.611 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:36.604+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:36.608938+0000) 2019-10-13T16:12:37.252 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:37.248+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.250381+0000) 2019-10-13T16:12:37.253 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:37.246+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.251149+0000) 2019-10-13T16:12:37.254 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:37.250+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.251884+0000) 2019-10-13T16:12:37.254 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:37.250+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.252217+0000) 2019-10-13T16:12:37.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:37.250+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.252529+0000) 2019-10-13T16:12:37.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:37.250+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.252895+0000) 2019-10-13T16:12:37.256 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:37.250+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.253911+0000) 2019-10-13T16:12:37.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:37.256+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:37.260947+0000) 2019-10-13T16:12:37.266 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:37.262+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238649 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:42.270 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:42.266+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.267575+0000) 2019-10-13T16:12:42.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:42.266+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.268581+0000) 2019-10-13T16:12:42.271 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:42.266+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.269175+0000) 2019-10-13T16:12:42.272 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:42.268+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.270183+0000) 2019-10-13T16:12:42.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:42.266+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.270500+0000) 2019-10-13T16:12:42.273 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:42.266+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.270952+0000) 2019-10-13T16:12:42.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:42.270+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.271550+0000) 2019-10-13T16:12:42.278 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:42.272+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.276086+0000) 2019-10-13T16:12:42.282 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:42.278+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:42.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:42.758+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:42.762083+0000) 2019-10-13T16:12:43.499 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:43.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:43.497235+0000) 2019-10-13T16:12:43.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:43.740+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:43.746066+0000) 2019-10-13T16:12:45.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:45.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:45.772101+0000) 2019-10-13T16:12:45.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:45.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:45.772103+0000) 2019-10-13T16:12:45.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:45.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:45.782466+0000) 2019-10-13T16:12:46.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:46.322+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:46.324117+0000) 2019-10-13T16:12:46.618 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:46.612+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:46.615443+0000) 2019-10-13T16:12:47.289 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:47.282+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.283346+0000) 2019-10-13T16:12:47.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:47.282+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.283846+0000) 2019-10-13T16:12:47.289 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:47.282+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.283405+0000) 2019-10-13T16:12:47.290 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:47.282+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.284524+0000) 2019-10-13T16:12:47.291 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:47.282+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.284765+0000) 2019-10-13T16:12:47.291 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:47.280+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.285074+0000) 2019-10-13T16:12:47.292 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:47.282+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.285474+0000) 2019-10-13T16:12:47.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:47.288+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:47.292159+0000) 2019-10-13T16:12:47.299 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:47.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238717 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:50.150 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:12:50.150 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:12:50.154 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:12:50.154 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:12:52.301 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:52.294+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.298358+0000) 2019-10-13T16:12:52.301 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:52.294+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.299117+0000) 2019-10-13T16:12:52.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:52.298+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.299363+0000) 2019-10-13T16:12:52.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:52.298+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.299796+0000) 2019-10-13T16:12:52.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:52.298+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.300736+0000) 2019-10-13T16:12:52.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:52.296+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.300953+0000) 2019-10-13T16:12:52.303 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:52.298+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.301265+0000) 2019-10-13T16:12:52.309 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:52.304+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.306857+0000) 2019-10-13T16:12:52.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:52.306+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238745 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:12:52.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:52.758+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:52.762266+0000) 2019-10-13T16:12:53.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:53.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:53.497435+0000) 2019-10-13T16:12:53.748 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:53.744+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:53.746327+0000) 2019-10-13T16:12:55.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:55.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:55.772307+0000) 2019-10-13T16:12:55.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:55.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:55.772310+0000) 2019-10-13T16:12:55.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:55.782+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:55.782694+0000) 2019-10-13T16:12:56.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:56.322+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:56.324324+0000) 2019-10-13T16:12:56.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:56.616+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:56.621810+0000) 2019-10-13T16:12:57.316 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:12:57.310+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.313734+0000) 2019-10-13T16:12:57.316 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:12:57.308+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.314023+0000) 2019-10-13T16:12:57.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:12:57.310+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.314716+0000) 2019-10-13T16:12:57.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:12:57.310+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.315167+0000) 2019-10-13T16:12:57.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:12:57.314+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.315807+0000) 2019-10-13T16:12:57.318 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:12:57.314+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.316291+0000) 2019-10-13T16:12:57.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:12:57.314+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.317165+0000) 2019-10-13T16:12:57.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:12:57.320+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:12:57.323225+0000) 2019-10-13T16:12:57.327 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:12:57.322+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238785 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:02.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:02.326+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.329549+0000) 2019-10-13T16:13:02.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:02.326+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.330172+0000) 2019-10-13T16:13:02.333 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:02.326+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.330225+0000) 2019-10-13T16:13:02.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:02.326+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.330709+0000) 2019-10-13T16:13:02.333 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:02.326+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.331198+0000) 2019-10-13T16:13:02.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:02.330+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.332476+0000) 2019-10-13T16:13:02.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:02.328+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.332170+0000) 2019-10-13T16:13:02.340 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:02.332+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.338020+0000) 2019-10-13T16:13:02.343 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:02.338+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238814 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:02.764 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:02.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:02.762453+0000) 2019-10-13T16:13:03.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:03.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:03.497668+0000) 2019-10-13T16:13:03.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:03.744+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:03.746534+0000) 2019-10-13T16:13:05.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:05.770+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:05.772533+0000) 2019-10-13T16:13:05.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:05.770+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:05.772521+0000) 2019-10-13T16:13:05.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:05.778+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:05.782881+0000) 2019-10-13T16:13:06.326 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:06.322+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:06.324526+0000) 2019-10-13T16:13:06.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:06.632+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:06.633357+0000) 2019-10-13T16:13:07.346 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:07.342+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.344315+0000) 2019-10-13T16:13:07.347 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:07.342+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.345150+0000) 2019-10-13T16:13:07.348 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:07.342+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.345784+0000) 2019-10-13T16:13:07.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:07.342+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.346372+0000) 2019-10-13T16:13:07.349 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:07.342+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.347043+0000) 2019-10-13T16:13:07.349 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:07.346+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.347867+0000) 2019-10-13T16:13:07.350 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:07.344+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.348159+0000) 2019-10-13T16:13:07.355 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:07.348+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:07.353258+0000) 2019-10-13T16:13:07.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:07.354+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238853 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:12.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:12.358+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.361662+0000) 2019-10-13T16:13:12.364 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:12.358+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.362190+0000) 2019-10-13T16:13:12.365 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:12.358+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.362554+0000) 2019-10-13T16:13:12.365 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:12.358+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.362895+0000) 2019-10-13T16:13:12.365 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:12.362+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.363564+0000) 2019-10-13T16:13:12.366 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:12.362+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.364221+0000) 2019-10-13T16:13:12.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:12.360+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.365623+0000) 2019-10-13T16:13:12.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:12.368+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.370278+0000) 2019-10-13T16:13:12.374 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:12.370+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238881 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:12.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:12.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:12.762635+0000) 2019-10-13T16:13:13.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:13.494+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:13.497900+0000) 2019-10-13T16:13:13.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:13.744+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:13.746729+0000) 2019-10-13T16:13:15.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:15.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:15.772741+0000) 2019-10-13T16:13:15.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:15.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:15.772733+0000) 2019-10-13T16:13:15.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:15.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:15.783080+0000) 2019-10-13T16:13:16.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:16.321+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:16.324772+0000) 2019-10-13T16:13:16.642 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:16.636+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:16.639728+0000) 2019-10-13T16:13:17.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:17.373+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.375899+0000) 2019-10-13T16:13:17.378 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:17.373+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.376543+0000) 2019-10-13T16:13:17.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:17.373+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.376914+0000) 2019-10-13T16:13:17.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:17.373+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.377752+0000) 2019-10-13T16:13:17.380 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:17.373+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.378247+0000) 2019-10-13T16:13:17.381 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:17.373+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.378964+0000) 2019-10-13T16:13:17.381 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:17.376+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.379630+0000) 2019-10-13T16:13:17.386 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:17.380+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:17.383935+0000) 2019-10-13T16:13:17.391 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:17.385+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:20.220 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:13:20.220 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:13:20.223 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:13:20.224 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:13:22.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:22.389+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.392367+0000) 2019-10-13T16:13:22.395 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:22.389+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.392840+0000) 2019-10-13T16:13:22.395 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:22.389+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.393301+0000) 2019-10-13T16:13:22.396 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:22.389+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.393693+0000) 2019-10-13T16:13:22.396 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:22.389+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.394523+0000) 2019-10-13T16:13:22.397 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:22.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.394979+0000) 2019-10-13T16:13:22.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:22.392+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.396100+0000) 2019-10-13T16:13:22.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:22.397+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238950 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:22.406 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:22.400+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.403873+0000) 2019-10-13T16:13:22.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:22.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:22.762816+0000) 2019-10-13T16:13:23.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:23.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:23.498095+0000) 2019-10-13T16:13:23.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:23.744+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:23.746926+0000) 2019-10-13T16:13:25.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:25.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:25.772954+0000) 2019-10-13T16:13:25.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:25.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:25.772952+0000) 2019-10-13T16:13:25.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:25.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:25.783295+0000) 2019-10-13T16:13:26.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:26.321+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:26.325033+0000) 2019-10-13T16:13:26.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:26.640+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:26.646117+0000) 2019-10-13T16:13:27.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:27.401+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.406461+0000) 2019-10-13T16:13:27.409 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:27.401+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.406828+0000) 2019-10-13T16:13:27.409 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:27.405+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.407545+0000) 2019-10-13T16:13:27.410 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:27.405+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.407949+0000) 2019-10-13T16:13:27.410 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:27.405+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.408188+0000) 2019-10-13T16:13:27.411 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:27.404+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.408511+0000) 2019-10-13T16:13:27.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:27.405+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.409801+0000) 2019-10-13T16:13:27.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:27.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:27.414940+0000) 2019-10-13T16:13:27.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:27.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 238989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:32.424 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:32.415+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.421617+0000) 2019-10-13T16:13:32.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:32.417+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.422103+0000) 2019-10-13T16:13:32.425 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:32.417+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.422618+0000) 2019-10-13T16:13:32.425 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:32.417+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.422849+0000) 2019-10-13T16:13:32.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:32.421+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.423474+0000) 2019-10-13T16:13:32.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:32.421+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.423576+0000) 2019-10-13T16:13:32.426 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:32.421+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.424289+0000) 2019-10-13T16:13:32.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:32.423+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.429994+0000) 2019-10-13T16:13:32.434 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:32.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239017 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:32.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:32.757+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:32.762997+0000) 2019-10-13T16:13:33.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:33.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:33.498281+0000) 2019-10-13T16:13:33.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:33.743+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:33.747122+0000) 2019-10-13T16:13:35.775 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:35.769+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:35.773154+0000) 2019-10-13T16:13:35.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:35.769+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:35.773153+0000) 2019-10-13T16:13:35.785 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:35.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:35.783561+0000) 2019-10-13T16:13:36.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:36.321+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:36.325237+0000) 2019-10-13T16:13:36.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:36.647+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:36.652579+0000) 2019-10-13T16:13:37.438 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:37.431+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.436215+0000) 2019-10-13T16:13:37.439 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:37.433+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.437077+0000) 2019-10-13T16:13:37.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:37.433+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.437428+0000) 2019-10-13T16:13:37.440 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:37.433+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.438071+0000) 2019-10-13T16:13:37.440 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:37.433+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.438138+0000) 2019-10-13T16:13:37.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:37.433+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.439005+0000) 2019-10-13T16:13:37.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:37.433+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.438561+0000) 2019-10-13T16:13:37.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:37.439+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:37.442990+0000) 2019-10-13T16:13:37.450 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:37.445+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239056 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:42.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:42.449+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.451416+0000) 2019-10-13T16:13:42.454 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:42.449+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.452057+0000) 2019-10-13T16:13:42.454 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:42.449+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.452421+0000) 2019-10-13T16:13:42.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:42.449+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.453319+0000) 2019-10-13T16:13:42.456 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:42.447+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.453752+0000) 2019-10-13T16:13:42.456 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:42.449+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.454401+0000) 2019-10-13T16:13:42.457 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:42.449+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.454513+0000) 2019-10-13T16:13:42.463 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:42.455+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.461570+0000) 2019-10-13T16:13:42.465 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:42.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239086 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:42.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:42.761+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:42.763216+0000) 2019-10-13T16:13:43.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:43.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:43.498434+0000) 2019-10-13T16:13:43.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:43.743+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:43.747347+0000) 2019-10-13T16:13:45.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:45.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:45.773357+0000) 2019-10-13T16:13:45.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:45.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:45.773358+0000) 2019-10-13T16:13:45.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:45.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:45.783779+0000) 2019-10-13T16:13:46.327 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:46.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:46.325433+0000) 2019-10-13T16:13:46.660 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:46.655+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:46.658378+0000) 2019-10-13T16:13:47.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:47.461+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.466340+0000) 2019-10-13T16:13:47.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:47.461+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.466863+0000) 2019-10-13T16:13:47.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:47.461+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.467051+0000) 2019-10-13T16:13:47.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:47.463+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.467547+0000) 2019-10-13T16:13:47.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:47.465+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.468948+0000) 2019-10-13T16:13:47.471 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:47.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.469634+0000) 2019-10-13T16:13:47.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:47.465+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.469938+0000) 2019-10-13T16:13:47.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:47.471+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:47.476380+0000) 2019-10-13T16:13:47.479 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:47.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239124 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:50.323 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:13:50.323 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:13:50.327 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:13:50.327 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:13:52.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:52.477+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.480901+0000) 2019-10-13T16:13:52.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:52.475+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.481232+0000) 2019-10-13T16:13:52.484 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:52.477+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.481729+0000) 2019-10-13T16:13:52.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:52.477+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.482453+0000) 2019-10-13T16:13:52.485 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:52.477+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.483049+0000) 2019-10-13T16:13:52.486 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:52.481+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.484326+0000) 2019-10-13T16:13:52.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:52.481+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.484656+0000) 2019-10-13T16:13:52.494 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:52.487+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.492466+0000) 2019-10-13T16:13:52.495 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:52.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:13:52.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:52.761+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:52.763430+0000) 2019-10-13T16:13:53.500 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:53.493+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:53.498627+0000) 2019-10-13T16:13:53.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:53.743+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:53.747548+0000) 2019-10-13T16:13:55.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:55.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:55.773561+0000) 2019-10-13T16:13:55.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:55.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:55.773563+0000) 2019-10-13T16:13:55.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:55.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:55.783985+0000) 2019-10-13T16:13:56.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:56.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:56.325631+0000) 2019-10-13T16:13:56.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:56.659+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:56.664729+0000) 2019-10-13T16:13:57.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:13:57.493+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.496057+0000) 2019-10-13T16:13:57.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:13:57.493+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.496549+0000) 2019-10-13T16:13:57.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:13:57.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.497331+0000) 2019-10-13T16:13:57.500 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:13:57.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.497825+0000) 2019-10-13T16:13:57.500 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:13:57.493+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.498418+0000) 2019-10-13T16:13:57.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:13:57.493+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.498430+0000) 2019-10-13T16:13:57.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:13:57.497+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.499685+0000) 2019-10-13T16:13:57.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:13:57.499+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:13:57.505102+0000) 2019-10-13T16:13:57.510 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:13:57.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239192 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:02.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:02.507+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.512089+0000) 2019-10-13T16:14:02.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:02.508+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.512545+0000) 2019-10-13T16:14:02.515 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:02.508+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.512783+0000) 2019-10-13T16:14:02.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:02.508+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.513223+0000) 2019-10-13T16:14:02.516 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:02.508+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.514522+0000) 2019-10-13T16:14:02.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:02.508+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.515113+0000) 2019-10-13T16:14:02.517 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:02.508+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.515074+0000) 2019-10-13T16:14:02.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:02.515+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.520350+0000) 2019-10-13T16:14:02.526 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:02.520+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239222 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:02.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:02.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:02.763608+0000) 2019-10-13T16:14:03.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:03.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:03.498824+0000) 2019-10-13T16:14:03.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:03.743+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:03.747764+0000) 2019-10-13T16:14:05.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:05.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:05.773795+0000) 2019-10-13T16:14:05.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:05.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:05.773800+0000) 2019-10-13T16:14:05.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:05.780+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:05.784207+0000) 2019-10-13T16:14:06.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:06.320+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:06.325871+0000) 2019-10-13T16:14:06.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:06.667+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:06.671163+0000) 2019-10-13T16:14:07.529 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:07.520+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.526711+0000) 2019-10-13T16:14:07.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:07.520+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.527234+0000) 2019-10-13T16:14:07.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:07.523+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.527905+0000) 2019-10-13T16:14:07.532 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:07.524+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.529130+0000) 2019-10-13T16:14:07.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:07.524+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.529350+0000) 2019-10-13T16:14:07.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:07.524+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.529759+0000) 2019-10-13T16:14:07.533 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:07.524+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.530501+0000) 2019-10-13T16:14:07.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:07.527+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:07.533855+0000) 2019-10-13T16:14:07.541 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:07.536+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239260 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:12.544 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:12.536+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.542101+0000) 2019-10-13T16:14:12.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:12.540+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.543337+0000) 2019-10-13T16:14:12.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:12.540+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.543652+0000) 2019-10-13T16:14:12.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:12.539+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.544542+0000) 2019-10-13T16:14:12.547 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:12.540+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.545112+0000) 2019-10-13T16:14:12.548 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:12.540+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.546090+0000) 2019-10-13T16:14:12.548 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:12.540+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.546505+0000) 2019-10-13T16:14:12.553 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:12.547+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.551057+0000) 2019-10-13T16:14:12.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:12.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:12.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:12.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:12.763788+0000) 2019-10-13T16:14:13.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:13.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:13.499003+0000) 2019-10-13T16:14:13.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:13.747+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:13.747991+0000) 2019-10-13T16:14:15.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:15.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:15.773995+0000) 2019-10-13T16:14:15.776 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:15.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:15.774067+0000) 2019-10-13T16:14:15.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:15.780+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:15.784435+0000) 2019-10-13T16:14:16.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:16.320+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:16.326112+0000) 2019-10-13T16:14:16.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:16.671+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:16.677488+0000) 2019-10-13T16:14:17.561 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:17.555+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.559325+0000) 2019-10-13T16:14:17.562 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:17.556+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.560476+0000) 2019-10-13T16:14:17.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:17.556+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.561203+0000) 2019-10-13T16:14:17.563 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:17.556+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.561757+0000) 2019-10-13T16:14:17.564 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:17.556+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.562362+0000) 2019-10-13T16:14:17.565 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:17.556+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.562394+0000) 2019-10-13T16:14:17.565 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:17.560+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.563505+0000) 2019-10-13T16:14:17.568 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:17.563+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:17.566288+0000) 2019-10-13T16:14:17.586 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:17.564+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239328 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:20.446 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:14:20.446 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:14:20.450 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:14:20.450 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:14:22.576 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:22.571+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.573797+0000) 2019-10-13T16:14:22.577 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:22.568+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.574703+0000) 2019-10-13T16:14:22.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:22.572+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.575620+0000) 2019-10-13T16:14:22.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:22.572+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.575891+0000) 2019-10-13T16:14:22.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:22.572+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.576621+0000) 2019-10-13T16:14:22.579 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:22.572+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.576964+0000) 2019-10-13T16:14:22.580 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:22.572+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.578066+0000) 2019-10-13T16:14:22.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:22.575+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.581419+0000) 2019-10-13T16:14:22.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:22.580+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239358 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:22.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:22.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:22.763986+0000) 2019-10-13T16:14:23.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:23.492+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:23.499176+0000) 2019-10-13T16:14:23.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:23.747+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:23.748188+0000) 2019-10-13T16:14:25.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:25.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:25.774215+0000) 2019-10-13T16:14:25.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:25.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:25.774310+0000) 2019-10-13T16:14:25.786 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:25.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:25.784633+0000) 2019-10-13T16:14:26.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:26.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:26.326364+0000) 2019-10-13T16:14:26.686 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:26.682+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:26.683793+0000) 2019-10-13T16:14:27.591 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:27.584+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.589199+0000) 2019-10-13T16:14:27.592 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:27.584+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.590563+0000) 2019-10-13T16:14:27.593 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:27.586+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.591087+0000) 2019-10-13T16:14:27.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:27.588+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.591854+0000) 2019-10-13T16:14:27.594 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:27.588+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.592264+0000) 2019-10-13T16:14:27.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:27.588+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.593185+0000) 2019-10-13T16:14:27.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:27.588+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.593892+0000) 2019-10-13T16:14:27.598 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:27.590+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:27.596703+0000) 2019-10-13T16:14:27.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:27.596+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239395 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:32.607 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:32.600+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.604874+0000) 2019-10-13T16:14:32.607 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:32.600+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.605442+0000) 2019-10-13T16:14:32.608 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:32.600+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.606203+0000) 2019-10-13T16:14:32.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:32.602+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.606968+0000) 2019-10-13T16:14:32.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:32.604+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.607542+0000) 2019-10-13T16:14:32.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:32.604+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.608135+0000) 2019-10-13T16:14:32.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:32.604+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.608621+0000) 2019-10-13T16:14:32.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:32.606+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.613009+0000) 2019-10-13T16:14:32.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:32.612+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:32.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:32.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:32.764159+0000) 2019-10-13T16:14:33.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:33.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:33.499384+0000) 2019-10-13T16:14:33.750 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:33.742+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:33.748384+0000) 2019-10-13T16:14:35.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:35.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:35.774453+0000) 2019-10-13T16:14:35.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:35.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:35.774518+0000) 2019-10-13T16:14:35.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:35.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:35.784832+0000) 2019-10-13T16:14:36.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:36.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:36.326570+0000) 2019-10-13T16:14:36.692 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:36.682+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:36.690086+0000) 2019-10-13T16:14:37.623 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:37.616+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.620795+0000) 2019-10-13T16:14:37.624 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:37.616+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.621063+0000) 2019-10-13T16:14:37.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:37.616+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.621788+0000) 2019-10-13T16:14:37.625 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:37.616+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.622421+0000) 2019-10-13T16:14:37.625 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:37.616+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.622950+0000) 2019-10-13T16:14:37.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:37.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.623763+0000) 2019-10-13T16:14:37.626 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:37.620+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.624636+0000) 2019-10-13T16:14:37.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:37.626+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:37.632446+0000) 2019-10-13T16:14:37.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:37.628+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239463 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:42.639 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:42.631+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.636371+0000) 2019-10-13T16:14:42.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:42.631+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.636772+0000) 2019-10-13T16:14:42.640 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:42.631+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.637985+0000) 2019-10-13T16:14:42.640 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:42.631+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.638277+0000) 2019-10-13T16:14:42.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:42.631+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.638776+0000) 2019-10-13T16:14:42.641 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:42.631+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.638938+0000) 2019-10-13T16:14:42.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:42.634+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.639551+0000) 2019-10-13T16:14:42.649 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:42.642+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.647085+0000) 2019-10-13T16:14:42.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:42.643+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:42.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:42.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:42.764345+0000) 2019-10-13T16:14:43.501 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:43.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:43.499532+0000) 2019-10-13T16:14:43.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:43.746+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:43.748588+0000) 2019-10-13T16:14:45.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:45.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:45.774651+0000) 2019-10-13T16:14:45.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:45.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:45.774717+0000) 2019-10-13T16:14:45.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:45.779+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:45.785019+0000) 2019-10-13T16:14:46.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:46.323+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:46.326768+0000) 2019-10-13T16:14:46.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:46.694+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:46.696312+0000) 2019-10-13T16:14:47.653 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:47.647+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.650735+0000) 2019-10-13T16:14:47.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:47.647+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.651837+0000) 2019-10-13T16:14:47.654 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:47.647+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.652173+0000) 2019-10-13T16:14:47.654 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:47.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.652450+0000) 2019-10-13T16:14:47.655 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:47.647+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.653720+0000) 2019-10-13T16:14:47.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:47.647+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.654126+0000) 2019-10-13T16:14:47.656 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:47.647+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.654662+0000) 2019-10-13T16:14:47.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:47.654+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:47.661978+0000) 2019-10-13T16:14:47.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:47.659+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239531 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:50.570 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:14:50.570 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:14:50.574 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:14:50.574 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:14:52.669 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:52.662+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.667451+0000) 2019-10-13T16:14:52.670 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:52.663+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.668040+0000) 2019-10-13T16:14:52.670 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:52.663+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.668030+0000) 2019-10-13T16:14:52.671 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:52.663+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.668580+0000) 2019-10-13T16:14:52.671 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:52.663+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.668832+0000) 2019-10-13T16:14:52.672 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:52.663+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.669133+0000) 2019-10-13T16:14:52.672 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:52.663+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.669485+0000) 2019-10-13T16:14:52.678 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:52.670+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.676043+0000) 2019-10-13T16:14:52.681 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:52.675+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239562 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:14:52.766 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:52.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:52.764568+0000) 2019-10-13T16:14:53.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:53.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:53.499706+0000) 2019-10-13T16:14:53.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:53.746+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:53.748788+0000) 2019-10-13T16:14:55.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:55.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:55.774858+0000) 2019-10-13T16:14:55.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:55.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:55.774946+0000) 2019-10-13T16:14:55.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:55.783+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:55.785223+0000) 2019-10-13T16:14:56.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:56.323+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:56.326991+0000) 2019-10-13T16:14:56.705 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:56.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:56.702660+0000) 2019-10-13T16:14:57.685 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:14:57.675+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.682895+0000) 2019-10-13T16:14:57.685 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:14:57.679+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.683520+0000) 2019-10-13T16:14:57.686 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:14:57.679+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.683863+0000) 2019-10-13T16:14:57.686 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:14:57.679+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.684233+0000) 2019-10-13T16:14:57.686 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:14:57.678+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.684524+0000) 2019-10-13T16:14:57.687 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:14:57.679+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.684763+0000) 2019-10-13T16:14:57.687 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:14:57.679+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.685696+0000) 2019-10-13T16:14:57.691 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:14:57.682+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:14:57.689728+0000) 2019-10-13T16:14:57.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:14:57.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:02.702 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:02.695+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.699681+0000) 2019-10-13T16:15:02.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:02.695+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.700198+0000) 2019-10-13T16:15:02.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:02.695+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.700490+0000) 2019-10-13T16:15:02.703 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:02.694+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.700901+0000) 2019-10-13T16:15:02.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:02.695+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.701972+0000) 2019-10-13T16:15:02.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:02.695+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.702663+0000) 2019-10-13T16:15:02.705 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:02.695+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.702909+0000) 2019-10-13T16:15:02.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:02.706+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.711440+0000) 2019-10-13T16:15:02.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:02.707+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239631 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:02.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:02.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:02.764751+0000) 2019-10-13T16:15:03.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:03.495+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:03.499893+0000) 2019-10-13T16:15:03.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:03.746+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:03.748983+0000) 2019-10-13T16:15:05.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:05.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:05.775055+0000) 2019-10-13T16:15:05.777 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:05.771+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:05.775140+0000) 2019-10-13T16:15:05.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:05.783+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:05.785422+0000) 2019-10-13T16:15:06.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:06.323+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:06.327186+0000) 2019-10-13T16:15:06.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:06.702+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:06.709094+0000) 2019-10-13T16:15:07.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:07.711+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.715676+0000) 2019-10-13T16:15:07.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:07.711+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.716268+0000) 2019-10-13T16:15:07.718 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:07.711+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.716707+0000) 2019-10-13T16:15:07.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:07.710+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.717278+0000) 2019-10-13T16:15:07.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:07.711+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.717595+0000) 2019-10-13T16:15:07.720 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:07.711+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.718163+0000) 2019-10-13T16:15:07.720 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:07.711+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.718675+0000) 2019-10-13T16:15:07.727 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:07.718+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:07.724810+0000) 2019-10-13T16:15:07.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:07.723+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:12.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:12.731+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.731703+0000) 2019-10-13T16:15:12.734 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:12.731+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.732663+0000) 2019-10-13T16:15:12.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:12.726+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.732948+0000) 2019-10-13T16:15:12.736 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:12.731+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.734144+0000) 2019-10-13T16:15:12.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:12.731+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.734312+0000) 2019-10-13T16:15:12.737 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:12.731+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.734437+0000) 2019-10-13T16:15:12.737 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:12.731+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.734792+0000) 2019-10-13T16:15:12.740 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:12.734+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.738741+0000) 2019-10-13T16:15:12.746 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:12.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239698 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:12.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:12.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:12.764905+0000) 2019-10-13T16:15:13.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:13.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:13.500100+0000) 2019-10-13T16:15:13.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:13.746+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:13.749180+0000) 2019-10-13T16:15:15.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:15.771+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:15.775246+0000) 2019-10-13T16:15:15.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:15.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:15.775364+0000) 2019-10-13T16:15:15.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:15.783+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:15.785611+0000) 2019-10-13T16:15:16.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:16.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:16.327386+0000) 2019-10-13T16:15:16.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:16.714+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:16.714980+0000) 2019-10-13T16:15:17.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:17.746+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.747154+0000) 2019-10-13T16:15:17.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:17.747+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.748287+0000) 2019-10-13T16:15:17.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:17.747+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.748554+0000) 2019-10-13T16:15:17.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:17.747+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.749127+0000) 2019-10-13T16:15:17.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:17.747+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.749463+0000) 2019-10-13T16:15:17.753 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:17.747+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.750748+0000) 2019-10-13T16:15:17.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:17.747+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.750970+0000) 2019-10-13T16:15:17.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:17.754+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:17.756194+0000) 2019-10-13T16:15:17.761 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:17.755+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:20.698 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:15:20.699 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:15:20.702 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:15:20.703 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:15:22.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:22.759+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.762626+0000) 2019-10-13T16:15:22.765 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:22.761+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.763532+0000) 2019-10-13T16:15:22.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:22.763+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.763882+0000) 2019-10-13T16:15:22.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:22.763+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.764431+0000) 2019-10-13T16:15:22.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:22.763+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.764921+0000) 2019-10-13T16:15:22.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:22.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.765034+0000) 2019-10-13T16:15:22.767 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:22.763+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.765557+0000) 2019-10-13T16:15:22.768 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:22.763+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.766450+0000) 2019-10-13T16:15:22.771 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:22.765+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:22.769332+0000) 2019-10-13T16:15:22.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:22.771+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239767 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:23.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:23.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:23.500269+0000) 2019-10-13T16:15:23.751 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:23.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:23.749377+0000) 2019-10-13T16:15:25.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:25.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:25.775506+0000) 2019-10-13T16:15:25.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:25.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:25.775587+0000) 2019-10-13T16:15:25.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:25.783+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:25.785845+0000) 2019-10-13T16:15:26.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:26.326+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:26.327540+0000) 2019-10-13T16:15:26.723 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:26.717+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:26.721405+0000) 2019-10-13T16:15:27.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:27.774+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.778163+0000) 2019-10-13T16:15:27.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:27.774+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.778534+0000) 2019-10-13T16:15:27.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:27.778+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.779371+0000) 2019-10-13T16:15:27.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:27.777+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.779760+0000) 2019-10-13T16:15:27.783 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:27.778+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.781036+0000) 2019-10-13T16:15:27.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:27.778+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.781359+0000) 2019-10-13T16:15:27.784 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:27.778+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.782279+0000) 2019-10-13T16:15:27.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:27.785+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:27.786836+0000) 2019-10-13T16:15:27.792 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:27.786+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239803 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:32.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:32.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.765221+0000) 2019-10-13T16:15:32.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:32.790+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.793581+0000) 2019-10-13T16:15:32.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:32.790+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.794197+0000) 2019-10-13T16:15:32.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:32.790+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.794457+0000) 2019-10-13T16:15:32.797 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:32.790+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.794909+0000) 2019-10-13T16:15:32.798 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:32.793+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.795172+0000) 2019-10-13T16:15:32.799 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:32.794+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.796790+0000) 2019-10-13T16:15:32.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:32.794+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.797131+0000) 2019-10-13T16:15:32.802 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:32.797+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:32.800387+0000) 2019-10-13T16:15:32.808 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:32.802+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239834 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:33.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:33.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:33.500464+0000) 2019-10-13T16:15:33.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:33.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:33.749564+0000) 2019-10-13T16:15:35.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:35.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:35.775711+0000) 2019-10-13T16:15:35.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:35.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:35.775808+0000) 2019-10-13T16:15:35.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:35.782+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:35.786041+0000) 2019-10-13T16:15:36.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:36.326+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:36.327765+0000) 2019-10-13T16:15:36.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:36.725+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:36.727429+0000) 2019-10-13T16:15:37.812 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:37.806+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.809550+0000) 2019-10-13T16:15:37.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:37.806+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.810065+0000) 2019-10-13T16:15:37.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:37.806+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.810801+0000) 2019-10-13T16:15:37.814 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:37.809+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.810493+0000) 2019-10-13T16:15:37.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:37.810+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.811493+0000) 2019-10-13T16:15:37.815 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:37.810+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.812079+0000) 2019-10-13T16:15:37.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:37.810+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.813307+0000) 2019-10-13T16:15:37.820 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:37.813+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:37.817931+0000) 2019-10-13T16:15:37.822 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:37.818+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:42.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:42.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.765423+0000) 2019-10-13T16:15:42.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:42.822+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.824153+0000) 2019-10-13T16:15:42.827 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:42.821+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.825206+0000) 2019-10-13T16:15:42.828 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:42.822+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.825653+0000) 2019-10-13T16:15:42.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:42.822+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.825958+0000) 2019-10-13T16:15:42.828 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:42.822+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.826723+0000) 2019-10-13T16:15:42.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:42.822+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.827140+0000) 2019-10-13T16:15:42.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:42.826+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.827843+0000) 2019-10-13T16:15:42.837 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:42.833+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:42.834759+0000) 2019-10-13T16:15:42.838 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:42.834+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:43.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:43.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:43.500629+0000) 2019-10-13T16:15:43.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:43.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:43.749713+0000) 2019-10-13T16:15:45.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:45.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:45.775911+0000) 2019-10-13T16:15:45.778 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:45.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:45.776004+0000) 2019-10-13T16:15:45.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:45.782+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:45.786258+0000) 2019-10-13T16:15:46.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:46.326+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:46.327918+0000) 2019-10-13T16:15:46.739 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:46.733+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:46.736681+0000) 2019-10-13T16:15:47.842 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:47.838+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.840091+0000) 2019-10-13T16:15:47.843 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:47.838+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.841588+0000) 2019-10-13T16:15:47.844 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:47.837+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.842011+0000) 2019-10-13T16:15:47.845 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:47.838+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.842602+0000) 2019-10-13T16:15:47.845 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:47.838+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.842968+0000) 2019-10-13T16:15:47.846 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:47.842+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.843780+0000) 2019-10-13T16:15:47.846 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:47.842+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.844576+0000) 2019-10-13T16:15:47.853 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:47.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239939 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:47.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:47.849+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:47.852307+0000) 2019-10-13T16:15:50.815 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:15:50.816 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:15:50.819 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:15:50.819 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:15:52.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:52.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.765624+0000) 2019-10-13T16:15:52.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:52.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.854456+0000) 2019-10-13T16:15:52.857 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:52.850+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.855203+0000) 2019-10-13T16:15:52.858 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:52.854+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.855914+0000) 2019-10-13T16:15:52.858 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:52.853+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.856789+0000) 2019-10-13T16:15:52.859 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:52.854+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.857299+0000) 2019-10-13T16:15:52.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:52.854+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.857496+0000) 2019-10-13T16:15:52.860 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:52.854+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.857884+0000) 2019-10-13T16:15:52.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:52.861+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:52.864818+0000) 2019-10-13T16:15:52.868 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:52.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 239970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:15:53.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:53.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:53.500804+0000) 2019-10-13T16:15:53.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:53.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:53.749900+0000) 2019-10-13T16:15:55.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:55.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:55.776119+0000) 2019-10-13T16:15:55.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:55.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:55.776233+0000) 2019-10-13T16:15:55.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:55.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:55.786450+0000) 2019-10-13T16:15:56.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:56.326+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:56.328117+0000) 2019-10-13T16:15:56.745 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:56.741+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:56.743107+0000) 2019-10-13T16:15:57.871 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:15:57.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.869381+0000) 2019-10-13T16:15:57.873 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:15:57.869+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.870945+0000) 2019-10-13T16:15:57.873 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:15:57.870+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.871473+0000) 2019-10-13T16:15:57.873 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:15:57.870+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.871709+0000) 2019-10-13T16:15:57.874 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:15:57.870+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.872222+0000) 2019-10-13T16:15:57.875 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:15:57.870+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.872736+0000) 2019-10-13T16:15:57.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:15:57.870+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.873195+0000) 2019-10-13T16:15:57.881 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:15:57.877+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:15:57.879458+0000) 2019-10-13T16:15:57.883 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:15:57.878+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240008 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:02.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:02.762+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.765841+0000) 2019-10-13T16:16:02.888 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:02.882+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.885281+0000) 2019-10-13T16:16:02.888 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:02.882+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.885641+0000) 2019-10-13T16:16:02.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:02.882+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.886244+0000) 2019-10-13T16:16:02.889 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:02.885+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.887568+0000) 2019-10-13T16:16:02.890 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:02.886+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.888014+0000) 2019-10-13T16:16:02.890 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:02.886+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.888490+0000) 2019-10-13T16:16:02.891 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:02.886+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.889851+0000) 2019-10-13T16:16:02.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:02.893+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:02.895016+0000) 2019-10-13T16:16:02.898 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:02.894+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:03.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:03.498+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:03.500994+0000) 2019-10-13T16:16:03.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:03.745+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:03.750066+0000) 2019-10-13T16:16:05.778 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:05.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:05.776348+0000) 2019-10-13T16:16:05.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:05.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:05.776464+0000) 2019-10-13T16:16:05.788 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:05.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:05.786644+0000) 2019-10-13T16:16:06.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:06.326+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:06.328321+0000) 2019-10-13T16:16:06.752 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:06.745+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:06.749553+0000) 2019-10-13T16:16:07.903 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:07.898+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.901061+0000) 2019-10-13T16:16:07.904 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:07.898+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.901135+0000) 2019-10-13T16:16:07.905 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:07.901+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.902381+0000) 2019-10-13T16:16:07.905 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:07.898+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.901932+0000) 2019-10-13T16:16:07.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:07.898+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.902822+0000) 2019-10-13T16:16:07.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:07.902+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.903743+0000) 2019-10-13T16:16:07.907 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:07.902+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.903964+0000) 2019-10-13T16:16:07.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:07.905+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:07.906387+0000) 2019-10-13T16:16:07.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:07.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:12.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:12.765+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.766049+0000) 2019-10-13T16:16:12.916 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:12.909+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.914061+0000) 2019-10-13T16:16:12.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:12.909+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.914755+0000) 2019-10-13T16:16:12.919 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:12.913+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.917281+0000) 2019-10-13T16:16:12.920 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:12.913+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.917939+0000) 2019-10-13T16:16:12.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:12.917+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.918367+0000) 2019-10-13T16:16:12.921 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:12.913+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.918719+0000) 2019-10-13T16:16:12.921 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:12.913+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.919124+0000) 2019-10-13T16:16:12.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:12.925+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:12.926436+0000) 2019-10-13T16:16:12.929 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:12.925+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240104 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:13.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:13.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:13.501168+0000) 2019-10-13T16:16:13.752 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:13.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:13.750238+0000) 2019-10-13T16:16:15.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:15.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:15.776552+0000) 2019-10-13T16:16:15.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:15.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:15.776642+0000) 2019-10-13T16:16:15.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:15.781+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:15.786840+0000) 2019-10-13T16:16:16.330 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:16.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:16.328566+0000) 2019-10-13T16:16:16.758 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:16.752+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:16.755892+0000) 2019-10-13T16:16:17.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:17.925+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.930554+0000) 2019-10-13T16:16:17.933 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:17.928+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.931049+0000) 2019-10-13T16:16:17.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:17.929+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.932127+0000) 2019-10-13T16:16:17.935 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:17.929+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.933346+0000) 2019-10-13T16:16:17.935 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:17.929+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.933474+0000) 2019-10-13T16:16:17.936 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:17.929+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.933872+0000) 2019-10-13T16:16:17.936 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:17.929+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.934763+0000) 2019-10-13T16:16:17.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:17.936+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:17.940572+0000) 2019-10-13T16:16:17.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:17.937+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:20.938 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:16:20.938 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:16:20.942 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:16:20.942 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:16:22.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:22.761+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.766285+0000) 2019-10-13T16:16:22.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:22.941+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.945284+0000) 2019-10-13T16:16:22.948 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:22.944+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.946199+0000) 2019-10-13T16:16:22.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:22.941+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.946632+0000) 2019-10-13T16:16:22.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:22.941+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.947101+0000) 2019-10-13T16:16:22.949 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:22.945+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.947474+0000) 2019-10-13T16:16:22.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:22.945+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.948572+0000) 2019-10-13T16:16:22.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:22.945+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.949070+0000) 2019-10-13T16:16:22.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:22.948+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:22.953922+0000) 2019-10-13T16:16:22.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:22.953+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240172 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:23.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:23.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:23.501344+0000) 2019-10-13T16:16:23.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:23.748+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:23.750414+0000) 2019-10-13T16:16:25.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:25.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:25.776752+0000) 2019-10-13T16:16:25.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:25.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:25.776863+0000) 2019-10-13T16:16:25.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:25.785+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:25.787039+0000) 2019-10-13T16:16:26.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:26.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:26.328760+0000) 2019-10-13T16:16:26.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:26.760+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:26.762300+0000) 2019-10-13T16:16:27.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:27.957+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.962188+0000) 2019-10-13T16:16:27.966 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:27.960+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.962623+0000) 2019-10-13T16:16:27.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:27.957+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.963013+0000) 2019-10-13T16:16:27.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:27.961+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.963628+0000) 2019-10-13T16:16:27.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:27.961+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.963785+0000) 2019-10-13T16:16:27.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:27.961+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.964135+0000) 2019-10-13T16:16:27.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:27.961+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.964944+0000) 2019-10-13T16:16:27.969 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:27.964+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:27.967408+0000) 2019-10-13T16:16:27.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:27.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240208 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:32.768 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:32.765+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.766512+0000) 2019-10-13T16:16:32.979 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:32.972+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.977138+0000) 2019-10-13T16:16:32.980 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:32.973+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.977860+0000) 2019-10-13T16:16:32.980 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:32.973+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.978667+0000) 2019-10-13T16:16:32.981 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:32.973+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.978906+0000) 2019-10-13T16:16:32.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:32.977+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.979583+0000) 2019-10-13T16:16:32.982 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:32.977+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.980333+0000) 2019-10-13T16:16:32.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:32.977+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.980826+0000) 2019-10-13T16:16:32.988 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:32.980+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:32.986002+0000) 2019-10-13T16:16:32.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:32.985+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:33.503 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:33.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:33.501519+0000) 2019-10-13T16:16:33.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:33.748+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:33.750604+0000) 2019-10-13T16:16:35.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:35.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:35.776979+0000) 2019-10-13T16:16:35.779 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:35.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:35.777086+0000) 2019-10-13T16:16:35.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:35.785+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:35.787253+0000) 2019-10-13T16:16:36.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:36.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:36.328956+0000) 2019-10-13T16:16:36.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:36.772+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:36.774026+0000) 2019-10-13T16:16:37.996 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:37.989+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.993516+0000) 2019-10-13T16:16:37.996 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:37.989+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.993666+0000) 2019-10-13T16:16:37.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:37.989+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.994570+0000) 2019-10-13T16:16:37.997 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:37.989+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.994799+0000) 2019-10-13T16:16:37.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:37.992+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.995177+0000) 2019-10-13T16:16:37.998 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:37.993+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.995484+0000) 2019-10-13T16:16:37.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:37.993+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:37.996105+0000) 2019-10-13T16:16:38.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:38.000+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:38.004790+0000) 2019-10-13T16:16:38.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:38.001+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240276 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:42.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:42.761+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:42.766705+0000) 2019-10-13T16:16:43.011 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:43.005+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.008475+0000) 2019-10-13T16:16:43.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:43.005+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.009133+0000) 2019-10-13T16:16:43.011 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:43.005+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.009414+0000) 2019-10-13T16:16:43.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:43.005+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.010280+0000) 2019-10-13T16:16:43.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:43.005+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.010989+0000) 2019-10-13T16:16:43.013 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:43.008+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.011304+0000) 2019-10-13T16:16:43.013 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:43.009+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.011786+0000) 2019-10-13T16:16:43.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:43.016+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.018841+0000) 2019-10-13T16:16:43.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:43.017+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240306 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:43.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:43.497+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.501689+0000) 2019-10-13T16:16:43.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:43.748+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:43.750796+0000) 2019-10-13T16:16:45.779 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:45.773+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:45.777173+0000) 2019-10-13T16:16:45.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:45.773+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:45.777288+0000) 2019-10-13T16:16:45.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:45.785+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:45.787503+0000) 2019-10-13T16:16:46.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:46.325+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:46.329193+0000) 2019-10-13T16:16:46.783 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:46.776+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:46.780433+0000) 2019-10-13T16:16:48.025 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:48.020+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.023524+0000) 2019-10-13T16:16:48.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:48.021+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.024055+0000) 2019-10-13T16:16:48.026 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:48.021+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.024433+0000) 2019-10-13T16:16:48.027 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:48.021+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.025316+0000) 2019-10-13T16:16:48.027 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:48.021+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.025909+0000) 2019-10-13T16:16:48.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:48.021+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.026625+0000) 2019-10-13T16:16:48.029 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:48.021+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.027081+0000) 2019-10-13T16:16:48.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:48.028+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:48.032403+0000) 2019-10-13T16:16:48.036 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:48.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240342 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:51.057 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:16:51.057 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:16:51.061 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:16:51.061 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:16:52.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:52.765+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:52.766912+0000) 2019-10-13T16:16:53.040 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:53.036+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.038221+0000) 2019-10-13T16:16:53.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:53.033+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.038670+0000) 2019-10-13T16:16:53.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:53.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.038956+0000) 2019-10-13T16:16:53.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:53.037+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.039402+0000) 2019-10-13T16:16:53.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:53.037+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.040231+0000) 2019-10-13T16:16:53.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:53.037+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.040779+0000) 2019-10-13T16:16:53.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:53.037+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.040943+0000) 2019-10-13T16:16:53.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:53.044+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.046914+0000) 2019-10-13T16:16:53.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:53.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240371 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:16:53.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:53.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.501890+0000) 2019-10-13T16:16:53.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:53.748+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:53.750966+0000) 2019-10-13T16:16:55.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:55.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:55.777371+0000) 2019-10-13T16:16:55.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:55.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:55.777497+0000) 2019-10-13T16:16:55.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:55.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:55.787704+0000) 2019-10-13T16:16:56.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:56.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:56.329420+0000) 2019-10-13T16:16:56.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:56.784+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:56.786745+0000) 2019-10-13T16:16:58.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:16:58.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.054409+0000) 2019-10-13T16:16:58.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:16:58.048+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.054819+0000) 2019-10-13T16:16:58.057 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:16:58.052+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.055713+0000) 2019-10-13T16:16:58.058 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:16:58.052+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.056372+0000) 2019-10-13T16:16:58.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:16:58.052+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.056648+0000) 2019-10-13T16:16:58.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:16:58.052+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.057074+0000) 2019-10-13T16:16:58.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:16:58.052+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.058294+0000) 2019-10-13T16:16:58.066 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:16:58.060+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:16:58.063991+0000) 2019-10-13T16:16:58.069 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:16:58.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240410 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:02.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:02.760+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:02.767116+0000) 2019-10-13T16:17:03.073 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:03.064+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.070475+0000) 2019-10-13T16:17:03.073 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:03.068+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.070551+0000) 2019-10-13T16:17:03.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:03.068+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.071804+0000) 2019-10-13T16:17:03.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:03.068+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.071918+0000) 2019-10-13T16:17:03.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:03.068+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.072595+0000) 2019-10-13T16:17:03.075 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:03.068+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.072823+0000) 2019-10-13T16:17:03.075 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:03.068+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.073409+0000) 2019-10-13T16:17:03.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:03.072+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.077763+0000) 2019-10-13T16:17:03.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:03.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240439 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:03.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:03.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.502097+0000) 2019-10-13T16:17:03.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:03.748+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:03.751169+0000) 2019-10-13T16:17:05.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:05.772+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:05.777543+0000) 2019-10-13T16:17:05.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:05.772+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:05.777683+0000) 2019-10-13T16:17:05.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:05.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:05.787854+0000) 2019-10-13T16:17:06.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:06.328+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:06.329624+0000) 2019-10-13T16:17:06.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:06.792+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:06.793020+0000) 2019-10-13T16:17:08.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:08.080+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.086652+0000) 2019-10-13T16:17:08.089 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:08.080+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.087155+0000) 2019-10-13T16:17:08.090 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:08.084+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.087449+0000) 2019-10-13T16:17:08.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:08.084+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.087911+0000) 2019-10-13T16:17:08.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:08.084+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.088284+0000) 2019-10-13T16:17:08.092 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:08.084+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.089704+0000) 2019-10-13T16:17:08.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:08.084+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.090769+0000) 2019-10-13T16:17:08.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:08.092+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:08.097304+0000) 2019-10-13T16:17:08.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:08.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240476 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:12.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:12.764+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:12.767355+0000) 2019-10-13T16:17:13.104 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:13.096+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.101739+0000) 2019-10-13T16:17:13.104 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:13.096+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.102580+0000) 2019-10-13T16:17:13.105 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:13.096+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.102946+0000) 2019-10-13T16:17:13.105 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:13.100+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.103347+0000) 2019-10-13T16:17:13.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:13.100+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.104480+0000) 2019-10-13T16:17:13.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:13.100+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.103984+0000) 2019-10-13T16:17:13.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:13.099+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.104515+0000) 2019-10-13T16:17:13.113 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:13.107+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.111604+0000) 2019-10-13T16:17:13.116 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:13.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240505 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:13.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:13.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.502274+0000) 2019-10-13T16:17:13.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:13.747+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:13.751343+0000) 2019-10-13T16:17:15.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:15.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:15.777744+0000) 2019-10-13T16:17:15.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:15.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:15.777848+0000) 2019-10-13T16:17:15.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:15.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:15.788048+0000) 2019-10-13T16:17:16.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:16.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:16.329820+0000) 2019-10-13T16:17:16.801 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:16.795+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:16.799191+0000) 2019-10-13T16:17:18.121 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:18.112+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.118701+0000) 2019-10-13T16:17:18.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:18.112+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.118976+0000) 2019-10-13T16:17:18.121 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:18.116+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.119405+0000) 2019-10-13T16:17:18.122 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:18.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.119931+0000) 2019-10-13T16:17:18.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:18.116+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.119900+0000) 2019-10-13T16:17:18.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:18.116+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.120354+0000) 2019-10-13T16:17:18.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:18.116+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.121465+0000) 2019-10-13T16:17:18.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:18.123+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:18.128266+0000) 2019-10-13T16:17:18.131 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:18.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240544 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:21.160 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:17:21.160 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:17:21.164 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:17:21.164 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:17:22.769 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:22.764+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:22.767551+0000) 2019-10-13T16:17:23.134 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:23.128+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.132305+0000) 2019-10-13T16:17:23.135 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:23.128+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.132796+0000) 2019-10-13T16:17:23.135 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:23.128+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.133342+0000) 2019-10-13T16:17:23.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:23.128+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.134677+0000) 2019-10-13T16:17:23.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:23.131+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.135058+0000) 2019-10-13T16:17:23.137 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:23.132+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.135581+0000) 2019-10-13T16:17:23.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:23.132+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.135794+0000) 2019-10-13T16:17:23.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:23.135+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.140355+0000) 2019-10-13T16:17:23.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:23.140+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240573 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:23.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:23.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.502469+0000) 2019-10-13T16:17:23.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:23.747+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:23.751514+0000) 2019-10-13T16:17:25.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:25.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:25.777944+0000) 2019-10-13T16:17:25.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:25.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:25.778011+0000) 2019-10-13T16:17:25.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:25.784+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:25.788243+0000) 2019-10-13T16:17:26.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:26.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:26.330043+0000) 2019-10-13T16:17:26.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:26.803+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:26.805546+0000) 2019-10-13T16:17:28.150 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:28.143+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.147877+0000) 2019-10-13T16:17:28.151 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:28.144+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.149212+0000) 2019-10-13T16:17:28.152 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:28.144+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.149954+0000) 2019-10-13T16:17:28.152 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:28.144+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.150134+0000) 2019-10-13T16:17:28.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:28.144+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.150996+0000) 2019-10-13T16:17:28.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:28.148+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.151681+0000) 2019-10-13T16:17:28.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:28.144+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.150560+0000) 2019-10-13T16:17:28.157 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:28.151+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:28.154965+0000) 2019-10-13T16:17:28.168 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:28.156+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240609 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:32.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:32.764+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:32.767753+0000) 2019-10-13T16:17:33.166 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:33.160+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.164328+0000) 2019-10-13T16:17:33.167 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:33.159+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.164521+0000) 2019-10-13T16:17:33.168 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:33.160+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.165038+0000) 2019-10-13T16:17:33.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:33.160+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.165270+0000) 2019-10-13T16:17:33.168 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:33.160+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.165533+0000) 2019-10-13T16:17:33.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:33.160+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.166532+0000) 2019-10-13T16:17:33.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:33.160+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.166840+0000) 2019-10-13T16:17:33.174 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:33.167+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.172078+0000) 2019-10-13T16:17:33.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:33.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240639 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:33.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:33.496+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.502671+0000) 2019-10-13T16:17:33.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:33.751+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:33.751720+0000) 2019-10-13T16:17:35.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:35.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:35.778179+0000) 2019-10-13T16:17:35.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:35.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:35.778180+0000) 2019-10-13T16:17:35.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:35.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:35.788476+0000) 2019-10-13T16:17:36.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:36.324+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:36.330268+0000) 2019-10-13T16:17:36.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:36.811+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:36.811850+0000) 2019-10-13T16:17:38.180 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:38.171+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.178242+0000) 2019-10-13T16:17:38.181 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:38.175+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.178815+0000) 2019-10-13T16:17:38.181 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:38.171+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.179104+0000) 2019-10-13T16:17:38.182 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:38.175+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.180130+0000) 2019-10-13T16:17:38.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:38.175+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.180322+0000) 2019-10-13T16:17:38.183 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:38.175+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.180680+0000) 2019-10-13T16:17:38.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:38.175+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.180899+0000) 2019-10-13T16:17:38.189 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:38.183+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:38.187099+0000) 2019-10-13T16:17:38.192 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:38.183+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:42.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:42.767+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:42.767952+0000) 2019-10-13T16:17:43.195 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:43.187+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.193181+0000) 2019-10-13T16:17:43.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:43.187+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.193892+0000) 2019-10-13T16:17:43.196 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:43.187+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.194212+0000) 2019-10-13T16:17:43.197 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:43.191+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.195549+0000) 2019-10-13T16:17:43.198 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:43.191+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.195588+0000) 2019-10-13T16:17:43.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:43.191+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.195876+0000) 2019-10-13T16:17:43.198 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:43.191+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.196394+0000) 2019-10-13T16:17:43.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:43.195+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.200851+0000) 2019-10-13T16:17:43.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:43.199+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:43.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:43.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.502851+0000) 2019-10-13T16:17:43.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:43.751+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:43.751923+0000) 2019-10-13T16:17:45.780 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:45.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:45.778402+0000) 2019-10-13T16:17:45.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:45.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:45.778402+0000) 2019-10-13T16:17:45.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:45.787+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:45.788675+0000) 2019-10-13T16:17:46.332 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:46.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:46.330517+0000) 2019-10-13T16:17:46.823 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:46.819+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:46.820674+0000) 2019-10-13T16:17:48.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:48.203+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.209110+0000) 2019-10-13T16:17:48.211 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:48.203+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.209440+0000) 2019-10-13T16:17:48.212 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:48.203+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.209952+0000) 2019-10-13T16:17:48.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:48.203+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.210870+0000) 2019-10-13T16:17:48.213 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:48.203+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.210301+0000) 2019-10-13T16:17:48.213 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:48.207+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.211291+0000) 2019-10-13T16:17:48.214 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:48.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.212913+0000) 2019-10-13T16:17:48.220 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:48.215+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:48.218323+0000) 2019-10-13T16:17:48.224 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:48.215+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:51.273 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:17:51.273 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:17:51.277 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:17:51.277 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:17:52.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:52.763+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:52.768176+0000) 2019-10-13T16:17:53.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:53.223+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.226202+0000) 2019-10-13T16:17:53.229 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:53.219+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.226724+0000) 2019-10-13T16:17:53.229 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:53.223+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.227286+0000) 2019-10-13T16:17:53.230 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:53.223+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.227737+0000) 2019-10-13T16:17:53.230 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:53.223+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.228120+0000) 2019-10-13T16:17:53.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:53.223+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.228455+0000) 2019-10-13T16:17:53.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:53.223+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.229040+0000) 2019-10-13T16:17:53.235 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:53.227+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.233306+0000) 2019-10-13T16:17:53.240 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:53.231+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:17:53.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:53.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.503005+0000) 2019-10-13T16:17:53.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:53.747+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:53.752087+0000) 2019-10-13T16:17:55.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:55.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:55.778627+0000) 2019-10-13T16:17:55.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:55.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:55.778626+0000) 2019-10-13T16:17:55.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:55.787+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:55.788873+0000) 2019-10-13T16:17:56.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:56.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:56.330717+0000) 2019-10-13T16:17:56.829 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:56.823+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:56.826982+0000) 2019-10-13T16:17:58.243 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:17:58.235+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.240618+0000) 2019-10-13T16:17:58.243 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:17:58.235+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.241044+0000) 2019-10-13T16:17:58.244 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:17:58.235+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.242074+0000) 2019-10-13T16:17:58.245 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:17:58.235+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.242494+0000) 2019-10-13T16:17:58.245 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:17:58.235+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.242736+0000) 2019-10-13T16:17:58.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:17:58.239+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.243690+0000) 2019-10-13T16:17:58.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:17:58.239+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.244158+0000) 2019-10-13T16:17:58.252 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:17:58.247+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:17:58.250503+0000) 2019-10-13T16:17:58.255 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:17:58.247+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240811 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:02.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:02.767+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:02.768371+0000) 2019-10-13T16:18:03.259 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:03.251+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.256705+0000) 2019-10-13T16:18:03.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:03.251+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.257023+0000) 2019-10-13T16:18:03.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:03.251+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.257462+0000) 2019-10-13T16:18:03.260 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:03.251+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.257872+0000) 2019-10-13T16:18:03.261 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:03.255+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.259449+0000) 2019-10-13T16:18:03.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:03.255+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.259993+0000) 2019-10-13T16:18:03.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:03.255+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.260538+0000) 2019-10-13T16:18:03.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:03.259+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.264382+0000) 2019-10-13T16:18:03.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:03.263+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:03.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:03.499+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.503182+0000) 2019-10-13T16:18:03.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:03.751+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:03.752264+0000) 2019-10-13T16:18:05.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:05.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:05.778854+0000) 2019-10-13T16:18:05.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:05.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:05.778854+0000) 2019-10-13T16:18:05.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:05.787+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:05.789128+0000) 2019-10-13T16:18:06.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:06.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:06.330920+0000) 2019-10-13T16:18:06.835 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:06.830+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:06.833298+0000) 2019-10-13T16:18:08.273 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:08.271+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.271376+0000) 2019-10-13T16:18:08.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:08.271+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.272310+0000) 2019-10-13T16:18:08.275 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:08.271+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.273176+0000) 2019-10-13T16:18:08.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:08.271+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.273287+0000) 2019-10-13T16:18:08.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:08.271+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.273893+0000) 2019-10-13T16:18:08.276 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:08.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.274415+0000) 2019-10-13T16:18:08.277 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:08.271+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.274671+0000) 2019-10-13T16:18:08.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:08.274+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:08.278880+0000) 2019-10-13T16:18:08.286 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:08.283+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240877 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:12.770 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:12.767+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:12.768574+0000) 2019-10-13T16:18:13.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:13.287+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.287738+0000) 2019-10-13T16:18:13.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:13.282+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.288492+0000) 2019-10-13T16:18:13.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:13.287+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.288773+0000) 2019-10-13T16:18:13.291 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:13.287+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.289351+0000) 2019-10-13T16:18:13.292 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:13.287+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.289108+0000) 2019-10-13T16:18:13.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:13.287+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.290266+0000) 2019-10-13T16:18:13.292 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:13.287+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.289871+0000) 2019-10-13T16:18:13.299 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:13.290+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.297493+0000) 2019-10-13T16:18:13.303 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:13.299+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:13.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:13.503+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.503391+0000) 2019-10-13T16:18:13.754 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:13.750+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:13.752426+0000) 2019-10-13T16:18:15.781 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:15.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:15.779084+0000) 2019-10-13T16:18:15.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:15.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:15.779084+0000) 2019-10-13T16:18:15.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:15.787+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:15.789320+0000) 2019-10-13T16:18:16.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:16.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:16.331117+0000) 2019-10-13T16:18:16.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:16.838+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:16.839621+0000) 2019-10-13T16:18:18.307 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:18.303+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.304971+0000) 2019-10-13T16:18:18.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:18.298+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.305267+0000) 2019-10-13T16:18:18.308 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:18.303+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.305934+0000) 2019-10-13T16:18:18.308 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:18.303+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.305765+0000) 2019-10-13T16:18:18.309 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:18.303+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.306288+0000) 2019-10-13T16:18:18.309 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:18.303+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.306532+0000) 2019-10-13T16:18:18.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:18.307+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.307356+0000) 2019-10-13T16:18:18.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:18.306+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:18.313490+0000) 2019-10-13T16:18:18.318 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:18.315+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240945 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:21.374 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:18:21.375 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:18:21.378 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:18:21.378 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:18:22.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:22.766+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:22.768772+0000) 2019-10-13T16:18:23.322 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:23.318+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.320219+0000) 2019-10-13T16:18:23.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:23.318+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.320576+0000) 2019-10-13T16:18:23.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:23.314+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.321073+0000) 2019-10-13T16:18:23.324 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:23.318+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.321942+0000) 2019-10-13T16:18:23.324 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:23.318+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.320861+0000) 2019-10-13T16:18:23.324 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:23.318+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.322778+0000) 2019-10-13T16:18:23.325 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:23.322+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.323723+0000) 2019-10-13T16:18:23.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:23.322+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.328631+0000) 2019-10-13T16:18:23.334 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:23.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 240975 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:23.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:23.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.503532+0000) 2019-10-13T16:18:23.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:23.750+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:23.752611+0000) 2019-10-13T16:18:25.781 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:25.774+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:25.779265+0000) 2019-10-13T16:18:25.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:25.774+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:25.779266+0000) 2019-10-13T16:18:25.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:25.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:25.789517+0000) 2019-10-13T16:18:26.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:26.330+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:26.331332+0000) 2019-10-13T16:18:26.848 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:26.842+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:26.845991+0000) 2019-10-13T16:18:28.337 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:28.330+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.335261+0000) 2019-10-13T16:18:28.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:28.334+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.335965+0000) 2019-10-13T16:18:28.338 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:28.330+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.336313+0000) 2019-10-13T16:18:28.339 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:28.334+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.336934+0000) 2019-10-13T16:18:28.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:28.334+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.337207+0000) 2019-10-13T16:18:28.340 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:28.334+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.337531+0000) 2019-10-13T16:18:28.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:28.334+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.338929+0000) 2019-10-13T16:18:28.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:28.338+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:28.344815+0000) 2019-10-13T16:18:28.349 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:28.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241010 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:32.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:32.766+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:32.768989+0000) 2019-10-13T16:18:33.352 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:33.346+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.350130+0000) 2019-10-13T16:18:33.353 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:33.346+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.350728+0000) 2019-10-13T16:18:33.354 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:33.346+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.351232+0000) 2019-10-13T16:18:33.354 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:33.350+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.352008+0000) 2019-10-13T16:18:33.355 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:33.350+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.352289+0000) 2019-10-13T16:18:33.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:33.350+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.353209+0000) 2019-10-13T16:18:33.356 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:33.350+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.353765+0000) 2019-10-13T16:18:33.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:33.354+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.358783+0000) 2019-10-13T16:18:33.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:33.358+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241042 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:33.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:33.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.503731+0000) 2019-10-13T16:18:33.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:33.750+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:33.752777+0000) 2019-10-13T16:18:35.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:35.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:35.779530+0000) 2019-10-13T16:18:35.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:35.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:35.779530+0000) 2019-10-13T16:18:35.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:35.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:35.789712+0000) 2019-10-13T16:18:36.333 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:36.330+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:36.331564+0000) 2019-10-13T16:18:36.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:36.850+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:36.852270+0000) 2019-10-13T16:18:38.368 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:38.362+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.366266+0000) 2019-10-13T16:18:38.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:38.362+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.367154+0000) 2019-10-13T16:18:38.369 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:38.366+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.367436+0000) 2019-10-13T16:18:38.370 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:38.366+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.368067+0000) 2019-10-13T16:18:38.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:38.366+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.368763+0000) 2019-10-13T16:18:38.372 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:38.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.369698+0000) 2019-10-13T16:18:38.372 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:38.366+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.369397+0000) 2019-10-13T16:18:38.376 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:38.366+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:38.373958+0000) 2019-10-13T16:18:38.381 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:38.374+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:42.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:42.766+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:42.769184+0000) 2019-10-13T16:18:43.384 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:43.378+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.381619+0000) 2019-10-13T16:18:43.385 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:43.378+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.382716+0000) 2019-10-13T16:18:43.385 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:43.378+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.383083+0000) 2019-10-13T16:18:43.385 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:43.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.383424+0000) 2019-10-13T16:18:43.386 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:43.382+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.384144+0000) 2019-10-13T16:18:43.386 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:43.382+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.384326+0000) 2019-10-13T16:18:43.387 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:43.382+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.384601+0000) 2019-10-13T16:18:43.392 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:43.382+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.389978+0000) 2019-10-13T16:18:43.397 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:43.390+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241109 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:43.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:43.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.503914+0000) 2019-10-13T16:18:43.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:43.750+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:43.752949+0000) 2019-10-13T16:18:45.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:45.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:45.779726+0000) 2019-10-13T16:18:45.782 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:45.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:45.779724+0000) 2019-10-13T16:18:45.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:45.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:45.789940+0000) 2019-10-13T16:18:46.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:46.330+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:46.331814+0000) 2019-10-13T16:18:46.861 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:46.858+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:46.858557+0000) 2019-10-13T16:18:48.400 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:48.394+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.398432+0000) 2019-10-13T16:18:48.401 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:48.394+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.398844+0000) 2019-10-13T16:18:48.402 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:48.394+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.399182+0000) 2019-10-13T16:18:48.402 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:48.398+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.399551+0000) 2019-10-13T16:18:48.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:48.398+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.399998+0000) 2019-10-13T16:18:48.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:48.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.400894+0000) 2019-10-13T16:18:48.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:48.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.402330+0000) 2019-10-13T16:18:48.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:48.402+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:48.408456+0000) 2019-10-13T16:18:48.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:48.406+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241144 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:51.506 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:18:51.506 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:18:51.510 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:18:51.510 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:18:52.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:52.766+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:52.769397+0000) 2019-10-13T16:18:53.416 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:53.410+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.413987+0000) 2019-10-13T16:18:53.416 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:53.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.414322+0000) 2019-10-13T16:18:53.417 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:53.410+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.414429+0000) 2019-10-13T16:18:53.417 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:53.414+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.415427+0000) 2019-10-13T16:18:53.418 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:53.414+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.415730+0000) 2019-10-13T16:18:53.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:53.414+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.416353+0000) 2019-10-13T16:18:53.418 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:53.414+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.416766+0000) 2019-10-13T16:18:53.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:53.418+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.423593+0000) 2019-10-13T16:18:53.426 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:53.422+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241176 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:18:53.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:53.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.504101+0000) 2019-10-13T16:18:53.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:53.750+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:53.753118+0000) 2019-10-13T16:18:55.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:55.778+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:55.779990+0000) 2019-10-13T16:18:55.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:55.778+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:55.779990+0000) 2019-10-13T16:18:55.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:55.786+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:55.790132+0000) 2019-10-13T16:18:56.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:56.330+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:56.332051+0000) 2019-10-13T16:18:56.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:56.862+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:56.865023+0000) 2019-10-13T16:18:58.430 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:18:58.426+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.428068+0000) 2019-10-13T16:18:58.430 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:18:58.426+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.428426+0000) 2019-10-13T16:18:58.431 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:18:58.426+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.428962+0000) 2019-10-13T16:18:58.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:18:58.426+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.429430+0000) 2019-10-13T16:18:58.432 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:18:58.426+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.430205+0000) 2019-10-13T16:18:58.433 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:18:58.426+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.431189+0000) 2019-10-13T16:18:58.434 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:18:58.430+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.432281+0000) 2019-10-13T16:18:58.438 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:18:58.434+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:18:58.436750+0000) 2019-10-13T16:18:58.442 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:18:58.438+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241212 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:02.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:02.766+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:02.769594+0000) 2019-10-13T16:19:03.445 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:03.442+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.443417+0000) 2019-10-13T16:19:03.446 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:03.442+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.444340+0000) 2019-10-13T16:19:03.447 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:03.442+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.444827+0000) 2019-10-13T16:19:03.447 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:03.441+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.445274+0000) 2019-10-13T16:19:03.448 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:03.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.445957+0000) 2019-10-13T16:19:03.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:03.442+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.446476+0000) 2019-10-13T16:19:03.449 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:03.442+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.447240+0000) 2019-10-13T16:19:03.457 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:03.454+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241243 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:03.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:03.453+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.456768+0000) 2019-10-13T16:19:03.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:03.502+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.504296+0000) 2019-10-13T16:19:03.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:03.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:03.753295+0000) 2019-10-13T16:19:05.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:05.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:05.780213+0000) 2019-10-13T16:19:05.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:05.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:05.780212+0000) 2019-10-13T16:19:05.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:05.785+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:05.790358+0000) 2019-10-13T16:19:06.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:06.329+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:06.332268+0000) 2019-10-13T16:19:06.871 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:06.865+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:06.869255+0000) 2019-10-13T16:19:08.462 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:08.457+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.459809+0000) 2019-10-13T16:19:08.462 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:08.457+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.459965+0000) 2019-10-13T16:19:08.463 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:08.457+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.460356+0000) 2019-10-13T16:19:08.463 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:08.457+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.459463+0000) 2019-10-13T16:19:08.463 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:08.457+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.460630+0000) 2019-10-13T16:19:08.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:08.457+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.461965+0000) 2019-10-13T16:19:08.464 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:08.457+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.462333+0000) 2019-10-13T16:19:08.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:08.465+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:08.468781+0000) 2019-10-13T16:19:08.473 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:08.465+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241278 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:12.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:12.765+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:12.769796+0000) 2019-10-13T16:19:13.477 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:13.473+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.475078+0000) 2019-10-13T16:19:13.477 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:13.473+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.475848+0000) 2019-10-13T16:19:13.478 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:13.473+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.476706+0000) 2019-10-13T16:19:13.479 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:13.473+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.476948+0000) 2019-10-13T16:19:13.479 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:13.473+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.477300+0000) 2019-10-13T16:19:13.480 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:13.473+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.477839+0000) 2019-10-13T16:19:13.480 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:13.473+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.478598+0000) 2019-10-13T16:19:13.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:13.481+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.482531+0000) 2019-10-13T16:19:13.489 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:13.481+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241310 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:13.506 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:13.501+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.504432+0000) 2019-10-13T16:19:13.755 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:13.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:13.753502+0000) 2019-10-13T16:19:15.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:15.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:15.780393+0000) 2019-10-13T16:19:15.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:15.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:15.780392+0000) 2019-10-13T16:19:15.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:15.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:15.790559+0000) 2019-10-13T16:19:16.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:16.329+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:16.332464+0000) 2019-10-13T16:19:16.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:16.873+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:16.875854+0000) 2019-10-13T16:19:18.493 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:18.485+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.490729+0000) 2019-10-13T16:19:18.493 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:18.489+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.491516+0000) 2019-10-13T16:19:18.494 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:18.489+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.492179+0000) 2019-10-13T16:19:18.494 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:18.489+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.492671+0000) 2019-10-13T16:19:18.495 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:18.489+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.492916+0000) 2019-10-13T16:19:18.495 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:18.489+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.493266+0000) 2019-10-13T16:19:18.496 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:18.489+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.493842+0000) 2019-10-13T16:19:18.498 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:18.493+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:18.496109+0000) 2019-10-13T16:19:18.504 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:18.497+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241345 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:21.638 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:19:21.639 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:19:21.642 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:19:21.642 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:19:22.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:22.765+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:22.770030+0000) 2019-10-13T16:19:23.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:23.501+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.505970+0000) 2019-10-13T16:19:23.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:23.501+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.505638+0000) 2019-10-13T16:19:23.509 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:23.501+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.506325+0000) 2019-10-13T16:19:23.510 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:23.501+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.507254+0000) 2019-10-13T16:19:23.510 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:23.505+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.507531+0000) 2019-10-13T16:19:23.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:23.505+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.508068+0000) 2019-10-13T16:19:23.511 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:23.505+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.508457+0000) 2019-10-13T16:19:23.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:23.513+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.515431+0000) 2019-10-13T16:19:23.519 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:23.513+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241377 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:23.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:23.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:23.753705+0000) 2019-10-13T16:19:25.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:25.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:25.780618+0000) 2019-10-13T16:19:25.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:25.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:25.780618+0000) 2019-10-13T16:19:25.792 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:25.785+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:25.790757+0000) 2019-10-13T16:19:26.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:26.329+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:26.332660+0000) 2019-10-13T16:19:26.884 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:26.881+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:26.882221+0000) 2019-10-13T16:19:28.523 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:28.517+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.521528+0000) 2019-10-13T16:19:28.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:28.517+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.521853+0000) 2019-10-13T16:19:28.524 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:28.517+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.522590+0000) 2019-10-13T16:19:28.525 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:28.517+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.522820+0000) 2019-10-13T16:19:28.525 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:28.517+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.523042+0000) 2019-10-13T16:19:28.525 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:28.521+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.523469+0000) 2019-10-13T16:19:28.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:28.521+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.524644+0000) 2019-10-13T16:19:28.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:28.529+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:28.530382+0000) 2019-10-13T16:19:28.535 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:28.529+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:30.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:30.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:30.254974+0000) 2019-10-13T16:19:32.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:32.769+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:32.770226+0000) 2019-10-13T16:19:33.539 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:33.533+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.536880+0000) 2019-10-13T16:19:33.540 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:33.533+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.537308+0000) 2019-10-13T16:19:33.541 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:33.533+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.538705+0000) 2019-10-13T16:19:33.541 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:33.533+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.538923+0000) 2019-10-13T16:19:33.541 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:33.537+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.539716+0000) 2019-10-13T16:19:33.542 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:33.537+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.540282+0000) 2019-10-13T16:19:33.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:33.537+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.540567+0000) 2019-10-13T16:19:33.547 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:33.541+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.544939+0000) 2019-10-13T16:19:33.551 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:33.545+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:33.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:33.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:33.753877+0000) 2019-10-13T16:19:35.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:35.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:35.780841+0000) 2019-10-13T16:19:35.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:35.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:35.780841+0000) 2019-10-13T16:19:35.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:35.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:35.790950+0000) 2019-10-13T16:19:36.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:36.329+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:36.332859+0000) 2019-10-13T16:19:36.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:36.885+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:36.888593+0000) 2019-10-13T16:19:38.555 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:38.549+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.552875+0000) 2019-10-13T16:19:38.556 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:38.549+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.553488+0000) 2019-10-13T16:19:38.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:38.549+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.553993+0000) 2019-10-13T16:19:38.556 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:38.549+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.554343+0000) 2019-10-13T16:19:38.557 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:38.549+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.555143+0000) 2019-10-13T16:19:38.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:38.553+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.555080+0000) 2019-10-13T16:19:38.558 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:38.553+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.556109+0000) 2019-10-13T16:19:38.565 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:38.561+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:38.562827+0000) 2019-10-13T16:19:38.567 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:38.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:40.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:40.249+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:40.255187+0000) 2019-10-13T16:19:42.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:42.769+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:42.770449+0000) 2019-10-13T16:19:43.571 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:43.565+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.569097+0000) 2019-10-13T16:19:43.572 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:43.565+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.569793+0000) 2019-10-13T16:19:43.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:43.569+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.570320+0000) 2019-10-13T16:19:43.573 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:43.565+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.570081+0000) 2019-10-13T16:19:43.573 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:43.569+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.571774+0000) 2019-10-13T16:19:43.574 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:43.569+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.572036+0000) 2019-10-13T16:19:43.574 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:43.569+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.572431+0000) 2019-10-13T16:19:43.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:43.573+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.577158+0000) 2019-10-13T16:19:43.583 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:43.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241511 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:43.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:43.749+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:43.754046+0000) 2019-10-13T16:19:45.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:45.777+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:45.781071+0000) 2019-10-13T16:19:45.783 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:45.777+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:45.781071+0000) 2019-10-13T16:19:45.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:45.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:45.791143+0000) 2019-10-13T16:19:46.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:46.329+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:46.333058+0000) 2019-10-13T16:19:46.897 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:46.893+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:46.894839+0000) 2019-10-13T16:19:48.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:48.581+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.584614+0000) 2019-10-13T16:19:48.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:48.581+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.586091+0000) 2019-10-13T16:19:48.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:48.581+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.586389+0000) 2019-10-13T16:19:48.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:48.581+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.586700+0000) 2019-10-13T16:19:48.589 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:48.585+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.587657+0000) 2019-10-13T16:19:48.590 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:48.585+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.588000+0000) 2019-10-13T16:19:48.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:48.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.589126+0000) 2019-10-13T16:19:48.597 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:48.593+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:48.595193+0000) 2019-10-13T16:19:48.598 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:48.593+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241545 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:50.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:50.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:50.255405+0000) 2019-10-13T16:19:51.795 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:19:51.795 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:19:51.799 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:19:51.799 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:19:52.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:52.768+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:52.770646+0000) 2019-10-13T16:19:53.601 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:53.596+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.599293+0000) 2019-10-13T16:19:53.602 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:53.597+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.600525+0000) 2019-10-13T16:19:53.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:53.596+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.601155+0000) 2019-10-13T16:19:53.604 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:53.596+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.601742+0000) 2019-10-13T16:19:53.604 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:53.596+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.601880+0000) 2019-10-13T16:19:53.604 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:53.596+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.602449+0000) 2019-10-13T16:19:53.605 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:53.596+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.602649+0000) 2019-10-13T16:19:53.613 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:53.604+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241578 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:19:53.614 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:53.609+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.612158+0000) 2019-10-13T16:19:53.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:53.753+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:53.754232+0000) 2019-10-13T16:19:55.783 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:55.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:55.781297+0000) 2019-10-13T16:19:55.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:55.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:55.781297+0000) 2019-10-13T16:19:55.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:55.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:55.791403+0000) 2019-10-13T16:19:56.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:56.332+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:56.333257+0000) 2019-10-13T16:19:56.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:56.896+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:56.901190+0000) 2019-10-13T16:19:58.616 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:19:58.608+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.614097+0000) 2019-10-13T16:19:58.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:19:58.608+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.614839+0000) 2019-10-13T16:19:58.617 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:19:58.608+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.615111+0000) 2019-10-13T16:19:58.619 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:19:58.612+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.616246+0000) 2019-10-13T16:19:58.619 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:19:58.612+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.615915+0000) 2019-10-13T16:19:58.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:19:58.612+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.615807+0000) 2019-10-13T16:19:58.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:19:58.612+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.617018+0000) 2019-10-13T16:19:58.626 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:19:58.620+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:19:58.623810+0000) 2019-10-13T16:19:58.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:19:58.620+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:00.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:00.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:00.255601+0000) 2019-10-13T16:20:02.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:02.768+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:02.770882+0000) 2019-10-13T16:20:03.631 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:03.624+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.629414+0000) 2019-10-13T16:20:03.632 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:03.624+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.630198+0000) 2019-10-13T16:20:03.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:03.624+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.630533+0000) 2019-10-13T16:20:03.633 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:03.624+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.630844+0000) 2019-10-13T16:20:03.633 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:03.628+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.631606+0000) 2019-10-13T16:20:03.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:03.628+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.631628+0000) 2019-10-13T16:20:03.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:03.628+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.633031+0000) 2019-10-13T16:20:03.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:03.636+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.638812+0000) 2019-10-13T16:20:03.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:03.636+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241645 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:03.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:03.752+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:03.754395+0000) 2019-10-13T16:20:05.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:05.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:05.781560+0000) 2019-10-13T16:20:05.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:05.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:05.781560+0000) 2019-10-13T16:20:05.793 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:05.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:05.791643+0000) 2019-10-13T16:20:06.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:06.328+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:06.333463+0000) 2019-10-13T16:20:06.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:06.904+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:06.907735+0000) 2019-10-13T16:20:08.646 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:08.640+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.644365+0000) 2019-10-13T16:20:08.647 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:08.640+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.645060+0000) 2019-10-13T16:20:08.647 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:08.640+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.645416+0000) 2019-10-13T16:20:08.648 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:08.640+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.646127+0000) 2019-10-13T16:20:08.649 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:08.644+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.646476+0000) 2019-10-13T16:20:08.649 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:08.640+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.646280+0000) 2019-10-13T16:20:08.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:08.644+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.647816+0000) 2019-10-13T16:20:08.655 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:08.648+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:08.653644+0000) 2019-10-13T16:20:08.659 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:08.652+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:10.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:10.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:10.255806+0000) 2019-10-13T16:20:12.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:12.764+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:12.771077+0000) 2019-10-13T16:20:13.662 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:13.656+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.660127+0000) 2019-10-13T16:20:13.663 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:13.656+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.660578+0000) 2019-10-13T16:20:13.663 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:13.656+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.661161+0000) 2019-10-13T16:20:13.664 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:13.656+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.661667+0000) 2019-10-13T16:20:13.664 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:13.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.662336+0000) 2019-10-13T16:20:13.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:13.660+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.663404+0000) 2019-10-13T16:20:13.665 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:13.660+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.663575+0000) 2019-10-13T16:20:13.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:13.664+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.668422+0000) 2019-10-13T16:20:13.673 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:13.668+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241712 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:13.757 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:13.752+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:13.754578+0000) 2019-10-13T16:20:15.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:15.776+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:15.781787+0000) 2019-10-13T16:20:15.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:15.776+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:15.781788+0000) 2019-10-13T16:20:15.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:15.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:15.791835+0000) 2019-10-13T16:20:16.335 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:16.328+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:16.333644+0000) 2019-10-13T16:20:16.916 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:16.912+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:16.914180+0000) 2019-10-13T16:20:18.677 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:18.668+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.674992+0000) 2019-10-13T16:20:18.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:18.672+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.675745+0000) 2019-10-13T16:20:18.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:18.672+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.675966+0000) 2019-10-13T16:20:18.678 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:18.672+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.676236+0000) 2019-10-13T16:20:18.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:18.672+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.676929+0000) 2019-10-13T16:20:18.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:18.672+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.677030+0000) 2019-10-13T16:20:18.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:18.672+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.678195+0000) 2019-10-13T16:20:18.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:18.680+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:18.685182+0000) 2019-10-13T16:20:18.689 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:18.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241747 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:20.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:20.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:20.255977+0000) 2019-10-13T16:20:21.844 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:20:21.844 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:20:21.847 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:20:21.848 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:20:22.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:22.768+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:22.771277+0000) 2019-10-13T16:20:23.692 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:23.684+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.690334+0000) 2019-10-13T16:20:23.693 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:23.684+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.691170+0000) 2019-10-13T16:20:23.694 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:23.688+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.692247+0000) 2019-10-13T16:20:23.694 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:23.688+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.692632+0000) 2019-10-13T16:20:23.695 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:23.688+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.693305+0000) 2019-10-13T16:20:23.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:23.688+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.693547+0000) 2019-10-13T16:20:23.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:23.688+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.693513+0000) 2019-10-13T16:20:23.700 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:23.692+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.697854+0000) 2019-10-13T16:20:23.704 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:23.696+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241779 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:23.757 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:23.752+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:23.754755+0000) 2019-10-13T16:20:25.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:25.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:25.782012+0000) 2019-10-13T16:20:25.784 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:25.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:25.782011+0000) 2019-10-13T16:20:25.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:25.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:25.792050+0000) 2019-10-13T16:20:26.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:26.332+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:26.333846+0000) 2019-10-13T16:20:26.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:26.916+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:26.920200+0000) 2019-10-13T16:20:28.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:28.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.705729+0000) 2019-10-13T16:20:28.709 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:28.700+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.706233+0000) 2019-10-13T16:20:28.710 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:28.700+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.706659+0000) 2019-10-13T16:20:28.710 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:28.700+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.706806+0000) 2019-10-13T16:20:28.710 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:28.700+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.707194+0000) 2019-10-13T16:20:28.711 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:28.704+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.707566+0000) 2019-10-13T16:20:28.711 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:28.704+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.708370+0000) 2019-10-13T16:20:28.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:28.712+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:28.714961+0000) 2019-10-13T16:20:28.721 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:28.712+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241813 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:30.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:30.252+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:30.256164+0000) 2019-10-13T16:20:32.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:32.767+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:32.771467+0000) 2019-10-13T16:20:33.725 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:33.715+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.722693+0000) 2019-10-13T16:20:33.725 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:33.719+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.723372+0000) 2019-10-13T16:20:33.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:33.719+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.723744+0000) 2019-10-13T16:20:33.726 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:33.720+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.724162+0000) 2019-10-13T16:20:33.727 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:33.719+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.724434+0000) 2019-10-13T16:20:33.727 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:33.719+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.724642+0000) 2019-10-13T16:20:33.727 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:33.719+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.724966+0000) 2019-10-13T16:20:33.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:33.724+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.728340+0000) 2019-10-13T16:20:33.737 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:33.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241846 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:33.757 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:33.752+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:33.754902+0000) 2019-10-13T16:20:35.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:35.775+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:35.782257+0000) 2019-10-13T16:20:35.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:35.775+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:35.782257+0000) 2019-10-13T16:20:35.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:35.787+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:35.792258+0000) 2019-10-13T16:20:36.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:36.327+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:36.334074+0000) 2019-10-13T16:20:36.929 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:36.924+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:36.926553+0000) 2019-10-13T16:20:38.741 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:38.731+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.738428+0000) 2019-10-13T16:20:38.741 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:38.731+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.738839+0000) 2019-10-13T16:20:38.742 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:38.735+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.740008+0000) 2019-10-13T16:20:38.742 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:38.735+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.740197+0000) 2019-10-13T16:20:38.743 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:38.735+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.740619+0000) 2019-10-13T16:20:38.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:38.735+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.741012+0000) 2019-10-13T16:20:38.744 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:38.736+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.741775+0000) 2019-10-13T16:20:38.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:38.744+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:38.747962+0000) 2019-10-13T16:20:38.753 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:38.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241881 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:40.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:40.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:40.256369+0000) 2019-10-13T16:20:42.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:42.767+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:42.771684+0000) 2019-10-13T16:20:43.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:43.747+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.753840+0000) 2019-10-13T16:20:43.756 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:43.747+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.754316+0000) 2019-10-13T16:20:43.757 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:43.751+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.755494+0000) 2019-10-13T16:20:43.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:43.751+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.756539+0000) 2019-10-13T16:20:43.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:43.752+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.757144+0000) 2019-10-13T16:20:43.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:43.751+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.757540+0000) 2019-10-13T16:20:43.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:43.751+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.757886+0000) 2019-10-13T16:20:43.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:43.760+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:43.763353+0000) 2019-10-13T16:20:43.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:43.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241913 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:45.784 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:45.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:45.782477+0000) 2019-10-13T16:20:45.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:45.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:45.782477+0000) 2019-10-13T16:20:45.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:45.791+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:45.792451+0000) 2019-10-13T16:20:46.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:46.331+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:46.334283+0000) 2019-10-13T16:20:46.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:46.932+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:46.932830+0000) 2019-10-13T16:20:48.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:48.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.770086+0000) 2019-10-13T16:20:48.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:48.768+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.770392+0000) 2019-10-13T16:20:48.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:48.763+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.770775+0000) 2019-10-13T16:20:48.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:48.767+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.772057+0000) 2019-10-13T16:20:48.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:48.767+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.772524+0000) 2019-10-13T16:20:48.775 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:48.767+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.772797+0000) 2019-10-13T16:20:48.775 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:48.767+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.773646+0000) 2019-10-13T16:20:48.778 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:48.772+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:48.776425+0000) 2019-10-13T16:20:48.784 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:48.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241947 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:50.258 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:50.251+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:50.256571+0000) 2019-10-13T16:20:50.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:50.500+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:50.505555+0000) 2019-10-13T16:20:52.027 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:20:52.027 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:20:52.031 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:20:52.031 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:20:52.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:52.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:52.771877+0000) 2019-10-13T16:20:53.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:53.779+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.786296+0000) 2019-10-13T16:20:53.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:53.783+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.787047+0000) 2019-10-13T16:20:53.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:53.779+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.786726+0000) 2019-10-13T16:20:53.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:53.783+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.787468+0000) 2019-10-13T16:20:53.790 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:53.783+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.788232+0000) 2019-10-13T16:20:53.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:53.783+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.788549+0000) 2019-10-13T16:20:53.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:53.783+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.789497+0000) 2019-10-13T16:20:53.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:53.791+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:53.796305+0000) 2019-10-13T16:20:53.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:53.791+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 241979 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:20:55.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:55.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:55.782703+0000) 2019-10-13T16:20:55.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:55.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:55.782703+0000) 2019-10-13T16:20:55.794 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:55.791+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:55.792663+0000) 2019-10-13T16:20:56.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:56.331+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:56.334516+0000) 2019-10-13T16:20:56.941 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:56.935+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:56.939455+0000) 2019-10-13T16:20:58.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:20:58.795+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.800727+0000) 2019-10-13T16:20:58.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:20:58.795+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.801236+0000) 2019-10-13T16:20:58.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:20:58.799+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.802257+0000) 2019-10-13T16:20:58.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:20:58.795+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.802791+0000) 2019-10-13T16:20:58.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:20:58.799+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.803478+0000) 2019-10-13T16:20:58.806 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:20:58.799+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.803759+0000) 2019-10-13T16:20:58.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:20:58.799+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.805114+0000) 2019-10-13T16:20:58.811 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:20:58.803+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:20:58.809545+0000) 2019-10-13T16:20:58.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:20:58.807+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242015 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:00.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:00.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:00.256802+0000) 2019-10-13T16:21:00.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:00.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:00.505750+0000) 2019-10-13T16:21:02.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:02.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:02.772133+0000) 2019-10-13T16:21:03.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:03.811+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.816945+0000) 2019-10-13T16:21:03.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:03.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.817286+0000) 2019-10-13T16:21:03.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:03.815+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.817745+0000) 2019-10-13T16:21:03.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:03.815+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.818323+0000) 2019-10-13T16:21:03.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:03.819+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.820028+0000) 2019-10-13T16:21:03.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:03.819+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.820287+0000) 2019-10-13T16:21:03.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:03.819+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.820554+0000) 2019-10-13T16:21:03.827 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:03.819+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:03.825399+0000) 2019-10-13T16:21:03.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:03.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242046 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:05.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:05.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:05.782929+0000) 2019-10-13T16:21:05.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:05.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:05.782930+0000) 2019-10-13T16:21:05.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:05.791+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:05.792891+0000) 2019-10-13T16:21:06.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:06.331+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:06.334707+0000) 2019-10-13T16:21:06.948 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:06.939+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:06.945780+0000) 2019-10-13T16:21:08.836 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:08.831+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.833924+0000) 2019-10-13T16:21:08.836 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:08.831+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.834302+0000) 2019-10-13T16:21:08.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:08.831+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.835235+0000) 2019-10-13T16:21:08.838 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:08.835+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.836010+0000) 2019-10-13T16:21:08.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:08.835+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.836367+0000) 2019-10-13T16:21:08.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:08.835+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.836783+0000) 2019-10-13T16:21:08.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:08.835+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.837140+0000) 2019-10-13T16:21:08.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:08.839+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:08.843639+0000) 2019-10-13T16:21:08.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:08.843+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:10.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:10.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:10.257003+0000) 2019-10-13T16:21:10.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:10.499+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:10.505960+0000) 2019-10-13T16:21:12.774 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:12.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:12.772363+0000) 2019-10-13T16:21:13.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:13.847+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.849481+0000) 2019-10-13T16:21:13.852 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:13.847+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.850112+0000) 2019-10-13T16:21:13.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:13.847+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.850794+0000) 2019-10-13T16:21:13.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:13.847+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.850688+0000) 2019-10-13T16:21:13.854 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:13.847+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.851060+0000) 2019-10-13T16:21:13.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:13.851+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.851515+0000) 2019-10-13T16:21:13.854 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:13.851+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.852268+0000) 2019-10-13T16:21:13.857 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:13.851+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:13.855563+0000) 2019-10-13T16:21:13.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:13.859+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:15.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:15.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:15.783189+0000) 2019-10-13T16:21:15.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:15.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:15.783189+0000) 2019-10-13T16:21:15.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:15.791+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:15.793112+0000) 2019-10-13T16:21:16.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:16.330+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:16.334905+0000) 2019-10-13T16:21:16.954 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:16.951+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:16.952254+0000) 2019-10-13T16:21:18.867 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:18.862+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.865425+0000) 2019-10-13T16:21:18.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:18.862+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.865889+0000) 2019-10-13T16:21:18.868 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:18.862+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.866567+0000) 2019-10-13T16:21:18.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:18.866+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.867394+0000) 2019-10-13T16:21:18.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:18.863+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.867449+0000) 2019-10-13T16:21:18.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:18.866+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.867849+0000) 2019-10-13T16:21:18.870 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:18.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.868062+0000) 2019-10-13T16:21:18.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:18.871+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:18.874778+0000) 2019-10-13T16:21:18.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:18.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:20.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:20.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:20.257191+0000) 2019-10-13T16:21:20.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:20.503+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:20.506172+0000) 2019-10-13T16:21:22.158 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:21:22.158 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:21:22.162 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:21:22.162 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:21:22.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:22.770+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:22.772560+0000) 2019-10-13T16:21:23.881 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:23.874+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.879067+0000) 2019-10-13T16:21:23.882 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:23.878+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.880091+0000) 2019-10-13T16:21:23.882 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:23.878+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.880545+0000) 2019-10-13T16:21:23.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:23.878+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.881174+0000) 2019-10-13T16:21:23.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:23.875+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.881681+0000) 2019-10-13T16:21:23.884 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:23.878+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.882575+0000) 2019-10-13T16:21:23.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:23.878+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.883152+0000) 2019-10-13T16:21:23.890 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:23.883+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:23.888676+0000) 2019-10-13T16:21:23.893 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:23.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:25.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:25.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:25.783418+0000) 2019-10-13T16:21:25.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:25.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:25.783418+0000) 2019-10-13T16:21:25.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:25.790+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:25.793343+0000) 2019-10-13T16:21:26.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:26.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:26.335104+0000) 2019-10-13T16:21:26.960 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:26.955+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:26.958585+0000) 2019-10-13T16:21:28.896 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:28.890+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.894505+0000) 2019-10-13T16:21:28.897 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:28.890+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.895016+0000) 2019-10-13T16:21:28.898 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:28.891+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.895518+0000) 2019-10-13T16:21:28.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:28.894+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.896064+0000) 2019-10-13T16:21:28.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:28.894+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.896491+0000) 2019-10-13T16:21:28.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:28.894+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.897723+0000) 2019-10-13T16:21:28.900 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:28.894+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.898462+0000) 2019-10-13T16:21:28.905 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:28.899+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:28.903633+0000) 2019-10-13T16:21:28.908 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:28.902+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242216 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:30.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:30.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:30.257346+0000) 2019-10-13T16:21:30.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:30.503+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:30.506362+0000) 2019-10-13T16:21:32.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:32.770+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:32.772780+0000) 2019-10-13T16:21:33.912 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:33.906+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.910067+0000) 2019-10-13T16:21:33.913 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:33.906+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.911169+0000) 2019-10-13T16:21:33.914 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:33.910+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.911671+0000) 2019-10-13T16:21:33.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:33.910+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.912068+0000) 2019-10-13T16:21:33.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:33.907+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.912449+0000) 2019-10-13T16:21:33.915 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:33.910+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.913357+0000) 2019-10-13T16:21:33.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:33.910+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.913431+0000) 2019-10-13T16:21:33.919 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:33.911+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:33.917034+0000) 2019-10-13T16:21:33.923 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:33.918+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242249 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:35.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:35.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:35.783642+0000) 2019-10-13T16:21:35.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:35.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:35.783642+0000) 2019-10-13T16:21:35.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:35.790+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:35.793558+0000) 2019-10-13T16:21:36.337 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:36.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:36.335360+0000) 2019-10-13T16:21:36.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:36.963+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:36.965020+0000) 2019-10-13T16:21:38.927 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:38.922+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.925468+0000) 2019-10-13T16:21:38.928 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:38.919+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.925900+0000) 2019-10-13T16:21:38.928 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:38.922+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.926264+0000) 2019-10-13T16:21:38.929 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:38.922+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.926891+0000) 2019-10-13T16:21:38.929 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:38.926+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.927377+0000) 2019-10-13T16:21:38.930 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:38.926+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.928248+0000) 2019-10-13T16:21:38.930 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:38.926+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.928609+0000) 2019-10-13T16:21:38.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:38.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:38.931713+0000) 2019-10-13T16:21:38.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:38.934+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:40.259 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:40.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:40.257541+0000) 2019-10-13T16:21:40.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:40.503+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:40.506514+0000) 2019-10-13T16:21:42.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:42.770+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:42.772995+0000) 2019-10-13T16:21:43.943 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:43.935+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.940482+0000) 2019-10-13T16:21:43.944 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:43.938+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.942446+0000) 2019-10-13T16:21:43.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:43.938+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.942349+0000) 2019-10-13T16:21:43.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:43.938+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.942799+0000) 2019-10-13T16:21:43.945 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:43.938+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.943125+0000) 2019-10-13T16:21:43.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:43.942+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.943644+0000) 2019-10-13T16:21:43.946 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:43.942+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.944497+0000) 2019-10-13T16:21:43.952 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:43.947+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:43.950293+0000) 2019-10-13T16:21:43.956 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:43.950+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242316 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:45.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:45.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:45.783863+0000) 2019-10-13T16:21:45.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:45.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:45.783863+0000) 2019-10-13T16:21:45.795 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:45.790+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:45.793757+0000) 2019-10-13T16:21:46.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:46.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:46.335603+0000) 2019-10-13T16:21:46.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:46.970+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:46.971427+0000) 2019-10-13T16:21:48.959 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:48.954+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.957120+0000) 2019-10-13T16:21:48.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:48.954+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.957960+0000) 2019-10-13T16:21:48.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:48.954+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.958125+0000) 2019-10-13T16:21:48.961 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:48.954+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.959007+0000) 2019-10-13T16:21:48.961 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:48.954+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.959212+0000) 2019-10-13T16:21:48.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:48.958+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.960339+0000) 2019-10-13T16:21:48.962 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:48.958+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.960775+0000) 2019-10-13T16:21:48.967 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:48.958+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:48.965069+0000) 2019-10-13T16:21:48.971 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:48.966+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:50.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:50.254+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:50.257767+0000) 2019-10-13T16:21:50.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:50.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:50.506724+0000) 2019-10-13T16:21:52.279 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:21:52.279 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:21:52.283 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:21:52.283 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:21:52.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:52.770+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:52.773199+0000) 2019-10-13T16:21:53.975 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:53.970+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.973264+0000) 2019-10-13T16:21:53.976 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:53.966+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.973920+0000) 2019-10-13T16:21:53.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:53.970+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.973715+0000) 2019-10-13T16:21:53.977 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:53.970+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.974835+0000) 2019-10-13T16:21:53.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:53.970+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.974909+0000) 2019-10-13T16:21:53.978 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:53.974+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.975467+0000) 2019-10-13T16:21:53.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:53.974+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.976023+0000) 2019-10-13T16:21:53.983 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:53.974+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:53.981305+0000) 2019-10-13T16:21:53.987 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:53.982+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:21:55.786 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:55.782+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:55.784090+0000) 2019-10-13T16:21:55.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:55.782+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:55.784090+0000) 2019-10-13T16:21:55.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:55.790+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:55.793953+0000) 2019-10-13T16:21:56.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:56.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:56.335824+0000) 2019-10-13T16:21:56.980 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:56.974+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:56.977710+0000) 2019-10-13T16:21:58.990 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:21:58.986+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.987719+0000) 2019-10-13T16:21:58.991 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:21:58.986+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.988969+0000) 2019-10-13T16:21:58.991 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:21:58.986+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.989337+0000) 2019-10-13T16:21:58.992 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:21:58.986+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.989910+0000) 2019-10-13T16:21:58.992 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:21:58.986+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.990109+0000) 2019-10-13T16:21:58.993 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:21:58.986+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.991456+0000) 2019-10-13T16:21:58.994 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:21:58.990+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.991975+0000) 2019-10-13T16:21:59.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:21:58.994+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:21:58.999180+0000) 2019-10-13T16:21:59.001 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:21:58.998+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242421 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:00.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:00.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:00.257963+0000) 2019-10-13T16:22:00.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:00.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:00.506932+0000) 2019-10-13T16:22:02.775 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:02.769+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:02.773393+0000) 2019-10-13T16:22:04.005 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:03.997+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.002676+0000) 2019-10-13T16:22:04.005 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:04.001+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.003348+0000) 2019-10-13T16:22:04.006 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:04.001+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.004294+0000) 2019-10-13T16:22:04.006 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:04.001+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.004848+0000) 2019-10-13T16:22:04.007 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:04.001+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.005104+0000) 2019-10-13T16:22:04.008 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:04.001+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.006680+0000) 2019-10-13T16:22:04.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:04.002+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.007119+0000) 2019-10-13T16:22:04.014 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:04.006+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:04.012442+0000) 2019-10-13T16:22:04.017 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:04.013+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242451 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:05.786 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:05.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:05.784339+0000) 2019-10-13T16:22:05.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:05.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:05.784340+0000) 2019-10-13T16:22:05.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:05.793+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:05.794174+0000) 2019-10-13T16:22:06.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:06.333+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:06.336055+0000) 2019-10-13T16:22:06.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:06.982+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:06.984067+0000) 2019-10-13T16:22:09.021 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:09.017+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.019299+0000) 2019-10-13T16:22:09.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:09.017+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.020127+0000) 2019-10-13T16:22:09.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:09.017+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.020830+0000) 2019-10-13T16:22:09.023 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:09.014+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.021131+0000) 2019-10-13T16:22:09.023 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:09.017+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.020506+0000) 2019-10-13T16:22:09.024 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:09.017+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.022136+0000) 2019-10-13T16:22:09.024 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:09.017+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.022781+0000) 2019-10-13T16:22:09.030 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:09.022+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:09.028000+0000) 2019-10-13T16:22:09.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:09.029+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:10.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:10.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:10.258158+0000) 2019-10-13T16:22:10.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:10.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:10.507132+0000) 2019-10-13T16:22:12.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:12.769+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:12.773592+0000) 2019-10-13T16:22:14.037 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:14.033+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.035392+0000) 2019-10-13T16:22:14.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:14.033+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.036112+0000) 2019-10-13T16:22:14.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:14.030+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.036242+0000) 2019-10-13T16:22:14.039 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:14.033+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.036881+0000) 2019-10-13T16:22:14.039 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:14.033+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.037614+0000) 2019-10-13T16:22:14.040 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:14.033+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.038285+0000) 2019-10-13T16:22:14.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:14.033+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.038620+0000) 2019-10-13T16:22:14.046 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:14.038+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:14.043922+0000) 2019-10-13T16:22:14.051 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:14.045+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:15.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:15.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:15.784567+0000) 2019-10-13T16:22:15.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:15.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:15.784568+0000) 2019-10-13T16:22:15.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:15.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:15.794370+0000) 2019-10-13T16:22:16.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:16.333+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:16.336273+0000) 2019-10-13T16:22:16.992 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:16.986+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:16.990424+0000) 2019-10-13T16:22:19.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:19.049+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.051919+0000) 2019-10-13T16:22:19.054 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:19.049+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.052422+0000) 2019-10-13T16:22:19.055 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:19.046+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.053456+0000) 2019-10-13T16:22:19.056 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:19.049+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.053815+0000) 2019-10-13T16:22:19.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:19.049+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.054446+0000) 2019-10-13T16:22:19.057 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:19.049+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.055232+0000) 2019-10-13T16:22:19.057 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:19.053+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.055666+0000) 2019-10-13T16:22:19.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:19.058+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:19.063169+0000) 2019-10-13T16:22:19.066 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:19.061+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242557 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:20.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:20.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:20.258399+0000) 2019-10-13T16:22:20.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:20.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:20.507300+0000) 2019-10-13T16:22:22.398 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:22:22.398 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:22:22.401 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:22:22.402 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:22:22.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:22.769+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:22.773793+0000) 2019-10-13T16:22:24.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:24.065+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.068405+0000) 2019-10-13T16:22:24.071 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:24.065+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.068846+0000) 2019-10-13T16:22:24.071 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:24.065+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.069508+0000) 2019-10-13T16:22:24.072 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:24.062+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.069639+0000) 2019-10-13T16:22:24.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:24.065+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.070023+0000) 2019-10-13T16:22:24.073 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:24.065+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.071230+0000) 2019-10-13T16:22:24.074 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:24.069+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.071987+0000) 2019-10-13T16:22:24.081 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:24.074+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:24.079571+0000) 2019-10-13T16:22:24.082 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:24.077+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242587 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:25.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:25.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:25.784793+0000) 2019-10-13T16:22:25.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:25.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:25.784793+0000) 2019-10-13T16:22:25.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:25.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:25.794564+0000) 2019-10-13T16:22:26.338 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:26.333+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:26.336472+0000) 2019-10-13T16:22:27.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:26.998+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:27.000160+0000) 2019-10-13T16:22:29.085 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:29.081+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.083390+0000) 2019-10-13T16:22:29.086 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:29.081+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.084576+0000) 2019-10-13T16:22:29.087 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:29.081+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.084358+0000) 2019-10-13T16:22:29.087 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:29.081+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.085045+0000) 2019-10-13T16:22:29.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:29.078+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.085968+0000) 2019-10-13T16:22:29.088 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:29.081+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.086522+0000) 2019-10-13T16:22:29.089 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:29.081+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.087159+0000) 2019-10-13T16:22:29.098 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:29.090+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:29.096012+0000) 2019-10-13T16:22:29.098 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:29.093+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242624 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:30.260 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:30.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:30.258638+0000) 2019-10-13T16:22:30.509 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:30.502+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:30.507492+0000) 2019-10-13T16:22:32.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:32.773+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:32.773995+0000) 2019-10-13T16:22:34.102 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:34.097+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.100594+0000) 2019-10-13T16:22:34.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:34.097+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.101297+0000) 2019-10-13T16:22:34.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:34.097+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.101903+0000) 2019-10-13T16:22:34.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:34.097+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.101945+0000) 2019-10-13T16:22:34.104 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:34.098+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.102311+0000) 2019-10-13T16:22:34.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:34.097+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.102986+0000) 2019-10-13T16:22:34.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:34.101+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.103941+0000) 2019-10-13T16:22:34.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:34.102+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:34.109055+0000) 2019-10-13T16:22:34.113 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:34.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242656 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:35.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:35.781+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:35.785020+0000) 2019-10-13T16:22:35.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:35.781+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:35.785019+0000) 2019-10-13T16:22:35.796 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:35.789+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:35.794753+0000) 2019-10-13T16:22:36.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:36.333+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:36.336673+0000) 2019-10-13T16:22:37.009 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:37.006+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:37.006554+0000) 2019-10-13T16:22:39.117 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:39.109+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.114849+0000) 2019-10-13T16:22:39.117 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:39.109+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.114903+0000) 2019-10-13T16:22:39.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:39.113+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.115867+0000) 2019-10-13T16:22:39.118 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:39.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.116644+0000) 2019-10-13T16:22:39.119 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:39.113+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.117432+0000) 2019-10-13T16:22:39.120 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:39.114+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.117764+0000) 2019-10-13T16:22:39.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:39.113+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.118277+0000) 2019-10-13T16:22:39.124 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:39.122+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:39.122437+0000) 2019-10-13T16:22:39.128 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:39.121+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242693 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:40.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:40.253+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:40.258838+0000) 2019-10-13T16:22:40.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:40.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:40.507714+0000) 2019-10-13T16:22:42.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:42.773+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:42.774207+0000) 2019-10-13T16:22:44.132 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:44.124+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.130006+0000) 2019-10-13T16:22:44.133 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:44.124+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.129842+0000) 2019-10-13T16:22:44.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:44.124+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.130594+0000) 2019-10-13T16:22:44.134 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:44.129+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.131061+0000) 2019-10-13T16:22:44.134 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:44.128+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.131520+0000) 2019-10-13T16:22:44.134 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:44.128+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.131997+0000) 2019-10-13T16:22:44.135 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:44.128+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.132602+0000) 2019-10-13T16:22:44.144 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:44.136+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242723 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:44.144 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:44.141+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:44.142801+0000) 2019-10-13T16:22:45.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:45.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:45.785263+0000) 2019-10-13T16:22:45.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:45.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:45.785263+0000) 2019-10-13T16:22:45.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:45.792+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:45.794960+0000) 2019-10-13T16:22:46.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:46.332+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:46.336895+0000) 2019-10-13T16:22:47.015 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:47.009+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:47.012984+0000) 2019-10-13T16:22:49.147 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:49.141+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.145047+0000) 2019-10-13T16:22:49.148 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:49.140+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.145403+0000) 2019-10-13T16:22:49.149 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:49.140+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.147058+0000) 2019-10-13T16:22:49.149 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:49.144+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.147352+0000) 2019-10-13T16:22:49.150 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:49.144+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.148542+0000) 2019-10-13T16:22:49.150 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:49.144+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.148977+0000) 2019-10-13T16:22:49.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:49.144+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.149319+0000) 2019-10-13T16:22:49.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:49.153+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:49.156496+0000) 2019-10-13T16:22:49.159 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:49.152+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242760 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:50.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:50.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:50.259091+0000) 2019-10-13T16:22:50.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:50.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:50.507909+0000) 2019-10-13T16:22:52.776 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:22:52.776 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:22:52.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:52.768+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:52.774405+0000) 2019-10-13T16:22:52.780 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:22:52.781 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:22:54.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:54.156+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.160390+0000) 2019-10-13T16:22:54.163 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:54.156+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.160912+0000) 2019-10-13T16:22:54.164 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:54.156+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.161252+0000) 2019-10-13T16:22:54.164 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:54.156+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.161838+0000) 2019-10-13T16:22:54.164 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:54.161+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.162439+0000) 2019-10-13T16:22:54.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:54.156+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.162980+0000) 2019-10-13T16:22:54.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:54.160+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.163937+0000) 2019-10-13T16:22:54.168 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:54.165+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:54.166282+0000) 2019-10-13T16:22:54.176 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:54.168+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242793 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:22:55.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:55.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:55.785486+0000) 2019-10-13T16:22:55.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:55.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:55.785486+0000) 2019-10-13T16:22:55.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:55.788+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:55.795162+0000) 2019-10-13T16:22:56.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:56.336+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:56.337092+0000) 2019-10-13T16:22:57.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:57.017+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:57.019468+0000) 2019-10-13T16:22:59.178 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:22:59.172+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.176449+0000) 2019-10-13T16:22:59.179 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:22:59.172+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.177172+0000) 2019-10-13T16:22:59.180 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:22:59.172+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.177968+0000) 2019-10-13T16:22:59.180 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:22:59.172+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.178722+0000) 2019-10-13T16:22:59.181 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:22:59.177+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.179464+0000) 2019-10-13T16:22:59.182 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:22:59.176+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.179985+0000) 2019-10-13T16:22:59.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:22:59.176+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.180442+0000) 2019-10-13T16:22:59.186 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:22:59.181+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:22:59.184760+0000) 2019-10-13T16:22:59.191 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:22:59.184+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242830 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:00.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:00.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:00.259332+0000) 2019-10-13T16:23:00.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:00.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:00.508098+0000) 2019-10-13T16:23:02.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:02.768+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:02.774625+0000) 2019-10-13T16:23:04.195 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:04.188+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.192611+0000) 2019-10-13T16:23:04.195 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:04.188+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.192999+0000) 2019-10-13T16:23:04.196 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:04.189+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.193584+0000) 2019-10-13T16:23:04.196 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:04.188+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.193299+0000) 2019-10-13T16:23:04.197 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:04.188+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.194868+0000) 2019-10-13T16:23:04.197 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:04.192+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.195529+0000) 2019-10-13T16:23:04.198 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:04.192+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.195825+0000) 2019-10-13T16:23:04.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:04.201+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:04.204519+0000) 2019-10-13T16:23:04.207 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:04.200+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242861 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:05.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:05.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:05.785715+0000) 2019-10-13T16:23:05.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:05.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:05.785714+0000) 2019-10-13T16:23:05.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:05.792+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:05.795355+0000) 2019-10-13T16:23:06.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:06.336+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:06.337305+0000) 2019-10-13T16:23:07.028 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:07.025+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:07.025793+0000) 2019-10-13T16:23:09.210 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:09.204+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.208100+0000) 2019-10-13T16:23:09.211 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:09.204+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.208888+0000) 2019-10-13T16:23:09.212 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:09.204+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.209944+0000) 2019-10-13T16:23:09.212 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:09.204+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.210408+0000) 2019-10-13T16:23:09.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:09.209+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.210822+0000) 2019-10-13T16:23:09.213 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:09.204+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.211197+0000) 2019-10-13T16:23:09.213 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:09.208+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.211844+0000) 2019-10-13T16:23:09.217 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:09.213+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:09.215552+0000) 2019-10-13T16:23:09.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:09.216+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242898 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:10.261 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:10.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:10.259538+0000) 2019-10-13T16:23:10.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:10.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:10.508323+0000) 2019-10-13T16:23:12.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:12.772+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:12.774834+0000) 2019-10-13T16:23:14.224 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:14.216+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.222092+0000) 2019-10-13T16:23:14.225 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:14.216+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.222393+0000) 2019-10-13T16:23:14.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:14.216+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.223104+0000) 2019-10-13T16:23:14.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:14.220+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.225214+0000) 2019-10-13T16:23:14.228 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:14.220+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.226370+0000) 2019-10-13T16:23:14.228 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:14.225+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.226623+0000) 2019-10-13T16:23:14.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:14.220+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.227157+0000) 2019-10-13T16:23:14.237 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:14.232+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242931 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:14.238 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:14.233+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:14.235964+0000) 2019-10-13T16:23:15.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:15.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:15.785947+0000) 2019-10-13T16:23:15.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:15.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:15.785947+0000) 2019-10-13T16:23:15.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:15.792+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:15.795579+0000) 2019-10-13T16:23:16.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:16.332+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:16.337507+0000) 2019-10-13T16:23:17.034 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:17.029+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:17.031789+0000) 2019-10-13T16:23:19.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:19.232+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.238411+0000) 2019-10-13T16:23:19.242 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:19.236+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.239962+0000) 2019-10-13T16:23:19.242 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:19.236+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.240357+0000) 2019-10-13T16:23:19.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:19.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.240687+0000) 2019-10-13T16:23:19.243 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:19.236+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.241908+0000) 2019-10-13T16:23:19.244 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:19.236+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.242866+0000) 2019-10-13T16:23:19.245 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:19.241+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.242943+0000) 2019-10-13T16:23:19.250 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:19.245+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:19.247880+0000) 2019-10-13T16:23:19.253 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:19.248+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:20.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:20.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:20.259738+0000) 2019-10-13T16:23:20.510 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:20.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:20.508548+0000) 2019-10-13T16:23:22.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:22.772+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:22.775076+0000) 2019-10-13T16:23:22.899 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:23:22.900 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:23:22.903 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:23:22.903 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:23:24.257 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:24.248+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.254467+0000) 2019-10-13T16:23:24.257 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:24.248+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.254888+0000) 2019-10-13T16:23:24.257 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:24.252+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.255293+0000) 2019-10-13T16:23:24.258 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:24.253+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.255980+0000) 2019-10-13T16:23:24.260 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:24.252+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.256813+0000) 2019-10-13T16:23:24.260 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:24.252+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.257170+0000) 2019-10-13T16:23:24.260 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:24.252+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.257661+0000) 2019-10-13T16:23:24.266 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:24.261+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:24.264394+0000) 2019-10-13T16:23:24.270 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:24.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 242999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:25.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:25.780+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:25.786175+0000) 2019-10-13T16:23:25.788 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:25.780+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:25.786175+0000) 2019-10-13T16:23:25.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:25.792+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:25.795737+0000) 2019-10-13T16:23:26.339 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:26.336+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:26.337667+0000) 2019-10-13T16:23:27.040 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:27.033+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:27.038100+0000) 2019-10-13T16:23:29.273 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:29.263+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.270847+0000) 2019-10-13T16:23:29.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:29.269+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.272049+0000) 2019-10-13T16:23:29.274 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:29.267+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.272442+0000) 2019-10-13T16:23:29.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:29.267+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.272748+0000) 2019-10-13T16:23:29.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:29.267+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.273271+0000) 2019-10-13T16:23:29.275 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:29.267+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.273624+0000) 2019-10-13T16:23:29.276 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:29.267+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.274012+0000) 2019-10-13T16:23:29.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:29.273+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:29.277459+0000) 2019-10-13T16:23:29.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:29.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243036 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:30.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:30.255+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:30.259939+0000) 2019-10-13T16:23:30.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:30.505+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:30.508735+0000) 2019-10-13T16:23:32.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:32.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:32.775294+0000) 2019-10-13T16:23:34.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:34.279+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.285842+0000) 2019-10-13T16:23:34.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:34.279+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.286885+0000) 2019-10-13T16:23:34.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:34.279+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.287194+0000) 2019-10-13T16:23:34.290 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:34.283+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.287582+0000) 2019-10-13T16:23:34.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:34.285+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.288494+0000) 2019-10-13T16:23:34.291 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:34.283+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.289167+0000) 2019-10-13T16:23:34.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:34.283+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.289531+0000) 2019-10-13T16:23:34.297 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:34.293+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:34.295153+0000) 2019-10-13T16:23:34.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:34.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243069 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:35.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:35.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:35.786422+0000) 2019-10-13T16:23:35.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:35.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:35.786422+0000) 2019-10-13T16:23:35.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:35.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:35.795946+0000) 2019-10-13T16:23:36.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:36.331+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:36.337875+0000) 2019-10-13T16:23:37.051 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:37.049+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:37.049455+0000) 2019-10-13T16:23:39.302 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:39.295+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.299814+0000) 2019-10-13T16:23:39.303 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:39.295+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.301021+0000) 2019-10-13T16:23:39.303 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:39.295+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.301088+0000) 2019-10-13T16:23:39.303 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:39.296+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.301408+0000) 2019-10-13T16:23:39.304 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:39.295+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.302264+0000) 2019-10-13T16:23:39.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:39.295+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.302598+0000) 2019-10-13T16:23:39.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:39.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.303185+0000) 2019-10-13T16:23:39.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:39.304+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:39.308098+0000) 2019-10-13T16:23:39.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:39.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243105 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:40.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:40.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:40.260154+0000) 2019-10-13T16:23:40.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:40.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:40.508925+0000) 2019-10-13T16:23:42.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:42.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:42.775524+0000) 2019-10-13T16:23:44.318 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:44.311+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.315516+0000) 2019-10-13T16:23:44.318 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:44.311+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.315942+0000) 2019-10-13T16:23:44.318 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:44.311+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.316667+0000) 2019-10-13T16:23:44.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:44.311+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.317168+0000) 2019-10-13T16:23:44.320 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:44.312+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.317797+0000) 2019-10-13T16:23:44.320 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:44.311+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.318109+0000) 2019-10-13T16:23:44.320 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:44.311+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.318488+0000) 2019-10-13T16:23:44.329 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:44.320+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:44.326081+0000) 2019-10-13T16:23:44.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:44.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243137 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:45.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:45.779+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:45.786652+0000) 2019-10-13T16:23:45.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:45.779+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:45.786653+0000) 2019-10-13T16:23:45.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:45.791+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:45.796156+0000) 2019-10-13T16:23:46.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:46.335+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:46.338109+0000) 2019-10-13T16:23:47.057 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:47.052+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:47.055314+0000) 2019-10-13T16:23:49.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:49.327+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.330205+0000) 2019-10-13T16:23:49.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:49.327+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.330996+0000) 2019-10-13T16:23:49.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:49.327+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.332039+0000) 2019-10-13T16:23:49.334 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:49.327+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.332157+0000) 2019-10-13T16:23:49.334 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:49.327+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.332244+0000) 2019-10-13T16:23:49.335 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:49.327+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.333454+0000) 2019-10-13T16:23:49.335 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:49.328+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.333500+0000) 2019-10-13T16:23:49.342 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:49.336+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:49.340186+0000) 2019-10-13T16:23:49.344 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:49.335+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:50.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:50.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:50.260334+0000) 2019-10-13T16:23:50.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:50.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:50.509119+0000) 2019-10-13T16:23:52.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:52.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:52.775730+0000) 2019-10-13T16:23:52.999 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:23:52.999 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:23:53.002 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:23:53.003 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:23:54.348 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:54.339+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.345689+0000) 2019-10-13T16:23:54.348 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:54.339+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.346528+0000) 2019-10-13T16:23:54.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:54.339+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.347189+0000) 2019-10-13T16:23:54.349 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:54.343+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.347650+0000) 2019-10-13T16:23:54.350 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:54.343+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.347996+0000) 2019-10-13T16:23:54.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:54.343+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.348632+0000) 2019-10-13T16:23:54.351 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:54.344+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.348891+0000) 2019-10-13T16:23:54.354 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:54.348+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:54.352646+0000) 2019-10-13T16:23:54.360 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:54.351+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243207 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:23:55.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:55.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:55.786882+0000) 2019-10-13T16:23:55.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:55.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:55.786881+0000) 2019-10-13T16:23:55.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:55.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:55.796351+0000) 2019-10-13T16:23:56.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:56.335+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:56.338306+0000) 2019-10-13T16:23:57.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:57.060+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:57.061781+0000) 2019-10-13T16:23:59.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:23:59.359+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.361783+0000) 2019-10-13T16:23:59.364 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:23:59.356+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.362012+0000) 2019-10-13T16:23:59.364 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:23:59.359+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.362658+0000) 2019-10-13T16:23:59.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:23:59.363+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.363624+0000) 2019-10-13T16:23:59.366 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:23:59.363+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.363822+0000) 2019-10-13T16:23:59.366 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:23:59.363+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.364153+0000) 2019-10-13T16:23:59.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:23:59.363+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.364629+0000) 2019-10-13T16:23:59.373 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:23:59.368+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:23:59.370913+0000) 2019-10-13T16:23:59.375 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:23:59.371+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243242 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:00.262 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:00.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:00.260520+0000) 2019-10-13T16:24:00.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:00.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:00.509374+0000) 2019-10-13T16:24:02.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:02.775+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:02.775939+0000) 2019-10-13T16:24:04.378 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:04.372+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.376107+0000) 2019-10-13T16:24:04.379 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:04.375+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.376642+0000) 2019-10-13T16:24:04.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:04.375+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.377555+0000) 2019-10-13T16:24:04.380 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:04.375+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.378336+0000) 2019-10-13T16:24:04.381 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:04.375+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.378990+0000) 2019-10-13T16:24:04.381 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:04.379+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.379427+0000) 2019-10-13T16:24:04.382 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:04.379+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.379887+0000) 2019-10-13T16:24:04.387 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:04.380+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:04.384950+0000) 2019-10-13T16:24:04.392 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:04.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243274 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:05.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:05.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:05.787097+0000) 2019-10-13T16:24:05.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:05.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:05.787097+0000) 2019-10-13T16:24:05.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:05.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:05.796581+0000) 2019-10-13T16:24:06.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:06.335+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:06.338509+0000) 2019-10-13T16:24:07.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:07.064+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:07.068291+0000) 2019-10-13T16:24:09.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:09.391+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.393917+0000) 2019-10-13T16:24:09.397 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:09.391+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.394034+0000) 2019-10-13T16:24:09.398 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:09.391+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.394727+0000) 2019-10-13T16:24:09.398 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:09.391+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.395130+0000) 2019-10-13T16:24:09.398 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:09.392+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.395729+0000) 2019-10-13T16:24:09.399 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:09.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.396759+0000) 2019-10-13T16:24:09.399 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:09.395+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.397092+0000) 2019-10-13T16:24:09.403 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:09.396+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:09.401649+0000) 2019-10-13T16:24:09.412 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:09.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243311 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:10.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:10.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:10.260732+0000) 2019-10-13T16:24:10.511 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:10.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:10.509614+0000) 2019-10-13T16:24:12.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:12.774+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:12.776145+0000) 2019-10-13T16:24:14.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:14.406+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.409223+0000) 2019-10-13T16:24:14.412 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:14.406+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.409721+0000) 2019-10-13T16:24:14.412 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:14.408+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.410320+0000) 2019-10-13T16:24:14.414 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:14.410+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.412118+0000) 2019-10-13T16:24:14.414 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:14.410+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.412679+0000) 2019-10-13T16:24:14.415 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:14.410+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.413603+0000) 2019-10-13T16:24:14.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:14.410+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.414033+0000) 2019-10-13T16:24:14.421 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:14.416+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:14.419036+0000) 2019-10-13T16:24:14.423 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:14.418+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243344 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:15.789 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:15.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:15.787366+0000) 2019-10-13T16:24:15.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:15.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:15.787351+0000) 2019-10-13T16:24:15.798 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:15.794+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:15.796773+0000) 2019-10-13T16:24:16.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:16.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:16.338707+0000) 2019-10-13T16:24:17.080 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:17.072+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:17.077668+0000) 2019-10-13T16:24:19.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:19.422+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.424426+0000) 2019-10-13T16:24:19.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:19.422+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.424920+0000) 2019-10-13T16:24:19.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:19.422+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.425218+0000) 2019-10-13T16:24:19.427 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:19.422+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.425793+0000) 2019-10-13T16:24:19.428 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:19.422+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.426146+0000) 2019-10-13T16:24:19.428 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:19.424+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.426514+0000) 2019-10-13T16:24:19.430 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:19.426+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.428128+0000) 2019-10-13T16:24:19.436 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:19.432+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:19.434312+0000) 2019-10-13T16:24:19.438 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:19.434+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243379 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:20.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:20.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:20.260934+0000) 2019-10-13T16:24:20.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:20.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:20.509836+0000) 2019-10-13T16:24:22.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:22.774+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:22.776345+0000) 2019-10-13T16:24:23.205 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:24:23.205 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:24:23.208 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:24:23.209 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:24:24.442 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:24.438+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.439925+0000) 2019-10-13T16:24:24.442 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:24.438+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.440539+0000) 2019-10-13T16:24:24.443 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:24.438+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.441411+0000) 2019-10-13T16:24:24.443 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:24.438+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.441833+0000) 2019-10-13T16:24:24.445 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:24.440+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.442813+0000) 2019-10-13T16:24:24.445 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:24.442+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.443910+0000) 2019-10-13T16:24:24.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:24.442+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.444196+0000) 2019-10-13T16:24:24.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:24.444+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:24.449945+0000) 2019-10-13T16:24:24.455 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:24.450+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243412 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:25.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:25.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:25.787569+0000) 2019-10-13T16:24:25.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:25.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:25.787569+0000) 2019-10-13T16:24:25.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:25.794+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:25.796970+0000) 2019-10-13T16:24:26.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:26.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:26.338918+0000) 2019-10-13T16:24:27.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:27.080+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:27.084079+0000) 2019-10-13T16:24:29.459 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:29.454+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.457353+0000) 2019-10-13T16:24:29.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:29.454+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.457823+0000) 2019-10-13T16:24:29.460 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:29.454+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.458360+0000) 2019-10-13T16:24:29.461 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:29.454+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.458662+0000) 2019-10-13T16:24:29.461 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:29.456+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.459091+0000) 2019-10-13T16:24:29.462 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:29.458+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.459518+0000) 2019-10-13T16:24:29.462 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:29.458+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.460128+0000) 2019-10-13T16:24:29.469 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:29.464+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:29.467303+0000) 2019-10-13T16:24:29.470 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:29.466+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:30.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:30.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:30.261126+0000) 2019-10-13T16:24:30.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:30.504+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:30.510030+0000) 2019-10-13T16:24:32.778 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:32.774+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:32.776541+0000) 2019-10-13T16:24:34.474 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:34.470+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.472348+0000) 2019-10-13T16:24:34.475 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:34.470+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.472406+0000) 2019-10-13T16:24:34.475 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:34.470+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.472768+0000) 2019-10-13T16:24:34.475 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:34.467+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.473231+0000) 2019-10-13T16:24:34.476 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:34.470+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.473478+0000) 2019-10-13T16:24:34.476 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:34.470+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.474959+0000) 2019-10-13T16:24:34.477 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:34.474+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.475725+0000) 2019-10-13T16:24:34.483 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:34.475+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:34.481257+0000) 2019-10-13T16:24:34.485 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:34.478+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243482 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:35.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:35.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:35.787797+0000) 2019-10-13T16:24:35.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:35.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:35.787796+0000) 2019-10-13T16:24:35.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:35.794+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:35.797168+0000) 2019-10-13T16:24:36.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:36.334+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:36.339146+0000) 2019-10-13T16:24:37.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:37.091+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:37.094972+0000) 2019-10-13T16:24:39.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:39.482+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.486706+0000) 2019-10-13T16:24:39.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:39.486+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.487843+0000) 2019-10-13T16:24:39.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:39.486+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.488323+0000) 2019-10-13T16:24:39.490 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:39.483+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.488813+0000) 2019-10-13T16:24:39.491 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:39.486+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.489312+0000) 2019-10-13T16:24:39.491 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:39.486+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.489722+0000) 2019-10-13T16:24:39.492 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:39.486+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.490587+0000) 2019-10-13T16:24:39.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:39.487+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:39.494000+0000) 2019-10-13T16:24:39.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:39.494+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243517 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:40.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:40.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:40.261312+0000) 2019-10-13T16:24:40.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:40.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:40.510230+0000) 2019-10-13T16:24:42.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:42.774+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:42.776759+0000) 2019-10-13T16:24:44.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:44.498+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.502700+0000) 2019-10-13T16:24:44.505 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:44.502+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.503386+0000) 2019-10-13T16:24:44.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:44.502+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.504400+0000) 2019-10-13T16:24:44.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:44.502+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.504610+0000) 2019-10-13T16:24:44.507 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:44.499+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.505046+0000) 2019-10-13T16:24:44.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:44.502+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.505424+0000) 2019-10-13T16:24:44.508 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:44.502+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.506150+0000) 2019-10-13T16:24:44.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:44.507+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:44.510388+0000) 2019-10-13T16:24:44.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:44.514+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:45.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:45.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:45.788021+0000) 2019-10-13T16:24:45.790 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:45.786+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:45.788021+0000) 2019-10-13T16:24:45.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:45.794+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:45.797357+0000) 2019-10-13T16:24:46.341 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:46.338+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:46.339344+0000) 2019-10-13T16:24:47.109 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:47.103+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:47.106674+0000) 2019-10-13T16:24:49.521 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:49.514+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.519039+0000) 2019-10-13T16:24:49.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:49.518+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.519759+0000) 2019-10-13T16:24:49.522 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:49.518+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.520544+0000) 2019-10-13T16:24:49.522 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:49.518+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.520612+0000) 2019-10-13T16:24:49.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:49.515+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.521187+0000) 2019-10-13T16:24:49.523 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:49.518+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.521523+0000) 2019-10-13T16:24:49.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:49.518+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.521840+0000) 2019-10-13T16:24:49.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:49.523+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:49.529410+0000) 2019-10-13T16:24:49.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:49.530+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243586 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:50.263 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:50.258+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:50.261447+0000) 2019-10-13T16:24:50.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:50.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:50.510421+0000) 2019-10-13T16:24:52.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:52.774+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:52.776992+0000) 2019-10-13T16:24:53.254 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:24:53.255 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:24:53.258 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:24:53.258 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:24:54.537 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:54.530+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.535122+0000) 2019-10-13T16:24:54.538 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:54.534+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.535929+0000) 2019-10-13T16:24:54.538 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:54.534+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.536335+0000) 2019-10-13T16:24:54.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:54.531+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.537989+0000) 2019-10-13T16:24:54.546 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:54.539+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.543010+0000) 2019-10-13T16:24:54.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:54.534+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.537158+0000) 2019-10-13T16:24:54.546 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:54.534+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.537673+0000) 2019-10-13T16:24:54.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:54.534+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:54.538765+0000) 2019-10-13T16:24:54.550 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:54.546+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243620 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:24:55.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:55.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:55.788252+0000) 2019-10-13T16:24:55.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:55.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:55.788252+0000) 2019-10-13T16:24:55.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:55.793+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:55.797596+0000) 2019-10-13T16:24:56.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:56.337+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:56.339547+0000) 2019-10-13T16:24:57.118 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:57.115+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:57.115856+0000) 2019-10-13T16:24:59.553 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:24:59.545+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.551047+0000) 2019-10-13T16:24:59.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:24:59.549+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.552644+0000) 2019-10-13T16:24:59.555 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:24:59.549+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.552746+0000) 2019-10-13T16:24:59.555 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:24:59.549+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.553091+0000) 2019-10-13T16:24:59.556 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:24:59.549+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.554267+0000) 2019-10-13T16:24:59.557 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:24:59.551+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.554986+0000) 2019-10-13T16:24:59.557 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:24:59.549+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.554799+0000) 2019-10-13T16:24:59.562 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:24:59.555+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:24:59.559963+0000) 2019-10-13T16:24:59.566 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:24:59.561+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243655 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:00.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:00.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:00.261656+0000) 2019-10-13T16:25:00.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:00.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:00.510613+0000) 2019-10-13T16:25:02.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:02.773+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:02.777144+0000) 2019-10-13T16:25:04.570 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:04.565+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.567642+0000) 2019-10-13T16:25:04.570 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:04.565+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.567918+0000) 2019-10-13T16:25:04.571 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:04.565+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.568266+0000) 2019-10-13T16:25:04.571 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:04.565+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.569046+0000) 2019-10-13T16:25:04.571 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:04.565+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.569718+0000) 2019-10-13T16:25:04.572 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:04.565+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.570288+0000) 2019-10-13T16:25:04.573 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:04.567+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.571416+0000) 2019-10-13T16:25:04.579 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:04.571+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:04.577053+0000) 2019-10-13T16:25:04.582 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:04.577+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243688 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:05.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:05.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:05.788509+0000) 2019-10-13T16:25:05.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:05.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:05.788509+0000) 2019-10-13T16:25:05.799 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:05.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:05.797820+0000) 2019-10-13T16:25:06.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:06.337+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:06.339744+0000) 2019-10-13T16:25:07.125 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:07.119+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:07.122552+0000) 2019-10-13T16:25:09.586 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:09.581+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.583980+0000) 2019-10-13T16:25:09.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:09.581+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.584275+0000) 2019-10-13T16:25:09.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:09.581+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.585053+0000) 2019-10-13T16:25:09.587 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:09.579+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.585217+0000) 2019-10-13T16:25:09.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:09.581+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.585859+0000) 2019-10-13T16:25:09.588 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:09.581+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.586207+0000) 2019-10-13T16:25:09.589 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:09.581+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.587073+0000) 2019-10-13T16:25:09.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:09.587+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:09.592855+0000) 2019-10-13T16:25:09.599 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:09.593+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243724 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:10.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:10.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:10.261856+0000) 2019-10-13T16:25:10.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:10.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:10.510811+0000) 2019-10-13T16:25:12.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:12.773+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:12.777372+0000) 2019-10-13T16:25:14.602 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:14.595+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.600311+0000) 2019-10-13T16:25:14.603 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:14.597+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.600710+0000) 2019-10-13T16:25:14.603 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:14.597+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.601426+0000) 2019-10-13T16:25:14.604 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:14.597+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.602382+0000) 2019-10-13T16:25:14.604 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:14.597+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.602249+0000) 2019-10-13T16:25:14.605 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:14.597+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.602672+0000) 2019-10-13T16:25:14.606 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:14.601+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.604143+0000) 2019-10-13T16:25:14.612 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:14.603+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:14.610046+0000) 2019-10-13T16:25:14.615 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:14.609+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243758 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:15.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:15.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:15.788768+0000) 2019-10-13T16:25:15.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:15.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:15.788768+0000) 2019-10-13T16:25:15.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:15.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:15.798049+0000) 2019-10-13T16:25:16.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:16.337+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:16.339944+0000) 2019-10-13T16:25:17.136 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:17.131+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:17.134309+0000) 2019-10-13T16:25:19.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:19.613+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.616749+0000) 2019-10-13T16:25:19.619 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:19.613+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.617025+0000) 2019-10-13T16:25:19.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:19.613+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.617893+0000) 2019-10-13T16:25:19.620 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:19.613+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.618101+0000) 2019-10-13T16:25:19.621 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:19.615+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.618992+0000) 2019-10-13T16:25:19.621 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:19.617+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.619654+0000) 2019-10-13T16:25:19.622 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:19.617+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.620066+0000) 2019-10-13T16:25:19.625 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:19.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:19.623472+0000) 2019-10-13T16:25:19.630 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:19.625+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243793 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:20.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:20.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:20.262036+0000) 2019-10-13T16:25:20.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:20.507+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:20.510998+0000) 2019-10-13T16:25:22.779 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:22.777+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:22.777630+0000) 2019-10-13T16:25:23.352 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:25:23.352 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:25:23.356 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:25:23.356 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:25:24.635 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:24.629+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.632817+0000) 2019-10-13T16:25:24.635 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:24.629+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.633328+0000) 2019-10-13T16:25:24.636 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:24.629+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.634173+0000) 2019-10-13T16:25:24.636 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:24.629+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.634356+0000) 2019-10-13T16:25:24.637 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:24.629+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.634768+0000) 2019-10-13T16:25:24.637 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:24.631+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.635354+0000) 2019-10-13T16:25:24.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:24.633+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.635789+0000) 2019-10-13T16:25:24.641 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:24.635+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:24.638966+0000) 2019-10-13T16:25:24.646 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:24.641+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243826 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:25.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:25.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:25.788995+0000) 2019-10-13T16:25:25.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:25.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:25.788995+0000) 2019-10-13T16:25:25.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:25.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:25.798245+0000) 2019-10-13T16:25:26.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:26.337+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:26.340135+0000) 2019-10-13T16:25:27.147 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:27.139+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:27.140825+0000) 2019-10-13T16:25:29.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:29.645+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.648970+0000) 2019-10-13T16:25:29.653 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:29.642+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.649290+0000) 2019-10-13T16:25:29.653 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:29.645+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.649944+0000) 2019-10-13T16:25:29.653 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:29.645+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.649872+0000) 2019-10-13T16:25:29.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:29.645+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.650568+0000) 2019-10-13T16:25:29.654 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:29.645+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.650191+0000) 2019-10-13T16:25:29.655 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:29.649+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.652153+0000) 2019-10-13T16:25:29.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:29.653+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243862 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:29.662 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:29.654+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:29.660363+0000) 2019-10-13T16:25:30.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:30.257+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:30.262215+0000) 2019-10-13T16:25:30.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:30.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:30.511190+0000) 2019-10-13T16:25:32.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:32.773+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:32.777831+0000) 2019-10-13T16:25:34.664 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:34.657+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.662253+0000) 2019-10-13T16:25:34.665 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:34.657+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.662756+0000) 2019-10-13T16:25:34.665 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:34.657+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.662890+0000) 2019-10-13T16:25:34.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:34.658+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.665068+0000) 2019-10-13T16:25:34.667 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:34.661+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.665486+0000) 2019-10-13T16:25:34.668 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:34.661+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.665849+0000) 2019-10-13T16:25:34.668 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:34.661+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.666135+0000) 2019-10-13T16:25:34.672 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:34.666+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:34.670729+0000) 2019-10-13T16:25:34.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:34.669+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243895 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:35.791 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:35.785+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:35.789225+0000) 2019-10-13T16:25:35.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:35.785+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:35.789225+0000) 2019-10-13T16:25:35.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:35.793+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:35.798440+0000) 2019-10-13T16:25:36.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:36.337+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:36.340365+0000) 2019-10-13T16:25:37.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:37.146+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:37.147220+0000) 2019-10-13T16:25:39.679 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:39.672+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.676904+0000) 2019-10-13T16:25:39.680 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:39.672+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.678067+0000) 2019-10-13T16:25:39.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:39.672+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.678689+0000) 2019-10-13T16:25:39.681 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:39.674+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.678968+0000) 2019-10-13T16:25:39.682 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:39.676+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.680021+0000) 2019-10-13T16:25:39.682 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:39.676+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.680463+0000) 2019-10-13T16:25:39.683 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:39.676+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.680786+0000) 2019-10-13T16:25:39.687 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:39.678+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:39.685631+0000) 2019-10-13T16:25:39.691 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:39.684+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243930 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:40.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:40.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:40.262399+0000) 2019-10-13T16:25:40.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:40.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:40.511362+0000) 2019-10-13T16:25:42.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:42.772+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:42.778060+0000) 2019-10-13T16:25:44.695 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:44.688+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.693580+0000) 2019-10-13T16:25:44.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:44.686+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.693803+0000) 2019-10-13T16:25:44.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:44.688+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.694368+0000) 2019-10-13T16:25:44.700 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:44.692+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.695348+0000) 2019-10-13T16:25:44.700 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:44.692+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.695838+0000) 2019-10-13T16:25:44.700 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:44.692+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.696389+0000) 2019-10-13T16:25:44.700 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:44.692+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.696826+0000) 2019-10-13T16:25:44.704 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:44.694+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:44.702023+0000) 2019-10-13T16:25:44.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:44.700+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243962 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:45.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:45.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:45.789454+0000) 2019-10-13T16:25:45.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:45.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:45.789453+0000) 2019-10-13T16:25:45.800 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:45.796+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:45.798674+0000) 2019-10-13T16:25:46.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:46.336+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:46.340568+0000) 2019-10-13T16:25:47.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:47.154+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:47.158765+0000) 2019-10-13T16:25:49.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:49.704+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.709643+0000) 2019-10-13T16:25:49.712 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:49.704+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.710446+0000) 2019-10-13T16:25:49.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:49.706+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.710926+0000) 2019-10-13T16:25:49.713 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:49.708+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.711547+0000) 2019-10-13T16:25:49.714 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:49.708+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.712331+0000) 2019-10-13T16:25:49.714 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:49.708+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.712708+0000) 2019-10-13T16:25:49.715 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:49.708+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.713041+0000) 2019-10-13T16:25:49.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:49.714+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:49.720491+0000) 2019-10-13T16:25:49.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:49.716+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 243998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:50.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:50.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:50.262616+0000) 2019-10-13T16:25:50.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:50.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:50.511566+0000) 2019-10-13T16:25:52.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:52.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:52.778264+0000) 2019-10-13T16:25:53.400 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:25:53.401 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:25:53.404 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:25:53.405 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:25:54.728 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:54.720+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.725727+0000) 2019-10-13T16:25:54.729 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:54.720+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.726298+0000) 2019-10-13T16:25:54.729 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:54.720+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.726929+0000) 2019-10-13T16:25:54.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:54.722+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.726615+0000) 2019-10-13T16:25:54.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:54.724+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.727524+0000) 2019-10-13T16:25:54.731 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:54.724+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.728164+0000) 2019-10-13T16:25:54.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:54.724+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.728714+0000) 2019-10-13T16:25:54.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:54.726+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:54.733411+0000) 2019-10-13T16:25:54.739 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:54.732+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244030 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:25:55.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:55.784+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:55.789694+0000) 2019-10-13T16:25:55.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:55.784+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:55.789693+0000) 2019-10-13T16:25:55.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:55.792+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:55.798867+0000) 2019-10-13T16:25:56.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:56.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:56.340808+0000) 2019-10-13T16:25:57.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:57.162+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:57.165356+0000) 2019-10-13T16:25:59.742 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:25:59.736+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.739995+0000) 2019-10-13T16:25:59.743 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:25:59.736+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.740668+0000) 2019-10-13T16:25:59.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:25:59.736+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.741295+0000) 2019-10-13T16:25:59.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:25:59.736+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.742173+0000) 2019-10-13T16:25:59.744 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:25:59.736+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.742686+0000) 2019-10-13T16:25:59.745 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:25:59.738+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.743106+0000) 2019-10-13T16:25:59.745 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:25:59.740+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.743516+0000) 2019-10-13T16:25:59.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:25:59.742+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:25:59.747957+0000) 2019-10-13T16:25:59.754 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:25:59.748+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:00.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:00.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:00.262792+0000) 2019-10-13T16:26:00.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:00.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:00.511737+0000) 2019-10-13T16:26:02.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:02.772+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:02.778469+0000) 2019-10-13T16:26:04.758 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:04.752+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.755717+0000) 2019-10-13T16:26:04.758 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:04.752+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.755966+0000) 2019-10-13T16:26:04.759 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:04.750+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.756758+0000) 2019-10-13T16:26:04.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:04.752+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.756577+0000) 2019-10-13T16:26:04.759 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:04.752+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.757382+0000) 2019-10-13T16:26:04.760 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:04.752+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.758109+0000) 2019-10-13T16:26:04.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:04.752+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.758533+0000) 2019-10-13T16:26:04.764 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:04.758+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:04.762621+0000) 2019-10-13T16:26:04.768 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:04.760+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244097 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:05.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:05.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:05.789925+0000) 2019-10-13T16:26:05.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:05.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:05.789924+0000) 2019-10-13T16:26:05.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:05.796+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:05.799079+0000) 2019-10-13T16:26:06.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:06.336+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:06.341001+0000) 2019-10-13T16:26:07.173 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:07.170+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:07.171038+0000) 2019-10-13T16:26:09.772 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:09.764+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.769666+0000) 2019-10-13T16:26:09.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:09.766+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.770449+0000) 2019-10-13T16:26:09.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:09.764+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.771017+0000) 2019-10-13T16:26:09.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:09.768+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.771581+0000) 2019-10-13T16:26:09.774 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:09.764+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.771127+0000) 2019-10-13T16:26:09.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:09.768+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.772207+0000) 2019-10-13T16:26:09.775 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:09.768+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.772685+0000) 2019-10-13T16:26:09.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:09.774+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:09.779125+0000) 2019-10-13T16:26:09.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:09.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244134 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:10.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:10.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:10.262925+0000) 2019-10-13T16:26:10.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:10.506+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:10.511925+0000) 2019-10-13T16:26:12.780 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:12.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:12.778666+0000) 2019-10-13T16:26:14.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:14.780+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.786698+0000) 2019-10-13T16:26:14.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:14.780+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.787205+0000) 2019-10-13T16:26:14.790 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:14.784+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.787734+0000) 2019-10-13T16:26:14.791 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:14.782+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.788971+0000) 2019-10-13T16:26:14.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:14.784+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.789319+0000) 2019-10-13T16:26:14.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:14.784+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.790191+0000) 2019-10-13T16:26:14.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:14.784+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.790933+0000) 2019-10-13T16:26:14.798 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:14.790+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:14.796550+0000) 2019-10-13T16:26:14.800 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:14.792+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244166 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:15.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:15.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:15.790147+0000) 2019-10-13T16:26:15.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:15.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:15.790147+0000) 2019-10-13T16:26:15.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:15.796+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:15.799333+0000) 2019-10-13T16:26:16.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:16.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:16.341179+0000) 2019-10-13T16:26:17.179 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:17.174+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:17.177458+0000) 2019-10-13T16:26:19.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:19.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.801773+0000) 2019-10-13T16:26:19.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:19.802+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.802374+0000) 2019-10-13T16:26:19.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:19.796+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.803165+0000) 2019-10-13T16:26:19.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:19.800+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.803822+0000) 2019-10-13T16:26:19.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:19.800+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.804368+0000) 2019-10-13T16:26:19.807 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:19.800+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.805410+0000) 2019-10-13T16:26:19.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:19.800+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.805787+0000) 2019-10-13T16:26:19.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:19.810+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:19.810407+0000) 2019-10-13T16:26:19.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:19.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:20.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:20.256+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:20.263155+0000) 2019-10-13T16:26:20.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:20.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:20.512115+0000) 2019-10-13T16:26:22.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:22.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:22.778904+0000) 2019-10-13T16:26:23.457 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:26:23.457 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:26:23.461 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:26:23.461 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:26:24.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:24.811+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.819011+0000) 2019-10-13T16:26:24.821 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:24.811+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.818860+0000) 2019-10-13T16:26:24.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:24.815+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.819799+0000) 2019-10-13T16:26:24.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:24.815+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.820238+0000) 2019-10-13T16:26:24.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:24.815+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.820646+0000) 2019-10-13T16:26:24.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:24.817+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.820863+0000) 2019-10-13T16:26:24.823 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:24.815+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.821723+0000) 2019-10-13T16:26:24.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:24.825+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:24.828133+0000) 2019-10-13T16:26:24.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:24.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244233 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:25.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:25.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:25.790335+0000) 2019-10-13T16:26:25.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:25.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:25.790335+0000) 2019-10-13T16:26:25.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:25.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:25.799562+0000) 2019-10-13T16:26:26.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:26.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:26.341412+0000) 2019-10-13T16:26:27.186 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:27.181+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:27.183841+0000) 2019-10-13T16:26:29.835 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:29.827+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.833239+0000) 2019-10-13T16:26:29.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:29.827+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.833591+0000) 2019-10-13T16:26:29.836 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:29.829+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.833907+0000) 2019-10-13T16:26:29.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:29.827+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.834338+0000) 2019-10-13T16:26:29.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:29.831+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.835344+0000) 2019-10-13T16:26:29.837 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:29.831+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.835810+0000) 2019-10-13T16:26:29.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:29.831+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.836189+0000) 2019-10-13T16:26:29.845 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:29.841+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:29.843319+0000) 2019-10-13T16:26:29.847 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:29.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244270 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:30.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:30.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:30.263334+0000) 2019-10-13T16:26:30.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:30.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:30.512283+0000) 2019-10-13T16:26:32.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:32.771+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:32.779096+0000) 2019-10-13T16:26:34.851 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:34.843+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.848844+0000) 2019-10-13T16:26:34.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:34.843+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.849952+0000) 2019-10-13T16:26:34.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:34.843+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.850258+0000) 2019-10-13T16:26:34.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:34.843+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.850748+0000) 2019-10-13T16:26:34.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:34.849+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.851030+0000) 2019-10-13T16:26:34.853 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:34.847+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.851575+0000) 2019-10-13T16:26:34.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:34.847+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.851917+0000) 2019-10-13T16:26:34.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:34.853+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:34.856964+0000) 2019-10-13T16:26:34.863 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:34.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244302 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:35.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:35.783+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:35.790530+0000) 2019-10-13T16:26:35.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:35.783+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:35.790531+0000) 2019-10-13T16:26:35.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:35.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:35.799732+0000) 2019-10-13T16:26:36.343 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:36.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:36.341620+0000) 2019-10-13T16:26:37.192 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:37.189+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:37.190335+0000) 2019-10-13T16:26:39.866 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:39.859+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.864377+0000) 2019-10-13T16:26:39.867 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:39.859+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.865088+0000) 2019-10-13T16:26:39.867 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:39.861+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.865730+0000) 2019-10-13T16:26:39.868 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:39.859+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.865495+0000) 2019-10-13T16:26:39.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:39.859+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.866251+0000) 2019-10-13T16:26:39.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:39.859+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.866519+0000) 2019-10-13T16:26:39.869 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:39.863+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.867649+0000) 2019-10-13T16:26:39.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:39.873+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:39.874263+0000) 2019-10-13T16:26:39.878 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:39.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:40.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:40.259+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:40.263516+0000) 2019-10-13T16:26:40.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:40.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:40.512495+0000) 2019-10-13T16:26:42.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:42.775+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:42.779332+0000) 2019-10-13T16:26:44.881 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:44.875+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.879579+0000) 2019-10-13T16:26:44.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:44.875+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.880367+0000) 2019-10-13T16:26:44.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:44.875+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.881247+0000) 2019-10-13T16:26:44.884 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:44.875+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.881735+0000) 2019-10-13T16:26:44.885 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:44.877+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.880837+0000) 2019-10-13T16:26:44.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:44.879+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.883590+0000) 2019-10-13T16:26:44.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:44.875+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.883258+0000) 2019-10-13T16:26:44.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:44.885+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:44.889414+0000) 2019-10-13T16:26:44.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:44.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244368 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:45.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:45.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:45.790735+0000) 2019-10-13T16:26:45.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:45.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:45.790735+0000) 2019-10-13T16:26:45.801 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:45.799+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:45.799914+0000) 2019-10-13T16:26:46.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:46.335+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:46.341862+0000) 2019-10-13T16:26:47.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:47.193+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:47.196754+0000) 2019-10-13T16:26:49.897 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:49.891+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.895035+0000) 2019-10-13T16:26:49.898 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:49.891+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.895941+0000) 2019-10-13T16:26:49.898 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:49.891+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.896309+0000) 2019-10-13T16:26:49.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:49.891+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.896713+0000) 2019-10-13T16:26:49.900 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:49.891+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.898170+0000) 2019-10-13T16:26:49.900 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:49.891+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.898343+0000) 2019-10-13T16:26:49.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:49.897+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.898921+0000) 2019-10-13T16:26:49.906 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:49.901+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:49.904894+0000) 2019-10-13T16:26:49.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:49.903+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244407 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:50.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:50.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:50.263695+0000) 2019-10-13T16:26:50.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:50.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:50.512664+0000) 2019-10-13T16:26:52.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:52.779+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:52.779534+0000) 2019-10-13T16:26:53.588 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:26:53.588 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:26:53.592 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:26:53.592 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:26:54.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:54.911+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.911536+0000) 2019-10-13T16:26:54.914 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:54.911+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.911849+0000) 2019-10-13T16:26:54.915 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:54.909+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.912796+0000) 2019-10-13T16:26:54.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:54.911+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.912781+0000) 2019-10-13T16:26:54.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:54.911+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.913649+0000) 2019-10-13T16:26:54.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:54.911+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.914018+0000) 2019-10-13T16:26:54.916 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:54.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.914142+0000) 2019-10-13T16:26:54.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:54.921+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:54.922159+0000) 2019-10-13T16:26:54.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:54.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244436 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:26:55.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:55.787+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:55.790882+0000) 2019-10-13T16:26:55.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:55.787+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:55.790882+0000) 2019-10-13T16:26:55.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:55.799+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:55.800088+0000) 2019-10-13T16:26:56.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:56.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:56.342105+0000) 2019-10-13T16:26:57.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:57.201+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:57.203269+0000) 2019-10-13T16:26:59.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:26:59.925+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.928395+0000) 2019-10-13T16:26:59.931 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:26:59.927+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.928781+0000) 2019-10-13T16:26:59.931 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:26:59.927+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.929127+0000) 2019-10-13T16:26:59.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:26:59.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.930002+0000) 2019-10-13T16:26:59.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:26:59.927+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.930460+0000) 2019-10-13T16:26:59.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:26:59.927+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.930761+0000) 2019-10-13T16:26:59.933 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:26:59.927+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.930934+0000) 2019-10-13T16:26:59.938 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:26:59.933+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:26:59.936092+0000) 2019-10-13T16:26:59.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:26:59.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244473 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:00.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:00.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:00.263898+0000) 2019-10-13T16:27:00.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:00.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:00.512837+0000) 2019-10-13T16:27:02.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:02.779+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:02.779733+0000) 2019-10-13T16:27:04.946 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:04.943+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.943965+0000) 2019-10-13T16:27:04.946 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:04.941+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.944035+0000) 2019-10-13T16:27:04.947 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:04.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.944992+0000) 2019-10-13T16:27:04.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:04.943+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.945351+0000) 2019-10-13T16:27:04.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:04.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.946030+0000) 2019-10-13T16:27:04.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:04.943+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.946572+0000) 2019-10-13T16:27:04.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:04.943+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.947091+0000) 2019-10-13T16:27:04.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:04.953+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:04.955012+0000) 2019-10-13T16:27:04.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:04.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244502 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:05.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:05.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:05.791075+0000) 2019-10-13T16:27:05.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:05.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:05.791075+0000) 2019-10-13T16:27:05.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:05.799+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:05.800268+0000) 2019-10-13T16:27:06.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:06.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:06.342363+0000) 2019-10-13T16:27:07.212 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:07.205+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:07.209882+0000) 2019-10-13T16:27:09.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:09.958+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.959909+0000) 2019-10-13T16:27:09.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:09.958+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.960632+0000) 2019-10-13T16:27:09.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:09.958+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.961832+0000) 2019-10-13T16:27:09.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:09.958+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.962001+0000) 2019-10-13T16:27:09.968 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:09.961+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.962679+0000) 2019-10-13T16:27:09.968 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:09.958+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.963099+0000) 2019-10-13T16:27:09.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:09.962+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.963626+0000) 2019-10-13T16:27:09.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:09.965+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:09.968493+0000) 2019-10-13T16:27:09.979 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:09.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:10.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:10.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:10.264102+0000) 2019-10-13T16:27:10.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:10.509+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:10.513065+0000) 2019-10-13T16:27:12.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:12.778+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:12.779895+0000) 2019-10-13T16:27:14.979 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:14.974+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.976906+0000) 2019-10-13T16:27:14.979 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:14.974+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.977985+0000) 2019-10-13T16:27:14.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:14.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.978357+0000) 2019-10-13T16:27:14.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:14.977+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.978657+0000) 2019-10-13T16:27:14.981 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:14.974+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.979051+0000) 2019-10-13T16:27:14.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:14.978+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.979807+0000) 2019-10-13T16:27:14.982 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:14.978+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.979534+0000) 2019-10-13T16:27:14.985 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:14.981+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:14.983509+0000) 2019-10-13T16:27:14.991 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:14.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244570 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:15.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:15.786+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:15.791274+0000) 2019-10-13T16:27:15.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:15.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:15.791274+0000) 2019-10-13T16:27:15.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:15.798+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:15.800440+0000) 2019-10-13T16:27:16.344 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:16.338+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:16.342574+0000) 2019-10-13T16:27:17.218 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:17.213+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:17.216441+0000) 2019-10-13T16:27:19.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:19.990+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.992365+0000) 2019-10-13T16:27:19.995 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:19.990+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.993149+0000) 2019-10-13T16:27:19.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:19.990+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.993345+0000) 2019-10-13T16:27:19.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:19.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.993945+0000) 2019-10-13T16:27:19.996 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:19.990+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.994364+0000) 2019-10-13T16:27:19.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:19.994+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.996365+0000) 2019-10-13T16:27:19.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:19.993+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.996775+0000) 2019-10-13T16:27:20.001 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:19.997+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:19.999082+0000) 2019-10-13T16:27:20.007 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:20.002+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244607 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:20.266 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:20.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:20.264317+0000) 2019-10-13T16:27:20.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:20.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:20.513242+0000) 2019-10-13T16:27:22.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:22.778+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:22.780099+0000) 2019-10-13T16:27:23.709 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:27:23.709 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:27:23.713 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:27:23.713 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:27:25.010 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:25.006+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.008443+0000) 2019-10-13T16:27:25.011 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:25.006+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.008707+0000) 2019-10-13T16:27:25.011 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:25.004+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.009467+0000) 2019-10-13T16:27:25.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:25.006+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.008985+0000) 2019-10-13T16:27:25.012 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:25.006+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.010119+0000) 2019-10-13T16:27:25.012 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:25.006+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.010412+0000) 2019-10-13T16:27:25.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:25.006+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.011130+0000) 2019-10-13T16:27:25.019 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:25.012+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.017163+0000) 2019-10-13T16:27:25.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:25.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244636 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:25.793 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:25.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.791555+0000) 2019-10-13T16:27:25.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:25.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.791555+0000) 2019-10-13T16:27:25.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:25.798+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:25.800618+0000) 2019-10-13T16:27:26.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:26.338+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:26.342805+0000) 2019-10-13T16:27:27.230 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:27.224+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:27.227930+0000) 2019-10-13T16:27:30.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:30.018+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.022653+0000) 2019-10-13T16:27:30.026 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:30.022+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.023897+0000) 2019-10-13T16:27:30.026 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:30.022+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.023596+0000) 2019-10-13T16:27:30.026 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:30.022+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.024209+0000) 2019-10-13T16:27:30.027 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:30.022+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.025121+0000) 2019-10-13T16:27:30.027 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:30.022+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.025665+0000) 2019-10-13T16:27:30.028 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:30.024+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.026187+0000) 2019-10-13T16:27:30.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:30.028+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.031713+0000) 2019-10-13T16:27:30.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:30.030+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244675 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:30.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:30.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.264516+0000) 2019-10-13T16:27:30.515 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:30.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:30.513415+0000) 2019-10-13T16:27:32.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:32.778+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:32.780312+0000) 2019-10-13T16:27:35.041 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:35.034+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.038876+0000) 2019-10-13T16:27:35.041 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:35.034+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.039253+0000) 2019-10-13T16:27:35.042 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:35.038+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.039822+0000) 2019-10-13T16:27:35.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:35.038+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.040241+0000) 2019-10-13T16:27:35.042 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:35.038+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.040625+0000) 2019-10-13T16:27:35.043 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:35.038+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.041132+0000) 2019-10-13T16:27:35.044 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:35.036+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.041982+0000) 2019-10-13T16:27:35.048 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:35.044+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.046716+0000) 2019-10-13T16:27:35.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:35.050+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:35.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:35.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.791753+0000) 2019-10-13T16:27:35.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:35.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.791753+0000) 2019-10-13T16:27:35.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:35.798+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:35.800822+0000) 2019-10-13T16:27:36.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:36.338+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:36.343006+0000) 2019-10-13T16:27:37.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:37.232+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:37.234524+0000) 2019-10-13T16:27:40.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:40.050+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.054636+0000) 2019-10-13T16:27:40.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:40.054+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.055652+0000) 2019-10-13T16:27:40.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:40.054+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.055969+0000) 2019-10-13T16:27:40.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:40.054+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.056336+0000) 2019-10-13T16:27:40.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:40.054+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.057296+0000) 2019-10-13T16:27:40.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:40.056+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.058207+0000) 2019-10-13T16:27:40.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:40.054+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.059022+0000) 2019-10-13T16:27:40.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:40.064+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.067321+0000) 2019-10-13T16:27:40.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:40.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244741 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:40.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:40.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.264723+0000) 2019-10-13T16:27:40.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:40.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:40.513616+0000) 2019-10-13T16:27:42.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:42.778+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:42.780537+0000) 2019-10-13T16:27:45.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:45.066+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.071182+0000) 2019-10-13T16:27:45.074 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:45.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.071600+0000) 2019-10-13T16:27:45.074 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:45.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.071875+0000) 2019-10-13T16:27:45.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:45.068+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.072203+0000) 2019-10-13T16:27:45.075 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:45.070+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.072652+0000) 2019-10-13T16:27:45.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:45.070+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.072876+0000) 2019-10-13T16:27:45.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:45.070+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.073597+0000) 2019-10-13T16:27:45.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:45.076+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.080773+0000) 2019-10-13T16:27:45.086 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:45.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:45.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:45.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.791956+0000) 2019-10-13T16:27:45.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:45.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.791955+0000) 2019-10-13T16:27:45.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:45.798+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:45.800996+0000) 2019-10-13T16:27:46.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:46.338+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:46.343240+0000) 2019-10-13T16:27:47.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:47.240+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:47.245609+0000) 2019-10-13T16:27:50.088 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:50.082+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.086491+0000) 2019-10-13T16:27:50.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:50.084+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.087537+0000) 2019-10-13T16:27:50.090 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:50.086+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.088146+0000) 2019-10-13T16:27:50.090 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:50.086+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.088633+0000) 2019-10-13T16:27:50.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:50.086+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.089103+0000) 2019-10-13T16:27:50.091 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:50.086+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.089413+0000) 2019-10-13T16:27:50.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:50.086+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.090712+0000) 2019-10-13T16:27:50.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:50.088+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.093777+0000) 2019-10-13T16:27:50.099 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:50.094+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244809 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:50.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:50.262+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.264940+0000) 2019-10-13T16:27:50.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:50.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:50.513817+0000) 2019-10-13T16:27:52.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:52.777+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:52.780750+0000) 2019-10-13T16:27:53.786 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:27:53.786 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:27:53.790 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:27:53.790 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:27:55.103 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:55.097+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.100807+0000) 2019-10-13T16:27:55.103 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:27:55.097+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.101256+0000) 2019-10-13T16:27:55.103 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:55.097+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.101470+0000) 2019-10-13T16:27:55.104 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:55.097+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.101806+0000) 2019-10-13T16:27:55.104 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:27:55.097+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.102317+0000) 2019-10-13T16:27:55.106 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:27:55.100+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.103971+0000) 2019-10-13T16:27:55.107 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:55.101+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.105213+0000) 2019-10-13T16:27:55.112 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:55.108+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.110599+0000) 2019-10-13T16:27:55.114 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:27:55.109+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244838 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:27:55.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:27:55.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.792152+0000) 2019-10-13T16:27:55.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:27:55.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.792151+0000) 2019-10-13T16:27:55.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:27:55.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:55.801170+0000) 2019-10-13T16:27:56.345 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:27:56.341+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:56.343440+0000) 2019-10-13T16:27:57.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:27:57.248+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:27:57.252157+0000) 2019-10-13T16:28:00.118 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:00.113+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.116097+0000) 2019-10-13T16:28:00.119 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:00.113+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.116355+0000) 2019-10-13T16:28:00.119 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:00.112+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.117320+0000) 2019-10-13T16:28:00.119 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:00.113+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.117063+0000) 2019-10-13T16:28:00.120 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:00.113+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.118330+0000) 2019-10-13T16:28:00.120 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:00.113+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.117814+0000) 2019-10-13T16:28:00.121 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:00.113+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.119048+0000) 2019-10-13T16:28:00.130 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:00.125+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:00.132 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:00.128+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.130822+0000) 2019-10-13T16:28:00.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:00.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.265117+0000) 2019-10-13T16:28:00.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:00.508+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:00.513989+0000) 2019-10-13T16:28:02.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:02.777+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:02.780968+0000) 2019-10-13T16:28:05.133 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:05.129+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.131525+0000) 2019-10-13T16:28:05.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:05.129+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.132124+0000) 2019-10-13T16:28:05.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:05.129+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.133109+0000) 2019-10-13T16:28:05.135 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:05.129+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.133956+0000) 2019-10-13T16:28:05.136 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:05.132+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.134378+0000) 2019-10-13T16:28:05.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:05.129+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.134784+0000) 2019-10-13T16:28:05.137 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:05.133+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.135880+0000) 2019-10-13T16:28:05.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:05.136+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.140017+0000) 2019-10-13T16:28:05.146 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:05.141+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244904 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:05.794 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:05.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.792350+0000) 2019-10-13T16:28:05.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:05.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.792350+0000) 2019-10-13T16:28:05.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:05.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:05.801348+0000) 2019-10-13T16:28:06.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:06.341+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:06.343634+0000) 2019-10-13T16:28:07.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:07.256+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:07.258581+0000) 2019-10-13T16:28:10.149 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:10.144+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.147503+0000) 2019-10-13T16:28:10.150 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:10.145+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.147895+0000) 2019-10-13T16:28:10.150 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:10.145+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.148366+0000) 2019-10-13T16:28:10.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:10.145+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.149922+0000) 2019-10-13T16:28:10.152 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:10.145+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.150456+0000) 2019-10-13T16:28:10.152 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:10.145+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.150668+0000) 2019-10-13T16:28:10.153 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:10.145+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.151085+0000) 2019-10-13T16:28:10.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:10.152+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.156145+0000) 2019-10-13T16:28:10.162 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:10.157+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:10.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:10.261+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.265294+0000) 2019-10-13T16:28:10.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:10.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:10.514186+0000) 2019-10-13T16:28:12.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:12.777+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:12.781205+0000) 2019-10-13T16:28:15.165 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:15.161+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.163572+0000) 2019-10-13T16:28:15.166 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:15.160+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.163995+0000) 2019-10-13T16:28:15.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:15.161+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.165023+0000) 2019-10-13T16:28:15.167 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:15.161+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.165359+0000) 2019-10-13T16:28:15.168 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:15.161+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.166014+0000) 2019-10-13T16:28:15.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:15.161+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.166280+0000) 2019-10-13T16:28:15.169 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:15.161+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.167162+0000) 2019-10-13T16:28:15.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:15.172+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.175546+0000) 2019-10-13T16:28:15.178 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:15.173+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 244972 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:15.794 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:15.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.792552+0000) 2019-10-13T16:28:15.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:15.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.792552+0000) 2019-10-13T16:28:15.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:15.801+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:15.801523+0000) 2019-10-13T16:28:16.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:16.341+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:16.343853+0000) 2019-10-13T16:28:17.267 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:17.263+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:17.265037+0000) 2019-10-13T16:28:20.181 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:20.177+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.179596+0000) 2019-10-13T16:28:20.182 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:20.177+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.179927+0000) 2019-10-13T16:28:20.182 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:20.175+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.180463+0000) 2019-10-13T16:28:20.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:20.177+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.181926+0000) 2019-10-13T16:28:20.185 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:20.177+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.182528+0000) 2019-10-13T16:28:20.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:20.177+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.183016+0000) 2019-10-13T16:28:20.185 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:20.177+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.182755+0000) 2019-10-13T16:28:20.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:20.183+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.188465+0000) 2019-10-13T16:28:20.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:20.189+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245007 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:20.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:20.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.265472+0000) 2019-10-13T16:28:20.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:20.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:20.514356+0000) 2019-10-13T16:28:22.783 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:22.777+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:22.781414+0000) 2019-10-13T16:28:23.917 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:28:23.917 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:28:23.921 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:28:23.921 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:28:25.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:25.193+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.195458+0000) 2019-10-13T16:28:25.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:25.193+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.196001+0000) 2019-10-13T16:28:25.198 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:25.193+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.196541+0000) 2019-10-13T16:28:25.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:25.193+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.197049+0000) 2019-10-13T16:28:25.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:25.193+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.197857+0000) 2019-10-13T16:28:25.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:25.195+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.198272+0000) 2019-10-13T16:28:25.201 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:25.197+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.199352+0000) 2019-10-13T16:28:25.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:25.199+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.204945+0000) 2019-10-13T16:28:25.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:25.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245038 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:25.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:25.789+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.792748+0000) 2019-10-13T16:28:25.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:25.789+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.792749+0000) 2019-10-13T16:28:25.803 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:25.797+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:25.801696+0000) 2019-10-13T16:28:26.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:26.341+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:26.344003+0000) 2019-10-13T16:28:27.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:27.267+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:27.271396+0000) 2019-10-13T16:28:30.215 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:30.209+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.212503+0000) 2019-10-13T16:28:30.215 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:30.209+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.212912+0000) 2019-10-13T16:28:30.216 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:30.209+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.213163+0000) 2019-10-13T16:28:30.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:30.209+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.213423+0000) 2019-10-13T16:28:30.216 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:30.207+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.213895+0000) 2019-10-13T16:28:30.217 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:30.209+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.214346+0000) 2019-10-13T16:28:30.217 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:30.209+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.215176+0000) 2019-10-13T16:28:30.224 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:30.215+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.221936+0000) 2019-10-13T16:28:30.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:30.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245072 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:30.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:30.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.265656+0000) 2019-10-13T16:28:30.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:30.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:30.514541+0000) 2019-10-13T16:28:32.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:32.781+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:32.781626+0000) 2019-10-13T16:28:35.229 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:35.223+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.227092+0000) 2019-10-13T16:28:35.231 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:35.224+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.228334+0000) 2019-10-13T16:28:35.231 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:35.224+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.228577+0000) 2019-10-13T16:28:35.231 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:35.224+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.229774+0000) 2019-10-13T16:28:35.232 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:35.224+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.230480+0000) 2019-10-13T16:28:35.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:35.224+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.230920+0000) 2019-10-13T16:28:35.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:35.228+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.231602+0000) 2019-10-13T16:28:35.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:35.231+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.234168+0000) 2019-10-13T16:28:35.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:35.236+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245103 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:35.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:35.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.792943+0000) 2019-10-13T16:28:35.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:35.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.792943+0000) 2019-10-13T16:28:35.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:35.796+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:35.801871+0000) 2019-10-13T16:28:36.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:36.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:36.344215+0000) 2019-10-13T16:28:37.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:37.271+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:37.277855+0000) 2019-10-13T16:28:40.247 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:40.240+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.244606+0000) 2019-10-13T16:28:40.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:40.240+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.244597+0000) 2019-10-13T16:28:40.247 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:40.239+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.245708+0000) 2019-10-13T16:28:40.248 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:40.240+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.246703+0000) 2019-10-13T16:28:40.249 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:40.240+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.247102+0000) 2019-10-13T16:28:40.249 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:40.244+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.247663+0000) 2019-10-13T16:28:40.250 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:40.244+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.248246+0000) 2019-10-13T16:28:40.253 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:40.247+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.251710+0000) 2019-10-13T16:28:40.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:40.252+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:40.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:40.260+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.265777+0000) 2019-10-13T16:28:40.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:40.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:40.514731+0000) 2019-10-13T16:28:42.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:42.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:42.781829+0000) 2019-10-13T16:28:45.261 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:45.256+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.259292+0000) 2019-10-13T16:28:45.262 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:45.256+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.260452+0000) 2019-10-13T16:28:45.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:45.255+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.261117+0000) 2019-10-13T16:28:45.263 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:45.256+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.261900+0000) 2019-10-13T16:28:45.264 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:45.256+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.262212+0000) 2019-10-13T16:28:45.264 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:45.256+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.262558+0000) 2019-10-13T16:28:45.265 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:45.256+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.263107+0000) 2019-10-13T16:28:45.269 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:45.263+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.267510+0000) 2019-10-13T16:28:45.274 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:45.268+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245167 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:45.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:45.788+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.793140+0000) 2019-10-13T16:28:45.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:45.788+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.793140+0000) 2019-10-13T16:28:45.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:45.800+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:45.802047+0000) 2019-10-13T16:28:46.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:46.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:46.344407+0000) 2019-10-13T16:28:47.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:47.283+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:47.284243+0000) 2019-10-13T16:28:50.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:50.264+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.265938+0000) 2019-10-13T16:28:50.278 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:50.272+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.275731+0000) 2019-10-13T16:28:50.278 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:50.272+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.276088+0000) 2019-10-13T16:28:50.279 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:50.268+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.275026+0000) 2019-10-13T16:28:50.279 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:50.272+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.277084+0000) 2019-10-13T16:28:50.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:50.272+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.277272+0000) 2019-10-13T16:28:50.280 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:50.271+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.278037+0000) 2019-10-13T16:28:50.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:50.276+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.279658+0000) 2019-10-13T16:28:50.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:50.279+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.285528+0000) 2019-10-13T16:28:50.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:50.284+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245200 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:50.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:50.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:50.514958+0000) 2019-10-13T16:28:52.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:52.780+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:52.782030+0000) 2019-10-13T16:28:53.990 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:28:53.991 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:28:53.994 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:28:53.994 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:28:55.294 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:55.288+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.291694+0000) 2019-10-13T16:28:55.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:28:55.288+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.292107+0000) 2019-10-13T16:28:55.294 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:55.288+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.292374+0000) 2019-10-13T16:28:55.295 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:55.288+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.293013+0000) 2019-10-13T16:28:55.295 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:28:55.288+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.293612+0000) 2019-10-13T16:28:55.296 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:28:55.287+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.293915+0000) 2019-10-13T16:28:55.297 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:55.288+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.295225+0000) 2019-10-13T16:28:55.302 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:55.295+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.299942+0000) 2019-10-13T16:28:55.305 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:28:55.296+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245231 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:28:55.795 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:28:55.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.793339+0000) 2019-10-13T16:28:55.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:28:55.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.793339+0000) 2019-10-13T16:28:55.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:28:55.800+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:55.802241+0000) 2019-10-13T16:28:56.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:28:56.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:56.344604+0000) 2019-10-13T16:28:57.293 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:28:57.287+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:28:57.290726+0000) 2019-10-13T16:29:00.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:00.264+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.266167+0000) 2019-10-13T16:29:00.307 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:00.300+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.305594+0000) 2019-10-13T16:29:00.308 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:00.300+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.306766+0000) 2019-10-13T16:29:00.309 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:00.304+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.307661+0000) 2019-10-13T16:29:00.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:00.304+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.308586+0000) 2019-10-13T16:29:00.311 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:00.303+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.308883+0000) 2019-10-13T16:29:00.311 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:00.304+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.309488+0000) 2019-10-13T16:29:00.312 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:00.304+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.310227+0000) 2019-10-13T16:29:00.315 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:00.307+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.313274+0000) 2019-10-13T16:29:00.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:00.312+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245263 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:00.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:00.511+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:00.515125+0000) 2019-10-13T16:29:02.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:02.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:02.782259+0000) 2019-10-13T16:29:05.324 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:05.319+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.322231+0000) 2019-10-13T16:29:05.328 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:05.316+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.322609+0000) 2019-10-13T16:29:05.328 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:05.316+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.322771+0000) 2019-10-13T16:29:05.328 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:05.320+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.323487+0000) 2019-10-13T16:29:05.329 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:05.320+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.324054+0000) 2019-10-13T16:29:05.329 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:05.320+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.324622+0000) 2019-10-13T16:29:05.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:05.320+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.325015+0000) 2019-10-13T16:29:05.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:05.323+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.328397+0000) 2019-10-13T16:29:05.336 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:05.328+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245295 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:05.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:05.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.793539+0000) 2019-10-13T16:29:05.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:05.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.793540+0000) 2019-10-13T16:29:05.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:05.800+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:05.802431+0000) 2019-10-13T16:29:06.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:06.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:06.344849+0000) 2019-10-13T16:29:07.304 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:07.299+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:07.302032+0000) 2019-10-13T16:29:10.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:10.264+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.266315+0000) 2019-10-13T16:29:10.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:10.332+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.337813+0000) 2019-10-13T16:29:10.340 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:10.331+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.337986+0000) 2019-10-13T16:29:10.341 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:10.332+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.338356+0000) 2019-10-13T16:29:10.341 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:10.332+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.338754+0000) 2019-10-13T16:29:10.341 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:10.336+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.339643+0000) 2019-10-13T16:29:10.342 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:10.336+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.340419+0000) 2019-10-13T16:29:10.343 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:10.336+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.341391+0000) 2019-10-13T16:29:10.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:10.339+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.344386+0000) 2019-10-13T16:29:10.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:10.344+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245327 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:10.517 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:10.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:10.515333+0000) 2019-10-13T16:29:12.784 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:12.776+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:12.782490+0000) 2019-10-13T16:29:15.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:15.348+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.353947+0000) 2019-10-13T16:29:15.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:15.350+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.354298+0000) 2019-10-13T16:29:15.357 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:15.348+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.354926+0000) 2019-10-13T16:29:15.357 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:15.348+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.354744+0000) 2019-10-13T16:29:15.358 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:15.352+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.355543+0000) 2019-10-13T16:29:15.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:15.352+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.356560+0000) 2019-10-13T16:29:15.359 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:15.352+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.357389+0000) 2019-10-13T16:29:15.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:15.358+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.362291+0000) 2019-10-13T16:29:15.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:15.360+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245359 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:15.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:15.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.793738+0000) 2019-10-13T16:29:15.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:15.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.793738+0000) 2019-10-13T16:29:15.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:15.800+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:15.802612+0000) 2019-10-13T16:29:16.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:16.340+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:16.345080+0000) 2019-10-13T16:29:17.310 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:17.306+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:17.308342+0000) 2019-10-13T16:29:20.268 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:20.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.266517+0000) 2019-10-13T16:29:20.371 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:20.363+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.369511+0000) 2019-10-13T16:29:20.373 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:20.363+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.371050+0000) 2019-10-13T16:29:20.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:20.367+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.371839+0000) 2019-10-13T16:29:20.374 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:20.367+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.372363+0000) 2019-10-13T16:29:20.375 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:20.366+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.372512+0000) 2019-10-13T16:29:20.375 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:20.367+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.372865+0000) 2019-10-13T16:29:20.375 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:20.367+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.373171+0000) 2019-10-13T16:29:20.382 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:20.374+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.380218+0000) 2019-10-13T16:29:20.384 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:20.375+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245390 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:20.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:20.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:20.515546+0000) 2019-10-13T16:29:22.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:22.775+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:22.782708+0000) 2019-10-13T16:29:24.104 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:29:24.104 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:29:24.107 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:29:24.108 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:29:25.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:25.379+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.385780+0000) 2019-10-13T16:29:25.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:25.379+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.386490+0000) 2019-10-13T16:29:25.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:25.383+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.387518+0000) 2019-10-13T16:29:25.390 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:25.382+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.387817+0000) 2019-10-13T16:29:25.390 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:25.383+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.388473+0000) 2019-10-13T16:29:25.390 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:25.383+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.388485+0000) 2019-10-13T16:29:25.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:25.383+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.389196+0000) 2019-10-13T16:29:25.395 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:25.386+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.393274+0000) 2019-10-13T16:29:25.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:25.391+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245423 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:25.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:25.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.793937+0000) 2019-10-13T16:29:25.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:25.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.793937+0000) 2019-10-13T16:29:25.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:25.795+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:25.802802+0000) 2019-10-13T16:29:26.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:26.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:26.345280+0000) 2019-10-13T16:29:27.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:27.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:27.315324+0000) 2019-10-13T16:29:30.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:30.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.266730+0000) 2019-10-13T16:29:30.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:30.395+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.401515+0000) 2019-10-13T16:29:30.404 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:30.395+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.402030+0000) 2019-10-13T16:29:30.404 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:30.395+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.402715+0000) 2019-10-13T16:29:30.405 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:30.395+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.403188+0000) 2019-10-13T16:29:30.405 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:30.399+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.403799+0000) 2019-10-13T16:29:30.406 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:30.399+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.404146+0000) 2019-10-13T16:29:30.407 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:30.398+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.404996+0000) 2019-10-13T16:29:30.415 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:30.407+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245455 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:30.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:30.410+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.414268+0000) 2019-10-13T16:29:30.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:30.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:30.515728+0000) 2019-10-13T16:29:32.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:32.779+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:32.782909+0000) 2019-10-13T16:29:35.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:35.411+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.416821+0000) 2019-10-13T16:29:35.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:35.411+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.417291+0000) 2019-10-13T16:29:35.420 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:35.411+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.418596+0000) 2019-10-13T16:29:35.421 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:35.414+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.418848+0000) 2019-10-13T16:29:35.422 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:35.415+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.419896+0000) 2019-10-13T16:29:35.422 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:35.415+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.420050+0000) 2019-10-13T16:29:35.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:35.415+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.420468+0000) 2019-10-13T16:29:35.426 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:35.418+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.424536+0000) 2019-10-13T16:29:35.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:35.423+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245487 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:35.796 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:35.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.794129+0000) 2019-10-13T16:29:35.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:35.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.794129+0000) 2019-10-13T16:29:35.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:35.799+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:35.802974+0000) 2019-10-13T16:29:36.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:36.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:36.345511+0000) 2019-10-13T16:29:37.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:37.314+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:37.321582+0000) 2019-10-13T16:29:40.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:40.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.266929+0000) 2019-10-13T16:29:40.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:40.427+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.432688+0000) 2019-10-13T16:29:40.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:40.427+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.433264+0000) 2019-10-13T16:29:40.435 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:40.427+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.433800+0000) 2019-10-13T16:29:40.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:40.430+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.434910+0000) 2019-10-13T16:29:40.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:40.427+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.435239+0000) 2019-10-13T16:29:40.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:40.431+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.436092+0000) 2019-10-13T16:29:40.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:40.431+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.436730+0000) 2019-10-13T16:29:40.440 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:40.434+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.438187+0000) 2019-10-13T16:29:40.446 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:40.439+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245518 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:40.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:40.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:40.515899+0000) 2019-10-13T16:29:42.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:42.775+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:42.783114+0000) 2019-10-13T16:29:45.450 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:45.443+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.447450+0000) 2019-10-13T16:29:45.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:45.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.447705+0000) 2019-10-13T16:29:45.451 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:45.443+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.449161+0000) 2019-10-13T16:29:45.452 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:45.443+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.450168+0000) 2019-10-13T16:29:45.452 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:45.446+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.450638+0000) 2019-10-13T16:29:45.453 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:45.443+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.451136+0000) 2019-10-13T16:29:45.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:45.447+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.451679+0000) 2019-10-13T16:29:45.462 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:45.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245551 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:45.464 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:45.458+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.462295+0000) 2019-10-13T16:29:45.796 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:45.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.794345+0000) 2019-10-13T16:29:45.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:45.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.794345+0000) 2019-10-13T16:29:45.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:45.799+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:45.803160+0000) 2019-10-13T16:29:46.347 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:46.339+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:46.345691+0000) 2019-10-13T16:29:47.330 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:47.326+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:47.327796+0000) 2019-10-13T16:29:50.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:50.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.267125+0000) 2019-10-13T16:29:50.466 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:50.458+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.464264+0000) 2019-10-13T16:29:50.467 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:50.463+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.464996+0000) 2019-10-13T16:29:50.468 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:50.463+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.466052+0000) 2019-10-13T16:29:50.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:50.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.466682+0000) 2019-10-13T16:29:50.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:50.467+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.467411+0000) 2019-10-13T16:29:50.469 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:50.463+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.467171+0000) 2019-10-13T16:29:50.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:50.467+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.467929+0000) 2019-10-13T16:29:50.474 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:50.466+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.472530+0000) 2019-10-13T16:29:50.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:50.475+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245583 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:50.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:50.510+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:50.516087+0000) 2019-10-13T16:29:52.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:52.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:52.783349+0000) 2019-10-13T16:29:54.236 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:29:54.236 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:29:54.240 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:29:54.240 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:29:55.481 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:55.475+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.479087+0000) 2019-10-13T16:29:55.482 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:55.479+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.480082+0000) 2019-10-13T16:29:55.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:29:55.479+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.481186+0000) 2019-10-13T16:29:55.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:29:55.479+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.481497+0000) 2019-10-13T16:29:55.483 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:29:55.474+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.481898+0000) 2019-10-13T16:29:55.484 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:55.479+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.482360+0000) 2019-10-13T16:29:55.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:55.479+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.482866+0000) 2019-10-13T16:29:55.488 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:55.478+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.485921+0000) 2019-10-13T16:29:55.492 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:29:55.487+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245615 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:29:55.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:29:55.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.794541+0000) 2019-10-13T16:29:55.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:29:55.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.794541+0000) 2019-10-13T16:29:55.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:29:55.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:55.803341+0000) 2019-10-13T16:29:56.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:29:56.343+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:56.345871+0000) 2019-10-13T16:29:57.336 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:29:57.334+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:29:57.334425+0000) 2019-10-13T16:30:00.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:00.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.267351+0000) 2019-10-13T16:30:00.495 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:00.491+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.493546+0000) 2019-10-13T16:30:00.496 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:00.491+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.494188+0000) 2019-10-13T16:30:00.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:00.491+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.494936+0000) 2019-10-13T16:30:00.498 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:00.495+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.496163+0000) 2019-10-13T16:30:00.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:00.495+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.496825+0000) 2019-10-13T16:30:00.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:00.494+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.497224+0000) 2019-10-13T16:30:00.499 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:00.495+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.497623+0000) 2019-10-13T16:30:00.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:00.502+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.504117+0000) 2019-10-13T16:30:00.508 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:00.503+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:00.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:00.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:00.516203+0000) 2019-10-13T16:30:02.785 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:02.782+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:02.783577+0000) 2019-10-13T16:30:05.512 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:05.506+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.510363+0000) 2019-10-13T16:30:05.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:05.506+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.511036+0000) 2019-10-13T16:30:05.513 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:05.510+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.511461+0000) 2019-10-13T16:30:05.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:05.510+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.511946+0000) 2019-10-13T16:30:05.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:05.510+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.512490+0000) 2019-10-13T16:30:05.516 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:05.510+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.514048+0000) 2019-10-13T16:30:05.516 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:05.510+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.514372+0000) 2019-10-13T16:30:05.524 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:05.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:05.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:05.522+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.523311+0000) 2019-10-13T16:30:05.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:05.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.794693+0000) 2019-10-13T16:30:05.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:05.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.794693+0000) 2019-10-13T16:30:05.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:05.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:05.803476+0000) 2019-10-13T16:30:06.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:06.342+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:06.346083+0000) 2019-10-13T16:30:07.347 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:07.341+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:07.345099+0000) 2019-10-13T16:30:10.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:10.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.267570+0000) 2019-10-13T16:30:10.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:10.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.516377+0000) 2019-10-13T16:30:10.528 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:10.522+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.525956+0000) 2019-10-13T16:30:10.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:10.522+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.526349+0000) 2019-10-13T16:30:10.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:10.525+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.527280+0000) 2019-10-13T16:30:10.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:10.526+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.527661+0000) 2019-10-13T16:30:10.530 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:10.526+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.528713+0000) 2019-10-13T16:30:10.531 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:10.526+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.528974+0000) 2019-10-13T16:30:10.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:10.526+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.530364+0000) 2019-10-13T16:30:10.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:10.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:10.534163+0000) 2019-10-13T16:30:10.539 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:10.534+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245711 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:12.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:12.782+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:12.783814+0000) 2019-10-13T16:30:15.543 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:15.538+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.541029+0000) 2019-10-13T16:30:15.543 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:15.538+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.541394+0000) 2019-10-13T16:30:15.544 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:15.538+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.542295+0000) 2019-10-13T16:30:15.544 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:15.538+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.542563+0000) 2019-10-13T16:30:15.545 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:15.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.543644+0000) 2019-10-13T16:30:15.546 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:15.542+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.544252+0000) 2019-10-13T16:30:15.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:15.541+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.544644+0000) 2019-10-13T16:30:15.555 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:15.550+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:15.556 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:15.549+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.553811+0000) 2019-10-13T16:30:15.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:15.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.794898+0000) 2019-10-13T16:30:15.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:15.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.794897+0000) 2019-10-13T16:30:15.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:15.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:15.803671+0000) 2019-10-13T16:30:16.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:16.342+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:16.346303+0000) 2019-10-13T16:30:17.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:17.353+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:17.356539+0000) 2019-10-13T16:30:20.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:20.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.267761+0000) 2019-10-13T16:30:20.518 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:20.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.516575+0000) 2019-10-13T16:30:20.559 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:20.553+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.557173+0000) 2019-10-13T16:30:20.560 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:20.554+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.558028+0000) 2019-10-13T16:30:20.560 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:20.554+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.558711+0000) 2019-10-13T16:30:20.561 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:20.554+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.558717+0000) 2019-10-13T16:30:20.561 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:20.554+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.559215+0000) 2019-10-13T16:30:20.562 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:20.558+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.559610+0000) 2019-10-13T16:30:20.562 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:20.558+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.560096+0000) 2019-10-13T16:30:20.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:20.565+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:20.566811+0000) 2019-10-13T16:30:20.571 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:20.566+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245774 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:22.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:22.782+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:22.784018+0000) 2019-10-13T16:30:24.360 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:30:24.360 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:30:24.366 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:30:24.366 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:30:25.576 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:25.573+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.574347+0000) 2019-10-13T16:30:25.577 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:25.570+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.574667+0000) 2019-10-13T16:30:25.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:25.570+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.575104+0000) 2019-10-13T16:30:25.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:25.574+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.575725+0000) 2019-10-13T16:30:25.578 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:25.574+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.576103+0000) 2019-10-13T16:30:25.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:25.574+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.576811+0000) 2019-10-13T16:30:25.579 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:25.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.577381+0000) 2019-10-13T16:30:25.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:25.581+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.583314+0000) 2019-10-13T16:30:25.588 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:25.582+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:25.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:25.790+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.795092+0000) 2019-10-13T16:30:25.797 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:25.790+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.795092+0000) 2019-10-13T16:30:25.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:25.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:25.803840+0000) 2019-10-13T16:30:26.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:26.342+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:26.346489+0000) 2019-10-13T16:30:27.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:27.361+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:27.362950+0000) 2019-10-13T16:30:30.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:30.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.267959+0000) 2019-10-13T16:30:30.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:30.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.516770+0000) 2019-10-13T16:30:30.591 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:30.586+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.589320+0000) 2019-10-13T16:30:30.592 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:30.589+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.590718+0000) 2019-10-13T16:30:30.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:30.586+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.591042+0000) 2019-10-13T16:30:30.593 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:30.590+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.591816+0000) 2019-10-13T16:30:30.594 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:30.590+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.591995+0000) 2019-10-13T16:30:30.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:30.590+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.592508+0000) 2019-10-13T16:30:30.595 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:30.590+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.593639+0000) 2019-10-13T16:30:30.603 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:30.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:30.604 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:30.601+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:30.602703+0000) 2019-10-13T16:30:32.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:32.782+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:32.784264+0000) 2019-10-13T16:30:35.608 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:35.602+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.605615+0000) 2019-10-13T16:30:35.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:35.602+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.605911+0000) 2019-10-13T16:30:35.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:35.605+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.606292+0000) 2019-10-13T16:30:35.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:35.606+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.607492+0000) 2019-10-13T16:30:35.610 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:35.606+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.607898+0000) 2019-10-13T16:30:35.610 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:35.606+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.608333+0000) 2019-10-13T16:30:35.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:35.606+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.608651+0000) 2019-10-13T16:30:35.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:35.609+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.611431+0000) 2019-10-13T16:30:35.619 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:35.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:35.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:35.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.795260+0000) 2019-10-13T16:30:35.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:35.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.795258+0000) 2019-10-13T16:30:35.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:35.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:35.804003+0000) 2019-10-13T16:30:36.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:36.342+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:36.346674+0000) 2019-10-13T16:30:37.371 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:37.365+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:37.369368+0000) 2019-10-13T16:30:40.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:40.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.268170+0000) 2019-10-13T16:30:40.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:40.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.516990+0000) 2019-10-13T16:30:40.622 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:40.617+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.620403+0000) 2019-10-13T16:30:40.623 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:40.618+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.621109+0000) 2019-10-13T16:30:40.624 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:40.618+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.622390+0000) 2019-10-13T16:30:40.625 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:40.622+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.623696+0000) 2019-10-13T16:30:40.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:40.622+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.624277+0000) 2019-10-13T16:30:40.626 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:40.622+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.624769+0000) 2019-10-13T16:30:40.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:40.622+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.625297+0000) 2019-10-13T16:30:40.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:40.629+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:40.630171+0000) 2019-10-13T16:30:40.635 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:40.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245902 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:42.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:42.782+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:42.784465+0000) 2019-10-13T16:30:45.638 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:45.634+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.635978+0000) 2019-10-13T16:30:45.639 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:45.633+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.636968+0000) 2019-10-13T16:30:45.639 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:45.634+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.637364+0000) 2019-10-13T16:30:45.640 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:45.634+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.638170+0000) 2019-10-13T16:30:45.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:45.634+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.638570+0000) 2019-10-13T16:30:45.641 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:45.634+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.639021+0000) 2019-10-13T16:30:45.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:45.638+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.639461+0000) 2019-10-13T16:30:45.648 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:45.645+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.646776+0000) 2019-10-13T16:30:45.650 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:45.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245934 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:45.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:45.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.795498+0000) 2019-10-13T16:30:45.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:45.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.795497+0000) 2019-10-13T16:30:45.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:45.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:45.804178+0000) 2019-10-13T16:30:46.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:46.342+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:46.346879+0000) 2019-10-13T16:30:47.383 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:47.377+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:47.380740+0000) 2019-10-13T16:30:50.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:50.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.268342+0000) 2019-10-13T16:30:50.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:50.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.517181+0000) 2019-10-13T16:30:50.654 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:50.649+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.652021+0000) 2019-10-13T16:30:50.654 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:50.649+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.652667+0000) 2019-10-13T16:30:50.655 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:50.649+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.653122+0000) 2019-10-13T16:30:50.655 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:50.649+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.653922+0000) 2019-10-13T16:30:50.657 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:50.649+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.655110+0000) 2019-10-13T16:30:50.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:50.653+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.655477+0000) 2019-10-13T16:30:50.658 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:50.653+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.655966+0000) 2019-10-13T16:30:50.661 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:50.657+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:50.659513+0000) 2019-10-13T16:30:50.665 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:50.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245968 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:52.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:52.781+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:52.784659+0000) 2019-10-13T16:30:54.483 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:30:54.484 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:30:54.487 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:30:54.488 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:30:55.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:30:55.661+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.666911+0000) 2019-10-13T16:30:55.670 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:30:55.665+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.667445+0000) 2019-10-13T16:30:55.670 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:55.665+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.667810+0000) 2019-10-13T16:30:55.671 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:55.665+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.669275+0000) 2019-10-13T16:30:55.672 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:30:55.665+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.669901+0000) 2019-10-13T16:30:55.672 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:55.665+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.670296+0000) 2019-10-13T16:30:55.672 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:55.665+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.670430+0000) 2019-10-13T16:30:55.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:55.673+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.675076+0000) 2019-10-13T16:30:55.682 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:30:55.677+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 245999 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:30:55.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:30:55.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.795698+0000) 2019-10-13T16:30:55.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:30:55.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.795690+0000) 2019-10-13T16:30:55.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:30:55.801+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:55.804387+0000) 2019-10-13T16:30:56.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:30:56.341+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:56.347022+0000) 2019-10-13T16:30:57.389 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:30:57.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:30:57.387272+0000) 2019-10-13T16:31:00.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:00.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.268531+0000) 2019-10-13T16:31:00.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:00.513+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.517413+0000) 2019-10-13T16:31:00.685 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:00.681+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.683352+0000) 2019-10-13T16:31:00.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:00.681+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.683651+0000) 2019-10-13T16:31:00.686 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:00.681+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.684743+0000) 2019-10-13T16:31:00.687 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:00.681+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.684703+0000) 2019-10-13T16:31:00.687 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:00.681+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.685164+0000) 2019-10-13T16:31:00.688 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:00.681+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.685611+0000) 2019-10-13T16:31:00.688 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:00.681+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.686061+0000) 2019-10-13T16:31:00.695 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:00.689+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:00.692877+0000) 2019-10-13T16:31:00.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:00.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246032 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:02.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:02.781+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:02.784878+0000) 2019-10-13T16:31:05.702 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:05.697+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.699823+0000) 2019-10-13T16:31:05.702 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:05.697+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.700531+0000) 2019-10-13T16:31:05.703 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:05.697+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.701387+0000) 2019-10-13T16:31:05.703 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:05.697+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.701531+0000) 2019-10-13T16:31:05.704 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:05.696+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.702076+0000) 2019-10-13T16:31:05.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:05.697+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.702387+0000) 2019-10-13T16:31:05.705 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:05.697+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.703009+0000) 2019-10-13T16:31:05.711 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:05.704+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.709083+0000) 2019-10-13T16:31:05.713 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:05.709+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246064 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:05.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:05.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.795871+0000) 2019-10-13T16:31:05.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:05.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.795901+0000) 2019-10-13T16:31:05.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:05.801+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:05.804561+0000) 2019-10-13T16:31:06.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:06.345+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:06.347173+0000) 2019-10-13T16:31:07.396 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:07.392+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:07.393716+0000) 2019-10-13T16:31:10.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:10.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.268757+0000) 2019-10-13T16:31:10.519 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:10.512+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.517636+0000) 2019-10-13T16:31:10.717 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:10.709+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.715036+0000) 2019-10-13T16:31:10.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:10.713+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.715738+0000) 2019-10-13T16:31:10.718 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:10.713+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.716019+0000) 2019-10-13T16:31:10.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:10.713+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.716504+0000) 2019-10-13T16:31:10.719 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:10.713+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.716902+0000) 2019-10-13T16:31:10.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:10.712+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.717299+0000) 2019-10-13T16:31:10.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:10.713+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.718483+0000) 2019-10-13T16:31:10.726 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:10.720+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:10.724588+0000) 2019-10-13T16:31:10.728 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:10.721+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246098 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:12.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:12.781+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:12.785074+0000) 2019-10-13T16:31:15.732 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:15.725+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.729713+0000) 2019-10-13T16:31:15.733 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:15.725+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.730518+0000) 2019-10-13T16:31:15.733 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:15.728+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.730964+0000) 2019-10-13T16:31:15.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:15.729+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.731403+0000) 2019-10-13T16:31:15.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:15.729+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.732088+0000) 2019-10-13T16:31:15.734 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:15.729+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.732434+0000) 2019-10-13T16:31:15.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:15.729+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.732725+0000) 2019-10-13T16:31:15.743 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:15.736+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.740005+0000) 2019-10-13T16:31:15.744 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:15.737+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246129 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:15.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:15.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.796078+0000) 2019-10-13T16:31:15.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:15.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.796058+0000) 2019-10-13T16:31:15.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:15.801+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:15.804741+0000) 2019-10-13T16:31:16.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:16.345+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:16.347362+0000) 2019-10-13T16:31:17.402 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:17.396+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:17.400068+0000) 2019-10-13T16:31:20.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:20.265+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.268956+0000) 2019-10-13T16:31:20.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:20.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.517831+0000) 2019-10-13T16:31:20.748 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:20.741+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.746537+0000) 2019-10-13T16:31:20.749 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:20.744+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.746754+0000) 2019-10-13T16:31:20.750 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:20.745+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.747497+0000) 2019-10-13T16:31:20.750 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:20.745+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.747783+0000) 2019-10-13T16:31:20.750 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:20.745+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.748200+0000) 2019-10-13T16:31:20.751 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:20.745+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.748386+0000) 2019-10-13T16:31:20.751 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:20.745+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.749387+0000) 2019-10-13T16:31:20.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:20.748+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:20.753248+0000) 2019-10-13T16:31:20.760 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:20.753+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246161 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:22.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:22.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:22.785300+0000) 2019-10-13T16:31:24.624 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:31:24.624 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:31:24.627 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:31:24.628 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:31:25.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:25.757+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.761529+0000) 2019-10-13T16:31:25.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:25.760+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.762357+0000) 2019-10-13T16:31:25.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:25.757+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.762676+0000) 2019-10-13T16:31:25.765 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:25.761+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.763437+0000) 2019-10-13T16:31:25.765 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:25.761+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.763564+0000) 2019-10-13T16:31:25.766 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:25.761+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.763927+0000) 2019-10-13T16:31:25.767 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:25.761+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.764922+0000) 2019-10-13T16:31:25.773 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:25.768+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.771365+0000) 2019-10-13T16:31:25.776 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:25.769+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246193 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:25.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:25.793+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.796264+0000) 2019-10-13T16:31:25.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:25.793+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.796275+0000) 2019-10-13T16:31:25.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:25.801+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:25.804909+0000) 2019-10-13T16:31:26.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:26.345+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:26.347548+0000) 2019-10-13T16:31:27.409 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:27.404+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:27.406530+0000) 2019-10-13T16:31:30.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:30.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.269111+0000) 2019-10-13T16:31:30.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:30.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.518015+0000) 2019-10-13T16:31:30.780 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:30.772+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.778195+0000) 2019-10-13T16:31:30.780 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:30.772+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.778313+0000) 2019-10-13T16:31:30.781 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:30.772+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.778823+0000) 2019-10-13T16:31:30.781 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:30.772+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.779204+0000) 2019-10-13T16:31:30.782 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:30.776+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.779517+0000) 2019-10-13T16:31:30.782 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:30.776+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.780205+0000) 2019-10-13T16:31:30.782 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:30.776+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.780663+0000) 2019-10-13T16:31:30.789 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:30.784+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:30.787599+0000) 2019-10-13T16:31:30.791 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:30.784+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:32.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:32.780+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:32.785535+0000) 2019-10-13T16:31:35.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:35.788+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.792750+0000) 2019-10-13T16:31:35.795 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:35.788+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.793181+0000) 2019-10-13T16:31:35.796 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:35.788+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.793629+0000) 2019-10-13T16:31:35.797 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:35.788+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.795065+0000) 2019-10-13T16:31:35.797 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:35.788+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.795114+0000) 2019-10-13T16:31:35.797 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:35.792+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.795393+0000) 2019-10-13T16:31:35.798 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:35.792+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.795759+0000) 2019-10-13T16:31:35.798 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:35.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.796403+0000) 2019-10-13T16:31:35.798 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:35.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.796404+0000) 2019-10-13T16:31:35.803 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:35.796+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.801013+0000) 2019-10-13T16:31:35.806 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:35.800+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:35.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:35.800+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:35.805047+0000) 2019-10-13T16:31:36.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:36.344+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:36.347744+0000) 2019-10-13T16:31:37.415 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:37.408+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:37.413089+0000) 2019-10-13T16:31:40.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:40.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.269306+0000) 2019-10-13T16:31:40.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:40.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.518213+0000) 2019-10-13T16:31:40.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:40.800+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.806885+0000) 2019-10-13T16:31:40.809 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:40.804+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.807782+0000) 2019-10-13T16:31:40.810 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:40.804+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.808223+0000) 2019-10-13T16:31:40.811 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:40.804+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.809167+0000) 2019-10-13T16:31:40.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:40.804+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.809589+0000) 2019-10-13T16:31:40.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:40.804+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.810048+0000) 2019-10-13T16:31:40.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:40.804+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.811128+0000) 2019-10-13T16:31:40.819 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:40.812+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:40.817014+0000) 2019-10-13T16:31:40.821 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:40.816+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:42.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:42.784+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:42.785754+0000) 2019-10-13T16:31:45.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:45.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.796551+0000) 2019-10-13T16:31:45.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:45.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.796620+0000) 2019-10-13T16:31:45.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:45.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.805266+0000) 2019-10-13T16:31:45.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:45.820+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.823416+0000) 2019-10-13T16:31:45.827 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:45.820+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.824901+0000) 2019-10-13T16:31:45.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:45.820+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.825058+0000) 2019-10-13T16:31:45.828 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:45.820+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.825378+0000) 2019-10-13T16:31:45.829 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:45.820+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.825863+0000) 2019-10-13T16:31:45.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:45.820+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.826303+0000) 2019-10-13T16:31:45.830 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:45.820+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.826977+0000) 2019-10-13T16:31:45.832 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:45.828+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:45.830411+0000) 2019-10-13T16:31:45.836 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:45.828+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:46.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:46.344+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:46.347929+0000) 2019-10-13T16:31:47.422 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:47.416+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:47.419610+0000) 2019-10-13T16:31:50.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:50.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.269508+0000) 2019-10-13T16:31:50.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:50.516+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.518455+0000) 2019-10-13T16:31:50.840 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:50.832+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.838349+0000) 2019-10-13T16:31:50.841 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:50.832+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.838405+0000) 2019-10-13T16:31:50.841 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:50.832+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.839167+0000) 2019-10-13T16:31:50.842 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:50.836+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.840183+0000) 2019-10-13T16:31:50.842 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:50.836+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.840701+0000) 2019-10-13T16:31:50.843 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:50.836+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.841186+0000) 2019-10-13T16:31:50.844 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:50.836+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.842471+0000) 2019-10-13T16:31:50.849 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:50.844+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:50.847333+0000) 2019-10-13T16:31:50.851 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:50.844+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246354 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:52.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:52.784+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:52.785982+0000) 2019-10-13T16:31:54.722 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:31:54.723 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:31:54.727 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:31:54.728 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:31:55.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:55.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.796748+0000) 2019-10-13T16:31:55.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:55.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.796819+0000) 2019-10-13T16:31:55.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:55.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.805462+0000) 2019-10-13T16:31:55.855 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:31:55.848+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.853433+0000) 2019-10-13T16:31:55.856 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:55.848+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.854731+0000) 2019-10-13T16:31:55.857 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:31:55.852+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.855152+0000) 2019-10-13T16:31:55.857 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:31:55.852+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.855600+0000) 2019-10-13T16:31:55.858 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:31:55.852+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.855849+0000) 2019-10-13T16:31:55.858 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:31:55.852+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.856262+0000) 2019-10-13T16:31:55.859 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:31:55.852+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.857133+0000) 2019-10-13T16:31:55.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:55.860+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:55.863217+0000) 2019-10-13T16:31:55.867 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:31:55.860+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246384 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:31:56.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:31:56.344+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:56.348115+0000) 2019-10-13T16:31:57.433 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:31:57.428+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:31:57.430827+0000) 2019-10-13T16:32:00.271 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:00.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.269709+0000) 2019-10-13T16:32:00.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:00.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.518656+0000) 2019-10-13T16:32:00.871 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:00.864+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.868946+0000) 2019-10-13T16:32:00.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:00.864+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.869566+0000) 2019-10-13T16:32:00.872 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:00.864+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.870394+0000) 2019-10-13T16:32:00.873 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:00.868+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.871668+0000) 2019-10-13T16:32:00.874 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:00.867+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.872057+0000) 2019-10-13T16:32:00.874 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:00.868+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.872885+0000) 2019-10-13T16:32:00.875 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:00.868+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.873598+0000) 2019-10-13T16:32:00.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:00.871+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:00.876597+0000) 2019-10-13T16:32:00.884 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:00.876+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246417 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:02.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:02.784+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:02.786182+0000) 2019-10-13T16:32:05.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:05.792+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.796954+0000) 2019-10-13T16:32:05.799 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:05.792+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.797024+0000) 2019-10-13T16:32:05.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:05.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.805647+0000) 2019-10-13T16:32:05.887 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:05.879+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.884856+0000) 2019-10-13T16:32:05.888 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:05.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.886050+0000) 2019-10-13T16:32:05.888 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:05.880+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.886287+0000) 2019-10-13T16:32:05.889 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:05.880+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.886861+0000) 2019-10-13T16:32:05.889 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:05.880+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.887166+0000) 2019-10-13T16:32:05.889 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:05.884+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.887455+0000) 2019-10-13T16:32:05.890 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:05.884+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.888411+0000) 2019-10-13T16:32:05.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:05.887+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:05.891624+0000) 2019-10-13T16:32:05.899 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:05.892+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:06.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:06.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:06.348329+0000) 2019-10-13T16:32:07.439 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:07.431+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:07.437178+0000) 2019-10-13T16:32:10.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:10.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.269905+0000) 2019-10-13T16:32:10.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:10.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.518866+0000) 2019-10-13T16:32:10.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:10.896+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.901446+0000) 2019-10-13T16:32:10.904 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:10.896+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.902139+0000) 2019-10-13T16:32:10.905 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:10.900+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.903616+0000) 2019-10-13T16:32:10.906 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:10.899+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.903946+0000) 2019-10-13T16:32:10.906 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:10.900+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.904320+0000) 2019-10-13T16:32:10.907 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:10.900+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.904808+0000) 2019-10-13T16:32:10.907 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:10.900+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.905160+0000) 2019-10-13T16:32:10.910 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:10.903+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:10.908753+0000) 2019-10-13T16:32:10.915 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:10.908+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:12.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:12.784+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:12.786383+0000) 2019-10-13T16:32:15.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:15.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.797189+0000) 2019-10-13T16:32:15.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:15.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.797234+0000) 2019-10-13T16:32:15.807 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:15.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.805874+0000) 2019-10-13T16:32:15.919 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:15.911+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.916692+0000) 2019-10-13T16:32:15.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:15.911+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.916942+0000) 2019-10-13T16:32:15.920 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:15.911+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.917791+0000) 2019-10-13T16:32:15.920 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:15.911+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.918102+0000) 2019-10-13T16:32:15.920 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:15.915+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.918352+0000) 2019-10-13T16:32:15.921 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:15.911+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.918975+0000) 2019-10-13T16:32:15.922 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:15.915+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.920310+0000) 2019-10-13T16:32:15.928 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:15.923+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:15.926617+0000) 2019-10-13T16:32:15.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:15.923+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246513 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:16.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:16.347+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:16.348510+0000) 2019-10-13T16:32:17.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:17.439+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:17.443604+0000) 2019-10-13T16:32:20.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:20.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.270080+0000) 2019-10-13T16:32:20.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:20.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.519075+0000) 2019-10-13T16:32:20.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:20.927+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.931981+0000) 2019-10-13T16:32:20.934 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:20.927+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.932586+0000) 2019-10-13T16:32:20.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:20.927+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.933607+0000) 2019-10-13T16:32:20.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:20.927+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.933855+0000) 2019-10-13T16:32:20.936 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:20.927+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.934545+0000) 2019-10-13T16:32:20.937 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:20.931+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.934909+0000) 2019-10-13T16:32:20.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:20.931+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.935898+0000) 2019-10-13T16:32:20.944 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:20.939+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:20.942310+0000) 2019-10-13T16:32:20.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:20.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246547 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:22.788 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:22.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:22.786556+0000) 2019-10-13T16:32:24.863 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:32:24.863 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:32:24.867 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:32:24.867 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:32:25.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:25.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.797392+0000) 2019-10-13T16:32:25.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:25.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.797434+0000) 2019-10-13T16:32:25.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:25.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.806070+0000) 2019-10-13T16:32:25.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:25.943+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.947982+0000) 2019-10-13T16:32:25.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:25.943+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.949016+0000) 2019-10-13T16:32:25.951 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:25.943+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.949581+0000) 2019-10-13T16:32:25.952 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:25.943+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.949909+0000) 2019-10-13T16:32:25.952 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:25.943+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.950232+0000) 2019-10-13T16:32:25.953 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:25.947+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.951102+0000) 2019-10-13T16:32:25.953 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:25.947+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.951527+0000) 2019-10-13T16:32:25.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:25.951+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:25.954918+0000) 2019-10-13T16:32:25.962 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:25.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246579 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:26.350 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:26.343+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:26.348686+0000) 2019-10-13T16:32:27.452 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:27.447+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:27.449887+0000) 2019-10-13T16:32:30.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:30.263+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.270278+0000) 2019-10-13T16:32:30.521 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:30.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.519288+0000) 2019-10-13T16:32:30.966 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:30.959+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.964472+0000) 2019-10-13T16:32:30.967 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:30.959+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.965158+0000) 2019-10-13T16:32:30.967 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:30.959+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.965195+0000) 2019-10-13T16:32:30.968 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:30.959+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.966031+0000) 2019-10-13T16:32:30.969 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:30.963+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.967026+0000) 2019-10-13T16:32:30.969 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:30.963+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.967555+0000) 2019-10-13T16:32:30.971 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:30.963+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.967999+0000) 2019-10-13T16:32:30.975 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:30.967+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:30.972989+0000) 2019-10-13T16:32:30.978 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:30.971+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246613 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:32.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:32.779+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:32.786754+0000) 2019-10-13T16:32:35.799 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:35.791+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.797595+0000) 2019-10-13T16:32:35.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:35.791+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.797646+0000) 2019-10-13T16:32:35.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:35.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.806273+0000) 2019-10-13T16:32:35.982 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:35.975+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.980145+0000) 2019-10-13T16:32:35.983 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:35.975+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.980501+0000) 2019-10-13T16:32:35.983 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:35.975+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.980752+0000) 2019-10-13T16:32:35.984 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:35.975+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.981347+0000) 2019-10-13T16:32:35.984 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:35.975+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.981967+0000) 2019-10-13T16:32:35.985 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:35.975+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.982816+0000) 2019-10-13T16:32:35.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:35.979+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.983829+0000) 2019-10-13T16:32:35.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:35.983+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:35.988672+0000) 2019-10-13T16:32:35.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:35.987+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246643 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:36.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:36.347+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:36.348867+0000) 2019-10-13T16:32:37.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:37.455+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:37.456397+0000) 2019-10-13T16:32:40.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:40.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.270501+0000) 2019-10-13T16:32:40.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:40.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.519539+0000) 2019-10-13T16:32:40.999 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:40.991+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.996986+0000) 2019-10-13T16:32:40.999 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:40.991+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.997368+0000) 2019-10-13T16:32:41.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:40.991+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.998453+0000) 2019-10-13T16:32:41.001 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:40.995+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.998903+0000) 2019-10-13T16:32:41.001 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:40.995+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.999611+0000) 2019-10-13T16:32:41.002 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:40.995+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:40.999736+0000) 2019-10-13T16:32:41.002 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:40.995+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:41.000115+0000) 2019-10-13T16:32:41.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:40.999+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:41.004081+0000) 2019-10-13T16:32:41.010 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:41.003+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246677 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:42.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:42.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:42.786958+0000) 2019-10-13T16:32:45.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:45.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:45.797788+0000) 2019-10-13T16:32:45.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:45.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:45.797867+0000) 2019-10-13T16:32:45.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:45.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:45.806509+0000) 2019-10-13T16:32:46.021 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:46.007+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.013773+0000) 2019-10-13T16:32:46.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:46.011+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.017677+0000) 2019-10-13T16:32:46.021 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:46.007+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.011043+0000) 2019-10-13T16:32:46.021 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:46.011+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.011900+0000) 2019-10-13T16:32:46.022 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:46.011+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.012631+0000) 2019-10-13T16:32:46.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:46.011+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.013738+0000) 2019-10-13T16:32:46.022 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:46.011+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.014116+0000) 2019-10-13T16:32:46.022 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:46.011+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.014658+0000) 2019-10-13T16:32:46.028 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:46.023+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:46.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:46.347+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:46.349047+0000) 2019-10-13T16:32:47.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:47.459+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:47.462930+0000) 2019-10-13T16:32:50.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:50.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:50.270708+0000) 2019-10-13T16:32:50.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:50.515+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:50.519776+0000) 2019-10-13T16:32:51.029 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:51.027+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.027377+0000) 2019-10-13T16:32:51.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:51.027+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.028102+0000) 2019-10-13T16:32:51.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:51.027+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.028700+0000) 2019-10-13T16:32:51.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:51.027+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.028899+0000) 2019-10-13T16:32:51.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:51.027+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.029796+0000) 2019-10-13T16:32:51.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:51.027+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.030615+0000) 2019-10-13T16:32:51.033 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:51.027+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.031026+0000) 2019-10-13T16:32:51.037 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:51.031+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:51.035375+0000) 2019-10-13T16:32:51.041 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:51.039+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246743 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:52.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:52.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:52.787136+0000) 2019-10-13T16:32:55.178 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:32:55.178 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:32:55.182 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:32:55.182 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:32:55.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:55.795+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:55.798008+0000) 2019-10-13T16:32:55.800 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:55.795+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:55.798087+0000) 2019-10-13T16:32:55.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:55.803+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:55.806708+0000) 2019-10-13T16:32:56.044 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:32:56.039+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.042315+0000) 2019-10-13T16:32:56.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:32:56.039+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.043070+0000) 2019-10-13T16:32:56.046 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:32:56.043+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.044011+0000) 2019-10-13T16:32:56.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:32:56.043+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.044474+0000) 2019-10-13T16:32:56.046 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:32:56.038+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.044667+0000) 2019-10-13T16:32:56.048 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:56.043+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.046142+0000) 2019-10-13T16:32:56.048 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:32:56.043+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.046203+0000) 2019-10-13T16:32:56.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:56.046+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.051885+0000) 2019-10-13T16:32:56.056 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:32:56.051+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:32:56.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:32:56.347+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:56.349224+0000) 2019-10-13T16:32:57.471 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:32:57.462+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:32:57.469249+0000) 2019-10-13T16:33:00.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:00.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:00.270914+0000) 2019-10-13T16:33:00.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:00.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:00.519971+0000) 2019-10-13T16:33:01.060 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:01.054+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.057794+0000) 2019-10-13T16:33:01.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:01.054+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.058329+0000) 2019-10-13T16:33:01.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:01.058+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.059763+0000) 2019-10-13T16:33:01.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:01.054+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.059856+0000) 2019-10-13T16:33:01.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:01.058+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.060165+0000) 2019-10-13T16:33:01.063 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:01.058+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.060423+0000) 2019-10-13T16:33:01.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:01.058+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.061694+0000) 2019-10-13T16:33:01.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:01.062+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:01.067240+0000) 2019-10-13T16:33:01.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:01.066+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:02.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:02.786+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:02.787357+0000) 2019-10-13T16:33:05.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:05.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:05.798228+0000) 2019-10-13T16:33:05.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:05.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:05.798308+0000) 2019-10-13T16:33:05.808 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:05.802+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:05.806909+0000) 2019-10-13T16:33:06.076 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:06.070+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.074072+0000) 2019-10-13T16:33:06.076 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:06.070+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.074473+0000) 2019-10-13T16:33:06.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:06.070+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.074890+0000) 2019-10-13T16:33:06.078 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:06.074+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.075644+0000) 2019-10-13T16:33:06.078 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:06.074+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.075675+0000) 2019-10-13T16:33:06.078 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:06.074+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.076082+0000) 2019-10-13T16:33:06.079 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:06.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.076362+0000) 2019-10-13T16:33:06.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:06.078+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.082872+0000) 2019-10-13T16:33:06.088 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:06.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:06.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:06.346+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:06.349407+0000) 2019-10-13T16:33:07.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:07.474+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:07.475704+0000) 2019-10-13T16:33:10.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:10.266+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:10.271112+0000) 2019-10-13T16:33:10.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:10.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:10.520168+0000) 2019-10-13T16:33:11.091 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:11.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.088928+0000) 2019-10-13T16:33:11.093 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:11.086+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.091240+0000) 2019-10-13T16:33:11.094 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:11.086+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.091004+0000) 2019-10-13T16:33:11.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:11.090+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.091768+0000) 2019-10-13T16:33:11.094 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:11.090+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.091548+0000) 2019-10-13T16:33:11.095 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:11.090+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.092371+0000) 2019-10-13T16:33:11.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:11.090+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.092064+0000) 2019-10-13T16:33:11.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:11.090+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:11.097136+0000) 2019-10-13T16:33:11.103 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:11.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246872 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:12.789 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:12.786+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:12.787590+0000) 2019-10-13T16:33:15.800 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:15.794+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:15.798426+0000) 2019-10-13T16:33:15.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:15.794+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:15.798549+0000) 2019-10-13T16:33:15.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:15.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:15.807122+0000) 2019-10-13T16:33:16.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:16.102+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.104436+0000) 2019-10-13T16:33:16.107 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:16.098+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.104874+0000) 2019-10-13T16:33:16.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:16.102+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.105386+0000) 2019-10-13T16:33:16.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:16.102+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.106896+0000) 2019-10-13T16:33:16.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:16.102+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.107078+0000) 2019-10-13T16:33:16.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:16.106+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.107888+0000) 2019-10-13T16:33:16.110 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:16.106+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.107896+0000) 2019-10-13T16:33:16.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:16.106+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.114001+0000) 2019-10-13T16:33:16.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:16.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246903 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:16.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:16.346+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:16.349636+0000) 2019-10-13T16:33:17.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:17.478+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:17.482061+0000) 2019-10-13T16:33:20.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:20.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:20.271338+0000) 2019-10-13T16:33:20.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:20.514+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:20.520393+0000) 2019-10-13T16:33:21.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:21.114+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.119474+0000) 2019-10-13T16:33:21.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:21.118+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.119987+0000) 2019-10-13T16:33:21.123 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:21.118+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.120865+0000) 2019-10-13T16:33:21.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:21.118+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.120298+0000) 2019-10-13T16:33:21.123 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:21.118+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.121119+0000) 2019-10-13T16:33:21.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:21.118+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.121815+0000) 2019-10-13T16:33:21.124 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:21.118+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.122946+0000) 2019-10-13T16:33:21.130 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:21.122+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:21.128158+0000) 2019-10-13T16:33:21.134 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:21.130+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246936 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:22.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:22.786+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:22.787760+0000) 2019-10-13T16:33:25.444 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:33:25.444 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:33:25.448 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:33:25.448 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:33:25.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:25.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:25.798630+0000) 2019-10-13T16:33:25.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:25.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:25.798776+0000) 2019-10-13T16:33:25.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:25.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:25.807338+0000) 2019-10-13T16:33:26.138 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:26.130+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.135814+0000) 2019-10-13T16:33:26.138 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:26.134+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.136432+0000) 2019-10-13T16:33:26.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:26.134+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.136939+0000) 2019-10-13T16:33:26.139 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:26.134+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.137070+0000) 2019-10-13T16:33:26.140 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:26.134+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.138091+0000) 2019-10-13T16:33:26.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:26.134+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.138314+0000) 2019-10-13T16:33:26.141 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:26.134+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.138720+0000) 2019-10-13T16:33:26.146 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:26.138+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.143978+0000) 2019-10-13T16:33:26.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:26.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 246969 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:26.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:26.346+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:26.349869+0000) 2019-10-13T16:33:27.495 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:27.490+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:27.493500+0000) 2019-10-13T16:33:30.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:30.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:30.271491+0000) 2019-10-13T16:33:30.522 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:30.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:30.520599+0000) 2019-10-13T16:33:31.154 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:31.150+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.151344+0000) 2019-10-13T16:33:31.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:31.146+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.151057+0000) 2019-10-13T16:33:31.154 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:31.150+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.152329+0000) 2019-10-13T16:33:31.155 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:31.146+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.152811+0000) 2019-10-13T16:33:31.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:31.150+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.153146+0000) 2019-10-13T16:33:31.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:31.150+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.153794+0000) 2019-10-13T16:33:31.156 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:31.150+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.154319+0000) 2019-10-13T16:33:31.161 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:31.154+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:31.159093+0000) 2019-10-13T16:33:31.164 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:31.158+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247002 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:32.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:32.786+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:32.787936+0000) 2019-10-13T16:33:35.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:35.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:35.798864+0000) 2019-10-13T16:33:35.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:35.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:35.799008+0000) 2019-10-13T16:33:35.809 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:35.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:35.807565+0000) 2019-10-13T16:33:36.168 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:36.162+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.165808+0000) 2019-10-13T16:33:36.168 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:36.162+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.166372+0000) 2019-10-13T16:33:36.169 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:36.162+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.166763+0000) 2019-10-13T16:33:36.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:36.162+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.167119+0000) 2019-10-13T16:33:36.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:36.162+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.168910+0000) 2019-10-13T16:33:36.171 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:36.166+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.167672+0000) 2019-10-13T16:33:36.171 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:36.166+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.169364+0000) 2019-10-13T16:33:36.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:36.170+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.175450+0000) 2019-10-13T16:33:36.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:36.174+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247033 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:36.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:36.346+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:36.350080+0000) 2019-10-13T16:33:37.506 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:37.502+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:37.503685+0000) 2019-10-13T16:33:40.273 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:40.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:40.271711+0000) 2019-10-13T16:33:40.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:40.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:40.520823+0000) 2019-10-13T16:33:41.183 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:41.178+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.181499+0000) 2019-10-13T16:33:41.184 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:41.178+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.182282+0000) 2019-10-13T16:33:41.185 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:41.178+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.182885+0000) 2019-10-13T16:33:41.185 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:41.178+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.183274+0000) 2019-10-13T16:33:41.186 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:41.182+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.183844+0000) 2019-10-13T16:33:41.186 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:41.182+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.184516+0000) 2019-10-13T16:33:41.187 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:41.182+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.184652+0000) 2019-10-13T16:33:41.191 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:41.186+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:41.188908+0000) 2019-10-13T16:33:41.195 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:41.190+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247066 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:42.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:42.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:42.788111+0000) 2019-10-13T16:33:45.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:45.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:45.799081+0000) 2019-10-13T16:33:45.801 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:45.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:45.799180+0000) 2019-10-13T16:33:45.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:45.805+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:45.807759+0000) 2019-10-13T16:33:46.199 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:46.193+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.196671+0000) 2019-10-13T16:33:46.199 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:46.193+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.197306+0000) 2019-10-13T16:33:46.200 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:46.194+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.198018+0000) 2019-10-13T16:33:46.200 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:46.193+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.198847+0000) 2019-10-13T16:33:46.201 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:46.197+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.199706+0000) 2019-10-13T16:33:46.202 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:46.197+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.200450+0000) 2019-10-13T16:33:46.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:46.197+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.200854+0000) 2019-10-13T16:33:46.207 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:46.202+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.205825+0000) 2019-10-13T16:33:46.212 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:46.205+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247102 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:46.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:46.349+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:46.350293+0000) 2019-10-13T16:33:47.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:47.510+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:47.510244+0000) 2019-10-13T16:33:50.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:50.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:50.271911+0000) 2019-10-13T16:33:50.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:50.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:50.521019+0000) 2019-10-13T16:33:51.216 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:51.209+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.213968+0000) 2019-10-13T16:33:51.216 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:51.209+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.214037+0000) 2019-10-13T16:33:51.216 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:51.209+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.214398+0000) 2019-10-13T16:33:51.217 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:51.213+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.215676+0000) 2019-10-13T16:33:51.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:51.213+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.216007+0000) 2019-10-13T16:33:51.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:51.213+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.216281+0000) 2019-10-13T16:33:51.219 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:51.213+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.217538+0000) 2019-10-13T16:33:51.225 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:51.221+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:51.223088+0000) 2019-10-13T16:33:51.229 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:51.221+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247135 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:52.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:52.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:52.788265+0000) 2019-10-13T16:33:55.571 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:33:55.572 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:33:55.575 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:33:55.575 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:33:55.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:55.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:55.799280+0000) 2019-10-13T16:33:55.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:55.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:55.799389+0000) 2019-10-13T16:33:55.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:55.805+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:55.807984+0000) 2019-10-13T16:33:56.232 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:33:56.225+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.230138+0000) 2019-10-13T16:33:56.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:56.225+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.231229+0000) 2019-10-13T16:33:56.233 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:33:56.225+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.231243+0000) 2019-10-13T16:33:56.234 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:33:56.229+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.231893+0000) 2019-10-13T16:33:56.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:33:56.229+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.232413+0000) 2019-10-13T16:33:56.235 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:33:56.229+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.232877+0000) 2019-10-13T16:33:56.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:33:56.229+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.233592+0000) 2019-10-13T16:33:56.245 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:33:56.237+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247169 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:33:56.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:56.241+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.243289+0000) 2019-10-13T16:33:56.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:33:56.349+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:56.350474+0000) 2019-10-13T16:33:57.518 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:33:57.513+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:33:57.516652+0000) 2019-10-13T16:34:00.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:00.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:00.272117+0000) 2019-10-13T16:34:00.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:00.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:00.521233+0000) 2019-10-13T16:34:01.248 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:01.241+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.245941+0000) 2019-10-13T16:34:01.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:01.241+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.246952+0000) 2019-10-13T16:34:01.252 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:01.245+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.247656+0000) 2019-10-13T16:34:01.253 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:01.245+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.248058+0000) 2019-10-13T16:34:01.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:01.245+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.248909+0000) 2019-10-13T16:34:01.253 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:01.245+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.249175+0000) 2019-10-13T16:34:01.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:01.245+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.247095+0000) 2019-10-13T16:34:01.257 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:01.253+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:01.255746+0000) 2019-10-13T16:34:01.262 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:01.257+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247202 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:02.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:02.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:02.788448+0000) 2019-10-13T16:34:05.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:05.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:05.799522+0000) 2019-10-13T16:34:05.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:05.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:05.799593+0000) 2019-10-13T16:34:05.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:05.805+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:05.808198+0000) 2019-10-13T16:34:06.265 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:06.261+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.263347+0000) 2019-10-13T16:34:06.267 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:06.261+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.264928+0000) 2019-10-13T16:34:06.268 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:06.261+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.265775+0000) 2019-10-13T16:34:06.268 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:06.261+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.266225+0000) 2019-10-13T16:34:06.268 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:06.265+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.266748+0000) 2019-10-13T16:34:06.269 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:06.265+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.267868+0000) 2019-10-13T16:34:06.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:06.265+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.268359+0000) 2019-10-13T16:34:06.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:06.269+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.271684+0000) 2019-10-13T16:34:06.277 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:06.273+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247238 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:06.352 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:06.349+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:06.350641+0000) 2019-10-13T16:34:07.525 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:07.521+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:07.523097+0000) 2019-10-13T16:34:10.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:10.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:10.272308+0000) 2019-10-13T16:34:10.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:10.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:10.521424+0000) 2019-10-13T16:34:11.281 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:11.273+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.279055+0000) 2019-10-13T16:34:11.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:11.277+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.279357+0000) 2019-10-13T16:34:11.282 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:11.277+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.280540+0000) 2019-10-13T16:34:11.283 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:11.277+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.281119+0000) 2019-10-13T16:34:11.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:11.277+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.281393+0000) 2019-10-13T16:34:11.284 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:11.277+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.281929+0000) 2019-10-13T16:34:11.284 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:11.277+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.282403+0000) 2019-10-13T16:34:11.287 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:11.281+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:11.285436+0000) 2019-10-13T16:34:11.294 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:11.289+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247270 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:12.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:12.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:12.788652+0000) 2019-10-13T16:34:15.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:15.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:15.799662+0000) 2019-10-13T16:34:15.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:15.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:15.799784+0000) 2019-10-13T16:34:15.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:15.805+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:15.808401+0000) 2019-10-13T16:34:16.298 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:16.293+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.296166+0000) 2019-10-13T16:34:16.299 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:16.293+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.296642+0000) 2019-10-13T16:34:16.299 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:16.293+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.296962+0000) 2019-10-13T16:34:16.299 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:16.293+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.297700+0000) 2019-10-13T16:34:16.301 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:16.297+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.298839+0000) 2019-10-13T16:34:16.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:16.297+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.299362+0000) 2019-10-13T16:34:16.301 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:16.297+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.299906+0000) 2019-10-13T16:34:16.307 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:16.301+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.305706+0000) 2019-10-13T16:34:16.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:16.305+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247305 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:16.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:16.345+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:16.350837+0000) 2019-10-13T16:34:17.532 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:17.525+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:17.529528+0000) 2019-10-13T16:34:20.274 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:20.269+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:20.272538+0000) 2019-10-13T16:34:20.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:20.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:20.521615+0000) 2019-10-13T16:34:21.313 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:21.305+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.311241+0000) 2019-10-13T16:34:21.314 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:21.309+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.311539+0000) 2019-10-13T16:34:21.314 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:21.309+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.311796+0000) 2019-10-13T16:34:21.314 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:21.309+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.312309+0000) 2019-10-13T16:34:21.315 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:21.309+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.313419+0000) 2019-10-13T16:34:21.315 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:21.309+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.313600+0000) 2019-10-13T16:34:21.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:21.309+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.314222+0000) 2019-10-13T16:34:21.323 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:21.317+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:21.320897+0000) 2019-10-13T16:34:21.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:21.321+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247338 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:22.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:22.785+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:22.788800+0000) 2019-10-13T16:34:25.624 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:34:25.624 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:34:25.628 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:34:25.628 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:34:25.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:25.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:25.799847+0000) 2019-10-13T16:34:25.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:25.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:25.799968+0000) 2019-10-13T16:34:25.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:25.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:25.808597+0000) 2019-10-13T16:34:26.329 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:26.320+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.326727+0000) 2019-10-13T16:34:26.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:26.324+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.327299+0000) 2019-10-13T16:34:26.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:26.324+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.327928+0000) 2019-10-13T16:34:26.330 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:26.324+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.328509+0000) 2019-10-13T16:34:26.331 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:26.325+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.328753+0000) 2019-10-13T16:34:26.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:26.324+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.329749+0000) 2019-10-13T16:34:26.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:26.324+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.330072+0000) 2019-10-13T16:34:26.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:26.332+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247374 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:26.343 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:26.337+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.338972+0000) 2019-10-13T16:34:26.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:26.344+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:26.351030+0000) 2019-10-13T16:34:27.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:27.533+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:27.535310+0000) 2019-10-13T16:34:30.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:30.268+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:30.272741+0000) 2019-10-13T16:34:30.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:30.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:30.521826+0000) 2019-10-13T16:34:31.345 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:31.336+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.342597+0000) 2019-10-13T16:34:31.345 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:31.336+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.343026+0000) 2019-10-13T16:34:31.345 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:31.340+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.343491+0000) 2019-10-13T16:34:31.346 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:31.340+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.344107+0000) 2019-10-13T16:34:31.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:31.341+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.344379+0000) 2019-10-13T16:34:31.347 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:31.340+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.344998+0000) 2019-10-13T16:34:31.347 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:31.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.344290+0000) 2019-10-13T16:34:31.349 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:31.345+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:31.347525+0000) 2019-10-13T16:34:31.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:31.348+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247406 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:32.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:32.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:32.788998+0000) 2019-10-13T16:34:35.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:35.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:35.800030+0000) 2019-10-13T16:34:35.802 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:35.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:35.800168+0000) 2019-10-13T16:34:35.810 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:35.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:35.808774+0000) 2019-10-13T16:34:36.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:36.344+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.351215+0000) 2019-10-13T16:34:36.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:36.352+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.357778+0000) 2019-10-13T16:34:36.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:36.352+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.358426+0000) 2019-10-13T16:34:36.360 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:36.352+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.358905+0000) 2019-10-13T16:34:36.361 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:36.357+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.359456+0000) 2019-10-13T16:34:36.362 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:36.356+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.360609+0000) 2019-10-13T16:34:36.364 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:36.356+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.361222+0000) 2019-10-13T16:34:36.364 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:36.356+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.361429+0000) 2019-10-13T16:34:36.369 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:36.365+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:36.367379+0000) 2019-10-13T16:34:36.372 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:36.364+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:37.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:37.541+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:37.541666+0000) 2019-10-13T16:34:40.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:40.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:40.272971+0000) 2019-10-13T16:34:40.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:40.517+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:40.522031+0000) 2019-10-13T16:34:41.375 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:41.368+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.373427+0000) 2019-10-13T16:34:41.376 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:41.369+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.373720+0000) 2019-10-13T16:34:41.377 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:41.368+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.375006+0000) 2019-10-13T16:34:41.378 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:41.372+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.375613+0000) 2019-10-13T16:34:41.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:41.372+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.376331+0000) 2019-10-13T16:34:41.379 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:41.372+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.376778+0000) 2019-10-13T16:34:41.379 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:41.372+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.377631+0000) 2019-10-13T16:34:41.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:41.381+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:41.383068+0000) 2019-10-13T16:34:41.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:41.380+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247474 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:42.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:42.784+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:42.789152+0000) 2019-10-13T16:34:45.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:45.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:45.800231+0000) 2019-10-13T16:34:45.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:45.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:45.800352+0000) 2019-10-13T16:34:45.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:45.808+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:45.808941+0000) 2019-10-13T16:34:46.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:46.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.351411+0000) 2019-10-13T16:34:46.391 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:46.384+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.389392+0000) 2019-10-13T16:34:46.392 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:46.384+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.390726+0000) 2019-10-13T16:34:46.393 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:46.384+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.390936+0000) 2019-10-13T16:34:46.393 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:46.384+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.391149+0000) 2019-10-13T16:34:46.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:46.388+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.392065+0000) 2019-10-13T16:34:46.394 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:46.388+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.392658+0000) 2019-10-13T16:34:46.395 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:46.388+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.393171+0000) 2019-10-13T16:34:46.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:46.396+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:46.399382+0000) 2019-10-13T16:34:46.404 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:46.396+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247510 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:47.550 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:47.544+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:47.548152+0000) 2019-10-13T16:34:50.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:50.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:50.273167+0000) 2019-10-13T16:34:50.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:50.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:50.522249+0000) 2019-10-13T16:34:51.407 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:51.400+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.405343+0000) 2019-10-13T16:34:51.408 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:51.404+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.406897+0000) 2019-10-13T16:34:51.409 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:51.400+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.407192+0000) 2019-10-13T16:34:51.410 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:51.404+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.407852+0000) 2019-10-13T16:34:51.410 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:51.404+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.408054+0000) 2019-10-13T16:34:51.410 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:51.404+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.408639+0000) 2019-10-13T16:34:51.411 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:51.404+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.409150+0000) 2019-10-13T16:34:51.419 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:51.412+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:51.416809+0000) 2019-10-13T16:34:51.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:51.412+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247542 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:52.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:52.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:52.789345+0000) 2019-10-13T16:34:55.747 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:34:55.747 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:34:55.751 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:34:55.751 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:34:55.802 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:55.796+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:55.800427+0000) 2019-10-13T16:34:55.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:55.796+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:55.800553+0000) 2019-10-13T16:34:55.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:55.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:55.809170+0000) 2019-10-13T16:34:56.353 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:56.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.351611+0000) 2019-10-13T16:34:56.424 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:34:56.416+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.422700+0000) 2019-10-13T16:34:56.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:34:56.416+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.423154+0000) 2019-10-13T16:34:56.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:34:56.420+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.423376+0000) 2019-10-13T16:34:56.426 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:34:56.420+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.423949+0000) 2019-10-13T16:34:56.426 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:34:56.420+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.424662+0000) 2019-10-13T16:34:56.427 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:34:56.420+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.424967+0000) 2019-10-13T16:34:56.427 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:34:56.420+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.425373+0000) 2019-10-13T16:34:56.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:56.424+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:56.427894+0000) 2019-10-13T16:34:56.435 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:34:56.428+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247577 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:34:57.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:34:57.552+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:34:57.554640+0000) 2019-10-13T16:35:00.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:00.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:00.273327+0000) 2019-10-13T16:35:00.524 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:00.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:00.522445+0000) 2019-10-13T16:35:01.439 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:01.432+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.436290+0000) 2019-10-13T16:35:01.440 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:01.432+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.437907+0000) 2019-10-13T16:35:01.440 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:01.432+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.437582+0000) 2019-10-13T16:35:01.441 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:01.432+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.438321+0000) 2019-10-13T16:35:01.441 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:01.432+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.438638+0000) 2019-10-13T16:35:01.441 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:01.436+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.439513+0000) 2019-10-13T16:35:01.442 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:01.436+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.439967+0000) 2019-10-13T16:35:01.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:01.440+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:01.443791+0000) 2019-10-13T16:35:01.451 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:01.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247610 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:02.791 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:02.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:02.789561+0000) 2019-10-13T16:35:05.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:05.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:05.800670+0000) 2019-10-13T16:35:05.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:05.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:05.800754+0000) 2019-10-13T16:35:05.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:05.804+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:05.809366+0000) 2019-10-13T16:35:06.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:06.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.351838+0000) 2019-10-13T16:35:06.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:06.448+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.451855+0000) 2019-10-13T16:35:06.455 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:06.448+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.452790+0000) 2019-10-13T16:35:06.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:06.448+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.453543+0000) 2019-10-13T16:35:06.456 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:06.448+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.453832+0000) 2019-10-13T16:35:06.457 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:06.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.454750+0000) 2019-10-13T16:35:06.457 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:06.452+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.455350+0000) 2019-10-13T16:35:06.457 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:06.452+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.455728+0000) 2019-10-13T16:35:06.462 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:06.456+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:06.460336+0000) 2019-10-13T16:35:06.467 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:06.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247646 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:07.563 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:07.556+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:07.561091+0000) 2019-10-13T16:35:10.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:10.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:10.273527+0000) 2019-10-13T16:35:10.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:10.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:10.522689+0000) 2019-10-13T16:35:11.470 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:11.464+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.468536+0000) 2019-10-13T16:35:11.471 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:11.463+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.469618+0000) 2019-10-13T16:35:11.472 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:11.463+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.469893+0000) 2019-10-13T16:35:11.472 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:11.463+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.470649+0000) 2019-10-13T16:35:11.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:11.463+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.470456+0000) 2019-10-13T16:35:11.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:11.463+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.471188+0000) 2019-10-13T16:35:11.474 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:11.467+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.472423+0000) 2019-10-13T16:35:11.480 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:11.476+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:11.478385+0000) 2019-10-13T16:35:11.483 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:11.475+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247678 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:12.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:12.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:12.789757+0000) 2019-10-13T16:35:15.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:15.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:15.800872+0000) 2019-10-13T16:35:15.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:15.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:15.800979+0000) 2019-10-13T16:35:15.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:15.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:15.809577+0000) 2019-10-13T16:35:16.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:16.351+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.352038+0000) 2019-10-13T16:35:16.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:16.479+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.484971+0000) 2019-10-13T16:35:16.487 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:16.480+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.485259+0000) 2019-10-13T16:35:16.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:16.479+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.485806+0000) 2019-10-13T16:35:16.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:16.479+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.486266+0000) 2019-10-13T16:35:16.489 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:16.479+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.486810+0000) 2019-10-13T16:35:16.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:16.479+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.487066+0000) 2019-10-13T16:35:16.490 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:16.483+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.488419+0000) 2019-10-13T16:35:16.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:16.488+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:16.493920+0000) 2019-10-13T16:35:16.498 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:16.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247713 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:17.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:17.564+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:17.567422+0000) 2019-10-13T16:35:20.275 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:20.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:20.273720+0000) 2019-10-13T16:35:20.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:20.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:20.522908+0000) 2019-10-13T16:35:21.502 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:21.496+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.500096+0000) 2019-10-13T16:35:21.503 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:21.495+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.499805+0000) 2019-10-13T16:35:21.503 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:21.495+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.500588+0000) 2019-10-13T16:35:21.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:21.495+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.501675+0000) 2019-10-13T16:35:21.504 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:21.495+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.501980+0000) 2019-10-13T16:35:21.505 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:21.495+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.502833+0000) 2019-10-13T16:35:21.505 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:21.499+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.503286+0000) 2019-10-13T16:35:21.512 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:21.504+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:21.509943+0000) 2019-10-13T16:35:21.514 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:21.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247747 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:22.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:22.787+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:22.789963+0000) 2019-10-13T16:35:25.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:25.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:25.801100+0000) 2019-10-13T16:35:25.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:25.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:25.801180+0000) 2019-10-13T16:35:25.811 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:25.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:25.809779+0000) 2019-10-13T16:35:25.887 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:35:25.887 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:35:25.891 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:35:25.891 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:35:26.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:26.347+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.352195+0000) 2019-10-13T16:35:26.519 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:26.511+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.516776+0000) 2019-10-13T16:35:26.519 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:26.511+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.517076+0000) 2019-10-13T16:35:26.519 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:26.511+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.517657+0000) 2019-10-13T16:35:26.520 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:26.512+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.517943+0000) 2019-10-13T16:35:26.521 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:26.511+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.518405+0000) 2019-10-13T16:35:26.521 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:26.511+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.518855+0000) 2019-10-13T16:35:26.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:26.515+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.519499+0000) 2019-10-13T16:35:26.530 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:26.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247782 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:26.531 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:26.524+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:26.528848+0000) 2019-10-13T16:35:27.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:27.572+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:27.573735+0000) 2019-10-13T16:35:30.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:30.267+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:30.273907+0000) 2019-10-13T16:35:30.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:30.520+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:30.523134+0000) 2019-10-13T16:35:31.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:31.527+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.532003+0000) 2019-10-13T16:35:31.534 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:31.527+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.532571+0000) 2019-10-13T16:35:31.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:31.528+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.533128+0000) 2019-10-13T16:35:31.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:31.527+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.532846+0000) 2019-10-13T16:35:31.536 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:31.527+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.533650+0000) 2019-10-13T16:35:31.536 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:31.527+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.534296+0000) 2019-10-13T16:35:31.536 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:31.527+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.534795+0000) 2019-10-13T16:35:31.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:31.536+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:31.541559+0000) 2019-10-13T16:35:31.546 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:31.539+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247815 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:32.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:32.783+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:32.790151+0000) 2019-10-13T16:35:35.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:35.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:35.801322+0000) 2019-10-13T16:35:35.803 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:35.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:35.801420+0000) 2019-10-13T16:35:35.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:35.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:35.810003+0000) 2019-10-13T16:35:36.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:36.351+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.352404+0000) 2019-10-13T16:35:36.550 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:36.543+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.548135+0000) 2019-10-13T16:35:36.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:36.543+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.548705+0000) 2019-10-13T16:35:36.551 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:36.543+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.549004+0000) 2019-10-13T16:35:36.551 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:36.543+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.549542+0000) 2019-10-13T16:35:36.552 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:36.543+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.549646+0000) 2019-10-13T16:35:36.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:36.548+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.551149+0000) 2019-10-13T16:35:36.553 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:36.547+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.552019+0000) 2019-10-13T16:35:36.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:36.556+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:36.559412+0000) 2019-10-13T16:35:36.563 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:36.555+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247849 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:37.582 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:37.576+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:37.580249+0000) 2019-10-13T16:35:40.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:40.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:40.274108+0000) 2019-10-13T16:35:40.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:40.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:40.523352+0000) 2019-10-13T16:35:41.566 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:41.559+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.564019+0000) 2019-10-13T16:35:41.567 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:41.563+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.564679+0000) 2019-10-13T16:35:41.567 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:41.563+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.565096+0000) 2019-10-13T16:35:41.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:41.563+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.565443+0000) 2019-10-13T16:35:41.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:41.563+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.566732+0000) 2019-10-13T16:35:41.569 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:41.563+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.566996+0000) 2019-10-13T16:35:41.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:41.567+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.567677+0000) 2019-10-13T16:35:41.574 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:41.567+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:41.571912+0000) 2019-10-13T16:35:41.578 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:41.575+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247883 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:42.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:42.787+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:42.790354+0000) 2019-10-13T16:35:45.803 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:45.799+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:45.801487+0000) 2019-10-13T16:35:45.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:45.799+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:45.801621+0000) 2019-10-13T16:35:45.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:45.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:45.810200+0000) 2019-10-13T16:35:46.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:46.351+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.352638+0000) 2019-10-13T16:35:46.582 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:46.579+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.579694+0000) 2019-10-13T16:35:46.582 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:46.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.580019+0000) 2019-10-13T16:35:46.582 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:46.579+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.580872+0000) 2019-10-13T16:35:46.583 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:46.575+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.580952+0000) 2019-10-13T16:35:46.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:46.579+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.581724+0000) 2019-10-13T16:35:46.584 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:46.579+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.581495+0000) 2019-10-13T16:35:46.584 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:46.579+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.582063+0000) 2019-10-13T16:35:46.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:46.587+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:46.592170+0000) 2019-10-13T16:35:46.595 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:46.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247917 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:47.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:47.583+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:47.586759+0000) 2019-10-13T16:35:50.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:50.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:50.274338+0000) 2019-10-13T16:35:50.525 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:50.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:50.523560+0000) 2019-10-13T16:35:51.599 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:51.595+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.596406+0000) 2019-10-13T16:35:51.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:51.595+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.597088+0000) 2019-10-13T16:35:51.599 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:51.595+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.596581+0000) 2019-10-13T16:35:51.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:51.595+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.598870+0000) 2019-10-13T16:35:51.601 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:51.595+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.598475+0000) 2019-10-13T16:35:51.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:51.595+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.599487+0000) 2019-10-13T16:35:51.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:51.599+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.600176+0000) 2019-10-13T16:35:51.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:51.603+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:51.608911+0000) 2019-10-13T16:35:51.612 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:51.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:52.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:52.787+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:52.790506+0000) 2019-10-13T16:35:55.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:55.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:55.801670+0000) 2019-10-13T16:35:55.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:55.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:55.801779+0000) 2019-10-13T16:35:55.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:55.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:55.810412+0000) 2019-10-13T16:35:56.010 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:35:56.011 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:35:56.014 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:35:56.015 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:35:56.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:56.350+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.352872+0000) 2019-10-13T16:35:56.616 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:35:56.610+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.613837+0000) 2019-10-13T16:35:56.616 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:35:56.610+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.614493+0000) 2019-10-13T16:35:56.617 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:35:56.610+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.615266+0000) 2019-10-13T16:35:56.617 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:35:56.614+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.615569+0000) 2019-10-13T16:35:56.618 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:35:56.614+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.615973+0000) 2019-10-13T16:35:56.618 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:35:56.614+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.616543+0000) 2019-10-13T16:35:56.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:35:56.611+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.616657+0000) 2019-10-13T16:35:56.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:56.619+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:56.625214+0000) 2019-10-13T16:35:56.628 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:35:56.622+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 247984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:35:57.595 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:35:57.587+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:35:57.593077+0000) 2019-10-13T16:36:00.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:00.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:00.274539+0000) 2019-10-13T16:36:00.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:00.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:00.523777+0000) 2019-10-13T16:36:01.631 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:01.626+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.629470+0000) 2019-10-13T16:36:01.632 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:01.626+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.630207+0000) 2019-10-13T16:36:01.632 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:01.626+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.630510+0000) 2019-10-13T16:36:01.634 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:01.630+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.631739+0000) 2019-10-13T16:36:01.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:01.630+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.632165+0000) 2019-10-13T16:36:01.634 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:01.630+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.632630+0000) 2019-10-13T16:36:01.635 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:01.627+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.633392+0000) 2019-10-13T16:36:01.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:01.635+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:01.638182+0000) 2019-10-13T16:36:01.644 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:01.638+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248019 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:02.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:02.786+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:02.790711+0000) 2019-10-13T16:36:05.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:05.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:05.801871+0000) 2019-10-13T16:36:05.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:05.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:05.802015+0000) 2019-10-13T16:36:05.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:05.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:05.810645+0000) 2019-10-13T16:36:06.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:06.350+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.353084+0000) 2019-10-13T16:36:06.649 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:06.642+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.646992+0000) 2019-10-13T16:36:06.649 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:06.646+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.647351+0000) 2019-10-13T16:36:06.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:06.646+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.648102+0000) 2019-10-13T16:36:06.650 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:06.643+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.648049+0000) 2019-10-13T16:36:06.651 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:06.646+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.649028+0000) 2019-10-13T16:36:06.651 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:06.646+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.649072+0000) 2019-10-13T16:36:06.651 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:06.646+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.648717+0000) 2019-10-13T16:36:06.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:06.651+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:06.655886+0000) 2019-10-13T16:36:06.660 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:06.654+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248053 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:07.606 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:07.599+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:07.604594+0000) 2019-10-13T16:36:10.276 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:10.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:10.274731+0000) 2019-10-13T16:36:10.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:10.519+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:10.523988+0000) 2019-10-13T16:36:11.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:11.658+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.662047+0000) 2019-10-13T16:36:11.664 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:11.658+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.662482+0000) 2019-10-13T16:36:11.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:11.658+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.662764+0000) 2019-10-13T16:36:11.665 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:11.659+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.663222+0000) 2019-10-13T16:36:11.666 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:11.658+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.663220+0000) 2019-10-13T16:36:11.666 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:11.662+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.663758+0000) 2019-10-13T16:36:11.666 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:11.662+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.664620+0000) 2019-10-13T16:36:11.676 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:11.667+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:11.672015+0000) 2019-10-13T16:36:11.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:11.670+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248087 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:12.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:12.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:12.790938+0000) 2019-10-13T16:36:15.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:15.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:15.802091+0000) 2019-10-13T16:36:15.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:15.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:15.802211+0000) 2019-10-13T16:36:15.812 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:15.806+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:15.810842+0000) 2019-10-13T16:36:16.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:16.350+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.353279+0000) 2019-10-13T16:36:16.680 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:16.671+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.677744+0000) 2019-10-13T16:36:16.680 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:16.674+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.678858+0000) 2019-10-13T16:36:16.681 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:16.674+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.679172+0000) 2019-10-13T16:36:16.681 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:16.678+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.679904+0000) 2019-10-13T16:36:16.682 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:16.678+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.680146+0000) 2019-10-13T16:36:16.683 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:16.678+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.681149+0000) 2019-10-13T16:36:16.683 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:16.678+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.681667+0000) 2019-10-13T16:36:16.688 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:16.679+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:16.685949+0000) 2019-10-13T16:36:16.692 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:16.686+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248120 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:17.613 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:17.607+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:17.611055+0000) 2019-10-13T16:36:20.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:20.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:20.274936+0000) 2019-10-13T16:36:20.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:20.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:20.524204+0000) 2019-10-13T16:36:21.696 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:21.690+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.693592+0000) 2019-10-13T16:36:21.696 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:21.691+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.694379+0000) 2019-10-13T16:36:21.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:21.690+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.695241+0000) 2019-10-13T16:36:21.697 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:21.694+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.695608+0000) 2019-10-13T16:36:21.698 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:21.694+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.695937+0000) 2019-10-13T16:36:21.698 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:21.694+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.696207+0000) 2019-10-13T16:36:21.699 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:21.694+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.697604+0000) 2019-10-13T16:36:21.708 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:21.703+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:21.706258+0000) 2019-10-13T16:36:21.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:21.702+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248155 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:22.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:22.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:22.791176+0000) 2019-10-13T16:36:25.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:25.798+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:25.802287+0000) 2019-10-13T16:36:25.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:25.798+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:25.802406+0000) 2019-10-13T16:36:25.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:25.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:25.811035+0000) 2019-10-13T16:36:26.144 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:36:26.144 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:36:26.148 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:36:26.148 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:36:26.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:26.350+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.353500+0000) 2019-10-13T16:36:26.711 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:26.706+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.709332+0000) 2019-10-13T16:36:26.712 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:26.703+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.709884+0000) 2019-10-13T16:36:26.712 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:26.706+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.710349+0000) 2019-10-13T16:36:26.713 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:26.710+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.711613+0000) 2019-10-13T16:36:26.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:26.710+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.712331+0000) 2019-10-13T16:36:26.714 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:26.710+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.712568+0000) 2019-10-13T16:36:26.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:26.710+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.713182+0000) 2019-10-13T16:36:26.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:26.715+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:26.719877+0000) 2019-10-13T16:36:26.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:26.718+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248189 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:27.619 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:27.611+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:27.617201+0000) 2019-10-13T16:36:30.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:30.270+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:30.275094+0000) 2019-10-13T16:36:30.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:30.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:30.524438+0000) 2019-10-13T16:36:31.727 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:31.719+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.725644+0000) 2019-10-13T16:36:31.728 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:31.722+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.726753+0000) 2019-10-13T16:36:31.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:31.726+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.727362+0000) 2019-10-13T16:36:31.730 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:31.726+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.728069+0000) 2019-10-13T16:36:31.730 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:31.726+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.728398+0000) 2019-10-13T16:36:31.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:31.726+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.728671+0000) 2019-10-13T16:36:31.731 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:31.726+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.729296+0000) 2019-10-13T16:36:31.737 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:31.731+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:31.734895+0000) 2019-10-13T16:36:31.740 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:31.734+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248223 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:32.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:32.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:32.791377+0000) 2019-10-13T16:36:35.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:35.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:35.802517+0000) 2019-10-13T16:36:35.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:35.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:35.802613+0000) 2019-10-13T16:36:35.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:35.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:35.811221+0000) 2019-10-13T16:36:36.355 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:36.350+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.353723+0000) 2019-10-13T16:36:36.743 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:36.734+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.741623+0000) 2019-10-13T16:36:36.744 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:36.738+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.742101+0000) 2019-10-13T16:36:36.745 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:36.738+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.742682+0000) 2019-10-13T16:36:36.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:36.742+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.743641+0000) 2019-10-13T16:36:36.746 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:36.742+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.743970+0000) 2019-10-13T16:36:36.747 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:36.742+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.744634+0000) 2019-10-13T16:36:36.747 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:36.742+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.745179+0000) 2019-10-13T16:36:36.754 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:36.746+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:36.751795+0000) 2019-10-13T16:36:36.755 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:36.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:37.628 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:37.618+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:37.623743+0000) 2019-10-13T16:36:40.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:40.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:40.275328+0000) 2019-10-13T16:36:40.526 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:40.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:40.524632+0000) 2019-10-13T16:36:41.758 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:41.750+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.756445+0000) 2019-10-13T16:36:41.759 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:41.753+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.757194+0000) 2019-10-13T16:36:41.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:41.753+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.757514+0000) 2019-10-13T16:36:41.760 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:41.753+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.758087+0000) 2019-10-13T16:36:41.760 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:41.753+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.758339+0000) 2019-10-13T16:36:41.761 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:41.753+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.758682+0000) 2019-10-13T16:36:41.762 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:41.757+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.760712+0000) 2019-10-13T16:36:41.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:41.758+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:41.765470+0000) 2019-10-13T16:36:41.771 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:41.765+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:42.793 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:42.789+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:42.791612+0000) 2019-10-13T16:36:45.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:45.797+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:45.802748+0000) 2019-10-13T16:36:45.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:45.797+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:45.802819+0000) 2019-10-13T16:36:45.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:45.809+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:45.811415+0000) 2019-10-13T16:36:46.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:46.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.353921+0000) 2019-10-13T16:36:46.774 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:46.769+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.771969+0000) 2019-10-13T16:36:46.774 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:46.766+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.772585+0000) 2019-10-13T16:36:46.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:46.769+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.773135+0000) 2019-10-13T16:36:46.776 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:46.769+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.774028+0000) 2019-10-13T16:36:46.776 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:46.769+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.774443+0000) 2019-10-13T16:36:46.777 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:46.769+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.774993+0000) 2019-10-13T16:36:46.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:46.773+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.775832+0000) 2019-10-13T16:36:46.781 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:46.774+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:46.779717+0000) 2019-10-13T16:36:46.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:46.781+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248326 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:47.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:47.626+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:47.630299+0000) 2019-10-13T16:36:50.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:50.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:50.275492+0000) 2019-10-13T16:36:50.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:50.518+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:50.524856+0000) 2019-10-13T16:36:51.789 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:51.782+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.787344+0000) 2019-10-13T16:36:51.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:51.785+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.788247+0000) 2019-10-13T16:36:51.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:51.785+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.789226+0000) 2019-10-13T16:36:51.792 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:51.785+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.788552+0000) 2019-10-13T16:36:51.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:51.785+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.789490+0000) 2019-10-13T16:36:51.792 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:51.785+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.789751+0000) 2019-10-13T16:36:51.793 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:51.785+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.790800+0000) 2019-10-13T16:36:51.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:51.786+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:51.793917+0000) 2019-10-13T16:36:51.802 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:51.797+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:52.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:52.789+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:52.791813+0000) 2019-10-13T16:36:55.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:55.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:55.802953+0000) 2019-10-13T16:36:55.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:55.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:55.803037+0000) 2019-10-13T16:36:55.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:55.809+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:55.811610+0000) 2019-10-13T16:36:56.274 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:36:56.274 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:36:56.278 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:36:56.278 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:36:56.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:56.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.354125+0000) 2019-10-13T16:36:56.805 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:36:56.797+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.802833+0000) 2019-10-13T16:36:56.805 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:36:56.801+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.803567+0000) 2019-10-13T16:36:56.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:36:56.801+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.804128+0000) 2019-10-13T16:36:56.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:36:56.798+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.804345+0000) 2019-10-13T16:36:56.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:36:56.801+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.804913+0000) 2019-10-13T16:36:56.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:36:56.801+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.805729+0000) 2019-10-13T16:36:56.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:36:56.801+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.806101+0000) 2019-10-13T16:36:56.815 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:56.806+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:56.813027+0000) 2019-10-13T16:36:56.817 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:36:56.809+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248394 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:36:57.639 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:36:57.634+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:36:57.636826+0000) 2019-10-13T16:37:00.277 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:00.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:00.275694+0000) 2019-10-13T16:37:00.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:00.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:00.525101+0000) 2019-10-13T16:37:01.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:01.813+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.817970+0000) 2019-10-13T16:37:01.820 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:01.814+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.818747+0000) 2019-10-13T16:37:01.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:01.813+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.819233+0000) 2019-10-13T16:37:01.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:01.817+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.819711+0000) 2019-10-13T16:37:01.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:01.817+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.819864+0000) 2019-10-13T16:37:01.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:01.817+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.820611+0000) 2019-10-13T16:37:01.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:01.817+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.822114+0000) 2019-10-13T16:37:01.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:01.822+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:01.826354+0000) 2019-10-13T16:37:01.833 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:01.825+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248430 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:02.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:02.789+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:02.792065+0000) 2019-10-13T16:37:05.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:05.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:05.803185+0000) 2019-10-13T16:37:05.805 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:05.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:05.803231+0000) 2019-10-13T16:37:05.813 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:05.809+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:05.811806+0000) 2019-10-13T16:37:06.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:06.349+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.354340+0000) 2019-10-13T16:37:06.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:06.829+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.834539+0000) 2019-10-13T16:37:06.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:06.830+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.835244+0000) 2019-10-13T16:37:06.837 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:06.829+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.835047+0000) 2019-10-13T16:37:06.838 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:06.833+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.836009+0000) 2019-10-13T16:37:06.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:06.833+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.836251+0000) 2019-10-13T16:37:06.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:06.833+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.836603+0000) 2019-10-13T16:37:06.840 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:06.833+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.837988+0000) 2019-10-13T16:37:06.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:06.838+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:06.842773+0000) 2019-10-13T16:37:06.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:06.841+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248464 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:07.645 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:07.638+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:07.643400+0000) 2019-10-13T16:37:10.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:10.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:10.275923+0000) 2019-10-13T16:37:10.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:10.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:10.525293+0000) 2019-10-13T16:37:11.852 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:11.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.850481+0000) 2019-10-13T16:37:11.853 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:11.845+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.850817+0000) 2019-10-13T16:37:11.854 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:11.849+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.851905+0000) 2019-10-13T16:37:11.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:11.849+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.852158+0000) 2019-10-13T16:37:11.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:11.849+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.852436+0000) 2019-10-13T16:37:11.856 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:11.849+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.853555+0000) 2019-10-13T16:37:11.856 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:11.849+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.853900+0000) 2019-10-13T16:37:11.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:11.857+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248499 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:11.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:11.858+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:11.862579+0000) 2019-10-13T16:37:12.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:12.789+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:12.792241+0000) 2019-10-13T16:37:15.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:15.801+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:15.803383+0000) 2019-10-13T16:37:15.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:15.801+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:15.803470+0000) 2019-10-13T16:37:15.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:15.809+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:15.812020+0000) 2019-10-13T16:37:16.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:16.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.354540+0000) 2019-10-13T16:37:16.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:16.861+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.865553+0000) 2019-10-13T16:37:16.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:16.862+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.866206+0000) 2019-10-13T16:37:16.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:16.861+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.866720+0000) 2019-10-13T16:37:16.869 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:16.865+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.867370+0000) 2019-10-13T16:37:16.869 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:16.865+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.867848+0000) 2019-10-13T16:37:16.870 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:16.865+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.868610+0000) 2019-10-13T16:37:16.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:16.865+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.869135+0000) 2019-10-13T16:37:16.878 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:16.870+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:16.875979+0000) 2019-10-13T16:37:16.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:16.873+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248532 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:17.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:17.646+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:17.649742+0000) 2019-10-13T16:37:20.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:20.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:20.276119+0000) 2019-10-13T16:37:20.527 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:20.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:20.525512+0000) 2019-10-13T16:37:21.884 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:21.876+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.882197+0000) 2019-10-13T16:37:21.885 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:21.876+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.882858+0000) 2019-10-13T16:37:21.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:21.880+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.883395+0000) 2019-10-13T16:37:21.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:21.878+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.884061+0000) 2019-10-13T16:37:21.886 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:21.880+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.884328+0000) 2019-10-13T16:37:21.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:21.880+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.885448+0000) 2019-10-13T16:37:21.887 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:21.880+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.885929+0000) 2019-10-13T16:37:21.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:21.882+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:21.889937+0000) 2019-10-13T16:37:21.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:21.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248568 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:22.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:22.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:22.792393+0000) 2019-10-13T16:37:25.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:25.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:25.803587+0000) 2019-10-13T16:37:25.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:25.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:25.803701+0000) 2019-10-13T16:37:25.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:25.808+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:25.812231+0000) 2019-10-13T16:37:26.356 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:26.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.354696+0000) 2019-10-13T16:37:26.556 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:37:26.556 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:37:26.561 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:37:26.561 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:37:26.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:26.892+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.897654+0000) 2019-10-13T16:37:26.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:26.894+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.897888+0000) 2019-10-13T16:37:26.901 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:26.892+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.898546+0000) 2019-10-13T16:37:26.901 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:26.892+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.899248+0000) 2019-10-13T16:37:26.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:26.896+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.899577+0000) 2019-10-13T16:37:26.902 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:26.896+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.900047+0000) 2019-10-13T16:37:26.903 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:26.896+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.900975+0000) 2019-10-13T16:37:26.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:26.906+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:26.907250+0000) 2019-10-13T16:37:26.911 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:26.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248602 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:27.658 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:27.654+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:27.655951+0000) 2019-10-13T16:37:30.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:30.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:30.276267+0000) 2019-10-13T16:37:30.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:30.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:30.525698+0000) 2019-10-13T16:37:31.914 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:31.908+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.912495+0000) 2019-10-13T16:37:31.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:31.908+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.912996+0000) 2019-10-13T16:37:31.915 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:31.908+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.913788+0000) 2019-10-13T16:37:31.916 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:31.913+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.914391+0000) 2019-10-13T16:37:31.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:31.908+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.914881+0000) 2019-10-13T16:37:31.917 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:31.908+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.915280+0000) 2019-10-13T16:37:31.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:31.912+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.915704+0000) 2019-10-13T16:37:31.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:31.917+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:31.921233+0000) 2019-10-13T16:37:31.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:31.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248637 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:32.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:32.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:32.792567+0000) 2019-10-13T16:37:35.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:35.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:35.803820+0000) 2019-10-13T16:37:35.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:35.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:35.803895+0000) 2019-10-13T16:37:35.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:35.808+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:35.812431+0000) 2019-10-13T16:37:36.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:36.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.354894+0000) 2019-10-13T16:37:36.932 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:36.929+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.930419+0000) 2019-10-13T16:37:36.933 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:36.924+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.930736+0000) 2019-10-13T16:37:36.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:36.928+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.931452+0000) 2019-10-13T16:37:36.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:36.928+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.931772+0000) 2019-10-13T16:37:36.934 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:36.928+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.931490+0000) 2019-10-13T16:37:36.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:36.928+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.932784+0000) 2019-10-13T16:37:36.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:36.928+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.932475+0000) 2019-10-13T16:37:36.936 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:36.933+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:36.934283+0000) 2019-10-13T16:37:36.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:36.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:37.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:37.661+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:37.662291+0000) 2019-10-13T16:37:40.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:40.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:40.276483+0000) 2019-10-13T16:37:40.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:40.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:40.525921+0000) 2019-10-13T16:37:41.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:41.940+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.946306+0000) 2019-10-13T16:37:41.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:41.940+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.946386+0000) 2019-10-13T16:37:41.949 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:41.944+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.947757+0000) 2019-10-13T16:37:41.950 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:41.944+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.948124+0000) 2019-10-13T16:37:41.950 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:41.945+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.948349+0000) 2019-10-13T16:37:41.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:41.944+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.948733+0000) 2019-10-13T16:37:41.951 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:41.944+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.949356+0000) 2019-10-13T16:37:41.957 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:41.953+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:41.955042+0000) 2019-10-13T16:37:41.960 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:41.952+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248707 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:42.794 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:42.792+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:42.792744+0000) 2019-10-13T16:37:45.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:45.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:45.804017+0000) 2019-10-13T16:37:45.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:45.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:45.804108+0000) 2019-10-13T16:37:45.814 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:45.812+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:45.812626+0000) 2019-10-13T16:37:46.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:46.348+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.355092+0000) 2019-10-13T16:37:46.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:46.956+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.961490+0000) 2019-10-13T16:37:46.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:46.957+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.962044+0000) 2019-10-13T16:37:46.964 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:46.956+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.962496+0000) 2019-10-13T16:37:46.965 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:46.956+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.963127+0000) 2019-10-13T16:37:46.965 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:46.960+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.963463+0000) 2019-10-13T16:37:46.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:46.960+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.964178+0000) 2019-10-13T16:37:46.967 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:46.960+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.965294+0000) 2019-10-13T16:37:46.974 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:46.969+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:46.972112+0000) 2019-10-13T16:37:46.976 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:46.968+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248740 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:47.670 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:47.665+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:47.668592+0000) 2019-10-13T16:37:50.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:50.272+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:50.276678+0000) 2019-10-13T16:37:50.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:50.521+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:50.526115+0000) 2019-10-13T16:37:51.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:51.972+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.978100+0000) 2019-10-13T16:37:51.981 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:51.972+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.978412+0000) 2019-10-13T16:37:51.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:51.977+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.979474+0000) 2019-10-13T16:37:51.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:51.972+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.978936+0000) 2019-10-13T16:37:51.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:51.972+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.979155+0000) 2019-10-13T16:37:51.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:51.976+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.980349+0000) 2019-10-13T16:37:51.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:51.976+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.980795+0000) 2019-10-13T16:37:51.990 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:51.985+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:51.987989+0000) 2019-10-13T16:37:51.992 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:51.984+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:52.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:52.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:52.792948+0000) 2019-10-13T16:37:55.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:55.800+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:55.804215+0000) 2019-10-13T16:37:55.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:55.800+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:55.804355+0000) 2019-10-13T16:37:55.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:55.808+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:55.812855+0000) 2019-10-13T16:37:56.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:56.352+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.355345+0000) 2019-10-13T16:37:56.630 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:37:56.631 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:37:56.634 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:37:56.634 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:37:56.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:37:56.989+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.993719+0000) 2019-10-13T16:37:56.996 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:37:56.988+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.994534+0000) 2019-10-13T16:37:56.997 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:37:56.988+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.994929+0000) 2019-10-13T16:37:56.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:37:56.992+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.995341+0000) 2019-10-13T16:37:56.998 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:37:56.992+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.995663+0000) 2019-10-13T16:37:56.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:37:56.992+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.996118+0000) 2019-10-13T16:37:56.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:37:56.992+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:56.997594+0000) 2019-10-13T16:37:57.004 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:57.001+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:57.002750+0000) 2019-10-13T16:37:57.008 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:37:57.000+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248806 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:37:57.677 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:37:57.673+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:37:57.675136+0000) 2019-10-13T16:38:00.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:00.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:00.276879+0000) 2019-10-13T16:38:00.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:00.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:00.526371+0000) 2019-10-13T16:38:02.012 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:02.004+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.010366+0000) 2019-10-13T16:38:02.013 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:02.004+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.010756+0000) 2019-10-13T16:38:02.013 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:02.008+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.011434+0000) 2019-10-13T16:38:02.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:02.008+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.011391+0000) 2019-10-13T16:38:02.014 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:02.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.012183+0000) 2019-10-13T16:38:02.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:02.008+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.012814+0000) 2019-10-13T16:38:02.015 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:02.008+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.013380+0000) 2019-10-13T16:38:02.021 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:02.017+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.018992+0000) 2019-10-13T16:38:02.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:02.016+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248842 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:02.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:02.788+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:02.793158+0000) 2019-10-13T16:38:05.806 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:05.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:05.804410+0000) 2019-10-13T16:38:05.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:05.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:05.804548+0000) 2019-10-13T16:38:05.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:05.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:05.813071+0000) 2019-10-13T16:38:06.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:06.351+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:06.355551+0000) 2019-10-13T16:38:07.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:07.025+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.027302+0000) 2019-10-13T16:38:07.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:07.023+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.027713+0000) 2019-10-13T16:38:07.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:07.023+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.028253+0000) 2019-10-13T16:38:07.030 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:07.023+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.028338+0000) 2019-10-13T16:38:07.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:07.023+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.028961+0000) 2019-10-13T16:38:07.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:07.023+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.029311+0000) 2019-10-13T16:38:07.032 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:07.023+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.029599+0000) 2019-10-13T16:38:07.035 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:07.029+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.033145+0000) 2019-10-13T16:38:07.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:07.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:07.683 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:07.677+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:07.681600+0000) 2019-10-13T16:38:10.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:10.271+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:10.277076+0000) 2019-10-13T16:38:10.528 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:10.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:10.526571+0000) 2019-10-13T16:38:12.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:12.035+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.042145+0000) 2019-10-13T16:38:12.045 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:12.035+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.042480+0000) 2019-10-13T16:38:12.045 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:12.039+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.043364+0000) 2019-10-13T16:38:12.046 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:12.039+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.044485+0000) 2019-10-13T16:38:12.047 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:12.041+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.044634+0000) 2019-10-13T16:38:12.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:12.039+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.045218+0000) 2019-10-13T16:38:12.048 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:12.039+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.045664+0000) 2019-10-13T16:38:12.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:12.045+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.046944+0000) 2019-10-13T16:38:12.056 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:12.047+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248908 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:12.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:12.791+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:12.793334+0000) 2019-10-13T16:38:15.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:15.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:15.804640+0000) 2019-10-13T16:38:15.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:15.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:15.804763+0000) 2019-10-13T16:38:15.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:15.807+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:15.813268+0000) 2019-10-13T16:38:16.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:16.351+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:16.355773+0000) 2019-10-13T16:38:17.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:17.051+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.057027+0000) 2019-10-13T16:38:17.060 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:17.051+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.057925+0000) 2019-10-13T16:38:17.060 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:17.057+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.058356+0000) 2019-10-13T16:38:17.061 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:17.051+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.059011+0000) 2019-10-13T16:38:17.061 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:17.055+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.059753+0000) 2019-10-13T16:38:17.062 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:17.055+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.060499+0000) 2019-10-13T16:38:17.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:17.055+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.061032+0000) 2019-10-13T16:38:17.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:17.061+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.064899+0000) 2019-10-13T16:38:17.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:17.063+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:17.690 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:17.685+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:17.688071+0000) 2019-10-13T16:38:20.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:20.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:20.277275+0000) 2019-10-13T16:38:20.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:20.525+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:20.526761+0000) 2019-10-13T16:38:22.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:22.067+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.072836+0000) 2019-10-13T16:38:22.076 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:22.067+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.074141+0000) 2019-10-13T16:38:22.076 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:22.067+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.074497+0000) 2019-10-13T16:38:22.077 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:22.067+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.075035+0000) 2019-10-13T16:38:22.077 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:22.073+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.075170+0000) 2019-10-13T16:38:22.077 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:22.071+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.075964+0000) 2019-10-13T16:38:22.078 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:22.071+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.076827+0000) 2019-10-13T16:38:22.084 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:22.081+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.082524+0000) 2019-10-13T16:38:22.089 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:22.079+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 248977 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:22.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:22.791+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:22.793546+0000) 2019-10-13T16:38:25.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:25.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:25.804882+0000) 2019-10-13T16:38:25.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:25.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:25.804987+0000) 2019-10-13T16:38:25.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:25.811+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:25.813489+0000) 2019-10-13T16:38:26.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:26.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:26.355969+0000) 2019-10-13T16:38:26.840 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:38:26.840 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:38:26.844 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:38:26.844 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:38:27.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:27.083+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.090683+0000) 2019-10-13T16:38:27.093 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:27.083+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.091206+0000) 2019-10-13T16:38:27.093 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:27.087+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.091907+0000) 2019-10-13T16:38:27.094 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:27.088+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.092289+0000) 2019-10-13T16:38:27.095 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:27.087+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.092903+0000) 2019-10-13T16:38:27.095 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:27.087+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.093226+0000) 2019-10-13T16:38:27.095 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:27.087+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.093444+0000) 2019-10-13T16:38:27.100 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:27.092+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.097787+0000) 2019-10-13T16:38:27.105 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:27.095+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:27.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:27.692+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:27.694397+0000) 2019-10-13T16:38:30.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:30.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:30.277470+0000) 2019-10-13T16:38:30.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:30.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:30.526969+0000) 2019-10-13T16:38:32.108 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:32.099+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.106162+0000) 2019-10-13T16:38:32.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:32.099+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.106888+0000) 2019-10-13T16:38:32.109 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:32.103+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.107635+0000) 2019-10-13T16:38:32.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:32.103+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.108175+0000) 2019-10-13T16:38:32.110 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:32.104+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.108767+0000) 2019-10-13T16:38:32.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:32.103+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.109693+0000) 2019-10-13T16:38:32.112 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:32.103+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.110520+0000) 2019-10-13T16:38:32.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:32.112+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.115516+0000) 2019-10-13T16:38:32.121 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:32.115+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249044 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:32.795 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:32.791+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:32.793723+0000) 2019-10-13T16:38:35.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:35.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:35.805084+0000) 2019-10-13T16:38:35.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:35.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:35.805222+0000) 2019-10-13T16:38:35.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:35.811+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:35.813668+0000) 2019-10-13T16:38:36.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:36.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:36.356164+0000) 2019-10-13T16:38:37.124 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:37.119+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.122591+0000) 2019-10-13T16:38:37.125 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:37.119+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.122910+0000) 2019-10-13T16:38:37.125 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:37.123+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.123589+0000) 2019-10-13T16:38:37.126 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:37.123+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.124525+0000) 2019-10-13T16:38:37.127 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:37.123+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.124996+0000) 2019-10-13T16:38:37.127 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:37.120+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.125358+0000) 2019-10-13T16:38:37.128 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:37.123+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.126358+0000) 2019-10-13T16:38:37.129 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:37.124+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.127754+0000) 2019-10-13T16:38:37.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:37.131+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249078 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:37.703 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:37.696+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:37.700673+0000) 2019-10-13T16:38:40.279 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:40.275+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:40.277671+0000) 2019-10-13T16:38:40.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:40.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:40.527181+0000) 2019-10-13T16:38:42.140 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:42.136+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.138433+0000) 2019-10-13T16:38:42.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:42.139+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.139365+0000) 2019-10-13T16:38:42.141 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:42.139+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.139441+0000) 2019-10-13T16:38:42.142 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:42.139+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.139780+0000) 2019-10-13T16:38:42.142 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:42.139+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.140114+0000) 2019-10-13T16:38:42.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:42.139+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.140544+0000) 2019-10-13T16:38:42.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:42.139+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.140810+0000) 2019-10-13T16:38:42.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:42.136+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.140882+0000) 2019-10-13T16:38:42.152 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:42.147+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249113 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:42.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:42.791+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:42.793895+0000) 2019-10-13T16:38:45.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:45.803+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:45.805285+0000) 2019-10-13T16:38:45.807 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:45.803+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:45.805456+0000) 2019-10-13T16:38:45.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:45.811+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:45.813889+0000) 2019-10-13T16:38:46.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:46.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:46.356363+0000) 2019-10-13T16:38:47.156 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:47.151+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.154381+0000) 2019-10-13T16:38:47.157 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:47.151+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.154602+0000) 2019-10-13T16:38:47.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:47.151+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.155221+0000) 2019-10-13T16:38:47.158 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:47.155+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.155872+0000) 2019-10-13T16:38:47.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:47.155+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.155969+0000) 2019-10-13T16:38:47.159 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:47.152+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.157215+0000) 2019-10-13T16:38:47.159 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:47.155+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.157831+0000) 2019-10-13T16:38:47.162 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:47.156+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.160096+0000) 2019-10-13T16:38:47.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:47.167+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249147 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:47.709 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:47.704+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:47.707649+0000) 2019-10-13T16:38:50.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:50.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:50.277863+0000) 2019-10-13T16:38:50.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:50.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:50.527402+0000) 2019-10-13T16:38:52.172 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:52.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.170405+0000) 2019-10-13T16:38:52.173 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:52.166+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.170733+0000) 2019-10-13T16:38:52.173 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:52.170+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.171982+0000) 2019-10-13T16:38:52.174 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:52.168+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.172273+0000) 2019-10-13T16:38:52.174 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:52.170+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.172792+0000) 2019-10-13T16:38:52.175 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:52.170+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.173574+0000) 2019-10-13T16:38:52.176 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:52.170+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.174265+0000) 2019-10-13T16:38:52.181 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:52.176+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.179466+0000) 2019-10-13T16:38:52.184 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:52.178+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249180 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:52.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:52.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:52.794075+0000) 2019-10-13T16:38:55.807 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:55.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:55.805479+0000) 2019-10-13T16:38:55.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:55.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:55.805639+0000) 2019-10-13T16:38:55.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:55.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:55.814084+0000) 2019-10-13T16:38:56.358 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:56.354+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:56.356576+0000) 2019-10-13T16:38:56.959 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:38:56.960 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:38:56.963 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:38:56.964 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:38:57.187 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:38:57.182+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.185174+0000) 2019-10-13T16:38:57.188 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:38:57.182+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.185861+0000) 2019-10-13T16:38:57.188 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:38:57.182+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.186661+0000) 2019-10-13T16:38:57.189 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:38:57.182+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.186996+0000) 2019-10-13T16:38:57.189 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:38:57.186+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.187656+0000) 2019-10-13T16:38:57.190 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:38:57.186+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.188291+0000) 2019-10-13T16:38:57.190 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:38:57.184+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.188654+0000) 2019-10-13T16:38:57.198 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:57.192+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.196034+0000) 2019-10-13T16:38:57.199 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:38:57.194+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249214 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:38:57.716 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:38:57.712+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:38:57.714232+0000) 2019-10-13T16:39:00.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:00.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:00.278090+0000) 2019-10-13T16:39:00.529 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:00.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:00.527594+0000) 2019-10-13T16:39:02.203 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:02.198+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.200992+0000) 2019-10-13T16:39:02.203 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:02.198+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.201609+0000) 2019-10-13T16:39:02.204 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:02.198+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.202092+0000) 2019-10-13T16:39:02.204 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:02.200+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.202772+0000) 2019-10-13T16:39:02.205 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:02.202+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.203842+0000) 2019-10-13T16:39:02.206 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:02.202+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.204457+0000) 2019-10-13T16:39:02.206 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:02.202+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.204716+0000) 2019-10-13T16:39:02.211 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:02.204+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.208965+0000) 2019-10-13T16:39:02.214 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:02.210+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:02.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:02.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:02.794244+0000) 2019-10-13T16:39:05.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:05.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:05.805692+0000) 2019-10-13T16:39:05.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:05.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:05.805828+0000) 2019-10-13T16:39:05.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:05.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:05.814301+0000) 2019-10-13T16:39:06.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:06.354+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:06.356772+0000) 2019-10-13T16:39:07.218 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:07.214+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.215787+0000) 2019-10-13T16:39:07.218 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:07.214+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.216217+0000) 2019-10-13T16:39:07.218 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:07.214+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.216489+0000) 2019-10-13T16:39:07.219 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:07.212+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.217286+0000) 2019-10-13T16:39:07.220 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:07.214+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.218621+0000) 2019-10-13T16:39:07.221 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:07.214+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.218350+0000) 2019-10-13T16:39:07.221 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:07.214+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.219235+0000) 2019-10-13T16:39:07.226 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:07.220+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.224525+0000) 2019-10-13T16:39:07.230 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:07.226+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249282 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:07.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:07.716+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:07.720254+0000) 2019-10-13T16:39:10.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:10.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:10.278300+0000) 2019-10-13T16:39:10.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:10.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:10.527810+0000) 2019-10-13T16:39:12.234 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:12.230+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.231749+0000) 2019-10-13T16:39:12.234 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:12.228+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.231951+0000) 2019-10-13T16:39:12.235 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:12.230+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.232384+0000) 2019-10-13T16:39:12.235 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:12.230+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.233082+0000) 2019-10-13T16:39:12.236 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:12.230+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.233728+0000) 2019-10-13T16:39:12.236 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:12.230+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.233997+0000) 2019-10-13T16:39:12.236 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:12.230+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.234258+0000) 2019-10-13T16:39:12.243 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:12.236+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.240968+0000) 2019-10-13T16:39:12.246 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:12.242+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:12.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:12.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:12.794444+0000) 2019-10-13T16:39:15.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:15.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:15.805881+0000) 2019-10-13T16:39:15.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:15.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:15.805992+0000) 2019-10-13T16:39:15.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:15.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:15.814514+0000) 2019-10-13T16:39:16.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:16.354+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:16.357001+0000) 2019-10-13T16:39:17.250 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:17.246+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.247789+0000) 2019-10-13T16:39:17.250 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:17.246+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.248007+0000) 2019-10-13T16:39:17.251 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:17.246+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.248446+0000) 2019-10-13T16:39:17.251 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:17.246+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.249007+0000) 2019-10-13T16:39:17.251 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:17.246+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.249309+0000) 2019-10-13T16:39:17.252 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:17.244+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.249435+0000) 2019-10-13T16:39:17.252 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:17.246+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.249855+0000) 2019-10-13T16:39:17.254 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:17.248+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.252543+0000) 2019-10-13T16:39:17.261 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:17.254+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249348 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:17.729 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:17.724+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:17.726700+0000) 2019-10-13T16:39:20.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:20.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:20.278498+0000) 2019-10-13T16:39:20.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:20.524+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:20.528017+0000) 2019-10-13T16:39:22.264 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:22.259+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.262498+0000) 2019-10-13T16:39:22.265 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:22.262+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.263363+0000) 2019-10-13T16:39:22.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:22.262+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.263968+0000) 2019-10-13T16:39:22.266 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:22.262+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.264356+0000) 2019-10-13T16:39:22.267 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:22.262+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.265377+0000) 2019-10-13T16:39:22.267 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:22.262+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.265755+0000) 2019-10-13T16:39:22.268 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:22.262+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.266894+0000) 2019-10-13T16:39:22.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:22.263+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.269708+0000) 2019-10-13T16:39:22.276 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:22.270+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249382 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:22.796 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:22.790+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:22.794620+0000) 2019-10-13T16:39:25.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:25.802+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:25.806100+0000) 2019-10-13T16:39:25.808 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:25.802+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:25.806219+0000) 2019-10-13T16:39:25.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:25.810+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:25.814741+0000) 2019-10-13T16:39:26.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:26.354+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:26.357198+0000) 2019-10-13T16:39:27.101 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:39:27.101 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:39:27.105 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:39:27.105 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:39:27.280 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:27.274+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.277915+0000) 2019-10-13T16:39:27.280 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:27.274+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.278679+0000) 2019-10-13T16:39:27.281 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:27.278+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.279609+0000) 2019-10-13T16:39:27.282 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:27.275+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.280388+0000) 2019-10-13T16:39:27.283 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:27.278+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.281172+0000) 2019-10-13T16:39:27.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:27.278+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.281629+0000) 2019-10-13T16:39:27.283 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:27.278+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.282030+0000) 2019-10-13T16:39:27.288 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:27.283+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.286419+0000) 2019-10-13T16:39:27.293 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:27.290+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249416 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:27.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:27.727+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:27.733249+0000) 2019-10-13T16:39:30.280 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:30.274+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:30.278651+0000) 2019-10-13T16:39:30.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:30.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:30.528237+0000) 2019-10-13T16:39:32.296 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:32.290+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.294274+0000) 2019-10-13T16:39:32.297 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:32.290+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.295206+0000) 2019-10-13T16:39:32.297 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:32.294+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.295571+0000) 2019-10-13T16:39:32.298 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:32.291+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.296070+0000) 2019-10-13T16:39:32.299 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:32.294+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.297602+0000) 2019-10-13T16:39:32.300 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:32.294+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.298167+0000) 2019-10-13T16:39:32.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:32.294+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.298533+0000) 2019-10-13T16:39:32.306 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:32.299+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.304163+0000) 2019-10-13T16:39:32.309 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:32.302+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249448 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:32.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:32.789+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:32.794836+0000) 2019-10-13T16:39:35.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:35.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:35.806300+0000) 2019-10-13T16:39:35.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:35.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:35.806423+0000) 2019-10-13T16:39:35.816 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:35.809+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:35.814936+0000) 2019-10-13T16:39:36.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:36.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:36.357421+0000) 2019-10-13T16:39:37.311 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:37.305+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.309709+0000) 2019-10-13T16:39:37.312 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:37.303+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.310086+0000) 2019-10-13T16:39:37.313 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:37.305+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.311277+0000) 2019-10-13T16:39:37.313 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:37.309+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.312063+0000) 2019-10-13T16:39:37.314 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:37.309+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.312622+0000) 2019-10-13T16:39:37.315 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:37.309+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.313157+0000) 2019-10-13T16:39:37.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:37.309+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.313877+0000) 2019-10-13T16:39:37.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:37.315+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.318198+0000) 2019-10-13T16:39:37.325 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:37.317+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249483 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:37.742 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:37.735+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:37.739709+0000) 2019-10-13T16:39:40.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:40.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:40.278845+0000) 2019-10-13T16:39:40.530 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:40.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:40.528465+0000) 2019-10-13T16:39:42.329 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:42.321+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.326828+0000) 2019-10-13T16:39:42.329 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:42.325+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.327370+0000) 2019-10-13T16:39:42.330 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:42.323+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.327594+0000) 2019-10-13T16:39:42.331 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:42.325+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.328045+0000) 2019-10-13T16:39:42.331 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:42.325+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.328343+0000) 2019-10-13T16:39:42.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:42.325+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.329020+0000) 2019-10-13T16:39:42.332 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:42.325+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.329848+0000) 2019-10-13T16:39:42.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:42.327+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.332498+0000) 2019-10-13T16:39:42.340 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:42.333+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249516 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:42.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:42.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:42.795014+0000) 2019-10-13T16:39:45.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:45.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:45.806499+0000) 2019-10-13T16:39:45.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:45.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:45.806617+0000) 2019-10-13T16:39:45.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:45.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:45.815130+0000) 2019-10-13T16:39:46.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:46.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:46.357641+0000) 2019-10-13T16:39:47.344 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:47.337+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.341944+0000) 2019-10-13T16:39:47.344 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:47.337+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.342253+0000) 2019-10-13T16:39:47.345 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:47.341+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.343442+0000) 2019-10-13T16:39:47.346 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:47.339+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.344100+0000) 2019-10-13T16:39:47.346 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:47.341+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.344378+0000) 2019-10-13T16:39:47.346 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:47.341+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.344743+0000) 2019-10-13T16:39:47.348 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:47.341+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.346189+0000) 2019-10-13T16:39:47.351 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:47.343+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.348931+0000) 2019-10-13T16:39:47.356 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:47.349+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249550 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:47.748 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:47.743+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:47.745957+0000) 2019-10-13T16:39:50.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:50.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:50.279050+0000) 2019-10-13T16:39:50.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:50.523+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:50.528693+0000) 2019-10-13T16:39:52.359 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:52.353+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.356734+0000) 2019-10-13T16:39:52.360 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:52.353+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.358366+0000) 2019-10-13T16:39:52.360 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:52.353+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.358751+0000) 2019-10-13T16:39:52.361 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:52.353+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.359178+0000) 2019-10-13T16:39:52.361 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:52.357+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.359553+0000) 2019-10-13T16:39:52.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:52.357+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.360228+0000) 2019-10-13T16:39:52.362 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:52.355+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.360478+0000) 2019-10-13T16:39:52.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:52.359+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.363897+0000) 2019-10-13T16:39:52.373 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:52.369+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249582 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:52.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:52.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:52.795198+0000) 2019-10-13T16:39:55.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:55.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:55.806726+0000) 2019-10-13T16:39:55.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:55.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:55.806827+0000) 2019-10-13T16:39:55.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:55.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:55.815349+0000) 2019-10-13T16:39:56.359 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:56.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:56.357835+0000) 2019-10-13T16:39:57.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:39:57.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:39:57.203 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:39:57.203 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:39:57.377 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:39:57.369+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.375096+0000) 2019-10-13T16:39:57.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:39:57.373+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.375788+0000) 2019-10-13T16:39:57.378 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:39:57.371+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.375971+0000) 2019-10-13T16:39:57.378 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:39:57.373+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.376420+0000) 2019-10-13T16:39:57.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:39:57.373+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.376992+0000) 2019-10-13T16:39:57.380 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:39:57.373+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.378086+0000) 2019-10-13T16:39:57.380 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:39:57.373+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.378386+0000) 2019-10-13T16:39:57.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:57.379+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.383563+0000) 2019-10-13T16:39:57.388 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:39:57.381+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249617 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:39:57.755 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:39:57.747+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:39:57.752920+0000) 2019-10-13T16:40:00.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:00.273+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:00.279244+0000) 2019-10-13T16:40:00.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:00.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:00.528883+0000) 2019-10-13T16:40:02.392 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:02.385+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.389518+0000) 2019-10-13T16:40:02.392 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:02.385+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.390285+0000) 2019-10-13T16:40:02.392 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:02.385+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.390672+0000) 2019-10-13T16:40:02.393 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:02.385+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.390877+0000) 2019-10-13T16:40:02.393 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:02.389+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.391452+0000) 2019-10-13T16:40:02.394 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:02.387+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.392083+0000) 2019-10-13T16:40:02.394 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:02.389+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.392984+0000) 2019-10-13T16:40:02.400 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:02.395+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.398164+0000) 2019-10-13T16:40:02.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:02.401+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249650 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:02.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:02.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:02.795415+0000) 2019-10-13T16:40:05.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:05.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:05.806925+0000) 2019-10-13T16:40:05.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:05.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:05.807049+0000) 2019-10-13T16:40:05.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:05.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:05.815550+0000) 2019-10-13T16:40:06.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:06.357+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:06.358040+0000) 2019-10-13T16:40:07.408 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:07.401+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.406478+0000) 2019-10-13T16:40:07.409 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:07.401+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.407098+0000) 2019-10-13T16:40:07.410 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:07.403+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.408333+0000) 2019-10-13T16:40:07.411 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:07.405+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.408713+0000) 2019-10-13T16:40:07.411 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:07.405+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.408970+0000) 2019-10-13T16:40:07.411 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:07.405+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.409538+0000) 2019-10-13T16:40:07.412 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:07.405+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.410025+0000) 2019-10-13T16:40:07.416 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:07.411+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.414147+0000) 2019-10-13T16:40:07.420 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:07.413+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249684 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:07.761 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:07.759+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:07.759226+0000) 2019-10-13T16:40:10.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:10.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:10.279462+0000) 2019-10-13T16:40:10.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:10.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:10.529072+0000) 2019-10-13T16:40:12.424 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:12.417+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.422361+0000) 2019-10-13T16:40:12.425 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:12.419+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.423033+0000) 2019-10-13T16:40:12.425 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:12.417+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.423008+0000) 2019-10-13T16:40:12.425 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:12.421+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.423478+0000) 2019-10-13T16:40:12.426 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:12.421+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.424067+0000) 2019-10-13T16:40:12.426 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:12.421+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.424479+0000) 2019-10-13T16:40:12.427 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:12.421+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.425416+0000) 2019-10-13T16:40:12.432 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:12.427+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.430593+0000) 2019-10-13T16:40:12.436 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:12.429+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249716 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:12.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:12.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:12.795629+0000) 2019-10-13T16:40:15.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:15.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:15.807137+0000) 2019-10-13T16:40:15.809 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:15.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:15.807256+0000) 2019-10-13T16:40:15.817 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:15.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:15.815780+0000) 2019-10-13T16:40:16.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:16.353+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:16.358272+0000) 2019-10-13T16:40:17.440 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:17.432+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.437763+0000) 2019-10-13T16:40:17.440 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:17.432+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.438195+0000) 2019-10-13T16:40:17.440 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:17.432+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.438713+0000) 2019-10-13T16:40:17.441 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:17.436+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.439518+0000) 2019-10-13T16:40:17.442 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:17.436+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.440245+0000) 2019-10-13T16:40:17.442 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:17.434+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.440836+0000) 2019-10-13T16:40:17.443 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:17.436+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.441251+0000) 2019-10-13T16:40:17.448 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:17.442+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.446647+0000) 2019-10-13T16:40:17.452 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:17.444+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249751 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:17.767 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:17.762+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:17.765423+0000) 2019-10-13T16:40:20.281 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:20.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:20.279662+0000) 2019-10-13T16:40:20.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:20.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:20.529263+0000) 2019-10-13T16:40:22.454 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:22.448+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.452722+0000) 2019-10-13T16:40:22.455 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:22.448+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.453399+0000) 2019-10-13T16:40:22.455 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:22.446+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.453542+0000) 2019-10-13T16:40:22.456 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:22.448+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.454378+0000) 2019-10-13T16:40:22.457 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:22.448+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.455177+0000) 2019-10-13T16:40:22.457 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:22.452+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.455725+0000) 2019-10-13T16:40:22.458 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:22.452+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.456028+0000) 2019-10-13T16:40:22.465 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:22.458+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.463651+0000) 2019-10-13T16:40:22.468 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:22.460+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249784 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:22.797 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:22.792+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:22.795767+0000) 2019-10-13T16:40:25.809 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:25.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:25.807337+0000) 2019-10-13T16:40:25.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:25.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:25.807491+0000) 2019-10-13T16:40:25.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:25.812+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:25.816014+0000) 2019-10-13T16:40:26.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:26.352+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:26.358471+0000) 2019-10-13T16:40:27.322 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:40:27.322 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:40:27.326 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:40:27.326 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:40:27.471 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:27.464+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.469724+0000) 2019-10-13T16:40:27.472 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:27.464+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.470281+0000) 2019-10-13T16:40:27.473 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:27.464+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.471035+0000) 2019-10-13T16:40:27.473 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:27.468+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.471524+0000) 2019-10-13T16:40:27.474 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:27.468+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.471868+0000) 2019-10-13T16:40:27.474 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:27.466+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.472276+0000) 2019-10-13T16:40:27.475 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:27.468+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.472693+0000) 2019-10-13T16:40:27.481 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:27.474+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.478919+0000) 2019-10-13T16:40:27.484 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:27.476+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249818 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:27.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:27.770+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:27.771803+0000) 2019-10-13T16:40:30.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:30.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:30.279879+0000) 2019-10-13T16:40:30.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:30.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:30.529454+0000) 2019-10-13T16:40:32.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:32.480+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.485725+0000) 2019-10-13T16:40:32.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:32.480+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.486227+0000) 2019-10-13T16:40:32.488 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:32.480+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.486575+0000) 2019-10-13T16:40:32.489 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:32.482+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.487311+0000) 2019-10-13T16:40:32.489 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:32.484+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.487568+0000) 2019-10-13T16:40:32.490 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:32.484+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.488407+0000) 2019-10-13T16:40:32.490 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:32.484+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.488871+0000) 2019-10-13T16:40:32.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:32.490+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.495193+0000) 2019-10-13T16:40:32.500 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:32.492+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249850 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:32.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:32.792+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:32.795946+0000) 2019-10-13T16:40:35.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:35.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:35.807555+0000) 2019-10-13T16:40:35.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:35.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:35.807724+0000) 2019-10-13T16:40:35.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:35.812+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:35.816252+0000) 2019-10-13T16:40:36.360 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:36.356+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:36.358699+0000) 2019-10-13T16:40:37.504 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:37.496+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.502217+0000) 2019-10-13T16:40:37.505 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:37.496+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.503227+0000) 2019-10-13T16:40:37.506 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:37.500+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.504456+0000) 2019-10-13T16:40:37.507 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:37.500+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.504671+0000) 2019-10-13T16:40:37.507 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:37.500+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.505084+0000) 2019-10-13T16:40:37.507 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:37.500+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.505473+0000) 2019-10-13T16:40:37.508 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:37.498+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.505769+0000) 2019-10-13T16:40:37.511 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:37.502+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.509579+0000) 2019-10-13T16:40:37.517 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:37.508+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249884 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:37.780 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:37.774+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:37.778488+0000) 2019-10-13T16:40:40.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:40.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:40.280102+0000) 2019-10-13T16:40:40.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:40.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:40.529647+0000) 2019-10-13T16:40:42.521 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:42.512+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.518814+0000) 2019-10-13T16:40:42.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:42.516+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.519552+0000) 2019-10-13T16:40:42.522 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:42.516+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.519598+0000) 2019-10-13T16:40:42.522 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:42.516+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.520043+0000) 2019-10-13T16:40:42.522 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:42.516+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.520463+0000) 2019-10-13T16:40:42.523 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:42.514+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.521679+0000) 2019-10-13T16:40:42.524 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:42.516+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.522351+0000) 2019-10-13T16:40:42.530 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:42.522+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.528693+0000) 2019-10-13T16:40:42.533 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:42.528+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249918 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:42.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:42.792+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:42.796157+0000) 2019-10-13T16:40:45.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:45.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:45.807803+0000) 2019-10-13T16:40:45.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:45.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:45.807900+0000) 2019-10-13T16:40:45.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:45.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:45.816478+0000) 2019-10-13T16:40:46.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:46.356+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:46.358914+0000) 2019-10-13T16:40:47.537 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:47.528+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.534930+0000) 2019-10-13T16:40:47.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:47.530+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.535405+0000) 2019-10-13T16:40:47.538 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:47.532+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.535503+0000) 2019-10-13T16:40:47.538 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:47.532+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.535995+0000) 2019-10-13T16:40:47.539 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:47.532+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.537209+0000) 2019-10-13T16:40:47.539 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:47.532+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.537501+0000) 2019-10-13T16:40:47.540 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:47.532+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.536705+0000) 2019-10-13T16:40:47.544 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:47.538+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.542343+0000) 2019-10-13T16:40:47.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:47.544+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249951 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:47.787 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:47.782+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:47.784807+0000) 2019-10-13T16:40:50.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:50.276+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:50.280292+0000) 2019-10-13T16:40:50.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:50.522+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:50.529840+0000) 2019-10-13T16:40:52.552 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:52.544+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.550853+0000) 2019-10-13T16:40:52.553 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:52.544+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.551276+0000) 2019-10-13T16:40:52.554 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:52.548+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.551953+0000) 2019-10-13T16:40:52.554 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:52.548+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.552208+0000) 2019-10-13T16:40:52.554 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:52.548+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.553071+0000) 2019-10-13T16:40:52.555 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:52.546+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.553820+0000) 2019-10-13T16:40:52.556 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:52.548+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.554117+0000) 2019-10-13T16:40:52.561 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:52.554+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.559760+0000) 2019-10-13T16:40:52.565 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:52.560+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 249984 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:52.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:52.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:52.796368+0000) 2019-10-13T16:40:55.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:55.804+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:55.807999+0000) 2019-10-13T16:40:55.810 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:55.804+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:55.808089+0000) 2019-10-13T16:40:55.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:55.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:55.816671+0000) 2019-10-13T16:40:56.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:56.352+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:56.359073+0000) 2019-10-13T16:40:57.446 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:40:57.446 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:40:57.450 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:40:57.450 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:40:57.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:40:57.560+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.565983+0000) 2019-10-13T16:40:57.568 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:40:57.560+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.566854+0000) 2019-10-13T16:40:57.569 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:40:57.564+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.567976+0000) 2019-10-13T16:40:57.570 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:40:57.564+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.568336+0000) 2019-10-13T16:40:57.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:57.562+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.569462+0000) 2019-10-13T16:40:57.572 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:40:57.564+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.569655+0000) 2019-10-13T16:40:57.572 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:40:57.564+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.570024+0000) 2019-10-13T16:40:57.572 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:40:57.562+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.569810+0000) 2019-10-13T16:40:57.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:40:57.576+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250018 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:40:57.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:40:57.790+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:40:57.790994+0000) 2019-10-13T16:41:00.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:00.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:00.280492+0000) 2019-10-13T16:41:00.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:00.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:00.530033+0000) 2019-10-13T16:41:02.585 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:02.579+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.583140+0000) 2019-10-13T16:41:02.585 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:02.579+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.583713+0000) 2019-10-13T16:41:02.586 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:02.579+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.584367+0000) 2019-10-13T16:41:02.586 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:02.578+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.584679+0000) 2019-10-13T16:41:02.587 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:02.579+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.585347+0000) 2019-10-13T16:41:02.587 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:02.579+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.585864+0000) 2019-10-13T16:41:02.588 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:02.579+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.586364+0000) 2019-10-13T16:41:02.594 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:02.586+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.592779+0000) 2019-10-13T16:41:02.597 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:02.591+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250052 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:02.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:02.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:02.796552+0000) 2019-10-13T16:41:05.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:05.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:05.808190+0000) 2019-10-13T16:41:05.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:05.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:05.808265+0000) 2019-10-13T16:41:05.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:05.815+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:05.816872+0000) 2019-10-13T16:41:06.361 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:06.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:06.359267+0000) 2019-10-13T16:41:07.601 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:07.598+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.598420+0000) 2019-10-13T16:41:07.601 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:07.591+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.597849+0000) 2019-10-13T16:41:07.602 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:07.591+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.598800+0000) 2019-10-13T16:41:07.602 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:07.595+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.599296+0000) 2019-10-13T16:41:07.602 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:07.595+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.600112+0000) 2019-10-13T16:41:07.603 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:07.595+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.601164+0000) 2019-10-13T16:41:07.610 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:07.606+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.606190+0000) 2019-10-13T16:41:07.610 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:07.595+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.601698+0000) 2019-10-13T16:41:07.614 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:07.607+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250085 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:07.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:07.794+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:07.797598+0000) 2019-10-13T16:41:10.282 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:10.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:10.280729+0000) 2019-10-13T16:41:10.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:10.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:10.530230+0000) 2019-10-13T16:41:12.617 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:12.611+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.615663+0000) 2019-10-13T16:41:12.618 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:12.613+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.616004+0000) 2019-10-13T16:41:12.618 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:12.611+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.616449+0000) 2019-10-13T16:41:12.619 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:12.611+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.617081+0000) 2019-10-13T16:41:12.619 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:12.611+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.617493+0000) 2019-10-13T16:41:12.619 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:12.611+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.617868+0000) 2019-10-13T16:41:12.620 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:12.611+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.618367+0000) 2019-10-13T16:41:12.627 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:12.621+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.625847+0000) 2019-10-13T16:41:12.629 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:12.623+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250118 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:12.798 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:12.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:12.796764+0000) 2019-10-13T16:41:15.810 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:15.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:15.808409+0000) 2019-10-13T16:41:15.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:15.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:15.808414+0000) 2019-10-13T16:41:15.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:15.811+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:15.817068+0000) 2019-10-13T16:41:16.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:16.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:16.359504+0000) 2019-10-13T16:41:17.632 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:17.629+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.630433+0000) 2019-10-13T16:41:17.633 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:17.627+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.631402+0000) 2019-10-13T16:41:17.634 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:17.627+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.632478+0000) 2019-10-13T16:41:17.635 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:17.627+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.633164+0000) 2019-10-13T16:41:17.635 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:17.627+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.633408+0000) 2019-10-13T16:41:17.636 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:17.627+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.634537+0000) 2019-10-13T16:41:17.637 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:17.631+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.635356+0000) 2019-10-13T16:41:17.640 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:17.637+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.638455+0000) 2019-10-13T16:41:17.645 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:17.639+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250152 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:17.806 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:17.801+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:17.803919+0000) 2019-10-13T16:41:20.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:20.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:20.280923+0000) 2019-10-13T16:41:20.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:20.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:20.530436+0000) 2019-10-13T16:41:22.649 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:22.643+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.646788+0000) 2019-10-13T16:41:22.649 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:22.643+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.647081+0000) 2019-10-13T16:41:22.650 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:22.643+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.648273+0000) 2019-10-13T16:41:22.650 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:22.643+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.648833+0000) 2019-10-13T16:41:22.651 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:22.645+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.649063+0000) 2019-10-13T16:41:22.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:22.643+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.650104+0000) 2019-10-13T16:41:22.652 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:22.643+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.650797+0000) 2019-10-13T16:41:22.654 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:22.649+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.652812+0000) 2019-10-13T16:41:22.661 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:22.651+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:22.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:22.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:22.796953+0000) 2019-10-13T16:41:25.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:25.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:25.808612+0000) 2019-10-13T16:41:25.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:25.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:25.808617+0000) 2019-10-13T16:41:25.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:25.815+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:25.817268+0000) 2019-10-13T16:41:26.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:26.355+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:26.359671+0000) 2019-10-13T16:41:27.565 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:41:27.565 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:41:27.572 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:41:27.572 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:41:27.664 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:27.655+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.662572+0000) 2019-10-13T16:41:27.665 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:27.655+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.662861+0000) 2019-10-13T16:41:27.665 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:27.659+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.663623+0000) 2019-10-13T16:41:27.666 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:27.659+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.663528+0000) 2019-10-13T16:41:27.666 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:27.659+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.664460+0000) 2019-10-13T16:41:27.667 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:27.661+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.664759+0000) 2019-10-13T16:41:27.667 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:27.659+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.665573+0000) 2019-10-13T16:41:27.673 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:27.669+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.671508+0000) 2019-10-13T16:41:27.677 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:27.667+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:27.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:27.809+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:27.810316+0000) 2019-10-13T16:41:30.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:30.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:30.281134+0000) 2019-10-13T16:41:30.532 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:30.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:30.530665+0000) 2019-10-13T16:41:32.680 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:32.675+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.678434+0000) 2019-10-13T16:41:32.681 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:32.675+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.679115+0000) 2019-10-13T16:41:32.681 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:32.675+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.678825+0000) 2019-10-13T16:41:32.683 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:32.677+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.679869+0000) 2019-10-13T16:41:32.683 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:32.679+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.679554+0000) 2019-10-13T16:41:32.684 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:32.679+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.680558+0000) 2019-10-13T16:41:32.684 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:32.679+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.681960+0000) 2019-10-13T16:41:32.689 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:32.685+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.687495+0000) 2019-10-13T16:41:32.693 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:32.691+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250252 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:32.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:32.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:32.797138+0000) 2019-10-13T16:41:35.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:35.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:35.808785+0000) 2019-10-13T16:41:35.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:35.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:35.808842+0000) 2019-10-13T16:41:35.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:35.815+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:35.817498+0000) 2019-10-13T16:41:36.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:36.359+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:36.359904+0000) 2019-10-13T16:41:37.696 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:37.691+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.694503+0000) 2019-10-13T16:41:37.697 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:37.691+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.695233+0000) 2019-10-13T16:41:37.697 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:37.695+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.695971+0000) 2019-10-13T16:41:37.698 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:37.695+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.696152+0000) 2019-10-13T16:41:37.698 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:37.695+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.696809+0000) 2019-10-13T16:41:37.699 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:37.695+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.697202+0000) 2019-10-13T16:41:37.699 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:37.693+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.697616+0000) 2019-10-13T16:41:37.706 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:37.701+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.704465+0000) 2019-10-13T16:41:37.708 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:37.703+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:37.818 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:37.813+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:37.816638+0000) 2019-10-13T16:41:40.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:40.279+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:40.281330+0000) 2019-10-13T16:41:40.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:40.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:40.530857+0000) 2019-10-13T16:41:42.712 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:42.707+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.710548+0000) 2019-10-13T16:41:42.713 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:42.709+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.711184+0000) 2019-10-13T16:41:42.713 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:42.707+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.710954+0000) 2019-10-13T16:41:42.714 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:42.711+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.711949+0000) 2019-10-13T16:41:42.714 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:42.711+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.712667+0000) 2019-10-13T16:41:42.715 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:42.711+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.713260+0000) 2019-10-13T16:41:42.715 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:42.711+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.713686+0000) 2019-10-13T16:41:42.722 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:42.717+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.719956+0000) 2019-10-13T16:41:42.724 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:42.719+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250320 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:42.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:42.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:42.797320+0000) 2019-10-13T16:41:45.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:45.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:45.808991+0000) 2019-10-13T16:41:45.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:45.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:45.809038+0000) 2019-10-13T16:41:45.819 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:45.814+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:45.817715+0000) 2019-10-13T16:41:46.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:46.358+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:46.360168+0000) 2019-10-13T16:41:47.729 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:47.722+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.726744+0000) 2019-10-13T16:41:47.729 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:47.722+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.726967+0000) 2019-10-13T16:41:47.729 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:47.726+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.727579+0000) 2019-10-13T16:41:47.730 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:47.726+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.728336+0000) 2019-10-13T16:41:47.730 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:47.725+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.728633+0000) 2019-10-13T16:41:47.731 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:47.726+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.729430+0000) 2019-10-13T16:41:47.731 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:47.726+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.729795+0000) 2019-10-13T16:41:47.735 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:47.729+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.733402+0000) 2019-10-13T16:41:47.741 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:47.738+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250352 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:47.830 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:47.825+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:47.828059+0000) 2019-10-13T16:41:50.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:50.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:50.281562+0000) 2019-10-13T16:41:50.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:50.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:50.531054+0000) 2019-10-13T16:41:52.743 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:52.738+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.741663+0000) 2019-10-13T16:41:52.744 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:52.738+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.742204+0000) 2019-10-13T16:41:52.744 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:52.738+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.742366+0000) 2019-10-13T16:41:52.745 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:52.742+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.743806+0000) 2019-10-13T16:41:52.746 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:52.742+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.744596+0000) 2019-10-13T16:41:52.746 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:52.742+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.745090+0000) 2019-10-13T16:41:52.747 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:52.741+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.745133+0000) 2019-10-13T16:41:52.751 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:52.745+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.749495+0000) 2019-10-13T16:41:52.756 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:52.750+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250386 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:52.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:52.794+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:52.797537+0000) 2019-10-13T16:41:55.811 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:55.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:55.809241+0000) 2019-10-13T16:41:55.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:55.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:55.809241+0000) 2019-10-13T16:41:55.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:55.814+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:55.817909+0000) 2019-10-13T16:41:56.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:56.358+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:56.360368+0000) 2019-10-13T16:41:57.693 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:41:57.693 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:41:57.719 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:41:57.719 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:41:57.763 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:41:57.754+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.758194+0000) 2019-10-13T16:41:57.763 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:41:57.754+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.758411+0000) 2019-10-13T16:41:57.764 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:41:57.757+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.759113+0000) 2019-10-13T16:41:57.764 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:41:57.758+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.759493+0000) 2019-10-13T16:41:57.765 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:41:57.758+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.760550+0000) 2019-10-13T16:41:57.765 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:41:57.758+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.760839+0000) 2019-10-13T16:41:57.766 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:41:57.758+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.761342+0000) 2019-10-13T16:41:57.774 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:57.765+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.768353+0000) 2019-10-13T16:41:57.775 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:41:57.766+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250419 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:41:57.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:41:57.833+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:41:57.834310+0000) 2019-10-13T16:42:00.283 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:00.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:00.281766+0000) 2019-10-13T16:42:00.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:00.529+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:00.531277+0000) 2019-10-13T16:42:02.775 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:02.770+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.772806+0000) 2019-10-13T16:42:02.776 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:02.770+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.773320+0000) 2019-10-13T16:42:02.776 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:02.770+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.774198+0000) 2019-10-13T16:42:02.777 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:02.773+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.774127+0000) 2019-10-13T16:42:02.777 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:02.774+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.775381+0000) 2019-10-13T16:42:02.778 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:02.774+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.775789+0000) 2019-10-13T16:42:02.778 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:02.770+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.773569+0000) 2019-10-13T16:42:02.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:02.781+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.782603+0000) 2019-10-13T16:42:02.787 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:02.782+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250454 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:02.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:02.794+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:02.797677+0000) 2019-10-13T16:42:05.811 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:05.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:05.809438+0000) 2019-10-13T16:42:05.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:05.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:05.809467+0000) 2019-10-13T16:42:05.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:05.814+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:05.818145+0000) 2019-10-13T16:42:06.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:06.358+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:06.360565+0000) 2019-10-13T16:42:07.790 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:07.786+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.788571+0000) 2019-10-13T16:42:07.791 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:07.786+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.789039+0000) 2019-10-13T16:42:07.791 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:07.786+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.789858+0000) 2019-10-13T16:42:07.792 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:07.786+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.790206+0000) 2019-10-13T16:42:07.792 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:07.786+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.790893+0000) 2019-10-13T16:42:07.793 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:07.786+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.791158+0000) 2019-10-13T16:42:07.793 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:07.788+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.791309+0000) 2019-10-13T16:42:07.799 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:07.792+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.797899+0000) 2019-10-13T16:42:07.803 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:07.798+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250488 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:07.842 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:07.836+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:07.840219+0000) 2019-10-13T16:42:10.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:10.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:10.281987+0000) 2019-10-13T16:42:10.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:10.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:10.531477+0000) 2019-10-13T16:42:12.799 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:12.794+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.797830+0000) 2019-10-13T16:42:12.806 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:12.800+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.804321+0000) 2019-10-13T16:42:12.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:12.802+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.804722+0000) 2019-10-13T16:42:12.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:12.802+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.805929+0000) 2019-10-13T16:42:12.808 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:12.802+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.806315+0000) 2019-10-13T16:42:12.808 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:12.802+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.806817+0000) 2019-10-13T16:42:12.814 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:12.808+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.810925+0000) 2019-10-13T16:42:12.814 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:12.806+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.807444+0000) 2019-10-13T16:42:12.815 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:12.806+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:12.807507+0000) 2019-10-13T16:42:12.818 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:12.814+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250520 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:15.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:15.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:15.809639+0000) 2019-10-13T16:42:15.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:15.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:15.809694+0000) 2019-10-13T16:42:15.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:15.814+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:15.818391+0000) 2019-10-13T16:42:16.362 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:16.358+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:16.360761+0000) 2019-10-13T16:42:17.822 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:17.818+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.819853+0000) 2019-10-13T16:42:17.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:17.818+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.820132+0000) 2019-10-13T16:42:17.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:17.818+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.821309+0000) 2019-10-13T16:42:17.823 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:17.816+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.821554+0000) 2019-10-13T16:42:17.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:17.818+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.822292+0000) 2019-10-13T16:42:17.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:17.818+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.822905+0000) 2019-10-13T16:42:17.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:17.822+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.823351+0000) 2019-10-13T16:42:17.831 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:17.824+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.829748+0000) 2019-10-13T16:42:17.834 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:17.830+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250555 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:17.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:17.840+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:17.844202+0000) 2019-10-13T16:42:20.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:20.278+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:20.282200+0000) 2019-10-13T16:42:20.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:20.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:20.531661+0000) 2019-10-13T16:42:22.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:22.794+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.798069+0000) 2019-10-13T16:42:22.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:22.830+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.835192+0000) 2019-10-13T16:42:22.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:22.834+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.836148+0000) 2019-10-13T16:42:22.838 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:22.832+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.836871+0000) 2019-10-13T16:42:22.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:22.834+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.837271+0000) 2019-10-13T16:42:22.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:22.834+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.837884+0000) 2019-10-13T16:42:22.841 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:22.834+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.839203+0000) 2019-10-13T16:42:22.841 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:22.838+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.839696+0000) 2019-10-13T16:42:22.846 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:22.840+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:22.844227+0000) 2019-10-13T16:42:22.849 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:22.846+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250588 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:25.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:25.806+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:25.809908+0000) 2019-10-13T16:42:25.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:25.806+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:25.809898+0000) 2019-10-13T16:42:25.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:25.818+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:25.818585+0000) 2019-10-13T16:42:26.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:26.358+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:26.360973+0000) 2019-10-13T16:42:27.835 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:42:27.835 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:42:27.838 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:42:27.839 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:42:27.850 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:27.844+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.848752+0000) 2019-10-13T16:42:27.854 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:27.848+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.852459+0000) 2019-10-13T16:42:27.855 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:27.850+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.853135+0000) 2019-10-13T16:42:27.856 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:27.850+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.853882+0000) 2019-10-13T16:42:27.861 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:27.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.853767+0000) 2019-10-13T16:42:27.862 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:27.850+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.853729+0000) 2019-10-13T16:42:27.862 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:27.850+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.854464+0000) 2019-10-13T16:42:27.863 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:27.850+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.854850+0000) 2019-10-13T16:42:27.863 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:27.856+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:27.861656+0000) 2019-10-13T16:42:27.866 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:27.862+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250622 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:30.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:30.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:30.282430+0000) 2019-10-13T16:42:30.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:30.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:30.531855+0000) 2019-10-13T16:42:32.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:32.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.798285+0000) 2019-10-13T16:42:32.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:32.865+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.868904+0000) 2019-10-13T16:42:32.871 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:32.865+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.869954+0000) 2019-10-13T16:42:32.872 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:32.868+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.870344+0000) 2019-10-13T16:42:32.872 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:32.865+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.870848+0000) 2019-10-13T16:42:32.873 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:32.865+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.871158+0000) 2019-10-13T16:42:32.873 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:32.869+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.871711+0000) 2019-10-13T16:42:32.874 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:32.869+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.872839+0000) 2019-10-13T16:42:32.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:32.872+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:32.875456+0000) 2019-10-13T16:42:32.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:32.877+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250654 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:35.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:35.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:35.810104+0000) 2019-10-13T16:42:35.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:35.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:35.810106+0000) 2019-10-13T16:42:35.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:35.817+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:35.818815+0000) 2019-10-13T16:42:36.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:36.357+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:36.361181+0000) 2019-10-13T16:42:37.854 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:37.848+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.852125+0000) 2019-10-13T16:42:37.884 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:37.877+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.882247+0000) 2019-10-13T16:42:37.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:37.877+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.882822+0000) 2019-10-13T16:42:37.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:37.880+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.883289+0000) 2019-10-13T16:42:37.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:37.877+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.883167+0000) 2019-10-13T16:42:37.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:37.877+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.883235+0000) 2019-10-13T16:42:37.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:37.881+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.883651+0000) 2019-10-13T16:42:37.887 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:37.881+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.883983+0000) 2019-10-13T16:42:37.892 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:37.888+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:37.890212+0000) 2019-10-13T16:42:37.897 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:37.893+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250689 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:40.284 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:40.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:40.282618+0000) 2019-10-13T16:42:40.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:40.528+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:40.532048+0000) 2019-10-13T16:42:42.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:42.793+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.798485+0000) 2019-10-13T16:42:42.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:42.893+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.898379+0000) 2019-10-13T16:42:42.901 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:42.896+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.898916+0000) 2019-10-13T16:42:42.901 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:42.897+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.899901+0000) 2019-10-13T16:42:42.902 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:42.897+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.900402+0000) 2019-10-13T16:42:42.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:42.897+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.900762+0000) 2019-10-13T16:42:42.903 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:42.897+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.901499+0000) 2019-10-13T16:42:42.904 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:42.897+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.902350+0000) 2019-10-13T16:42:42.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:42.904+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:42.907156+0000) 2019-10-13T16:42:42.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:42.905+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250722 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:45.812 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:45.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:45.810341+0000) 2019-10-13T16:42:45.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:45.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:45.810339+0000) 2019-10-13T16:42:45.820 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:45.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:45.819013+0000) 2019-10-13T16:42:46.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:46.357+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:46.361431+0000) 2019-10-13T16:42:47.860 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:47.856+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.858645+0000) 2019-10-13T16:42:47.915 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:47.909+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.913282+0000) 2019-10-13T16:42:47.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:47.909+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.914581+0000) 2019-10-13T16:42:47.917 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:47.909+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.915244+0000) 2019-10-13T16:42:47.917 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:47.913+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.915465+0000) 2019-10-13T16:42:47.918 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:47.912+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.915845+0000) 2019-10-13T16:42:47.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:47.913+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.916154+0000) 2019-10-13T16:42:47.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:47.913+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.916942+0000) 2019-10-13T16:42:47.922 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:47.916+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:47.920207+0000) 2019-10-13T16:42:47.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:47.921+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250756 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:50.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:50.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:50.282818+0000) 2019-10-13T16:42:50.534 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:50.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:50.532257+0000) 2019-10-13T16:42:52.800 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:52.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.798725+0000) 2019-10-13T16:42:52.930 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:52.924+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.927861+0000) 2019-10-13T16:42:52.930 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:52.925+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.928372+0000) 2019-10-13T16:42:52.931 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:52.925+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.929504+0000) 2019-10-13T16:42:52.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:52.925+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.929951+0000) 2019-10-13T16:42:52.932 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:52.925+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.930340+0000) 2019-10-13T16:42:52.932 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:52.925+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.930833+0000) 2019-10-13T16:42:52.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:52.929+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.932116+0000) 2019-10-13T16:42:52.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:52.928+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:52.933467+0000) 2019-10-13T16:42:52.940 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:52.933+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250788 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:55.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:55.805+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:55.810501+0000) 2019-10-13T16:42:55.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:55.805+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:55.810606+0000) 2019-10-13T16:42:55.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:55.813+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:55.819211+0000) 2019-10-13T16:42:56.363 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:56.357+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:56.361673+0000) 2019-10-13T16:42:57.867 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:57.864+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.865378+0000) 2019-10-13T16:42:57.944 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:42:57.937+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.942118+0000) 2019-10-13T16:42:57.944 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:42:57.940+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.942753+0000) 2019-10-13T16:42:57.945 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:42:57.937+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.942990+0000) 2019-10-13T16:42:57.945 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:42:57.941+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.943478+0000) 2019-10-13T16:42:57.945 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:42:57.941+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.943838+0000) 2019-10-13T16:42:57.946 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:42:57.941+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.944561+0000) 2019-10-13T16:42:57.946 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:42:57.941+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.944782+0000) 2019-10-13T16:42:57.949 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:42:57.944+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:42:57.947913+0000) 2019-10-13T16:42:57.952 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:42:57.952 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:42:57.954 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:42:57.949+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250824 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:42:57.957 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:42:57.957 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:43:00.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:00.277+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:00.283016+0000) 2019-10-13T16:43:00.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:00.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:00.532456+0000) 2019-10-13T16:43:02.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:02.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.798969+0000) 2019-10-13T16:43:02.957 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:02.953+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.955826+0000) 2019-10-13T16:43:02.958 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:02.953+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.956414+0000) 2019-10-13T16:43:02.958 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:02.953+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.956772+0000) 2019-10-13T16:43:02.959 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:02.953+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.957208+0000) 2019-10-13T16:43:02.960 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:02.955+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.958276+0000) 2019-10-13T16:43:02.960 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:02.953+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.958890+0000) 2019-10-13T16:43:02.961 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:02.953+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.959176+0000) 2019-10-13T16:43:02.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:02.959+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:02.962255+0000) 2019-10-13T16:43:02.970 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:02.965+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250855 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:05.812 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:05.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:05.810700+0000) 2019-10-13T16:43:05.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:05.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:05.810798+0000) 2019-10-13T16:43:05.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:05.817+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:05.819441+0000) 2019-10-13T16:43:06.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:06.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:06.361906+0000) 2019-10-13T16:43:07.874 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:07.867+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.871994+0000) 2019-10-13T16:43:07.973 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:07.965+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.971188+0000) 2019-10-13T16:43:07.974 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:07.969+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.972800+0000) 2019-10-13T16:43:07.975 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:07.969+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.973490+0000) 2019-10-13T16:43:07.975 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:07.967+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.973762+0000) 2019-10-13T16:43:07.976 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:07.969+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.973580+0000) 2019-10-13T16:43:07.976 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:07.969+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.975108+0000) 2019-10-13T16:43:07.977 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:07.973+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.975366+0000) 2019-10-13T16:43:07.982 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:07.975+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:07.980838+0000) 2019-10-13T16:43:07.986 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:07.981+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250890 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:10.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:10.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:10.283193+0000) 2019-10-13T16:43:10.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:10.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:10.532647+0000) 2019-10-13T16:43:12.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:12.792+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.799159+0000) 2019-10-13T16:43:12.989 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:12.984+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.987788+0000) 2019-10-13T16:43:12.991 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:12.984+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.989476+0000) 2019-10-13T16:43:12.992 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:12.984+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.990162+0000) 2019-10-13T16:43:12.993 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:12.988+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.991408+0000) 2019-10-13T16:43:12.993 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:12.984+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.990890+0000) 2019-10-13T16:43:12.994 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:12.987+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.991493+0000) 2019-10-13T16:43:12.994 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:12.988+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.992031+0000) 2019-10-13T16:43:12.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:12.991+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:12.996507+0000) 2019-10-13T16:43:13.002 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:12.996+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250921 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:15.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:15.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:15.810876+0000) 2019-10-13T16:43:15.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:15.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:15.811006+0000) 2019-10-13T16:43:15.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:15.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:15.819648+0000) 2019-10-13T16:43:16.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:16.356+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:16.362065+0000) 2019-10-13T16:43:17.880 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:17.875+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:17.878581+0000) 2019-10-13T16:43:18.006 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:18.000+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.004644+0000) 2019-10-13T16:43:18.008 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:18.000+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.005464+0000) 2019-10-13T16:43:18.008 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:18.000+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.005527+0000) 2019-10-13T16:43:18.008 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:18.000+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.005819+0000) 2019-10-13T16:43:18.008 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:18.000+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.006082+0000) 2019-10-13T16:43:18.009 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:18.003+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.006415+0000) 2019-10-13T16:43:18.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:18.000+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.006629+0000) 2019-10-13T16:43:18.013 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:18.007+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:18.011786+0000) 2019-10-13T16:43:18.018 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:18.012+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250958 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:20.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:20.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:20.283394+0000) 2019-10-13T16:43:20.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:20.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:20.532838+0000) 2019-10-13T16:43:22.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:22.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:22.799364+0000) 2019-10-13T16:43:23.022 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:23.015+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.019812+0000) 2019-10-13T16:43:23.022 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:23.016+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.020661+0000) 2019-10-13T16:43:23.023 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:23.016+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.021658+0000) 2019-10-13T16:43:23.024 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:23.016+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.022219+0000) 2019-10-13T16:43:23.024 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:23.016+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.022412+0000) 2019-10-13T16:43:23.025 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:23.016+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.022797+0000) 2019-10-13T16:43:23.025 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:23.020+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.023449+0000) 2019-10-13T16:43:23.029 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:23.023+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:23.026965+0000) 2019-10-13T16:43:23.034 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:23.028+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 250989 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:25.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:25.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:25.811073+0000) 2019-10-13T16:43:25.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:25.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:25.811233+0000) 2019-10-13T16:43:25.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:25.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:25.819856+0000) 2019-10-13T16:43:26.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:26.360+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:26.362313+0000) 2019-10-13T16:43:27.887 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:27.883+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:27.885160+0000) 2019-10-13T16:43:28.038 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:28.032+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.035899+0000) 2019-10-13T16:43:28.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:28.032+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.036247+0000) 2019-10-13T16:43:28.038 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:28.031+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.036678+0000) 2019-10-13T16:43:28.039 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:28.032+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.037609+0000) 2019-10-13T16:43:28.040 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:28.032+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.038152+0000) 2019-10-13T16:43:28.040 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:28.032+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.038412+0000) 2019-10-13T16:43:28.041 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:28.036+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.039884+0000) 2019-10-13T16:43:28.043 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:28.035+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:28.041466+0000) 2019-10-13T16:43:28.050 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:28.044+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:28.063 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:43:28.063 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:43:28.067 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:43:28.067 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:43:30.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:30.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:30.283588+0000) 2019-10-13T16:43:30.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:30.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:30.533029+0000) 2019-10-13T16:43:32.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:32.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:32.799557+0000) 2019-10-13T16:43:33.054 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:33.048+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.052215+0000) 2019-10-13T16:43:33.055 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:33.048+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.053740+0000) 2019-10-13T16:43:33.056 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:33.048+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.054486+0000) 2019-10-13T16:43:33.056 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:33.048+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.054921+0000) 2019-10-13T16:43:33.057 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:33.051+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.054883+0000) 2019-10-13T16:43:33.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:33.052+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.055429+0000) 2019-10-13T16:43:33.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:33.052+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.055596+0000) 2019-10-13T16:43:33.064 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:33.059+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:33.062168+0000) 2019-10-13T16:43:33.067 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:33.060+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:35.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:35.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:35.811282+0000) 2019-10-13T16:43:35.813 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:35.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:35.811467+0000) 2019-10-13T16:43:35.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:35.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:35.820016+0000) 2019-10-13T16:43:36.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:36.360+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:36.362514+0000) 2019-10-13T16:43:37.894 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:37.887+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:37.891724+0000) 2019-10-13T16:43:38.072 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:38.064+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.069835+0000) 2019-10-13T16:43:38.072 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:38.064+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.070418+0000) 2019-10-13T16:43:38.072 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:38.064+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.070614+0000) 2019-10-13T16:43:38.073 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:38.068+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.071893+0000) 2019-10-13T16:43:38.074 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:38.067+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.072262+0000) 2019-10-13T16:43:38.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:38.068+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.072751+0000) 2019-10-13T16:43:38.075 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:38.068+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.073375+0000) 2019-10-13T16:43:38.079 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:38.071+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:38.077039+0000) 2019-10-13T16:43:38.083 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:38.076+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251092 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:40.285 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:40.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:40.283804+0000) 2019-10-13T16:43:40.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:40.531+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:40.533255+0000) 2019-10-13T16:43:42.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:42.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:42.799757+0000) 2019-10-13T16:43:43.086 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:43.080+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.084573+0000) 2019-10-13T16:43:43.087 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:43.080+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.085617+0000) 2019-10-13T16:43:43.087 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:43.080+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.086024+0000) 2019-10-13T16:43:43.088 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:43.083+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.086658+0000) 2019-10-13T16:43:43.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:43.080+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.086952+0000) 2019-10-13T16:43:43.089 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:43.084+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.087502+0000) 2019-10-13T16:43:43.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:43.084+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.089526+0000) 2019-10-13T16:43:43.097 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:43.091+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:43.094990+0000) 2019-10-13T16:43:43.100 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:43.092+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251123 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:45.813 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:45.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:45.811576+0000) 2019-10-13T16:43:45.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:45.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:45.811686+0000) 2019-10-13T16:43:45.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:45.816+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:45.820260+0000) 2019-10-13T16:43:46.364 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:46.360+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:46.362725+0000) 2019-10-13T16:43:47.900 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:47.895+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:47.898330+0000) 2019-10-13T16:43:48.104 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:48.096+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.102460+0000) 2019-10-13T16:43:48.105 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:48.096+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.102758+0000) 2019-10-13T16:43:48.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:48.099+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.103167+0000) 2019-10-13T16:43:48.106 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:48.100+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.103939+0000) 2019-10-13T16:43:48.106 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:48.100+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.104324+0000) 2019-10-13T16:43:48.106 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:48.100+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.104945+0000) 2019-10-13T16:43:48.107 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:48.100+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.105344+0000) 2019-10-13T16:43:48.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:48.111+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:48.114324+0000) 2019-10-13T16:43:48.117 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:48.108+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251158 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:50.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:50.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:50.284021+0000) 2019-10-13T16:43:50.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:50.527+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:50.533446+0000) 2019-10-13T16:43:52.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:52.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:52.799978+0000) 2019-10-13T16:43:53.121 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:53.112+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.118853+0000) 2019-10-13T16:43:53.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:53.116+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.119493+0000) 2019-10-13T16:43:53.122 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:53.116+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.119839+0000) 2019-10-13T16:43:53.122 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:53.112+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.119156+0000) 2019-10-13T16:43:53.122 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:53.116+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.120769+0000) 2019-10-13T16:43:53.123 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:53.115+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.121684+0000) 2019-10-13T16:43:53.124 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:53.116+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.122289+0000) 2019-10-13T16:43:53.131 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:53.123+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:53.129086+0000) 2019-10-13T16:43:53.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:53.124+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251188 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:55.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:55.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:55.811822+0000) 2019-10-13T16:43:55.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:55.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:55.811921+0000) 2019-10-13T16:43:55.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:55.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:55.820454+0000) 2019-10-13T16:43:56.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:56.359+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:56.362944+0000) 2019-10-13T16:43:57.907 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:57.902+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:57.904941+0000) 2019-10-13T16:43:58.135 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:43:58.127+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.133521+0000) 2019-10-13T16:43:58.137 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:43:58.131+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.135496+0000) 2019-10-13T16:43:58.138 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:43:58.131+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.136061+0000) 2019-10-13T16:43:58.138 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:43:58.131+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.136225+0000) 2019-10-13T16:43:58.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:43:58.131+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.137450+0000) 2019-10-13T16:43:58.139 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:43:58.130+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.137797+0000) 2019-10-13T16:43:58.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:43:58.131+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.136947+0000) 2019-10-13T16:43:58.143 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:43:58.134+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:43:58.141634+0000) 2019-10-13T16:43:58.148 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:43:58.139+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251226 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:43:58.171 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:43:58.171 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:43:58.175 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:43:58.175 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:44:00.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:00.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:00.284210+0000) 2019-10-13T16:44:00.535 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:00.526+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:00.533637+0000) 2019-10-13T16:44:02.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:02.795+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:02.800177+0000) 2019-10-13T16:44:03.152 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:03.143+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.150210+0000) 2019-10-13T16:44:03.152 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:03.143+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.150953+0000) 2019-10-13T16:44:03.153 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:03.147+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.151639+0000) 2019-10-13T16:44:03.153 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:03.147+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.151975+0000) 2019-10-13T16:44:03.154 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:03.146+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.152089+0000) 2019-10-13T16:44:03.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:03.147+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.153588+0000) 2019-10-13T16:44:03.155 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:03.147+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.153842+0000) 2019-10-13T16:44:03.166 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:03.155+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251256 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:03.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:03.154+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:03.158858+0000) 2019-10-13T16:44:05.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:05.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:05.812038+0000) 2019-10-13T16:44:05.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:05.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:05.812117+0000) 2019-10-13T16:44:05.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:05.815+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:05.820658+0000) 2019-10-13T16:44:06.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:06.359+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:06.363140+0000) 2019-10-13T16:44:07.913 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:07.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:07.911495+0000) 2019-10-13T16:44:08.167 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:08.159+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.165477+0000) 2019-10-13T16:44:08.168 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:08.159+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.166204+0000) 2019-10-13T16:44:08.168 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:08.159+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.166416+0000) 2019-10-13T16:44:08.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:08.159+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.167164+0000) 2019-10-13T16:44:08.169 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:08.163+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.167614+0000) 2019-10-13T16:44:08.170 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:08.162+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.168010+0000) 2019-10-13T16:44:08.170 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:08.163+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.168585+0000) 2019-10-13T16:44:08.175 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:08.166+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:08.173018+0000) 2019-10-13T16:44:08.177 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:08.171+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251292 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:10.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:10.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:10.284417+0000) 2019-10-13T16:44:10.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:10.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:10.533828+0000) 2019-10-13T16:44:12.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:12.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:12.800412+0000) 2019-10-13T16:44:13.182 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:13.175+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.179820+0000) 2019-10-13T16:44:13.182 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:13.175+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.180446+0000) 2019-10-13T16:44:13.183 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:13.175+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.181329+0000) 2019-10-13T16:44:13.183 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:13.175+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.182026+0000) 2019-10-13T16:44:13.184 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:13.175+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.182844+0000) 2019-10-13T16:44:13.185 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:13.178+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.183415+0000) 2019-10-13T16:44:13.185 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:13.179+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.183946+0000) 2019-10-13T16:44:13.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:13.182+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:13.188443+0000) 2019-10-13T16:44:13.193 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:13.187+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:15.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:15.807+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:15.812258+0000) 2019-10-13T16:44:15.814 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:15.807+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:15.812310+0000) 2019-10-13T16:44:15.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:15.815+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:15.820852+0000) 2019-10-13T16:44:16.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:16.359+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:16.363338+0000) 2019-10-13T16:44:17.921 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:17.910+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:17.918071+0000) 2019-10-13T16:44:18.197 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:18.191+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.195631+0000) 2019-10-13T16:44:18.198 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:18.191+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.196011+0000) 2019-10-13T16:44:18.198 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:18.191+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.196314+0000) 2019-10-13T16:44:18.199 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:18.191+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.197372+0000) 2019-10-13T16:44:18.199 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:18.190+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.197367+0000) 2019-10-13T16:44:18.200 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:18.191+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.197697+0000) 2019-10-13T16:44:18.205 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:18.198+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.203286+0000) 2019-10-13T16:44:18.205 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:18.191+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:18.198644+0000) 2019-10-13T16:44:18.210 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:18.203+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251361 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:20.286 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:20.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:20.284646+0000) 2019-10-13T16:44:20.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:20.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:20.534016+0000) 2019-10-13T16:44:22.802 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:22.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:22.800680+0000) 2019-10-13T16:44:23.213 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:23.207+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.210830+0000) 2019-10-13T16:44:23.213 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:23.206+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.211136+0000) 2019-10-13T16:44:23.215 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:23.207+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.213182+0000) 2019-10-13T16:44:23.215 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:23.207+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.212982+0000) 2019-10-13T16:44:23.215 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:23.207+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.214133+0000) 2019-10-13T16:44:23.216 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:23.207+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.214576+0000) 2019-10-13T16:44:23.216 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:23.207+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.215003+0000) 2019-10-13T16:44:23.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:23.214+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:23.220222+0000) 2019-10-13T16:44:23.226 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:23.219+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251392 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:25.814 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:25.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:25.812459+0000) 2019-10-13T16:44:25.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:25.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:25.812544+0000) 2019-10-13T16:44:25.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:25.819+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:25.821003+0000) 2019-10-13T16:44:26.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:26.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:26.363499+0000) 2019-10-13T16:44:27.926 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:27.922+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:27.924684+0000) 2019-10-13T16:44:28.229 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:28.227+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.227428+0000) 2019-10-13T16:44:28.229 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:28.227+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.227540+0000) 2019-10-13T16:44:28.230 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:28.227+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.228193+0000) 2019-10-13T16:44:28.230 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:28.227+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.229127+0000) 2019-10-13T16:44:28.231 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:28.222+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.229514+0000) 2019-10-13T16:44:28.232 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:28.227+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.229963+0000) 2019-10-13T16:44:28.233 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:28.231+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.231351+0000) 2019-10-13T16:44:28.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:28.230+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:28.237240+0000) 2019-10-13T16:44:28.242 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:28.239+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:28.277 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:44:28.277 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:44:28.281 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:44:28.281 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:44:30.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:30.283+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:30.284842+0000) 2019-10-13T16:44:30.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:30.530+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:30.534211+0000) 2019-10-13T16:44:32.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:32.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:32.800883+0000) 2019-10-13T16:44:33.246 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:33.243+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.244243+0000) 2019-10-13T16:44:33.246 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:33.243+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.244983+0000) 2019-10-13T16:44:33.247 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:33.243+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.245425+0000) 2019-10-13T16:44:33.247 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:33.243+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.245689+0000) 2019-10-13T16:44:33.248 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:33.238+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.245913+0000) 2019-10-13T16:44:33.248 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:33.243+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.246436+0000) 2019-10-13T16:44:33.248 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:33.243+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.246617+0000) 2019-10-13T16:44:33.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:33.246+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:33.253359+0000) 2019-10-13T16:44:33.258 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:33.255+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251458 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:35.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:35.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:35.812695+0000) 2019-10-13T16:44:35.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:35.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:35.812755+0000) 2019-10-13T16:44:35.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:35.819+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:35.821229+0000) 2019-10-13T16:44:36.365 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:36.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:36.363768+0000) 2019-10-13T16:44:37.933 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:37.930+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:37.931176+0000) 2019-10-13T16:44:38.262 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:38.259+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.260377+0000) 2019-10-13T16:44:38.263 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:38.254+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.260970+0000) 2019-10-13T16:44:38.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:38.259+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.261445+0000) 2019-10-13T16:44:38.264 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:38.259+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.261800+0000) 2019-10-13T16:44:38.264 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:38.259+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.262751+0000) 2019-10-13T16:44:38.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:38.259+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.262999+0000) 2019-10-13T16:44:38.265 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:38.259+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.263060+0000) 2019-10-13T16:44:38.271 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:38.262+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:38.269888+0000) 2019-10-13T16:44:38.273 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:38.271+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251497 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:40.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:40.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:40.285038+0000) 2019-10-13T16:44:40.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:40.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:40.534360+0000) 2019-10-13T16:44:42.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:42.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:42.801079+0000) 2019-10-13T16:44:43.277 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:43.270+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.274858+0000) 2019-10-13T16:44:43.277 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:43.270+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.275870+0000) 2019-10-13T16:44:43.278 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:43.274+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.276742+0000) 2019-10-13T16:44:43.279 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:43.274+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.277043+0000) 2019-10-13T16:44:43.279 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:43.274+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.277504+0000) 2019-10-13T16:44:43.279 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:43.274+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.277688+0000) 2019-10-13T16:44:43.280 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:43.274+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.278711+0000) 2019-10-13T16:44:43.284 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:43.278+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:43.282841+0000) 2019-10-13T16:44:43.289 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:43.286+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251528 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:45.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:45.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:45.812955+0000) 2019-10-13T16:44:45.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:45.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:45.812946+0000) 2019-10-13T16:44:45.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:45.818+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:45.821441+0000) 2019-10-13T16:44:46.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:46.362+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:46.363960+0000) 2019-10-13T16:44:47.940 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:47.934+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:47.937739+0000) 2019-10-13T16:44:48.293 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:48.286+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.291164+0000) 2019-10-13T16:44:48.293 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:48.290+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.291567+0000) 2019-10-13T16:44:48.294 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:48.290+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.291955+0000) 2019-10-13T16:44:48.294 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:48.290+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.292369+0000) 2019-10-13T16:44:48.295 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:48.290+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.292712+0000) 2019-10-13T16:44:48.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:48.290+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.293526+0000) 2019-10-13T16:44:48.295 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:48.290+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.293781+0000) 2019-10-13T16:44:48.301 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:48.298+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:48.299255+0000) 2019-10-13T16:44:48.304 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:48.298+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:50.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:50.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:50.285239+0000) 2019-10-13T16:44:50.536 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:50.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:50.534555+0000) 2019-10-13T16:44:52.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:52.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:52.801287+0000) 2019-10-13T16:44:53.308 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:53.302+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.305653+0000) 2019-10-13T16:44:53.308 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:53.302+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.306090+0000) 2019-10-13T16:44:53.309 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:53.305+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.306633+0000) 2019-10-13T16:44:53.309 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:53.306+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.307384+0000) 2019-10-13T16:44:53.310 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:53.306+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.307568+0000) 2019-10-13T16:44:53.310 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:53.306+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.307795+0000) 2019-10-13T16:44:53.310 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:53.306+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.309033+0000) 2019-10-13T16:44:53.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:53.313+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:53.314946+0000) 2019-10-13T16:44:53.320 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:53.314+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:55.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:55.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:55.813155+0000) 2019-10-13T16:44:55.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:55.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:55.813154+0000) 2019-10-13T16:44:55.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:55.818+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:55.821672+0000) 2019-10-13T16:44:56.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:56.362+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:56.364160+0000) 2019-10-13T16:44:57.946 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:57.941+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:57.944292+0000) 2019-10-13T16:44:58.322 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:44:58.318+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.320798+0000) 2019-10-13T16:44:58.323 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:44:58.318+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.321332+0000) 2019-10-13T16:44:58.324 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:44:58.318+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.322249+0000) 2019-10-13T16:44:58.325 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:44:58.321+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.322351+0000) 2019-10-13T16:44:58.325 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:44:58.318+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.322911+0000) 2019-10-13T16:44:58.326 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:44:58.322+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.324279+0000) 2019-10-13T16:44:58.326 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:44:58.322+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.325091+0000) 2019-10-13T16:44:58.334 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:44:58.329+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:44:58.331901+0000) 2019-10-13T16:44:58.335 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:44:58.330+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251635 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:44:58.377 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:44:58.377 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:44:58.380 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:44:58.380 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:45:00.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:00.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:00.285441+0000) 2019-10-13T16:45:00.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:00.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:00.534776+0000) 2019-10-13T16:45:02.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:02.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:02.801487+0000) 2019-10-13T16:45:03.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:03.334+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.337163+0000) 2019-10-13T16:45:03.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:03.333+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.337457+0000) 2019-10-13T16:45:03.340 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:03.334+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.338185+0000) 2019-10-13T16:45:03.340 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:03.334+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.338329+0000) 2019-10-13T16:45:03.341 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:03.334+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.338879+0000) 2019-10-13T16:45:03.341 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:03.338+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.339530+0000) 2019-10-13T16:45:03.342 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:03.338+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.340290+0000) 2019-10-13T16:45:03.348 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:03.345+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:03.346370+0000) 2019-10-13T16:45:03.352 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:03.346+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:05.815 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:05.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:05.813376+0000) 2019-10-13T16:45:05.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:05.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:05.813377+0000) 2019-10-13T16:45:05.823 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:05.818+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:05.821868+0000) 2019-10-13T16:45:06.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:06.362+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:06.364376+0000) 2019-10-13T16:45:07.953 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:07.949+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:07.950962+0000) 2019-10-13T16:45:08.355 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:08.350+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.353053+0000) 2019-10-13T16:45:08.356 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:08.350+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.353964+0000) 2019-10-13T16:45:08.356 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:08.353+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.354183+0000) 2019-10-13T16:45:08.357 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:08.350+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.353703+0000) 2019-10-13T16:45:08.357 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:08.354+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.355463+0000) 2019-10-13T16:45:08.357 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:08.354+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.355923+0000) 2019-10-13T16:45:08.358 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:08.354+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.356130+0000) 2019-10-13T16:45:08.364 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:08.361+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:08.362805+0000) 2019-10-13T16:45:08.368 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:08.362+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:10.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:10.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:10.285641+0000) 2019-10-13T16:45:10.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:10.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:10.534968+0000) 2019-10-13T16:45:12.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:12.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:12.801713+0000) 2019-10-13T16:45:13.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:13.366+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.369271+0000) 2019-10-13T16:45:13.371 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:13.365+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.369649+0000) 2019-10-13T16:45:13.372 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:13.366+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.370109+0000) 2019-10-13T16:45:13.372 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:13.366+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.370702+0000) 2019-10-13T16:45:13.373 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:13.366+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.371040+0000) 2019-10-13T16:45:13.373 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:13.370+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.371365+0000) 2019-10-13T16:45:13.374 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:13.370+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.372493+0000) 2019-10-13T16:45:13.381 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:13.377+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:13.379026+0000) 2019-10-13T16:45:13.383 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:13.378+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:15.815 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:15.810+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:15.813576+0000) 2019-10-13T16:45:15.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:15.810+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:15.813577+0000) 2019-10-13T16:45:15.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:15.818+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:15.822096+0000) 2019-10-13T16:45:16.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:16.362+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:16.364606+0000) 2019-10-13T16:45:17.959 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:17.953+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:17.957424+0000) 2019-10-13T16:45:18.386 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:18.381+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.384625+0000) 2019-10-13T16:45:18.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:18.382+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.385790+0000) 2019-10-13T16:45:18.388 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:18.382+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.385858+0000) 2019-10-13T16:45:18.388 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:18.382+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.386233+0000) 2019-10-13T16:45:18.388 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:18.382+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.386606+0000) 2019-10-13T16:45:18.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:18.382+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.387257+0000) 2019-10-13T16:45:18.390 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:18.386+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.388198+0000) 2019-10-13T16:45:18.397 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:18.393+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:18.395238+0000) 2019-10-13T16:45:18.400 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:18.394+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251773 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:20.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:20.282+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:20.285884+0000) 2019-10-13T16:45:20.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:20.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:20.535166+0000) 2019-10-13T16:45:22.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:22.798+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:22.801915+0000) 2019-10-13T16:45:24.058 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:23.397+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.401516+0000) 2019-10-13T16:45:24.058 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:23.405+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.409275+0000) 2019-10-13T16:45:24.058 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:23.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.401073+0000) 2019-10-13T16:45:24.058 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:23.398+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.402380+0000) 2019-10-13T16:45:24.058 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:23.398+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.402879+0000) 2019-10-13T16:45:24.059 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:23.402+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.403674+0000) 2019-10-13T16:45:24.059 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:23.402+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.404065+0000) 2019-10-13T16:45:24.059 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:23.402+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:23.404463+0000) 2019-10-13T16:45:24.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:23.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251804 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:25.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:25.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:25.813813+0000) 2019-10-13T16:45:25.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:25.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:25.813819+0000) 2019-10-13T16:45:25.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:25.817+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:25.822307+0000) 2019-10-13T16:45:26.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:26.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:26.364805+0000) 2019-10-13T16:45:27.966 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:27.961+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:27.963961+0000) 2019-10-13T16:45:28.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:28.413+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.417781+0000) 2019-10-13T16:45:28.420 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:28.417+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.418376+0000) 2019-10-13T16:45:28.421 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:28.413+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.418225+0000) 2019-10-13T16:45:28.421 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:28.413+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.418986+0000) 2019-10-13T16:45:28.421 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:28.417+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.419757+0000) 2019-10-13T16:45:28.422 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:28.417+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.420588+0000) 2019-10-13T16:45:28.422 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:28.417+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.420849+0000) 2019-10-13T16:45:28.428 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:28.425+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:28.426845+0000) 2019-10-13T16:45:28.433 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:28.425+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:28.613 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:45:28.613 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:45:28.616 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:45:28.617 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:45:30.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:30.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:30.286074+0000) 2019-10-13T16:45:30.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:30.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:30.535393+0000) 2019-10-13T16:45:32.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:32.797+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:32.802110+0000) 2019-10-13T16:45:33.436 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:33.429+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.433988+0000) 2019-10-13T16:45:33.436 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:33.429+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.434691+0000) 2019-10-13T16:45:33.437 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:33.433+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.435457+0000) 2019-10-13T16:45:33.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:33.429+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.435153+0000) 2019-10-13T16:45:33.438 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:33.433+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.436158+0000) 2019-10-13T16:45:33.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:33.433+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.436253+0000) 2019-10-13T16:45:33.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:33.433+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.437555+0000) 2019-10-13T16:45:33.447 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:33.441+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:33.445633+0000) 2019-10-13T16:45:33.448 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:33.441+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:35.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:35.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:35.814039+0000) 2019-10-13T16:45:35.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:35.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:35.814040+0000) 2019-10-13T16:45:35.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:35.821+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:35.822535+0000) 2019-10-13T16:45:36.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:36.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:36.365003+0000) 2019-10-13T16:45:37.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:37.969+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:37.970448+0000) 2019-10-13T16:45:38.451 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:38.445+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.449603+0000) 2019-10-13T16:45:38.452 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:38.445+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.450373+0000) 2019-10-13T16:45:38.453 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:38.445+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.450946+0000) 2019-10-13T16:45:38.453 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:38.449+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.451736+0000) 2019-10-13T16:45:38.454 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:38.449+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.452427+0000) 2019-10-13T16:45:38.454 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:38.449+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.452848+0000) 2019-10-13T16:45:38.455 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:38.449+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.453438+0000) 2019-10-13T16:45:38.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:38.457+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:38.458466+0000) 2019-10-13T16:45:38.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:38.457+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251910 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:40.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:40.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:40.286272+0000) 2019-10-13T16:45:40.537 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:40.533+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:40.535585+0000) 2019-10-13T16:45:42.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:42.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:42.802310+0000) 2019-10-13T16:45:43.467 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:43.461+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.465118+0000) 2019-10-13T16:45:43.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:43.461+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.465870+0000) 2019-10-13T16:45:43.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:43.461+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.466276+0000) 2019-10-13T16:45:43.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:43.461+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.466943+0000) 2019-10-13T16:45:43.469 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:43.465+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.467017+0000) 2019-10-13T16:45:43.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:43.465+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.468375+0000) 2019-10-13T16:45:43.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:43.465+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.468884+0000) 2019-10-13T16:45:43.476 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:43.469+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:43.474096+0000) 2019-10-13T16:45:43.480 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:43.473+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251942 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:45.816 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:45.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:45.814286+0000) 2019-10-13T16:45:45.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:45.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:45.814288+0000) 2019-10-13T16:45:45.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:45.817+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:45.822754+0000) 2019-10-13T16:45:46.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:46.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:46.365201+0000) 2019-10-13T16:45:47.979 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:47.972+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:47.976891+0000) 2019-10-13T16:45:48.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:48.477+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.481393+0000) 2019-10-13T16:45:48.486 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:48.476+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.481647+0000) 2019-10-13T16:45:48.487 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:48.477+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.482703+0000) 2019-10-13T16:45:48.487 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:48.481+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.483335+0000) 2019-10-13T16:45:48.487 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:48.481+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.483655+0000) 2019-10-13T16:45:48.488 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:48.481+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.484889+0000) 2019-10-13T16:45:48.488 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:48.481+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.485221+0000) 2019-10-13T16:45:48.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:48.484+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:48.489124+0000) 2019-10-13T16:45:48.497 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:48.489+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 251978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:50.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:50.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:50.286496+0000) 2019-10-13T16:45:50.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:50.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:50.535804+0000) 2019-10-13T16:45:52.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:52.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:52.802512+0000) 2019-10-13T16:45:53.500 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:53.493+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.497425+0000) 2019-10-13T16:45:53.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:53.493+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.497973+0000) 2019-10-13T16:45:53.501 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:53.496+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.498370+0000) 2019-10-13T16:45:53.502 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:53.493+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.498842+0000) 2019-10-13T16:45:53.502 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:53.493+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.499138+0000) 2019-10-13T16:45:53.502 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:53.497+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.499537+0000) 2019-10-13T16:45:53.503 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:53.497+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.500133+0000) 2019-10-13T16:45:53.508 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:53.504+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:53.506329+0000) 2019-10-13T16:45:53.512 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:53.505+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:55.816 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:55.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:55.814491+0000) 2019-10-13T16:45:55.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:55.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:55.814479+0000) 2019-10-13T16:45:55.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:55.821+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:55.822948+0000) 2019-10-13T16:45:56.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:56.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:56.365440+0000) 2019-10-13T16:45:57.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:57.980+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:57.983668+0000) 2019-10-13T16:45:58.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:45:58.509+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.513683+0000) 2019-10-13T16:45:58.516 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:45:58.509+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.514386+0000) 2019-10-13T16:45:58.516 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:45:58.512+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.514734+0000) 2019-10-13T16:45:58.517 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:45:58.513+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.515873+0000) 2019-10-13T16:45:58.518 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:45:58.513+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.515969+0000) 2019-10-13T16:45:58.518 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:45:58.513+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.516247+0000) 2019-10-13T16:45:58.518 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:45:58.513+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.516897+0000) 2019-10-13T16:45:58.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:45:58.520+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:45:58.524049+0000) 2019-10-13T16:45:58.528 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:45:58.521+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252048 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:45:58.665 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:45:58.665 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:45:58.669 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:45:58.669 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:46:00.288 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:00.281+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:00.286716+0000) 2019-10-13T16:46:00.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:00.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:00.536028+0000) 2019-10-13T16:46:02.804 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:02.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:02.802749+0000) 2019-10-13T16:46:03.532 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:03.525+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.530215+0000) 2019-10-13T16:46:03.532 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:03.525+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.530655+0000) 2019-10-13T16:46:03.533 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:03.528+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.531262+0000) 2019-10-13T16:46:03.535 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:03.529+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.531705+0000) 2019-10-13T16:46:03.535 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:03.529+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.532247+0000) 2019-10-13T16:46:03.535 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:03.529+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.532283+0000) 2019-10-13T16:46:03.535 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:03.529+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.533503+0000) 2019-10-13T16:46:03.539 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:03.532+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:03.537211+0000) 2019-10-13T16:46:03.544 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:03.537+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252080 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:05.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:05.809+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:05.814709+0000) 2019-10-13T16:46:05.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:05.809+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:05.814712+0000) 2019-10-13T16:46:05.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:05.821+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:05.823144+0000) 2019-10-13T16:46:06.367 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:06.361+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:06.365639+0000) 2019-10-13T16:46:07.992 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:07.988+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:07.990195+0000) 2019-10-13T16:46:08.547 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:08.540+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.545230+0000) 2019-10-13T16:46:08.548 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:08.540+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.545949+0000) 2019-10-13T16:46:08.548 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:08.540+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.547051+0000) 2019-10-13T16:46:08.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:08.544+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.547361+0000) 2019-10-13T16:46:08.550 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:08.544+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.548436+0000) 2019-10-13T16:46:08.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:08.544+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.548953+0000) 2019-10-13T16:46:08.551 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:08.544+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.549419+0000) 2019-10-13T16:46:08.557 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:08.552+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:08.555128+0000) 2019-10-13T16:46:08.560 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:08.552+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:10.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:10.280+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:10.286917+0000) 2019-10-13T16:46:10.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:10.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:10.536200+0000) 2019-10-13T16:46:12.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:12.796+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:12.802951+0000) 2019-10-13T16:46:13.563 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:13.556+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.561402+0000) 2019-10-13T16:46:13.564 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:13.556+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.562445+0000) 2019-10-13T16:46:13.565 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:13.556+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.563076+0000) 2019-10-13T16:46:13.565 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:13.560+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.563997+0000) 2019-10-13T16:46:13.566 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:13.560+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.564250+0000) 2019-10-13T16:46:13.566 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:13.560+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.565033+0000) 2019-10-13T16:46:13.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:13.560+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.565436+0000) 2019-10-13T16:46:13.571 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:13.564+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:13.569754+0000) 2019-10-13T16:46:13.576 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:13.568+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:15.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:15.808+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:15.814943+0000) 2019-10-13T16:46:15.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:15.808+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:15.814912+0000) 2019-10-13T16:46:15.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:15.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:15.823341+0000) 2019-10-13T16:46:16.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:16.360+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:16.365833+0000) 2019-10-13T16:46:17.998 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:17.996+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:17.996553+0000) 2019-10-13T16:46:18.580 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:18.572+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.578013+0000) 2019-10-13T16:46:18.581 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:18.576+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.579468+0000) 2019-10-13T16:46:18.582 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:18.576+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.580165+0000) 2019-10-13T16:46:18.582 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:18.576+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.580547+0000) 2019-10-13T16:46:18.583 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:18.576+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.581452+0000) 2019-10-13T16:46:18.584 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:18.576+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.581633+0000) 2019-10-13T16:46:18.584 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:18.576+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.582003+0000) 2019-10-13T16:46:18.589 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:18.584+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:18.587241+0000) 2019-10-13T16:46:18.592 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:18.584+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252186 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:20.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:20.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:20.287109+0000) 2019-10-13T16:46:20.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:20.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:20.536396+0000) 2019-10-13T16:46:22.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:22.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:22.803162+0000) 2019-10-13T16:46:23.596 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:23.592+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.594327+0000) 2019-10-13T16:46:23.597 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:23.588+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.594745+0000) 2019-10-13T16:46:23.597 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:23.592+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.595472+0000) 2019-10-13T16:46:23.598 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:23.592+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.595774+0000) 2019-10-13T16:46:23.598 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:23.592+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.596311+0000) 2019-10-13T16:46:23.599 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:23.592+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.597310+0000) 2019-10-13T16:46:23.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:23.592+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.598003+0000) 2019-10-13T16:46:23.607 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:23.600+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:23.604920+0000) 2019-10-13T16:46:23.608 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:23.600+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252218 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:25.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:25.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:25.815146+0000) 2019-10-13T16:46:25.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:25.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:25.815148+0000) 2019-10-13T16:46:25.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:25.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:25.823577+0000) 2019-10-13T16:46:26.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:26.364+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:26.366072+0000) 2019-10-13T16:46:28.005 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:28.000+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.003236+0000) 2019-10-13T16:46:28.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:28.604+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.609821+0000) 2019-10-13T16:46:28.612 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:28.604+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.610060+0000) 2019-10-13T16:46:28.612 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:28.604+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.610753+0000) 2019-10-13T16:46:28.613 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:28.608+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.611484+0000) 2019-10-13T16:46:28.614 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:28.608+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.611802+0000) 2019-10-13T16:46:28.614 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:28.608+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.612309+0000) 2019-10-13T16:46:28.615 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:28.608+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.613307+0000) 2019-10-13T16:46:28.624 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:28.616+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:28.624 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:28.616+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:28.622129+0000) 2019-10-13T16:46:28.819 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:46:28.820 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:46:28.823 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:46:28.824 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:46:30.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:30.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:30.287335+0000) 2019-10-13T16:46:30.538 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:30.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:30.536607+0000) 2019-10-13T16:46:32.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:32.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:32.803409+0000) 2019-10-13T16:46:33.627 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:33.620+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.625412+0000) 2019-10-13T16:46:33.628 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:33.620+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.625952+0000) 2019-10-13T16:46:33.628 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:33.620+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.626299+0000) 2019-10-13T16:46:33.629 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:33.620+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.626760+0000) 2019-10-13T16:46:33.629 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:33.620+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.627031+0000) 2019-10-13T16:46:33.630 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:33.624+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.627801+0000) 2019-10-13T16:46:33.630 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:33.624+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.628262+0000) 2019-10-13T16:46:33.635 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:33.628+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:33.633800+0000) 2019-10-13T16:46:33.639 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:33.632+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:35.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:35.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:35.815365+0000) 2019-10-13T16:46:35.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:35.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:35.815365+0000) 2019-10-13T16:46:35.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:35.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:35.823776+0000) 2019-10-13T16:46:36.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:36.364+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:36.366253+0000) 2019-10-13T16:46:38.012 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:38.008+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.009773+0000) 2019-10-13T16:46:38.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:38.636+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.640879+0000) 2019-10-13T16:46:38.644 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:38.636+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.642209+0000) 2019-10-13T16:46:38.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:38.636+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.642495+0000) 2019-10-13T16:46:38.645 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:38.636+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.642868+0000) 2019-10-13T16:46:38.645 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:38.640+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.643912+0000) 2019-10-13T16:46:38.646 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:38.640+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.644200+0000) 2019-10-13T16:46:38.646 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:38.640+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.644813+0000) 2019-10-13T16:46:38.652 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:38.644+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:38.650080+0000) 2019-10-13T16:46:38.656 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:38.648+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252323 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:40.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:40.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:40.287523+0000) 2019-10-13T16:46:40.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:40.532+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:40.536799+0000) 2019-10-13T16:46:42.805 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:42.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:42.803640+0000) 2019-10-13T16:46:43.659 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:43.652+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.657024+0000) 2019-10-13T16:46:43.659 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:43.652+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.658026+0000) 2019-10-13T16:46:43.660 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:43.652+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.658759+0000) 2019-10-13T16:46:43.661 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:43.655+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.659061+0000) 2019-10-13T16:46:43.661 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:43.656+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.659928+0000) 2019-10-13T16:46:43.662 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:43.656+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.660340+0000) 2019-10-13T16:46:43.662 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:43.656+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.660403+0000) 2019-10-13T16:46:43.667 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:43.659+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:43.665683+0000) 2019-10-13T16:46:43.676 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:43.664+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252356 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:45.817 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:45.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:45.815603+0000) 2019-10-13T16:46:45.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:45.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:45.815597+0000) 2019-10-13T16:46:45.825 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:45.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:45.823975+0000) 2019-10-13T16:46:46.368 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:46.360+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:46.366481+0000) 2019-10-13T16:46:48.018 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:48.015+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.016361+0000) 2019-10-13T16:46:48.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:48.668+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.671601+0000) 2019-10-13T16:46:48.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:48.667+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.672176+0000) 2019-10-13T16:46:48.675 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:48.668+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.673762+0000) 2019-10-13T16:46:48.676 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:48.668+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.674327+0000) 2019-10-13T16:46:48.676 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:48.668+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.674607+0000) 2019-10-13T16:46:48.677 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:48.672+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.675478+0000) 2019-10-13T16:46:48.677 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:48.672+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.675916+0000) 2019-10-13T16:46:48.681 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:48.675+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:48.679132+0000) 2019-10-13T16:46:48.687 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:48.680+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:50.289 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:50.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:50.287718+0000) 2019-10-13T16:46:50.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:50.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:50.536967+0000) 2019-10-13T16:46:52.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:52.799+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:52.803903+0000) 2019-10-13T16:46:53.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:53.683+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.688425+0000) 2019-10-13T16:46:53.691 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:53.683+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.688903+0000) 2019-10-13T16:46:53.691 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:53.683+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.689609+0000) 2019-10-13T16:46:53.692 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:53.683+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.689971+0000) 2019-10-13T16:46:53.692 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:53.683+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.690258+0000) 2019-10-13T16:46:53.693 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:53.687+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.691702+0000) 2019-10-13T16:46:53.693 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:53.687+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.691690+0000) 2019-10-13T16:46:53.698 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:53.691+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:53.696004+0000) 2019-10-13T16:46:53.702 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:53.695+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:55.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:55.811+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:55.815802+0000) 2019-10-13T16:46:55.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:55.811+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:55.815805+0000) 2019-10-13T16:46:55.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:55.823+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:55.824230+0000) 2019-10-13T16:46:56.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:56.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:56.366733+0000) 2019-10-13T16:46:58.025 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:58.019+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.022929+0000) 2019-10-13T16:46:58.705 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:46:58.699+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.703643+0000) 2019-10-13T16:46:58.706 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:46:58.699+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.704460+0000) 2019-10-13T16:46:58.707 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:46:58.699+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.705924+0000) 2019-10-13T16:46:58.708 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:46:58.699+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.706096+0000) 2019-10-13T16:46:58.708 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:46:58.699+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.706928+0000) 2019-10-13T16:46:58.709 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:46:58.699+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.707214+0000) 2019-10-13T16:46:58.709 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:46:58.703+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.707728+0000) 2019-10-13T16:46:58.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:46:58.707+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:46:58.712842+0000) 2019-10-13T16:46:58.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:46:58.711+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252461 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:46:59.025 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:46:59.025 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:46:59.028 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:46:59.028 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:47:00.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:00.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:00.287920+0000) 2019-10-13T16:47:00.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:00.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:00.537158+0000) 2019-10-13T16:47:02.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:02.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:02.804120+0000) 2019-10-13T16:47:03.722 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:03.715+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.719994+0000) 2019-10-13T16:47:03.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:03.715+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.720258+0000) 2019-10-13T16:47:03.723 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:03.715+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.720638+0000) 2019-10-13T16:47:03.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:03.715+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.721189+0000) 2019-10-13T16:47:03.724 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:03.715+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.722621+0000) 2019-10-13T16:47:03.725 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:03.719+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.723350+0000) 2019-10-13T16:47:03.725 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:03.719+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.724034+0000) 2019-10-13T16:47:03.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:03.723+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:03.728469+0000) 2019-10-13T16:47:03.734 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:03.727+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252494 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:05.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:05.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:05.816025+0000) 2019-10-13T16:47:05.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:05.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:05.816024+0000) 2019-10-13T16:47:05.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:05.823+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:05.824447+0000) 2019-10-13T16:47:06.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:06.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:06.366938+0000) 2019-10-13T16:47:08.036 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:08.031+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.034179+0000) 2019-10-13T16:47:08.737 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:08.731+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.735050+0000) 2019-10-13T16:47:08.737 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:08.731+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.735547+0000) 2019-10-13T16:47:08.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:08.731+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.736963+0000) 2019-10-13T16:47:08.739 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:08.731+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.737100+0000) 2019-10-13T16:47:08.739 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:08.731+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.738112+0000) 2019-10-13T16:47:08.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:08.731+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.738318+0000) 2019-10-13T16:47:08.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:08.731+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.739053+0000) 2019-10-13T16:47:08.749 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:08.743+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:08.747276+0000) 2019-10-13T16:47:08.750 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:08.743+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:10.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:10.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:10.288146+0000) 2019-10-13T16:47:10.539 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:10.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:10.537352+0000) 2019-10-13T16:47:12.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:12.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:12.804347+0000) 2019-10-13T16:47:13.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:13.747+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.752224+0000) 2019-10-13T16:47:13.755 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:13.747+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.752490+0000) 2019-10-13T16:47:13.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:13.747+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.753856+0000) 2019-10-13T16:47:13.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:13.747+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.754043+0000) 2019-10-13T16:47:13.756 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:13.747+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.754418+0000) 2019-10-13T16:47:13.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:13.747+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.754829+0000) 2019-10-13T16:47:13.757 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:13.747+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.755219+0000) 2019-10-13T16:47:13.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:13.755+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:13.761119+0000) 2019-10-13T16:47:13.766 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:13.759+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:15.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:15.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:15.816226+0000) 2019-10-13T16:47:15.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:15.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:15.816228+0000) 2019-10-13T16:47:15.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:15.823+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:15.824665+0000) 2019-10-13T16:47:16.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:16.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:16.367143+0000) 2019-10-13T16:47:18.042 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:18.035+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.040697+0000) 2019-10-13T16:47:18.771 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:18.763+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.768851+0000) 2019-10-13T16:47:18.771 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:18.763+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.769090+0000) 2019-10-13T16:47:18.772 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:18.763+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.769512+0000) 2019-10-13T16:47:18.772 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:18.763+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.769319+0000) 2019-10-13T16:47:18.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:18.763+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.769940+0000) 2019-10-13T16:47:18.773 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:18.763+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.770336+0000) 2019-10-13T16:47:18.773 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:18.763+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.770734+0000) 2019-10-13T16:47:18.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:18.771+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:18.774654+0000) 2019-10-13T16:47:18.783 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:18.775+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252599 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:20.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:20.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:20.288343+0000) 2019-10-13T16:47:20.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:20.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:20.537543+0000) 2019-10-13T16:47:22.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:22.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:22.804510+0000) 2019-10-13T16:47:23.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:23.783+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.784950+0000) 2019-10-13T16:47:23.787 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:23.783+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.785790+0000) 2019-10-13T16:47:23.788 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:23.783+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.786463+0000) 2019-10-13T16:47:23.788 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:23.783+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.786333+0000) 2019-10-13T16:47:23.789 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:23.783+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.787059+0000) 2019-10-13T16:47:23.789 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:23.787+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.787433+0000) 2019-10-13T16:47:23.790 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:23.787+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.787822+0000) 2019-10-13T16:47:23.795 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:23.787+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:23.793203+0000) 2019-10-13T16:47:23.799 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:23.795+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:25.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:25.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:25.816439+0000) 2019-10-13T16:47:25.818 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:25.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:25.816439+0000) 2019-10-13T16:47:25.826 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:25.823+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:25.824858+0000) 2019-10-13T16:47:26.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:26.367+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:26.367354+0000) 2019-10-13T16:47:28.050 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:28.047+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.047309+0000) 2019-10-13T16:47:28.803 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:28.799+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.800802+0000) 2019-10-13T16:47:28.804 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:28.795+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.801459+0000) 2019-10-13T16:47:28.804 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:28.799+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.801831+0000) 2019-10-13T16:47:28.804 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:28.799+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.802305+0000) 2019-10-13T16:47:28.806 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:28.799+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.802876+0000) 2019-10-13T16:47:28.806 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:28.799+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.802948+0000) 2019-10-13T16:47:28.806 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:28.799+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.803253+0000) 2019-10-13T16:47:28.812 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:28.807+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:28.810338+0000) 2019-10-13T16:47:28.816 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:28.811+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:29.136 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:47:29.136 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:47:29.140 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:47:29.141 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:47:30.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:30.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:30.288534+0000) 2019-10-13T16:47:30.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:30.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:30.537780+0000) 2019-10-13T16:47:32.806 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:32.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:32.804751+0000) 2019-10-13T16:47:33.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:33.815+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.818574+0000) 2019-10-13T16:47:33.821 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:33.815+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.819006+0000) 2019-10-13T16:47:33.824 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:33.819+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.819846+0000) 2019-10-13T16:47:33.824 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:33.819+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.821058+0000) 2019-10-13T16:47:33.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:33.819+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.821125+0000) 2019-10-13T16:47:33.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:33.819+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.821527+0000) 2019-10-13T16:47:33.825 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:33.815+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.819366+0000) 2019-10-13T16:47:33.828 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:33.823+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:33.826747+0000) 2019-10-13T16:47:33.832 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:33.827+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252701 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:35.818 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:35.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:35.816645+0000) 2019-10-13T16:47:35.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:35.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:35.816644+0000) 2019-10-13T16:47:35.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:35.822+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:35.825061+0000) 2019-10-13T16:47:36.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:36.366+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:36.367561+0000) 2019-10-13T16:47:38.056 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:38.050+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.053908+0000) 2019-10-13T16:47:38.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:38.830+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.834036+0000) 2019-10-13T16:47:38.836 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:38.830+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.834729+0000) 2019-10-13T16:47:38.837 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:38.830+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.834870+0000) 2019-10-13T16:47:38.838 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:38.834+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.835694+0000) 2019-10-13T16:47:38.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:38.834+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.836575+0000) 2019-10-13T16:47:38.839 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:38.834+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.837083+0000) 2019-10-13T16:47:38.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:38.834+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.837282+0000) 2019-10-13T16:47:38.844 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:38.838+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:38.842317+0000) 2019-10-13T16:47:38.848 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:38.842+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252737 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:40.290 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:40.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:40.288731+0000) 2019-10-13T16:47:40.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:40.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:40.538009+0000) 2019-10-13T16:47:42.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:42.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:42.804953+0000) 2019-10-13T16:47:43.852 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:43.846+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.850339+0000) 2019-10-13T16:47:43.853 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:43.846+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.850952+0000) 2019-10-13T16:47:43.853 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:43.846+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.850953+0000) 2019-10-13T16:47:43.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:43.850+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.851907+0000) 2019-10-13T16:47:43.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:43.850+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.851904+0000) 2019-10-13T16:47:43.854 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:43.850+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.852587+0000) 2019-10-13T16:47:43.855 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:43.850+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.852827+0000) 2019-10-13T16:47:43.865 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:43.858+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252770 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:43.865 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:43.854+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:43.858631+0000) 2019-10-13T16:47:45.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:45.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:45.816865+0000) 2019-10-13T16:47:45.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:45.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:45.816859+0000) 2019-10-13T16:47:45.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:45.822+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:45.825315+0000) 2019-10-13T16:47:46.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:46.366+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:46.367760+0000) 2019-10-13T16:47:48.062 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:48.058+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.060424+0000) 2019-10-13T16:47:48.868 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:48.862+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.866423+0000) 2019-10-13T16:47:48.869 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:48.862+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.866899+0000) 2019-10-13T16:47:48.869 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:48.866+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.867465+0000) 2019-10-13T16:47:48.870 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:48.866+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.868222+0000) 2019-10-13T16:47:48.870 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:48.866+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.868410+0000) 2019-10-13T16:47:48.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:48.866+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.869128+0000) 2019-10-13T16:47:48.871 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:48.866+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.869337+0000) 2019-10-13T16:47:48.876 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:48.870+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:48.874897+0000) 2019-10-13T16:47:48.881 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:48.874+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:50.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:50.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:50.288926+0000) 2019-10-13T16:47:50.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:50.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:50.538205+0000) 2019-10-13T16:47:52.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:52.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:52.805153+0000) 2019-10-13T16:47:53.885 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:53.878+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.882942+0000) 2019-10-13T16:47:53.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:53.878+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.883252+0000) 2019-10-13T16:47:53.886 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:53.882+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.883694+0000) 2019-10-13T16:47:53.886 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:53.878+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.884312+0000) 2019-10-13T16:47:53.887 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:53.882+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.884593+0000) 2019-10-13T16:47:53.887 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:53.882+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.885475+0000) 2019-10-13T16:47:53.887 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:53.882+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.885991+0000) 2019-10-13T16:47:53.893 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:53.886+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:53.891302+0000) 2019-10-13T16:47:53.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:53.890+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252839 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:55.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:55.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:55.817103+0000) 2019-10-13T16:47:55.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:55.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:55.817099+0000) 2019-10-13T16:47:55.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:55.822+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:55.825553+0000) 2019-10-13T16:47:56.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:56.366+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:56.367964+0000) 2019-10-13T16:47:58.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:58.062+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.066940+0000) 2019-10-13T16:47:58.899 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:47:58.890+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.896950+0000) 2019-10-13T16:47:58.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:47:58.894+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.897441+0000) 2019-10-13T16:47:58.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:47:58.894+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.898191+0000) 2019-10-13T16:47:58.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:47:58.894+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.898770+0000) 2019-10-13T16:47:58.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:47:58.894+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.898985+0000) 2019-10-13T16:47:58.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:47:58.898+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.900223+0000) 2019-10-13T16:47:58.903 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:47:58.898+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.900959+0000) 2019-10-13T16:47:58.909 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:47:58.902+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:47:58.907678+0000) 2019-10-13T16:47:58.912 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:47:58.906+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252875 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:47:59.249 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:47:59.249 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:47:59.253 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:47:59.253 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:48:00.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:00.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:00.289125+0000) 2019-10-13T16:48:00.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:00.534+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:00.538420+0000) 2019-10-13T16:48:02.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:02.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:02.805367+0000) 2019-10-13T16:48:03.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:03.910+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.913738+0000) 2019-10-13T16:48:03.916 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:03.910+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.914259+0000) 2019-10-13T16:48:03.916 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:03.910+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.914975+0000) 2019-10-13T16:48:03.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:03.910+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.915238+0000) 2019-10-13T16:48:03.917 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:03.910+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.915046+0000) 2019-10-13T16:48:03.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:03.914+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.916419+0000) 2019-10-13T16:48:03.918 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:03.914+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.916884+0000) 2019-10-13T16:48:03.924 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:03.918+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:03.922204+0000) 2019-10-13T16:48:03.928 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:03.922+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252907 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:05.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:05.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:05.817301+0000) 2019-10-13T16:48:05.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:05.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:05.817306+0000) 2019-10-13T16:48:05.827 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:05.822+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:05.825808+0000) 2019-10-13T16:48:06.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:06.366+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:06.368114+0000) 2019-10-13T16:48:08.075 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:08.070+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.073321+0000) 2019-10-13T16:48:08.932 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:08.926+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.930218+0000) 2019-10-13T16:48:08.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:08.926+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.930855+0000) 2019-10-13T16:48:08.933 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:08.930+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.931852+0000) 2019-10-13T16:48:08.934 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:08.930+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.932609+0000) 2019-10-13T16:48:08.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:08.930+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.932988+0000) 2019-10-13T16:48:08.935 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:08.926+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.933781+0000) 2019-10-13T16:48:08.936 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:08.930+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.934339+0000) 2019-10-13T16:48:08.939 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:08.930+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:08.937499+0000) 2019-10-13T16:48:08.944 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:08.938+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:10.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:10.286+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:10.289275+0000) 2019-10-13T16:48:10.540 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:10.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:10.538646+0000) 2019-10-13T16:48:12.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:12.802+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:12.805569+0000) 2019-10-13T16:48:13.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:13.942+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.945827+0000) 2019-10-13T16:48:13.948 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:13.942+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.946568+0000) 2019-10-13T16:48:13.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:13.942+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.947191+0000) 2019-10-13T16:48:13.949 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:13.946+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.947671+0000) 2019-10-13T16:48:13.950 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:13.946+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.948068+0000) 2019-10-13T16:48:13.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:13.946+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.948502+0000) 2019-10-13T16:48:13.951 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:13.946+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.949701+0000) 2019-10-13T16:48:13.956 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:13.946+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:13.954109+0000) 2019-10-13T16:48:13.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:13.954+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 252976 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:15.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:15.814+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:15.817531+0000) 2019-10-13T16:48:15.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:15.814+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:15.817506+0000) 2019-10-13T16:48:15.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:15.822+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:15.826023+0000) 2019-10-13T16:48:16.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:16.366+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:16.368309+0000) 2019-10-13T16:48:18.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:18.074+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.079794+0000) 2019-10-13T16:48:18.965 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:18.961+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.963343+0000) 2019-10-13T16:48:18.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:18.961+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.963835+0000) 2019-10-13T16:48:18.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:18.961+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.964022+0000) 2019-10-13T16:48:18.966 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:18.961+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.964473+0000) 2019-10-13T16:48:18.967 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:18.958+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.965005+0000) 2019-10-13T16:48:18.967 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:18.961+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.965338+0000) 2019-10-13T16:48:18.968 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:18.961+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.966319+0000) 2019-10-13T16:48:18.970 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:18.962+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:18.968825+0000) 2019-10-13T16:48:18.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:18.969+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253013 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:20.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:20.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:20.289464+0000) 2019-10-13T16:48:20.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:20.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:20.538838+0000) 2019-10-13T16:48:22.807 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:22.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:22.805777+0000) 2019-10-13T16:48:23.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:23.974+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.978175+0000) 2019-10-13T16:48:23.980 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:23.973+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.978860+0000) 2019-10-13T16:48:23.981 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:23.977+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.979379+0000) 2019-10-13T16:48:23.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:23.977+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.980068+0000) 2019-10-13T16:48:23.982 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:23.977+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.980291+0000) 2019-10-13T16:48:23.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:23.977+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.980911+0000) 2019-10-13T16:48:23.983 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:23.977+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.981034+0000) 2019-10-13T16:48:23.987 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:23.978+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:23.985779+0000) 2019-10-13T16:48:23.994 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:23.989+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253045 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:25.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:25.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:25.817718+0000) 2019-10-13T16:48:25.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:25.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:25.817725+0000) 2019-10-13T16:48:25.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:25.821+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:25.826263+0000) 2019-10-13T16:48:26.370 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:26.365+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:26.368529+0000) 2019-10-13T16:48:28.088 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:28.086+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.086183+0000) 2019-10-13T16:48:28.998 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:28.993+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.995825+0000) 2019-10-13T16:48:28.998 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:28.993+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.996401+0000) 2019-10-13T16:48:28.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:28.993+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.996936+0000) 2019-10-13T16:48:28.999 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:28.993+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.997560+0000) 2019-10-13T16:48:28.999 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:28.994+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.997742+0000) 2019-10-13T16:48:29.000 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:28.993+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.998233+0000) 2019-10-13T16:48:29.000 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:28.993+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:28.999063+0000) 2019-10-13T16:48:29.007 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:29.002+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:29.005645+0000) 2019-10-13T16:48:29.010 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:29.005+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:29.584 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:48:29.584 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:48:29.588 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:48:29.588 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:48:30.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:30.285+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:30.289662+0000) 2019-10-13T16:48:30.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:30.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:30.539055+0000) 2019-10-13T16:48:32.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:32.801+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:32.805929+0000) 2019-10-13T16:48:34.030 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:34.009+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.012451+0000) 2019-10-13T16:48:34.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:34.017+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.021491+0000) 2019-10-13T16:48:34.069 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:34.009+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.011498+0000) 2019-10-13T16:48:34.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:34.009+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.013029+0000) 2019-10-13T16:48:34.070 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:34.009+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.013647+0000) 2019-10-13T16:48:34.070 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:34.009+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.013973+0000) 2019-10-13T16:48:34.070 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:34.009+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.014655+0000) 2019-10-13T16:48:34.070 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:34.009+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:34.014976+0000) 2019-10-13T16:48:34.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:34.021+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253115 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:35.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:35.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:35.817948+0000) 2019-10-13T16:48:35.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:35.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:35.817950+0000) 2019-10-13T16:48:35.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:35.821+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:35.826477+0000) 2019-10-13T16:48:36.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:36.365+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:36.368743+0000) 2019-10-13T16:48:38.095 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:38.089+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:38.092698+0000) 2019-10-13T16:48:39.029 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:39.025+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.027594+0000) 2019-10-13T16:48:39.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:39.025+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.029007+0000) 2019-10-13T16:48:39.031 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:39.025+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.029270+0000) 2019-10-13T16:48:39.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:39.025+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.028734+0000) 2019-10-13T16:48:39.032 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:39.029+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.030354+0000) 2019-10-13T16:48:39.032 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:39.025+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.030847+0000) 2019-10-13T16:48:39.033 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:39.029+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.031462+0000) 2019-10-13T16:48:39.039 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:39.033+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:39.037465+0000) 2019-10-13T16:48:39.043 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:39.037+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253151 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:40.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:40.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:40.289883+0000) 2019-10-13T16:48:40.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:40.537+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:40.539257+0000) 2019-10-13T16:48:42.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:42.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:42.806128+0000) 2019-10-13T16:48:44.046 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:44.041+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.044715+0000) 2019-10-13T16:48:44.047 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:44.041+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.045442+0000) 2019-10-13T16:48:44.048 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:44.041+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.046415+0000) 2019-10-13T16:48:44.048 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:44.045+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.046687+0000) 2019-10-13T16:48:44.049 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:44.041+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.047081+0000) 2019-10-13T16:48:44.049 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:44.045+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.047604+0000) 2019-10-13T16:48:44.050 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:44.045+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.047980+0000) 2019-10-13T16:48:44.052 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:44.049+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:44.050777+0000) 2019-10-13T16:48:44.059 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:44.053+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253184 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:45.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:45.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:45.818202+0000) 2019-10-13T16:48:45.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:45.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:45.818203+0000) 2019-10-13T16:48:45.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:45.825+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:45.826680+0000) 2019-10-13T16:48:46.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:46.365+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:46.368971+0000) 2019-10-13T16:48:48.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:48.097+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:48.099300+0000) 2019-10-13T16:48:49.062 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:49.057+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.060436+0000) 2019-10-13T16:48:49.063 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:49.057+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.061027+0000) 2019-10-13T16:48:49.064 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:49.057+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.062444+0000) 2019-10-13T16:48:49.064 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:49.057+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.063048+0000) 2019-10-13T16:48:49.065 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:49.061+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.063413+0000) 2019-10-13T16:48:49.065 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:49.061+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.064055+0000) 2019-10-13T16:48:49.066 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:49.061+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.064778+0000) 2019-10-13T16:48:49.070 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:49.065+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:49.068519+0000) 2019-10-13T16:48:49.076 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:49.069+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:50.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:50.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:50.290078+0000) 2019-10-13T16:48:50.541 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:50.537+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:50.539454+0000) 2019-10-13T16:48:52.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:52.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:52.806327+0000) 2019-10-13T16:48:54.080 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:54.073+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.078655+0000) 2019-10-13T16:48:54.081 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:54.073+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.078987+0000) 2019-10-13T16:48:54.081 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:54.077+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.079166+0000) 2019-10-13T16:48:54.082 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:54.077+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.080229+0000) 2019-10-13T16:48:54.082 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:54.077+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.080617+0000) 2019-10-13T16:48:54.083 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:54.077+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.081131+0000) 2019-10-13T16:48:54.083 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:54.077+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.081538+0000) 2019-10-13T16:48:54.087 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:54.081+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:54.085770+0000) 2019-10-13T16:48:54.092 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:54.085+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:55.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:55.813+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:55.818425+0000) 2019-10-13T16:48:55.820 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:55.813+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:55.818415+0000) 2019-10-13T16:48:55.828 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:55.825+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:55.826889+0000) 2019-10-13T16:48:56.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:56.365+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:56.369183+0000) 2019-10-13T16:48:58.107 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:58.101+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:58.105648+0000) 2019-10-13T16:48:59.096 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:48:59.089+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.094481+0000) 2019-10-13T16:48:59.097 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:48:59.093+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.095337+0000) 2019-10-13T16:48:59.097 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:48:59.093+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.095420+0000) 2019-10-13T16:48:59.097 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:48:59.093+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.095666+0000) 2019-10-13T16:48:59.098 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:48:59.093+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.096172+0000) 2019-10-13T16:48:59.098 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:48:59.093+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.096673+0000) 2019-10-13T16:48:59.099 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:48:59.093+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.097310+0000) 2019-10-13T16:48:59.102 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:48:59.097+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:48:59.100851+0000) 2019-10-13T16:48:59.108 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:48:59.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253289 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:48:59.712 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:48:59.712 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:48:59.715 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:48:59.716 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:49:00.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:00.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:00.290308+0000) 2019-10-13T16:49:00.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:00.537+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:00.539684+0000) 2019-10-13T16:49:02.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:02.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:02.806527+0000) 2019-10-13T16:49:04.111 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:04.104+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.109437+0000) 2019-10-13T16:49:04.112 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:04.104+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.110800+0000) 2019-10-13T16:49:04.113 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:04.104+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.111225+0000) 2019-10-13T16:49:04.113 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:04.108+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.111763+0000) 2019-10-13T16:49:04.114 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:04.108+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.112216+0000) 2019-10-13T16:49:04.114 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:04.109+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.112658+0000) 2019-10-13T16:49:04.115 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:04.108+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.113283+0000) 2019-10-13T16:49:04.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:04.113+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:04.118010+0000) 2019-10-13T16:49:04.124 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:04.116+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253321 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:05.820 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:05.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:05.818621+0000) 2019-10-13T16:49:05.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:05.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:05.818620+0000) 2019-10-13T16:49:05.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:05.820+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:05.827113+0000) 2019-10-13T16:49:06.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:06.368+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:06.369428+0000) 2019-10-13T16:49:08.114 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:08.109+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:08.112014+0000) 2019-10-13T16:49:09.130 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:09.125+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.126543+0000) 2019-10-13T16:49:09.130 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:09.120+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.125051+0000) 2019-10-13T16:49:09.130 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:09.120+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.126216+0000) 2019-10-13T16:49:09.130 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:09.120+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.126325+0000) 2019-10-13T16:49:09.130 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:09.124+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.127792+0000) 2019-10-13T16:49:09.131 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:09.124+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.128211+0000) 2019-10-13T16:49:09.131 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:09.124+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.128533+0000) 2019-10-13T16:49:09.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:09.129+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:09.133482+0000) 2019-10-13T16:49:09.139 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:09.132+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:10.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:10.284+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:10.290512+0000) 2019-10-13T16:49:10.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:10.537+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:10.539895+0000) 2019-10-13T16:49:12.808 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:12.800+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:12.806726+0000) 2019-10-13T16:49:14.143 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:14.137+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.140814+0000) 2019-10-13T16:49:14.143 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:14.136+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.141284+0000) 2019-10-13T16:49:14.144 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:14.136+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.141798+0000) 2019-10-13T16:49:14.144 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:14.136+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.141894+0000) 2019-10-13T16:49:14.144 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:14.136+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.142746+0000) 2019-10-13T16:49:14.145 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:14.140+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.143863+0000) 2019-10-13T16:49:14.146 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:14.140+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.144375+0000) 2019-10-13T16:49:14.152 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:14.149+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:14.150609+0000) 2019-10-13T16:49:14.156 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:14.148+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:15.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:15.812+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:15.818845+0000) 2019-10-13T16:49:15.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:15.812+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:15.818844+0000) 2019-10-13T16:49:15.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:15.824+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:15.827331+0000) 2019-10-13T16:49:16.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:16.364+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:16.369622+0000) 2019-10-13T16:49:18.120 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:18.117+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:18.118555+0000) 2019-10-13T16:49:19.159 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:19.152+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.157457+0000) 2019-10-13T16:49:19.160 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:19.152+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.157542+0000) 2019-10-13T16:49:19.160 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:19.152+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.158036+0000) 2019-10-13T16:49:19.160 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:19.152+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.158643+0000) 2019-10-13T16:49:19.161 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:19.156+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.159898+0000) 2019-10-13T16:49:19.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:19.156+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.160468+0000) 2019-10-13T16:49:19.162 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:19.157+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.160749+0000) 2019-10-13T16:49:19.167 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:19.161+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:19.165794+0000) 2019-10-13T16:49:19.171 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:19.164+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253427 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:20.292 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:20.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:20.290709+0000) 2019-10-13T16:49:20.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:20.537+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:20.540131+0000) 2019-10-13T16:49:22.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:22.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:22.806957+0000) 2019-10-13T16:49:24.174 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:24.168+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.172649+0000) 2019-10-13T16:49:24.175 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:24.168+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.173533+0000) 2019-10-13T16:49:24.176 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:24.168+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.174278+0000) 2019-10-13T16:49:24.176 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:24.168+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.174838+0000) 2019-10-13T16:49:24.178 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:24.172+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.176200+0000) 2019-10-13T16:49:24.178 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:24.173+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.176076+0000) 2019-10-13T16:49:24.178 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:24.172+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.176513+0000) 2019-10-13T16:49:24.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:24.177+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:24.181953+0000) 2019-10-13T16:49:24.187 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:24.180+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253459 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:25.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:25.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:25.819049+0000) 2019-10-13T16:49:25.821 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:25.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:25.819051+0000) 2019-10-13T16:49:25.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:25.824+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:25.827530+0000) 2019-10-13T16:49:26.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:26.368+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:26.369825+0000) 2019-10-13T16:49:28.127 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:28.125+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:28.125100+0000) 2019-10-13T16:49:29.190 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:29.184+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.188632+0000) 2019-10-13T16:49:29.191 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:29.184+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.189630+0000) 2019-10-13T16:49:29.191 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:29.184+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.189867+0000) 2019-10-13T16:49:29.192 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:29.184+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.190511+0000) 2019-10-13T16:49:29.193 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:29.184+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.191080+0000) 2019-10-13T16:49:29.193 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:29.188+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.191644+0000) 2019-10-13T16:49:29.194 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:29.188+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.192089+0000) 2019-10-13T16:49:29.200 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:29.196+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:29.198364+0000) 2019-10-13T16:49:29.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:29.196+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:29.783 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:49:29.783 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:49:29.786 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:49:29.787 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:49:30.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:30.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:30.290912+0000) 2019-10-13T16:49:30.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:30.536+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:30.540336+0000) 2019-10-13T16:49:32.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:32.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:32.807152+0000) 2019-10-13T16:49:34.207 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:34.200+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.205560+0000) 2019-10-13T16:49:34.208 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:34.200+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.205819+0000) 2019-10-13T16:49:34.208 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:34.200+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.206455+0000) 2019-10-13T16:49:34.209 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:34.200+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.206870+0000) 2019-10-13T16:49:34.209 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:34.204+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.207325+0000) 2019-10-13T16:49:34.209 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:34.204+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.207737+0000) 2019-10-13T16:49:34.210 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:34.204+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.208329+0000) 2019-10-13T16:49:34.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:34.212+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:34.214194+0000) 2019-10-13T16:49:34.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:34.212+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:35.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:35.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:35.819246+0000) 2019-10-13T16:49:35.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:35.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:35.819252+0000) 2019-10-13T16:49:35.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:35.824+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:35.827763+0000) 2019-10-13T16:49:36.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:36.368+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:36.370043+0000) 2019-10-13T16:49:38.133 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:38.128+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:38.131551+0000) 2019-10-13T16:49:39.225 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:39.216+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.222434+0000) 2019-10-13T16:49:39.225 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:39.216+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.222118+0000) 2019-10-13T16:49:39.225 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:39.216+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.222681+0000) 2019-10-13T16:49:39.226 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:39.220+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.223027+0000) 2019-10-13T16:49:39.227 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:39.220+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.223831+0000) 2019-10-13T16:49:39.227 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:39.220+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.224027+0000) 2019-10-13T16:49:39.227 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:39.220+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.224110+0000) 2019-10-13T16:49:39.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:39.224+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:39.227792+0000) 2019-10-13T16:49:39.236 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:39.228+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253565 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:40.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:40.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:40.291127+0000) 2019-10-13T16:49:40.542 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:40.536+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:40.540546+0000) 2019-10-13T16:49:42.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:42.804+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:42.807385+0000) 2019-10-13T16:49:44.240 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:44.232+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.237672+0000) 2019-10-13T16:49:44.241 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:44.232+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.238351+0000) 2019-10-13T16:49:44.241 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:44.232+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.238604+0000) 2019-10-13T16:49:44.242 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:44.232+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.238848+0000) 2019-10-13T16:49:44.242 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:44.236+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.239401+0000) 2019-10-13T16:49:44.242 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:44.236+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.239909+0000) 2019-10-13T16:49:44.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:44.236+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.241008+0000) 2019-10-13T16:49:44.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:44.244+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:44.246383+0000) 2019-10-13T16:49:44.251 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:44.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253597 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:45.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:45.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:45.819535+0000) 2019-10-13T16:49:45.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:45.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:45.819528+0000) 2019-10-13T16:49:45.829 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:45.824+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:45.827995+0000) 2019-10-13T16:49:46.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:46.368+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:46.370249+0000) 2019-10-13T16:49:48.141 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:48.132+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:48.137949+0000) 2019-10-13T16:49:49.254 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:49.247+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.252551+0000) 2019-10-13T16:49:49.255 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:49.247+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.252920+0000) 2019-10-13T16:49:49.255 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:49.247+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.253723+0000) 2019-10-13T16:49:49.256 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:49.247+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.254445+0000) 2019-10-13T16:49:49.256 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:49.247+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.254951+0000) 2019-10-13T16:49:49.257 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:49.252+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.255641+0000) 2019-10-13T16:49:49.258 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:49.251+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.255980+0000) 2019-10-13T16:49:49.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:49.256+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:49.259936+0000) 2019-10-13T16:49:49.267 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:49.259+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253633 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:50.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:50.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:50.291333+0000) 2019-10-13T16:49:50.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:50.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:50.540716+0000) 2019-10-13T16:49:52.809 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:52.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:52.807583+0000) 2019-10-13T16:49:54.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:54.263+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.269415+0000) 2019-10-13T16:49:54.272 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:54.263+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.270137+0000) 2019-10-13T16:49:54.272 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:54.263+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.270506+0000) 2019-10-13T16:49:54.273 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:54.263+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.270873+0000) 2019-10-13T16:49:54.274 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:54.267+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.271504+0000) 2019-10-13T16:49:54.274 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:54.267+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.272076+0000) 2019-10-13T16:49:54.274 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:54.268+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.272072+0000) 2019-10-13T16:49:54.280 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:54.276+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:54.278740+0000) 2019-10-13T16:49:54.284 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:54.275+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253667 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:55.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:55.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:55.819774+0000) 2019-10-13T16:49:55.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:55.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:55.819773+0000) 2019-10-13T16:49:55.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:55.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:55.828186+0000) 2019-10-13T16:49:56.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:56.367+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:56.370471+0000) 2019-10-13T16:49:58.151 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:58.148+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:58.149280+0000) 2019-10-13T16:49:59.289 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:49:59.279+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.287063+0000) 2019-10-13T16:49:59.289 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:49:59.283+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.287442+0000) 2019-10-13T16:49:59.290 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:49:59.284+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.287866+0000) 2019-10-13T16:49:59.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:49:59.283+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.287911+0000) 2019-10-13T16:49:59.290 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:49:59.283+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.288667+0000) 2019-10-13T16:49:59.291 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:49:59.283+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.288961+0000) 2019-10-13T16:49:59.291 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:49:59.283+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.289704+0000) 2019-10-13T16:49:59.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:49:59.292+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:49:59.296237+0000) 2019-10-13T16:49:59.300 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:49:59.291+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253703 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:49:59.951 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:49:59.951 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:49:59.955 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:49:59.955 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:50:00.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:00.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:00.291526+0000) 2019-10-13T16:50:00.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:00.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:00.540936+0000) 2019-10-13T16:50:02.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:02.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:02.807785+0000) 2019-10-13T16:50:04.304 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:04.295+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.302061+0000) 2019-10-13T16:50:04.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:04.295+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.302534+0000) 2019-10-13T16:50:04.305 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:04.295+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.303075+0000) 2019-10-13T16:50:04.305 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:04.299+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.303525+0000) 2019-10-13T16:50:04.306 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:04.299+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.304161+0000) 2019-10-13T16:50:04.307 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:04.300+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.304989+0000) 2019-10-13T16:50:04.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:04.299+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.305746+0000) 2019-10-13T16:50:04.316 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:04.307+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:04.317 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:04.312+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:04.315072+0000) 2019-10-13T16:50:05.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:05.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:05.819994+0000) 2019-10-13T16:50:05.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:05.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:05.819969+0000) 2019-10-13T16:50:05.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:05.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:05.828378+0000) 2019-10-13T16:50:06.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:06.363+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:06.370666+0000) 2019-10-13T16:50:08.158 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:08.152+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:08.155728+0000) 2019-10-13T16:50:09.319 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:09.311+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.317571+0000) 2019-10-13T16:50:09.320 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:09.311+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.318324+0000) 2019-10-13T16:50:09.322 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:09.315+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.320134+0000) 2019-10-13T16:50:09.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:09.315+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.320261+0000) 2019-10-13T16:50:09.323 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:09.315+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.320657+0000) 2019-10-13T16:50:09.323 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:09.315+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.321077+0000) 2019-10-13T16:50:09.323 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:09.316+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.321401+0000) 2019-10-13T16:50:09.327 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:09.320+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:09.325317+0000) 2019-10-13T16:50:09.332 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:09.323+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:10.293 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:10.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:10.291731+0000) 2019-10-13T16:50:10.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:10.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:10.541134+0000) 2019-10-13T16:50:12.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:12.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:12.808005+0000) 2019-10-13T16:50:14.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:14.327+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.333719+0000) 2019-10-13T16:50:14.336 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:14.327+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.334218+0000) 2019-10-13T16:50:14.337 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:14.331+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.335632+0000) 2019-10-13T16:50:14.338 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:14.331+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.335420+0000) 2019-10-13T16:50:14.338 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:14.331+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.336070+0000) 2019-10-13T16:50:14.338 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:14.331+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.337010+0000) 2019-10-13T16:50:14.339 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:14.332+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.337372+0000) 2019-10-13T16:50:14.344 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:14.340+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:14.342653+0000) 2019-10-13T16:50:14.348 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:14.339+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253805 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:15.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:15.815+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:15.820180+0000) 2019-10-13T16:50:15.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:15.815+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:15.820197+0000) 2019-10-13T16:50:15.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:15.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:15.828573+0000) 2019-10-13T16:50:16.372 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:16.367+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:16.370857+0000) 2019-10-13T16:50:18.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:18.160+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:18.162236+0000) 2019-10-13T16:50:19.352 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:19.347+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.349919+0000) 2019-10-13T16:50:19.353 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:19.347+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.350509+0000) 2019-10-13T16:50:19.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:19.348+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.351151+0000) 2019-10-13T16:50:19.354 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:19.351+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.351890+0000) 2019-10-13T16:50:19.355 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:19.351+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.351870+0000) 2019-10-13T16:50:19.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:19.351+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.352364+0000) 2019-10-13T16:50:19.355 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:19.351+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.353110+0000) 2019-10-13T16:50:19.361 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:19.356+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:19.359517+0000) 2019-10-13T16:50:19.365 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:19.363+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253841 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:20.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:20.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:20.291931+0000) 2019-10-13T16:50:20.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:20.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:20.541331+0000) 2019-10-13T16:50:22.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:22.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:22.808205+0000) 2019-10-13T16:50:24.369 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:24.363+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.366963+0000) 2019-10-13T16:50:24.369 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:24.363+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.367765+0000) 2019-10-13T16:50:24.370 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:24.367+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.368217+0000) 2019-10-13T16:50:24.370 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:24.367+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.368454+0000) 2019-10-13T16:50:24.371 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:24.367+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.368866+0000) 2019-10-13T16:50:24.371 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:24.367+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.369731+0000) 2019-10-13T16:50:24.371 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:24.367+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.369795+0000) 2019-10-13T16:50:24.378 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:24.371+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:24.376187+0000) 2019-10-13T16:50:24.382 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:24.379+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253873 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:25.822 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:25.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:25.820363+0000) 2019-10-13T16:50:25.822 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:25.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:25.820405+0000) 2019-10-13T16:50:25.830 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:25.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:25.828802+0000) 2019-10-13T16:50:26.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:26.367+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:26.371085+0000) 2019-10-13T16:50:28.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:28.167+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:28.168815+0000) 2019-10-13T16:50:29.386 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:29.383+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.384535+0000) 2019-10-13T16:50:29.387 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:29.383+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.385357+0000) 2019-10-13T16:50:29.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:29.383+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.385839+0000) 2019-10-13T16:50:29.388 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:29.383+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.385894+0000) 2019-10-13T16:50:29.388 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:29.383+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.386681+0000) 2019-10-13T16:50:29.389 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:29.383+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.387245+0000) 2019-10-13T16:50:29.389 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:29.387+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.387986+0000) 2019-10-13T16:50:29.393 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:29.387+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:29.391064+0000) 2019-10-13T16:50:29.398 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:29.395+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253909 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:30.062 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:50:30.062 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:50:30.065 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:50:30.066 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:50:30.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:30.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:30.292106+0000) 2019-10-13T16:50:30.543 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:30.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:30.541524+0000) 2019-10-13T16:50:32.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:32.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:32.808421+0000) 2019-10-13T16:50:34.401 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:34.398+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.399665+0000) 2019-10-13T16:50:34.402 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:34.398+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.400072+0000) 2019-10-13T16:50:34.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:34.398+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.400399+0000) 2019-10-13T16:50:34.403 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:34.398+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.400830+0000) 2019-10-13T16:50:34.404 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:34.395+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.401949+0000) 2019-10-13T16:50:34.404 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:34.398+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.402419+0000) 2019-10-13T16:50:34.404 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:34.398+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.402942+0000) 2019-10-13T16:50:34.410 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:34.403+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:34.407986+0000) 2019-10-13T16:50:34.414 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:34.410+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253943 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:35.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:35.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:35.820567+0000) 2019-10-13T16:50:35.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:35.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:35.820655+0000) 2019-10-13T16:50:35.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:35.826+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:35.829001+0000) 2019-10-13T16:50:36.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:36.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:36.371342+0000) 2019-10-13T16:50:38.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:38.171+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:38.175243+0000) 2019-10-13T16:50:39.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:39.411+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.415574+0000) 2019-10-13T16:50:39.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:39.414+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.416228+0000) 2019-10-13T16:50:39.418 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:39.414+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.416603+0000) 2019-10-13T16:50:39.419 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:39.414+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.417251+0000) 2019-10-13T16:50:39.419 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:39.414+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.417864+0000) 2019-10-13T16:50:39.420 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:39.414+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.418437+0000) 2019-10-13T16:50:39.420 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:39.414+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.419048+0000) 2019-10-13T16:50:39.425 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:39.419+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:39.423216+0000) 2019-10-13T16:50:39.431 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:39.426+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 253978 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:40.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:40.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:40.292290+0000) 2019-10-13T16:50:40.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:40.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:40.541718+0000) 2019-10-13T16:50:42.810 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:42.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:42.808624+0000) 2019-10-13T16:50:44.435 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:44.430+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.433500+0000) 2019-10-13T16:50:44.436 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:44.430+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.434127+0000) 2019-10-13T16:50:44.436 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:44.430+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.434447+0000) 2019-10-13T16:50:44.437 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:44.431+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.434550+0000) 2019-10-13T16:50:44.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:44.430+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.435065+0000) 2019-10-13T16:50:44.438 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:44.434+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.435473+0000) 2019-10-13T16:50:44.438 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:44.434+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.436074+0000) 2019-10-13T16:50:44.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:44.435+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:44.439120+0000) 2019-10-13T16:50:44.447 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:44.442+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254011 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:45.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:45.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:45.820800+0000) 2019-10-13T16:50:45.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:45.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:45.820887+0000) 2019-10-13T16:50:45.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:45.826+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:45.829226+0000) 2019-10-13T16:50:46.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:46.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:46.371554+0000) 2019-10-13T16:50:48.184 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:48.179+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:48.181771+0000) 2019-10-13T16:50:49.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:49.446+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.448456+0000) 2019-10-13T16:50:49.451 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:49.446+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.448927+0000) 2019-10-13T16:50:49.451 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:49.443+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.449099+0000) 2019-10-13T16:50:49.452 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:49.446+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.450075+0000) 2019-10-13T16:50:49.453 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:49.446+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.451100+0000) 2019-10-13T16:50:49.453 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:49.450+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.451363+0000) 2019-10-13T16:50:49.453 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:49.450+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.452111+0000) 2019-10-13T16:50:49.459 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:49.451+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:49.456683+0000) 2019-10-13T16:50:49.463 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:49.458+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254047 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:50.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:50.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:50.292427+0000) 2019-10-13T16:50:50.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:50.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:50.541906+0000) 2019-10-13T16:50:52.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:52.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:52.808826+0000) 2019-10-13T16:50:54.466 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:54.462+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.464783+0000) 2019-10-13T16:50:54.467 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:54.459+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.465191+0000) 2019-10-13T16:50:54.467 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:54.462+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.465844+0000) 2019-10-13T16:50:54.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:54.462+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.466702+0000) 2019-10-13T16:50:54.469 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:54.466+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.467656+0000) 2019-10-13T16:50:54.470 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:54.466+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.468083+0000) 2019-10-13T16:50:54.470 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:54.466+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.468562+0000) 2019-10-13T16:50:54.475 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:54.467+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:54.473144+0000) 2019-10-13T16:50:54.478 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:54.474+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254081 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:50:55.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:55.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:55.821033+0000) 2019-10-13T16:50:55.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:55.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:55.821106+0000) 2019-10-13T16:50:55.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:55.826+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:55.829447+0000) 2019-10-13T16:50:56.373 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:56.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:56.371791+0000) 2019-10-13T16:50:58.190 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:58.187+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:58.188293+0000) 2019-10-13T16:50:59.482 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:50:59.475+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.479924+0000) 2019-10-13T16:50:59.482 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:50:59.478+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.481059+0000) 2019-10-13T16:50:59.483 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:50:59.478+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.481394+0000) 2019-10-13T16:50:59.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:50:59.478+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.482045+0000) 2019-10-13T16:50:59.484 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:50:59.478+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.482344+0000) 2019-10-13T16:50:59.485 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:50:59.478+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.482550+0000) 2019-10-13T16:50:59.485 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:50:59.478+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.482950+0000) 2019-10-13T16:50:59.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:50:59.483+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:50:59.489668+0000) 2019-10-13T16:50:59.493 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:50:59.490+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254116 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:00.199 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:51:00.199 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:51:00.203 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:51:00.203 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:51:00.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:00.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:00.292604+0000) 2019-10-13T16:51:00.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:00.535+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:00.542101+0000) 2019-10-13T16:51:02.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:02.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:02.809028+0000) 2019-10-13T16:51:04.497 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:04.490+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.494969+0000) 2019-10-13T16:51:04.497 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:04.494+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.495812+0000) 2019-10-13T16:51:04.498 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:04.494+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.496493+0000) 2019-10-13T16:51:04.499 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:04.491+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.496456+0000) 2019-10-13T16:51:04.499 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:04.494+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.496037+0000) 2019-10-13T16:51:04.499 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:04.494+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.497665+0000) 2019-10-13T16:51:04.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:04.494+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.497858+0000) 2019-10-13T16:51:04.505 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:04.499+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:04.503557+0000) 2019-10-13T16:51:04.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:04.502+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254149 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:05.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:05.818+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:05.821234+0000) 2019-10-13T16:51:05.823 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:05.818+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:05.821323+0000) 2019-10-13T16:51:05.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:05.826+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:05.829673+0000) 2019-10-13T16:51:06.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:06.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:06.372009+0000) 2019-10-13T16:51:08.197 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:08.191+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:08.194821+0000) 2019-10-13T16:51:09.513 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:09.506+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.510959+0000) 2019-10-13T16:51:09.513 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:09.510+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.511686+0000) 2019-10-13T16:51:09.514 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:09.510+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.511872+0000) 2019-10-13T16:51:09.514 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:09.510+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.512227+0000) 2019-10-13T16:51:09.514 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:09.507+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.512596+0000) 2019-10-13T16:51:09.515 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:09.510+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.513296+0000) 2019-10-13T16:51:09.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:09.510+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.513632+0000) 2019-10-13T16:51:09.522 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:09.515+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:09.520259+0000) 2019-10-13T16:51:09.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:09.518+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254185 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:10.294 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:10.290+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:10.292816+0000) 2019-10-13T16:51:10.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:10.539+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:10.542334+0000) 2019-10-13T16:51:12.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:12.806+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:12.809233+0000) 2019-10-13T16:51:14.528 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:14.521+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.526858+0000) 2019-10-13T16:51:14.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:14.525+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.527566+0000) 2019-10-13T16:51:14.530 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:14.525+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.528512+0000) 2019-10-13T16:51:14.530 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:14.525+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.528890+0000) 2019-10-13T16:51:14.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:14.523+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.529161+0000) 2019-10-13T16:51:14.532 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:14.525+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.529653+0000) 2019-10-13T16:51:14.532 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:14.525+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.529986+0000) 2019-10-13T16:51:14.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:14.531+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:14.536409+0000) 2019-10-13T16:51:14.540 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:14.533+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:15.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:15.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:15.821438+0000) 2019-10-13T16:51:15.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:15.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:15.821524+0000) 2019-10-13T16:51:15.831 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:15.825+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:15.829868+0000) 2019-10-13T16:51:16.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:16.369+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:16.372166+0000) 2019-10-13T16:51:18.203 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:18.195+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:18.201283+0000) 2019-10-13T16:51:19.543 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:19.537+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.541287+0000) 2019-10-13T16:51:19.543 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:19.537+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.541475+0000) 2019-10-13T16:51:19.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:19.534+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.541559+0000) 2019-10-13T16:51:19.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:19.541+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.543468+0000) 2019-10-13T16:51:19.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:19.541+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.543969+0000) 2019-10-13T16:51:19.546 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:19.541+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.544166+0000) 2019-10-13T16:51:19.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:19.541+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.545053+0000) 2019-10-13T16:51:19.552 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:19.546+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:19.550643+0000) 2019-10-13T16:51:19.557 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:19.549+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:20.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:20.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:20.293013+0000) 2019-10-13T16:51:20.544 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:20.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:20.542532+0000) 2019-10-13T16:51:22.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:22.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:22.809457+0000) 2019-10-13T16:51:24.567 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:24.553+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.558242+0000) 2019-10-13T16:51:24.567 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:24.554+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.558516+0000) 2019-10-13T16:51:24.568 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:24.557+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.559576+0000) 2019-10-13T16:51:24.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:24.557+0000 7fb4ea307700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.559801+0000) 2019-10-13T16:51:24.568 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:24.557+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.560170+0000) 2019-10-13T16:51:24.569 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:24.557+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.560590+0000) 2019-10-13T16:51:24.569 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:24.557+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.561411+0000) 2019-10-13T16:51:24.576 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:24.566+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:24.570358+0000) 2019-10-13T16:51:24.577 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:24.565+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254287 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:25.823 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:25.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:25.821662+0000) 2019-10-13T16:51:25.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:25.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:25.821727+0000) 2019-10-13T16:51:25.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:25.825+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:25.830080+0000) 2019-10-13T16:51:26.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:26.369+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:26.372352+0000) 2019-10-13T16:51:28.209 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:28.202+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:28.207384+0000) 2019-10-13T16:51:29.578 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:29.569+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.574947+0000) 2019-10-13T16:51:29.578 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:29.573+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.575843+0000) 2019-10-13T16:51:29.578 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:29.570+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.574216+0000) 2019-10-13T16:51:29.579 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:29.573+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.576229+0000) 2019-10-13T16:51:29.579 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:29.573+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.576576+0000) 2019-10-13T16:51:29.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:29.573+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.577636+0000) 2019-10-13T16:51:29.580 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:29.573+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.578596+0000) 2019-10-13T16:51:29.583 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:29.574+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:29.581439+0000) 2019-10-13T16:51:29.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:29.581+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254322 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:30.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:30.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:30.293193+0000) 2019-10-13T16:51:30.318 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:51:30.318 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:51:30.322 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:51:30.322 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:51:30.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:30.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:30.542748+0000) 2019-10-13T16:51:32.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:32.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:32.809677+0000) 2019-10-13T16:51:34.592 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:34.585+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.590616+0000) 2019-10-13T16:51:34.593 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:34.585+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.591025+0000) 2019-10-13T16:51:34.593 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:34.589+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.591842+0000) 2019-10-13T16:51:34.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:34.589+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.592196+0000) 2019-10-13T16:51:34.595 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:34.589+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.593407+0000) 2019-10-13T16:51:34.595 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:34.586+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.593607+0000) 2019-10-13T16:51:34.596 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:34.589+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.594594+0000) 2019-10-13T16:51:34.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:34.597+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254357 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:34.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:34.598+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:34.603341+0000) 2019-10-13T16:51:35.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:35.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:35.821869+0000) 2019-10-13T16:51:35.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:35.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:35.821935+0000) 2019-10-13T16:51:35.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:35.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:35.830294+0000) 2019-10-13T16:51:36.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:36.369+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:36.372571+0000) 2019-10-13T16:51:38.216 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:38.210+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:38.213912+0000) 2019-10-13T16:51:39.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:39.601+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.606143+0000) 2019-10-13T16:51:39.608 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:39.601+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.606667+0000) 2019-10-13T16:51:39.609 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:39.605+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.607355+0000) 2019-10-13T16:51:39.609 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:39.602+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.607851+0000) 2019-10-13T16:51:39.610 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:39.605+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.608245+0000) 2019-10-13T16:51:39.610 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:39.605+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.608728+0000) 2019-10-13T16:51:39.611 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:39.605+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.609228+0000) 2019-10-13T16:51:39.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:39.606+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:39.613693+0000) 2019-10-13T16:51:39.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:39.613+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254391 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:40.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:40.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:40.293374+0000) 2019-10-13T16:51:40.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:40.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:40.542946+0000) 2019-10-13T16:51:42.811 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:42.805+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:42.809914+0000) 2019-10-13T16:51:44.624 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:44.617+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.622033+0000) 2019-10-13T16:51:44.624 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:44.617+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.622261+0000) 2019-10-13T16:51:44.625 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:44.618+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.622422+0000) 2019-10-13T16:51:44.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:44.617+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.622984+0000) 2019-10-13T16:51:44.626 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:44.621+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.624583+0000) 2019-10-13T16:51:44.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:44.621+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.624575+0000) 2019-10-13T16:51:44.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:44.621+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.624078+0000) 2019-10-13T16:51:44.634 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:44.626+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:44.631900+0000) 2019-10-13T16:51:44.636 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:44.629+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254425 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:45.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:45.817+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:45.822129+0000) 2019-10-13T16:51:45.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:45.817+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:45.822105+0000) 2019-10-13T16:51:45.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:45.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:45.830478+0000) 2019-10-13T16:51:46.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:46.369+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:46.372764+0000) 2019-10-13T16:51:48.222 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:48.218+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:48.220646+0000) 2019-10-13T16:51:49.640 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:49.633+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.638170+0000) 2019-10-13T16:51:49.640 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:49.633+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.638738+0000) 2019-10-13T16:51:49.641 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:49.637+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.639779+0000) 2019-10-13T16:51:49.641 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:49.633+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.638964+0000) 2019-10-13T16:51:49.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:49.634+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.640039+0000) 2019-10-13T16:51:49.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:49.637+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.640511+0000) 2019-10-13T16:51:49.643 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:49.637+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.640994+0000) 2019-10-13T16:51:49.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:49.638+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:49.645556+0000) 2019-10-13T16:51:49.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:49.645+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254460 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:50.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:50.289+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:50.293576+0000) 2019-10-13T16:51:50.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:50.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:50.543121+0000) 2019-10-13T16:51:52.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:52.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:52.810147+0000) 2019-10-13T16:51:54.656 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:54.646+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.653974+0000) 2019-10-13T16:51:54.656 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:54.649+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.654659+0000) 2019-10-13T16:51:54.657 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:54.649+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.654948+0000) 2019-10-13T16:51:54.657 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:54.653+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.655818+0000) 2019-10-13T16:51:54.658 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:54.653+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.656577+0000) 2019-10-13T16:51:54.658 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:54.653+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.656510+0000) 2019-10-13T16:51:54.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:54.653+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.657691+0000) 2019-10-13T16:51:54.664 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:54.658+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:54.662829+0000) 2019-10-13T16:51:54.667 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:54.661+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254495 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:51:55.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:55.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:55.822308+0000) 2019-10-13T16:51:55.824 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:55.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:55.822360+0000) 2019-10-13T16:51:55.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:55.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:55.830682+0000) 2019-10-13T16:51:56.374 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:56.369+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:56.372959+0000) 2019-10-13T16:51:58.229 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:58.226+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:58.227399+0000) 2019-10-13T16:51:59.671 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:51:59.664+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.669291+0000) 2019-10-13T16:51:59.671 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:51:59.664+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.669559+0000) 2019-10-13T16:51:59.672 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:51:59.664+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.670285+0000) 2019-10-13T16:51:59.672 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:51:59.666+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.670605+0000) 2019-10-13T16:51:59.673 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:51:59.664+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.671052+0000) 2019-10-13T16:51:59.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:51:59.664+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.671280+0000) 2019-10-13T16:51:59.673 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:51:59.668+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.671690+0000) 2019-10-13T16:51:59.680 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:51:59.674+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:51:59.678216+0000) 2019-10-13T16:51:59.683 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:51:59.676+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254529 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:00.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:00.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:00.293771+0000) 2019-10-13T16:52:00.421 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:52:00.421 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:52:00.445 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:52:00.445 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:52:00.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:00.538+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:00.543292+0000) 2019-10-13T16:52:02.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:02.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:02.810351+0000) 2019-10-13T16:52:04.686 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:04.680+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.684412+0000) 2019-10-13T16:52:04.687 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:04.680+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.685079+0000) 2019-10-13T16:52:04.687 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:04.680+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.685777+0000) 2019-10-13T16:52:04.688 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:04.680+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.686465+0000) 2019-10-13T16:52:04.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:04.682+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.687113+0000) 2019-10-13T16:52:04.689 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:04.684+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.687654+0000) 2019-10-13T16:52:04.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:04.684+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.687941+0000) 2019-10-13T16:52:04.694 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:04.686+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:04.692226+0000) 2019-10-13T16:52:04.698 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:04.692+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254563 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:05.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:05.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:05.822500+0000) 2019-10-13T16:52:05.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:05.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:05.822588+0000) 2019-10-13T16:52:05.832 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:05.828+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:05.830868+0000) 2019-10-13T16:52:06.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:06.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:06.373176+0000) 2019-10-13T16:52:08.236 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:08.230+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:08.233893+0000) 2019-10-13T16:52:09.701 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:09.696+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.699600+0000) 2019-10-13T16:52:09.702 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:09.696+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.699919+0000) 2019-10-13T16:52:09.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:09.696+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.700917+0000) 2019-10-13T16:52:09.703 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:09.696+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.701372+0000) 2019-10-13T16:52:09.703 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:09.696+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.701678+0000) 2019-10-13T16:52:09.704 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:09.696+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.702518+0000) 2019-10-13T16:52:09.705 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:09.702+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.702942+0000) 2019-10-13T16:52:09.713 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:09.706+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:09.709089+0000) 2019-10-13T16:52:09.714 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:09.708+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254598 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:10.295 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:10.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:10.293901+0000) 2019-10-13T16:52:10.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:10.542+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:10.543471+0000) 2019-10-13T16:52:12.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:12.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:12.810554+0000) 2019-10-13T16:52:14.717 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:14.712+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.715685+0000) 2019-10-13T16:52:14.718 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:14.712+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.716002+0000) 2019-10-13T16:52:14.718 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:14.712+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.716750+0000) 2019-10-13T16:52:14.719 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:14.712+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.716908+0000) 2019-10-13T16:52:14.719 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:14.713+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.717775+0000) 2019-10-13T16:52:14.720 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:14.712+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.718456+0000) 2019-10-13T16:52:14.720 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:14.712+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.718635+0000) 2019-10-13T16:52:14.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:14.721+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:14.723635+0000) 2019-10-13T16:52:14.730 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:14.724+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254632 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:15.824 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:15.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:15.822703+0000) 2019-10-13T16:52:15.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:15.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:15.822792+0000) 2019-10-13T16:52:15.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:15.828+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:15.831085+0000) 2019-10-13T16:52:16.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:16.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:16.373397+0000) 2019-10-13T16:52:18.242 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:18.237+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:18.240307+0000) 2019-10-13T16:52:19.733 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:19.724+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.731242+0000) 2019-10-13T16:52:19.734 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:19.728+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.731872+0000) 2019-10-13T16:52:19.734 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:19.728+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.732528+0000) 2019-10-13T16:52:19.734 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:19.728+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.732896+0000) 2019-10-13T16:52:19.735 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:19.729+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.733608+0000) 2019-10-13T16:52:19.736 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:19.728+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.734283+0000) 2019-10-13T16:52:19.736 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:19.728+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.735037+0000) 2019-10-13T16:52:19.742 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:19.737+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:19.739937+0000) 2019-10-13T16:52:19.747 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:19.740+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254666 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:20.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:20.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:20.294083+0000) 2019-10-13T16:52:20.545 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:20.541+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:20.543638+0000) 2019-10-13T16:52:22.812 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:22.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:22.810766+0000) 2019-10-13T16:52:24.751 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:24.744+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.749038+0000) 2019-10-13T16:52:24.751 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:24.744+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.749671+0000) 2019-10-13T16:52:24.752 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:24.744+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.749810+0000) 2019-10-13T16:52:24.752 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:24.744+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.750120+0000) 2019-10-13T16:52:24.753 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:24.744+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.750555+0000) 2019-10-13T16:52:24.753 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:24.749+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.750703+0000) 2019-10-13T16:52:24.753 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:24.748+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.751893+0000) 2019-10-13T16:52:24.759 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:24.753+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:24.757584+0000) 2019-10-13T16:52:24.765 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:24.756+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254699 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:25.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:25.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:25.822931+0000) 2019-10-13T16:52:25.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:25.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:25.822967+0000) 2019-10-13T16:52:25.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:25.824+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:25.831278+0000) 2019-10-13T16:52:26.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:26.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:26.373602+0000) 2019-10-13T16:52:28.248 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:28.245+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:28.246420+0000) 2019-10-13T16:52:29.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:29.760+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.766611+0000) 2019-10-13T16:52:29.769 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:29.764+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.767523+0000) 2019-10-13T16:52:29.769 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:29.764+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.767911+0000) 2019-10-13T16:52:29.770 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:29.765+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.768498+0000) 2019-10-13T16:52:29.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:29.764+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.769196+0000) 2019-10-13T16:52:29.771 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:29.764+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.769560+0000) 2019-10-13T16:52:29.772 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:29.764+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.770140+0000) 2019-10-13T16:52:29.776 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:29.773+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:29.774812+0000) 2019-10-13T16:52:29.782 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:29.776+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254734 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:30.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:30.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:30.294293+0000) 2019-10-13T16:52:30.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:30.541+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:30.543838+0000) 2019-10-13T16:52:30.766 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:52:30.766 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:52:30.770 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:52:30.770 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:52:32.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:32.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:32.810967+0000) 2019-10-13T16:52:34.785 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:34.776+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.783196+0000) 2019-10-13T16:52:34.786 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:34.781+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.784000+0000) 2019-10-13T16:52:34.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:34.780+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.784812+0000) 2019-10-13T16:52:34.787 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:34.780+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.785097+0000) 2019-10-13T16:52:34.787 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:34.780+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.785320+0000) 2019-10-13T16:52:34.788 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:34.780+0000 7f914e6fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.786802+0000) 2019-10-13T16:52:34.790 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:34.784+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.787457+0000) 2019-10-13T16:52:34.792 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:34.789+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:34.790298+0000) 2019-10-13T16:52:34.797 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:34.792+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:35.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:35.816+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:35.823092+0000) 2019-10-13T16:52:35.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:35.816+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:35.823161+0000) 2019-10-13T16:52:35.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:35.828+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:35.831452+0000) 2019-10-13T16:52:36.375 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:36.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:36.373837+0000) 2019-10-13T16:52:38.255 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:38.249+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:38.253036+0000) 2019-10-13T16:52:39.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:39.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.798607+0000) 2019-10-13T16:52:39.801 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:39.792+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.799242+0000) 2019-10-13T16:52:39.801 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:39.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.799762+0000) 2019-10-13T16:52:39.804 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:39.796+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.800797+0000) 2019-10-13T16:52:39.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:39.796+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.801503+0000) 2019-10-13T16:52:39.804 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:39.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.802165+0000) 2019-10-13T16:52:39.805 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:39.797+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.801078+0000) 2019-10-13T16:52:39.808 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:39.805+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:39.806741+0000) 2019-10-13T16:52:39.814 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:39.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254802 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:40.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:40.288+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:40.294466+0000) 2019-10-13T16:52:40.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:40.541+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:40.544003+0000) 2019-10-13T16:52:42.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:42.803+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:42.811197+0000) 2019-10-13T16:52:44.817 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:44.813+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.815562+0000) 2019-10-13T16:52:44.819 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:44.811+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.817321+0000) 2019-10-13T16:52:44.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:44.811+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.818013+0000) 2019-10-13T16:52:44.820 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:44.811+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.817707+0000) 2019-10-13T16:52:44.820 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:44.811+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.818674+0000) 2019-10-13T16:52:44.821 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:44.811+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.818803+0000) 2019-10-13T16:52:44.821 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:44.815+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.819354+0000) 2019-10-13T16:52:44.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:44.821+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:44.822620+0000) 2019-10-13T16:52:44.830 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:44.823+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254835 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:45.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:45.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:45.823298+0000) 2019-10-13T16:52:45.825 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:45.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:45.823390+0000) 2019-10-13T16:52:45.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:45.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:45.831663+0000) 2019-10-13T16:52:46.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:46.371+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:46.374035+0000) 2019-10-13T16:52:48.261 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:48.257+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:48.259493+0000) 2019-10-13T16:52:49.833 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:49.827+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.831383+0000) 2019-10-13T16:52:49.834 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:49.829+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.831959+0000) 2019-10-13T16:52:49.835 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:49.827+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.833051+0000) 2019-10-13T16:52:49.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:49.827+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.833204+0000) 2019-10-13T16:52:49.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:49.827+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.834043+0000) 2019-10-13T16:52:49.836 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:49.827+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.834730+0000) 2019-10-13T16:52:49.837 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:49.827+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.835028+0000) 2019-10-13T16:52:49.841 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:49.837+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:49.839856+0000) 2019-10-13T16:52:49.850 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:49.839+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254870 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:50.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:50.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:50.294657+0000) 2019-10-13T16:52:50.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:50.541+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:50.544132+0000) 2019-10-13T16:52:52.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:52.807+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:52.811395+0000) 2019-10-13T16:52:54.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:54.843+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.848883+0000) 2019-10-13T16:52:54.851 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:54.845+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.848903+0000) 2019-10-13T16:52:54.852 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:54.843+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.849427+0000) 2019-10-13T16:52:54.852 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:54.843+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.850271+0000) 2019-10-13T16:52:54.852 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:54.843+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.850884+0000) 2019-10-13T16:52:54.854 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:54.847+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.851708+0000) 2019-10-13T16:52:54.854 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:54.847+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.851878+0000) 2019-10-13T16:52:54.858 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:54.853+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:54.856743+0000) 2019-10-13T16:52:54.864 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:54.855+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254904 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:52:55.825 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:55.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:55.823544+0000) 2019-10-13T16:52:55.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:55.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:55.823589+0000) 2019-10-13T16:52:55.833 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:55.831+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:55.831868+0000) 2019-10-13T16:52:56.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:56.371+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:56.374236+0000) 2019-10-13T16:52:58.273 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:58.269+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:58.270847+0000) 2019-10-13T16:52:59.868 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:52:59.859+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.866094+0000) 2019-10-13T16:52:59.868 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:52:59.859+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.866377+0000) 2019-10-13T16:52:59.869 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:52:59.859+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.866775+0000) 2019-10-13T16:52:59.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:52:59.863+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.867765+0000) 2019-10-13T16:52:59.870 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:52:59.865+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.868025+0000) 2019-10-13T16:52:59.870 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:52:59.863+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.868407+0000) 2019-10-13T16:52:59.870 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:52:59.863+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.868893+0000) 2019-10-13T16:52:59.877 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:52:59.873+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:52:59.875202+0000) 2019-10-13T16:52:59.879 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:52:59.871+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254938 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:00.296 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:00.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:00.294876+0000) 2019-10-13T16:53:00.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:00.541+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:00.544309+0000) 2019-10-13T16:53:00.900 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:53:00.901 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:53:00.906 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:53:00.907 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:53:02.813 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:02.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:02.811592+0000) 2019-10-13T16:53:04.883 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:04.877+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.881083+0000) 2019-10-13T16:53:04.884 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:04.875+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.881901+0000) 2019-10-13T16:53:04.884 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:04.875+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.882055+0000) 2019-10-13T16:53:04.885 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:04.879+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.883771+0000) 2019-10-13T16:53:04.886 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:04.879+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.884086+0000) 2019-10-13T16:53:04.886 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:04.879+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.884319+0000) 2019-10-13T16:53:04.886 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:04.879+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.884784+0000) 2019-10-13T16:53:04.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:04.885+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:04.889598+0000) 2019-10-13T16:53:04.896 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:04.887+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 254970 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:05.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:05.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:05.823793+0000) 2019-10-13T16:53:05.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:05.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:05.823761+0000) 2019-10-13T16:53:05.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:05.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:05.832081+0000) 2019-10-13T16:53:06.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:06.367+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:06.374462+0000) 2019-10-13T16:53:08.279 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:08.277+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:08.277332+0000) 2019-10-13T16:53:09.899 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:09.891+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.897464+0000) 2019-10-13T16:53:09.900 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:09.891+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.897965+0000) 2019-10-13T16:53:09.900 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:09.896+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.898804+0000) 2019-10-13T16:53:09.901 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:09.895+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.899730+0000) 2019-10-13T16:53:09.902 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:09.895+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.899562+0000) 2019-10-13T16:53:09.902 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:09.895+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.900988+0000) 2019-10-13T16:53:09.903 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:09.895+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.901916+0000) 2019-10-13T16:53:09.908 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:09.904+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:09.906772+0000) 2019-10-13T16:53:09.913 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:09.907+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255006 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:10.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:10.287+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:10.295027+0000) 2019-10-13T16:53:10.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:10.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:10.544455+0000) 2019-10-13T16:53:12.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:12.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:12.811816+0000) 2019-10-13T16:53:14.917 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:14.912+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.914750+0000) 2019-10-13T16:53:14.917 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:14.915+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.915478+0000) 2019-10-13T16:53:14.918 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:14.915+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.916320+0000) 2019-10-13T16:53:14.918 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:14.915+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.916319+0000) 2019-10-13T16:53:14.919 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:14.915+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.916710+0000) 2019-10-13T16:53:14.919 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:14.915+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.917165+0000) 2019-10-13T16:53:14.919 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:14.915+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.917512+0000) 2019-10-13T16:53:14.927 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:14.920+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:14.925546+0000) 2019-10-13T16:53:14.930 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:14.927+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255039 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:15.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:15.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:15.823964+0000) 2019-10-13T16:53:15.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:15.823+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:15.823986+0000) 2019-10-13T16:53:15.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:15.831+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:15.832257+0000) 2019-10-13T16:53:16.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:16.371+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:16.374673+0000) 2019-10-13T16:53:18.286 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:18.280+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:18.283944+0000) 2019-10-13T16:53:19.933 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:19.927+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.930913+0000) 2019-10-13T16:53:19.934 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:19.928+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.931266+0000) 2019-10-13T16:53:19.934 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:19.931+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.931856+0000) 2019-10-13T16:53:19.934 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:19.931+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.932837+0000) 2019-10-13T16:53:19.935 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:19.931+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.933169+0000) 2019-10-13T16:53:19.937 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:19.931+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.934283+0000) 2019-10-13T16:53:19.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:19.931+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.934391+0000) 2019-10-13T16:53:19.943 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:19.936+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:19.941413+0000) 2019-10-13T16:53:19.945 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:19.939+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255074 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:20.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:20.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:20.295156+0000) 2019-10-13T16:53:20.546 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:20.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:20.544583+0000) 2019-10-13T16:53:22.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:22.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:22.812020+0000) 2019-10-13T16:53:24.948 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:24.943+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.946618+0000) 2019-10-13T16:53:24.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:24.944+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.946750+0000) 2019-10-13T16:53:24.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:24.947+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.947386+0000) 2019-10-13T16:53:24.950 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:24.947+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.948506+0000) 2019-10-13T16:53:24.951 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:24.947+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.948602+0000) 2019-10-13T16:53:24.951 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:24.947+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.949043+0000) 2019-10-13T16:53:24.951 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:24.947+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.949369+0000) 2019-10-13T16:53:24.958 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:24.952+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:24.956231+0000) 2019-10-13T16:53:24.961 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:24.955+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255106 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:25.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:25.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:25.824186+0000) 2019-10-13T16:53:25.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:25.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:25.824184+0000) 2019-10-13T16:53:25.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:25.830+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:25.832429+0000) 2019-10-13T16:53:26.376 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:26.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:26.374877+0000) 2019-10-13T16:53:28.292 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:28.288+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:28.290387+0000) 2019-10-13T16:53:29.964 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:29.956+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.961987+0000) 2019-10-13T16:53:29.964 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:29.958+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.962567+0000) 2019-10-13T16:53:29.965 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:29.958+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.963099+0000) 2019-10-13T16:53:29.966 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:29.962+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.963929+0000) 2019-10-13T16:53:29.966 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:29.962+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.964287+0000) 2019-10-13T16:53:29.966 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:29.962+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.964785+0000) 2019-10-13T16:53:29.967 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:29.962+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.965216+0000) 2019-10-13T16:53:29.973 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:29.968+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:29.971894+0000) 2019-10-13T16:53:29.977 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:29.970+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255142 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:30.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:30.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:30.295364+0000) 2019-10-13T16:53:30.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:30.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:30.544753+0000) 2019-10-13T16:53:31.032 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:53:31.033 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:53:31.036 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:53:31.036 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:53:32.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:32.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:32.812176+0000) 2019-10-13T16:53:34.981 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:34.974+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.978995+0000) 2019-10-13T16:53:34.981 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:34.978+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.979378+0000) 2019-10-13T16:53:34.982 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:34.978+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.979761+0000) 2019-10-13T16:53:34.982 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:34.978+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.980085+0000) 2019-10-13T16:53:34.982 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:34.976+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.980425+0000) 2019-10-13T16:53:34.983 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:34.978+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.980965+0000) 2019-10-13T16:53:34.983 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:34.978+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.981928+0000) 2019-10-13T16:53:34.989 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:34.984+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:34.987514+0000) 2019-10-13T16:53:34.996 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:34.986+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255174 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:35.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:35.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:35.824381+0000) 2019-10-13T16:53:35.826 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:35.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:35.824373+0000) 2019-10-13T16:53:35.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:35.830+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:35.832603+0000) 2019-10-13T16:53:36.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:36.370+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:36.375075+0000) 2019-10-13T16:53:38.299 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:38.292+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:38.296729+0000) 2019-10-13T16:53:39.996 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:39.990+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.994565+0000) 2019-10-13T16:53:39.997 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:39.990+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.994885+0000) 2019-10-13T16:53:39.997 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:39.994+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.996044+0000) 2019-10-13T16:53:39.998 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:39.992+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.996245+0000) 2019-10-13T16:53:39.999 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:39.994+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.997395+0000) 2019-10-13T16:53:39.999 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:39.994+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.997660+0000) 2019-10-13T16:53:40.000 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:39.994+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:39.998152+0000) 2019-10-13T16:53:40.006 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:40.000+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:40.003763+0000) 2019-10-13T16:53:40.009 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:40.006+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255209 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:40.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:40.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:40.295503+0000) 2019-10-13T16:53:40.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:40.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:40.544938+0000) 2019-10-13T16:53:42.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:42.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:42.812382+0000) 2019-10-13T16:53:45.013 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:45.010+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.011383+0000) 2019-10-13T16:53:45.014 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:45.010+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.011885+0000) 2019-10-13T16:53:45.014 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:45.010+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.012331+0000) 2019-10-13T16:53:45.014 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:45.010+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.013160+0000) 2019-10-13T16:53:45.015 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:45.008+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.013440+0000) 2019-10-13T16:53:45.015 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:45.010+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.013939+0000) 2019-10-13T16:53:45.016 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:45.010+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.014420+0000) 2019-10-13T16:53:45.023 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:45.016+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.020903+0000) 2019-10-13T16:53:45.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:45.018+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255240 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:45.826 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:45.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.824592+0000) 2019-10-13T16:53:45.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:45.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.824596+0000) 2019-10-13T16:53:45.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:45.830+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:45.832819+0000) 2019-10-13T16:53:46.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:46.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:46.375281+0000) 2019-10-13T16:53:48.305 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:48.300+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:48.303303+0000) 2019-10-13T16:53:50.028 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:50.022+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.025788+0000) 2019-10-13T16:53:50.029 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:50.024+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.027042+0000) 2019-10-13T16:53:50.030 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:50.026+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.028037+0000) 2019-10-13T16:53:50.030 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:50.026+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.028395+0000) 2019-10-13T16:53:50.030 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:50.026+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.028787+0000) 2019-10-13T16:53:50.031 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:50.026+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.029436+0000) 2019-10-13T16:53:50.031 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:50.026+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.029946+0000) 2019-10-13T16:53:50.033 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:50.028+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.031093+0000) 2019-10-13T16:53:50.040 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:50.034+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255277 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:50.297 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:50.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.295721+0000) 2019-10-13T16:53:50.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:50.544+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:50.545130+0000) 2019-10-13T16:53:52.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:52.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:52.812587+0000) 2019-10-13T16:53:55.043 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:53:55.036+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.041796+0000) 2019-10-13T16:53:55.044 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:55.038+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.042568+0000) 2019-10-13T16:53:55.045 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:53:55.038+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.043243+0000) 2019-10-13T16:53:55.045 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:55.042+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.043629+0000) 2019-10-13T16:53:55.046 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:55.042+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.045008+0000) 2019-10-13T16:53:55.047 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:55.042+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.045124+0000) 2019-10-13T16:53:55.047 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:53:55.042+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.046019+0000) 2019-10-13T16:53:55.054 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:55.048+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.052060+0000) 2019-10-13T16:53:55.057 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:53:55.054+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255307 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:53:55.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:53:55.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.824792+0000) 2019-10-13T16:53:55.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:53:55.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.824799+0000) 2019-10-13T16:53:55.834 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:53:55.830+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:55.833031+0000) 2019-10-13T16:53:56.377 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:53:56.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:56.375543+0000) 2019-10-13T16:53:58.312 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:53:58.304+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:53:58.309852+0000) 2019-10-13T16:54:00.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:00.054+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.058411+0000) 2019-10-13T16:54:00.061 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:00.054+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.059191+0000) 2019-10-13T16:54:00.061 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:00.058+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.059721+0000) 2019-10-13T16:54:00.063 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:00.056+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.061151+0000) 2019-10-13T16:54:00.063 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:00.058+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.061452+0000) 2019-10-13T16:54:00.064 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:00.058+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.061985+0000) 2019-10-13T16:54:00.065 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:00.058+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.062805+0000) 2019-10-13T16:54:00.069 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:00.064+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.067353+0000) 2019-10-13T16:54:00.073 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:00.070+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255343 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:00.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:00.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.295898+0000) 2019-10-13T16:54:00.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:00.540+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:00.545302+0000) 2019-10-13T16:54:01.132 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:54:01.133 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:54:01.136 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:54:01.137 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:54:02.814 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:02.810+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:02.812787+0000) 2019-10-13T16:54:05.076 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:05.070+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.074774+0000) 2019-10-13T16:54:05.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:05.074+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.075981+0000) 2019-10-13T16:54:05.078 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:05.074+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.076420+0000) 2019-10-13T16:54:05.078 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:05.071+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.076765+0000) 2019-10-13T16:54:05.079 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:05.074+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.077174+0000) 2019-10-13T16:54:05.080 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:05.074+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.078078+0000) 2019-10-13T16:54:05.080 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:05.074+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.078663+0000) 2019-10-13T16:54:05.086 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:05.079+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.083864+0000) 2019-10-13T16:54:05.090 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:05.086+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255373 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:05.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:05.822+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.824975+0000) 2019-10-13T16:54:05.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:05.822+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.824994+0000) 2019-10-13T16:54:05.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:05.830+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:05.833207+0000) 2019-10-13T16:54:06.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:06.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:06.375784+0000) 2019-10-13T16:54:08.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:08.315+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:08.318176+0000) 2019-10-13T16:54:10.093 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:10.085+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.090709+0000) 2019-10-13T16:54:10.094 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:10.089+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.091794+0000) 2019-10-13T16:54:10.094 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:10.089+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.092414+0000) 2019-10-13T16:54:10.095 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:10.089+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.093502+0000) 2019-10-13T16:54:10.095 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:10.087+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.093536+0000) 2019-10-13T16:54:10.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:10.089+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.094190+0000) 2019-10-13T16:54:10.096 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:10.089+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.094860+0000) 2019-10-13T16:54:10.101 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:10.095+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.099256+0000) 2019-10-13T16:54:10.106 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:10.101+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255411 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:10.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:10.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.296076+0000) 2019-10-13T16:54:10.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:10.543+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:10.545475+0000) 2019-10-13T16:54:12.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:12.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:12.812985+0000) 2019-10-13T16:54:15.109 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:15.101+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.107202+0000) 2019-10-13T16:54:15.109 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:15.105+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.107863+0000) 2019-10-13T16:54:15.110 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:15.105+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.109146+0000) 2019-10-13T16:54:15.111 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:15.105+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.109609+0000) 2019-10-13T16:54:15.111 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:15.103+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.109699+0000) 2019-10-13T16:54:15.112 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:15.105+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.110202+0000) 2019-10-13T16:54:15.112 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:15.105+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.110418+0000) 2019-10-13T16:54:15.116 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:15.111+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.114296+0000) 2019-10-13T16:54:15.120 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:15.113+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255441 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:15.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:15.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.825154+0000) 2019-10-13T16:54:15.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:15.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.825177+0000) 2019-10-13T16:54:15.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:15.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:15.833417+0000) 2019-10-13T16:54:16.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:16.373+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:16.376006+0000) 2019-10-13T16:54:18.326 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:18.323+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:18.324685+0000) 2019-10-13T16:54:20.125 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:20.117+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.123113+0000) 2019-10-13T16:54:20.125 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:20.121+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.123781+0000) 2019-10-13T16:54:20.126 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:20.119+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.124028+0000) 2019-10-13T16:54:20.126 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:20.121+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.124690+0000) 2019-10-13T16:54:20.127 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:20.121+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.125145+0000) 2019-10-13T16:54:20.127 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:20.121+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.125518+0000) 2019-10-13T16:54:20.127 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:20.121+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.125855+0000) 2019-10-13T16:54:20.135 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:20.127+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.133274+0000) 2019-10-13T16:54:20.136 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:20.129+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:20.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:20.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.296285+0000) 2019-10-13T16:54:20.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:20.539+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:20.545647+0000) 2019-10-13T16:54:22.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:22.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:22.813209+0000) 2019-10-13T16:54:25.140 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:25.133+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.138107+0000) 2019-10-13T16:54:25.140 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:25.133+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.138503+0000) 2019-10-13T16:54:25.141 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:25.133+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.139282+0000) 2019-10-13T16:54:25.141 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:25.137+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.139897+0000) 2019-10-13T16:54:25.142 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:25.135+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.140233+0000) 2019-10-13T16:54:25.142 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:25.137+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.140098+0000) 2019-10-13T16:54:25.143 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:25.137+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.141068+0000) 2019-10-13T16:54:25.149 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:25.143+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.147815+0000) 2019-10-13T16:54:25.153 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:25.149+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255507 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:25.827 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:25.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.825349+0000) 2019-10-13T16:54:25.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:25.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.825345+0000) 2019-10-13T16:54:25.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:25.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:25.833627+0000) 2019-10-13T16:54:26.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:26.373+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:26.376217+0000) 2019-10-13T16:54:28.333 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:28.327+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:28.331112+0000) 2019-10-13T16:54:30.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:30.149+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.154832+0000) 2019-10-13T16:54:30.157 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:30.149+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.155209+0000) 2019-10-13T16:54:30.157 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:30.153+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.155864+0000) 2019-10-13T16:54:30.158 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:30.153+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.156064+0000) 2019-10-13T16:54:30.158 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:30.153+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.156596+0000) 2019-10-13T16:54:30.159 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:30.153+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.158002+0000) 2019-10-13T16:54:30.160 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:30.155+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.158587+0000) 2019-10-13T16:54:30.166 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:30.159+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.164186+0000) 2019-10-13T16:54:30.169 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:30.165+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255545 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:30.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:30.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.296462+0000) 2019-10-13T16:54:30.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:30.543+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:30.545851+0000) 2019-10-13T16:54:31.344 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:54:31.345 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:54:31.348 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:54:31.348 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:54:32.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:32.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:32.813419+0000) 2019-10-13T16:54:35.183 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:35.169+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.172049+0000) 2019-10-13T16:54:35.183 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:35.169+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.172401+0000) 2019-10-13T16:54:35.184 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:35.169+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.172787+0000) 2019-10-13T16:54:35.184 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:35.169+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.173251+0000) 2019-10-13T16:54:35.185 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:35.169+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.173670+0000) 2019-10-13T16:54:35.186 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:35.167+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.173919+0000) 2019-10-13T16:54:35.186 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:35.169+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.174530+0000) 2019-10-13T16:54:35.188 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:35.171+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.177866+0000) 2019-10-13T16:54:35.196 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:35.181+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255575 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:35.835 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:35.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.825544+0000) 2019-10-13T16:54:35.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:35.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.825548+0000) 2019-10-13T16:54:35.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:35.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:35.833805+0000) 2019-10-13T16:54:36.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:36.373+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:36.376420+0000) 2019-10-13T16:54:38.339 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:38.335+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:38.337616+0000) 2019-10-13T16:54:40.189 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:40.185+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.187925+0000) 2019-10-13T16:54:40.190 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:40.185+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.188364+0000) 2019-10-13T16:54:40.190 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:40.185+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.189010+0000) 2019-10-13T16:54:40.191 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:40.185+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.189373+0000) 2019-10-13T16:54:40.191 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:40.183+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.189673+0000) 2019-10-13T16:54:40.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:40.185+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.190222+0000) 2019-10-13T16:54:40.192 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:40.185+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.190715+0000) 2019-10-13T16:54:40.199 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:40.191+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.197283+0000) 2019-10-13T16:54:40.201 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:40.197+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255611 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:40.298 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:40.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.296653+0000) 2019-10-13T16:54:40.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:40.539+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:40.546048+0000) 2019-10-13T16:54:42.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:42.809+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:42.813611+0000) 2019-10-13T16:54:45.204 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:45.197+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.202820+0000) 2019-10-13T16:54:45.205 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:45.201+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.203414+0000) 2019-10-13T16:54:45.205 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:45.201+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.203627+0000) 2019-10-13T16:54:45.206 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:45.201+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.203905+0000) 2019-10-13T16:54:45.206 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:45.199+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.204314+0000) 2019-10-13T16:54:45.207 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:45.201+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.204779+0000) 2019-10-13T16:54:45.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:45.207+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.213586+0000) 2019-10-13T16:54:45.221 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:45.201+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.205825+0000) 2019-10-13T16:54:45.221 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:45.213+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255641 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:45.827 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:45.821+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.825717+0000) 2019-10-13T16:54:45.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:45.821+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.825748+0000) 2019-10-13T16:54:45.835 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:45.829+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:45.833997+0000) 2019-10-13T16:54:46.378 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:46.373+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:46.376626+0000) 2019-10-13T16:54:48.346 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:48.343+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:48.344100+0000) 2019-10-13T16:54:50.220 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:50.213+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.218563+0000) 2019-10-13T16:54:50.221 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:50.217+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.219594+0000) 2019-10-13T16:54:50.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:50.217+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.220056+0000) 2019-10-13T16:54:50.222 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:50.217+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.220430+0000) 2019-10-13T16:54:50.223 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:50.217+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.221270+0000) 2019-10-13T16:54:50.223 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:50.215+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.221621+0000) 2019-10-13T16:54:50.224 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:50.217+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.222395+0000) 2019-10-13T16:54:50.227 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:50.219+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.225835+0000) 2019-10-13T16:54:50.233 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:50.229+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255679 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:50.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:50.293+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.296840+0000) 2019-10-13T16:54:50.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:50.543+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:50.546217+0000) 2019-10-13T16:54:52.815 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:52.813+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:52.813813+0000) 2019-10-13T16:54:55.236 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:54:55.228+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.234554+0000) 2019-10-13T16:54:55.237 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:54:55.228+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.235077+0000) 2019-10-13T16:54:55.238 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:54:55.231+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.235915+0000) 2019-10-13T16:54:55.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:55.232+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.236306+0000) 2019-10-13T16:54:55.238 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:55.232+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.236562+0000) 2019-10-13T16:54:55.239 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:55.232+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.237835+0000) 2019-10-13T16:54:55.240 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:55.232+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.238269+0000) 2019-10-13T16:54:55.245 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:55.239+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.243680+0000) 2019-10-13T16:54:55.249 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:54:55.244+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255709 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:54:55.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:54:55.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.825937+0000) 2019-10-13T16:54:55.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:54:55.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.825927+0000) 2019-10-13T16:54:55.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:54:55.828+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:55.834125+0000) 2019-10-13T16:54:56.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:54:56.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:56.376850+0000) 2019-10-13T16:54:58.353 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:54:58.347+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:54:58.350603+0000) 2019-10-13T16:55:00.252 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:00.244+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.250590+0000) 2019-10-13T16:55:00.253 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:00.246+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.250666+0000) 2019-10-13T16:55:00.253 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:00.244+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.251166+0000) 2019-10-13T16:55:00.254 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:00.248+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.251936+0000) 2019-10-13T16:55:00.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:00.248+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.252935+0000) 2019-10-13T16:55:00.255 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:00.248+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.253843+0000) 2019-10-13T16:55:00.256 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:00.248+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.254574+0000) 2019-10-13T16:55:00.263 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:00.254+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.261685+0000) 2019-10-13T16:55:00.265 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:00.260+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255742 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:00.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:00.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.297015+0000) 2019-10-13T16:55:00.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:00.542+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:00.546392+0000) 2019-10-13T16:55:01.388 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:55:01.388 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:55:01.392 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:55:01.392 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:55:02.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:02.812+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:02.814014+0000) 2019-10-13T16:55:05.269 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:05.262+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.267339+0000) 2019-10-13T16:55:05.270 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:05.264+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.267932+0000) 2019-10-13T16:55:05.270 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:05.264+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.268266+0000) 2019-10-13T16:55:05.271 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:05.264+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.268518+0000) 2019-10-13T16:55:05.271 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:05.264+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.268965+0000) 2019-10-13T16:55:05.271 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:05.264+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.269343+0000) 2019-10-13T16:55:05.272 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:05.264+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.269905+0000) 2019-10-13T16:55:05.281 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:05.266+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.273562+0000) 2019-10-13T16:55:05.292 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:05.276+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255772 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:05.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:05.820+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.826106+0000) 2019-10-13T16:55:05.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:05.820+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.826105+0000) 2019-10-13T16:55:05.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:05.828+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:05.834298+0000) 2019-10-13T16:55:06.386 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:06.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:06.377056+0000) 2019-10-13T16:55:08.359 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:08.354+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:08.357132+0000) 2019-10-13T16:55:10.286 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:10.280+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.283719+0000) 2019-10-13T16:55:10.286 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:10.280+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.284065+0000) 2019-10-13T16:55:10.287 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:10.278+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.284753+0000) 2019-10-13T16:55:10.287 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:10.280+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.284613+0000) 2019-10-13T16:55:10.287 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:10.280+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.285323+0000) 2019-10-13T16:55:10.287 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:10.280+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.285667+0000) 2019-10-13T16:55:10.288 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:10.280+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.286281+0000) 2019-10-13T16:55:10.298 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:10.292+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255807 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:10.298 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:10.290+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.296147+0000) 2019-10-13T16:55:10.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:10.292+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.297151+0000) 2019-10-13T16:55:10.548 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:10.542+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:10.546564+0000) 2019-10-13T16:55:12.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:12.812+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:12.814221+0000) 2019-10-13T16:55:15.305 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:15.296+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.299884+0000) 2019-10-13T16:55:15.306 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:15.296+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.300177+0000) 2019-10-13T16:55:15.306 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:15.296+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.300949+0000) 2019-10-13T16:55:15.306 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:15.296+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.301268+0000) 2019-10-13T16:55:15.306 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:15.296+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.301969+0000) 2019-10-13T16:55:15.307 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:15.296+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.302464+0000) 2019-10-13T16:55:15.308 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:15.294+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.300408+0000) 2019-10-13T16:55:15.308 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:15.298+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.304485+0000) 2019-10-13T16:55:15.312 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:15.304+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255837 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:15.828 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:15.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.826282+0000) 2019-10-13T16:55:15.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:15.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.826308+0000) 2019-10-13T16:55:15.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:15.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:15.834470+0000) 2019-10-13T16:55:16.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:16.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:16.377278+0000) 2019-10-13T16:55:18.365 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:18.358+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:18.363601+0000) 2019-10-13T16:55:20.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:20.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.297326+0000) 2019-10-13T16:55:20.315 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:20.308+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.313680+0000) 2019-10-13T16:55:20.316 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:20.308+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.313940+0000) 2019-10-13T16:55:20.316 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:20.308+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.314740+0000) 2019-10-13T16:55:20.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:20.312+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.315603+0000) 2019-10-13T16:55:20.318 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:20.310+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.316514+0000) 2019-10-13T16:55:20.319 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:20.312+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.317437+0000) 2019-10-13T16:55:20.319 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:20.312+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.317839+0000) 2019-10-13T16:55:20.324 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:20.318+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.322676+0000) 2019-10-13T16:55:20.329 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:20.324+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255871 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:20.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:20.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:20.546759+0000) 2019-10-13T16:55:22.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:22.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:22.814441+0000) 2019-10-13T16:55:25.333 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:25.324+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.330852+0000) 2019-10-13T16:55:25.333 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:25.328+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.331380+0000) 2019-10-13T16:55:25.333 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:25.328+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.332058+0000) 2019-10-13T16:55:25.334 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:25.328+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.332409+0000) 2019-10-13T16:55:25.336 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:25.328+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.333818+0000) 2019-10-13T16:55:25.336 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:25.328+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.333396+0000) 2019-10-13T16:55:25.336 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:25.326+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.333955+0000) 2019-10-13T16:55:25.341 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:25.334+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.339114+0000) 2019-10-13T16:55:25.346 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:25.340+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255901 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:25.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:25.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.826510+0000) 2019-10-13T16:55:25.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:25.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.826506+0000) 2019-10-13T16:55:25.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:25.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:25.834647+0000) 2019-10-13T16:55:26.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:26.376+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:26.377477+0000) 2019-10-13T16:55:28.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:28.366+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:28.370098+0000) 2019-10-13T16:55:30.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:30.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.297528+0000) 2019-10-13T16:55:30.349 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:30.340+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.346964+0000) 2019-10-13T16:55:30.350 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:30.344+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.348175+0000) 2019-10-13T16:55:30.350 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:30.344+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.348699+0000) 2019-10-13T16:55:30.351 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:30.344+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.349370+0000) 2019-10-13T16:55:30.351 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:30.344+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.349353+0000) 2019-10-13T16:55:30.352 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:30.344+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.349963+0000) 2019-10-13T16:55:30.353 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:30.346+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.351160+0000) 2019-10-13T16:55:30.357 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:30.350+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.355830+0000) 2019-10-13T16:55:30.362 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:30.356+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255935 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:30.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:30.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:30.546952+0000) 2019-10-13T16:55:31.544 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:55:31.544 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:55:31.550 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:55:31.550 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:55:32.816 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:32.808+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:32.814653+0000) 2019-10-13T16:55:35.366 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:35.360+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.364364+0000) 2019-10-13T16:55:35.367 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:35.360+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.364450+0000) 2019-10-13T16:55:35.367 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:35.360+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.364738+0000) 2019-10-13T16:55:35.367 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:35.358+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.365478+0000) 2019-10-13T16:55:35.368 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:35.360+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.366286+0000) 2019-10-13T16:55:35.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:35.360+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.366644+0000) 2019-10-13T16:55:35.368 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:35.360+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.366951+0000) 2019-10-13T16:55:35.374 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:35.366+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.372104+0000) 2019-10-13T16:55:35.378 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:35.372+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255965 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:35.828 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:35.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.826696+0000) 2019-10-13T16:55:35.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:35.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.826698+0000) 2019-10-13T16:55:35.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:35.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:35.834855+0000) 2019-10-13T16:55:36.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:36.372+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:36.377679+0000) 2019-10-13T16:55:38.379 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:38.374+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:38.376624+0000) 2019-10-13T16:55:40.299 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:40.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.297777+0000) 2019-10-13T16:55:40.382 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:40.375+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.379912+0000) 2019-10-13T16:55:40.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:40.375+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.380715+0000) 2019-10-13T16:55:40.383 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:40.375+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.381150+0000) 2019-10-13T16:55:40.383 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:40.375+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.381445+0000) 2019-10-13T16:55:40.384 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:40.378+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.382448+0000) 2019-10-13T16:55:40.385 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:40.375+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.382948+0000) 2019-10-13T16:55:40.385 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:40.375+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.383184+0000) 2019-10-13T16:55:40.390 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:40.382+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.388095+0000) 2019-10-13T16:55:40.395 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:40.387+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 255998 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:40.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:40.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:40.547149+0000) 2019-10-13T16:55:42.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:42.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:42.814862+0000) 2019-10-13T16:55:45.399 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:45.391+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.396773+0000) 2019-10-13T16:55:45.399 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:45.391+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.397654+0000) 2019-10-13T16:55:45.400 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:45.391+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.398310+0000) 2019-10-13T16:55:45.400 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:45.398+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.398266+0000) 2019-10-13T16:55:45.401 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:45.395+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.399800+0000) 2019-10-13T16:55:45.402 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:45.395+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.400270+0000) 2019-10-13T16:55:45.403 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:45.395+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.401055+0000) 2019-10-13T16:55:45.407 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:45.398+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.405286+0000) 2019-10-13T16:55:45.411 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:45.403+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256029 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:45.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:45.823+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.826875+0000) 2019-10-13T16:55:45.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:45.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.826876+0000) 2019-10-13T16:55:45.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:45.831+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:45.835033+0000) 2019-10-13T16:55:46.379 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:46.371+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:46.377906+0000) 2019-10-13T16:55:48.385 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:48.382+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:48.383134+0000) 2019-10-13T16:55:50.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:50.291+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.298032+0000) 2019-10-13T16:55:50.416 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:50.407+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.414328+0000) 2019-10-13T16:55:50.417 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:50.407+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.414453+0000) 2019-10-13T16:55:50.417 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:50.407+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.414970+0000) 2019-10-13T16:55:50.418 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:50.407+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.415176+0000) 2019-10-13T16:55:50.418 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:50.411+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.415916+0000) 2019-10-13T16:55:50.418 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:50.414+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.416012+0000) 2019-10-13T16:55:50.420 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:50.411+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.417821+0000) 2019-10-13T16:55:50.427 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:50.422+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.424976+0000) 2019-10-13T16:55:50.428 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:50.419+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256063 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:50.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:50.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:50.547314+0000) 2019-10-13T16:55:52.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:52.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:52.815064+0000) 2019-10-13T16:55:55.432 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:55:55.423+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.430161+0000) 2019-10-13T16:55:55.432 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:55.423+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.430505+0000) 2019-10-13T16:55:55.433 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:55:55.423+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.431183+0000) 2019-10-13T16:55:55.433 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:55.427+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.431452+0000) 2019-10-13T16:55:55.433 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:55.427+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.431605+0000) 2019-10-13T16:55:55.434 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:55:55.429+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.432063+0000) 2019-10-13T16:55:55.435 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:55.427+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.433817+0000) 2019-10-13T16:55:55.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:55.437+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.438995+0000) 2019-10-13T16:55:55.445 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:55:55.435+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256093 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:55:55.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:55:55.819+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.827080+0000) 2019-10-13T16:55:55.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:55:55.819+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.827086+0000) 2019-10-13T16:55:55.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:55:55.827+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:55.835202+0000) 2019-10-13T16:55:56.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:55:56.371+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:56.378088+0000) 2019-10-13T16:55:58.391 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:55:58.385+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:55:58.389630+0000) 2019-10-13T16:56:00.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:00.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.298245+0000) 2019-10-13T16:56:00.448 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:00.439+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.446414+0000) 2019-10-13T16:56:00.449 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:00.445+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.446680+0000) 2019-10-13T16:56:00.449 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:00.443+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.447661+0000) 2019-10-13T16:56:00.450 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:00.443+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.448003+0000) 2019-10-13T16:56:00.450 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:00.443+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.448671+0000) 2019-10-13T16:56:00.450 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:00.443+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.448466+0000) 2019-10-13T16:56:00.451 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:00.443+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.449033+0000) 2019-10-13T16:56:00.458 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:00.453+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.456756+0000) 2019-10-13T16:56:00.461 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:00.455+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256126 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:00.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:00.545+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:00.547473+0000) 2019-10-13T16:56:01.599 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:56:01.599 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:56:01.603 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:56:01.603 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:56:02.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:02.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:02.815324+0000) 2019-10-13T16:56:05.464 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:05.461+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.462647+0000) 2019-10-13T16:56:05.465 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:05.459+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.463595+0000) 2019-10-13T16:56:05.465 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:05.459+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.463987+0000) 2019-10-13T16:56:05.466 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:05.459+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.464914+0000) 2019-10-13T16:56:05.468 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:05.459+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.465954+0000) 2019-10-13T16:56:05.468 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:05.459+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.466002+0000) 2019-10-13T16:56:05.468 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:05.459+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.466197+0000) 2019-10-13T16:56:05.472 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:05.469+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.470690+0000) 2019-10-13T16:56:05.477 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:05.471+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256157 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:05.829 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:05.823+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.827338+0000) 2019-10-13T16:56:05.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:05.823+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.827288+0000) 2019-10-13T16:56:05.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:05.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:05.835413+0000) 2019-10-13T16:56:06.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:06.375+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:06.378270+0000) 2019-10-13T16:56:08.401 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:08.393+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:08.396043+0000) 2019-10-13T16:56:10.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:10.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.298457+0000) 2019-10-13T16:56:10.481 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:10.477+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.478830+0000) 2019-10-13T16:56:10.481 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:10.479+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.479928+0000) 2019-10-13T16:56:10.482 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:10.479+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.480452+0000) 2019-10-13T16:56:10.483 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:10.479+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.481059+0000) 2019-10-13T16:56:10.483 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:10.479+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.481763+0000) 2019-10-13T16:56:10.483 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:10.479+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.482167+0000) 2019-10-13T16:56:10.484 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:10.479+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.482766+0000) 2019-10-13T16:56:10.490 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:10.485+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.488734+0000) 2019-10-13T16:56:10.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:10.491+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256191 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:10.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:10.545+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:10.547650+0000) 2019-10-13T16:56:12.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:12.815+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:12.815494+0000) 2019-10-13T16:56:15.497 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:15.493+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.495079+0000) 2019-10-13T16:56:15.497 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:15.495+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.495694+0000) 2019-10-13T16:56:15.498 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:15.495+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.496809+0000) 2019-10-13T16:56:15.500 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:15.495+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.497424+0000) 2019-10-13T16:56:15.500 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:15.495+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.497339+0000) 2019-10-13T16:56:15.500 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:15.495+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.497771+0000) 2019-10-13T16:56:15.501 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:15.495+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.498469+0000) 2019-10-13T16:56:15.507 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:15.501+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.505448+0000) 2019-10-13T16:56:15.509 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:15.507+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256221 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:15.829 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:15.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.827496+0000) 2019-10-13T16:56:15.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:15.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.827557+0000) 2019-10-13T16:56:15.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:15.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:15.835571+0000) 2019-10-13T16:56:16.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:16.375+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:16.378450+0000) 2019-10-13T16:56:18.404 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:18.401+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:18.402516+0000) 2019-10-13T16:56:20.300 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:20.295+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.298661+0000) 2019-10-13T16:56:20.512 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:20.505+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.509869+0000) 2019-10-13T16:56:20.513 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:20.511+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.511694+0000) 2019-10-13T16:56:20.514 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:20.511+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.512402+0000) 2019-10-13T16:56:20.514 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:20.511+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.513139+0000) 2019-10-13T16:56:20.515 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:20.511+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.513667+0000) 2019-10-13T16:56:20.515 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:20.511+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.514162+0000) 2019-10-13T16:56:20.516 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:20.511+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.514467+0000) 2019-10-13T16:56:20.520 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:20.517+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.518222+0000) 2019-10-13T16:56:20.525 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:20.523+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256253 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:20.549 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:20.545+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:20.547767+0000) 2019-10-13T16:56:22.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:22.814+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:22.815717+0000) 2019-10-13T16:56:25.529 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:25.522+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.526861+0000) 2019-10-13T16:56:25.529 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:25.526+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.527460+0000) 2019-10-13T16:56:25.529 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:25.526+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.528066+0000) 2019-10-13T16:56:25.530 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:25.526+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.528771+0000) 2019-10-13T16:56:25.531 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:25.526+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.529179+0000) 2019-10-13T16:56:25.531 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:25.525+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.529328+0000) 2019-10-13T16:56:25.532 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:25.526+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.530241+0000) 2019-10-13T16:56:25.537 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:25.533+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.535840+0000) 2019-10-13T16:56:25.542 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:25.538+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256285 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:25.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:25.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.827711+0000) 2019-10-13T16:56:25.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:25.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.827742+0000) 2019-10-13T16:56:25.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:25.834+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:25.835765+0000) 2019-10-13T16:56:26.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:26.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:26.378624+0000) 2019-10-13T16:56:28.411 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:28.405+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:28.408770+0000) 2019-10-13T16:56:30.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:30.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.298851+0000) 2019-10-13T16:56:30.545 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:30.542+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.543457+0000) 2019-10-13T16:56:30.546 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:30.542+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.544422+0000) 2019-10-13T16:56:30.547 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:30.541+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.544732+0000) 2019-10-13T16:56:30.547 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:30.542+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.545578+0000) 2019-10-13T16:56:30.547 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:30.542+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.545616+0000) 2019-10-13T16:56:30.548 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:30.542+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.546313+0000) 2019-10-13T16:56:30.549 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:30.542+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.547191+0000) 2019-10-13T16:56:30.550 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:30.545+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.547917+0000) 2019-10-13T16:56:30.558 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:30.554+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256318 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:30.559 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:30.553+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:30.556623+0000) 2019-10-13T16:56:31.680 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:56:31.681 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:56:31.685 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:56:31.685 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:56:32.817 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:32.814+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:32.815910+0000) 2019-10-13T16:56:35.562 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:35.558+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.560518+0000) 2019-10-13T16:56:35.563 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:35.558+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.561258+0000) 2019-10-13T16:56:35.563 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:35.558+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.561507+0000) 2019-10-13T16:56:35.563 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:35.557+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.561640+0000) 2019-10-13T16:56:35.564 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:35.558+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.562246+0000) 2019-10-13T16:56:35.568 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:35.558+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.562975+0000) 2019-10-13T16:56:35.568 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:35.558+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.562390+0000) 2019-10-13T16:56:35.569 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:35.565+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.567265+0000) 2019-10-13T16:56:35.573 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:35.570+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:35.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:35.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.827922+0000) 2019-10-13T16:56:35.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:35.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.827921+0000) 2019-10-13T16:56:35.837 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:35.834+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:35.835949+0000) 2019-10-13T16:56:36.380 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:36.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:36.378805+0000) 2019-10-13T16:56:38.417 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:38.413+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:38.415314+0000) 2019-10-13T16:56:40.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:40.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.299046+0000) 2019-10-13T16:56:40.550 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:40.545+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.548088+0000) 2019-10-13T16:56:40.577 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:40.574+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.575297+0000) 2019-10-13T16:56:40.577 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:40.574+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.575794+0000) 2019-10-13T16:56:40.578 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:40.574+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.576158+0000) 2019-10-13T16:56:40.579 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:40.574+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.577140+0000) 2019-10-13T16:56:40.579 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:40.573+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.577386+0000) 2019-10-13T16:56:40.580 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:40.574+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.577849+0000) 2019-10-13T16:56:40.580 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:40.574+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.578435+0000) 2019-10-13T16:56:40.585 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:40.581+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:40.583836+0000) 2019-10-13T16:56:40.589 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:40.586+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256381 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:42.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:42.814+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:42.816125+0000) 2019-10-13T16:56:45.593 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:45.586+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.591279+0000) 2019-10-13T16:56:45.594 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:45.589+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.591285+0000) 2019-10-13T16:56:45.594 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:45.590+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.591758+0000) 2019-10-13T16:56:45.594 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:45.590+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.592188+0000) 2019-10-13T16:56:45.595 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:45.590+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.592553+0000) 2019-10-13T16:56:45.595 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:45.590+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.592779+0000) 2019-10-13T16:56:45.595 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:45.590+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.593996+0000) 2019-10-13T16:56:45.601 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:45.597+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.599488+0000) 2019-10-13T16:56:45.604 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:45.598+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:45.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:45.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.828124+0000) 2019-10-13T16:56:45.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:45.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.828124+0000) 2019-10-13T16:56:45.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:45.834+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:45.836151+0000) 2019-10-13T16:56:46.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:46.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:46.378982+0000) 2019-10-13T16:56:48.424 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:48.417+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:48.421824+0000) 2019-10-13T16:56:50.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:50.294+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.299269+0000) 2019-10-13T16:56:50.550 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:50.544+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.548282+0000) 2019-10-13T16:56:50.608 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:50.602+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.606315+0000) 2019-10-13T16:56:50.608 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:50.602+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.606545+0000) 2019-10-13T16:56:50.609 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:50.602+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.606824+0000) 2019-10-13T16:56:50.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:50.606+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.607645+0000) 2019-10-13T16:56:50.610 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:50.604+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.608430+0000) 2019-10-13T16:56:50.611 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:50.606+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.608610+0000) 2019-10-13T16:56:50.611 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:50.606+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.608830+0000) 2019-10-13T16:56:50.617 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:50.612+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:50.615230+0000) 2019-10-13T16:56:50.620 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:50.614+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256446 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:52.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:52.814+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:52.816286+0000) 2019-10-13T16:56:55.624 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:55.618+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.622502+0000) 2019-10-13T16:56:55.625 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:56:55.618+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.622588+0000) 2019-10-13T16:56:55.625 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:55.622+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.623650+0000) 2019-10-13T16:56:55.626 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:56:55.622+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.624147+0000) 2019-10-13T16:56:55.626 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:56:55.620+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.624575+0000) 2019-10-13T16:56:55.627 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:55.622+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.625406+0000) 2019-10-13T16:56:55.627 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:55.622+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.625735+0000) 2019-10-13T16:56:55.632 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:55.628+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.630798+0000) 2019-10-13T16:56:55.637 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:56:55.630+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256477 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:56:55.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:56:55.826+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.828327+0000) 2019-10-13T16:56:55.830 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:56:55.826+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.828323+0000) 2019-10-13T16:56:55.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:56:55.834+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:55.836322+0000) 2019-10-13T16:56:56.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:56:56.374+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:56.379194+0000) 2019-10-13T16:56:58.430 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:56:58.424+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:56:58.428338+0000) 2019-10-13T16:57:00.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:00.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.299494+0000) 2019-10-13T16:57:00.550 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:00.544+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.548474+0000) 2019-10-13T16:57:00.641 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:00.638+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.639436+0000) 2019-10-13T16:57:00.642 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:00.638+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.639935+0000) 2019-10-13T16:57:00.642 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:00.638+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.640334+0000) 2019-10-13T16:57:00.642 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:00.636+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.640652+0000) 2019-10-13T16:57:00.643 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:00.638+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.641383+0000) 2019-10-13T16:57:00.643 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:00.638+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.641669+0000) 2019-10-13T16:57:00.644 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:00.638+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.642090+0000) 2019-10-13T16:57:00.647 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:00.640+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:00.645326+0000) 2019-10-13T16:57:00.652 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:00.646+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256509 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:01.747 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:57:01.747 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:57:01.751 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:57:01.751 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:57:02.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:02.814+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:02.816517+0000) 2019-10-13T16:57:05.656 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:05.649+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.654056+0000) 2019-10-13T16:57:05.657 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:05.652+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.654247+0000) 2019-10-13T16:57:05.657 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:05.649+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.655248+0000) 2019-10-13T16:57:05.658 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:05.653+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.656903+0000) 2019-10-13T16:57:05.659 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:05.653+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.656560+0000) 2019-10-13T16:57:05.659 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:05.653+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.658037+0000) 2019-10-13T16:57:05.660 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:05.653+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.657546+0000) 2019-10-13T16:57:05.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:05.660+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.662649+0000) 2019-10-13T16:57:05.669 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:05.665+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256541 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:05.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:05.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.828517+0000) 2019-10-13T16:57:05.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:05.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.828511+0000) 2019-10-13T16:57:05.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:05.833+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:05.836498+0000) 2019-10-13T16:57:06.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:06.377+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:06.379383+0000) 2019-10-13T16:57:08.442 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:08.436+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:08.440113+0000) 2019-10-13T16:57:10.301 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:10.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.299716+0000) 2019-10-13T16:57:10.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:10.548+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.548669+0000) 2019-10-13T16:57:10.673 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:10.665+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.671162+0000) 2019-10-13T16:57:10.673 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:10.669+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.671826+0000) 2019-10-13T16:57:10.674 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:10.668+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.672048+0000) 2019-10-13T16:57:10.674 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:10.669+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.672995+0000) 2019-10-13T16:57:10.675 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:10.669+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.673737+0000) 2019-10-13T16:57:10.675 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:10.669+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.673844+0000) 2019-10-13T16:57:10.676 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:10.669+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.674888+0000) 2019-10-13T16:57:10.682 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:10.676+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:10.680747+0000) 2019-10-13T16:57:10.685 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:10.681+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:12.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:12.813+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:12.816731+0000) 2019-10-13T16:57:15.688 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:15.681+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.686634+0000) 2019-10-13T16:57:15.689 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:15.684+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.687159+0000) 2019-10-13T16:57:15.689 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:15.685+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.688061+0000) 2019-10-13T16:57:15.690 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:15.685+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.688316+0000) 2019-10-13T16:57:15.690 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:15.685+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.688663+0000) 2019-10-13T16:57:15.691 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:15.685+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.689007+0000) 2019-10-13T16:57:15.691 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:15.685+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.690125+0000) 2019-10-13T16:57:15.700 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:15.693+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256605 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:15.701 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:15.696+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.698978+0000) 2019-10-13T16:57:15.830 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:15.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.828651+0000) 2019-10-13T16:57:15.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:15.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.828693+0000) 2019-10-13T16:57:15.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:15.833+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:15.836679+0000) 2019-10-13T16:57:16.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:16.377+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:16.379602+0000) 2019-10-13T16:57:18.449 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:18.444+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:18.446657+0000) 2019-10-13T16:57:20.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:20.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.299927+0000) 2019-10-13T16:57:20.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:20.544+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.548862+0000) 2019-10-13T16:57:20.704 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:20.697+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.702032+0000) 2019-10-13T16:57:20.704 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:20.697+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.702553+0000) 2019-10-13T16:57:20.705 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:20.697+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.703001+0000) 2019-10-13T16:57:20.706 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:20.701+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.704327+0000) 2019-10-13T16:57:20.706 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:20.701+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.704988+0000) 2019-10-13T16:57:20.707 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:20.700+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.704888+0000) 2019-10-13T16:57:20.707 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:20.701+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.706060+0000) 2019-10-13T16:57:20.714 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:20.708+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:20.712658+0000) 2019-10-13T16:57:20.718 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:20.713+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256638 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:22.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:22.813+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:22.816932+0000) 2019-10-13T16:57:25.721 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:25.717+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.719511+0000) 2019-10-13T16:57:25.722 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:25.717+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.719904+0000) 2019-10-13T16:57:25.722 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:25.717+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.720251+0000) 2019-10-13T16:57:25.723 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:25.717+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.720657+0000) 2019-10-13T16:57:25.723 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:25.717+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.721005+0000) 2019-10-13T16:57:25.723 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:25.717+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.722007+0000) 2019-10-13T16:57:25.724 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:25.720+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.722331+0000) 2019-10-13T16:57:25.730 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:25.724+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.728390+0000) 2019-10-13T16:57:25.734 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:25.729+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256670 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:25.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:25.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.828832+0000) 2019-10-13T16:57:25.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:25.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.828865+0000) 2019-10-13T16:57:25.838 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:25.833+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:25.836854+0000) 2019-10-13T16:57:26.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:26.377+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:26.379742+0000) 2019-10-13T16:57:28.460 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:28.456+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:28.457957+0000) 2019-10-13T16:57:30.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:30.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.300122+0000) 2019-10-13T16:57:30.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:30.548+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.549058+0000) 2019-10-13T16:57:30.737 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:30.729+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.735204+0000) 2019-10-13T16:57:30.738 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:30.733+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.735562+0000) 2019-10-13T16:57:30.738 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:30.733+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.736218+0000) 2019-10-13T16:57:30.738 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:30.732+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.736565+0000) 2019-10-13T16:57:30.739 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:30.733+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.737380+0000) 2019-10-13T16:57:30.740 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:30.733+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.738514+0000) 2019-10-13T16:57:30.740 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:30.733+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.739177+0000) 2019-10-13T16:57:30.749 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:30.745+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256704 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:30.750 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:30.744+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:30.747974+0000) 2019-10-13T16:57:31.804 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:57:31.804 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:57:31.808 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:57:31.808 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:57:32.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:32.813+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:32.817147+0000) 2019-10-13T16:57:35.752 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:35.745+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.750477+0000) 2019-10-13T16:57:35.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:35.749+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.752254+0000) 2019-10-13T16:57:35.754 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:35.749+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.752609+0000) 2019-10-13T16:57:35.755 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:35.749+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.752984+0000) 2019-10-13T16:57:35.755 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:35.749+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.753626+0000) 2019-10-13T16:57:35.756 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:35.748+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.753935+0000) 2019-10-13T16:57:35.756 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:35.749+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.754524+0000) 2019-10-13T16:57:35.763 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:35.756+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.761607+0000) 2019-10-13T16:57:35.764 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:35.757+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256735 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:35.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:35.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.829061+0000) 2019-10-13T16:57:35.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:35.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.829062+0000) 2019-10-13T16:57:35.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:35.833+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:35.837028+0000) 2019-10-13T16:57:36.381 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:36.377+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:36.379932+0000) 2019-10-13T16:57:38.466 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:38.464+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:38.464347+0000) 2019-10-13T16:57:40.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:40.297+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.300315+0000) 2019-10-13T16:57:40.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:40.548+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.549267+0000) 2019-10-13T16:57:40.770 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:40.761+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.766310+0000) 2019-10-13T16:57:40.771 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:40.761+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.766921+0000) 2019-10-13T16:57:40.771 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:40.764+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.767392+0000) 2019-10-13T16:57:40.772 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:40.765+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.768203+0000) 2019-10-13T16:57:40.773 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:40.765+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.768764+0000) 2019-10-13T16:57:40.774 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:40.765+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.769144+0000) 2019-10-13T16:57:40.774 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:40.765+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.769386+0000) 2019-10-13T16:57:40.784 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:40.776+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:40.779413+0000) 2019-10-13T16:57:40.785 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:40.777+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256768 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:42.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:42.813+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:42.817352+0000) 2019-10-13T16:57:45.785 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:45.777+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.782842+0000) 2019-10-13T16:57:45.785 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:45.781+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.783386+0000) 2019-10-13T16:57:45.786 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:45.781+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.783765+0000) 2019-10-13T16:57:45.786 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:45.781+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.784154+0000) 2019-10-13T16:57:45.787 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:45.779+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.784456+0000) 2019-10-13T16:57:45.787 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:45.781+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.784587+0000) 2019-10-13T16:57:45.787 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:45.781+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.785752+0000) 2019-10-13T16:57:45.793 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:45.787+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.791535+0000) 2019-10-13T16:57:45.798 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:45.793+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256800 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:45.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:45.825+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.829239+0000) 2019-10-13T16:57:45.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:45.825+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.829246+0000) 2019-10-13T16:57:45.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:45.833+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:45.837207+0000) 2019-10-13T16:57:46.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:46.377+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:46.380149+0000) 2019-10-13T16:57:48.478 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:48.475+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:48.476161+0000) 2019-10-13T16:57:50.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:50.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.300510+0000) 2019-10-13T16:57:50.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:50.543+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.549458+0000) 2019-10-13T16:57:50.801 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:50.792+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.799272+0000) 2019-10-13T16:57:50.802 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:50.795+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.799475+0000) 2019-10-13T16:57:50.802 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:50.796+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.800060+0000) 2019-10-13T16:57:50.804 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:50.796+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.800626+0000) 2019-10-13T16:57:50.804 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:50.796+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.801247+0000) 2019-10-13T16:57:50.804 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:50.796+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.801937+0000) 2019-10-13T16:57:50.805 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:50.796+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.802186+0000) 2019-10-13T16:57:50.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:50.803+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:50.808614+0000) 2019-10-13T16:57:50.813 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:50.808+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256833 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:52.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:52.816+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:52.817555+0000) 2019-10-13T16:57:55.817 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:57:55.808+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.815161+0000) 2019-10-13T16:57:55.817 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:55.812+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.815989+0000) 2019-10-13T16:57:55.818 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:57:55.812+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.816354+0000) 2019-10-13T16:57:55.818 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:55.812+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.816651+0000) 2019-10-13T16:57:55.819 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:57:55.811+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.817015+0000) 2019-10-13T16:57:55.819 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:55.812+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.817433+0000) 2019-10-13T16:57:55.819 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:55.812+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.817896+0000) 2019-10-13T16:57:55.826 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:55.819+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.824467+0000) 2019-10-13T16:57:55.829 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:57:55.824+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256863 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:57:55.831 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:57:55.824+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.829342+0000) 2019-10-13T16:57:55.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:57:55.824+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.829465+0000) 2019-10-13T16:57:55.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:57:55.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:55.837353+0000) 2019-10-13T16:57:56.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:57:56.376+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:56.380343+0000) 2019-10-13T16:57:58.484 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:57:58.479+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:57:58.482574+0000) 2019-10-13T16:58:00.302 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:00.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.300713+0000) 2019-10-13T16:58:00.551 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:00.547+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.549651+0000) 2019-10-13T16:58:00.833 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:00.827+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.830806+0000) 2019-10-13T16:58:00.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:00.828+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.831458+0000) 2019-10-13T16:58:00.834 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:00.828+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.832099+0000) 2019-10-13T16:58:00.834 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:00.828+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.832330+0000) 2019-10-13T16:58:00.835 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:00.828+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.833979+0000) 2019-10-13T16:58:00.836 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:00.828+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.834421+0000) 2019-10-13T16:58:00.836 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:00.828+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.834858+0000) 2019-10-13T16:58:00.843 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:00.835+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:00.841636+0000) 2019-10-13T16:58:00.845 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:00.840+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256896 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:01.861 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:58:01.861 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:58:01.865 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:58:01.865 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:58:02.819 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:02.812+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:02.817758+0000) 2019-10-13T16:58:05.831 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:05.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.829650+0000) 2019-10-13T16:58:05.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:05.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.830740+0000) 2019-10-13T16:58:05.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:05.836+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.837547+0000) 2019-10-13T16:58:05.848 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:05.840+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.846608+0000) 2019-10-13T16:58:05.849 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:05.843+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.846996+0000) 2019-10-13T16:58:05.849 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:05.840+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.847053+0000) 2019-10-13T16:58:05.850 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:05.844+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.848584+0000) 2019-10-13T16:58:05.850 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:05.844+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.848891+0000) 2019-10-13T16:58:05.851 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:05.844+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.849495+0000) 2019-10-13T16:58:05.851 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:05.844+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.849893+0000) 2019-10-13T16:58:05.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:05.847+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:05.853779+0000) 2019-10-13T16:58:05.860 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:05.852+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256927 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:06.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:06.376+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:06.380538+0000) 2019-10-13T16:58:08.491 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:08.487+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:08.488927+0000) 2019-10-13T16:58:10.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:10.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.300909+0000) 2019-10-13T16:58:10.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:10.547+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.549850+0000) 2019-10-13T16:58:10.863 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:10.855+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.861445+0000) 2019-10-13T16:58:10.864 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:10.856+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.862311+0000) 2019-10-13T16:58:10.864 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:10.856+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.862969+0000) 2019-10-13T16:58:10.865 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:10.860+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.863561+0000) 2019-10-13T16:58:10.866 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:10.860+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.863294+0000) 2019-10-13T16:58:10.866 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:10.860+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.864349+0000) 2019-10-13T16:58:10.867 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:10.860+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.865236+0000) 2019-10-13T16:58:10.873 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:10.867+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:10.870957+0000) 2019-10-13T16:58:10.877 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:10.872+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256961 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:12.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:12.816+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:12.817963+0000) 2019-10-13T16:58:15.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:15.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.829892+0000) 2019-10-13T16:58:15.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:15.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.830929+0000) 2019-10-13T16:58:15.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:15.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.837696+0000) 2019-10-13T16:58:15.880 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:15.875+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.878631+0000) 2019-10-13T16:58:15.881 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:15.876+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.879582+0000) 2019-10-13T16:58:15.882 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:15.876+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.879882+0000) 2019-10-13T16:58:15.882 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:15.876+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.880722+0000) 2019-10-13T16:58:15.883 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:15.876+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.880986+0000) 2019-10-13T16:58:15.883 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:15.876+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.881812+0000) 2019-10-13T16:58:15.883 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:15.876+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.881806+0000) 2019-10-13T16:58:15.891 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:15.883+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:15.889497+0000) 2019-10-13T16:58:15.894 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:15.888+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 256991 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:16.382 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:16.380+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:16.380729+0000) 2019-10-13T16:58:18.497 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:18.491+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:18.495386+0000) 2019-10-13T16:58:20.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:20.296+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.301069+0000) 2019-10-13T16:58:20.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:20.543+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.550039+0000) 2019-10-13T16:58:20.897 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:20.891+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.895236+0000) 2019-10-13T16:58:20.898 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:20.892+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.896430+0000) 2019-10-13T16:58:20.899 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:20.892+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.897271+0000) 2019-10-13T16:58:20.899 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:20.892+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.897691+0000) 2019-10-13T16:58:20.899 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:20.892+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.898087+0000) 2019-10-13T16:58:20.900 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:20.892+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.898483+0000) 2019-10-13T16:58:20.901 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:20.896+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.899786+0000) 2019-10-13T16:58:20.903 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:20.895+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:20.901919+0000) 2019-10-13T16:58:20.910 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:20.904+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257024 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:22.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:22.816+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:22.818170+0000) 2019-10-13T16:58:25.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:25.828+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.830112+0000) 2019-10-13T16:58:25.832 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:25.828+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.831133+0000) 2019-10-13T16:58:25.839 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:25.832+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.837925+0000) 2019-10-13T16:58:25.913 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:25.908+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.911859+0000) 2019-10-13T16:58:25.914 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:25.907+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.912122+0000) 2019-10-13T16:58:25.914 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:25.908+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.912586+0000) 2019-10-13T16:58:25.915 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:25.908+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.913585+0000) 2019-10-13T16:58:25.915 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:25.908+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.914179+0000) 2019-10-13T16:58:25.916 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:25.908+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.914529+0000) 2019-10-13T16:58:25.917 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:25.908+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.915182+0000) 2019-10-13T16:58:25.923 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:25.915+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:25.920946+0000) 2019-10-13T16:58:25.926 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:25.920+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257055 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:26.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:26.380+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:26.380962+0000) 2019-10-13T16:58:28.504 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:28.499+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:28.501810+0000) 2019-10-13T16:58:30.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:30.300+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.301294+0000) 2019-10-13T16:58:30.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:30.547+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.550255+0000) 2019-10-13T16:58:30.929 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:30.924+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.927994+0000) 2019-10-13T16:58:30.930 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:30.924+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.928400+0000) 2019-10-13T16:58:30.931 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:30.923+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.929448+0000) 2019-10-13T16:58:30.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:30.924+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.929928+0000) 2019-10-13T16:58:30.932 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:30.924+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.930407+0000) 2019-10-13T16:58:30.932 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:30.924+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.931014+0000) 2019-10-13T16:58:30.933 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:30.928+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.931872+0000) 2019-10-13T16:58:30.935 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:30.927+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:30.933906+0000) 2019-10-13T16:58:30.942 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:30.936+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257089 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:31.917 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:58:31.917 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:58:31.921 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:58:31.921 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:58:32.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:32.815+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:32.818379+0000) 2019-10-13T16:58:35.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:35.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.830310+0000) 2019-10-13T16:58:35.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:35.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.831344+0000) 2019-10-13T16:58:35.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:35.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.838124+0000) 2019-10-13T16:58:35.947 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:35.939+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.944073+0000) 2019-10-13T16:58:35.948 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:35.939+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.944857+0000) 2019-10-13T16:58:35.948 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:35.939+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.945976+0000) 2019-10-13T16:58:35.949 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:35.943+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.946119+0000) 2019-10-13T16:58:35.949 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:35.939+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.946507+0000) 2019-10-13T16:58:35.949 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:35.939+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.947126+0000) 2019-10-13T16:58:35.949 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:35.943+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.947657+0000) 2019-10-13T16:58:35.955 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:35.947+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:35.953020+0000) 2019-10-13T16:58:35.958 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:35.951+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257119 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:36.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:36.379+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:36.381162+0000) 2019-10-13T16:58:38.510 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:38.507+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:38.508386+0000) 2019-10-13T16:58:40.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:40.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.301512+0000) 2019-10-13T16:58:40.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:40.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.550415+0000) 2019-10-13T16:58:40.962 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:40.955+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.960456+0000) 2019-10-13T16:58:40.963 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:40.955+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.960194+0000) 2019-10-13T16:58:40.963 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:40.955+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.960802+0000) 2019-10-13T16:58:40.963 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:40.955+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.961841+0000) 2019-10-13T16:58:40.964 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:40.955+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.962155+0000) 2019-10-13T16:58:40.964 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:40.955+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.962817+0000) 2019-10-13T16:58:40.965 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:40.958+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.963395+0000) 2019-10-13T16:58:40.972 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:40.966+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:40.970657+0000) 2019-10-13T16:58:40.973 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:40.967+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257153 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:42.825 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:42.811+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:42.818586+0000) 2019-10-13T16:58:45.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:45.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.830536+0000) 2019-10-13T16:58:45.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:45.827+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.831535+0000) 2019-10-13T16:58:45.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:45.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.838323+0000) 2019-10-13T16:58:45.976 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:45.967+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.974718+0000) 2019-10-13T16:58:45.977 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:45.967+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.975244+0000) 2019-10-13T16:58:45.978 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:45.971+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.976332+0000) 2019-10-13T16:58:45.978 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:45.971+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.976662+0000) 2019-10-13T16:58:45.980 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:45.971+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.978072+0000) 2019-10-13T16:58:45.980 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:45.970+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.977403+0000) 2019-10-13T16:58:45.980 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:45.971+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.978881+0000) 2019-10-13T16:58:45.986 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:45.978+0000 7f859e3a8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:45.984479+0000) 2019-10-13T16:58:45.989 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:45.983+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257183 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:46.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:46.379+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:46.381347+0000) 2019-10-13T16:58:48.517 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:48.514+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:48.514898+0000) 2019-10-13T16:58:50.303 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:50.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.301711+0000) 2019-10-13T16:58:50.552 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:50.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.550566+0000) 2019-10-13T16:58:50.993 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:50.987+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.991051+0000) 2019-10-13T16:58:50.993 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:50.987+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.991750+0000) 2019-10-13T16:58:50.994 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:50.987+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.992869+0000) 2019-10-13T16:58:50.995 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:50.987+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.993279+0000) 2019-10-13T16:58:50.995 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:50.987+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.993675+0000) 2019-10-13T16:58:50.995 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:50.987+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.994057+0000) 2019-10-13T16:58:50.996 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:50.990+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.994746+0000) 2019-10-13T16:58:51.000 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:50.994+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:50.998222+0000) 2019-10-13T16:58:51.006 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:50.999+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257219 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:52.820 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:52.815+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:52.818823+0000) 2019-10-13T16:58:55.832 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:55.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:55.830738+0000) 2019-10-13T16:58:55.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:55.831+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:55.831743+0000) 2019-10-13T16:58:55.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:55.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:55.838552+0000) 2019-10-13T16:58:56.009 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:58:56.003+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.007365+0000) 2019-10-13T16:58:56.010 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:58:56.003+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.007872+0000) 2019-10-13T16:58:56.010 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:58:56.003+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.008338+0000) 2019-10-13T16:58:56.011 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:56.003+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.009347+0000) 2019-10-13T16:58:56.011 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:58:56.003+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.009699+0000) 2019-10-13T16:58:56.012 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:58:56.006+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.010257+0000) 2019-10-13T16:58:56.012 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:58:56.003+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.010802+0000) 2019-10-13T16:58:56.017 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:56.010+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.014985+0000) 2019-10-13T16:58:56.022 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:58:56.015+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257248 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:58:56.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:58:56.379+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:56.381534+0000) 2019-10-13T16:58:58.523 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:58:58.518+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:58:58.521380+0000) 2019-10-13T16:59:00.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:00.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:00.301944+0000) 2019-10-13T16:59:00.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:00.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:00.550775+0000) 2019-10-13T16:59:01.025 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:01.019+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.023542+0000) 2019-10-13T16:59:01.026 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:01.018+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.024109+0000) 2019-10-13T16:59:01.026 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:01.019+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.024820+0000) 2019-10-13T16:59:01.027 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:01.019+0000 7f914def9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.025099+0000) 2019-10-13T16:59:01.028 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:01.019+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.026411+0000) 2019-10-13T16:59:01.028 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:01.019+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.026773+0000) 2019-10-13T16:59:01.028 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:01.019+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.027211+0000) 2019-10-13T16:59:01.032 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:01.026+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:01.030448+0000) 2019-10-13T16:59:01.037 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:01.031+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257283 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:01.967 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:59:01.967 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:59:01.971 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:59:01.971 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:59:02.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:02.815+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:02.819077+0000) 2019-10-13T16:59:05.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:05.827+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:05.830939+0000) 2019-10-13T16:59:05.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:05.831+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:05.831914+0000) 2019-10-13T16:59:05.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:05.835+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:05.838778+0000) 2019-10-13T16:59:06.041 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:06.035+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.038964+0000) 2019-10-13T16:59:06.041 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:06.035+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.039267+0000) 2019-10-13T16:59:06.042 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:06.034+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.039540+0000) 2019-10-13T16:59:06.042 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:06.039+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.040119+0000) 2019-10-13T16:59:06.042 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:06.039+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.040907+0000) 2019-10-13T16:59:06.043 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:06.039+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.041734+0000) 2019-10-13T16:59:06.044 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:06.039+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.042628+0000) 2019-10-13T16:59:06.049 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:06.042+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.047707+0000) 2019-10-13T16:59:06.053 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:06.051+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257314 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:06.383 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:06.379+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:06.381717+0000) 2019-10-13T16:59:08.530 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:08.526+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:08.528028+0000) 2019-10-13T16:59:10.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:10.299+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:10.302129+0000) 2019-10-13T16:59:10.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:10.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:10.550968+0000) 2019-10-13T16:59:11.057 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:11.055+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.055397+0000) 2019-10-13T16:59:11.057 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:11.055+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.055861+0000) 2019-10-13T16:59:11.058 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:11.055+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.056386+0000) 2019-10-13T16:59:11.059 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:11.050+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.056510+0000) 2019-10-13T16:59:11.059 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:11.055+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.057025+0000) 2019-10-13T16:59:11.060 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:11.055+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.058169+0000) 2019-10-13T16:59:11.060 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:11.055+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.058773+0000) 2019-10-13T16:59:11.065 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:11.058+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:11.063299+0000) 2019-10-13T16:59:11.070 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:11.067+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257349 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:12.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:12.815+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:12.819276+0000) 2019-10-13T16:59:15.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:15.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:15.831143+0000) 2019-10-13T16:59:15.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:15.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:15.832124+0000) 2019-10-13T16:59:15.840 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:15.834+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:15.838972+0000) 2019-10-13T16:59:16.074 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:16.070+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.072320+0000) 2019-10-13T16:59:16.074 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:16.070+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.072758+0000) 2019-10-13T16:59:16.075 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:16.066+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.072921+0000) 2019-10-13T16:59:16.075 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:16.070+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.073444+0000) 2019-10-13T16:59:16.076 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:16.070+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.073912+0000) 2019-10-13T16:59:16.076 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:16.070+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.074471+0000) 2019-10-13T16:59:16.077 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:16.070+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.075209+0000) 2019-10-13T16:59:16.082 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:16.074+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.080899+0000) 2019-10-13T16:59:16.085 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:16.082+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257378 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:16.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:16.378+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:16.381902+0000) 2019-10-13T16:59:18.536 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:18.534+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:18.534564+0000) 2019-10-13T16:59:20.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:20.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:20.302360+0000) 2019-10-13T16:59:20.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:20.546+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:20.551125+0000) 2019-10-13T16:59:21.089 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:21.082+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.087037+0000) 2019-10-13T16:59:21.089 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:21.086+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.087572+0000) 2019-10-13T16:59:21.090 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:21.086+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.088400+0000) 2019-10-13T16:59:21.091 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:21.086+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.089218+0000) 2019-10-13T16:59:21.091 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:21.086+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.090037+0000) 2019-10-13T16:59:21.092 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:21.086+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.090404+0000) 2019-10-13T16:59:21.092 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:21.086+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.091042+0000) 2019-10-13T16:59:21.099 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:21.090+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:21.096935+0000) 2019-10-13T16:59:21.101 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:21.098+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257413 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:22.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:22.818+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:22.819510+0000) 2019-10-13T16:59:25.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:25.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:25.831399+0000) 2019-10-13T16:59:25.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:25.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:25.832340+0000) 2019-10-13T16:59:25.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:25.838+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:25.839178+0000) 2019-10-13T16:59:26.105 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:26.098+0000 7fa847216700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.103255+0000) 2019-10-13T16:59:26.105 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:26.098+0000 7f89e6a58700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.103511+0000) 2019-10-13T16:59:26.106 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:26.102+0000 7ffb2fa3b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.104297+0000) 2019-10-13T16:59:26.106 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:26.102+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.104692+0000) 2019-10-13T16:59:26.107 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:26.102+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.105330+0000) 2019-10-13T16:59:26.107 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:26.102+0000 7f09211f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.105887+0000) 2019-10-13T16:59:26.109 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:26.102+0000 7f3ff7ca3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.107265+0000) 2019-10-13T16:59:26.111 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:26.102+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.108882+0000) 2019-10-13T16:59:26.118 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:26.114+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257444 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:26.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:26.378+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:26.382110+0000) 2019-10-13T16:59:28.543 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:28.538+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:28.541118+0000) 2019-10-13T16:59:30.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:30.298+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:30.302556+0000) 2019-10-13T16:59:30.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:30.550+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:30.551324+0000) 2019-10-13T16:59:31.121 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:31.118+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.118835+0000) 2019-10-13T16:59:31.121 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:31.118+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.119521+0000) 2019-10-13T16:59:31.122 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:31.118+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.120280+0000) 2019-10-13T16:59:31.122 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:31.118+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.121165+0000) 2019-10-13T16:59:31.123 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:31.118+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.121751+0000) 2019-10-13T16:59:31.124 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:31.118+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.122154+0000) 2019-10-13T16:59:31.124 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:31.118+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.122468+0000) 2019-10-13T16:59:31.128 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:31.122+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:31.125964+0000) 2019-10-13T16:59:31.132 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:31.126+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257479 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:32.116 INFO:teuthology.orchestra.run.smithi071:Running: 2019-10-13T16:59:32.116 INFO:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:59:32.119 INFO:teuthology.orchestra.run.smithi129:Running: 2019-10-13T16:59:32.120 INFO:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T16:59:32.821 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:32.818+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:32.819729+0000) 2019-10-13T16:59:35.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:35.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:35.831592+0000) 2019-10-13T16:59:35.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:35.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:35.832570+0000) 2019-10-13T16:59:35.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:35.838+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:35.839377+0000) 2019-10-13T16:59:36.136 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:36.130+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.134216+0000) 2019-10-13T16:59:36.136 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:36.130+0000 7fb4e9b06700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.134616+0000) 2019-10-13T16:59:36.137 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:36.133+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.135588+0000) 2019-10-13T16:59:36.138 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:36.134+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.136299+0000) 2019-10-13T16:59:36.138 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:36.134+0000 7ffb3023c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.136578+0000) 2019-10-13T16:59:36.139 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:36.134+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.137059+0000) 2019-10-13T16:59:36.139 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:36.134+0000 7fa847a17700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.137368+0000) 2019-10-13T16:59:36.142 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:36.137+0000 7f859dba7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.140336+0000) 2019-10-13T16:59:36.149 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:36.146+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257508 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:36.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:36.378+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:36.382296+0000) 2019-10-13T16:59:38.554 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:38.549+0000 7f859839c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:38.552527+0000) 2019-10-13T16:59:40.304 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:40.302+0000 7fa844210700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:40.302784+0000) 2019-10-13T16:59:40.553 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:40.549+0000 7f89e3251700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:40.551515+0000) 2019-10-13T16:59:41.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:41.146+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.151031+0000) 2019-10-13T16:59:41.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:41.150+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.151637+0000) 2019-10-13T16:59:41.154 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:41.150+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.152080+0000) 2019-10-13T16:59:41.154 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:41.149+0000 7f89e6257700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.152167+0000) 2019-10-13T16:59:41.155 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:41.150+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.152774+0000) 2019-10-13T16:59:41.155 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:41.150+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.153064+0000) 2019-10-13T16:59:41.156 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:41.150+0000 7f3ff74a2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.153912+0000) 2019-10-13T16:59:41.164 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:41.161+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:41.162467+0000) 2019-10-13T16:59:41.165 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:41.162+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257543 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:42.822 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:42.818+0000 7f0912d84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:42.819933+0000) 2019-10-13T16:59:45.833 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:45.830+0000 7ffb20dc6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:45.831793+0000) 2019-10-13T16:59:45.834 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:45.830+0000 7f913f284700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:45.832763+0000) 2019-10-13T16:59:45.841 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:45.838+0000 7fb4db692700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:45.839572+0000) 2019-10-13T16:59:46.168 INFO:tasks.ceph.mds.a.smithi129.stderr:2019-10-13T16:59:46.162+0000 7fa848218700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.166242+0000) 2019-10-13T16:59:46.169 INFO:tasks.ceph.osd.2.smithi129.stderr:2019-10-13T16:59:46.162+0000 7ffb2f23a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.166951+0000) 2019-10-13T16:59:46.169 INFO:tasks.ceph.osd.0.smithi129.stderr:2019-10-13T16:59:46.166+0000 7f914d6f8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.167700+0000) 2019-10-13T16:59:46.170 INFO:tasks.ceph.osd.3.smithi129.stderr:2019-10-13T16:59:46.166+0000 7fb4eab08700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.167983+0000) 2019-10-13T16:59:46.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:46.166+0000 7f3ff6ca1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.168618+0000) 2019-10-13T16:59:46.171 INFO:tasks.ceph.mds.b.smithi071.stderr:2019-10-13T16:59:46.165+0000 7f89e7259700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.168555+0000) 2019-10-13T16:59:46.171 INFO:tasks.ceph.osd.1.smithi129.stderr:2019-10-13T16:59:46.166+0000 7f09221fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.169061+0000) 2019-10-13T16:59:46.177 INFO:tasks.ceph.mgr.x.smithi071.stderr:2019-10-13T16:59:46.173+0000 7f859d3a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.175536+0000) 2019-10-13T16:59:46.180 INFO:tasks.ceph.mon.a.smithi129.stderr:2019-10-13T16:59:46.174+0000 7f0f07edd700 -1 mon.a@1(probing) e1 get_health_metrics reporting 257574 slow ops, oldest is osd_alive(want up_thru 24 have 24) 2019-10-13T16:59:46.384 INFO:tasks.ceph.mgr.y.smithi129.stderr:2019-10-13T16:59:46.382+0000 7f3ff1c97700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T15:59:46.382478+0000) 2019-10-13T16:59:47.423 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-13T16:59:47.423 DEBUG:teuthology.task.console_log:Killing console logger for smithi129 2019-10-13T16:59:47.424 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2019-10-13T16:59:47.424 DEBUG:teuthology.task.console_log:Killing console logger for smithi129 2019-10-13T16:59:47.424 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2019-10-13T16:59:47.425 DEBUG:teuthology.exit:Finished running handlers